Three bugs that passed every test I had
A market snapshot split across two HTTP calls, a pandas timestamp cast that assumed the wrong resolution, and an FX session boundary I was certain about — three bugs a green test suite was perfectly happy with.
My snapshot tool printed expected slippage against mid of 0.14 basis points. Mid was 2485.78, best ask 2485.80. I had already worked out, roughly, that the number should land near 0.08.
Nothing had thrown. The suite was green. The arithmetic in the code was correct. And 0.14 bps is small enough that if I hadn't had a figure in mind before reading the output, I would have accepted it and moved on.
That gap is worth naming up front, because all three of last week's real bugs lived in it. A test asserts that your code matches your expectations. It is silent on whether your expectations match the world. Two different claims. For anything touching real market data, only the second one decides whether the results mean anything — and the first one delivers a very convincing feeling of having checked.
I'm building a crypto and FX research stack from scratch: ingestion, storage, validation, point-in-time loading, as the foundation for backtesting. Here are the three, in roughly the order they got harder to see.
1. A market snapshot assembled from two HTTP calls
The snapshot command printed a quote and simulated walking the order book for a market buy. It got each from its own network call:
quote = exchange.fetch_ticker(symbol) # one moment
book = exchange.fetch_order_book(symbol) # a slightly later moment
Two calls, two moments. The market ticked between them, so the quote described one state of the book and the walk described another. The snapshot disagreed with itself, and the disagreement surfaced as that 0.06 bps discrepancy.
The fix is to stop asking twice:
book = exchange.fetch_order_book(symbol)
bid, ask = book["bids"][0][0], book["asks"][0][0]
mid = (bid + ask) / 2
One read, one consistent view of the world, every derived figure agreeing with every other.
In a live monitor the original version is a cosmetic wobble — the market really is moving, and a tenth of a basis point either way changes nothing you'd act on. As backtest input it's a correctness bug, because you're feeding the model a market state that never existed at any instant. The error is small, and it is also not real.
2. An FX session boundary I was certain about
Validating FX data means knowing the weekend calendar, so the validator can tell a legitimate session gap from a data quality problem.
I knew this one without looking it up. Forex closes Friday at 5pm US Eastern and reopens Sunday at 5pm. 21:00 UTC on both sides, symmetric.
# first version — textbook, symmetric, asserted from memory
FX_CLOSE_UTC = time(21, 0)
FX_OPEN_UTC = time(21, 0)
That version also over-excluded the whole of Sunday, including bars after the reopen. I found the Sunday bug, fixed it, and kept the 21:00 — the part I was confident about went unexamined precisely because I was confident about it.
Second attempt, still reasoning from convention rather than checking: last Friday bar at 21:00, first Sunday bar at 23:00. Asymmetric now, which was closer, and still asserted.
Then I looked at the actual bars.
# confirmed against live vendor data, not convention
FX_CLOSE_UTC = time(22, 0) # last Friday bar
FX_OPEN_UTC = time(23, 0) # first Sunday bar
Not the textbook split. Not symmetric. And — the part that actually mattered — not a fact about foreign exchange at all. A fact about this vendor's bars. Another provider may well draw the boundary somewhere else, and would be no more wrong.
The constants now carry a comment saying they were confirmed empirically, which exists mainly to stop a future version of me from "correcting" them back to the clean 21:00 that feels right.
3. A pandas timestamp cast that assumed nanoseconds
This is the one I'd have shipped.
Storing OHLCV bars means turning a pandas DatetimeIndex into millisecond epochs. The standard move:
df.index.astype("int64") // 1_000_000
That divide assumes nanosecond resolution, which is pandas' default and is true nearly all the time. The FX feed returned datetime64[s, Europe/London]. Seconds.
Dividing second-resolution integers by a million doesn't raise. It produces numbers — off by a factor of a billion, and still integers, still monotonic, still correctly ordered relative to each other, still perfectly storable in parquet. Every structural property you'd think to assert on a timestamp column survives intact. Only the values are meaningless.
df.index.as_unit("ms").astype("int64")
That's the fix, and it isn't the interesting part. How it surfaced is: by printing raw rows and comparing the epoch values against a wall clock.
No test I would have written catches this. I'd have built the fixture from the same assumption that produced the bug — nanosecond input, nanosecond expectation — and watched it go green. The test and the bug would have been two expressions of one misunderstanding, agreeing with each other enthusiastically. That's the failure mode worth internalising: a test can only be as correct as the model you wrote it from, and when the model is the thing that's wrong, the test doesn't dissent.
What actually caught all three
Three bugs, three layers of the stack, one shared property. Each produced output that looked entirely reasonable. No exceptions, no red tests, nothing a careful read of the code would surface — because in every case the code was doing exactly what I had told it to do.
And each was caught by the same instrument: a reference value that came from outside my own code. Arithmetic done in my head before reading the output. A wall clock held next to an epoch. Raw vendor rows held next to my assumption about them. Not one of them came from the test suite, and the test suite was never going to produce one, because a test built from my model can't audit my model.
For backtesting specifically the failure mode is nastier than it looks, because all three of these fail quietly and in your favour. Garbage epochs sort fine. An inconsistent snapshot is still a snapshot. A wrong session boundary just means the validator stops complaining about gaps. Nothing turns red. What you get is a model that trains and an equity curve that looks plausible, built on a market that never happened — and you find out later, with money on it.
So the rule for the rest of this build: before reading any output a decision depends on, have a number in mind that it ought to be near. Where no such number exists, go find an external reference — the wall clock, the venue's own documentation, the raw rows — and check against that rather than against another part of my own code.
Not a clever practice. Just a slower one.
I write a weekly log of what I'm building — crypto payments infrastructure, a quant research stack, and lately Solidity. Subscribe if you want the next one.