The base-rate mirage: how a 60% hit rate can be worth nothing

A 60% hit rate is the most seductive number in quantitative trading. It sits far enough above a coin flip to look like an edge, yet stays modest enough to escape the suspicion a 90% claim would attract. We spent eight months learning that on its own it carries no information — and then found a bound which says that for daily direction on crypto perpetuals, the thing we were trying to measure is about fifteen times smaller than the smallest thing our data could ever have resolved.

1. The market already did it

Crypto perpetuals move together. On a red day almost everything closes lower, so a strategy that is always short posts a magnificent hit rate on exactly those days while telling you nothing about whether it can select.

Written out, the problem is obvious. For a one-sided short strategy, accuracy decomposes into the market's move plus whatever selection adds:

accuracy = P(down | selected)
          = P(down)  +  [ P(down | selected) − P(down) ]
            ↑ the tape    ↑ everything you actually contributed

We publish a card each morning grading the previous day's flagged coins, and that is where it bit us. On 12 August 2026 the screen flagged 8 coins and 6 closed lower — 75%, the kind of number that goes in a headline. That same day 71.7% of all perpetuals on the venue closed lower. The contribution was three points, not twenty-five.

A coin flip is not the benchmark. On a day when the whole market fell, the benchmark is what a strategy that blindly shorted everything would have scored.

The benchmark is not stable either, which is what makes eyeballing it hopeless. Across 50 days of forward-recorded live calls our universe closed lower 53.4% of the time overall (n=26,541 graded coin-days) — but the daily figure ranges from 18% to 87%.

2. Your sample is 900 observations, not 117,000

Our panel holds 116,935 coin-days with a realized next-day return. That number is a trap. Because coins move together, the unit that varies independently is not the row — it is the day. Two hundred coins on the same red Tuesday are close to one observation wearing two hundred hats.

This is measurable rather than rhetorical. The intraclass correlation of the up/down label across days is 0.2634: a full 26.4% of the label's variance is a pure day effect, and 36% of days are more than 80% one-sided. Plugging that into the standard design effect for clustered samples:

DEFF  = 1 + (n̄ − 1) · ICC
      = 1 + (481.2 − 1) × 0.2634  =  127.5

n_eff = 116,935 / 127.5  ≈  917        √DEFF = 11.3

Once the unit of observation is the day, the resolution limit follows immediately. A top-1 selector makes one decision per day, so each day is one Bernoulli draw:

SE  = √( p(1−p) / n_days ) = √( 0.73 × 0.27 / 243 ) ≈ 0.0285
MDE ≈ 2 · SE ≈ 5.7 percentage points

3. Three ways we manufactured results that weren't there

An outcome column reached the features.Twice we produced a model that had read the answer; the first scored an AUC of 1.0000. Our sample rows carry realized forward returns for grading, and a feature list built by convenience picked them up. The lesson is not “be careful” — we were being careful. It is that any AUC above ~0.7 on daily financial direction should trigger a leak hunt rather than a celebration, and the hunt should be automatic: correlate every feature with the target and read the top of that list before you read the score.

A stable sort invented 8% a day.Trees with little signal emit near-constant probabilities — on some slices 40% to 98% of a day's predictions are the identical value. We picked the best coin per day by sorting on probability; ties broke by whatever order rows happened to arrive in, so arrival order, not the model, was choosing the trades. The backtest reported +8.45% per day. If your selector produces ties, the tie-break is part of your strategy: make it explicit, make it random, and if the result depends on it, you do not have a result.

Averaging looked like signal.A rank-blend of weak signals lifted our rule's top-1 hit rate by 3.18 points. Then we ran the control that should have come first: same columns, signs randomly flipped, blended again — inputs pointing nowhere by construction. The noise blend produced +1.62 points of hit rate and +1.05 points of return per trade. Most of the apparent discovery was the mechanical effect of averaging ranks. A related result from the same stretch: selecting which signals to blend inside each fold did significantly worse than a fixed pair chosen once. Fitting destroyed what averaging preserved.

4. The limit case: when accuracy is the base rate

Those are partial mirages — some real contribution, inflated by a missing baseline. The cleanest illustration we own is the total case, running in our own production system.

Our cluster-routed direction model produces a calibrated probability of “up” for every coin, every day. Over the last 30 days of live, self-graded predictions:

The model never once said up. And once a classifier is constant, its accuracy stops being a measurement and becomes a restatement of the base rate, by definition:

if  prediction ≡ DOWN  for every row, then
    accuracy = P(correct) = P(actual = DOWN) = base rate
    edge     = accuracy − base rate ≡ 0

The precomputed walk-forward tables agree, with more decimals. At the 1-day horizon, for every behaviour cluster:

It is tempting to read those zeros as a bug. They are the opposite. The probabilities pass through isotonic calibration, and a calibrated probability is a promise: when the model says 55%, the event should happen 55% of the time. If the features carry no directional information, the only honest calibrated output is the unconditional base rate — and since that sits below 50% for “up” on most days, the argmax is DOWN on every row. The machinery behaved exactly as designed. It reported “I don't know” in the only vocabulary a probability has.

A model with nothing to say should collapse onto the base rate. The failure worth writing about is not the collapse — it is presenting what collapsed as a forecast.

5. Why it was never going to work

Everything above is empirical: we looked, and did not find. That is a weak kind of claim — absence of evidence. What changed our mind was a result that turns it into something stronger.

Christoffersen and Diebold (Management Science, 2006) show that for a return series, the conditional probability of a positive return is

P( r > 0 | information )  =  Φ( μ / σ )

Both quantities are measurable on our own panel, so the band the conditional sign probability can physically occupy is not a matter of opinion:

Set that against the detection floor from section 2. The quantity we were trying to estimate is 0.38pp; the smallest thing 243 days can resolve is 5.7pp. The target is about fifteen times smaller than the instrument's resolution.

Inverting the relation makes the scale vivid. For a genuine 55.7% call you would need:

Φ ≥ 0.557   ⟹   μ/σ ≥ 0.1434
μ ≥ 0.1434 × 7.541%  =  1.081% per day  =  +4,965% per year

This is the difference between “we could not find it” and “it is not there”. More data does not help, because the binding parameter is a property of the instrument, not of the sample.

6. The reformulations don't escape it

The natural response is to change the target. Don't predict up-or-down; predict a dump— a fall of 10% or more. We measured AUC 0.683 on that, against 0.603 for the mirror-image “pump” target. It looks like a directional edge finally appearing.

It is not, and the algebra says why. “Fall ≥ 10%” is a conjunction — big and down — so it factors:

P(fall ≥ 10%)  =  P(|move| ≥ 10%)  ×  P(down | big move)

AUC measures ranking. If one factor varies enormously across rows and the other barely does, the ranking is the first factor's ranking. Across our deciles, P(|move| ≥ 10%) spans 7× while P(down | big) spans 3.6× — so the “dump” ordering separates coins by 25×, and neither factor is direction.

The decisive test: we trained a model on the symmetric target |move| ≥ 10%, which by construction contains zero directional information, and scored it out-of-sample on the directional ones:

There is a trap here that will regenerate this illusion forever, and it is worth naming. A fixed 10% barrier is a 5σ shock for a coin with 5% daily range and a 0.6σ wiggle for one with 16%. Those are different populations with different sign skews, so any volatility proxy appears to predict the sign merely by predicting which population a row came from. Sort by ATR and the pooled sign-AUC is 0.5844; stratify within predicted-magnitude deciles and it collapses to 0.5013 — chance, exactly.

Any “this will move big” selector, used directionally, buys tail risk. It correctly identifies which instruments are dangerous. Being right about that is not the same as knowing which way.

7. What this instrument can and cannot see

The same arithmetic that bounds our hit rates bounds what we could validate from anyone else. Converting the detection floor into Sharpe terms, 244 days of daily returns can only certify an annualised Sharpe above roughly 2.41. Set that against the published literature:

That reframes what a negative result from us means. We are not entitled to say “crypto momentum does not work”; we are entitled to say our panel cannot see it. The direction bound is different in kind — it does not depend on sample size at all.

What survived

Three things, none of them a per-coin sign forecast: a volatility model that ranks which coins will move hard (AUC 0.61–0.72) without claiming a side; a cross-sectional relative-strength model measured against a genuine 50% null; and one hand-written rule behind a funding gate, which moves the base rate by 15–19 points rather than ranking within it.

That last distinction turned out to be the most useful thing we learned. Gates and rankers are different objects with different sample requirements — a gate shifts the base rate by enough to clear the 5.7pp floor, while a ranker operates in the 1–3pp band that is invisible here. Training a model on our own gated cohort reproduced the gate and added +0.003 AUC, which is inside the noise control from section 3.

And when testing anything new, benchmark against the dumb version rather than against zero. Several conditions looked like strong predictors of large moves until we checked what one line of code does on the same panel:

Why publish this

Applied honestly, these checks killed most of what we built. We publish that anyway, because the opposite kind of page is everywhere: “our model is 90% accurate”, with no mention of what it was compared against, how many days it was tested on, or how wide the error bars are. That is not a strong claim. It is not a claim at all — there is nothing in it you could check, and nothing that could turn out to be false.

A number only starts to mean something once you know three things about it: what it beat, how many independent observations it rests on, and what the author did to try to prove it wrong. Everything above is our attempt to hold our own numbers to that standard, including when the answer was that we had nothing.

The useful question about any backtest is not “how accurate is it”. It is “compared to what, over how many independent days, and what did you try that would have shown you it was wrong?”