experiment · evergreen

The reranker that hurt

Recall@40 left 17 points on the table; bge-reranker-v2-m3 spent them and added seconds.

The textbook says: retrieve 40, rerank with a cross-encoder, hand over 8. On this corpus the textbook was wrong, and the numbers said so.

SettingRecall@8MRRTR prosep50
auto + dense (no rerank)0.7860.6900.68434 ms
hybrid, k=40 candidates0.9520.89539 ms
hybrid + bge-reranker-v2-m30.7620.5080.5794389 ms
auto + bge-reranker-v2-m30.7620.5140.5792050 ms
  • The headroom is real. Recall@40 is 0.95 — the right chunk is almost always in the candidate pool. A good reranker has 17 points to win.
  • This reranker spent them. MRR dropped from 0.690 to 0.514 and the Turkish slice fell furthest. It also cost two to four seconds per query.
  • As a "no answer" gate it was worse. It caught 12/12 negatives — and flagged 12/42 real answers as junk. A 29% false alarm rate.

A reranker is a hypothesis about your corpus. Test it like one.

#rerank #cross-encoder #latency

See this note on the whiteboard →