experiment Β· evergreen
AST chunks vs. plain windows
Same repo, same questions: one question of recall, MRR +0.09 β almost all of it symbol ranking.
What does syntax-aware chunking actually buy? We indexed the same repo twice β once with tree-sitter units, once with blank-line windows of the same size β and ran the same 42 questions.
| Chunker | Recall@8 | MRR | EN prose | TR prose | symbol R / MRR |
|---|---|---|---|---|---|
| tree-sitter units | 0.786 | 0.690 | 0.842 / 0.744 | 0.684 / 0.570 | 1.0 / 1.0 |
| blank-line windows | 0.762 | 0.598 | 0.895 / 0.737 | 0.632 / 0.518 | 0.75 / 0.32 |
Reading it honestly
- Recall moved by one question. MRR moved by 0.09 β and almost all of it is symbol queries (0.32 β 1.0) plus the Turkish slice.
- On English prose the plain windows were equal or better.
- So AST chunking does not find more. It puts the right piece on top and knows its name. That is worth having; it is not a 20-point lever.
The literature agrees
The cAST paper reports +1 to +4 points over same-size line windows. An independent controlled replication found β 0, and a third found the fixed windows slightly ahead. Naive function-per-chunk is reliably worse (β4 to β6 pp) β the value sits in merging small siblings and capping size, not in the syntactic boundary itself. Reranking and LLM-written context move retrieval 3β10Γ more.