schema · evergreen
A chunk is a code unit
tree-sitter boundaries, ≤ 2000 bytes, big classes split, small pieces merged, header only in the index.
A chunk is a function, class, method, interface, type or exported constant — with boundaries from a real parser, not from counting braces.
parsetree-sitter
units≤ 2000 B
splitbig class → members
mergesmall ones ≥ 200 B
headeronly for the index
The four rules
- Big containers split into members. A 5 KB class becomes a header chunk
plus one chunk per method, each carrying
parent = class. - Small things merge with a neighbour. One-line types, short consts, import blocks and doc comments stick to the next unit.
- Nothing exceeds ~2000 bytes (≈ 500 tokens). The model accepts 8192, but a long chunk's embedding averages toward nothing. A single oversized function is windowed by lines and keeps its symbol.
- Text stays clean. The "what am I, where do I live" header — file, symbol, parent, imports — is prepended only to the text that gets embedded and BM25-indexed. Citations show real source.
Identifiers for BM25
The standard analyser keeps handleAuthCallback as one token, so "auth
callback" never touches it. Every chunk's identifiers are split into lowercase
sub-words and appended to the indexed text.