Graph Indexer benchmarks

Graph Indexer matches the language’s own compiler on references: 99.7% precision and 96.2% recall against the TypeScript compiler on 400 sampled nestjs symbols, where grep for the same names reaches 12.8% precision. In real Claude Code sessions, agents with Graph Indexer answered 12 of 12 code questions exactly against 9 of 12 without, and were cheaper than with no index; in sub-agent rounds they were also cheaper than grep alone. Every number below comes from the public benchmark documents and the agentic benchmark, measured on Graph Indexer 3.0.0.

How are the Graph Indexer benchmarks measured?

Graph Indexer is graded against the real compilers, not against opinions. A seeded random sample of functions, methods, classes and interfaces is drawn from each repository. For every symbol, an oracle built on the language’s own front end lists every line that truly refers to it, and the answer of find_references is compared with it at file:line granularity.

Two oracles are derived from the compiler’s answer. The dispatch oracle counts references to the member itself, to the members it overrides or implements (a call through the base type may land there) and to its own overrides. This is what find_references promises, and all tables below use it with micro averages unless stated. The rename oracle is stricter: everything the compiler would touch in a rename, including calls to sibling overrides.

Two baselines run on the same sample. grep is every line containing the name as a whole word, imports excluded. Name-only is every syntactic reference with that name, with no binding to a definition. Precision is the share of returned lines that are real references; recall is the share of real references that were returned.

Which samples were used?

The TypeScript sample is 400 symbols from nestjs packages/core and packages/common (seed 11), plus a second sample of 149 symbols (seed 7). Go and Java use seed 7 with 111 to 200 symbols per repository. Twenty uses 400 symbols from packages/twenty-server/src (seed 11). All repositories are public and pinned to exact commits.

What is the precision and recall of Graph Indexer per repository?

Graph Indexer reaches 95.5% to 100% precision and 86.1% to 98.9% recall across eight repositories in three languages, while grep precision ranges from 0.9% to 60.9%. Each row is measured against the language’s own compiler or type checker.

Reading the table: on nestjs, a name search returns roughly seven false hits for every real reference. Graph Indexer binds each reference to the definition it points at, so the false hits are mostly gone. On nestjs the exact reference set matches the compiler for 91.5% of symbols; on gin, 98.6%.

Grep recall is about 99% in these samples by construction, because it returns every textual match. Graph Indexer recall is lower, and the missed references are listed as same-name call sites it could not bind, so an agent knows what to check.

Does it hold on a large monorepo?

Yes. On Twenty’s server code (28,512 files, 2.3 million lines) Graph Indexer reaches 99.9% precision and 92.4% recall, against 0.9% precision for grep. Remaining misses there are union and registry-object calls, object-literal implementers and fluent builders through generic this.

How do agents perform with and without Graph Indexer?

In real Claude Code sessions, an agent with Graph Indexer answered 12 of 12 code questions exactly on nestjs, against 9 of 12 without it; the three misses were questions about callers two levels away. Sessions are graded by a hidden oracle (the TypeScript compiler, the Go type checker, or hidden tests of real fixes), with the same model, task and repository in every arm.

Cost is measured in input-equivalent tokens (input, cache writes, cache reads and output weighted by price). Times and costs compare paired runs of the same task. The Go round was graded by the Go type checker on caddy 2.8.4.

Why does a smaller model gain the most?

With a smaller model, grep-only agents burned many more calls: 29.9 per question against 9.0 with Graph Indexer. Exact answers from find_references replace long chains of searches and file reads. The 7 questions in that round are a small sample, and the accuracy difference (7/7 against 5/7) is not statistically significant.

Where does Graph Indexer not save money?

When an agent fixes a real issue from a bug report, the saving was not measurable. In those sessions the agent often did not call the index at all, and the cost was within a few percent of no index. The short lookup rules that init writes into the agent instructions carry most of the gain there, not the tools. Graph Indexer is built for code questions, refactors and renames, where an exact answer replaces many guesses.

How fast is Graph Indexer?

Graph Indexer indexes more than 45,000 lines of code per second: the Twenty monorepo (28,512 files, 2,327,185 lines, 170,126 symbols, 554,975 references) takes 51 seconds to index fully and the index takes 304 MB. After the first build, only changed files are re-parsed, so re-syncing an unchanged nestjs takes about 0.2 seconds.

Warm tool calls on nestjs (median of 5) take 1.1 ms for find_references, 1.2 ms for get_symbol, 1.4 ms for call_graph and 3.7 ms for change_impact, so every graph tool answers in under 4 ms. outline takes about 18 ms and search_code about 30 ms. The eight tool definitions cost about 2,400 tokens of agent context and typical replies are 400 to 1,500 tokens. See the 8 MCP tools and how it works.

What are the limits of these benchmarks?

The numbers are strong, and they come with published limits. Treat them as follows.

• Not a type checker. Graph Indexer infers types well enough to bind most references, but generics, reflection, any-typed values and dependency-injection tokens stay partly invisible. They appear as lower recall, not as wrong answers.
• Precision is a lower bound on nestjs. The fixture has no node_modules, so library-typed values are any to the compiler. Some references Graph Indexer finds correctly count as false positives.
• Fixes were tuned on these samples. Accuracy work iterated on failures from the same samples. The second TypeScript sample (seed 7) shows what is left on fresh symbols: 99.9% precision and 88.7% recall.
• Java is the weakest language measured. Recall drops to 86.1% on commons-collections.
• Agent samples are small. Most rounds use one run per task and arm. Cost differences under about 20% are inside the noise, and with the model used nearly every task is solved in every arm, so the benchmark mostly measures cost and time.
• No saving when fixing real issues. See above.

For the list of other languages and how deep each is supported, see supported languages.

How can I reproduce the Graph Indexer benchmarks?

Clone the repository, install Node.js 22.5 or later and run the scripts in the bench/ folder. The repositories are public and pinned to exact commits; once cloned, the oracles run locally.

node bench/fixtures.mjs                      # clone the pinned repositories into test/fixtures/
node bench/eval-graph.mjs --n 400 --seed 11  # TypeScript compiler oracle (needs the typescript package)
node bench/eval-graph-go.mjs --n 150         # go/types oracle (needs Go 1.22+)
node bench/eval-graph-java.mjs               # javac front end oracle (needs a JDK 17+)

The agent rounds are driven by bench/agentic/; the exact commands are in the Reproducing section of the agentic benchmark. To compare against grep yourself, read Graph Indexer vs grep and the landing page results.