Reachability corpus — method and measured results

Status: fourth measured run, 2026-08-16 (sprint 20260802_001 — the R3 definition settled, and svc-03 retiered R3 → R0 as a consequence). Third run: 2026-08-16, 20260816_001 RNC-4 (negative-class semantic decided and implemented; corpus unchanged). Second run: 2026-08-15, RCE-10 (corpus grew from 8 cases to 16). First run: 2026-08-11, RCE-8. Corpus content digest: sha256:1c71eb7dee66edd7a5d48a23c349ea0a78467fdefc92a77abcc779e75bf55685 (was sha256:0721b670… through the third run).

Two changes landed the same day and they are separable, so read them separately. The third run moved the analyzer against an unchanged answer key (8/16 → 10/16). The fourth moved the answer key — one case — against an unchanged analyzer (10/16 → 11/16). Both deltas are broken out below, and the answer-key change is the one that deserves your suspicion, so it is argued in full rather than summarised.

This page describes how we measure the accuracy of the shipped reachability analyzer, and publishes the numbers that measurement produced. It exists because the product’s thesis is “do not trust us, replay it yourself”, and that has to apply to our own accuracy claims before it applies to anything else.

Everything here is reproducible offline from a clean checkout with one command. Nothing on this page is an estimate, a projection, or a figure from a different build.

Read the sample size first. The corpus is sixteen cases, two per language, across eight languages and four verdict classes. That is enough that no single case can carry a class or a language on its own, and not nearly enough for a per-language accuracy claim — two cases is a pair of examples, not a rate. We do not make one. What this page supports: a reproducible method, an honest current number, and a named list of the analyzer’s failure classes, each tied to the cases that expose it.

The command

dotnet run --project src/Scanner/__Benchmarks/StellaOps.Scanner.ReachabilityCorpus.Evaluator \
  -- --output ./corpus-report

Writes report.json (machine artifact, the determinism subject) and report.md (this page’s source data) into ./corpus-report. Options: --corpus <dir> to point at a different corpus root, --help for the rest.

Exit codes are the gate, and a buyer replaying offline gets the identical signal: 0 every case evaluated; 2 any case failed to evaluate — so CI cannot go green while silently measuring fewer cases than the corpus contains; 3 the negative (not-observed) class has no ground-truth support left, so the corpus has reverted to the positives-only shape that cannot support a precision claim at all. The run also prints the negative class’s precision, recall and support on every invocation, whether or not they are flattering.

No network, no database, no running service, no language toolchain. The run takes about a second.

What is being measured

The evaluator drives the shipped analysis path, not a reimplementation of it. For each corpus case:

1. the real per-language ICallGraphExtractor reads the toy service’s source directory and produces a call graph;
2. the real per-language ISymbolNormalizer resolves the advisory’s canonical symbol ids against that graph’s nodes;
3. the real ReachabilityAnalyzer runs its BFS from the graph’s entrypoints;
4. the real ReachabilityStackEvaluator and PublicReachabilityVerdictComposer produce a PublicReachabilityVerdictObject.

The evaluator supplies inputs and scores outputs. It classifies nothing. The report records the match-rule id and version that bound (or failed to bind) each advisory symbol, so every number traces to a named rule.

Ground truth

src/Scanner/__Tests/__Datasets/toys/ holds sixteen toy services, each with a labels.yaml (schema v2) stating the expected reachability tier R0–R4, a rationale, and the advisory’s affected symbols with their provenance. The corpus README documents the schema, the per-language canonical-id shapes, and how to add a case.

Every label states a checkable reason, not a belief. The tier is a claim about that source tree — “the entrypoint reaches this symbol through this function”, or “no call site for this symbol exists anywhere in the tree” — so a reviewer can confirm or refute it by reading two files, without trusting the author and without running the analyzer.

Four advisories deliberately appear on two services each, with the advisory and the affected symbol held constant and only the usage varied:

A version matcher cannot tell the members of a pair apart — both declare the same vulnerable package at the same version, and an installation reports the same component for both. Only reachability can, which makes each pair the sharpest single test in the corpus of whether the analyzer is adding anything.

Scoring space, and the honest limits of it

Scoring happens in the analyzer’s own output space — the public verdict lattice (reachable:proven, reachable:likely, not-observed, not-present, unknown). Expected values are derived from the corpus tier through the shipped maps (tier → confidence tier → public verdict); the evaluator keeps no mapping of its own.

That mapping is not injective. R1 (“present in the dependency closure”) and R2 (“imported but not called”) both collapse onto unknown. So this benchmark cannot separate R1 from R2, and we do not reconstruct a predicted tier to pretend otherwise — inverting the collapse would fabricate a distinction the analyzer never made.

Results

Sixteen cases, all sixteen evaluated, eleven exact matches (accuracy 0.6875).

“Support” is the number of corpus cases carrying that verdict as ground truth. Read every precision and recall figure against it: five cases is five cases, not a rate.

not-observed precision reads 1.0000 and you should not quote it. It rests on three predictions. A perfect score over three attempts is the most misquotable number on this page, and the not-present row two lines below — a vacuous 1.0000 from zero predictions — is the cautionary example sitting in the same table. Recall 0.6000 over five cases is the more honest half, and it is still not a rate.

The two prior positions, kept for comparison

Second position — 2026-08-15, before the negative-class semantic. Same sixteen cases, same answer key, earlier analyzer. 8/16 (0.5000).

Third position — 2026-08-16, analyzer changed, answer key unchanged. 10/16 (0.6250).

What each step did. The analyzer step (second → third) touched only what the analyzer does when it is about to say no: not-observed went from 0/4 with two false positives to 2/4 with one, unknown precision rose, and both reachable classes were untouched. The answer-key step (third → fourth) moved exactly one case — svc-03, retiered R3 → R0 — which turned the last not-observed false positive into a true positive and moved one case out of the reachable:likely support into the not-observed support. No analyzer prediction changed between the third and fourth runs; every per-case Actual column is identical. Only the Expected column for svc-03 moved.

What the negative class asserts now, and why it is still 2/4 rather than 4/4

The full decision is docs/modules/scanner/reachability-sink-strategy.md §7.0a. In one sentence: not-observed now asserts that the analyzer searched and that the search was capable of exhibiting the symbol had it been called.

Two gates, both failing closed to unknown:

1. An entrypoint witness. A BFS rooted at an empty entrypoint set cannot fail to find a path for any reason but its own emptiness. This removed an overclaim rather than adding a claim — see failure class 4.
2. A package coverage witness, for the case where the advisory symbol bound to no graph node at all: the graph must contain a node attributed to that symbol’s own package, proving on this artifact that the lane resolves call sites into it. It is a positive control, not a completeness proof — no static analyzer can assert a graph is exhaustive, and this one does not try.

The two negatives the analyzer now gets right (svc-15 php, svc-16 rust) both call some other symbol of the advisory’s package, which is what supplies the control. The two it still declines are declined for reasons this corpus can demonstrate rather than assert:

• svc-05 (dotnet) — DotNetLightweightCallGraphExtractor attributes every external node to the literal package "unknown", so no coverage witness can exist in that lane. Its own pair svc-13 proves the point: the same advisory symbol fails to bind there even though it is called. Granting svc-05 a negative verdict would defer a patch on a symbol the analyzer cannot see.
• svc-09 (java) — empty graph, no_graph_for_language. A different cause inside the same class (failure class 1), and not folded in.

We measured the cheap alternative rather than asserting it was bad. Patching the analyzer to relabel every symbol_unresolved as not-observed — the move this exercise exists to prevent — was implemented, run, and reverted. It does not score 4/4: svc-09 still fails at the earlier empty-graph gate, so it reaches 3/4. It lands on the same headline 10/16 as the principled gate, and gets there by breaking two cases that were previously right or honest — not-observed precision falls to 0.4286 with four false positives, including svc-04, where it would claim “no path observed” for a Go module (text/template) that appears nowhere in the graph. Same accuracy, half the precision, and the errors land exactly on the verdict a team defers a patch on.

Three analyzers, all scored against the same 2026-08-15 answer key (i.e. before svc-03 was retiered), so the comparison isolates the analyzer choice and nothing else:

The shipped row reads 10/16 rather than the headline 11/16 for that reason: the R3 answer-key correction landed afterwards and is accounted for separately.

The not-present row has no ground-truth case and never will. No corpus tier maps onto that verdict and PublicReachabilityVerdictMapper never emits it — it is reserved for L2 linker/symbol-table facts no internal vocabulary carries yet. Its perfect score is an artifact of the harness convention that a class with zero expected and zero predicted scores 1.0. It means nothing, and growing the corpus cannot change that; only an L2 evidence source could.

What moved in the RCE-10 growth pass (2026-08-15), and why

Historical: this section compares the eight-case run (2026-08-11) with the sixteen-case run (2026-08-15). The 2026-08-16 delta is the section above.

The headline accuracy is unchanged at 0.5000 — 4/8 became 8/16 — and that coincidence is worth taking apart, because the underlying picture changed a lot:

• reachable:proven recall rose from 0.5000 to 0.7143 (2/4 → 5/7). The three new positive cases (node, python, go) were all predicted correctly. Before this run the node, python and go lanes had never produced a correct positive verdict on this corpus; now each has.
• not-observed recall stayed at 0.0000, over four cases instead of one. This is the real result of the growth. What looked like one .NET oddity is a systematic behaviour reproduced in four languages: see failure class 5.
• unknown precision fell from 0.5000 to 0.2500. The analyzer emitted unknown eight times against two ground-truth cases. It is not that the analyzer got worse — the corpus simply now contains more cases it declines to answer, and precision is the honest way that shows up.
• reachable:likely was predicted correctly for the first time (svc-14), moving recall from 0.0000 to 0.3333. Before this run that class was unpredictable by construction — see “Two evaluator defects” below.

Both directions are in the table because both happened. If the growth had been chosen to flatter the number, the three easy positives would have gone in and the four not-observed cases would not have.

Per case

Three reason codes are new in this run and they are worth reading as a group, because they are the same abstention split into causes that were previously indistinguishable: package_not_covered (the lane cannot show it sees that package here — svc-04, svc-05, svc-13), no_entrypoints (the search was vacuous — svc-06), and the retained no_graph_for_language (there was nothing to search — svc-01, svc-02, svc-09). Only the last of those existed before.

Three of the sixteen cases produced an empty call graph and therefore measure nothing about the analyzer: svc-01, svc-02 and svc-09. The report states this in its own body, on the same page as the metrics, because a match on an empty graph is a coincidence and a reader is entitled to know how many of the eight matches are of that kind. One is: svc-02.

Known failure classes

Each of these was diagnosed from the run, not assumed. They are the answer to “what are your known false-negative classes”, and they are the reason we publish the measurement rather than a headline.

1. The Java lane cannot analyze a source-only tree. (svc-01 and svc-09, both false negatives.) JavaCallGraphExtractor reads .class, .jar and .war files — it is a bytecode analyzer. The toy services ship .java and nothing else, so the extractor produced an empty graph and both verdicts degraded to unknown / no_graph_for_language. A real deployment scans built images and does have bytecode, so this is a corpus/analyzer interface mismatch rather than a production blind spot — but it means this corpus measures nothing about Java accuracy, and adding a second Java case did not change that: it produced a second empty graph, exactly as the first did. Closing this needs compiled bytecode in the corpus, which conflicts with the current “no toolchain, nothing compiled, nothing fetched” build contract; it is the largest single hole in the sample.

2. Log4Shell’s path runs through the dependency’s own code. (svc-01, second cause.) Even with bytecode, the labeled symbol org.apache.logging.log4j.core.lookup::JndiLookup.lookup is not in the toy service’s own code; the path from Log.error(...) to the lookup lives inside log4j. A source-only corpus cannot express that. This is the general shape of the hardest reachability cases and it is honest to say the corpus does not yet cover it.

3. The JavaScript lane finds nothing in a script with no function declarations. (svc-02, a match for the wrong reason.) src/index.js is top-level statements only, so the extractor produced an empty graph and the verdict was unknown / no_graph_for_language. The expected value for an R2 case is also unknown, so this scores as a match — coincidentally. It is counted as a match because the scoring rule is mechanical, and flagged here because a reader deserves to know one of the eight matches carries no evidence.

The RCE-10 growth confirmed this is about the shape of the script, not the lane: svc-10 wraps the same kind of code in main() and renderGreeting(), and the extractor produced a 3-node graph and the correct reachable:proven. The JS lane works; a script that declares no functions gives it nothing to work with.

4. Ruby entrypoint classification misses script-style entrypoints. (svc-06, false negative.) app.rb guards its call with if __FILE__ == $PROGRAM_NAME, which is not a method definition. RubyEntrypointClassifier recognises HTTP handlers, message handlers, rake tasks, and run/call/execute on a class named *CLI or *Command — none of which match. The graph had zero entrypoints. The advisory symbol erb::ERB.new did resolve; only the entrypoint was missing. This one is a genuine analyzer gap.

Corrected 2026-08-16 — and this case was worse than recorded. Through the 2026-08-15 run the verdict here was not-observed: the analyzer told a reader it had looked for a path and found none, for a service that is genuinely R4 reachable, on the strength of a BFS that searched nothing. That is not a missing answer, it is a wrong one, and it is the exact failure mode — a negative verdict a team defers a patch on — that the negative-class work was funded to prevent. It reads as a “false negative” in the class-4 sense above, but it was also a false positive of the negative class, which is why the old not-observed row showed 2 predictions against 0 true positives.

The §7.0a entrypoint gate closes it: a negative claim now requires at least one classified entrypoint, so svc-06 answers unknown / no_entrypoints. The case is still a miss — the expected verdict is reachable:proven and the Ruby entrypoint gap that causes it is untouched — but the analyzer no longer makes an unfalsifiable claim about it, and the reason code now names the real defect instead of hiding it inside a verdict.

Now measured from both sides: svc-14 calls the same ERB.new from a ReportCommand#run, which the classifier does recognise, and the analyzer answered correctly. The pair isolates the defect precisely — same advisory, same symbol, same gem, one entrypoint shape recognised and one not.

5. unknown and not-observed are different claims, and the analyzer is stricter than the label. (svc-05, svc-09, svc-15, svc-16 — all four not-observed cases, all scored as misses.) This was one case before the growth and read as a .NET quirk. It is not: it is the analyzer’s uniform behaviour, reproduced independently in .NET, Java, PHP and Rust. When a service declares a vulnerable package but never calls the affected symbol, the symbol does not resolve against the call graph, and the analyzer returns unknown / symbol_unresolved rather than not-observed.

The not-observed class therefore scored 0/4, and that single fact was the most consequential result on this page. “The package is present and nothing calls it” is the most common real-world shape of a vulnerable dependency, and it is the one where reachability analysis has the most value to add — it is the verdict that lets a team defer a patch. Read strictly, the analyzer never delivered it for the source-only shape.

RESOLVED 2026-08-16 (sprint 20260816_001). The open question this class raised — whether not-observed means “I looked for a path and found none” or “an exhaustive graph with no call site is evidence of no path” — was decided, and the answer is neither verbatim. not-observed now asserts that the analyzer searched and that the search was capable of exhibiting the symbol had it been called, which is a checkable property where “exhaustive” is not. The decision, the two rejected options, and the residual risk are recorded in docs/modules/scanner/reachability-sink-strategy.md §7.0a.

The class now scores 3/5 at precision 1.0000 (2/4 at 0.6667 before the R3 resolution moved svc-03 into it — see the answer-key section, and do not quote a perfect score standing on three predictions). Two of the five are still declined, and the reasons are demonstrated by the corpus rather than asserted: svc-05’s lane cannot attribute external calls to packages at all (its pair svc-13 proves the same symbol fails to bind even when it is called), and svc-09 has no graph to search. Both remain scored as misses. The honest ceiling for this corpus, under this analyzer, is 2/4 — and it is a ceiling made of two named, fixable analyzer gaps rather than a semantic disagreement.

6. The .NET lightweight lane cannot bind an instance-method advisory symbol. (svc-13, false negative — new in this run.) svc-13 calls serializer.Deserialize(reader) — the only way C# can call that member, since XmlSerializer.Deserialize is an instance method. DotNetSymbolNormalizer documents that variable-receiver text never binds, because inferring that serializer has type XmlSerializer is type inference the lexical tier does not have. The graph was built (4 nodes, 3 edges, 1 entrypoint) and the path is there; the symbol just could not be attached to it.

Measured rather than reasoned: rewriting the call site as XmlSerializer.Deserialize(reader) makes the same case bind and produce the expected reachable:likely. That spelling is not valid C# for an instance member, so no idiomatic .NET source can bind this advisory symbol in this lane — the corpus keeps the idiomatic form and takes the miss. Closing it needs the Roslyn semantic tier, which resolves receiver types.

7. unknown cannot be scored against. (svc-02 and svc-04, both matches.) Tiers R1 and R2 both collapse onto unknown, and the analyzer also returns unknown whenever it cannot bind a symbol — which is what happens in an R1/R2 service by definition, since the symbol is never called. Both unknown cases in the corpus therefore match for a reason that carries no information, and this is structural, not fixable by adding cases. It is why the RCE-10 growth added no further R1/R2 cases: they would have raised the accuracy figure by two while measuring nothing.

Two evaluator defects the growth exposed

Both were found by adding cases, both are fixed, and both are recorded because they change how an earlier number should be read.

The reachable:likely class was unpredictable by construction. The shipped composer decides that verdict by testing a sink mapping’s evidence URI against PublicReachabilityVerdictComposer.PackageSymbolsEvidenceUriPrefix. The corpus adapter emitted every symbol under its own stella://reachability-corpus/… scheme, so a label declaring the package-symbols tier was invisible to that test and the verdict could never be produced for any case. The 2026-08-11 run’s reachable:likely recall of 0.0000 was therefore not a measurement of the analyzer at all. The adapter now emits the tier-specific URI shape the production bridge (VulnSurfaceCveSymbolMappingService) uses, and svc-14 is the first case ever to be answered reachable:likely correctly.

Corpus-wide symbol scoping would have corrupted the paired cases. The adapter keyed ground truth by (CVE, purl) for the whole corpus, which was harmless while every case used a distinct advisory. The moment svc-13 and svc-05 shared one, the ordinal-later service silently overwrote the earlier one’s answer key — measuring a case against another case’s ground truth. The evaluator now builds one mapping set per case. CorpusCveSymbolMappingServiceTests pins both the fix and the hazard.

Neither fix moved any pre-existing case: the eight-case corpus produced an identical per-case table before and after, checked by diff.

The answer-key change: what R3 means, and why svc-03 moved

This is the one place on this page where we changed a label, so it gets the full argument rather than a summary. Read it sceptically; it is worth exactly one case, and it went in the direction that flatters us.

The disagreement. Two committed definitions of R3 coexisted for months: the corpus README said “called but not reachable from entrypoint” (a negative), while ReachabilityTier.R3 said “Likely reachable (heuristic call path)” (a positive). The scored map always followed the second. Settled 2026-08-16 in favour of the second, on four grounds, none of which is about the score:

1. The ladder is monotone. ReachabilityTier’s own summary line reads “R0 unreachable … R4 confirmed-reachable”. A negative at R3 sits between R2 and R4 and breaks that ordering.
2. It would add a second non-injective pair. R0 and R3 would both map to not-observed. This corpus already documents R1/R2 → unknown as a structural limit and explicitly refused to add more cases of that shape; deliberately creating another would be worse.
3. An R3 case under the losing reading is unmatchable by construction — the analyzer answers not-observed for that shape and can never answer reachable:likely. Such a case measures the definition, not the analyzer.
4. It would leave reachable:likely vacuous. No corpus tier would map onto a shipped, emittable verdict class — the same emptiness that makes the not-present row meaningless. svc-13 and svc-14 were added in RCE-10 precisely to give that class support.

Why svc-03 had to move, and the check that it was not tuning. Under the decided reading, svc-03’s source tree does not express R3: main() only prints, so nothing reaches unsafe_deserialize, so there is no entrypoint path at all. A label must be true of its tree — the corpus’s own authoring rule — so leaving it at R3 would have left the answer key asserting something false about its own source. It is now R0, under the second admissible R0 shape (call site present, unreachable from any entrypoint), which is checkable by reading one file.

The honest disclosure: this is the only relabel, and it improves the number, 10/16 → 11/16, and takes not-observed precision from 0.6667 to 1.0000 by converting the class’s last false positive. Three things bound that:

• The losing reading also scores svc-03 as a match (its not-observed is correct under either reading). What the two readings actually disagree about is svc-14: it matches under the winning reading and would become a miss under the loser. So the decision is worth 11/16 vs 10/16 — one case — and the arguments above were settled before that was costed.
• The alternative that would have scored the same was rejected. The corpus README blesses “fix the source so it expresses the tier you claimed”. Rewriting svc-03 to have a real entrypoint path would have kept it at R3 and also scored a match — but it would have destroyed the corpus’s only case of the no-path-to-resolved-sink negative shape, leaving that shape unmeasured. The label moved because the label was wrong, not because it was cheaper.
• It cost something too. reachable:likely support drops from 3 cases to 2, the floor below which this corpus refuses to read a class as anything at all.

R0 now has two admissible shapes and that is deliberate, mirroring the analyzer: the public lattice merges “no call site anywhere” and “call site, no entrypoint path” onto not-observed and records which shape it was in the coverage statement (§7.0a). A label space that split them while the verdict space merged them would manufacture the non-injectivity that argument 2 rejects. The corpus README carries the authoring rule and requires each R0 case to state its shape.

What would overturn this. A demonstration that the ladder is not meant to be monotone — i.e. an owner ruling that ReachabilityTier is a taxonomy of evidence kinds rather than a confidence ordering. That would reopen argument 1 and, with it, arguments 2 and 4. Nothing about the measurement would overturn it; the measurement is downstream of the definition, which is the whole reason this was settled before the page was regenerated rather than after.

Determinism

Two consecutive runs over the same corpus produce byte-identical report.json and report.md. This is guarded by ReachabilityCorpusEvaluatorTests.Evaluation_TwoFullRuns_ProduceByteIdenticalReports, which was red-proved: making the case emit order depend on ambient state fails it at byte 2027. The report body carries no wall-clock time, no elapsed duration, and no absolute path; the corpus is identified by a content digest so a report is comparable across machines and a silent corpus edit cannot masquerade as an analyzer change.

The measurement is also independent of the checkout’s line endings: the same corpus evaluated with CRLF working-tree sources and with the committed LF bytes produced byte-identical report.json (measured 2026-08-14, RCE-9). That matters because a buyer’s clone is not guaranteed to match ours byte for byte, and a benchmark whose number moved with core.autocrlf would not be replayable.

Re-measured on the sixteen-case corpus (2026-08-15): two consecutive runs produced byte-identical report.json (17388 bytes, sha256:07ed1e070d97a824968ddbec0be530d33e85f43107867809c162b139d7bd6b7e) and report.md (7028 bytes).

Re-measured after the negative-class change (2026-08-16, third run): two consecutive runs produced byte-identical report.json (17399 bytes, sha256:615953909c4fb662b61a9593b32e34d62f9e16260889f46818b08e0a9a88c152) and report.md (7023 bytes, sha256:e51c75e516e17f439673bf4d776a44fb7db7a68acc5bcda2e3a6822cc05dc926). The corpus content digest was unchanged from the previous run, so that report delta is attributable to the analyzer alone.

Re-measured again after the R3 resolution (2026-08-16, fourth run): two consecutive runs produced byte-identical report.json (17388 bytes, sha256:b75cb6833196e4b3c977f5b0a23e803334903f3419f346d579fb49e19e000364) and report.md (7020 bytes, sha256:8505b6cf37980539fa0e426b1e78a9b6c959ea0c5cb226e9104a541459d1981f). Here the corpus digest did change (0721b670… → 1c71eb7d…) and the analyzer did not — the inverse of the previous step, and the reason the digest is published at all: it is what lets a reader tell the two kinds of change apart without trusting this page’s account of them.

The negative class is now a permanent gate

The corpus was positives-only until 2026-08-15, and that shape is why a verdict class could score 0/4 unnoticed: a sample that never asks the analyzer to decline can measure recall and nothing else. Removing the five negative cases today would raise the headline accuracy from 0.6875 to 0.7273 while measuring strictly less — which is exactly how this regresses, quietly and while looking like an improvement. (Measured, not projected: the eleven-case copy scores 8/11.)

So it is gated rather than trusted:

• every run prints the negative class’s precision, recall and support, whatever they are;
• the evaluator exits 3when that class has no ground-truth support left, with a message that says to restore the cases rather than lower the gate;
• .gitea/workflows/reachability-negative-class.yml runs the documented command on any change to the corpus, the evaluator, the analyzer or the call-graph libraries, and is a blocking lane — deliberately not part of .NET PR Tests, which is report-only and would not gate anything;
• the gate is red-proofed both ways: NegativeClassGate_FailsWhenTheClassIsPresentButHasNoGroundTruth perturbs a real report into the vacuous shape, and a positives-only corpus copy was run end to end and exited 3.

The gate deliberately checks ground-truth support, not the presence of the row. Every lattice member gets a row in every report, so a hollow row would otherwise sail through — precisely how not-present scores a meaningless 1.0000.

It is not, however, independent of where the corpus sits on disk — and that is a caveat, not a footnote. The shipped extractors skip files under directory segments named build, dist, tmp, test, tests, coverage, vendor, log, node_modules, env, venv and a few more (the lists differ per language). A corpus copy under such a path yields empty call graphs with no warning: measured 2026-08-15, evaluating a byte-identical copy from tmp/build/… scored svc-03 and svc-06 at zero nodes where the in-repo path scored them at 3 and 2. The in-repo path is safe (the comparison is ordinal, so __Tests is not tests), and a buyer running from a clean checkout is fine — but a buyer who clones into ~/build/ would silently measure less than they think. The report now states how many cases produced an empty graph, which is the signal to check the path.

Running it through your own installation

The measurement above consumes source directories. A buyer’s installation takes an image digest, so each service is also materialized as a container image: a Dockerfile per service, and src/Scanner/__Tests/__Datasets/toys/index.yaml recording the manifest and config digest of each.

bash tools/scripts/build-reachability-corpus-images.sh            # build, print digests
bash tools/scripts/build-reachability-corpus-images.sh --verify   # rebuild and diff against index.yaml

Every image is FROM scratch — nothing is fetched at build time or at scan time, and there is no base image to pin, to drift, or to licence-gate. Two consecutive --no-cache rebuilds of all eight produced identical manifest and config digests; index.yaml records what is deterministic by construction and the three inputs the digest still depends on.

Fourteen of the sixteen images resolve their labeled package to a versioned purl — the form an advisory range can match — and that is asserted against the real shipped analyzers by ReachabilityCorpusScannableArtifactTests. The two exceptions are honest and permanent: svc-03’s pickle and svc-04’s text/template are standard library, so no SBOM can express them as third-party components, and the corpus refuses to fabricate a distribution to make them appear. Those two cases can demonstrate the source-level verdict but not image-level matching. Every RCE-10 growth case resolves, including the first scannable Python case (svc-11, Pillow) and the first scannable Go case (svc-12, golang.org/x/crypto).

The four paired services are the most useful thing to run through an installation: each pair reports the same vulnerable component at the same version, so anything that distinguishes them is reachability and nothing else.

One thing the images deliberately do not change: they carry manifests that declare the vulnerable package, not the package’s own code. The Java false-negative class below is therefore unchanged — log4shell’s path runs through log4j’s own bytecode, which is absent from the corpus by design.

Growing the corpus

src/Scanner/__Tests/__Datasets/toys/README.md documents the growth path. Only one step needs judgement: supplying the advisory’s affected symbols in the language’s canonical-id shape. The evaluator needs no change to pick up a new case — the language lane comes from the label.

A new case needs a Dockerfile and an index.yaml entry as well, or it will be measurable but not scannable; the corpus README’s growth checklist carries both steps.

Sprint 20260802_001 is complete: RCE-9 (the image artifacts described above) landed 2026-08-14 and RCE-10 (this growth pass) landed 2026-08-15.

Where the sample should grow next, in the order that would buy the most:

1. Java bytecode. Two Java cases currently measure nothing. Until the corpus can carry compiled classes, the largest ecosystem in the sample is untested.
2. A vulnerable dependency’s own code. Every case here declares the package without containing it, so the hardest and most realistic shape — the path running through the dependency’s internals, as Log4Shell’s does — is unrepresented.
3. A negative case with the package declared and entirely unused. The §7.0a coverage witness needs something else in the artifact to call into the advisory’s package, so the purest real-world shape — a dependency declared and never touched at all — still answers unknown. That bound is stated in §7.0a and is the single largest limit on the negative class’s real-world reach. Closing it needs a per-lane, machine-readable “this lane models this ecosystem’s externals” signal, not more cases; a case would measure the gap rather than close it.
4. A third case per language, so a per-language figure starts to mean something. Two is enough to stop a single case carrying a language; it is not enough to be a rate.