Scanner Reachability Ground-Truth Corpus

This document defines the deterministic toy-service corpus used to validate reachability tier classification quality in Scanner tests.

Location

• src/Scanner/__Tests/__Datasets/toys/

Service Set

• svc-01-log4shell-java
• svc-02-prototype-pollution-node
• svc-03-pickle-deserialization-python
• svc-04-text-template-go
• svc-05-xmlserializer-dotnet
• svc-06-erb-injection-ruby
• svc-07-time-segfault-rust (RUSTSEC-2020-0071 / CVE-2020-26235, tier R4 — added by Sprint 20260610_002 LANG-RUST)
• svc-08-psr7-header-parse-php (GHSA-q7rv-6hp3-vh96 / CVE-2022-24775, tier R4 — added by Sprint 20260610_002 LANG-PHP)

Each service contains:

• Minimal source code with a known vulnerability pattern.
• labels.yaml with tier ground truth for one or more CVEs.

labels.yaml Contract (v1)

• Required top-level fields: schema_version, service, language, entrypoint, cves.
• Each CVE entry requires: id, package, tier, rationale.
• Allowed tier values:
  • R0: unreachable
  • R1: present in dependency only
  • R2: imported but not called
  • R3: called but not reachable from entrypoint
  • R4: reachable from entrypoint

Deterministic Validation Harness

• Test suite: src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/Benchmarks/ReachabilityTierCorpusTests.cs
• Harness capabilities:
  • Validates corpus structure and required schema fields.
  • Verifies R0..R4 coverage across the toy corpus.
  • Maps R0..R4 into Scanner confidence tiers for compatibility checks.
  • Computes precision, recall, and F1 per tier using deterministic ordering.
  • Identifies each labelled case by the composite service|package|CVE key. A CVE may occur in more than one service or package without one case replacing another.

Current Scope and Limitations

The current harness validates the corpus oracle and metric arithmetic only. It does not invoke a Scanner analyzer, Worker stage, WebService endpoint, or CLI against the toy source trees. Its metric tests receive predictions supplied by the test, including one perfect-score test that copies the expected labels. Consequently, these tests do not measure current Scanner classification quality.

There is no active corpus regression workflow or threshold gate. The archived benchmark-vs-competitors.yml workflow contains TODO placeholders for Scanner execution and metric calculation. The older .gitea/scripts/metrics/compute-reachability-metrics.sh targets a different, obsolete corpus path and CLI contract and is not a forcing path for this corpus.

Before this corpus can act as an end-to-end quality gate, a supported offline Scanner execution path must derive predictions from every toy, normalize them to the composite case identity, report per-tier and per-language metrics, and fail an active CI job when documented thresholds regress.

Offline Posture

• No external network access is required for corpus loading or metric computation.
• Dataset files are copied into test output for stable local/CI execution.