Stella-vs-Trivy benchmark enhancement — consolidated findings

Status: research capture (pre-plan). Date: 2026-06-21. Purpose: single durable reference consolidating four investigations, so the implementation sprint is built on complete knowledge. No code changed yet. Source workflows (full transcripts under the session subagents dir): gap analysis w99iukh19, scoring recon w9km5w72n, oracle-data acquisition w38ebc66u. Working dir of the comparer: tools/benchmarks/stella-vs-trivy/.

1. Current state of the comparer

The lane builds 500 real OSS projects into OCI images and scans each with three scanners — Stella, Trivy (Aqua), Grype (Anchore). It measures only:

• PURL counts per tool (arrays + counts + purlDeltaPct) — compare-project.sh SBOM block.
• Alias-aware CVE-ID overlap, three pairwise sets (Stella↔Trivy, Stella↔Grype, Trivy↔Grype) — compare-project.sh:402-510.
• Stella-only reachability estimate — regex binning of a reachability string, compare-project.sh:436-439.
• Trivy source-tree FP annotation — annotate-trivy-fixture-fps.py (one-directional).

There is no ground truth — every metric is tool-vs-tool. The harness’s own analysis-problem-and-winning-groups.md documents a prior wrong “Trivy wins” conclusion that was a methodology confound (deployed-vs-all scope). This is the core motivation: tool-vs-tool counting is not just incomplete, it has actively misled.

2. The 10 evaluation dimensions vs current coverage

3. Locked decisions

• Tools to add: osv-scanner (independent OSV lineage), Syft (SBOM-interop producer). Clair dropped (Postgres cost; marginal RHEL-only value).
• Lineages for consensus: Stella (Concelier) · Trivy (Aqua) · Grype (Anchore) · osv-scanner (OSV). Grype+osv-scanner partly correlate on OSV → consensus is a fallback proxy only.
• Oracle = rule-based, no hand-labeling. Truth derived from advisory version-ranges + N-of-M agreement + distro backport rules.
• Label precedence on conflict: distro tracker wins for OS packages (deb/apk/rpm); advisory-range wins for language packages.
• Oracle data = LOCAL Concelier Postgres stellaops_platform.vuln.advisory_affected (rich version_range JSONB + versions_fixed[] + database_specific, independent of the compact DB the matcher reads). No internet fetch — confirmed present locally and measured (§10). Benchmark projects are latest-clone + cached (already the harness model). The fetch-not-vendor / oracle-sources.lock.json approach (§7–9) is retained only as a fallback for environments without a populated Concelier; if ever used, data stays in a git-ignored cache with only URLs+digests committed (removes the redistribution trigger incl. RHEL — no counsel gate).
• Comparer is developer-only: extend the existing lane scripts (not a parallel system); the scorer is a C# dev-only tool built/run by the lane, referencing product libs but shipped in no release artifact.
• Gating: report-and-trend first; add CI fail-gates once oracle coverage/label-stability is proven.
• VEX: annotate by default; explicit flag to suppress. Capsule signing: local key, not Sigstore/Rekor.

4. Reporter / product gaps (Stella sbom check) — 14

Recurring theme: Stella computes/stores the signal but the offline read path projects a thin finding.

v3 compact-DB bump batches EVID-03 + KEV-01 (new columns); RPT-02 rides the same re-export (exporter fix only). One re-export + mirror refresh covers all three.

Product open questions: EPSS source table for the join; EVID-05 confidence rubric; AUDIT-01 VEX default; AUDIT-02 capsule signing key; EVID-04 CPE synthesis policy.

5. Harness gaps — 17

5a. Reachability dimension — “does the code reach the vulnerable call” (REACH-01…08)

Scope: genuine call-path reachability (the app’s code graph reaching the vulnerable function/sink), not scoring a metric. No competitor emits it (Trivy/Grype/osv-scanner) → Stella-only axis whose value is noise reduction: “of N found vulns, M are reachable-proven.” Source workflow wpbgy1si3.

Headline defect: the lane never runs the engine. compare-project.sh:292 calls sbom check with no --reachability/--source, so findings.reachability is always null and the regex estimate (:436-439) bins null into a permanent zero. The current reachability number is dead.

Reporting model: funnel found → assessed → reachable (proven+likely) → proven, gated on reachabilityEngine.ran; headline reachableProvenRate = proven/assessed, noiseReductionRate = notObserved/assessed. Competitor columns stay N/A by design (that is the story). Coverage honestly bounded to Go+Rust today (9,077 sinks); all other cohorts shown as “reachability: not measured (no sinks)”.

Ground truth (no hand-labeling): verdict is self-certifying — proven requires a non-empty witness path; unknown(no-sinks) is rule-derived. Correctness/regression stays on the curated tier corpus (ReachabilityTierMetricHarness, promoted out of test asm — the same metrics engine ORACLE-04 reuses) + a precision-only “unreachable-by-rule” cross-check (advisory sink symbol provably absent from the call graph). Exploitability funnel: found → reachable:proven → KEV-listed → high-EPSS (reads KEV-01/RPT-02 fields).

6. Scoring reuse map (the big finding: ORACLE-04 is mostly assembly)

Substantial scoring already exists — and the highest-value pieces are all C#. ORACLE-04 becomes assembly of tested code + one new rule engine.

Do NOT reuse for evaluation: Policy EWS / AdvancedScoringEngine and Findings RiskEngine — those are risk prioritization (0-100 danger), orthogonal to correctness. Do NOT extend the doc’s stella bench run plural-Benchmarks path — it is unwired (not in CommandFactory; ICorpusRunner has no prod impl).

Genuinely net-new: the 5-tier TruthTier enum + the rule engine assigning labels per (image,pkg,CVE) (ORACLE-01/02/03 — nothing in the repo does (image,pkg,CVE) truth-labeling); the composite key; an osv-scanner + Stella-native adapter; promote the tier-metric harness; revive the orphaned Benchmark lib.

7. Oracle data — language/library lineages

Decision (2026-06-21): the oracle reads these sources from the local Concelier Postgres (vuln.advisory_affected), NOT by re-fetching. The URLs/formats below remain the authoritative map and the fallback for un-populated environments. Measured local coverage is in §10.

Authoritative URL/format map is already in the repo: every per-ecosystem language connector is a no-op alias of the single real OSV fetcher.

Range-evaluator independence (recommended): reuse EcosystemVersionFilter.IsStillVulnerable (EcosystemVersionFilter.cs:36) fed raw upstream data — shares the algorithm, not the data, keeping it non-circular. A neutral re-implementation is only warranted to audit the comparator itself (then run both and diff).

8. Oracle data — distro lineages

Cohort distro exposure comes from the runtime base images the lane builds.

License-by-reference confirmed clean for all (incl. RHEL — fetch-by-reference, no committed data = consumer, not redistributor). Document the snapshot date per AGENTS.md §2.6.

9. Integration into the existing lane

• New scripts/resolve-oracle-data.shmodeled byte-for-byte on resolve-grype.sh (same --cache-dir/--metadata/--env flags). Reads the lock, GETs each pinned artifact, verifies sha256 (fail-closed on drift), writes oracle.env + oracle-metadata.json (stellaops.bench.oracle/v1).
• Cache: $BENCH_CACHE_DIR/oracle/ — sibling to trivy/ and grype/ (project-runner.sh:556,560), already outside the repo on CI; add a .gitignore rule for the local tmp/**/oracle per §2.9.
• Launch wiring: add --resolve-oracle-data/--no-... flags + a resolve block after the Grype block in launch-trivy-vs-stella-ops.sh (mirror lines ~24-25,88-91,660-672); add to the bash -n syntax-check (~436-437) and the script-copy list (project-runner.sh:71-73).
• Workflow: add a “Resolve oracle data” step before “Run benchmark” in .gitea/workflows/stella-vs-trivy-benchmark.yml (~after line 140); optional cache key on the lock digests.
• C# dev tool (e.g. src/Tools/Bench/StellaOps.Bench.Oracle, built/run by the lane, shipped in no release): reads each sbom.cdx.json + ORACLE_DATA_DIR, derives distro+release from OS-package PURLs, reuses connector parser semantics (DebianTrackerJsonParser, AlpineSecDbDto, RedHat/SUSE CSAF shapes) + EcosystemVersionFilter for ranges, emits oracle.json next to compare.json.
• Release detection: OS-package PURLs in sbom.cdx.json (parsed from /etc/os-release by the OS analyzer) key each image to the right tracker branch — confirm PURLs are release-qualified (pkg:deb/debian-12/... vs bare pkg:deb/debian/...).

oracle-sources.lock.json (committed — URLs + digests + versions ONLY)

{
  "schema": "stellaops.bench.oracleSources/v1",
  "generatedAt": "2026-06-21T00:00:00Z",
  "sources": [
    { "id": "debian-tracker-json", "distro": "debian", "releases": ["bookworm","bullseye","trixie"],
      "format": "json", "url": "https://security-tracker.debian.org/tracker/data/json",
      "sha256": "<hex>", "version": "snapshot-2026-06-21", "approxBytes": 78643200,
      "cachePath": "oracle/debian/data.json" },
    { "id": "alpine-secdb-v3.20-main", "distro": "alpine", "releases": ["v3.20"], "repo": "main",
      "format": "secdb", "url": "https://secdb.alpinelinux.org/v3.20/main.json",
      "sha256": "<hex>", "version": "v3.20", "cachePath": "oracle/alpine/v3.20/main.json" },
    { "id": "osv-npm", "ecosystem": "npm", "format": "osv-zip",
      "url": "https://osv-vulnerabilities.storage.googleapis.com/npm/all.zip",
      "sha256": "<hex>", "etag": "<etag>", "approxBytes": 217000000, "cachePath": "oracle/osv/npm/all.zip" }
    // redhat/suse marked "lazy": true (fetch only when an rpm PURL appears)
  ]
}

10. Coverage summary

Measured from the local store (stellaops_platform.vuln.advisory_affected, 2026-06-21) — distinct advisories per ecosystem:

So NuGet & Swift ARE present locally (the OSV-bucket concern is moot — GHSA backfill is already ingested); dart/elixir/swift are thin-but-present; cpp is the only true gap; distros deb/rpm/apk are strong. All from local data, zero fetch. Estimates below are superseded by the measured table above.

• Libraries (estimates, superseded): OSV directly ~11/15; +GHSA → ~13/15 (~87%). Hard gap: cpp (no clean PURL-ranged ecosystem). dotnet/swift depend on GHSA enrichment, not a dedicated OSV bucket.
• Distros: Alpine + Debian + Ubuntu cover ≈100% of the crafted base images (where OS backport FPs arise). RHEL/SUSE ≈0% of the current cohort (lazy, future).
• Scoring: P/R/F1 math, ingestion, regression gate, consensus all exist (C#); net-new = the (image,pkg,CVE) truth-label rule engine.

11. Open questions / to-confirm before/while planning

Data realities (only npm’s ~207 MB is verified in-repo; rest are estimates):

1. Actual on-disk sizes of non-npm OSV zips, Debian data.json, Ubuntu vex-all.tar.xz, Alpine per-repo.
2. Do NuGet (dotnet) and Swift have working OSV all.zip buckets now? (OSV connector deliberately omits them — in-repo signal they were unreliable.)
3. cpp authoritative truth source — likely mark cpp findings oracle-low-confidence.
4. Ubuntu canonical fixed-in source: USN JSON vs CSAF vex-all vs per-release OVAL.
5. Are bench SBOM OS-package PURLs release-qualified (so branch selection is unambiguous)?

Provisioning / determinism / posture: 6. GitHub token (GHSA) + NVD API key — where do they come from in the air-gapped/CI lane? 7. Snapshot-freeze: trackers update continuously → lock pins a snapshot, resolve fails-closed on drift; lock refreshed deliberately, not per run. 8. Re-confirm RHEL/SUSE fetch-by-reference posture + record snapshot date (§2.6/§2.8).

Scorer: 9. Granularity (image,pkg,CVE) — confirm every scanner carries package identity at compare time (Trivy PkgName / Grype artifact.name / Stella component). 10. Which (scanner,tier) the regression gate fires on; is coverage% gated or report-only.

12. Sources

• Gap analysis — workflow w99iukh19 (14 reporter + 17 harness gaps).
• Scoring recon — workflow w9km5w72n (9 bench-CLI + 10 repo-wide scoring assets).
• Oracle-data acquisition — workflow w38ebc66u (4 library + 5 distro sources).
• Harness baseline — tools/benchmarks/stella-vs-trivy/{README.md,ci-lane.md,scripts/compare-project.sh}; sprint docs/implplan/SPRINT_20260618_001_Tools_stella_vs_trivy_expanded_benchmark_ci.md.