Evaluation Pilot — a staged acceptance test you can run against us

This guide is the test we think you should run before trusting Stella Ops with a release path. It is staged so the first stage is cheap to fail: a two-day checkpoint that either earns the rest of the pilot or kills it early. Every step names the observable outcome, what passing proves, and what it does not prove.

Everything here runs on the Free tier: 3 environments cover a dev → staging → production promotion chain, and the 100 new-digest deep scans per rolling 24 hours are far above what a pilot consumes. The signed bundle and container images are public and fetch anonymously — no account, no token.

Stage 1 — the two-day checkpoint

Budget: two engineer-days. The kill criterion for the whole stage: if any step below needs vendor intervention beyond the published documentation and community channels, stop the pilot and record which step failed. A control plane you cannot stand up from its own documentation is not ready for your release path, and finding that out cost you two days.

What passing proves: the product installs, scans, gates and deploys from its own documentation, on your artifacts, without us.

What it does not prove: anything about analysis quality, failure behaviour, or evidence portability. That is Stage 2.

Stage 2 — the adversarial week

Only if Stage 1 passed. Budget: four to eight engineer-days depending on how representative you make the test images. Each test is designed so that we do not control the outcome — you pick the images, you cause the failures, you verify on hardware we never touch.

2.1 A reachable and an unreachable vulnerability, chosen by you

Pick two of your own images: one where a known CVE sits in code your application actually calls, and one where the vulnerable package is present but the vulnerable path is not invoked. Scan both.

Expected: the reachable finding carries reachability evidence and ranks as the exposure that blocks; the not-invoked finding is reported with its evidence state shown — NOT OBSERVED is displayed as exactly that, never as “not vulnerable” and never silently dropped.

Then run our own ground-truth corpus through your installation. Your images test us on artifacts we have never seen; this tests us on artifacts we have published a measured accuracy number for — so you can check both the analyzer and the number.

The corpus is sixteen labeled toy services, two per language, in the source tree at src/Scanner/__Tests/__Datasets/toys/. Each carries a labels.yaml stating its expected tier R0–R4 and the rationale for it. Build them as images and scan them:

bash tools/scripts/build-reachability-corpus-images.sh   # 16 images, offline, ~seconds

Scan the four paired services first — they are the sharpest test. svc-01 and svc-09 both declare log4j-core 2.14.1; svc-05 and svc-13 both declare System.Xml.XmlSerializer 4.3.0; svc-06 and svc-14 both declare erb 2.7.0; svc-07 and svc-16 both declare time 0.1.43. Within each pair the component your installation reports must be identical and the reachability verdict must differ, because the only thing that changes is whether the service actually reaches the vulnerable symbol. Any tool that treats the two members of a pair the same is doing version matching, whatever it calls itself — including ours, where it fails to tell them apart.

Every image is FROM scratch and nothing is fetched at build or scan time. The manifest and config digest of each is recorded in src/Scanner/__Tests/__Datasets/toys/index.yaml; --verify rebuilds and diffs against that record, so you can confirm you are scanning the same artifacts our numbers are about before you compare anything.

Expected, stated in advance so we cannot move the goalposts afterwards:

• Fourteen of the sixteen report the labeled vulnerable package as a versioned component. The other two — svc-03’s pickle and svc-04’s text/template — are standard library, which no SBOM can express as a third-party component; we do not fabricate a distribution to make them appear, and you should treat any tool that does report them as a finding with suspicion.
• Our measured agreement with these labels is 8 of 16, published with the seven failure classes that produce the misses at reachability corpus method. If your run reproduces roughly that, the published number is honest. If it is better than ours, something is wrong — tell us.
• Expect us to fail every not-observed case: svc-05, svc-09, svc-15 and svc-16 all score 0. Where a service declares a vulnerable package and never calls the affected symbol, our analyzer answers unknown, not “no path observed”. That is stated here in advance because it is the shape you most want a reachability tool to get right, and today ours does not deliver the verdict for it. The method page argues the case for why unknown is the stricter answer; you should judge that argument for yourself rather than take it from us.
• Three cases measure nothing at all: svc-01, svc-02 and svc-09 produce an empty call graph (the Java lane needs bytecode; svc-02’s script declares no functions). One of our eight matches — svc-02 — is a coincidence for that reason, and the report says so in its own body.

What this proves: the accuracy claim we publish is one you re-derived on your own hardware from artifacts you rebuilt yourself. What it does not prove: accuracy on your estate — sixteen cases, two per language, is enough that no single case carries a language or a verdict class, and nowhere near enough to be a per-language rate. The method page says so beside every figure.

2.2 Missing evidence

Gate a digest for which some required evidence does not exist (for example, a policy that requires reachability evidence, evaluated against a scan that has none).

Expected: the check reports NOT EVALUATED and the gate treats it as non-passing. A check that could not run is never counted as a pass. If you can make missing evidence look like success, we want the reproduction.

2.3 Control-plane outage during a release

Stop the Stella Ops stack while a pipeline is waiting on the gate.

Expected: the gate blocks — the product fails closed. Then exercise the recovery path without starting the stack: a gate exception is a signed, scoped, time-limited object requiring a permission granted to nobody by default, a written justification and an expiry. Confirm that releasing during the outage required that explicit, attributable act — and that nothing released silently.

2.4 Drift between approved and running

Deploy an approved digest to a watched environment, then start a different digest on the same Docker daemon.

Expected: Watch reports the mismatch between the approved digest and the running digest for that service and environment. Note the scope honestly: Watch compares digests on the Docker daemons the agent service can reach — it is not an agent installed on your hosts, and it does not verify non-containerised processes.

2.5 Evidence that survives us

Export the Decision Capsule for one gated release (stella evidence — see the auditor guide). Copy it to a machine with no Stella Ops installation and no network path to one.

Expected: signature verification (cosign verify-blob) and content hashes (sha256sum) pass offline; the manifest pins every input by digest; replaying the decision from its sealed inputs reproduces the same verdict. Then alter one frozen input and confirm verification fails. A capsule that only verifies on the installation that produced it would prove nothing; this one is portable by design.

Reading the result

What a fully green pilot proves: the product does what this site says it does, on your artifacts, under failure, with evidence you can carry away.

What it deliberately does not prove: analysis accuracy on estates unlike your test set, behaviour at scales you did not run, and the operational questions — support capacity, continuity, references — that belong to due diligence rather than to a technical pilot. Bring those to sales@stella-ops.org with your pilot results in hand; a pilot report is the strongest opening position a buyer can have.