Evaluation Pilot — a staged acceptance test you can run against us
This guide is the test we think you should run before trusting Stella Ops with a release path. It is staged so the first stage is cheap to fail: a two-day checkpoint that either earns the rest of the pilot or kills it early. Every step names the observable outcome, what passing proves, and what it does not prove.
Everything here runs on the Free tier: 3 environments cover a dev → staging → production promotion chain, and the 100 new-digest deep scans per rolling 24 hours are far above what a pilot consumes. The signed bundle and container images are public and fetch anonymously — no account, no token.
Stage 1 — the two-day checkpoint
Budget: two engineer-days. The kill criterion for the whole stage: if any step below needs vendor intervention beyond the published documentation and community channels, stop the pilot and record which step failed. A control plane you cannot stand up from its own documentation is not ready for your release path, and finding that out cost you two days.
| # | Step | Observable outcome |
|---|---|---|
| 1 | Download the signed bundle, verify the signature, run ./install.sh | The stack reports healthy |
| 2 | Run stella doctor run --format json | Checks pass, or name the concrete blocker with evidence |
| 3 | Scan one image from your own registry by digest | SBOM, vulnerability findings and an exposure working set for that digest |
| 4 | Wire the promotion gate into one existing CI pipeline | The pipeline halts or proceeds on the gate’s exit code — see the exit-code contract and the CI/CD gate flow |
| 5 | Promote one release into a staging environment | The promotion records the digest it deployed |
What passing proves: the product installs, scans, gates and deploys from its own documentation, on your artifacts, without us.
What it does not prove: anything about analysis quality, failure behaviour, or evidence portability. That is Stage 2.
Stage 2 — the adversarial week
Only if Stage 1 passed. Budget: four to eight engineer-days depending on how representative you make the test images. Each test is designed so that we do not control the outcome — you pick the images, you cause the failures, you verify on hardware we never touch.
2.1 A reachable and an unreachable vulnerability, chosen by you
Pick two of your own images: one where a known CVE sits in code your application actually calls, and one where the vulnerable package is present but the vulnerable path is not invoked. Scan both.
Expected: the reachable finding carries reachability evidence and ranks as the exposure that blocks; the not-invoked finding is reported with its evidence state shown — NOT OBSERVED is displayed as exactly that, never as “not vulnerable” and never silently dropped.
The tier vocabulary the analyzer is held to is published: the labeled ground-truth corpus in the source tree (src/Scanner/__Tests/__Datasets/toys/) defines tiers R0–R4 with a labels.yaml per service stating the expected tier and the rationale for it, across eight languages. Those labels are the contract our own tests hold the classifier to, and they are public — compare what your installation reports against them.
2.2 Missing evidence
Gate a digest for which some required evidence does not exist (for example, a policy that requires reachability evidence, evaluated against a scan that has none).
Expected: the check reports NOT EVALUATED and the gate treats it as non-passing. A check that could not run is never counted as a pass. If you can make missing evidence look like success, we want the reproduction.
2.3 Control-plane outage during a release
Stop the Stella Ops stack while a pipeline is waiting on the gate.
Expected: the gate blocks — the product fails closed. Then exercise the recovery path without starting the stack: a gate exception is a signed, scoped, time-limited object requiring a permission granted to nobody by default, a written justification and an expiry. Confirm that releasing during the outage required that explicit, attributable act — and that nothing released silently.
2.4 Drift between approved and running
Deploy an approved digest to a watched environment, then start a different digest on the same Docker daemon.
Expected: Watch reports the mismatch between the approved digest and the running digest for that service and environment. Note the scope honestly: Watch compares digests on the Docker daemons the agent service can reach — it is not an agent installed on your hosts, and it does not verify non-containerised processes.
2.5 Evidence that survives us
Export the Decision Capsule for one gated release (stella evidence — see the auditor guide). Copy it to a machine with no Stella Ops installation and no network path to one.
Expected: signature verification (cosign verify-blob) and content hashes (sha256sum) pass offline; the manifest pins every input by digest; replaying the decision from its sealed inputs reproduces the same verdict. Then alter one frozen input and confirm verification fails. A capsule that only verifies on the installation that produced it would prove nothing; this one is portable by design.
Reading the result
| If this failed | It means |
|---|---|
| Stage 1, any step | Not ready for your estate today. Cheap to have learned. |
| 2.1 | The analysis vocabulary and your expectation disagree — compare against the published corpus labels before concluding which of us is wrong, and send us the case either way. |
| 2.2 or 2.3 | Report it as a security defect. These are the product’s core claims. |
| 2.4 | Check the daemon is one the agent can reach; if it is, report it. |
| 2.5 | Report it. Portable evidence is the product’s reason to exist. |
What a fully green pilot proves: the product does what this site says it does, on your artifacts, under failure, with evidence you can carry away.
What it deliberately does not prove: analysis accuracy on estates unlike your test set, behaviour at scales you did not run, and the operational questions — support capacity, continuity, references — that belong to due diligence rather than to a technical pilot. Bring those to sales@stella-ops.org with your pilot results in hand; a pilot report is the strongest opening position a buyer can have.