Tester Architecture (ops-tester QA harness)

Technical architecture specification for src/Tester/— the stellaops-tester service.

STOP: this is not a product surface. stellaops-tester is a QA-isolation harness. It ships in a single opt-in compose overlay, it authenticates nobody, and it exposes credential-relay and fixture-injection endpoints. It must never run in a production stack. Everything below documents that boundary as much as the service itself.

0) Mission & boundaries

Mission. Own all QA fixture-import and per-scenario action endpoints, so those endpoints can be physically removed from production module images instead of living inside them behind an env-flag (src/Tester/AGENTS.md “Module Scope & Working Directory”). The premise is stated there bluntly: the presence of test endpoints inside production binaries “means a misconfigured prod deployment can expose test surface area”.

It also hosts the determinism oracle — POST /api/v1/tester/replay-and-compare — which fetches the same canonical artefacts for two run identifiers, canonicalises and hashes them, and returns identical, drift, or indeterminate (the scope’s upstream is not run-scoped, so no determinism claim can be made — see §5.1).

Boundaries.

• Stateless. No PostgreSQL, no migrations, no persistence library of any kind. The .csproj references exactly two StellaOps libraries — StellaOps.Configuration and StellaOps.Hosting (StellaOps.Tester.WebService.csproj) — plus the StellaOps.Router.AspNet reference that Directory.Build.props auto-injects into every Web SDK project.
• Public REST only. Every interaction with another module goes through that module’s published HTTP surface via a named HttpClient. There is not one ProjectReference to another module’s domain, repository, or storage assembly. If a scenario needs internal access, the receiving module must first grow a public, documented, test-shaped API (this is how Findings’ POST /api/v1/findings/ledger/advanced-assurance/append came to exist).
• Not the Replay engine. src/Replay/ is the production deterministic-replay service (tokens, manifests, feed snapshots, verdict replay — see replay/architecture.md). The Tester’s oracle is a thin REST comparator over other services’ read surfaces; it makes no verdicts, stores nothing, and knows nothing about replay tokens.

1) Why it has its own dossier (and why it is not folded into Replay or Tools)

Three properties force a standalone home:

1. It is a deployable service — its own image (stellaops/tester:dev), its own Dockerfile (devops/docker/Dockerfile.tester), its own compose overlay, its own Router serviceName (tester), and 19 routes. tools/ documents CLI utilities with no service surface; this is not one.
2. Only 1 of its 9 endpoint families is replay-related. Folding it into replay/ would orphan the Notify / Scanner / Findings / EvidenceLocker / Concelier / Authority / audit / cleanup families.
3. Most importantly: Replay is a production module. Documenting a “must never reach production” harness inside a production module’s dossier invites exactly the mistake the isolation invariant exists to prevent.

2) Source layout

src/Tester/
├─ AGENTS.md                                   # Production-isolation invariants (binding)
├─ StellaOps.Tester.WebService/
│  ├─ Program.cs                               # Host, named HttpClients, Router HELLO, route mapping
│  └─ Endpoints/
│     ├─ ReplayCompareEndpoints.cs             # /replay-and-compare + CanonicalJson
│     ├─ FixtureReadbackEndpoints.cs           # advanced-assurance-golden readback (seed-dir backed)
│     ├─ NotifyFixtureEndpoints.cs             # notify-import, timeline-readiness, delivery-state
│     ├─ ScannerFixtureEndpoints.cs            # reachability-import, sbom-import, self-sbom-import
│     ├─ FindingsFixtureEndpoints.cs           # findings-ledger-append
│     ├─ EvidenceFixtureEndpoints.cs           # evidence-import, portable-pack-build
│     ├─ ConcelierFixtureEndpoints.cs          # concelier-advisory-import
│     ├─ ReleaseFixtureEndpoints.cs            # release-seal, release-rollback
│     ├─ AuditFixtureEndpoints.cs              # audit inject-batch, synth-escalation-chain
│     ├─ AuthorityFixtureEndpoints.cs          # proxy-login (X-Forwarded-For injection)
│     └─ TesterCleanupEndpoints.cs             # cleanup/tenant/{tenantId}
└─ __Tests/StellaOps.Tester.WebService.Tests/  # 13 spec files; downstreams are stubbed (see §10)

3) Deployment: how the isolation is actually enforced

Runtime facts: container port 127.1.0.60:80 → 8080; network alias tester.stella-ops.local; seed volume stellaops-tester-seed mounted read-only at /var/lib/tester/seed (STELLAOPS_ADVANCED_ASSURANCE_GOLDEN_SEED_DIR); non-root UID/GID 10001.

4) Endpoint surface

All routes sit under /api/v1/tester/* (plus /health, /health/ready).

Ceilings worth knowing: audit inject-batch caps at 5 000 NDJSON lines and 512 chars per text field (AuditFixtureEndpoints.cs:32-33); proxy-login rejects CR/LF in the injected X-Forwarded-For.

5) The determinism oracle — POST /api/v1/tester/replay-and-compare

Contract. Body { runIdA, runIdB, scope[] }; headers Authorization: Bearer … + X-StellaOps-TenantId. For each requested scope the oracle first checks whether that scope’s upstream is actually run-scoped (ScopeContract.RunIdFiltered). If it is not, the scope is not fetched and not hashed — it returns indeterminate with the reason (§5.1). Otherwise the oracle issues GET {scopePath}?runId={runId} against the owning service — twice, once per run id — then:

1. Extracts records — root array, or the first matching envelope key (items / findings / events / packs / …), else the object itself as a single record.
2. Strips volatile fields recursively, per a per-scope allow-list (timestamps, etag, exportId, correlationId, …). For findings the physical row ids chainId/eventId are stripped too — the in-code rationale is that two physical runs of logically identical findings necessarily assign different ids, while eventHash and every logical content field stay in the hash, so drift detection is not weakened.
3. Canonicalises — RFC-8785-style: object keys sorted ordinal, no indentation, G17 for non-integral numbers, BOM stripped (CanonicalJson, ReplayCompareEndpoints.cs:735-847).
4. Sorts records by the scope’s sort key, then by canonical form, and SHA-256s the canonical array.
5. Compares the two digests → identical | drift; on drift it emits onlyInA / onlyInB / changed (with the changed field names).

Verdict precedence (top-level): drift > indeterminate > identical. The middle rung is what makes a false green impossible — the overall verdict can only be identical when every requested scope was actually measured against a run-scoped upstream. A real drift still outranks an unmeasurable scope, so indeterminate can never soften a genuine drift into “we could not tell”.

Error taxonomy (the response shape is a flat error object, not ProblemDetails): authorization_required 401 · tenant_required / request_required / run_id_required / scope_required / unsupported_scope 400 · run_not_found 404 (any upstream 404) · upstream_unavailable 502 (any other non-2xx, transport failure, or unparseable body).

5.1 Scope coverage — only findings is measurable (fixed: the rest now fail closed)

The oracle’s four scopes are not equally real. Verified against the upstream route tables at HEAD on 2026-07-12 (W3-E, SPRINT_20260712_008). RunIdFiltered is declared per scope in ScopeContracts (ReplayCompareEndpoints.cs) and pinned by ReplayCompareEndpointsTests.ScopeContract_DeclaresExactlyOneRunScopedUpstream:

What this used to do (the defect W3-E closed). The oracle fetched every scope unconditionally and hashed whatever came back:

• evidence-packs was a FALSE GREEN — the single worst failure mode an evidence system can have. Two completely different runs both read the same unfiltered pack list, the digests matched, and the scope reported identical: a determinism proof the oracle never performed. Executed and asserted against the pre-fix code, not argued (W3-E red-before-green probe).
• timeline-events did not 405, as previously documented here. ASP.NET route selection binds GET /api/v1/timeline/events to the parameterised GET /timeline/{eventId} with eventId = "events" (verified empirically); the lookup misses and Timeline answers 404 — which the oracle reported as run_not_found, i.e. it blamed the operator’s run ids for a scope that was never implemented.
• linksets 404’d on a path that does not exist → also run_not_found.
• Because run_not_found short-circuits the whole response, the invocation the runbook documented (all four scopes) returned HTTP 404 for every call, discarding even a healthy findings verdict.

Behaviour now — fail closed. A scope whose upstream is not run-scoped is never fetched, never hashed, and never identical. It returns:

{ "scope": "evidence-packs", "verdict": "indeterminate", "runIdFilteringSupported": false,
  "service": "EvidenceLocker", "reason": "…accepts no runId parameter; it lists every pack for the tenant…" }

with no sha256A/sha256B/rowCountA/rowCountB— an unmeasured scope makes no claim about a run — and it degrades the overall verdict to indeterminate. PerScopeDiff.RunIdFilteringSupported (the flag that was designed to warn about exactly this and was itself declared and never assigned) is now populated on every diff.

Consequence for anyone reading a replay verdict: scope: ["findings"] is still the only scope that yields a determinism proof. The difference is that the oracle now says so instead of fabricating a green. To make another scope measurable, the upstream must accept and filter runId (and the one-word RunIdFiltered flip must be accompanied by re-verifying its route in the guard test).

6) Auth & tenancy — the harness model (and why it is disqualifying for production)

This is the section to read before anyone proposes “just enable the tester in staging”.

• The service authenticates nobody. There is no AddAuthentication, no AddAuthorization, no UseAuthentication/UseAuthorization, and no RequireAuthorization anywhere in the project. Every route is explicitly .AllowAnonymous(). It carries no Authority resource-server library at all.
• Bearer tokens are relayed, never validated. replay-and-compare demands an Authorization: Bearer header, but TryGetBearerToken (ReplayCompareEndpoints.cs:635-654) only parses the scheme — the token is forwarded verbatim to the downstream service, which is what actually enforces scope. Authorization is therefore entirely downstream; the tester is a relay.
• Tenant comes from a caller-supplied header (X-StellaOps-TenantId, legacy X-Stella-Tenant), not from a stellaops:tenant claim. That inverts the platform’s tenancy invariant. It is a deliberate harness affordance — a fixture driver must be able to say which tenant it is seeding — and it is precisely why the surface cannot be reachable by an untrusted caller.
• POST /api/v1/tester/authority/proxy-login is an unauthenticated credential relay with source-IP spoofing: it takes {username, password, forwardedFor} and posts to Authority’s /connect/token with an attacker-chosen X-Forwarded-For (AuthorityFixtureEndpoints.cs:29-40, 83-90). It exists to drive Authority’s IP-based lockout/anomaly scenarios. Reachable in production, it is a brute-force proxy that defeats source-IP throttling.
• The Authority client_credentials flow is not implemented. In code the secret is read and discarded (Program.cs, _ = notifyAuthorityClientSecret;, with a comment deferring token acquisition to a later sprint). In practice the tester forwards the caller’s bearer, or a statically configured Tester:Notify:BearerToken. src/Tester/AGENTS.md invariant 3 used to assert the flow exists; it was corrected in W3-A, and the stellaops-tester Authority client is now disabled in the seed baseline (§8.2). See §11.

7) Invariants (binding — src/Tester/AGENTS.md)

1. Stateless — no schema ownership, no migrations, no persistence libs; seed artefacts come from a read-only mount.
2. Public REST only — no repository references, no internal write-throughs, no shared-schema reads into another module. A blocked scenario is resolved by the receiving module publishing a documented test-shaped API, never by reaching around it.
3. Opt-in compose only — never added to docker-compose.stella-ops.yml / docker-compose.stella-services.yml or any include: list.
4. Audit actor tester— QA-driven events stay filterable out of compliance views.
5. Any endpoint migrated out of a production module must land with Tier-2 integration tests proving parity before the legacy embedded path is removed.

8) Residual production footprint (status: the two headline residues are CLOSED)

The migration table in src/Tester/AGENTS.md marks all six stages DONE, but “DONE” there means the tester can now drive the scenario, not the production module has shed its test surface. The two residues recorded when this dossier was written were closed by SPRINT_20260712_008 / W3-A (2026-07-12); what remains is listed at the end.

8.1 QA fixture readback — now OFF by default in the canonical stack (FIXED)

Platform’s QaFixtureEndpoints.cs (and Notify’s / Scanner’s embedded fixture-import endpoints) still exist — the stage 6–7 soak (flag off, then delete) has not finished. But the default is fixed:

• Code was always default-deny. QaFixtureEndpoints.IsEnabled (:397-406) returns false unless Platform:QaFixtures:Enabled or STELLAOPS_QA_FIXTURES_ENABLED is explicitly set; the routes answer 503 "QA fixture readback disabled". The endpoints also sit behind RequireAuthorization (setup-admin / assurance-case-read) — they were never anonymous.
• The canonical compose stack was the thing that turned them on. devops/compose/docker-compose.stella-services.yml (platform-web) shipped STELLAOPS_QA_FIXTURES_ENABLED: "${STELLAOPS_QA_FIXTURES_ENABLED:-true}", so a production compose did answer GET /api/qa/fixtures/advanced-assurance-golden. That default is now false.
• Opt-in (QA / lab): either export STELLAOPS_QA_FIXTURES_ENABLED=true (the env indirection is preserved — one flag), or add the dedicated overlay: docker compose -f docker-compose.stella-ops.yml -f docker-compose.qa-fixtures.override.yml up -d, which sets the flag and the seed dir on platform-web explicitly. See devops/compose/README.md.

Guarded by QaFixtureEndpointsTests (Platform): the golden-readback, assurance-case and scenario-transaction routes all 503 when the flag is unset.

8.2 The stellaops-tester Authority client — disabled in the baseline (FIXED)

The original finding said this client shipped “with a guessable default secret”. That mechanism does not exist, and the truth was worse. Verified against src/ on 2026-07-12:

• stellaops-tester appears in no bootstrapClients list (devops/etc/authority/plugins/standard.yaml and the two sibling plugin configs). bootstrapClients + StandardClientProvisioningStore.Apply is the only code path that materialises a secret_hash from an env var (see migration 005). So nothing ever hashed TESTER_AUTHORITY_CLIENT_SECRET into Authority — that env var is read only by the tester container itself, which discards it. The S001 seed comment claiming the hash “is created on first boot by the bootstrap recovery routine” is false.
• The row (S001_v1_authority_operational_baseline.sql:261-277) shipped enabled = true, grant client_credentials, require_client_secret = true — and never set client_type, leaving it NULL. The token path gates the secret on the client type (ClientCredentialsHandlers.cs:337), never on require_client_secret. With client_type NULL and secret_hash NULL, a client_credentials request for stellaops-tester carrying no secret at all passed validation and minted a token with the row’s scopes — which the folded-in S010/S011 blocks had widened to findings:write, evidence:create, export.operator, export.viewer for tenant default. It was the only client_credentials client in the seed not covered by bootstrapClients, hence the only one left NULL-typed and secret-less.
• Fix — S039_disable_tester_client_baseline.sql(new forward-only seed migration; an applied migration’s checksum is immutable, so S001 was not edited). Two independent fail-closed layers: enabled = FALSE (→ Disabled → invalid_client at ClientCredentialsHandlers.cs:220), and client_type = 'confidential' so that a later re-enable still rejects without a provisioned secret ("Client secret is not configured.", :340-345). Scopes are left intact: the row is inert while disabled, and this keeps the lab opt-in a single reversible action.
• Nothing in src/ authenticates as stellaops-tester, so disabling it breaks no shipped path. The harness scopes it shared with stellaops-cli (S010/S011) remain reachable through stellaops-cli.
• Opt-in (lab only): add a bootstrapClients entry for stellaops-tester with clientSecretEnvironmentVariable: "TESTER_AUTHORITY_CLIENT_SECRET" and a real per-environment secret; the provisioning store then re-enables it, types it confidential, and hashes the secret on every boot. There is no default secret any more — the change-me-tester-secret literal was removed from Program.cs and from the compose overlay.

Proven by TesterClientSeedHardeningTests (StellaOps.Authority.Tests), which drives the real ValidateClientCredentialsHandler: the pre-S039 shape mints a token with no secret (characterization), the post-S039 shape is rejected invalid_client, and a re-enabled row still fails closed.

8.3 What is still open

The embedded fixture endpoints themselves (stages 6–7: delete). Notify’s AdvancedAssuranceFixtureImportEndpoints.cs and Scanner’s AdvancedAssuranceReachabilityFixtureEndpoints.cs / AdvancedAssuranceSbomFixtureEndpoints.cs carry no enable-flag at all — they are gated only by scope (notify.operator, scans:write), so they answer in a production stack for any caller holding those scopes. Platform’s flag does not cover them. Deleting them (or gating them behind the same flag) is a Notify/Scanner source change and is out of W3-A’s scope.

9) Configuration surface

Env prefix TESTER_; optional YAML at ../etc/tester.yaml / tester.yaml (Program.cs:34-43).

All defaults are in-cluster aliases — no public URL is a shipped runtime default.

10) Testing

dotnet build src/Tester/StellaOps.Tester.WebService/StellaOps.Tester.WebService.csproj
dotnet test  src/Tester/__Tests/StellaOps.Tester.WebService.Tests/StellaOps.Tester.WebService.Tests.csproj
docker compose -f devops/compose/docker-compose.tester.yml config      # overlay dry-run

Know what the suite does and does not prove. The specs run the real Program through WebApplicationFactory<Program> but replace every named HttpClient with a stub HttpMessageHandler (ReplayCompareEndpointsTests.ReplayStubHandler). They are a strong check on the oracle’s own logic — ReplayCompareDeterminismTests asserts that key-order + volatile-field differences still hash identically, that the same input hashes identically on every invocation, and that a real content change still drifts (no over-stripping).

A stub cannot prove an upstream exists — and this suite once asserted a world that did not. The original WithIdenticalRuns() stub keyed every scope’s response by runId, fabricating exactly the run-scoping that three of the four upstreams do not implement. The suite was green because it modelled the bug away; that is how the §5.1 false green survived. Two guards now stand in its place:

• ReplayDownstreamStub.WithRealisticUpstreams() models the upstreams as they actually behave (EvidenceLocker returns the same unfiltered list for every run id; Timeline/Concelier 404), and NonRunScopedUpstreams_AreIndeterminate_NeverIdentical asserts the oracle refuses to call them identical.
• ScopeContract_DeclaresExactlyOneRunScopedUpstream pins the RunIdFiltered set, so declaring a scope run-scoped is a deliberate act that forces you back to the upstream route table.

A stub still cannot tell you a route exists: any change to a scope contract needs a live-stack probe, not a green suite.

11) Known gaps / doc-vs-code drift

12) References

• Invariants (binding): src/Tester/AGENTS.md
• Original decision record: docs/qa/feature-checks/OPS_TESTER_SERVICE_ARCH_DECISION.md
• QA pipeline: docs/qa/feature-checks/FLOW.md
• Fixture contract: docs/qa/feature-checks/fixtures/advanced-assurance-golden-fixture-contract.md
• Production replay engine (distinct module): docs/modules/replay/architecture.md
• Audit ingest sink used by the audit fixtures: docs/modules/audit/architecture.md
• Overlay: devops/compose/docker-compose.tester.yml · Image: devops/docker/Dockerfile.tester