Operator Verify Runbook

Audience: operators and QA engineers collecting release-trust evidence from a running Stella Ops stack.

The Gateway owns the operator verification surface under /api/v1/operator/verify/*. These endpoints give operators a single front-door for release-trust evidence — buildinfo rollout, SBOM-to-digest binding, ledger chain integrity, token revocation, CRA provenance, and replay comparison — that previously required per-service APIs, CLI tools, or direct database access. This runbook documents each endpoint’s auth scope, headers, caching, status vocabulary, and the evidence-capture layout.

Evidence captures for Sprint 052 VERIFY-AGG-011 belong under:

docs/qa/feature-checks/runs/2026-05-18-operator-verify/evidence/

For each endpoint, keep one happy-path response and one service-down response. Name files by endpoint and condition, for example buildinfo-happy.json, buildinfo-service-down.json, replay-compare-happy.json, and replay-compare-service-down.json.

Shared Operator Rules

Base URL:

https://stella-ops.local

Authentication:

Headers:

Status vocabulary:

Buildinfo Rollout

Endpoint:

GET /api/v1/operator/verify/buildinfo

Auth scope: anonymous.

Cache and timeout:

Operator-trust gap:

Feature-check consumers:

Sample curl:

curl -sk https://stella-ops.local/api/v1/operator/verify/buildinfo \
  -o docs/qa/feature-checks/runs/2026-05-18-operator-verify/evidence/buildinfo-happy.json

Representative success response:

{
  "generatedAtUtc": "2026-05-18T10:00:00.0000000Z",
  "status": "ok",
  "services": [
    {
      "service": "platform",
      "status": "ok",
      "buildinfo": {
        "module": "platform",
        "gitSha": "abc123",
        "imageBuiltAt": "2026-05-18T08:42:00Z"
      }
    }
  ]
}

Representative status:missing response:

{
  "generatedAtUtc": "2026-05-18T10:00:00.0000000Z",
  "status": "ok",
  "services": [
    {
      "service": "scanner",
      "status": "missing"
    }
  ]
}

What missing means:

Wire-up reminder:

Sprint 052 VERIFY-AGG-002 follows Sprint 041’s BUILDINFO-WIDEN-001 pattern: add the StellaOps.Hosting project reference, map MapBuildInfoEndpoint next to the service’s health/readiness routes, and whitelist /buildinfo.json and /api/v1/buildinfo in any tenant-gating middleware. Sprint 052 did not rebuild images; live missing counts only drop after rebuilt images are deployed.

SBOM By Digest

Endpoint:

GET /api/v1/operator/verify/sbom-by-digest/{sha256}

Auth scope: operator.verify.read.

Tenant/header requirements:

Cache and timeout:

Upstream:

Operator-trust gap:

Feature-check consumers:

Sample curl:

TOKEN='<operator.verify.read bearer>'
DIGEST='sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'
curl -sk \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-StellaOps-TenantId: default" \
  "https://stella-ops.local/api/v1/operator/verify/sbom-by-digest/$DIGEST" \
  -o docs/qa/feature-checks/runs/2026-05-18-operator-verify/evidence/sbom-by-digest-happy.json

Representative success response:

{
  "generatedAtUtc": "2026-05-18T10:05:00Z",
  "status": "ok",
  "digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
  "sbomIds": [
    "sbom:stellaops/scanner-web@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
  ],
  "lastIndexedAt": "2026-05-18T09:58:17Z"
}

Representative missing response for an unknown digest:

{
  "generatedAtUtc": "2026-05-18T10:05:00Z",
  "status": "missing",
  "digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
  "sbomIds": [],
  "lastIndexedAt": null
}

What missing means:

Ledger Chain

Endpoint:

GET /api/v1/operator/verify/ledger-chain/{ledgerId}

Auth scope: operator.verify.read.

Header requirements:

Cache and timeout:

Upstream:

Operator-trust gap:

Feature-check consumers:

Sample curl:

TOKEN='<operator.verify.read bearer>'
LEDGER_ID='11111111-1111-1111-1111-111111111111'
curl -sk \
  -H "Authorization: Bearer $TOKEN" \
  "https://stella-ops.local/api/v1/operator/verify/ledger-chain/$LEDGER_ID" \
  -o docs/qa/feature-checks/runs/2026-05-18-operator-verify/evidence/ledger-chain-happy.json

Representative verified response:

{
  "generatedAtUtc": "2026-05-18T10:10:00Z",
  "status": "verified",
  "rootHash": "sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
  "verifiedAt": "2026-05-18T10:09:59Z",
  "entryCount": 128,
  "firstTamperedSeq": null
}

Representative tampered response:

{
  "generatedAtUtc": "2026-05-18T10:10:00Z",
  "status": "tampered",
  "rootHash": "sha256:changed",
  "verifiedAt": "2026-05-18T10:09:59Z",
  "entryCount": 128,
  "firstTamperedSeq": 42
}

Representative status:missing response:

{
  "generatedAtUtc": "2026-05-18T10:10:00Z",
  "status": "missing",
  "rootHash": null,
  "verifiedAt": null,
  "entryCount": 0,
  "firstTamperedSeq": null
}

What missing means:

Token Revocation

Endpoint:

GET /api/v1/operator/verify/token-revocation

Auth scope: operator.verify.read.

Header requirements:

Cache and timeout:

Upstreams:

Operator-trust gap:

Feature-check consumers:

Sample curl:

TOKEN='<operator.verify.read bearer>'
curl -sk \
  -H "Authorization: Bearer $TOKEN" \
  https://stella-ops.local/api/v1/operator/verify/token-revocation \
  -o docs/qa/feature-checks/runs/2026-05-18-operator-verify/evidence/token-revocation-happy.json

Representative success response:

{
  "generatedAtUtc": "2026-05-18T10:15:00Z",
  "status": "ok",
  "bundle": {
    "entryCount": 12,
    "digest": "sha256:0123456789abcdef",
    "issuedAt": "2026-05-18T10:14:30Z",
    "signingKid": "authority-revocation-2026-05"
  },
  "propagation": [
    {
      "service": "gateway",
      "lastSeenRevocationAt": "2026-05-18T10:14:35Z",
      "lagSeconds": 25,
      "status": "ok"
    }
  ]
}

Representative status:missing response:

{
  "generatedAtUtc": "2026-05-18T10:15:00Z",
  "status": "missing",
  "bundle": {
    "entryCount": 0,
    "digest": null,
    "issuedAt": null,
    "signingKid": null
  },
  "propagation": []
}

Propagation telemetry:

Sprint 052 VERIFY-AGG-006 ships the Authority status endpoint, the Gateway read wrapper, and the gateway-side propagation writer. The Gateway runs a RevocationPropagationBeaconService (BackgroundService) that publishes its own verify:revocation:propagation:gateway Valkey hash entry every 30 s with a 15-minute TTL. The dispatch path is NOT instrumented for revoked-token interception (no safe way to inspect bundle membership at request time without caching bearer values), so the gateway row is a heartbeat-style attestation that the gateway is alive and would observe revocations. Downstream services may add their own IRevocationPropagationWriter implementations in follow-up work; the verify endpoint surfaces additional rows transparently as they appear.

What missing or stale means:

CRA Provenance

Endpoint:

GET /api/v1/operator/verify/cra-provenance/{tenantId}

Auth scope: operator.verify.read.

Tenant/header requirements:

Cache and timeout:

Upstream:

Operator-trust gap:

Feature-check consumers:

Sample curl:

TOKEN='<operator.verify.read bearer>'
curl -sk \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-StellaOps-TenantId: default" \
  https://stella-ops.local/api/v1/operator/verify/cra-provenance/default \
  -o docs/qa/feature-checks/runs/2026-05-18-operator-verify/evidence/cra-provenance-happy.json

Representative success response:

{
  "tenantId": "default",
  "exportId": "cra-techdoc-20260518-001",
  "generatedAtUtc": "2026-05-18T10:20:00Z",
  "status": "ok",
  "fields": [
    {
      "path": "manufacturer.name",
      "sourceRef": "platform:tenant/default/compliance-profile",
      "signerRef": "signer:authority/kid-1",
      "signedAt": "2026-05-18T10:19:00Z"
    }
  ],
  "unprovenancedFieldCount": 0
}

Representative unprovenanced response:

{
  "tenantId": "default",
  "exportId": "cra-techdoc-20260518-001",
  "generatedAtUtc": "2026-05-18T10:20:00Z",
  "status": "unprovenanced",
  "fields": [
    {
      "path": "support.eolDate",
      "sourceRef": null,
      "signerRef": null,
      "signedAt": null
    }
  ],
  "unprovenancedFieldCount": 1
}

Representative status:missing response:

{
  "tenantId": "default",
  "exportId": null,
  "generatedAtUtc": null,
  "status": "missing",
  "fields": [],
  "unprovenancedFieldCount": 0
}

What missing or unprovenanced means:

Replay Compare

Endpoint:

POST /api/v1/operator/verify/replay-compare

Auth scope: operator.verify.admin.

Header requirements:

Cache and timeout:

Upstream:

Operator-trust gap:

Feature-check consumers:

Sample curl:

TOKEN='<operator.verify.admin bearer>'
curl -sk -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-StellaOps-TenantId: default" \
  -H "Content-Type: application/json" \
  --data '{"runIdA":"run-a","runIdB":"run-b","scope":["evidence-packs","findings","timeline-events","linksets"]}' \
  https://stella-ops.local/api/v1/operator/verify/replay-compare \
  -o docs/qa/feature-checks/runs/2026-05-18-operator-verify/evidence/replay-compare-happy.json

Representative identical response:

{
  "verdict": "identical",
  "runIdA": "run-a",
  "runIdB": "run-b",
  "computedAtUtc": "2026-05-18T10:25:00Z",
  "perScopeDiffs": []
}

Representative drift response:

{
  "verdict": "drift",
  "runIdA": "run-a",
  "runIdB": "run-b",
  "computedAtUtc": "2026-05-18T10:25:00Z",
  "perScopeDiffs": [
    {
      "scope": "timeline-events",
      "expectedHash": "sha256:old",
      "actualHash": "sha256:new"
    }
  ]
}

Representative status:missing response:

{
  "generatedAtUtc": "2026-05-18T10:25:00Z",
  "status": "missing",
  "verdict": null,
  "perScopeDiffs": []
}

What missing means:

VERIFY-AGG-011 Capture Checklist

For each of the six endpoint sections above: