Scanner SCN-8 tenant-feed cutover and bootstrap

Audience: Scanner, Authority, Findings, Integrations, Policy, database, and live-deploy operators executing SPRINT_20260722_029 SCN-8 in an owner-attended window.

Purpose: after sprint 017 SCN-5 has moved Scanner to stellaops_scanner, activate SCN-7’s tenant-partitioned completion feed, rebuild the configured consumer tenants through supported forced scans, prove X11-X13 without cross-service database access, and retain a forward-only rollback boundary.

This is an operator procedure, not authorization to execute the live window. Repository-side staging is complete only when SCN-5’s source, data-move, revocation, and forcing-function receipt is green. Image publication, service recreation, secret changes, forced live scans, and old-stream deletion remain confirm-first actions. Stop at the first failed assertion. Never expose a secret value in a transcript; this runbook names only FINDINGS_SCANNER_PROJECTION_CLIENT_SECRET, INTEGRATIONS_AUTHORITY_CLIENT_SECRET, POLICY_ENGINE_CLIENT_SECRET, and STELLAOPS_RELEASE_DISPATCH_CLIENT_SECRET.

Invariants and stop conditions

Evidence directory and secure bearer handling

Create a unique receipt directory under tmp/qa/scn-8/<window-id>/ and first prove it is ignored with git check-ignore. Record commands and redacted responses there. Keep tenant bearers in a session-only protected file or credential helper outside the repository; do not put a bearer in a shell command line, environment dump, Compose render, or receipt. Destroy the session material when the window closes.

Phase 0 — read-only inventory

  1. Capture the source revision, Compose project/config-file labels, candidate image digests, current image digests, health, and the configured tenant of every intended Findings and Integrations consumer. Record only whether each named secret is present, never its value. Keep two ordered Compose lists: the live-label list for evidence/rollback and the candidate list for recreation. Every candidate path must exist at the candidate revision.

    Record configured consumer intent separately from the preliminary successful-target inventory. An empty tenant on a disabled Findings instance is not a default tenant selection. Policy’s Scanner owner client selects the tenant per evaluation; its catalog-replication auth tenant is not a Scanner consumer inventory. Neither configuration nor a descriptor allow-list freezes the authoritative bootstrap manifest; Phase 1 does that only after fencing and stopping the producer.

    Check every path in each service’s live-label list, not only a known retired overlay. Record existence, source provenance and the disposition of each missing input. Trace a tracked retirement through its delete/fold commit and verify the replacement service fragment in the candidate source. Known Integrations retirements are:

    Historical input under devops/compose/Source fold into docker-compose.stella-services.yml
    docker-compose.integrations-consolidated.yml413ec660c6: own-database, projection and doctor configuration
    docker-compose.integrations-plugins.override.yml42d910621e: signed plugin composition

    This table is not an exhaustive live inventory. Preserve the exact historical bytes and SHA-256 in the ignored receipt directory for recovery only, recording the original path, provenance and recovery-copy mapping. Prefer an existing verified deployed-input capture; a pre-delete Git blob is sufficient only when its deployed-byte provenance is established. An ignored input with no recoverable verified bytes, or an unexplained missing path, is a stop. Do not replace it with plausible current YAML or restore a retired file to the candidate tree.

    Omit only source-proven retired inputs from the candidate list. Verify its rendered service preserves the intended owner-connection names, signed bundle mounts, trust root, scratch volume, process-isolation settings and ScannerProjection__* keys, with the required :8444 owner URL. Record every other candidate-list change and its parity proof; do not reorder files from memory.

  2. From Scanner’s own stellaops_scanner database connection, export a preliminary newest successful, resolvable row for every exact tenant/digest pair. Do not use another service role for this query:

    SELECT DISTINCT ON (tenant_id, target_digest)
       tenant_id, scan_id, target_reference, target_digest, updated_at
FROM scanner.scan_runtime_state
WHERE status = 'Succeeded'
  AND target_reference IS NOT NULL
  AND target_digest IS NOT NULL
ORDER BY tenant_id, target_digest, updated_at DESC, scan_id DESC;

    Resolve null reference/digest cases before proceeding: POST /api/v1/scans needs an addressable target. This is inventory only. The authoritative bootstrap manifest is exported again after the intake fence is proven and the worker is stopped in Phase 1.

  3. Capture tenant-stream state from Scanner’s own stellaops_scanner database. SCN-5 intentionally copied no eventing.*, so this query must not be presented as evidence about the old global stream:

    SELECT stream, stream_epoch, next_seq, updated_at
FROM eventing.stream_state
WHERE stream LIKE 'scanner.scans.tenant.%'
ORDER BY stream;

SELECT stream, stream_epoch, count(*) AS rows, min(seq) AS min_seq, max(seq) AS max_seq
FROM eventing.outbox
WHERE stream LIKE 'scanner.scans.tenant.%'
GROUP BY stream, stream_epoch
ORDER BY stream, stream_epoch;

SELECT stream, consumer_id, stream_epoch, seq, registered_at, reported_at
FROM eventing.remote_stream_consumers
WHERE stream LIKE 'scanner.scans.tenant.%'
ORDER BY stream, consumer_id;

  4. Separately, through the database-operator read-only path, capture the retired global stream from source stellaops_platform. Never give this connection to Scanner or a consumer service:

    SELECT stream, stream_epoch, next_seq, updated_at
FROM eventing.stream_state
WHERE stream = 'scanner.scans';

SELECT stream, stream_epoch, count(*) AS rows,
       min(seq) AS min_seq, max(seq) AS max_seq,
       min(occurred_at) AS oldest_event, max(occurred_at) AS newest_event
FROM eventing.outbox
WHERE stream = 'scanner.scans'
GROUP BY stream, stream_epoch
ORDER BY stream_epoch;

SELECT stream, consumer_id, stream_epoch, seq, registered_at, reported_at
FROM eventing.remote_stream_consumers
WHERE stream = 'scanner.scans'
ORDER BY consumer_id;

  5. Record the complete effective Scanner retention declaration from the candidate render and from the running scanner-web environment, redacting no values because these are durations/counts, not secrets. Require enabled, window, batch size, maximum batches/pass, interval, RequirePublished=false, superseded-epoch policy, and a positive remote-consumer lease shorter than the window in the candidate. A running estate with no declaration is an explicit activation delta; a partial running declaration, or any missing/partial candidate declaration, is a NO-GO. Identify the configuration provider for each value, including any image JSON/YAML or command-line override; an absent environment key alone does not establish the effective setting. The ScannerScansRetentionActivation.ValidateConfiguration source returns without registering a pruner when Enabled is false or absent. Successful startup therefore does not prove a bounded declaration or reject every partial disabled configuration. This operator gate requires the complete deployment-owned declaration; library defaults and commented examples are not choices.

  6. Confirm exact Authority tenant membership and audiences for every client used by each manifest tenant. Descriptor YAML is desired startup configuration, not evidence of the running Authority catalog. The SCN-5 preflight found only tenant default; do not manufacture e2e-lab from a descriptor allow-list. Query authority.tenants, authority.client_tenants, and each client’s persisted audience property in the live Authority database. Include another tenant only when the tenant row and the exact client membership both exist. A missing membership or non-intersecting Scanner audience is a stop, not a reason to select another tenant or add a tenant header.

Phase 1 — freeze and converge

  1. Before stopping the producer, recreate scanner-web only on its current immutable image and exact current configuration plus an ignored overlay setting SCANNER_SCANNER__CUTOVER__INTAKEWINDOWID=<window-id>. Keep scanner-worker unchanged. Submit a valid request without stellaops.cutover.window and one with the wrong marker; both must return 503 problem+json with detail=scan_intake_fenced, and neither may create a runtime row. Do not record the configured marker in either response. Use the recorded live-label list for this current-image recreate. If that service’s list has a missing input, use only the byte-verified recovery copy and explicit path mapping accepted in Phase 0; verify the resulting service parity before recreation. Do not substitute the candidate list before the fence is proven.
  2. Stop Scanner worker, then Findings worker, Integrations web, and Policy evaluation traffic. Re-export the Scanner query from Phase 0 step 2 and repeat both database inventories from steps 3 and 4 now. This post-fence CSV is the authoritative bootstrap manifest; review and hash it plus both state receipts, and do not widen the manifest during execution.
  3. Build the required candidate images locally and record their immutable image IDs/digests. Do not publish an image to any registry or other outward artifact destination in SCN-8. Recreate the candidate services only inside the recognized live window after the live-deploy operator confirms those recorded identities. Use the validated candidate Compose list from Phase 0, not a missing retired overlay and not a plausible list reconstructed from memory.
  4. Start Authority first and wait for startup migration S053. Verify the stable permission id now names scanner:projection:read; the three named clients have the expected audiences; Findings and Integrations have only the tenant projection scope; Policy has scanner:read; none retains scanner:projection:read-all.
  5. Start Scanner web with the same intake fence and worker, then Findings web and worker, Integrations web, and Policy engine. Verify Findings migration 012_findings_scanner_tenant_projection.sql and all normal startup migrations converged. An enabled consumer with an empty tenant, authority endpoint, or named secret must fail closed; do not bypass validation.
  6. Confirm all three clients resolve Scanner directly on :8444, and that no active Findings, Integrations, or Policy process contains a Scanner DSN.
  7. Prove the RAR-11 route contraction on the exact scanner-web candidate. The authenticated scan submit/status, completion-feed, consumer-cursor, and digest-detail routes must exist. Direct Scanner requests to /api/v1/registries/images/search and /api/v1/registries/images/digests must return 404; Platform owns those registry contracts. Never restore Scanner’s deleted RegistryEndpoints.cs or use those paths to resolve manifest targets.

Phase 2 — per-tenant bootstrap through Scanner

There is no safe bulk republish command. stella scan submit-list is not a substitute: its source route records an SBOM-source run but does not dispatch a Scanner job. For each manifest tenant, mint the existing stellaops-release-dispatch confidential client a token with the exact tenant and only the two Scanner scopes needed for enqueue plus status polling:

POST /connect/token
grant_type=client_credentials
client_id=stellaops-release-dispatch
client_secret=<from STELLAOPS_RELEASE_DISPATCH_CLIENT_SECRET>
scope=scanner:scan scanner:read
tenant=<exact-tenant>

Verify the token’s exact tenant claim and audience before use, without recording the token. Then call the normal enqueue route for each reviewed row with a fresh stable clientRequestId:

POST http://scanner.stella-ops.local:8444/api/v1/scans
Authorization: Bearer <session-only tenant bearer>
Content-Type: application/json

{
  "image": {
    "reference": "<target_reference>",
    "digest": "<target_digest>"
  },
  "force": true,
  "clientRequestId": "scn8:<window-id>:<tenant-safe-index>:<target-safe-index>",
  "metadata": {
    "stellaops.cutover.window": "<window-id>",
    "scn8.bootstrap": "true",
    "scn8.window": "<window-id>"
  }
}

Use the secure bearer mechanism from the evidence-directory section rather than copying the placeholder command literally. Record the accepted scan id without the bearer. Poll GET /api/v1/scans/{scanId} with the same tenant identity until Succeeded; a failed or wrong-tenant response stops that target. Then prove exactly one v2 completion is visible on that tenant’s feed, with the expected tenant, digest, purlSetSha256, and findingSetSha256. Use the actual Findings or Integrations projection identity for that feed proof: release-dispatch does not carry scanner:projection:read.

Complete every target for a tenant before activating that tenant’s consumers. The first fresh clientRequestId creates a distinct deterministic scan id. If the response is lost or that target must be retried ambiguously, reuse the same id: force: true then resets/re-enqueues the same durable row. Do not invent another id for an ambiguous retry. The production coordinator, not the operator, creates the Pending state and eventual outbox row. Keep the intake fence armed throughout bootstrap, consumer activation, and the pre-unfence parity checks; the exact window metadata above is the only admitted submit path.

Phase 3 — activate and prove X11-X13

For every configured consumer instance/tenant:

  1. X11 Findings: activate the Scanner security projection. Verify its tenant-specific lease, inbox, projection, and checkpoint in stellaops_findings; then verify Scanner’s remote cursor advances only after the local transaction is visible from a second connection. A token for tenant A must not read or report tenant B’s stream.

  2. X12 Integrations: activate the registry-image Scanner projection. Verify its tenant-specific projection and checkpoint in stellaops_integrations, the same post-commit cursor ordering, and the same wrong-tenant refusal. Do not treat a healthy process with a stationary cursor as proof.

  3. X13 Policy: make one real tenant-bound evaluation that requires Scanner digest detail and record the bounded owner response and decision. Then perform a safe negative by making the owner seam unavailable to the candidate instance or using an invalid tenant identity; Policy must fail closed before evaluation and must not reuse another tenant’s facts.

  4. For each tenant, compare eventing.stream_state.next_seq - 1 with both expected remote consumer cursors. Investigate any gap rather than advancing a cursor manually.

  5. Run the read-only ADR-039 matrix and verify the printed coverage includes every service database on the shared server:

    bash tools/scripts/deploy/postgres/probe-database-isolation.sh \
  --env-file <window env file> --container <postgres container>

    Exit zero is insufficient if the printed coverage omits a database. Append missing connection values only from the live container into an ignored temporary env file as documented in the script; never copy values into this runbook or a receipt.

Phase 4 — soak, closure, and old-stream disposition

Rollback

If failure occurs before any tenant completion is emitted, stop the candidate consumers and Scanner worker, restore the exact recorded prior images/configuration (including byte-verified historical input copies only through their Phase 0 mapping and only for the service lists that used them), and verify their prior behavior. Keep Scanner intake fenced until that verification finishes. If a tenant completion has been emitted, do not restore the global feed or downgrade Authority: freeze producers/consumers, preserve both databases and receipts, and fix forward from the recorded tenant epoch/head. Never delete tenant events or manufacture cursor progress to make rollback look clean.