Scanner Hub Advisory v2 cutover, acceptance, rollback, and soak

SCN-ID-5 is source-ready, not live-accepted. Run this procedure only in the recorded Scanner reachability window. It changes the scanner-worker image/configuration and refreshes the ReleaseOrchestrator consumer to the same accepted source before using fresh test scans/releases. It does not authorize a Scanner database/schema change, evidence rewrite, Concelier retirement, or SCN-ID-6. Q-32 carries one narrow Hub-owner exception: vulnerabilities-web startup may apply pending forward migration 005 only after the exact VULN-B9 target/dependent preflight below passes.

Owner sprint: SPRINT_20260722_017, task SCN-ID-5.

Offline record validator: verify-scanner-hub-v2-cutover-record.ps1. The validator reads only local captures, accepts no credentials, performs no network/Docker/Git/database operation, and fails closed before each phase.

Safety invariants

• Deploy a published commit only after its blocking Gitea jobs are final and green for that exact SHA. Queued, pending, cancelled, skipped, or a green run for another SHA is not acceptance.
• Select both scanner-worker and release-orchestrator by immutable repository digest and record each digest and local image ID. Both /app/buildinfo.json files must name the same full SHA, worktreeState=clean, and dirtyFileCount=0.
• Capture the current worker image ID, exact ordered compose -f list, its analyzer/public-trust cohort, and only the reachability-related configuration before any recreate. Never dump the container’s full environment; it can contain secrets.
• The current honest rollback posture has reachability and Hub v2 emission both disabled. The retired Concelier endpoint is not a fallback.
• Each forcing case uses a new scan ID, new release ID, and freshly emitted stellaops.reachability-report@v2 row. Never reuse or edit an evidence row.
• Rollback restores the exact captured image ID, analyzer/public-trust cohort, and compose/configuration fingerprint. It reads the three stored v2 rows again and proves their hashes are unchanged. It never deletes, updates, converts, or backfills evidence.
• Tokens are session-only input through the operator’s approved secret flow. Do not put a password/token in the command line, record, capture, shell history, or committed file.

Any Scanner startup failure that cites this file is a stop condition. Correct the selected mode/URL or restore the baseline; do not bypass the validator.

Record directory and phase gate

Create an ignored directory such as tmp/scratch/scn-id-5/<window-id>/. Keep the acceptance record and every referenced capture below that directory. Confirm it is ignored before writing:

git check-ignore -q -- 'tmp/scratch/scn-id-5/probe'
if ($LASTEXITCODE -ne 0) { throw 'tmp/scratch/scn-id-5 is not ignored' }

The record has schemaVersion: 1 and changeId: "SCN-ID-5". The validator is the executable field contract; it reports the first missing or inconsistent field and links back here:

pwsh tools/scripts/deploy/customer/verify-scanner-hub-v2-cutover-record.ps1 `
  -RecordPath tmp/scratch/scn-id-5/<window-id>/acceptance-record.json `
  -Phase Preflight

Advance through Preflight, Baseline, Consumer, Forcing, Rollback, then Soak. A later phase revalidates every earlier phase.

1. Pin source, CI, and candidate images

Record:

1. source.gitSha: the full lower-case 40-hex commit SHA.

2. source.publishedRef: refs/heads/main.

3. source.ci: the same SHA, status=completed, conclusion=success, final=true, the Gitea job/run receipt URL, a reviewed normalized receipt file beneath the record directory, and its SHA-256. Open the final job/run, verify those four facts in Gitea, and transcribe only this stable nonsecret projection from that result (do not copy the duplicate record fields back into the receipt):

   {
     "gitSha": "<full-lower-case-40-hex-sha>",
     "status": "completed",
     "conclusion": "success",
     "final": true,
     "receipt": "<exact-Gitea-job-or-run-URL>"
   }
   

   The validator parses the saved projection and cross-checks every field against source.gitSha and source.ci; hashing arbitrary bytes is not sufficient.

4. candidateImage.reference and .repoDigest: the same Scanner Worker repository@sha256:<64-hex> selector; .imageId: the exact local image ID.

5. Scanner Worker docker image inspect and /app/buildinfo.json captures plus their SHA-256 values. Record the buildinfo SHA, clean state, and zero dirty files. The validator parses both captures, requires exactly one inspect row, and cross-checks its Id and RepoDigests plus the buildinfo SHA/state/count.

6. releaseOrchestratorImage carries the same candidate fields and captured receipts for a clean ReleaseOrchestrator image from that exact SHA. It also records the outgoing immutable image ID/reference and exact ordered Compose list so a failed consumer refresh can be rolled back before forcing.

Use a networkless, non-root read of each candidate for buildinfo. Do not start either application during this proof. Repeat this capture for the Worker and ReleaseOrchestrator using distinct filenames:

$candidate = '<repository>@sha256:<digest>'
docker image inspect -- $candidate |
  Set-Content -LiteralPath '<record-dir>/candidate-image-inspect.json' -Encoding utf8NoBOM
docker run --rm --network none --entrypoint /bin/sh -- $candidate -c 'cat /app/buildinfo.json' |
  Set-Content -LiteralPath '<record-dir>/candidate-buildinfo.json' -Encoding utf8NoBOM

Hash the exact saved bytes. Do not hash a later pretty-printed copy.

1.1 Complete the Policy owner-read prerequisite before recapturing the baseline

Policy must resolve persisted release components through RO’s authenticated GET /api/v1/release-orchestrator/releases/{releaseId}/components, using its own service identity with release:read. It must not query an RO table through Policy’s database. The local origin is HTTPS on port8443; preserve normal TLS and exact tenant propagation. Missing, empty or malformed owner responses are unavailable, not clean-zero findings. Verify against source bf9deddee1eae37e8610882df0dda7653ef11543 with the Policy ReleaseComponentsClientTests / ReleaseComponentsLookupIntegrationTests and RO ReleaseComponentsParityTests before using another source.

devops/compose/local-scn-runtime.json is the sole runtime selection for Policy, RO and Worker; the dated RAR manifest now owns only Scanner Web, Platform and Gateway. The SCN controller’s Policy/RO phases do not activate Worker. An image-only Worker pin is not a complete cutover command: matching analyzer/trust cohorts and every phase below remain required.

Before replacing either outgoing Policy/RO container, use the canonical tools/scripts/deploy/preserve-policy-ro-runtime-secrets.cjs from primary main. Its metadata-only preflight binds the exact original, helper bytes, runtime model and selected unmounted stores. Reviewed preservation stops but does not remove that original, copies only its allowed key-ring/token-cache files into restricted .local-secrets homes, and verifies stopped-source and destination hashes. It rejects private worktrees and uncommitted helper bytes. Old preflights are invalid after a helper change or original restart.

The three new binds are Policy’s DataProtection key ring and RO’s key ring and token cache. They must retain the original destinations, use RW access and create_host_path:false. No secret payload belongs in a receipt. Before adoption, prove through each bind that the actual image’s uid10001 can read the retained files and create, rename and remove a unique nonsecret marker; then recheck that the retained secret-file hashes are unchanged. Host ACL checks alone do not establish Linux access or key rotation.

Do not treat the Compose flag as a missing-path detector. A create-only probe on Compose5.3.1/Docker Desktop29.6.2 created absent host directories even with explicit false. The controller independently requires existing, regular retained inputs and complete byte inventories, then repeats mounted-input and source checks immediately before recreation. Existing unchanged binds may retain their omitted creation setting; this is not interpreted as false or allowed for new stores.

Adopt and verify the canonical baseline profiles first, then use separately reviewed image-only Policy/RO candidates. Preserve all other environment values, mount/volume/network identities and process settings, including RO’s disabled federation posture. A failed or uncertain swap requires exact-state inspection and explicit scoped recovery, never an estate-wide retry. After successful prerequisite deployment, capture new SCN Baseline and Consumer evidence. Earlier RAR or pre-prerequisite receipts cannot be reused as those phase captures.

For the subsequent Policy tenant/Router repair, use the manifest’s bounded policy-engine.update generation. All-null source/image fields mean preparation only. After independently verifying the new clean Policy image and exact-source CI, the policy-update phase admits only that image over the retained 2a61 candidate chain. policy-update-recovery returns to 2a61; the original Policy 3a17 and RO ebd3 recovery paths remain separate. Do not replace the original manifest rows or re-run their already completed secret preservation. Keep the existing protected key-ring home and current contents.

The reviewed prior-receipt bridge binds the original successful receipts and retained commit bytes. It allows the exact pinned Platform-only queue addition to the common Compose file, after checking the entire remaining YAML and unchanged Policy/RO rendered models. Changed protected inputs, another service setting, an unpinned file revision or a different observed container still fails. Run a fresh actual image RW proof and reviewed preflight; offline bridge controls and a synthetic-candidate audit are not deployment admission.

Prerequisite recovery limits

The controller automates recovery only from a positively identified candidate replacement and its matching apply receipt. Failed baseline adoption, an absent or unobserved replacement, an unexpected identity, or failed recovery returns manual-recovery-required. Do not feed such a receipt into the candidate-only recovery path or retry an unscoped Compose command.

Before stopping an original, retain its exact ID/image, the successful pre-stop inspection, the reviewed admission preflight and both canonical chains. For an unsupported failure, inspect the exact original plus every container with the selected service/project labels and retain that state:

• If the unchanged original still exists and no replacement is present, the bounded fallback is to start that exact original ID, then verify its health and identity. This invalidates the stopped-source preservation capture; it does not complete baseline adoption. Keep the window open and retain the protected copy until its disposition and a new capture are verified.
• If the original is gone, do not infer rollback from an image tag. Reconcile the failed receipt with the actual replacement or proved absence, verify the recorded engine/network/volume identities and unchanged non-target services, and revalidate all canonical/protected input hashes. Preserve current key-ring and token-cache contents; never restore an older copy over live operational state. Repeat the baseline-image uid10001 read/write/rename proof as applicable. Only after this explicit operator review may the selected service be restored using its manifest rollbackFiles, project stellaops, the two named protected environment files, and up -d --no-deps --force-recreate --no-build --pull never. The target is only policy-engine or release-orchestrator; no other service, volume deletion, network recreation or orphan removal is authorized.

If any preserved input, target identity or physical resource cannot be proved, stop at that exact state. Record manual recovery separately; it is not an automated controller acceptance receipt. Verify the resulting immutable image, canonical labels, configuration parity, health and owner-API behavior before resuming. This fallback must not be represented as a rollback drill that ran.

2. Capture the reversible baseline

If the Worker still names temporary Compose definitions, first prepare a committed, Worker-only baseline chain and prove that it preserves the existing image, all environment values, mounts, process/security/health settings, ports and physical networks. The local baseline profile is devops/compose/docker-compose.local-scanner-worker-baseline.yml, layered after docker-compose.stella-ops.yml and before docker-compose.existing-networks.override.yml. It is opt-in preparation, not an activation command; do not use the four-service RAR replay helper for Worker. Recreate only Worker after the applicable source gates and complete replay preflight pass, then capture the new actual baseline below. Freeze those baseline file bytes through rollback/reapply. A capture of the old container or old list cannot be reused after canonicalization.

Image and mount parity alone do not establish safe recreation. Inspect the Worker’s writable layer and classify every populated path against its actual producer/consumer: a path named tmp is not evidence of disposability. For surface artifact caches, verify that every cached artifact belongs to a published manifest and freshly read/hash-check that manifest and all referenced durable objects. Preserve named CAS/secret-provider volumes without reading credential payloads. Unlinked artifacts or other unclassified state are a stop. After graceful Worker quiescence, repeat the nonsecret file inventory and durable readback before discarding the old container. Use metadata-only checks for empty credential-cache directories, never copy their contents. If this check fails before recreation, restart the untouched original container and record the failure; an image rollback cannot recover an unverified discarded writable layer.

Resolve the running container explicitly; the normal local name is stellaops-scanner-worker. Generate one stable normalized baseline snapshot from a reviewed docker inspect result. The configuration snapshot has this nonsecret shape; the separately required analyzer-cohort capture follows below:

{
  "capturedAtUtc": "<ISO-8601-UTC>",
  "workerContainer": "<resolved-container-name>",
  "composeProject": "<com.docker.compose.project-label>",
  "imageId": "sha256:<64-hex>",
  "imageReference": "<same-image-ID-or-repository-digest>",
  "composeConfigFiles": ["<first-file>", "<second-file>"],
  "reachabilityEnabled": false,
  "reachabilityEnabledSource": "<environment:false-or-compiled-default:false>",
  "hubAdvisoryV2EmissionEnabled": false,
  "hubAdvisoryV2EmissionEnabledSource": "<environment:false-or-compiled-default:false>"
}

Extract the values only from the inspected container and its image: resolve .Name, .Image, com.docker.compose.project, and the ordered com.docker.compose.project.config_files label; select only the two exact environment entries named below. Do not serialize .Config.Env or the raw container inspection. Use .Image itself as imageReference, or replace it with a repository digest proven to resolve to that same image ID. Review the normalized snapshot before saving it. It contains:

• capture time in UTC;
• container name and exact .Image ID;
• an immutable rollback selector (sha256:<image-id> or a repository digest);
• the ordered com.docker.compose.project.config_files label split into its exact files;
• the effective values and proof source of only Scanner__Worker__Reachability__Enabled and Scanner__Worker__Reachability__HubAdvisoryV2EmissionEnabled.

Both booleans must be false on this estate. For each key, record source environment:false only when the inspected container carries the exact key with value false. If the key is absent, record compiled-default:false only after the exact candidate source proves the corresponding non-nullable Boolean property has no initializer (therefore the CLR default is false) and the ScannerReachabilityRuntimeConfigurationValidatorTests disabled/default case passes for that same SHA. Any other value/source, or an appsettings file that overrides the default, aborts. The current SCN-5 worker has both keys absent; absence is not itself the proof and must never be transcribed as an observed environment value.

Copy every snapshot value into the corresponding baseline.* record field, hash the exact snapshot bytes into baseline.configurationSha256, and record its relative path as baseline.snapshotPath. The validator parses the snapshot and cross-checks every copied value, including the compose project, ordered file list, and the two proof-source fields. Abort if the container, image, compose list, either Boolean, or either proof source cannot be established. Do not infer configuration from committed YAML.

2.1 Retain and bind the Worker analyzer cohorts

An image-only Worker swap is not sufficient. The signed analyzer manifests pin the host-contract DLL bytes; the baseline and candidate may require different bundle cohorts even when every assembly version is the same. Retain the exact baseline and candidate base bundles separately beneath the record directory, outside any disposable worktree or swept tmp/build directory. Never overwrite the live baseline bind, restore only tracked manifest/checksum files over fresh DLLs, or copy individual contract DLLs into an already verified image.

This procedure preserves the base-only profile: exactly DotNet, Go, Java, Node, Python, Apk, Dpkg and Rpm. The full-analyzer overlay is not activated by this window. Capture both actual read-only bind identities from the baseline Worker: /app/plugins/scanner/base and /app/trust-roots/plugins/scanner. The candidate uses its separately installed base/trust directories through a Worker-only override. Follow the release installer’s existing site-owned external bind pattern (STELLA_PLUGIN_ROOT / STELLA_TRUST_ROOT), scoped to Worker only: select a new digest-specific absolute installation directory outside the source checkout, copy the complete verified cohort without overwriting an existing installation, and verify the installed bytes and non-root mount visibility. Do not apply the all-service mounted-plugin overlay or globally repoint its variables. Missing installation paths must fail closed (create_host_path: false). Other Worker mounts and environment inputs remain unchanged.

The cohort receipt’s absolute mount source is this installed runtime directory; its relative retainedRoot is a separate byte-identical evidence copy beneath the record directory. Build output remains under tmp/build/, not in the installation or devops/plugins/ source tree. Keep the installed source identity fixed through forcing, rollback reapply, soak and final canonical convergence. This placement does not relax any signature, inventory, stamp or freshness check.

Set baseline.analyzerCohort and candidateImage.analyzerCohort to independent { "receiptPath": "<relative-json-path>", "receiptSha256": "sha256:<64-hex>" } pointers. Each normalized receipt contains:

• schemaVersion: 1, the exact imageId, UTC capturedAtUtc, and captureKind: observed for the running baseline, staged for the undeployed candidate;
• bundleIds: the exact eight directory/manifest IDs, not only a count;
• mounts: the analyzer-base and public-trust roles, each with actual or staged source, exact destination, type: "bind", readOnly: true, a relative retainedRoot, and a complete files array of relative path and sha256 entries;
• hostContractsRoot: the relative directory holding the four DLLs read from that exact image; hostContracts: their assembly name, version, relative path and sha256 (Lang, LangAdapter, OS and Plugin.Contracts);
• verificationReceipt: another captured-file path/hash pointer to the signature/payload verification result for these exact bytes.

The verification receipt has schemaVersion: 1, manifestSignaturesVerified: true, payloadHashesVerified: true, and semanticCohort. The latter repeats the exact imageId, bundleIds, mount identities/file inventories and host-contract values; omit only retained-root paths, capture time and capture kind. Produce it from successful signature and payload verification of the retained files, never by asserting the two flags. Use the existing Scanner signing/verification tooling and the captured public trust anchor. Do not reimplement signature verification in an operator script.

The offline record validator re-hashes every retained file and rejects missing, duplicate, escaping, reparse or unlisted paths. It checks manifest/directory identity, signed entry-DLL hashes and per-kind host-contract stamps. It also binds the verification receipt to the same semantic cohort. The external verification receipt proves cryptographic verification; inventory hashes alone do not. Staging never proves runtime admission.

Capture the actual mounted public trust, not the signer’s development directory. Only index.json, README.md, *.public.ed25519 and *.public.b64 belong in this public capture. Never copy seed.hex, a private key or a credential. Check for additional mounts/profile roots before proceeding; an unexpected full-analyzer mount is a stop, not an omitted inventory entry.

After each live phase, recapture the actual Worker image, mount identities and mounted bytes. Store a new observed cohort pointer in forcing.analyzerCohort, rollback.restoredAnalyzerCohort, rollback.reappliedAnalyzerCohort and soak.finalAnalyzerCohort. Forcing/reapply/final cohorts must match the staged candidate; restored must match the saved baseline. Their timestamps and retention-copy paths may differ, but image, mount identity, bundle IDs, complete file inventories and host-contract values may not. Never relabel a staged receipt as observed or use the candidate cohort as rollback evidence.

Capture timestamps also enforce phase freshness; a matching old observed receipt is not a new measurement:

Retain both cohorts and their verification captures throughout the window. Only after accepted final convergence may they follow the recorded retention and evidence-distillation policy; worktree cleanup must not remove a live bind. These temporary paths are single-window inputs, not permanent runtime homes. Before closing the window, land the long-lived Compose definition in committed devops/compose/ and prove that no running service still depends on a temporary override or a cohort directory that will be swept. Do not delete the baseline recovery bytes or a live-bound cohort to satisfy cleanup.

Capture the outgoing ReleaseOrchestrator image ID/reference and ordered Compose label list into releaseOrchestratorImage.liveComposeConfigFiles. That label is drift evidence, not automatically a rollback command. When a historical ignored path is missing, record it in missingLiveComposeConfigFiles, omit only that path from a replay list, and append one ignored baseline overlay selecting the exact captured image ID with pull_policy: never. Record that renderable ordered list in baselineComposeConfigFiles; every path in that replay list must exist.

Capture a separate, hashed, nonsecret baseline-posture receipt. Set its relative path and digest in releaseOrchestratorImage.baselinePostureReceiptPath and baselinePostureReceiptSha256. It contains capturedAtUtc, imageId, federationEnabled, federationEnabledSource, originalContainerId, originalNamespace, scratchPlan and policyObserver. Copy the exact original full Docker ID into releaseOrchestratorImage.originalContainerId too. Do not use a container name or truncated ID as its identity.

The Boolean is the effective baseline, not a desired value. Read STELLAOPS_FEATURE_FEDERATION from the inspected container/configuration. Record environment:true/false or configuration:true/false only with that corresponding proof. An absent environment key alone is not evidence: compiled-default:false requires verifying the exact baseline image’s default and absence of any overriding configuration provider. Unknown posture is a stop. If already enabled, preserve true; do not disable an existing host.

In particular, current source FederationHostExtensions defaults this flag to false, while the base Compose profile supplies a true default and the accepted local profile pins false. An added name is therefore not harmless parity. Pin the verified baseline Boolean explicitly in the replay/candidate override. Retain that pin through rollback/reapply and final canonical convergence; do not enable federation as a side effect of this Scanner window. STELLAOPS_FEDERATION_REGION does not authorize activation. Review the behavior of every other added setting too; unrelated capability activation is not authorized by this replay procedure.

Before accepting the replacement list, compare its rendered release-orchestrator service against the running container entirely in process memory. Require every running Compose-controlled environment name in the replay, no removals, and only source-reviewed additions listed in baselineReplayExpectedEnvironmentAdditions. Require keyed-HMAC parity for common nonsecret literal values, mount parity, process/user/command parity, and network parity. Secret-backed values are compared by name only. Never write raw environment maps, secret values or the HMAC key; the named federation Boolean above is an explicit nonsecret posture field. Added-name approval does not waive that posture check. Static parity does not prove the outgoing image can boot with current file bytes; the next phase supplies that forcing function. This follows SCN-5’s established rule: a historical config_files label is evidence, not a reconstructed rollback command.

Prove the candidate binary has the SCN-ID-5 unavailable-source consumer path; a dual reader without BuildUnavailableAdvisorySourceContribution is not sufficient. If an inherited baseline Compose file contains image-selector interpolation, bind only those selectors ephemerally to the captured immutable baseline IDs before rendering. The historical post-SCN-5 list required SCN5_WEB_CANDIDATE_IMAGE and SCN5_WORKER_CANDIDATE_IMAGE; a canonical chain may no longer require them. Derive the required names from the actual selected files. Neither value is a secret and neither may be guessed from a tag.

Run the Preflight validator now. Do not recreate anything unless it passes.

2.2 Record the scratch namespace and owner-read boundary

Choose and review one scratch plan before stopping RO. Both scratch phases use that plan, but must create different containers. The original, Baseline scratch and Consumer scratch IDs must be pairwise distinct; their image IDs may legitimately be identical.

Record the actual internal listeners as [8080,8443]. Preserve every original physical network and alias, particularly release-orchestrator.stella-ops.local on stellaops. The plan adds only the new container name on each network and its unique alias on stellaops; it does not replace the canonical owner alias. Explicitly replace the complete host-port list with reviewed distinct ports. Compose’s additive merge must not silently retain an old published port. Check availability again immediately before creation. Certificate, internal TLS port8443 and normal trust remain intact.

Both hashed BaselineReplay and Consumer deployment receipts require environmentNamesParity, nonSecretEnvironmentHmacParity, mountParity, processParity and networkParity, all true, plus namespaceSnapshot containing:

• capturedAtUtc while the scratch container is running;
• original:{containerId,imageId,running:false,stoppedAtUtc} from the stopped original;
• scratch:{containerId,imageId,startedAtUtc,namespace} from the actual scratch container;
• policyOwnerRead, captured from the unchanged Policy observer namespace.

The Consumer snapshot also contains predecessor:{containerId,removed:true,removedAtUtc} for the Baseline scratch. Verify absence after removal; do not infer it from a planned command. The five parity predicates still compare all non-namespace settings. The explicit namespace model accounts for only the reviewed project/name/port/alias differences; it is not a waiver for another environment, mount, process, security or network change.

policyOwnerRead records capturedAtUtc, observerContainerId, observerNetworkId, origin, method, releaseId, requestPath, remoteAddress, statusCode, tlsVerified, redirected, tokenClaims, contentType, componentCount, allResponseReleaseIdsMatch, responseSha256, projectionPath and projectionSha256. Require all of the following:

• The Policy namespace performs GET to https://release-orchestrator.stella-ops.local:8443/api/v1/release-orchestrator/releases/{escapedReleaseId}/components.
• Normal service authentication has claim subset {clientId:"policy-engine",tenantId:<intended tenant>,scopes:["release:read"]}. Retain that subset, never the bearer or credential value.
• TLS/hostname verification succeeds without a redirect, HTTP status is exactly200, normalized content type is application/json, and the remote address matches the scratch stellaops address.
• Parse a nonempty response in memory and check every returned release ID. Retain only the strict nonsecret identity projection {rawResponseSha256,releaseId,components:[{releaseId,id,digest}]} at the relative projectionPath; its bytes must match projectionSha256. All hashes use sha256:<lowercase hex>. Raw component DTOs can contain configuration overrides or credential-bearing URLs and must not be retained. The projection binds the captured raw-response digest; the validator does not claim to reconstruct or rehash an omitted raw body.

Capture chronology is mandatory: baseline posture, original stop, Baseline scratch start, Policy probe, Baseline namespace snapshot, scratch removal and original restart, final Baseline receipt, next original stop, Consumer scratch start, fresh Policy probe and Consumer snapshot, then final Consumer receipt. Same-time probe/snapshot/receipt captures are allowed where the validator specifies it; stop/start/removal boundaries must be later than their prerequisites. Keep earlier failed receipts unchanged. A ready container or all-true parity flags cannot replace these actual identity, namespace, owner-read and cleanup observations.

3. Prove the baseline replay without consuming the rollback container

Before stopping RO, inventory the intended tenant’s existing immutable reachability rows through its owner API. If both exact v1 and v2 inventories are empty, Q-30 permits one controlled producer bootstrap: first capture/revalidate the exact Worker baseline, configuration/cohort and accepted bf9 Worker/RO candidate inputs; activate only the candidate Worker against unchanged canonical RO; submit one scan tied to an unpromoted Draft acceptance release; verify genuine producer-emitted v2 through the owner API; then restore and prove the exact Worker baseline/cohort. Only after restoration may the normal Baseline phase start. The bootstrap counts toward no acceptance case or soak scan. No promotion, approval, manual evidence insertion/backfill, sibling database access, new publication or automatic retry is allowed. On failure, restore baseline and stop with evidence. An unrelated packet remains invalid.

Current Q-32 execution boundary (2026-09-01): Q-30 and Q-31 are both consumed. Draft rel-a07073f remains unpromoted. Q-30 scan 5aee072040ba82b371c94c15520c84a64a91af06 is terminal Failed with no evidence. Q-31 scan 92a3690bf0411197430b424a9699e660026316f6 reached Succeeded and emitted one genuine, release-bound v2, but that row is explicitly unavailable: 36 product keys, 35 available and one hub-product-issues-product-key-disagreement. Do not treat it as Baseline input and do not resubmit under Q-31. The exact original Worker baseline/cohort is restored healthy and the candidate is stopped/retained. Source now repairs the Hub owner route’s second percent decode by recovering Kestrel’s raw request segment and decoding transport exactly once; the repair is not deployed. Q-32 is approved for an exact-source vulnerabilities-web-only local deployment and one final bootstrap scan against the same Draft/digest. The Hub image carries pending forward-only 005_malformed_vulnerability_id_convergence.sql; immediately before the recreate, the owner-database read-only preflight must reproduce exactly 17 doomed ids / 27 fact_current, with dependent counts 27 fact_revision, 27 fact_provenance, 18 issue_linkset, 27 dist.statement, 0 binary_fact and 0 rebuild_queue. Any count or identity drift stops the window. Migration 005 must delete every doomed dependent before its fact, including the currently empty binary index and rebuild queue classes; it does not authorize any Scanner DB/schema or evidence rewrite. Preserve both scans, the unavailable v2 and every runtime identity. Before Scanner mutation, retain the outgoing Hub image, render/read its exact live chain, recreate only vulnerabilities-web, and require the deployed percent-encoded product read to preserve the canonical key. A failed route probe stops the window. A green probe permits one new client request ID only. Success requires one newer non-vacuous all-products-available v2; a missing or non-available result restores baseline and stops without another submission. After success or failure restore the exact Worker baseline. The final scan is bootstrap-only and earns no acceptance credit. Its normal local referrer attachment and producer-emitted evidence are expected; no image push or new/external registry publication occurs. A post-startup zero owner-store census is only the point-in-time migration result: web-only activation neither runs the Worker-owned Alpine repair nor revisits unchanged content-addressed documents behind the normalization checkpoint. VULN-B9 still requires fixed-Worker activation, qualifying reprocessing, a retained publication and repeat hub/downstream censuses.

Create an ignored scratch override implementing the reviewed §2.2 namespace: distinct container name, explicitly replaced host ports, and one added unique alias while preserving every original alias. Use the separate Compose project, the replay list, and the exact baseline image pin. Stop—but do not remove or recreate—the original ReleaseOrchestrator, then start only the scratch baseline service. No other service is recreated. If the scratch service is not ready, remove it and restart the untouched original container.

Prove the scratch baseline is ready and can read one existing immutable reachability evidence row. Prefer v2 when present; a frozen v1 row is allowed only for this baseline-replay drill because it proves the old consumer’s own read path without claiming the later v2 gate. Stop/remove the scratch project and restart the original before continuing. Save only a normalized receipt containing capture time, baseline image ID, live/replay file counts, missing paths, reviewed environment additions, an empty removals array, the five parity booleans, scratch container ID, baselineReplayReady=true, baselineReplayEvidenceReadable=true, and secretValuesCaptured=false, plus the §2.2 namespaceSnapshot and owner-read projection. The snapshot is taken before scratch removal; the enclosing receipt is finalized after cleanup and original restart. Also capture federationEnabled and federationEnabledSource from the scratch container’s effective configuration; the validator requires the baseline Boolean. Hash it into baselineReplayReceiptSha256, then run the Baseline validator. This scratch drill is the rollback forcing function; source review or a successful render alone is not.

4. Refresh and prove the ReleaseOrchestrator consumer

Create one ignored candidate overlay that selects both accepted repository digests:

services:
  release-orchestrator:
    image: <release-orchestrator-repository>@sha256:<digest>
  scanner-worker:
    image: <scanner-worker-repository>@sha256:<digest>

Reuse the isolated project/alias shape proven by the Baseline phase. Stop the original container without removing it, render the replay list plus the candidate overlay, and start only the candidate scratch consumer. Prove from inspection and a bounded readiness capture that the new container uses releaseOrchestratorImage.imageId, serves the candidate buildinfo SHA, is ready, and has no unexpected restart. Save a normalized JSON receipt with capturedAtUtc, containerId, imageId, imageReference, buildinfoGitSha, ready, federationEnabled, federationEnabledSource, all five §2.2 parity flags and its fresh namespaceSnapshot (including the removed Baseline predecessor). Copy the deployment identity fields and its byte hash into releaseOrchestratorImage.deployment.

Measure federation posture from this candidate container, not from the requested overlay. Both the inline deployment fields and the hashed receipt must preserve the baseline Boolean. The detector’s positive and negative controls run offline with:

pwsh tools/scripts/deploy/customer/verify-scanner-hub-v2-cutover-record.ps1 -SelfTest

If the consumer is not ready, remove the scratch candidate and restart the untouched original container. Do not deploy the Worker. Run the Consumer validator after the clean consumer is ready. This phase is mandatory because the unavailable-v2 blocking comparison cannot be accepted against a consumer that predates the SCN-ID-5 availability gate.

5. Apply the Scanner candidate reversibly

Use the exact baseline compose file list, then append the committed devops/compose/docker-compose.reachability.yml and the same ignored two-service candidate overlay used for the consumer refresh. That overlay must also select the separately installed candidate base and public trust directories for only scanner-worker, as read-only binds at the two destinations in section 2.1. Keep its configuration/other mounts unchanged and verify the rendered mount identities before recreation. Preserve the original baseline bundle/trust directories for the unmodified baseline Compose chain. Keep ReleaseOrchestratorBaseUrl on the canonical in-network owner origin from the committed reachability overlay. Both namespace proofs must show the scratch instance owns the preserved canonical alias while the exact original is stopped; the fresh owner read must resolve to the scratch IP. Do not replace this durable origin with the scratch-only unique alias: section7 removes that scratch before the Worker rollback/reapply, which would leave the reapplied Worker publishing to a nonexistent name during soak. The unique alias remains an additional drill identity, not a permanent publisher address.

Render docker compose config first. Relative to the accepted Consumer render, refuse any new change outside scanner-worker; the ReleaseOrchestrator selector must remain byte-identical. Then recreate only the Worker with --no-deps --force-recreate. Record the full command, ordered -f list, overlay bytes and hash, resulting container ID, image ID, and readiness output.

After recreate, prove from container inspection—not from the overlay—that:

• .Image equals candidateImage.imageId;
• both reachability and Hub v2 emission are true;
• HubAdvisoryBaseUrl names the Vulnerabilities service;
• PublishReleaseEvidence=true and ReleaseOrchestratorBaseUrl names the in-cluster ReleaseOrchestrator service;
• the effective compose file list is the intended candidate list;
• actual analyzer/public-trust mounts and bytes match the staged candidate cohort; retain this initial check, then recapture forcing.analyzerCohort after all three forcing cases as specified below;
• the worker is ready before a scan is submitted.

An image tag, a healthy old container, or source tests do not substitute for these checks.

6. Run the three fresh forcing cases

Before preparing the three fresh Ready releases, resolve the exact existing target environment and verify RequiresHumanApproval=true, at least one enrolled target, and an explicit blocking reachability binding. A target name that does not resolve is a stop: it cannot prove the human-approval override. Do not create a dummy target or weaken a policy to make this test run.

The owner approved creating a dedicated non-deploying acceptance environment on 2026-08-30. Use the owner API only, preserve existing environment protection, and verify the enrolled lab agent’s current health and assignment semantics before creating its target. This approval does not authorize stealing another environment’s target or bypassing normal policy revision approval. Resolve a real compiled policy revision before creating the environment; a pack name with an empty versions array is not an executable policy. Keep the binding absent while obtaining the required real Policy proof. After that proof and normal revision activation, create the binding explicitly enabled with ReachabilityBlocking=true.

Use POST /api/v1/release-orchestrator/environments, not the topology facade’s POST /api/v1/environments: the latter does not carry RequiresHumanApproval and its default is false. Explicitly set requiresHumanApproval:true, a positive requiredApprovals, autoPromoteFrom:null and protectionClass:2 (Protected). The owner DTO uses the numeric ProtectionClass enum; do not send a string label. Derive a free order index and validate the remaining owner DTO fields through the normal /environments/validate route. Read the actual created record back; neither its name nor an Archived/Disposable lifecycle label prevents deployment. Environment creation does not require an Active policy lookup. In source, DeploymentClassGate rejects an unbound Protected environment, while the human approval flag separately prevents automatic promotion approval. These are scoped controls, not a universal deployment kill switch; verify their deployed configuration before use. No deployment call is authorized.

Create the new target without agentId, then assign the verified agent through PUT /api/v1/release-orchestrator/targets/{targetId}/agent. That route validates same-tenant membership, Active status and executor capability; an inline creation-time agent ID does not run those eligibility checks. Verify the old target still retains its original environment and agent association. Heartbeat health is not Docker connectivity or deployment proof; this procedure performs neither a deployment nor a connectivity-command dispatch.

For this consumer, verify GET /api/policy/packs: RO calls the selected revision’s evaluate route, whose repository reads the same pack/version store. There is no registry or local-file fallback. Do not substitute arbitrary sample DSL: rules unrecognized by CveDenylistExtractor can default-allow without consulting the findings source. On 2026-08-31 the owner approved the dedicated scn-id-acceptance pack/revision: high/critical severity AND proven reachability denies, and findings source unavailability remains fail-closed. This approval does not permit changing the shared default pack or bypassing review/activation controls.

Verify the deployed tenant transport before staging. A request token naming default does not prove an older Policy image stored its pack under that tenant: Router dispatch previously missed the Policy context bridge and selected public, while owner-role reads bypassed RLS. After the corrected filter and explicit tenant predicates are deployed, stage the exact approved definition through the normal owner API in the intended tenant. Leave the prior public row unchanged; this procedure does not authorize tenant reassignment or deletion. Pack summaries and revision numbers are not activation receipts. Require the actual activation response and the subsequent selected-revision consumer proof.

Create the revision metadata before compiling the bundle. Omit requiresTwoPersonApproval from the creation request so the configured requirement is resolved normally; record and honor the returned value. Compiling first can implicitly create a Draft revision without that requirement. Use the normal review/activation lifecycle, preserve a pending second approval, and never invent an actor identity. Record the exact definition bytes and returned bundle digest.

Verified against the Policy lifecycle/evaluator source at ce820e9724d77200ee9b0add5d0cf4de2902738b (identical audited files at 06841c8aa69d9b315d8ac4f8ac714617335702e9). Re-verify endpoint, repository and activation-setting behavior before applying this procedure to a different image.

Prove the definition through the real compiler, recognized-rule extractor and runtime deny/pass/unavailable controls locally, then repeat with actual persisted acceptance release and Scanner evidence through the owner API before binding. The retained definition is src/Policy/__Tests/StellaOps.Policy.Engine.Tests/Fixtures/scn-advisory-v2-acceptance.dsl; ScnAdvisoryV2AcceptancePolicyTests compiles those exact bytes, checks recognition, and distinguishes test-injected findings from live evidence. Do not substitute a different sample after running those tests. Re-run: pwsh ./tools/scripts/test-targeted-xunit.ps1 -Project src/Policy/__Tests/StellaOps.Policy.Engine.Tests/StellaOps.Policy.Engine.Tests.csproj -Class "*ScnAdvisoryV2AcceptancePolicyTests*" -BuildProjectReferences -Restore. The evaluation request accepts pack/version/subject, not injected findings. A missing release or unfed findings path cannot establish a clean-zero case. Do not point Policy at another service’s database or seed a synthetic projection to make the test pass. These gates do not authorize release approval or deployment.

The currently implemented durable Engine lifecycle is Draft revision, bundle compilation, an explicit initialStatus:"Approved" revision transition, then activation. It uses policy:edit and policy:activate; do not substitute the separate Registry client’s unserved publish route. Record independent definition review and test evidence, and honor any second-actor activation gate. This is not proof that a submitted/reviewer/shadow workflow ran. If that workflow is required by the selected deployment’s governance, keep the revision Draft until it is actually available and fulfilled.

Promotion is a state-changing control-plane request, not a read-only evaluator. ReleaseEndpoints.RequestPromotion passes the environment’s human-approval override to PromotionRequestProcessor; without it, a passing decision can auto-approve and DispatchOnApprovalReleaseTruthRepository can enqueue a deployment. Do not approve these requests, call a deployment endpoint, or change the target’s approval setting during this window. Positive cases should remain pending with their reachability gate passed; an unexpected approved, auto-approved or deploying state is a stop, not acceptance. This procedure does not authorize deploying the test releases to enrolled targets.

Keep the Policy-denial proof separate from the non-empty reachability-pass case. In the accepted RO source bf9deddee1eae37e8610882df0dda7653ef11543, Program.cs registers the approval/Policy gate before the reachability gate; PolicyDrivenGateDecisionEvaluator marks a Policy deny as terminal and PluginBackedGateDecisionEvaluator stops that chain. A Policy-denied promotion therefore cannot prove that its later reachability gate passed. Before selecting the non-empty forcing input, require its actual persisted Scanner findings to be non-empty and its selected Policy revision to allow; use a different, verified input for the real Policy-denial proof when necessary. Do not disable the binding, relax the approved definition, reorder gates or invent a missing gate row to satisfy the comparison. If no available input meets both conditions, record the input gap and hold that forcing case.

Verify Scanner intake permits the window and the scanner:queue:redis:connectionString setting is configured, by name/presence only: without queue registration, the same scan API merely persists Pending state. Submit each scan through POST /api/v1/scans with force=true, a unique clientRequestId, and metadata:{"release.id":"<fresh-release-id>"}. Use a new client request ID for every soak scan too. Scan IDs are deterministic over the target, tenant, force flag, client request ID and metadata; force=true alone does not guarantee a fresh ID. Use the normal token flow and API routes; do not query or seed PostgreSQL.

Classify the exact versionless PURLs against the real Hub product-issues route before choosing immutable inputs. Require a successful, well-formed response echoing the exact key; only an actual empty issues array proves clean-zero. Source-only inverse fixtures retain their dependencies and are not clean-zero inputs. An SBOM with no PURLs is not a clean lookup, and a fixture’s CVE label does not prove current Hub coverage. Retain the input digests and lookup proof; recheck them after the final coordinated runtime handoff.

The input must also be readable through the Worker’s actual registry transport: verify the immutable manifest, selected platform config and layer blobs. A local Docker image ID or its daemon-local RepoDigests does not prove registry availability. The existing GoText fixture is source-only (FROM scratch) for the static extractor; local construction needs no package downloads. Publishing it to a registry still follows the separate publication authorization boundary. For this window only, the owner approved publication to the existing local Zot registry on 2026-08-30. The verified GoText input is 127.1.2.9:5000/scn-id/go-text@sha256:d75fbffadbcec6bd87d5494957e931a0952f4c36aed26c6b0b3ebf6cd146e9aa. Manifest/index, platform config, layers and the two source payloads were read back and hash-verified; Worker-origin manifest HEAD returned 200 and the same digest. Recheck transport and product coverage after the coordinated runtime handoff. This publication is not a completed scan, promotion, deployment or soak, and does not authorize publication to an external registry. The older forward/inverse symbol fixture comments do not define Hub v2 outcomes: v2 matches product issues to component PURLs. Verify the actual emitted findings, paths and current gate options instead of treating the inverse as clean-zero or changing policy to obtain a passing result.

For each terminal scan:

1. List ReleaseOrchestrator evidence with releaseId=<id> and type=stellaops.reachability-report@v2; require exactly the fresh row.
2. Download GET /api/release-orchestrator/evidence/{id}/raw without transformation into <case>-evidence.json; hash those bytes.
3. Assert schemaVersion=2, the expected source status/reason, exact finding count, and absence of canonicalId.
4. After the human-approval precondition above is verified, request promotion for that release and exact target blocking environment. A denied promotion returns HTTP 409 with approvalId, not gateResults. For both positive and denied cases, retrieve the stored approval through GET /api/release-orchestrator/approvals/{approvalId} and save the single unchanged gateResults[] object whose type=reachability-gate as <case>-reachability-gate.json. Record the response and approval state too; do not substitute an error body or fabricate a gate row. The approval GET performs single-row expiry/deploying-state hygiene, so it is not a pure datastore read; use only the fresh approval created for this case.
5. Record the capture paths, IDs, hash, timestamp, source status/reason, finding count, gate status, witness ID/type, and normalized gate source status.

The three scan, release, and evidence IDs must all be distinct. Inspect the fresh release evidence set and record forcing.newV1WritesObserved=false; any new stellaops.reachability-report@v1 row while Hub v2 is selected aborts the window.

After the three cases finish and the normal candidate URL is restored, capture the actual mounted cohort again into forcing.analyzerCohort. Its timestamp must be at or after every case capture; do not reuse the initial post-recreate check. Run the Forcing validator. Its non-empty/clean-zero pass versus unavailable deny is the required ReleaseOrchestrator comparison.

7. Exercise exact rollback and prove evidence immutability

ReleaseOrchestrator rollback never reconstructs the missing historical list: stop/remove the scratch candidate and restart the untouched original container. Prove its exact baseline image ID, health, and immutable evidence read before continuing the Worker rollback below. For the current bf9deddee1eae37e8610882df0dda7653ef11543 window, the captured original RO already runs the same accepted source/image as the scratch consumer. Re-verify that full-source equality and a fresh canonical owner read after restoration; the reapplied Worker then publishes to this retained canonical RO. Do not generalize this to an older original: if its source differs from the Worker candidate, stop before reapply and prepare a separately reviewed consumer reapplication. An old deployment receipt alone does not prove the restored path.

Before rollback, hash the three raw v2 files again. Invoke Compose with exactly baseline.composeConfigFiles in their recorded order and no additional image or reachability overlay. Bind any inherited image-selector variables ephemerally to the captured baseline image IDs, render first, and require the Worker to resolve to baseline.imageId before recreating only scanner-worker with --no-deps --force-recreate. Appending a rollback overlay is forbidden: it would make the post-rollback Compose list differ from the baseline list and make this gate impossible by construction.

Prove after recreate:

• container .Image equals baseline.imageId;
• the selected compose list equals baseline.composeConfigFiles byte-for-byte and in order;
• the selected reachability booleans are both false, with the same two proof sources recorded at baseline;
• the nonsecret configuration snapshot hashes to baseline.configurationSha256;
• the actual analyzer/public-trust cohort matches the baseline; capture rollback.restoredAnalyzerCohort after recreation, not from the candidate;
• the worker is ready.

Do not submit a rollback-mode scan: the old source is stopped, and a fresh v1 write would be false evidence. Instead download each of the three existing v2 evidence rows again. Record one rollback.evidenceChecks row per evidence ID, with readable=true, the same before/after SHA-256, and the after-read path.

Finally reapply the same candidate repository digest with the same candidate compose list, recreate only scanner-worker, and prove the resulting image ID equals candidateImage.imageId. Record both the rollback and reapply selectors. Recapture rollback.reappliedAnalyzerCohort and require the exact candidate mount identities and retained byte inventory before starting the soak. Record rollback.completedAtUtc only after that reapply capture, and start the soak at or after this completion time. The restored/reapplied captures must each be strictly later than the preceding phase capture. Run the Rollback validator.

8. Soak after the successful rollback drill

The soak begins only after the exact candidate is reapplied and ready. Record the observed UTC start and completion timestamps; the validator derives the elapsed interval, requires completion to be later than start, and refuses a durationMinutes value larger than the derived interval. Minimum acceptance is:

• 60 continuous minutes;
• readiness sampled at start, every five minutes or less, and end—at least 13 samples;
• at least 10 fresh ordinary scans while Hub v2 is selected;
• zero worker restarts;
• zero unexpected hub request failures outside the intentional unavailable forcing case;
• zero v2 parse/unsupported-version failures;
• zero v1 writes while Hub v2 is selected;
• zero release-evidence publish failures;
• zero non-available witnesses reported as passed;
• zero evidence hash mismatches.

An ordinary scan carries no release association, and release evidence is published per release: a soak of ordinary scans therefore adds no new v2 evidence rows (measured 2026-09-02: ten soak scans, v2 delta zero). Growth in the v2 count is not a soak threshold and its absence is not an abort condition; the release-scoped forcing cases are what prove emission. Zero v1 writes stays a threshold, and it is meaningful for ordinary scans.

An access token minted at soak start expires before the hour is out. Re-mint during the soak rather than treating the resulting 401 as an estate failure; record only the mint count, never the credential.

Capture nonsecret health and bounded relevant log output beneath the record directory. Do not capture authorization headers, tokens, full environments, or unrelated logs.

Abort immediately—without waiting for the hour—on any worker unready sample, restart, new v1 write, evidence publish/parse failure, changed stored hash, or a non-available passing gate. Execute the rollback steps above and do not reapply the candidate after an abort. Record the reason; SCN-ID-5 remains DOING.

If all thresholds pass, record the final exact candidate ID/digest and a fresh soak.finalAnalyzerCohort matching the candidate, then run:

pwsh tools/scripts/deploy/customer/verify-scanner-hub-v2-cutover-record.ps1 `
  -RecordPath '<record-dir>/acceptance-record.json' -Phase Soak

After the Soak validator passes, stop the scratch consumer, converge the candidate image through the canonical stellaops Compose project using the same parity-proven replay list, and prove its canonical name, aliases, image ID, buildinfo, readiness, and one immutable evidence read. Remove the scratch project only after that canonical convergence. If convergence fails, restart the proven baseline scratch service; never delete the preserved original until the canonical candidate is accepted.

Completion and durable evidence

SCN-ID-5 can move to DONE only after -Phase Soak passes, canonical convergence is verified, and no temporary input still defines the long-lived service. Distill the reviewed nonsecret result, key small captures and hashes under docs-archive/qa/scanner-hub-v2-cutover/<window-id>/; never commit the raw bulk or analyzer DLL payloads. Keep the complete record and retained cohorts together for validation during the window. On close, verify raw files are no longer live or recovery inputs before applying their retention class. Add the exact source SHA, CI receipt, Worker and ReleaseOrchestrator image IDs/digests, scan/release/evidence IDs, rollback receipt, soak interval, and validator result to sprint 017.

SCN-ID-6 remains blocked until that soak is explicitly accepted. Its later source change removes the old writer/implementation edge but permanently keeps the frozen v1 reader and literal fixtures.

The older tools/scripts/deploy/customer/scan-reachability.ps1 and devops/fixtures/reachability-live-go/run-live-proof.ps1 exercise legacy verdict/direct-database paths. They do not prove the Hub v2 trio, release gate, rollback, or soak and must not be used as SCN-ID-5 acceptance.