Deploying the consolidated Findings service (findings-web / findings-worker)

Owner sprint: SPRINT_20260722_010 FND-8 — Companion: the route swap in findings-cutover-fnd5-route-swap.md, which executes in the same FND-9 window.

ROUTE LEG RETIRED — 2026-09-04. The route step of this procedure must not be run. The owner ruling of 2026-08-28 (“all microservices needs to autopublish no router configuration for microservices endpoints”) was executed by SPRINT_20260809_001 RAR-5/RAR-7 (81e276c2a3, 83bfc9359f, f4b1774cf6). Both gateway route tables now hold 22 routes and none of the eight rows the FND-5 manifest replaces still exists. Findings is served by the auto-published layer instead. A static gateway entry out-ranks publication, so reinstating those eight rows — by applying the FND-5 swap, or by “reconciling” the manifest until -Check passes and then applying — would claw the Findings plane back off the published layer: a regression, not a no-op. (Concretely today the scripts cannot do this on their own — with nothing to match they write nothing, then falsely report a successful apply and exit 0; see the window steps.)

The script now says so itself. Since ced8a2d286 fnd5-swap-findings-routes.ps1 refuses on both invocations: -Check and a bare run each print FND-5 route swap: RETIRED -- nothing applied, nothing checked. and exit 2 (measured 2026-09-04 at HEAD). It also records what the older text here got wrong: the swap ALREADY RAN — d25921373d landed it in all three configs — and RAR then retired the consolidated group it installed, so ^/api/findings/v1(.*) is in the retired-route registry. Step 1 below demanded “9 of 9 logical routes”; that number was never right either — the manifest tools/scripts/fnd5-findings-route-swap.json has always held 8 replacedRoutes, and none of the eight survives.

Do instead: confirm the auto-published layer serves the native Findings surfaces (/api/findings/v1/** answered through the gateway with a real payload, and rejected unauthenticated), rather than swapping any route. The retired end state is recorded in src/Router/__Tests/StellaOps.Gateway.WebService.Tests/Configuration/retired-routes.json and enforced by RetiredRouteRegistryTests. Everything in this runbook that is not the route swap — the compose flip, the database move, the parity and revocation gates — still stands.

The two roles are replica roles of ONE deployable family sharing ONE database, stellaops_findings (CoC §8.2). They replace five compose keys — findings-ledger-web, riskengine-web, riskengine-worker, findings-security-web, findings-vulncorrelation — which are deleted at FND-10, not before.

Current status: the consolidated family is LIVE and SERVING (updated 2026-08-27)

The re-opened FND-9 window (after the 010×023 seam shipped as 3eeed9b925) cut the family over: findings-web + findings-worker serve against stellaops_findings (images stamped gitSha=3eeed9b925, worktreeState=clean), the FND-5 route swap is applied in all three gateway configs, the Console runs the five flipped prefixes, the predecessor data is copied with row-count + content-hash parity (7/7 tables), and the five predecessors are STOPPED and preserved until FND-10. Facts a future operator needs:

1. The Graph asset-registry seam is ARMED live on findings-web (Findings__GraphAssetRegistry__Enabled=true, FeedBaseUrl http://graph.stella-ops.local, shared envelope key): the admission passed against the deployed graph-api feed, the empty upstream log is the contract’s valid zero-event state, and the copied asset_registry_projection_offsets checkpoint (the 1970 epoch-zero sentinel — the producer has never emitted an event on this estate) is byte-faithful. The pre-seam facts (unconditional worker, 42P01 fail-closed against a graph-less DB when the seam is DISABLED) still hold for a disabled boot.
2. The security plane SERVES against a live corpus generation (measured 2026-08-27). Active generation: nvd generation 47, sha256:eccf0366ca24f5751ebf4c545280fb8fcf21cd409b716c56fe4cf98f20bfb6ed, publication seq 13, activated 2026-08-27T10:41:27.979523Z, consumer checkpoint epoch 1787755709627 seq 13 = head, lag 0, corpus health “Local advisory corpus is current.”; sha256:8865ed55… (nvd 46) is the retained prior. Both containers are healthy. /security/findings serves 200 authenticated on the canonical prefixes, direct and through Router; unauthenticated is 401 through Router and 400 tenant_missing direct (the host trusts network position for auth but still requires a tenant). artifact-triage remains a typed 503 (security_artifact_triage_projection_unavailable) BY DESIGN until the Release and Scanner owner projections exist — that one is not a fault. The cold typed-503 contract still applies to an estate with no generation.
3. The target database is converged at HEAD (Findings 001–008 + Eventing 001–004) after the window re-baselined the empty pre-provisioned target — a9f48fd0a5 had edited applied migration 005’s executable SQL, which the checksum gate caught. Do not re-baseline again.
4. The owner role carries BYPASSRLS and CREATEROLE. CREATEROLE is required because 001’s COMMENT ON ROLE findings_ledger_admin is unguarded; scratch rehearsals masked this by running findings as the scratch cluster’s superuser.
5. findings-web needs the Router transport block (inlined x-router-microservice-defaults + the four plugin/trust-root mounts in the overlay, consumer group findings) — without it the host sends no HELLO and the consolidated Microservice group answers 503 no-instances.
6. Grant revocation is a measured no-op on this estate: all five predecessors connected as the stellaops superuser and zero explicit grants exist on the four old schemas (or on graph), so there is nothing to revoke; the effective revocation is the stopped predecessor containers plus FND-10’s deletion. On an estate with per-service roles the revocation step is real.

Artifacts

Build

bash devops/docker/build-service-publish.sh findings-web findings-worker

Base-image note (corrected 2026-08-06, measured on the first scratch boot). STELLAOPS_PUBLISH_BASE_MODE defaults to clean since SPRINT_20260727_002 PKG-1, so both keys build FROM the service-INDEPENDENT shared base stellaops/service-base:<flavour>-<rev> — this run used stellaops/service-base:default-e5c197de300a. The predecessor tags in the SERVICES[] triple (findings-ledger-web:dev, riskengine-worker:dev) are the legacy base field and are read only when STELLAOPS_PUBLISH_BASE_MODE=legacy. Practical consequence: you do not need either predecessor :dev image present to build these keys — findings-ledger-web:dev was absent on the reference host and both builds succeeded. Only the hardened shared runtime base is inherited; the app payload arrives by publish+COPY.

Rehearse on an isolated scratch stack (do this before the window)

Never rehearse against the live network or the live cluster. The family overlay names no postgres service — by design, because stellaops_findings lives on the existing control-plane cluster (one database per service is the hard rule; one cluster per service is not) — so an isolated boot needs a throwaway cluster. That is what docker-compose.findings.scratch.yml provides, and it is scratch-only: never compose it in the FND-9 window, or the service converges into a second, empty stellaops_findings.

cd devops/compose
STELLAOPS_FINDINGS_NETWORK=fndscratch \
FINDINGS_POSTGRES_PASSWORD=<scratch-password> \
STELLAOPS_POSTGRES_FINDINGS_CONNECTION='Host=findings-db.stella-ops.local;Port=5432;Database=stellaops_findings;Username=findings;Password=<scratch-password>;Maximum Pool Size=16' \
KESTREL_CERT_PASSWORD=<dev-cert-password> \
POSTGRES_IMAGE=<audited digest-pinned image> \
  docker compose -p fndscratch \
    -f docker-compose.findings.yml \
    -f docker-compose.findings.scratch.yml up -d

The password appears twice (cluster + connection string) because compose does not expand ${VAR} inside env-file values (ENV-001). Pass --env-file /dev/null if you want the rehearsal fully insulated from devops/compose/.env; without it, compose loads that file and the estate’s values leak into the scratch project.

Verify — a forcing function, not “the container is up”:

# 1. Fresh-DB convergence, counted the way the family baseline counts
#    (relkind r/p, non-partition, excluding schema_migrations):
docker compose -p fndscratch exec findings-postgres \
  psql -U findings -d stellaops_findings -c \
  "SELECT n.nspname, count(*) FROM pg_class c JOIN pg_namespace n ON n.oid=c.relnamespace
   WHERE n.nspname IN ('findings','findings_security','analytics','riskengine','eventing')
     AND c.relkind IN ('r','p') AND c.relispartition=false AND c.relname<>'schema_migrations'
   GROUP BY n.nspname ORDER BY n.nspname"

# 2. The doctor guard still applies. The hardened base ships NEITHER curl NOR
#    wget — only bash — so probe over /dev/tcp (the same fallback
#    devops/docker/healthcheck.sh uses). Unauthenticated MUST be 401.
docker compose -p fndscratch exec findings-web bash -c '
  exec 3<>/dev/tcp/127.0.0.1/8080
  printf "GET /doctor/findings-web/checks HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n" >&3
  while IFS= read -r -t 5 line <&3; do case "$line" in HTTP/*|WWW-Authenticate*) echo "$line";; esac; done'

Expected table counts match ConsolidatedFindingsSchemaTests exactly: findings 25, findings_security 18, analytics 7, riskengine 1 and eventing 6 when counted by the query above (non-partition relations, excluding the two migration ledgers); findings_ledger_app and findings_archive exist with 0 tables. Every owned table is EMPTY on a fresh converge (AGENTS.md §2.11).

Teardown drops the scratch volume — it is scratch data by construction:

docker compose -p fndscratch -f docker-compose.findings.yml \
  -f docker-compose.findings.scratch.yml down -v

Deploy (FND-9 window only)

The database is not created by compose. Read-only preflight on 2026-08-24 found stellaops_findings already provisioned on the shared PostgreSQL installation: owner role findings (login, non-superuser), 11 MB, zero active sessions, Findings migrations 001–005 and Eventing 001–004. Its exact empty topology is 59 regular relations plus nine partitioned parents; all 57 non-ledger regular relations contain zero rows. Treat this as an empty target, not as evidence that FND-9 ran: re-verify sibling-database CONNECT isolation, RLS/default privileges and provisioning authority before starting either host. Do not recreate, drop or overwrite it merely because the old runbook said it was absent.

Reversible deploy/canary/route phase

Run this phase only after FND-X18-7’s owner projections, direct/Router forcing and prior-generation rollback prerequisites are green. Before any predecessor data is copied or old grant is revoked:

1. Re-run the source and database checks. The route check is retired (see the banner at the top): pwsh tools/scripts/fnd5-swap-findings-routes.ps1 -Check exits 2 and refuses, which is the correct end state — the eight rows were removed by RAR-5/RAR-7 and Findings is auto-published. Do not reconcile the manifest to make it pass, and do not treat the refusal as a blocker on this phase. (The earlier text here demanded “9 of 9 logical routes”; the manifest holds 8, so that gate was unsatisfiable even before RAR.)
2. Preserve (do not remove) the five predecessor containers, then stop them for the alias hand-off. findings.stella-ops.local is contended, so predecessor and target must never run together on the live network.
3. Start findings-web and findings-worker with the Findings overlay and the target-specific connection. Canary health, doctor, native canonical routes, identity envelope and Router HELLO against the target before changing routes.
4. Do not apply the companion Router swap — it is retired (banner above). Confirm instead that the auto-published layer already serves the canonical Findings paths through the gateway. Then apply the Console prefix flip, recreate the gateway with its own recorded config_files, and run authorized, unauthorized and four-persona forcing checks.
5. If any pre-revocation check fails, stop the targets, restore Router/Console config and restart the preserved predecessors. Only a successful canary may advance to data parity, grant revocation and soak; after revocation the rollback contract changes to fix-forward.

docker compose \
  -f devops/compose/docker-compose.stella-infra.yml \
  -f devops/compose/docker-compose.stella-services.yml \
  -f devops/compose/docker-compose.findings.yml \
  up -d findings-web findings-worker

Same-tag images do NOT swap on up -d alone. A rebuild that keeps the :dev tag needs --force-recreate --no-deps <key>, or the old container keeps running and every symptom you then chase is a stale image:

docker compose ... up -d --force-recreate --no-deps findings-web

Recreate each service with its OWN config_files label. The label records the exact -f list a container was created with; recreating a service with a different list silently rewrites its configuration. Compose the overlay only for the two findings keys.

The findings.stella-ops.local alias is contended. It belongs to findings-ledger-web in the canonical services file and is claimed by findings-web in the overlay. Never run both: the window stops the legacy hosts first. FindingsConsolidationConformanceTests fails if either file gains the other’s keys before the window.

After a restart, a service re-sends HELLO to the gateway. A gateway 503 "no instances available" right after a deploy is that handshake, and one docker restart of the gateway fixes it — do not go looking for a routing bug.

Verify

1. docker compose ... ps — both containers healthy. The web healthcheck probes /health/ready. The worker has a healthcheck and must be healthy too (verified 2026-08-06). The hardened runtime base declares HEALTHCHECK CMD /usr/local/bin/healthcheck.sh and every image inherits it, so a worker serving no HTTP surface reports unhealthy forever — measured on the staged JobEngine sibling’s first scratch boot, where its born-new worker sat unhealthy while riskengine-worker and every other estate worker stayed healthy through the shared StellaOps.Worker.Health endpoints. StellaOps.Findings.Worker now hosts those endpoints (/health/liveness + /health/readiness, its only HTTP surface) and the compose key points LIVENESS_PATH / READINESS_PATH at them. Pinned by FindingsConsolidationConformanceTests.ConsolidatedWorker_AnswersTheHealthcheckItsBaseImageInherits. If it goes unhealthy, fix those values — do not delete the healthcheck.

2. Fresh-database convergence, which is the thing worth actually checking on a first deploy:

   docker exec stellaops-postgres psql -U stellaops -d stellaops_findings \
     -c "select schema_name from information_schema.schemata"
   

   Expect findings, findings_ledger_app, findings_archive, findings_security, riskengine, analytics, eventing. A missing set means the host started against the wrong database or the migration host did not run.

3. The doctor surface, which is the honest forcing function available at this stage — it exercises auth, the connection, the migration ledger and the two family checks in one call:

   curl -s -H "Authorization: Bearer <token>" http://findings.stella-ops.local/doctor/findings-web/checks | jq .
   

   Unauthenticated must be rejected; the authorized call returns five checks (doctor.standard.db.connection, .db.migration-status, .db.size-budget, doctor.findings.ledger.tenant-partition-coverage, doctor.findings.ledger.rls-posture). The token needs ops.health.

   Read doctor.findings.ledger.rls-posture before declaring the deploy good. It is the check that catches the one failure mode this family has that most services do not: the host starts, converges its schema, passes its connection check, and still cannot read a single ledger row because fourteen tables carry FORCE row-level security. See the grant note below.

Advisory-generation recovery authority (Q-21 ruled 2026-08-25: dedicated-recovery)

Corpus-generation recovery (hold / rollback / restore / resume) is governed by a dedicated global opt-in scope, findings:recovery, granted to no base persona and no service client. An operator receives it only by explicit incident assignment. Every transition requires a non-empty reason and an incident/change reference; the server records subject, time, operation id, expected/selected digests, checkpoint, and outcome in append-audit evidence. Rollback atomically establishes a durable, non-expiring activation hold (the ordinary projection worker cannot activate a generation past it); restore stays gated by the exact witness/checkpoint test; resume is a separate audited operation admitted only after the active/staging witness and direct canaries are current. There is no automatic expiry and no hand-written-SQL escape. findings:write and vuln:operate must NOT be used for recovery.

Implemented 2026-08-26 (SPRINT_20260722_010 FND-X18-7; settlement receipt in that sprint’s Decisions & Risks). The consolidated findings-web host serves four POST endpoints under /api/findings/v1/security/advisory-generations/recovery/{hold|rollback|restore|resume}, each gated by the findings:recovery policy (policy name == scope claim value). Request bodies carry reason + incidentReference (rollback adds expectedCurrentManifestDigest; restore adds expectedRollbackManifestDigest + expectedCurrentManifestDigest); the subject is derived from the authenticated principal only — a caller without a derivable subject is refused 403 even if network-position bypass satisfied the scope, because an audit row that cannot name its subject is not evidence. Successful transitions and their state change commit in ONE transaction together with the hold row (findings_security.advisory_generation_recovery_hold, singleton, no expiry column) and the append-only audit row (findings_security.advisory_generation_recovery_audit); refused transitions append a failure row carrying the stable code as its outcome. Two operational notes:

• The adapter is fail-closed twice over: without the scope every call is 401/403, and while findings-web’s Findings:AdvisoryCorpus section stays disabled (the shipped default — the corpus flag lives on the worker role) every operation answers the typed advisory_generation_recovery_disabled refusal. Arming the adapter is a deploy-window step: supply the same Findings__AdvisoryCorpus__* values the worker receives to findings-web too (the web host runs no projector from them — it never calls the projection registration).
• Scope issuance is estate work the window owns: the scope is catalogued (S051_findings_recovery_scanner_projection_scopes.sql) and granted to no role or service client; assign it to the named incident operator for the window and revoke after.
• Mint through stellaops-break-glass-operator, not stellaops-cli(SPRINT_20260722_016 AUTH-15, 2026-09-05). That public password-grant client (tenant default, audience stellaops) carries openid profile and nothing else at rest, and no bootstrap client carries findings:recovery (RepositoryStandardDescriptors_GrantFindingsRecoveryToNoClient). Why nothing at rest: the password grant resolves scopes against the client ceiling only (ClientCredentialsHandlers.ResolveGrantedScopes; there is no user-permission intersection), so a standing findings:recovery on any public client would be mintable by every user who can log in. The window is therefore: (1) append findings:recovery to stellaops-break-glass-operator — Console client editor, or standard.yaml + Authority restart; (2) assign the incident-findings-recovery role to the named operator; (3) mint (grant_type=password&client_id=stellaops-break-glass-operator&authority_provider=standard&tenant=default&scope=openid profile findings:recovery); (4) run the audited transitions; (5) revoke the role, revoke the token (POST /console/admin/tokens/revoke, authority:tokens.revoke — minted through stellaops-estate-admin-operator); (6) reset the break-glass row in the database — removing the scope from the YAML alone does nothing because the bootstrapper unions and never shrinks:

  UPDATE authority.clients
     SET allowed_scopes = ARRAY['openid','profile'],
         properties = jsonb_set(properties, '{allowedScopes}', to_jsonb('openid profile'::text), true),
         updated_at = NOW()
   WHERE client_id = 'stellaops-break-glass-operator';
  

  The product CLI client can never be the vehicle again: StellaOpsCliClientScopeCeilingTests fails the commit and migration S058 strips the scope from the database. Census and identity table: docs/modules/authority/operations/stellaops-cli-scope-census.md.

What “revoke after” actually does, and what it needs to be true

Revoking the grant does not, by itself, withdraw a token that has already been issued. Every StellaOps service validates its bearer token locally against Authority’s signing key and none of them consults Authority’s revocation state per call. Measured on the live estate 2026-08-27: with the scope removed from the client, the role removed from the operator, the operator disabled, and POST /console/admin/tokens/revoke returning revokedCount: 1, the already-issued token still established an activation hold. Authority’s own response says so — “It is rejected at the Authority immediately; access tokens already issued expire on their own schedule.” Re-minting is correctly refused invalid_scope; the outstanding token is the gap. Residual measured at 30 minutes (STELLAOPS_AUTHORITY_AUTHORITY__ACCESSTOKENLIFETIME; note devops/etc/authority.yaml still declares 00:02:00, which the deployed env var overrides).

The gateway closes that window for findings:recovery specifically, and only when it is armed. Owner decision D2 (2026-08-28, route (B), implemented 2026-08-29): the router-gateway introspects every request whose validated token carries findings:recovery against Authority’s RFC 7662 /introspect, per call and uncached, and refuses 401 break_glass_token_inactive the moment Authority reports the token inactive. It runs at the gateway because the gateway is the only layer that still holds the token — the four recovery endpoints are reached through a regex route, which strips Authorization and forwards a signed identity envelope carrying neither the token nor its jti, so findings-web has nothing to introspect. No other scope is affected, and a request without findings:recovery makes no round trip.

Operationally this means:

1. A revoke now takes effect on the operator’s NEXT recovery call, not at token expiry — but only for findings:recovery. For every other scope in the estate the 30-minute residual above is still the honest number. Do not generalize this section.

2. The check fails closed. If Authority is unreachable, returns anything other than a 200 carrying active, or the gateway’s introspection credential is missing, break-glass is refused, not admitted. That is deliberate: the capability cannot be obtained without Authority either, so failing closed removes nothing that was genuinely available. It does mean an operator holding a pre-minted token loses it if Authority goes down mid-incident.

3. Confirm it is armed before you rely on it. The gateway states its posture once at startup: Break-glass introspection ARMED for scopes findings:recovery against … (or a warning naming what is missing).

   docker logs stellaops-router-gateway 2>&1 | grep -i "Break-glass introspection"
   

   ARMED is the only line that means the control is live. DISARMED means no scope is guarded; FAIL-CLOSED but incomplete means every break-glass call will be refused 503.

4. The credential is operator-supplied. GATEWAY_INTROSPECTION_CLIENT_SECRET in the git-ignored devops/compose/.env feeds both the authority service (which seeds the confidential stellaops-gateway-introspection client from devops/etc/authority/plugins/standard.yaml) and the router-gateway service (which presents it). They rotate together; restart both. Compose declares it ${…:?required} on both services, so an unset value fails the deploy loudly rather than starting a gateway that silently refuses recovery during an incident. A fresh devops/compose/.env renders it from stellaops.env.local-dev.example via devops/compose/scripts/init-local-env.sh; an existing .env must have the line added by hand before the next docker compose up.

5. A 503 break_glass_introspection_not_authorized_party is a configuration fault, not a revocation. Authority answers an introspection request only for a caller named among the token’s audiences, and tells everyone else exactly what it tells the holder of a revoked token. The gateway checks that precondition itself so the two cannot be confused; if you see this code, the token’s audiences no longer include the gateway’s introspection client id (stellaops-gateway-introspection) — fix allowedAudiences on the minting client rather than re-issuing tokens.

   Mint through stellaops-break-glass-operator and nothing else (AUTH-17, 2026-09-08). That client is the only one whose allowedAudiences names the introspection client, so it is the only identity whose tokens the gateway can introspect. Until 2026-09-08 the introspection client was registered under stellaops — the audience every token in the estate carries — so any client’s token was introspectable and the choice of minting identity did not matter. It does now: a findings:recovery token minted through some other client is refused 503 on this precondition, before any revocation question is asked. The estate-wide stellaops client is deleted, not disabled, so there is no fallback to fall back to.

Verification, both directions, in one window: mint a findings:recovery token the documented way, call POST /api/findings/v1/security/advisory-generations/recovery/hold and confirm 200; revoke it with POST /console/admin/tokens/revoke; repeat the same call and require a refusal on the next call rather than after 30 minutes. One token, one window, both directions — a run that only shows the refusal has not distinguished a working control from a broken one.

The grant decision this deploy depends on (FND-9 owns it)

Fourteen findings tables are ENABLE + FORCE RLS with a policy calling findings_ledger_app.require_current_tenant(), which raises rather than filtering. Measured on the converged baseline (FindingsRoleGrantTests, 2026-08-05):

• GRANT findings_ledger_admin TO <owner> does nothing — BYPASSRLS is a role attribute and attributes are never inherited through membership.
• SET ROLE findings_ledger_admin does confer the attribute and simultaneously loses every privilege, because the baseline creates that role with no grants: the read then fails 42501 permission denied for schema findings.
• ALTER ROLE <owner> BYPASSRLS works unchanged, and is the only one-step remedy that does. The alternative is setting app.current_tenant per session (R-FND3-8 option b), which the service must then do on every connection.

Also measured: GRANT ... ON ALL TABLES IN SCHEMA is a one-time snapshot and partitions do not inherit privileges from their parent, so every per-tenant partition created after the window is unreadable until it is granted explicitly or ALTER DEFAULT PRIVILEGES is in place.

Doctor registration (LIVE since 2026-08-26)

The FND-9 window flipped registration ON for the live stack (FINDINGS_DOCTOR_REGISTRATION_ENABLED=true + FINDINGS_DOCTOR_CLIENT_ID=stellaops-findings-doctor

• the secret in the live .env): the handshake completed and platform.doctor_capabilities holds findings-web’s five checks, heartbeating. The default-off guidance below stays for fresh estates.

Doctor__Registration__Enabled defaults to false. Turning it on before this service’s Authority client is allowed platform:doctor:register produces a 403 loop against Platform, which is exactly what the default-off posture exists to prevent. The identity is seeded in desired state: both repository Standard descriptors declare the least-privilege stellaops-findings-doctor client (platform:doctor:register only; secret from FINDINGS_DOCTOR_CLIENT_SECRET), so the grant lands with the next Authority reconciliation rather than needing a hand-created client. When you do turn registration on, set FINDINGS_DOCTOR_CLIENT_ID=stellaops-findings-doctor plus the secret, and keep Doctor__Registration__TokenEndpoint explicit — Authority serves HTTP discovery in labs but only accepts token requests over HTTPS (ID2083).

Scanner security projection (FND-X18-7) stays default-off

The worker’s hosted Scanner projection pass and the direct owner client share one gate: Findings__ScannerSecurityOwner__Enabled (default false; there is no second worker flag whose value could drift from the network client). While it is off, the boot registers no owner client, no token client, no hosted pass, and no extra database pool. Arming it is a deploy-window step — the shipped stellaops-vs-api://scanner Scanner audience alignment is a live-activation prerequisite. SCN-7 also hard-cuts the producer to exact-tenant physical streams, so a live window must converge Authority migration S053, recreate Scanner/Findings on the new contract, and run the recorded per-tenant rescan/bootstrap before enabling the pass. When you do arm it, supply on findings-worker:

• FINDINGS_SCANNER_SECURITY_PROJECTION_ENABLED=true — the single gate (Findings__ScannerSecurityOwner__Enabled).
• FINDINGS_SCANNER_SECURITY_CONTROL_TENANT — the opaque control-tenant claim (Findings__ScannerSecurityOwner__ControlTenant; empty is accepted only while disabled).
• FINDINGS_SCANNER_PROJECTION_AUTHORITY (and normally FINDINGS_SCANNER_PROJECTION_TOKEN_ENDPOINT — Authority serves HTTP discovery in labs but only accepts token requests over HTTPS, ID2083) — the dedicated token client’s Authority endpoints (Findings__ScannerSecurityProjection__Authority / __TokenEndpoint).
• FINDINGS_SCANNER_PROJECTION_CLIENT_SECRET — the confidential secret for the pinned client id stellaops-findings-scanner-projection. This is the same env key both Authority Standard descriptors read on their side; the Authority container needs it before its next restart (the Standard plugin fails closed on a declared confidential client without its secret), and the worker needs it to mint scanner:projection:read tokens.

The source ships no deployment values: an enabled boot with any of these missing refuses startup naming the exact key. The pass runs under the family’s P6 fenced lease (findings:security-scanner-projection:tenant:<exact-tenant-token>), advances its durable tenant-specific checkpoint only through ScannerSecurityProjectionConsumer’s fenced transaction, refuses sequence gaps and unresolvable epoch/retention states fail-closed (watch for Scanner projection pass stopped fail-closed in the worker log — that state needs the recorded bootstrap/replay procedure, never a hand-advanced checkpoint), and skip-and-counts pre-cutover v1 residue.

Graph asset-registry feed seam (010x023) stays default-off

stellaops_findings contains no graph schema BY DESIGN, so findings-web must never run the predecessor’s same-database graph.asset_registry_events read — the 2026-08-26 FND-9 canary failed closed on exactly that. The consolidated host instead consumes graph-api’s asset-registry event feed behind one gate: Findings__GraphAssetRegistry__Enabled (default false). While it is off, the boot registers none of the asset-registry projection plane — no worker, no projector, no source, no HTTP client — and the host boots healthy against its own database; the dormancy is this recorded deployment gate, not a missing capability. When you arm it, supply on findings-web:

• FINDINGS_GRAPH_ASSET_REGISTRY_ENABLED=true — the single gate (Findings__GraphAssetRegistry__Enabled).
• FINDINGS_GRAPH_ASSET_REGISTRY_FEED_URL — graph-api’s base URL (Findings__GraphAssetRegistry__FeedBaseUrl; in the stack that is http://graph.stella-ops.local). The source ships no value.
• The shared identity-envelope signing key (STELLAOPS_IDENTITY_ENVELOPE_SIGNING_KEY / Router__IdentityEnvelopeSigningKey) on this host: the seam authenticates the internal hop with a self-signed gateway-shaped envelope (subject stellaops-findings-graph-projection, sole machine scope graph:asset-registry:read-all — catalogued by S052, granted to no Authority client by design), which graph-api verifies with the same key.

An enabled boot with a missing/invalid value refuses startup naming the exact key. Once armed, the projection worker’s startup admission proves the local seam relation (findings.asset_registry_feed_state, migration 008) AND the remote feed contract, and every contract-grade failure — refused identity, absent route, stream-epoch divergence, out-of-order page — fails the host closed with the typed refusal routed to the source-unavailable runbook; ordinary transport transients keep bounded retry. Live-activation prerequisite: the deployed graph-api build must carry the feed route (GET /graph/asset-registry/events).

Tenant lifecycle (SCR-4) stays report-only

Catalog__Replication__TenantLifecycle__Enabled is false and DryRun is true. The handler archives-then-detaches this family’s Merkle partitions and never deletes them, but tenant offboarding disposal is not something a deploy switches on. Turning the drain on without flipping DryRun gives you the report with nothing touched, which is the right first step.

FND-X18 advisory generation rollback and catch-up

The one-way door is closed at the source — the adapter now pre-flights its own exit (FND-X18-7, 2026-08-27). hold and rollback establish a durable, non-expiring activation hold whose ONLY lawful removal is the audited resume, and resume is admitted only while the active generation and every catch-up-eligible staging generation carries an SBOM fence witness equal to the canonical checkpoint. Until this change, a staging generation that could never obtain that witness made the hold unremovable and would have frozen the security plane permanently — measured on this estate as SPRINT_20260722_010 F-R0827-CP-8.

Both operations now refuse UPFRONT with the typed code advisory_generation_resume_unreachable(HTTP 409), naming the blocking generation, when resume’s admission predicate is unsatisfiable for a reason the operation itself does not fix. The test is exact rather than cautious: a generation is unreachable only when it has no current witness, is outside the materialization pass’s work-discovery predicate (the sole writer of a fence), AND its publication is at or below the durable corpus checkpoint — which the projector advances only when it activates, so a live import always sits strictly above it. A running import is therefore admitted, and only a generation the consumer has provably moved past is refused.

When you do get that refusal, the remedy is not to force anything. The corpus projector writes an explicit terminal marker (abandoned_at / abandonment_code, migration 010) on its next bootstrap/re-derive, after which the generation stops counting as pending for resume admission, for the served X-StellaOps-Source-State header, and for prune eligibility — and the operation is admitted. The marker is never written while a hold exists, so a rollback’s own staging row cannot be marked or pruned.

The earlier DO-NOT-INVOKE warning for this estate is retired, and what settled it is a cycle, not an argument. In order: the marker and the pre-flight shipped (ed546b4744, 1526cdd5dc) and were deployed live (gitSha=7216591f4359fc84757c0fa6ca9d00f784ac9a34, worktreeState=clean; migration 010 applied on startup 2026-08-27T12:55:39.168Z); the two stranded generations sha256:9357b712… (redhat-csaf 45, artifact_reclaimed) and sha256:d918d118… (kev 48, checkpoint_rederived) were marked terminal, which alone flipped every security read from X-StellaOps-Source-State: stale back to current on an unchanged corpus digest; and then the full cycle was run on this estate: hold → rollback → restore → resume, all 200, with resume ADMITTED — the exact operation that would have been refused forever the day before. The two stranded generations were reclaimed afterwards by the shipped prune statements. Retire a warning like this by exercising the thing it warns about, once the reason for the warning is gone.

A correction to the retired warning’s own settling condition, recorded because a later reader checking the old text would have drawn the wrong conclusion. It said to retire the box once the blocking generation “has left staging”. That named the wrong observable: the row stayed staging right up to the moment it was reclaimed, and the marker excludes it while it remains there. That is a better design than the condition assumed — a terminal generation does not need a state transition to stop blocking, because abandoned_at carries the fact directly and the three consumers read it. The condition should have read “is terminal or otherwise non-blocking”, and the machine-checkable form of it is the query below with its abandoned_at IS NULL filter.

Diagnostic — the adapter checks this for you, but this is what it is checking:

docker exec stellaops-postgres psql -U stellaops -d stellaops_findings -c "
SELECT left(g.manifest_digest,20) AS dg, g.state, g.source_key||' '||g.source_generation AS src,
       g.abandonment_code,
       (w.stream_epoch IS NOT NULL AND w.stream_seq IS NOT NULL
        AND w.stream_epoch = c.stream_epoch AND w.stream_seq = c.seq) AS witness_current
FROM findings_security.advisory_projection_generation g
LEFT JOIN findings_security.advisory_sbom_match_fence w ON w.manifest_digest = g.manifest_digest
CROSS JOIN (SELECT stream_epoch, seq FROM eventing.consumer_checkpoints
            WHERE consumer_id='findings-advisory-sbom-v1') c
WHERE g.state IN ('active','staging') AND g.abandoned_at IS NULL
ORDER BY g.state, g.manifest_digest"

A row with witness_current = f is either catching up (fine — wait for the materialization pass) or stranded (the adapter will refuse and name it). Rows carrying an abandonment_code are excluded by the abandoned_at IS NULL filter because they are terminal, not pending.

This procedure is authorized ONLY through the audited findings:recovery adapter described in the previous section — the Q-21 ruling and its FND-X18-7 implementation. Invoking the internal IAdvisoryCorpusGenerationRecovery primitive any other way, or flipping generation state by hand, is prohibited. AdvisoryCorpusGenerationRecoveryException includes this section’s repository path in every operational failure; do not turn one of those failures into a hand-written SQL pointer flip.

Procedure (each step is one audited POST from the previous section):

1. POST .../recovery/hold — establishes the durable activation hold and records the exact active manifest and canonical findings-advisory-sbom-v1 checkpoint in the hold and audit rows. A route stop alone is not an activation hold; the ordinary projector refuses the staging -> active flip while the hold row exists and reports it in its health detail. (Not required before rollback — rollback establishes the hold itself — but holding first freezes activation while you record digests and plan.)
2. Keep the SbomService owner catch-up and Findings SBOM projection available. Do not stop owner events: both the selected rollback generation and the former current generation must continue to receive tenant inventory changes.
3. Keep both consolidated roles on the same Findings database. Do not drop/recreate vuln, run a down migration, copy owner rows, or edit generation state manually.
4. POST .../recovery/rollback with the just-recorded expected current digest. The code chooses only the newest retained lower generation from the same source; an arbitrary digest is not accepted. The two-row flip, the durable hold, and the audit row commit in one transaction.

The rollback call explicitly rematerializes that retained manifest at the locked canonical SBOM checkpoint. It then acquires the corpus fencing lease and commits exactly two local state changes in one transaction: current active -> staging, prior retained -> active. No generation, corpus, metadata, match, SBOM version, or component row is deleted. Reads already in flight remain pinned to their repeatable-read snapshot; the next request sees the selected prior metadata and matches as one generation.

During the hold, the normal materialization pass discovers both active and staging when their witness lags the owner checkpoint. Let it catch both up. A restore attempt before the former current staging witness equals the locked checkpoint fails with advisory_generation_sbom_witness_stale and performs no state change. Once caught up, POST .../recovery/restore with both recorded digests. It commits rollback active -> retained and former-current staging -> active atomically; the hold stays. Re-run the direct tenant detail/aggregate canary and verify one active row and matching checkpoint/witness. Then POST .../recovery/resume — the separate audited transition that removes the hold; it is admitted only while the active generation’s witness (and every staging generation’s witness) equals the canonical checkpoint, so a premature resume fails advisory_generation_sbom_witness_stale and keeps the hold.

Stable failure codes are:

• advisory_generation_recovery_disabled: the retained-corpus source invariant is not enabled (on findings-web this also means the adapter has not been armed — see the recovery-authority section above);
• advisory_generation_hold_already_established / advisory_generation_hold_absent: the durable hold is single and explicit — establish it once, resume it once;
• advisory_generation_sbom_lease_unavailable / advisory_generation_corpus_lease_unavailable: another writer owns the canonical fence; wait and re-read state before retrying;
• advisory_generation_active_mismatch / advisory_generation_prior_unavailable: the recorded digests are stale or no compatible prior remains; stop and re-plan, never select another row by hand;
• advisory_generation_prior_reclaiming: a compatible prior generation EXISTS and is named in the message, but it is being reclaimed and its projected rows are partially deleted, so it is not a safe rollback target. This is a deliberate refusal, not a recovery malfunction, and it needs no operator action. Corpus reclamation runs as a bounded background pass (FND-18); when it finishes, the generation row disappears and the same call refuses with advisory_generation_prior_unavailable instead — which is the answer the pre-FND-18 code gave immediately, because reclamation then happened inside the activation transaction. So this code marks a WINDOW, not a lost capability: rollback availability in the settled state is unchanged, and what is new is only that the doomed row stays visible in advisory_projection_generation for a while after activation. Confirm rather than assume with SELECT manifest_digest, state, reclaim_started_at FROM findings_security.advisory_projection_generation WHERE reclaim_started_at IS NOT NULL; — a non-null reclaim_started_at is the claim, it is terminal, and it is never cleared. Wait for the row to disappear (default pass cadence 30 s, 60 s budget, 20,000 rows per statement) and re-plan against the real answer. Never clear reclaim_started_at by hand to make a rollback proceed — the generation’s rows are already partially deleted, so promoting it would activate a corpus with silent gaps;
• advisory_generation_not_ready / advisory_generation_sbom_witness_stale: imports or the target witness are incomplete; keep the existing active generation and repair/catch up the named source;
• advisory_generation_resume_unreachable: refused UPFRONT by hold/rollback because the activation hold they would establish could never be removed — a named catch-up-eligible generation has no current SBOM witness, is outside the materialization pass’s work-discovery predicate, AND sits at or below the durable corpus checkpoint, so no mechanism can ever give it one. Nothing changed; settle that generation first (the projector marks it terminal on its next re-derive) and retry. A generation whose import is merely still RUNNING sits ABOVE the checkpoint and is admitted, so this code never fires for ordinary in-flight work;
• advisory_generation_selection_conflict: state moved during admission; the transaction rolled back, so re-read the two generations and checkpoint before any retry.

If the database schema itself must roll back, use the ADR-004 PostgreSQL snapshot-restore procedure. That is separate from this data-generation pointer recovery.

The corpus consumer falls behind: hub artifact retention vs. the event stream

Measured live 2026-08-27 during the FND-9 closure window. Read this before diagnosing a findings-worker that is unhealthy with the security plane serving last-good.

The hub protects its event stream and its artifacts under two policies that do not know about each other.

So a consumer that lags by more than RetainedPublicationCount publications is handed envelopes whose bytes no longer exist, on a page that validates as contiguous and healthy with a null retention block. Measured on this estate: 13 publications between 2026-08-26 14:32:13Z and 2026-08-27 07:12:09Z (~77 min mean interval, one observed burst of 4 in 71 min), 2 artifacts retained (~11.1 GB each, 5 sections), Findings registered as findings-advisory-corpus-v1 and pinned at reported seq 8 while the head was 13. A full generation import measured end-to-end at 36m26s. Retention depth 2 gives roughly 154 minutes of grace, so steady state fits and any stall does not.

What the consumer now does about it (2026-08-27). A manifest or section answering 404/410 raises AdvisoryCorpusArtifactReclaimedException, which is deliberately NOT an input failure: the generation is never marked failed, because nothing was wrong with it. The projector instead takes the remedy the owner names in its own retention.detail — re-derive from GET /api/vulnerabilities/v1/corpus/publications/current and resume at headSeq. Publications between the checkpoint and the head are not replayed; the log line says so by name. Nothing is deleted (the verified-absence path runs only when the owner reports no current publication) and the stream epoch is unchanged. Pinned by AdvisoryCorpusConsumer_ReclaimedArtifactReDerivesFromCurrentRatherThanStalling.

Operator note. This makes the consumer self-healing, but it does not make the gap free: the skipped generations are never projected, so any tenant fact that existed only in them is re-derived from the current publication rather than replayed. The durable fix belongs to the hub — artifact retention should either honour eventing.remote_stream_consumers the way outbox retention does, or publish an artifact-retention horizon the consumer can act on before it walks into a 404. Recorded as a finding in SPRINT_20260722_010 Decisions & Risks; same class as Q-22’s mandatory scanner.scans declared-retention companion.

Producer values the destination schema refuses to store

Same window, and the reason the consumer was stuck in the first place.

Migration 002 shipped ck_advisory_metadata_projection_times (modified_at >= published_at), an invariant upstream advisory sources do not hold. The hub’s first post-backfill advisory-metadata section carried rows published AFTER their last modification — e.g. GHSA-22h7-7wwg-qmgg published 2020-09-04T17:56:39Z, modified 2020-08-31T19:00:24Z. The corrected rates are: corpus-wide 217 of the active generation’s 669,579 metadata rows (0.032%); at the producer input 218 of 865,027 (0.025%), every one from osv, max skew 246 days; and among GHSA-* identifiers specifically 196 of 3,877 = 5.055%. GHSA is 0.58% of the corpus but carries ~90% of all inversions, which is why a GHSA-dense mid-file sample read 3.9% and the corpus does not. Verified at the producer: 0 of the 217 pair timestamps from two different source rows, so every inversion is a single source row published that way — the hub invents nothing, and the selector’s independent field picking (which could in principle manufacture an inversion) demonstrably did not.

Why earlier sections passed is NOT “their metadata was all NULL” — that was the first explanation recorded and it is wrong. The genuine all-null baseline is publication seq 1 at 330 B/row; the generation active at the time (nvd 46, seq 8) was 812 B/row, 2.46× that, because cve finished backfilling and contributed 204,481 candidates (30.5% of rows) before it published. Seq 8 already carried real advisory metadata for roughly the cve third of the corpus. It passed because cve rows carry no inversions; seq 9 broke because it was the first publication after osv — which owns every inversion — completed.

PostgresException was in no classified-failure branch, so the projector neither marked the generation nor stopped retrying: it re-imported 16,062,882 matcher rows every ~4 minutes for over 12 hours, never advanced past checkpoint seq 8, and reported only a generic Degraded readiness. Two fixes, both live:

• Migration 009 drops the constraint. The projection stores what the producer asserts; clamping or swapping the two instants would fabricate a fact the owner never published.
• SQLSTATE class 22 plus 23502/23514 are now corrupt input — typed refusal, generation marked failed with the distinct code invalid_projected_value, previous active generation preserved. Deliberately narrow: 23503/23505 can mean the consumer’s own bookkeeping is wrong, and 40001/40P01 must stay retryable, so none of them are classified. Pinned (with its transient negative control) by AdvisoryCorpusConsumer_RejectedProjectedValueFailsTypedWhileTransientsStayRetryable.

Diagnosing this class. docker logs was useless during this incident — the estate wedge left findings-web, findings-worker and sbomservice with no log stream at all from 2026-08-26T21:40Z onward while other containers logged normally. Do not read log silence as an idle worker. Use the database instead:

# is the projector alive? the lease heartbeat is the answer, not the log
docker exec stellaops-postgres psql -U stellaops -d stellaops_findings -c \
  "SELECT lease_name, holder_id, fencing_token, expires_at FROM eventing.leases"

# is it making progress, or looping? sample twice ~60s apart
docker exec stellaops-postgres psql -U stellaops -d stellaops_findings -c \
  "SELECT left(manifest_digest,20), section_name, import_state, imported_line_count, updated_at
     FROM findings_security.advisory_corpus_section_import ORDER BY updated_at DESC LIMIT 10"

# how far behind is it, and what does the owner still retain?
docker exec stellaops-postgres psql -U stellaops -d stellaops_vuln -c \
  "SELECT publication_sequence, source_key, source_generation FROM runtime.corpus_publication
   ORDER BY publication_sequence"
docker exec stellaops-postgres psql -U stellaops -d stellaops_vuln -c \
  "SELECT * FROM eventing.remote_stream_consumers"

A section whose updated_at advances while imported_line_count returns to the same value is the retry loop, not progress.

Service rollback — CLOSED. Never restart a predecessor.

The service-rollback path is gone as of the 2026-08-27 FND-9 closure. Restarting any of the five predecessor containers (findings-ledger-web, riskengine-web, riskengine-worker, findings-security-web, findings-vulncorrelation) would fork history and is prohibited — the contract is fix-forward.

Measured 2026-08-27: those five containers and their images are gone from the host entirely — docker ps -a lists none of them and docker image ls carries no stellaops/findings-ledger*, *riskengine*, *findings-security* or *findings-vulncorrelation*. Not caused by any command in that window; cause undetermined (SPRINT_20260722_010 F-R0827-CP-9). The compose keys and the four old schemas in stellaops_platform remain, so FND-10’s deletion work is unaffected. The practical consequence: a docker compose up on one of those keys no longer restores anything — it attempts a registry pull of stellaops/<key>:dev, which either fails or resolves to something unrelated.

The hazard is at the ROUTING layer, not the database — and the routing one is worse. An earlier version of this section said a restarted predecessor “would write into the same stellaops_findings database from a stale in-process view.” That is wrong, and wrong in a way that invites the unsafe move (“I will restart it pointed somewhere else and be safe”). findings-ledger-web binds ConnectionStrings__Default, ConnectionStrings__FindingsLedger and findings__ledger__Database__ConnectionString all to the generic platform connection (retired estate-wide on 2026-09-14; it is no longer defined anywhere), i.e. the frozen pre-cutover schemas in stellaops_platform. It never touches stellaops_findings.

What actually bites is that findings.stella-ops.local is a contended network alias. It is declared by findings-ledger-web in docker-compose.stella-services.yml and claimed by findings-web in docker-compose.findings.yml, whose own header already warns that “these two files must never be composed together while BOTH services run”. A restarted predecessor contends for that alias and can silently take live gateway traffic, answering from the frozen platform schemas while the real data sits in stellaops_findings. Nothing errors. Reads simply go stale and writes land in a database nothing reads any more — a split brain at the routing layer, which is harder to notice than a database fork because there is no exception to find.

Independently of that, they were never a rollback target: since the cutover they have missed every ledger append, every disposition change and, since 2026-08-27, an entire advisory corpus generation.

Note what “revocation” does and does not mean here, because the old wording implied a gate that never existed on this estate: grant revocation was MEASURED a structural no-op — all five predecessors connected as the stellaops superuser and zero explicit grants exist on the four old schemas or on graph, so there was nothing to revoke and no moment at which revocation closed the door. What actually closes it is the stopped containers plus FND-10’s deletion. On an estate with per-service roles the revocation step is real and would be the boundary.

Recovery instead of rollback, in order of preference:

1. Redeploy the consolidated pair at a known-good commit. Each build-service-publish.sh run preserves the outgoing image as stellaops/<key>:dev-<timestamp>; append a pin last in the compose chain (never edit an existing pin — the last one wins) and force-recreate the two services with --no-deps.
2. Corpus-generation recovery through the audited findings:recovery adapter — but read the STOP box in the rollback section above first, because an abandoned staging generation makes the hold unremovable.
3. Database rollback is the ADR-004 PostgreSQL snapshot-restore procedure, which is separate from both of the above.