Runbook — moving Policy onto its own database (stellaops_policy)

Audience: operators performing the ADR-039 database-per-service move for policy-engine, and anyone standing up a NEW installation who wants to know why the gate projection is empty on day one.

Sprint: SPRINT_20260722_007 POL-F6. Standards: ADR-039, CODE_OF_CONDUCT §8.2, AGENTS.md §2.7 (auto-migration) and §2.11 (no fixture data in migrations).

STATUS 2026-08-15: this cutover is DONE on the local estate. The third attempt HELD, after two rollbacks on 2026-08-14. policy-engine runs on stellaops_policy, the §0 prerequisite (SCR-3 tenant replication) is flipped and verified live, the corpus seed completed (10,499,248 issues derived into 10,499,248 decision rows, ~4h42m), and the KEV forcing function fired: CVE-2014-0160 on pkg:deb/debian/openssl → HTTP 403 blockedBy: KevReachable, attested, WITH its policy.gate_decisions history row — the step attempt 2 never reached. Isolation 25/25, coverage 7/7.

One verification step below is NOT satisfied and is knowingly outstanding: §5 step 3 (lag returns to 0). Measured 2026-08-15: the drain applies ~6,638 events/min while the hub produces ~6,421/min, so the ~490k backlog that accrued during the 4h42m seed is draining at a NET ~217/min — order of 37 hours, converging but slow. This is a drain-vs-hub throughput property, not a symptom of the database move: the checkpoint advances, rows agree with it exactly, and a lagging projection serves last-applied state by contract (T8_WhileLagging_GatesServeLastAppliedState_AndCatchUpIsOrdered). Re-check before treating lag as evidence of anything else.

Everything below remains the procedure for a NEW installation.

The one thing that will surprise you. A freshly converged stellaops_policy has the correct schema and cannot yet project anything, because the gate projection derives one decision row per (tenant, issue) and a fresh Policy database has no tenants. The tenant catalog is operator data. Until an operator creates the first policy pack, the projector correctly refuses to do anything and says so in the log. This is not a defect and there is nothing to repair — it is step 4 below.

0. Blocking prerequisite: SCR-3 tenant replication (SPRINT_20260722_027)

DISCHARGED — this prerequisite is met and the legacy path it describes no longer exists. policy-engine has resolved tenants from its own replica since 2026-08-15 (catalog.replica.tenants converged in stellaops_policy), and SCR-3 retired PolicyStellaOpsTenantResolver on 2026-09-12; the host now registers AddStellaOpsTenantResolverReplicaOnly, so there is no branch left that could reach shared.tenants. The section is KEPT because the failure it documents is the reason the replica exists, and because anyone reading this runbook against an older image still needs it. Flag-off is now a refusal naming the flag, not a 42P01.

Historically, Policy resolved tenant slugs to UUIDs by reading Platform’s shared.tenants. PolicyStellaOpsTenantResolver issued SELECT … FROM shared.tenants, and that relation lives in stellaops_platform. It does not exist in stellaops_policy and must not be created there — it is Authority/Platform-owned data, and copying it would be the cross-service duplication ADR-039 exists to prevent.

Measured on the 2026-08-14 attempt: a gate evaluation against stellaops_policy resolved its baseline, materialized the target snapshot, computed the delta, decided Allow, and created an attested verdict — then failed with 42P01: relation "shared.tenants" does not exist inside GateDecisionHistoryRecorder.RecordAsync, so the caller got HTTP 500 and the decision history was never written. The identical sequence on stellaops_platform returns HTTP 200, which is what attributes the failure to the move rather than to the gate.

The designed exit already exists and is a config flip, not new code: Program.cs registers AddStellaOpsTenantResolverDualMode, so setting Catalog:Replication:Tenants:Enabled=true plus a FeedBaseAddress serves the same IStellaOpsTenantResolver seam from Policy’s own replica of Authority’s tenants catalog.

0.1 What SCR-3 landed on 2026-08-14, so you do not re-derive it

The flip is now a single staged overlay, devops/compose/docker-compose.policy-catalog-replica.yml — deliberately a separate file from docker-compose.policy-db.yml so you can apply it alone on the shared database first. Applying both at once gives a failure two candidate causes and no rollback that isolates either. It carries the three settings that were each a measured live failure at the SCR-3 proof: the unprefixed key form, the gateway https feed address, and AuthTenant: default.

Three prior items on this list are closed:

0.2 What is still on YOU before the flip

Flip and verify SCR-3 first, on the shared database, where a regression is visible and reversible. Then run §4.

1. What “the tenant catalog” actually is

IProjectionTenantCatalog is the projector’s answer to “which tenants must this decision be derived for”. The shipped implementation (PostgresProjectionTenantCatalog, in src/Policy/__Libraries/StellaOps.Policy.Persistence/Postgres/VulnGate/VulnGateProjector.cs) reads Policy’s own tables and nothing else:

SELECT tenant_id FROM policy.packs
UNION
SELECT tenant_id FROM policy.vuln_tenant_overlay

A tenant with no policy pack has nothing to gate, which is what keeps the per-event fan-out proportional to real tenants rather than to every string that ever appeared in a tenant column.

So there are exactly two legitimate ways a tenant enters the catalog:

SourceHow it is populatedWho does it
policy.packsPOST /api/policy/packs (scope policy:edit) → PostgresPolicyPackRepositoryan operator, or any tooling holding policy:edit
policy.vuln_tenant_overlayDC-32 inbox materialization of Findings-ledger overlay events (tenant VEX decisions / trust overrides)the tenant, through Findings

And one illegitimate way, which is why this page exists: seeding a pack in a migration. AGENTS.md §2.11 forbids QA/demo/fixture rows in any runtime migration, and a policy pack is operator content — a migration that invented one would create a phantom tenant on every real installation, air-gap estate and CI database the moment the service started. There is deliberately no bootstrap flag that fabricates one either; *_BOOTSTRAP_ENABLED is explicitly not an exemption.

Therefore: at least one policy pack (or one tenant overlay) is a documented prerequisite of a meaningful gate projection — on a cutover and on a brand-new installation alike.

2. Preconditions

3. Order of operations, and why the order is load-bearing

The projection has two ways to fill: a one-shot corpus bulk seed (one ordered pass over the hub’s consensus-inputs section — minutes to hours) and the event drain (increments, event by event). On a fresh database the seed is what converges the ~10M-issue corpus; the drain would need days.

The seed’s only skip condition is “a durable checkpoint exists”, and the drain WRITES that checkpoint. So the two race for first fill, and the drain would always win — it runs continuously while the seed is attempted per lease acquisition. That race cost a real cutover window on 2026-08-14: the operator’s first pack appeared mid-loop, the running drain immediately had a tenant to derive for, consumed the small live backlog and checkpointed it, and the next start skipped the seed for “a checkpoint already exists” — leaving a projection holding 84,826 rows instead of 10.5M, behind a consumer that reported itself caught up.

Both halves are now enforced in code, so the sequence below is safe rather than merely recommended:

4. The cutover

  1. Repoint. Append the staged overlay to the stack’s own -f chain — read the chain off the live container, never hand-reconstruct it:

    cfg=$(docker inspect stellaops-policy-engine \
  --format '{{index .Config.Labels "com.docker.compose.project.config_files"}}')
IFS=',' read -ra files <<< "$cfg"; args=(); for f in "${files[@]}"; do args+=(-f "$f"); done
args+=(-f devops/compose/docker-compose.policy-db.yml)
docker compose -p stellaops --project-directory devops/compose "${args[@]}" \
  up -d --force-recreate --no-deps policy-engine

    Once the cutover is permanent, the canonical variable lives in docker-compose.stella-services.yml and the overlay is no longer needed.

  2. Confirm the schema converged in the NEW database (59 base tables + 7 views in policy, 7 in eventing) and that policy.schema_migrations lists every migration through 006_exploit_evidence_projection.sql.

  3. Confirm the projector is refusing honestly, not failing:

    [WRN] Gate projection has NO active tenants, so the corpus seed is a no-op and is
      being skipped: `policy.packs` and `policy.vuln_tenant_overlay` are both empty
      in THIS database …

    eventing.consumer_checkpoints must be empty at this point. A checkpoint here would mean the seed’s window has already closed.

    If that warning appears on an estate that does have packs, the projection resolved a different database than the rest of the service. That is the split-brain defect this runbook’s precondition guards: check PolicyPersistenceConnection and the image version.

  4. Create the first policy pack — the operator action that establishes the tenant catalog. Any policy:edit holder can do it; the pack is real configuration, so name it for what it is:

    curl -sk -X POST https://<host>/api/policy/packs \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"packId":"default-release-gate","displayName":"Default release gate"}'

    Verify the row landed in stellaops_policy.policy.packs (which also independently proves the write path repointed).

  5. Let the seed run. Within one poll interval the host re-attempts the seed and logs progress with both counts — issues derived and decision rows written. They must move together; that pairing exists because a version that reported only the issue count once claimed 10,504,824 issues derived while persisting nothing:

    [INF] Corpus seed progress: 300000 issues derived across 1 tenant(s), 300000 decision writes.

    Expect roughly 500 issues/s (≈ 5–7 h for a ~10M-issue corpus). The completion checkpoint is written only after the last row commits, so a seed interrupted half way is resumed by re-running, never mistaken for a finished one.

  6. Gate posture during the seed is fail-closed, by design. With AdvisorySource=PolicyProjection and no checkpoint yet, the findings lookup reports the source as unavailable rather than as “nothing affects this release”, and FindingsLookup:OnUnavailable=Deny blocks. Releases are refused during the window; nothing is certified against an empty projection.

5. Verifying it actually worked

A healthy container is not evidence. All five of these, in order:

  1. The tenants replica converged (SCR-3, and the reason attempt 2 rolled back — check this BEFORE anything else, because everything downstream needs a tenant UUID):

    docker logs stellaops-policy-engine 2>&1 | grep -i "tenants replica"
#  expect: "Tenants replica bootstrap drain applied N change(s) before startup completed."

    -- in stellaops_policy
SELECT catalog, key, version, deleted FROM catalog_replica.replica_rows WHERE catalog = 'tenants';
SELECT consumer_id, stream, stream_epoch, seq FROM eventing.consumer_checkpoints
WHERE consumer_id = 'catalog.replica.tenants';

    The checkpoint row is the load-bearing one, not the replica rows: it is what distinguishes “this tenant is unknown” from “this replica never drained”, and the resolver refuses to treat a miss as an answer without it. No checkpoint ⇒ every tenant-bound request fails with CatalogReplicaUnavailableException naming the flag, the catalog:replicate grant and the drain. That is correct behaviour — go fix the grant or the feed address, do not read it as a code bug.

  2. Rows exist: SELECT count(*) FROM policy.vuln_gate_current > 0 in stellaops_policy, and in the same order of magnitude as the corpus.

  3. The checkpoint agrees with the rows: exactly one row in eventing.consumer_checkpoints for policy.vuln_gate, written after the seed finished. A checkpoint over an empty projection is the one state a release gate must never serve from.

  4. Lag returns to 0: Gate projection lag: N hub events outstanding falls to 0 and stays there.

  5. A real gate decision, not a health probe. The strongest available forcing function is the KEV gate: a release subject carrying a KEV-listed, reachable vulnerability must evaluate to HTTP 403 with blockedBy: KevReachable, and a clean subject must Pass. Both verdicts are attested. The is_kev signal comes from policy.exploit_evidence, mirrored from the hub’s KEV catalog on the projection lease — check it is populated (~1.6k rows) rather than assuming it.

    Pick the subject from the live corpus, not from a previous run’s notes. The 2026-08-13 forcing function used pkg:maven/org.apache.logging.log4j/log4j-core; the corpus was re-ingested from upstream in the topology window and no longer carries a maven product key for CVE-2021-44228 at all, so that subject now yields unmatched — honest, and useless as a block proof. Query for a subject the projection actually holds:

    SELECT g.vulnerability_id, g.product_key
FROM policy.vuln_gate_current g
JOIN policy.exploit_evidence e
  ON e.vulnerability_id = g.vulnerability_id AND e.kind = 'exploited_in_wild'
WHERE g.effective_status = 'affected'
LIMIT 10;

    Verified working 2026-08-14 on the shared database: CVE-2014-0160 (Heartbleed) on pkg:deb/debian/openssl → clean subject HTTP 200 status 0, KEV subject HTTP 403 blockedBy: KevReachable, condition is_kev = true AND delta_reachable > 0.

    The chain that produces those two verdicts, in order: POST /policy/orchestrator/jobs (scope policy:run, body carries batch_items[{component_purl, advisory_id}]) → wait for status: completed → POST /policy/ledger/export (scope policy:audit, body {"tenant_id":"default"}) → POST /api/v1/policy/gate/evaluate with missingBaselineMode: "bootstrap". Run it once with a subject the projection answers unmatched/not_affected for to bootstrap the baseline, then again with the KEV subject.

  6. The decision HISTORY row landed. This is the step attempt 2 skipped, and it is the difference between “the gate decided” and “the gate is auditable” — the 2026-08-14 rollback had a correct Allow, an attested verdict, and no history row, because the audit write ran after the verdict and 500ed on its own.

    -- in stellaops_policy, after each evaluate above
SELECT decision_id, tenant_id, gate_status, bom_ref, evaluated_at
FROM policy.gate_decisions ORDER BY evaluated_at DESC LIMIT 5;

    One row per evaluation, tenant_id = the canonical UUID the replica served (not null, not a synthetic value). An HTTP 200/403 with no row here means tenant resolution failed after the decision — re-read step 0.

6. Isolation — the boundary is the revoke, not the connection string

bash tools/scripts/deploy/postgres/probe-database-isolation.sh

Every role must reach exactly its own database. Read the printed role count: a family whose connection string is assembled in a compose file rather than .env is invisible to the probe, and the script’s coverage gate now refuses to exit 0 on a partial matrix rather than reporting a green boundary it never tested. Follow the recipe in the script header to include such a family (the vulnerability hub is one).

7. Rollback

The rollback is the connection string, and it is honest only while the old database still holds a consistent projection:

Re-running the cutover after a rollback is safe. If a previous attempt left a drain-written checkpoint on the new database, the seed will skip; clear that consumer’s checkpoint and inbox rows (consumer_id = 'policy.vuln_gate') and the derived policy.vuln_gate_current rows so first fill is unambiguously the seed’s.

Rollback ORDERING with the catalog flip (measured 2026-08-17, routed from the SCR-3 lane): the catalog flip rolls back LAST, never first. Since the 08-15 window, Catalog__Replication__Tenants__Enabled: "true" is a literal in canonical compose and shared.tenants does NOT exist in stellaops_policy (to_regclass → NULL, verified live). Rolling back the catalog flip while Policy remains on its own database reintroduces the exact 42P01 → HTTP 500 audit-write failure that forced both 08-14 rollbacks — a gate that decides correctly and then fails its history write. The database-move rollback is safe in isolation (catalog_replica remains converged in stellaops_platform); the two rollbacks are NOT independent. Sequence: connection string first, catalog flip only after Policy is back on a database where shared.tenants resolves — or leave the flip in place entirely, which is always safe. Those three are all derived data — the packs and the exploit-evidence mirror are not, and must be kept.