Graph — building and deploying the staged consolidated stack (GRA-8)

Sprint: SPRINT_20260722_023 (GRA-8 stages; GRA-9 cuts over; GRA-10 deletes).

Nothing here changes the running stack. graph-api and reachgraph-web keep serving from docker-compose.stella-services.yml (lines 2378 and 2447) until the GRA-9 window. reachgraph-web survives GRA-9 as a container and retires at GRA-10, not before.

Why this family adds no publish or ownership keys — and the one flip that must wait for GRA-10

Findings, JobEngine and the vulnerability hub registered consolidated deployable keys at their S8 because their hosts were born new (zero violation pairs, so the keys were free). The AdvisoryAI sibling could not, because a second key over one continuation host re-counts the same source graph. This family needs neither: GRA-3 chose continuation, so graph-api — the surviving key — already exists and already carries the closure. DeployableKeys_AreUnchanged_BecauseTheSurvivingKeyAlreadyExists pins that, in both directions, against the publish helper and the ownership manifest.

One flip that looks like GRA-8’s job is not. The src/Graph/__Libraries/StellaOps.ReachGraph.Application rule in docs/architecture/build-boundary/ownership-manifest.json carries owner reachgraph, and its note used to say the D14 re-classification to graph “lands with GRA-8”. It does not, and doing it here breaks the MBI-2 gate — the note has been corrected in place, but the instruction may survive in other people’s notes, so here is the measurement. 2026-08-05 the owner was flipped on the working tree, tools/scripts/build-boundary/generate-build-boundary-report.ps1 -Check was run, and the tree restored:

violation pairs : 136   (from 135)
unpinned pairs  : 1
  UNPINNED violation: reachgraph-web -> graph (1 projects;
  witness: src/ReachGraph/StellaOps.ReachGraph.WebService/StellaOps.ReachGraph.WebService.csproj
        -> src/Graph/__Libraries/StellaOps.ReachGraph.Application/StellaOps.ReachGraph.Application.csproj)
exit 1

The cause is structural, not a tuning problem: reachgraph-web is still a deployable and it still references the lifted library, so any owner other than reachgraph makes its own host’s edge cross-family. The re-classification is correct only once GRA-10 deletes the host, the compose key and the publish key — at which point the library is in graph-api’s closure alone. Restored state re-verified: 135 pairs, 0 unpinned, OK.

EXECUTED 2026-09-08 (GRA-10). The host, its compose service block, its publish-helper key and its services-matrix row went in one change with the four-project re-classification and the StellaOps.ReachGraph.Contracts producer flip, in that order. The regenerated report reads 48 deployable keys, 34 violation pairs, 0 unpinned, 0 stale, OK— the graph-api|reachgraph approvedMergeEdges entry was deleted in the same change because the pair left the derived graph and the gate reported it STALE. This section is kept rather than deleted: it is the measurement that set the ordering, and the ordering is the reusable part.

What the deployed image serves versus current source (read this before rehearsing)

The live graph-api image is still the pre-GRA-9 host. Current source adds the GRA-9 pre-live composition, but nothing below is live until that image is rebuilt and recreated. The old image serves its legacy Graph endpoints plus one thing GRA-7 added:

• GET /doctor/graph-api/checks — doctor-check/v1, five checks, guarded by the ops.health scope (GraphPolicies.HealthRead).

The current/live distinction is load-bearing:

1. Current source serves /api/graph/v1/**; the deployed image does not. The source now explicitly composes the lifted controllers and publishes every subPrefixMap plane as real RouteEndpoint aliases before Router’s HELLO refresh. Middleware rewriting is deliberately not used because Router-dispatched calls bypass middleware. The 2026-08-05 live-image probe remains 404, so the window must probe the newly built host DLL and recreated container before it relies on that plane. Route/WAF tests alone do not prove the published host (D-GRA4-1). (The probe obligation is unchanged; the route swap it once gated is retired — see “Abort criteria”.)
2. Current source serves the lifted reachability CAS; the deployed image does not. Until the rebuild/recreate and forcing function pass, reachgraph-web keeps running and owns its direct Scanner alias — see “The two direct wires” below.
3. Current source arms the consolidated baseline through Graph’s own connection; the deployed image does not. A fresh database reached through the new image and canonical key converges the GRA-3 baseline with the legacy authorities stood down together. Prove that on the composed stack; the standalone overlay still lacks the live Router identity-envelope configuration.
4. Its own-database-cutover check reports “Pre-cutover”. That is the correct staging answer at Info/healthy, not a defect. See the next section.

The connection fan-out — the one thing in the overlay that must not be “simplified”

GRA-7 found that the two old hosts used different legacy keys. GRA-9’s graph-api source now resolves STELLAOPS_POSTGRES_GRAPH_CONNECTION first and uses one effective value for its migration authority, repositories, saved views and doctor data source. reachgraph-web still reads ConnectionStrings:PostgreSQL until GRA-10, so the overlay continues fanning the one operator-supplied value onto the legacy host key during the two-role window.

The default live compose file still carries Graph’s historical generic fallback, and compose merges environment maps. The new resolver does not consult that fallback when the canonical Graph key is present, while ${STELLAOPS_POSTGRES_GRAPH_CONNECTION:?…} makes compose refuse the window start when it is unset or empty. Still verify doctor.graph.database.own-database-cutover: it measures current_database() and reports Critical when the canonical value and the actual connection disagree.

reachgraph-web is deliberately different from graph-api during this two-role window: it retains the direct reachgraph alias but is no longer a migration authority on the target. The overlay must set both ReachGraph__ConsolidatedTarget__Enabled=true and ReachGraph__StartupMigrations__Enabled=false. A startup guard registered before the legacy migration host rejects a target database other than stellaops_graph, a missing/mismatched target declaration, or a missing/true migration flag before reachgraph.schema_migrations can be created.

Build the images

bash devops/docker/build-service-publish.sh graph-api
bash devops/docker/build-service-publish.sh reachgraph-web

Both keys are pre-existing and self-referencing (stellaops/graph-api:dev, stellaops/reachgraph-web:dev), so unlike the JobEngine sibling there is no predecessor-base-image step.

The doctor SDK is prune-sensitive. StellaOps.Doctor.Plugin.Abstractions.dll matches the image pruner’s *.Plugin.*.dll pattern by NAME and survives only because it is allowlisted in build-service-publish.sh (line 370). graph-api ships it as of GRA-7, which turns a pre-existing estate allowlist into a dependency of this family: if graph-api ever crash-loops at boot with a FileNotFoundException for that assembly, the allowlist regressed — do not “fix” it by removing the doctor adoption.

Same-tag images do not swap on up -d. After a rebuild use --force-recreate --no-deps <key>, and expect the service to re-send HELLO over Valkey; a gateway 503 "no instances available" right after a swap is that handshake, not a broken deploy, and one docker restart fixes it. Embedded-SQL or plugin changes need an image rebuild, not a recreate — the migration .sql files are embedded resources.

Bring up an isolated scratch stack

Never against the live network before the window: the overlay claims the graph.stella-ops.local, cartographer.stella-ops.local and reachgraph.stella-ops.local aliases that the live services already hold, so composing both while both run is a split-brain on the aliases as well as on the schemas.

cd devops/compose
STELLAOPS_GRAPH_NETWORK=grascratch \
STELLAOPS_GRAPH_NETWORK_EXTERNAL=false \
STELLAOPS_FRONTDOOR_NETWORK=grascratchfd \
STELLAOPS_FRONTDOOR_NETWORK_EXTERNAL=false \
GRAPH_ROUTER_ENABLED=false \
STELLAOPS_POSTGRES_GRAPH_CONNECTION='Host=…;Database=stellaops_graph;Username=…;Password=…' \
KESTREL_CERT_PASSWORD=… \
STELLAOPS_CERT_VOLUME=../etc/authority/keys:/app/etc/certs:ro \
  docker compose -p grascratch -f docker-compose.graph.yml up -d

The overlay is NOT a self-sufficient environment, and the gap is auth-shaped

The overlay’s x-router-transport anchor carries three Router__* keys. The live services file gives both hosts ~22, from x-router-microservice-defaults (docker-compose.stella-services.yml:55-83). The ones that only exist there include Router__Gateways__0__*, Router__Messaging__Transport, Router__Messaging__valkey__ConnectionString, Router__Frame__Envelope__Keys__primary and — the one that changes what you can test — Router__IdentityEnvelopeSigningKey: "${STELLAOPS_IDENTITY_ENVELOPE_SIGNING_KEY}".

graph-api’s only identity sources are an upstream-authenticated principal and a gateway-signed identity envelope (X-StellaOps-Identity-Envelope + signature; GraphHeaderAuthenticationHandler). It does not accept a raw Bearer token, and with no signing key configured it refuses to authenticate at all and logs “Identity envelope received but no signing key is configured”. So on a standalone scratch stack every authenticated Graph endpoint answers 401 regardless of the credential you present, /doctor/graph-api/checks included. That is the missing key, not a broken guard.

Two honest options, and they are different tools:

• Connectivity / schema / ownership rehearsal — standalone overlay with GRAPH_ROUTER_ENABLED=false as above (reachgraph-web needs no such variable: since SPRINT_20260904_003 ROA-7 both compose files hard-code Router__Enabled: "false" for that role, because graph-api owns its route prefixes). Verifies the hosts start, resolve their own database and converge a schema. Cannot exercise an authenticated endpoint.
• Anything auth-, routing- or doctor-shaped — compose the overlay with docker-compose.stella-services.yml, which is where the Router environment lives. This is also exactly what the window does.

Neither path was rehearsed while writing this runbook (docs-only task, no Docker); the statements above are read off the two compose files and the handler’s source, and the failure mode is named so a rehearsal that hits it is diagnosed in one step instead of read as a regression.

Joining the overlay to the live stack (the window’s compose step)

The overlay enters the config_files set at the window and not before. Three gates:

1. Use the measured live frontdoor network name. The estate has stellaops_frontdoor; it does not have a network named merely frontdoor. Before rendering any service chain, require the real network and pass its name explicitly:

   docker network inspect stellaops_frontdoor >/dev/null
   export STELLAOPS_FRONTDOOR_NETWORK=stellaops_frontdoor
   export STELLAOPS_FRONTDOOR_NETWORK_EXTERNAL=true
   # The exact per-service -f chain is assembled in step 2; render it before up.
   docker compose -p stellaops --project-directory <repo>/devops/compose \
     <-f each file from the service label> -f docker-compose.graph.yml config \
     | grep -F 'name: stellaops_frontdoor'
   

   A missing network, a rendered name: frontdoor, or any value other than stellaops_frontdoor is an abort before the first recreate.

2. There is no canonical -f list. On the live dev stack the com.docker.compose.project.config_files label is not uniform across containers — each service was last recreated by a different process with a different -f set. Read the label off the container you are about to recreate, append -f docker-compose.graph.yml, and recreate that service with its own list:

   docker inspect stellaops-graph-api \
     --format '{{ index .Config.Labels "com.docker.compose.project.config_files" }}'
   # then, with that list + the overlay, from devops/compose:
   docker compose -p stellaops --project-directory <repo>/devops/compose \
     <-f each file from the label> -f docker-compose.graph.yml \
     up -d --force-recreate --no-deps graph-api
   

   Using a guessed list either fails no such service or silently recomputes a different effective config. Orphan-container warnings for services absent from a partial set are benign — do not pass --remove-orphans.

3. The overlay must never reach a default bring-up before the window. StagedComposeOverlay_IsNotWiredIntoAnyDefaultBringUp enumerates every .sh/.yml/.yaml under devops/ and fails if anything references the overlay filename, and it also asserts both legacy keys are still in the canonical services file — because the overlay is staged, the canonical definitions are what actually runs.

The two direct wires — and only one of them is live today

Both of this family’s “break equals silent data loss” contracts bypass the gateway, so no route test can cover either one and the dependency they rest on is a compose ALIAS. They are not in the same state, and the difference decides what the window’s forcing functions prove.

Consequences the window must not get wrong:

• The cartographer.stella-ops.local alias stays on graph-api, the surviving host (TheCartographerAlias_StaysOnTheSurvivingHost). Its round-trip is a real, runnable forcing function today.
• The reachgraph.stella-ops.local alias stays on reachgraph-webuntil the GRA-9 window proves the lifted CAS on graph-api (TheReachGraphAlias_StaysOnTheDyingHost_UntilTheWindowMovesTheDirectWire). Moving it before the new image and forcing function are proven can still point the publisher at the old host image’s 404, while the scan itself succeeds. The mechanism is not what the sprint prose says, and the difference changes where you would look. HttpReachGraphPublisher handles a non-success status correctly — it logs and throws (:105-116). The loss happens one layer up: ReachabilityPublishStageExecutor:122-127 wraps the publish in catch (Exception) and logs “continuing without cross-module store row” at Warning. So a broken wire leaves one warning line in the scanner-worker log and a green scan, and there is nothing to fix in the publisher’s status handling. With PublishToReachGraph off there is not even a warning, because the whole block is skipped. 2026-09-14 (SPRINT_20260914_001 VRP-5): the block, the publisher and the keys are gone; the paragraph above is the record of why the wire was never a usable forcing function.
• “A live scan whose subgraph is then queryable” proves nothing unless the publisher is switched on. GRA-1 named that as the S8 forcing function and the prose since then has read as though the wire were hot; it is not. To exercise it, compose docker-compose.sprint057-smoke.override.yml or set the two keys explicitly, and note that the smoke override’s base URL is http://reachgraph.stella-ops.local:8080 — the alias with the port the app actually listens on.
• Do not probe reachgraph-web on 127.1.0.22:80. Its live port mapping is "127.1.0.22:80:80" while ASPNETCORE_URLS is http://+:8080, so that published port targets a container port nothing listens on. graph-api’s mapping (127.1.0.20:80:8080) is correct. Probe reachgraph-web from inside the network on :8080, or docker compose exec. Recorded rather than fixed: the container retires at GRA-10, and editing a live port mapping is not a docs task.

Verification points

Run these against the composed stack, not a standalone overlay (the auth gap above).

1. The doctor surface answers with this host’s posture. Anonymous must be 401 — a 200 means the ops.health guard was lost. An authorized call needs an ops.health scope through the gateway, because the envelope is what graph-api authenticates.
2. doctor.graph.database.own-database-cutover— the fail-open detector. Expected Info/healthy with “Pre-cutover: connected to the shared database …” before GRA-9, and Info/healthy “Connected to the family’s own database ‘stellaops_graph’ on the complete consolidated migration set” after. Critical means the repoint silently did not take: evidence carries config.canonical-variable.set, config.host-key.set and database.connected, which name the failure directly. A second Critical shape — “connected to stellaops_graph but the consolidated baseline has never been applied there” — means the database was created but GRA-9’s wire-up did not run. The second required ledger row is 002_force_reachgraph_tenant_rls.sql; its absence is separately Critical.
3. doctor.graph.reachgraph.cas-tenant-isolation— the CAS relies on row-level security for tenant isolation and PostgresReachGraphRepository carries no tenant predicate of its own, so this is the only forcing function for the GRA-9 grant decision. Three distinct failures, all Critical, and they fail differently: the connecting role bypasses RLS (BYPASSRLS or superuser); RLS disabled or table absent; RLS enabled but not FORCED (a table-owning runtime role bypasses it); RLS enabled with no policy, which denies everything rather than leaking and presents as an empty CAS. Pre-cutover, before the reachgraph schema exists in the target database, it reports Info/healthy “nothing to measure” — honest, not a pass.
4. Both aliases resolve to the hosts named in the table above — from another container on the stellaops network, not from the host.
5. A Scheduler-driven Cartographer round trip. POST /graphs/build on scheduler-web (scope scheduler:operate), then confirm graph-api received POST /api/graphs/builds. This is the wire that is live, so a failure here is a real regression.
6. A reachability CAS PUT/GET by digest, direct against the serving host. Note what this does and does not prove: it exercises the CAS. (Until 2026-09-14 only enabling PublishToReachGraph proved X20’s publisher wire; SPRINT_20260914_001 VRP-5 retired that publisher, so the CAS round-trip is the whole proof.)
7. Route probe on the real host, never WebApplicationFactory(D-GRA4-1: under the factory the dependency context is the test assembly’s, so the suite is structurally incapable of catching a controller-discovery regression). Booting notes: reachgraph-web needs ASPNETCORE_ENVIRONMENT=Testing plus any connection string; graph-api needs Testing and no connection string at all — with one set the host blocks on the migrator for 60s and exits non-zero. With no connection string the doctor data source is not registered (it is registered only when a connection resolves), so both family checks report Warning / not-healthy “Service registers no database”. That is the expected probe posture.

Abort criteria

Abort the window — do not improvise — if any of these is true:

• pwsh tools/scripts/gra5-swap-graph-routes.ps1 -Check does not print RESULT: manifest and live route configs agree. and exit 0. WITHDRAWN 2026-09-04 — this criterion can no longer be met and is not the gate. Since ced8a2d286 the script refuses on both invocations: -Check and a bare run each report the GRA-5 swap as RETIRED and exit 2 (measured at HEAD). RAR-7 (86f5ba998b) retired the rows the swap replaced, and graph-api publishes /api/graph/v1 natively, so there is no manifest/table agreement left to assert. Confirm the end state with RouteSwapManifestStateTests instead — see docs/runbooks/graph/graph-cutover-gra5-route-swap.md.
• generate-build-boundary-report.ps1 -Check reports unpinned or grown pairs. One unpinned pair with witness reachgraph-web -> graph means someone executed the ownership-manifest note early; revert that edit rather than pinning it.
• The host does not yet answer /api/graph/v1/* natively when the swap is about to be applied. Verify with a real call. Swapping first gives a 404 farm with green tests.
• doctor.graph.database.own-database-cutover is Critical after the repoint, or doctor.graph.reachgraph.cas-tenant-isolation reports a bypassing role or an unforced table. The first means the move did not take; the second means tenant isolation is gone.
• the rendered frontdoor network is not exactly stellaops_frontdoor, or the retained reachgraph-web target/migration flags are not exactly true/false respectively.
• reachgraph-web has lost the reachgraph alias while graph-api does not yet serve v1/reachgraphs.

Rollback

The overlay is not in the default stack, so before the window there is nothing to roll back in the running estate. For a scratch stack: docker compose -p grascratch -f devops/compose/docker-compose.graph.yml down -v (the -v drops scratch volumes; they are scratch by construction).

Rolling back the window’s compose join is a recreate of the two services without -f docker-compose.graph.yml, using each container’s own config_files list. The route swap rolls back separately using the verified timestamped copies prepared immediately before the swap, then config validation, release-bundle restaging and a gateway recreate — see the GRA-5 runbook. Do not use git checkout in the shared repository.

What GRA-9 and GRA-10 still owe after this stack is healthy

GRA-9 (the window):

1. Create stellaops_graph and its owner role on the existing control-plane cluster. The role must not be granted BYPASSRLS and must not be a superuser — a superuser bypasses RLS unconditionally, so any check run as one reports a posture it did not measure. Converge both startup migrations and prove all three CAS tables have relrowsecurity=true and relforcerowsecurity=true as the same table-owning runtime role.
2. Pre-live source complete: StellaOps.Graph.Persistence is wired behind Graph’s own connection gate and the lifted ReachGraph application is explicitly composed on graph-api, without a legacy ReachGraph startup migrator. The window must prove the newly built image and consolidated baseline on the composed stack; the deployed image remains old until recreated.
3. Copy graph (~496 kB) and reachgraph (~200 kB) with per-schema row-count parity.
4. Published-main commit 133c6a51f8 (the integrated equivalent of original lane commit 4c01e95fa0) already SOURCE-STAGES deletion of the GraphMigrationModulePlugin and ReachGraphMigrationModulePlugin registrations and their two foreign project references (DC-26, deferred here by D-GRA3-4). Only after step 3 parity is green, build/promote/recreate the affected Platform/central-migrator artifact and verify the candidate omits both plugins; retain the predecessor artifact for rollback. Source staging alone does not complete this step. 2026-09-14 (SPRINT_20260722_021 PLT-4, DC-26): the central-migrator mechanism itself is now deleted at source (IMigrationModulePlugin, discovery, registry, ReleaseMigrationRunner, stella system migrations-*); a platform-web image built from that commit carries no plugin at all, which satisfies the “candidate omits both plugins” check by construction.
5. Deploy the pre-live native /api/graph/v1 host surface and apply the route swap. The route leg is RETIRED, not pending (2026-09-04): gra5-swap-graph-routes.ps1 refuses on both invocations and exits 2, and GRA-9’s own status line carries the 2026-08-28 owner ruling that re-scopes the leg to a DELETION gated on a HELLO coverage proof. Do not run the swap here. Published-main commit 133c6a51f8 also SOURCE-STAGES only API_PATH_PREFIXES.graph at /api/graph/v1; run its wire specs against the candidate and build/promote/deploy Console in this same window. Do not flip graphReachability or graphReachabilityDrift: their findings/symbols/scans/drift contracts do not exist on the consolidated host and remain a separately recorded blocker.
6. Move the reachgraph alias onto graph-api only after step 5 is in place. (The second half of this step, repointing Scanner’s ReachGraphBaseUrl, lapsed 2026-09-14: SPRINT_20260914_001 VRP-5 retired the option with the publisher.)

Still NO-GO before the live window: Findings Ledger directly selects graph.asset_registry_events from the shared database. Moving Graph to stellaops_graph strands that reader. GRA-9 must not repoint or swap routes until the 023×010 owner event/API successor (or an explicitly approved fail-closed freeze) replaces that cross-database read. This runbook does not silently disable or rewrite the Findings path.

GRA-10 — DONE 2026-09-08, source half. The instruction below was executed as written: delete src/ReachGraph/, the reachgraph-web compose/publish/ownership keys, and then re-classify the entire ReachGraph implementation closure to owner graph: StellaOps.ReachGraph.Application, StellaOps.ReachGraph, StellaOps.ReachGraph.Cache and StellaOps.ReachGraph.Persistence. Update the surviving StellaOps.ReachGraph.Contracts producer classification in the same retirement change; doing any subset early leaves a false family boundary or a new reverse edge. StellaOps.Graph.Core + its self-only test project are already frozen under src/__Obsoleted/Graph/ and absent from live build/test routing (D-GRA4-5, 2026-08-24). Rename the lifted types out of the StellaOps.ReachGraph.WebService.* namespaces and the five misnamed InMemory* query services (D-GRA4-3); retire the two external TODOs naming IReachabilityDeltaService (ReleaseOrchestrator.EvidenceThread/Services/EvidenceNodeCollector.cs:28,292 and SbomService/Services/LineageCompareService.cs:327), pointing them at Scanner’s drift API.

Doctor registration — leave it OFF until the window

Doctor__Registration__Enabled defaults to false and must stay there until graph-api’s Authority client actually holds platform:doctor:register. Enabling it first produces a 403 loop against Platform; the default-off posture exists so an un-granted service degrades to local-only checks (docs/modules/doctor/adoption-guide.md §2). GRA-9 precondition 2 now places the remaining values directly in the staged overlay and declares graph-api-doctor in both Standard Authority descriptors. The client is confidential, first-party, tenant default, audience stellaops, client_credentials, and holds only platform:doctor:register. Authority and graph-api receive the same environment-backed secret; the token endpoint is explicitly /connect/token. This is source readiness only: registration remains OFF.

This provisioning gate completed 2026-08-22 while registration stayed OFF. The service-scoped, no-dependency Authority recreate changed only Authority and returned healthy on the same image; the client row had exactly the declared tenant/audience/grant/scope, and a fresh token was HTTP 200 with tenant default, audience stellaops, and platform:doctor:register. The secret and token were never printed. Graph’s container/image/routes/database and Platform capability row did not change. Re-run those exact proofs at the window boundary and abort on any extra scope/audience, failed token, non-Authority container recreation, or early capability row. Only a current graph-api image serving /doctor/graph-api/checks may open the later registration flip.

Three traps when the window flips it:

• Set the values, do not just declare them. The reference wiring guards Authority/ClientId/ClientSecret with ?? throw, which accepts an empty string, and compose passes unset variables through as ${VAR:-}. This host rejects blank as well as missing values and refuses to start, so a failed start immediately after the flip means one of the three is empty — not that the scope grant is missing.
• Authority rejects non-HTTPS token requests (ID2083): set the explicit Doctor__Registration__TokenEndpoint and make sure the CA bundle is mounted.
• Boot-time decisions cannot be injected through WebApplicationFactory’s ConfigureAppConfiguration — Program.cs reads the registration gate eagerly, before those sources are appended. Tests must drive it through UseSetting. This is the D-GRA4-1 trap in different clothes: it made an earlier version of the blank-credential test pass for the wrong reason.