Container Deployment Guide — Aggregation-Only Contract (AOC)

Audience: DevOps Guild and platform operators deploying Stella Ops services. Purpose: Cover the AOC-specific deployment considerations for Concelier and Excititor: the always-on write guard and its startup migrations, the pre-flight duplicate audit, AOC metrics, and verifier identities for the HTTP /aoc/verify endpoint. (Earlier drafts described “guard environment flags” — those do not exist; see § 2.)

This guide supplements existing deployment manuals with AOC-specific configuration. It assumes familiarity with the supported Compose and Offline Kit manifests described in docs/operations/devops/architecture.md. Kubernetes and Helm are not supported deployment targets.

Reconciliation note (verified against src/ on 2026-05-30). The AOC guard, its metrics, and the /aoc/verify surface are all implemented and live; however, several operational details earlier drafts described — a separately-applied CHECK-constraint script, a family of AOC_* environment toggles, an advisory → advisory_backup_* view swap, and a dedicated stella-aoc-verify OAuth client wired to the CLI — are not present in the codebase. Each such claim is annotated inline below. Authoritative behaviour: the guard ships always-on inside Concelier/Excititor (no enablement flag), AOC field/idempotency enforcement is part of the service’s auto-applied startup migrations (AGENTS.md §2.7), and the stella aoc verify CLI reads PostgreSQL directly (it does not authenticate through Authority).

1 · Schema constraint enablement

1.1 PostgreSQL constraints

CORRECTED — no separate constraint script exists. There is no apply-aoc-constraints.sql anywhere in the repository, and AOC enforcement is not applied as standalone DB CHECK constraints. Per the database auto-migration requirement (AGENTS.md §2.7), each service converges its own schema on startup from embedded SQL. The raw-advisory table and its idempotency guarantee ship in migration src/Concelier/__Libraries/StellaOps.Concelier.Persistence/Migrations/030_add_advisory_raw_store.sql, which creates vuln.advisory_raw with a unique dedupe index uq_advisory_raw_content ON vuln.advisory_raw(tenant_id, source_vendor, upstream_id, content_hash). The VEX raw store and its lineage live under src/Concelier/__Libraries/StellaOps.Excititor.Persistence/Migrations/ (e.g. 011_excititor_raw_lineage.sql, 013_vex_raw_source_revisions.sql). Operators do not run a manual constraint step — deploying the current Concelier/Excititor images is sufficient.

1.2 Migration order

  1. Run the duplicate audit (§ 1.1) in a maintenance window and resolve any reported groups.

  2. Roll out the current Concelier/Excititor images. The startup migrations create/converge the raw tables and dedupe index, and the AOC write guard is registered unconditionally — AddConcelierAocGuards() / AddExcititorAocGuards() run on every boot (see src/Concelier/StellaOps.Concelier.WebService/Program.cs:556 and src/Concelier/StellaOps.Excititor.WebService/Program.cs:110). There is no AOC_GUARD_ENABLED flag to set — the guard cannot be toggled off via configuration.

  3. Run a pre-flight check (stella aoc verify --dry-run validates configuration without touching the database) before resuming production ingestion.

    NOT IMPLEMENTED — stella sources ingest --dry-run. No such command/fixture was found; use the AOC verify dry-run for a pre-flight check instead.

1.3 Supersedes backfill verification

  1. Duplicate audit: Confirm psql -d concelier -v LIMIT=200 -f devops/tools/ops-scripts/check-advisory-raw-duplicates.sql reports no conflicts before restarting Concelier with the new migrations.

  2. Lineage spot-check: Inspect a sample set to confirm append-only chaining. The real table is vuln.advisory_raw; the supersede pointer column is supersedes_id and the identity columns are tenant_id / source_vendor / upstream_id:

    SELECT id, supersedes_id
FROM vuln.advisory_raw
WHERE upstream_id IS NOT NULL
ORDER BY tenant_id, source_vendor, upstream_id, retrieved_at
LIMIT 5;

    Each revision should reference the previous id via supersedes_id (or NULL for the first revision). Record findings in the change ticket before proceeding to production.

    REMOVED — advisory → advisory_backup_20251028 view check. Earlier drafts told operators to validate that an advisory view points to a table named advisory_backup_20251028. Neither the view nor the backup table exists anywhere in src/; the check was orphaned and has been deleted. Column names were also corrected: the table is vuln.advisory_raw (schema-qualified), the supersede column is supersedes_id (not supersedes), and the tenant column is tenant_id (not tenant).

2 · Container environment flags

NOT IMPLEMENTED — the AOC_* environment variables below do not exist. A repo-wide search (src/, devops/) finds no reference to AOC_GUARD_ENABLED, AOC_ALLOW_SUPERSEDES_RETROFIT, AOC_METRICS_ENABLED, AOC_TENANT_HEADER, or AOC_VERIFIER_USER outside this document. Setting them has no effect. The table is retained below for historical reference only, annotated with the actual mechanism.

Variable (NOT WIRED)Doc defaultActual mechanism in src/
AOC_GUARD_ENABLEDtrueNo toggle. The guard is registered unconditionally via AddConcelierAocGuards() / AddExcititorAocGuards() and always intercepts raw writes. There is no supported way to disable it through configuration.
AOC_ALLOW_SUPERSEDES_RETROFITfalseNo such flag. Supersede lineage is recorded via the supersedes_id column on vuln.advisory_raw; there is no retrofit toggle.
AOC_METRICS_ENABLEDtrueNo toggle. Metrics ingestion_write_total and aoc_violation_total are always emitted (src/__Libraries/StellaOps.Ingestion.Telemetry/IngestionTelemetry.Metrics.cs; Concelier Diagnostics/IngestionMetrics.cs).
AOC_TENANT_HEADERX-StellaOps-TenantIdThe header value X-StellaOps-TenantId is correct, but it is a hardcoded constant (TenantHeaderName in src/Concelier/StellaOps.Excititor.WebService/Program.Helpers.cs:17), not configurable via env var. (A legacy X-Stella-Tenant constant also exists.)
AOC_VERIFIER_USERstella-aoc-verifyNot read anywhere. The HTTP verify endpoint authorizes by scope (advisory:read + aoc:verify), not by a named user env var; the CLI verify path uses no Authority identity at all (see § 3).

Operator takeaway: no AOC-specific environment variables need to be added to Concelier/Excititor. Deploying the current images enables the always-on guard and metrics. Multi-tenant routing already supplies X-StellaOps-TenantId from the Gateway.

3 · Verifier identity

PARTIALLY IMPLEMENTED — verify which surface you are securing. The two AOC verification paths differ in their auth model:

  • HTTP /aoc/verify(Concelier Program.cs:3173; Excititor has an equivalent) does enforce scopes. Its policy Concelier.Aoc.Verify requires both advisory:read and aoc:verify (Program.cs:1138). All three scopes referenced below — aoc:verify, advisory:read, vex:read — are real, canonical scopes (src/Authority/StellaOps.Authority/StellaOps.Auth.Abstractions/StellaOpsScopes.cs). A dedicated read-only client for Console / API consumers of this endpoint is therefore a sound design.
  • CLI stella aoc verify(src/Cli/__Libraries/StellaOps.Cli.Plugins.Aoc/AocCliCommandModule.cs) does not authenticate through Authority at all. It connects directly to PostgreSQL (the --postgres / -p connection string is a required option) and reads vuln.advisory_raw itself. So the claim that the stella-aoc-verify client is “bound to stella aoc verify CI jobs” is incorrect — CI jobs running the CLI need a Postgres connection string, not an OAuth client.

No client named stella-aoc-verify is seeded anywhere in src/ or devops/ today; the block below is a recommended (draft) Authority bootstrap entry for the HTTP endpoint, not a description of existing configuration.

clients:
  - clientId: stella-aoc-verify
    grantTypes: [client_credentials]
    # advisory:read + aoc:verify are both required by the Concelier.Aoc.Verify policy;
    # add vex:read for the Excititor verify endpoint.
    scopes: [aoc:verify, advisory:read, vex:read]
    tenants: [default]

4 · Deployment steps

  1. Pre-checks: Confirm database backups, alerting in maintenance mode, and staging environment validated.

  2. Audit duplicates: Run the duplicate-audit helper (there is no separate “validator” script to apply — schema/idempotency converge via the service’s startup migrations) per § 1.1.

  3. Update manifests: No AOC-specific environment variables are required (see Section 2). Confirm the Gateway forwards X-StellaOps-TenantId to Concelier/Excititor.

  4. Redeploy services: Restart Concelier/Excititor through Docker Compose or the host service manager. The always-on AOC guard and metrics take effect on boot. Monitor ingestion_write_total for steady throughput.

  5. Seed verifier (HTTP endpoint only): If you secure /aoc/verify for Console/API consumers, deploy the read-only client from Section 3 and store credentials. (Not needed for the CLI path.)

  6. Run verification: Execute stella aoc verify --since <ISO-8601 timestamp> --postgres <conn-string> and ensure exit code 0.

    CORRECTED — --since and --postgres. --since requires an ISO-8601 timestamp (e.g. 2026-05-29T00:00:00Z); relative durations such as 24h are rejected by the parser (AocVerifyOptionsParser.TryParseSince). The --postgres connection string is a required option — the CLI queries PostgreSQL directly and does not call the HTTP endpoint. (The HTTP /aoc/verify endpoint, by contrast, accepts a since timestamp in the request body and defaults to a 24-hour window when omitted.)

  7. Update dashboards: Point Grafana panels to the metrics (ingestion_write_total, aoc_violation_total).

  8. Record handoff: Capture console screenshots and verification logs for release notes.

5 · Offline Kit updates

NOT IMPLEMENTED — offline-kit/scripts/apply-aoc-validators.js. No such file exists in the repository, and there is no separate validator-script step for AOC (enforcement is in-process + auto-migrated, per § 1.1). Do not ship or reference this script.

6 · Rollback plan

CORRECTED — the AOC guard cannot be disabled by configuration. There is no AOC_GUARD_ENABLED flag and no --remove validator script (see § 1.1 and § 2). The guard is always-on for the deployed image; “rollback” means redeploying a prior image build that predates the guard, not flipping an env var. Forward-only migrations apply (ADR-004) — DB-level rollback is recovered by PostgreSQL snapshot restore, not by dropping constraints.

  1. Roll back the image: Redeploy the previously-pinned (digest-addressed) Concelier/Excititor image if the guard behaviour must be reverted. There is no in-place disable.
  2. Restore schema if needed: Recover the database from a PostgreSQL snapshot per the migration-recovery runbook; migrations are forward-only (ADR-004).
  3. Pause verification jobs to prevent noise.
  4. Investigate and remediate upstream issues before redeploying the guard-enabled image.

7 · References

CORRECTED LINKS — relative paths were broken. This file lives at docs/operations/deployment/, so each ../ reaches docs/operations/. The prior links (../aoc/…, ../security/…, ../observability/…, ../modules/…) resolved to non-existent paths. Targets and depth are fixed below (verified to exist on disk 2026-05-31).

8 · Compliance checklist

Last updated: 2026-05-30 — deep doc/code reconciliation against src/ (Concelier/Excititor AOC guard, StellaOps.Aoc, scope catalog, CLI AOC plugin). Original draft: 2025-10-26, Sprint 19).