StellaOps Architecture Overview
Ownership: Architecture Guild • Docs Guild Audience: Service owners, platform engineers, solution architects Related: High-Level Architecture, Concelier Architecture, Policy Engine Architecture, Aggregation-Only Contract
This dossier summarises the end-to-end runtime topology of the Stella Ops release control plane after the Aggregation-Only Contract (AOC) rollout. It highlights where raw facts live, how ingest services enforce guardrails, and how downstream components consume those facts to derive policy decisions and user-facing experiences. Read it as the starting map before diving into the module-specific dossiers linked from §6.
Orientation & further reading
- New here? Start with the Developer Quickstart — core repositories, determinism checks, DSSE conventions, and starter tasks.
- Writing production code? The Determinism analyzer contributor guide covers STELLA0110–STELLA0114, suppression grammar, and
IRandomProviderremediation. - Testing & CI lanes:
docs/technical/testing/testing-strategy-models.md(source catalog indocs/technical/testing/TEST_CATALOG.yml). - Tenant behaviour inventory for Platform endpoints:
docs/modules/platform/tenant-endpoint-classification.md.
1 · System landscape
graph TD
subgraph Edge["Clients & Automation"]
CLI[stella CLI]
UI[Console SPA]
APIClients[CI / API Clients]
end
Gateway[API Gateway
(JWT + DPoP scopes)]
subgraph Scanner["Fact Collection"]
ScannerWeb[Scanner.WebService]
ScannerWorkers[Scanner.Workers]
Agent[Agent Runtime]
end
subgraph Ingestion["Aggregation-Only Ingestion (AOC)"]
Concelier[Concelier.WebService]
Excititor[Excititor.WebService]
RawStore[(PostgreSQL
advisory_raw / vex_raw)]
end
subgraph Derivation["Policy & Overlay"]
Policy[Policy Engine]
Scheduler[Scheduler Services]
Notify[Notifier]
end
subgraph Experience["UX & Export"]
PlatformSvc[Platform Service
(Console Backend)]
Exporters[Export / Offline Kit]
end
Observability[Telemetry Stack]
CLI --> Gateway
UI --> Gateway
APIClients --> Gateway
Gateway --> ScannerWeb
ScannerWeb --> ScannerWorkers
ScannerWorkers --> Concelier
ScannerWorkers --> Excititor
Concelier --> RawStore
Excititor --> RawStore
RawStore --> Policy
Policy --> Scheduler
Policy --> Notify
Policy --> PlatformSvc
Scheduler --> PlatformSvc
PlatformSvc --> Exporters
Exporters --> CLI
Exporters --> Offline[Offline Kit]
Observability -.-> ScannerWeb
Observability -.-> Concelier
Observability -.-> Excititor
Observability -.-> Policy
Observability -.-> Scheduler
Observability -.-> Notify
Platform Service (StellaOps.Platform.WebService) aggregates cross-service status for the Console UI (health, quotas, onboarding, preferences, global search) and does not mutate raw evidence. Pack22 environment posture surfaces also consume Platform v2 read-model adapters, including /api/v2/evidence/packs, with comma-delimited region and environment scope sets preserved end to end so multi-select Console context is not collapsed by the backend.
Key boundaries:
- AOC border. Everything inside the Ingestion subgraph writes only immutable raw facts plus link hints. Derived severity, consensus, and risk remain outside the border.
- Policy-only derivation. Policy Engine materialises
effective_finding_*collections and emits overlays; other services consume but never mutate them. - Tenant enforcement. Authority-issued DPoP scopes flow through Gateway to every service; raw stores and overlays include
tenantstrictly. - Envelope-bound tenant identity (binding, 2026-04-30). Tenant identity is resolved exclusively from the envelope-attached
stellaops:tenantclaim viaStellaOps.Auth.ServerIntegration.Tenancy.IStellaOpsTenantAccessor. RawX-Tenant-Id/X-StellaOps-TenantId/X-StellaOps-TenantIdheaders are stripped at ingress and never honoured as a tenancy source by service code. Architectural conformance is enforced byTenantHeaderCallSiteConformanceTests(src/Authority/StellaOps.Authority/StellaOps.Auth.ServerIntegration.Tests/TenantHeaderCallSiteConformanceTests.cs); see audit finding A7 andSPRINT_20260430_007_MultiTenant_identity_sweepfor the rollout plan and the legacy-offender snapshot. - Router weighted-canary control. Router Gateway can split traffic across health-, region-, and ping-tied service instances by advertised or overridden routing weight. Instances advertise their default weight with
Router__Microservice__Weight; operators tune process-local overrides through the scopedGET/POST /api/v1/router/routing/weightsruntime API (router:routing:read/router:routing:write) and observe per-version request rate, errors, latency, and instance counts with the weighted canary deployment runbook and the Grafana dashboard. - Tenancy model. Platform tenant mechanics operate inside the single-operator posture defined by ADR-005: tenants are environments, business units, or logical scopes that share one operator trust boundary. They are not unrelated, mutually adversarial SaaS customers on one shared deployment.
- Platform-side tenant directory.
shared.tenantsis the platform database directory used by F11-style slug resolvers, tenant-scoped foreign keys, and the Platform-owned tenant residency default. Thedefault_region varchar(16) NOT NULL DEFAULT 'unspecified'column uses the same lowercase region-token shape EvidenceLocker accepts ([a-z0-9][a-z0-9_-]{0,15}) so downstream residency enforcement can resolve a tenant default without Router involvement. Authority owns tenant lifecycle writes and propagates create/update/suspend/resume into this table throughISharedTenantsPropagator; see Authority Tenant Model. Operators can rundevops/compose/scripts/reconcile-tenants.*once after upgrading older stacks to backfill any Authority tenant that predates the propagation hook. - Empty topology is tenant truth, not a fallback trigger. The Platform topology synchronizer enumerates every active
shared.tenantsrow, including tenants with no release environments, and records successful zero-row projection watermarks. Once those watermarks exist,/api/v2/topology/*returns a typed empty tenant-scoped result. It must not fall back to the globalplatform.context_*catalog, because that would surface another tenant’s environments. The pre-sync bundle-derived fallback remains available only while no successful inventory watermark exists. - Hybrid reachability attestations. Scanner/Attestor always publish graph-level DSSE for reachability graphs; optional edge-bundle DSSEs capture high-risk/runtime/init edges. Policy/Signals consume both, with graph DSSE as the minimum bar and edge bundles used for quarantine/dispute flows.
- Assurance packs. NIS2, CRA, and future compliance/standards support are optional Assurance packs over the same Policy, EvidenceLocker, ExportCenter, Notify, Graph, Authority, Web, and CLI primitives. Product architecture for this layer lives in Assurance Runtime. Do not introduce a strategic
src/Europemodule or duplicate pack-specific persistence when a shared primitive already owns the data. - Cryptography and compliance defaults. First-run setup captures installation-scoped regional crypto, FIPS mode, and NIS2/DORA/CRA evidence defaults in
platform.environment_settings; see Platform Cryptography and Compliance Defaults. Per-tenant compliance profile changes remain separate from those installation defaults, and nullable NIS2/DORA/CRA pack overrides onplatform.tenant_compliance_profileresolve over the installation defaults for the advanced/setup/system/cryptographysurface. - Uniform runtime plugin extensibility (fail-closed). Toolchain-agnostic integrations and per-module extensions are delivered as signed plugin bundles mounted at runtime, not baked into service images. The model is uniform across the suite: a module-owned loader hands a bundle’s assembly to .NET only after a fail-closed admission chain passes — manifest binding (
id== directory name, module, contract version, declared capability, well-formed assembly descriptor), per-DLL sha256 with a path-traversal guard, and a detached signature withAllowUnsigned = false. The cross-module chokepoint isStellaOps.Plugin.Security.SignedRuntimePluginAdmission(detached RSA-PKCS1-SHA256), called directly (or via a thin module facade) by AdvisoryAI, SmRemote (SM crypto), and the Router messaging backend; BinaryIndex, Doctor, and ExportCenter run the same chain and the same RSA/SHA-256 primitive through their own inline admission implementations (BinaryIndex adds a payload-digest binding), and Scanner’s language/OS analyzer hosts run a parallel Ed25519-signed path with the same “no signature ⇒ no load” rule. The olderPluginHost-based loaders (Concelier, Excititor, Notify, Integrations, Authority) reach the same posture viaEnforceSignatureVerification = trueplus a config-bound verifier (cosign/offline-dev-rsa-sha256). The authoritative module list is the-Moduleset ofdevops/build/package-runtime-plugins.ps1; the per-module loader-status table and admission details live in the Plugin development guide, with loader internals in Plugin architecture. Policy/VEX/task packs stay on the separate declarative (non-executable) path. Status: loaders and admission are in source; production cosign signing and the live flip to mounted mode are still pending, so the model is implemented-and-tested but not yet production-proven.
2 · Aggregation-Only Contract focus
2.1 Responsibilities at the boundary
| Area | Services | Responsibilities under AOC | Forbidden under AOC |
|---|---|---|---|
| Ingestion (Concelier / Excititor) | StellaOps.Concelier.WebService, StellaOps.Excititor.WebService | Fetch upstream advisories/VEX, verify signatures, compute linksets, append immutable documents to advisory_raw / vex_raw, emit observability signals, expose raw read APIs. | Computing severity, consensus, suppressions, or policy hints; merging upstream sources into a single derived record; mutating existing documents. |
| Policy & Overlay | StellaOps.Policy.Engine, Scheduler | Join SBOM inventory with raw advisories/VEX, evaluate policies, issue effective_finding_* overlays, drive remediation workflows. | Writing to raw collections; bypassing guard scopes; running without recorded provenance. |
| Experience layers | Console, CLI, Exporters | Surface raw facts + policy overlays; run stella aoc verify; render AOC dashboards and reports. | Accepting ingestion payloads that lack provenance or violate guard results. |
2.2 Raw stores
| Collection | Purpose | Key fields | Notes |
|---|---|---|---|
advisory_raw | Immutable vendor/ecosystem advisory documents. | _id, tenant, source.*, upstream.*, content.raw, linkset, supersedes. | Idempotent by (source.vendor, upstream.upstream_id, upstream.content_hash). |
vex_raw | Immutable vendor VEX statements. | Mirrors advisory_raw; identifiers.statements summarises affected components. | Maintains supersedes chain identical to advisory flow. |
Logical replication (advisory_raw_stream, vex_raw_stream) | Feed Policy Engine and Scheduler. | operationType, documentKey, fullDocument, tenant, traceId. | Scope filtered per tenant before delivery. |
2.3 Guarded ingestion sequence
sequenceDiagram
participant Upstream as Upstream Source
participant Connector as Concelier/Excititor Connector
participant Guard as AOCWriteGuard
participant PG as PostgreSQL (advisory_raw / vex_raw)
participant Stream as Logical Replication
participant Policy as Policy Engine
Upstream-->>Connector: CSAF / OSV / VEX document
Connector->>Connector: Normalize transport, compute content_hash
Connector->>Guard: Candidate raw doc (source + upstream + content + linkset)
Guard-->>Connector: ERR_AOC_00x on violation
Guard->>PG: Append immutable row (with tenant & supersedes)
PG-->>Stream: Replication event (tenant scoped)
Stream->>Policy: Raw delta payload
Policy->>Policy: Evaluate policies, compute effective findings
2.4 Authority scopes & tenancy
| Scope | Holder | Purpose | Notes |
|---|---|---|---|
advisory:ingest / vex:ingest | Concelier / Excititor collectors | Append raw documents through ingestion endpoints. | Paired with tenant claims; requests without tenant are rejected. |
advisory:read / vex:read | DevOps verify identity, CLI | Run stella aoc verify or call /aoc/verify. | Read-only; cannot mutate raw docs. |
effective:write | Policy Engine | Materialise effective_finding_* overlays. | Only Policy Engine identity may hold; ingestion contexts receive ERR_AOC_006 if they attempt. |
findings:read | Console, CLI, exports | Consume derived findings. | Enforced by Gateway and downstream services. |
3 · Data & control flow highlights
- Ingestion: Concelier / Excititor connectors fetch upstream documents, compute linksets, and hand payloads to
AOCWriteGuard. Guards validate schema, provenance, forbidden fields, supersedes pointers, and append-only rules before writing to PostgreSQL. - Verification:
stella aoc verify(CLI/CI) and/aoc/verifyendpoints replay guard checks against stored documents, mappingERR_AOC_00xcodes to exit codes for automation. - Policy evaluation: PostgreSQL logical replication delivers tenant-scoped raw deltas. Policy Engine joins SBOM inventory (via BOM Index), executes deterministic policies, writes overlays, and emits events to Scheduler/Notify.
- Experience surfaces: Console renders an AOC dashboard showing ingestion latency, guard violations, and supersedes depth. CLI exposes raw-document fetch helpers for auditing. Offline Kit bundles raw collections alongside guard configs to keep air-gapped installs verifiable.
- Observability: All services emit
ingestion_write_total,aoc_violation_total{code},ingestion_latency_seconds, and trace spansingest.fetch,ingest.transform,ingest.write,aoc.guard. Logs correlate viatraceId,tenant,source.vendor, andcontent_hash.
3.1 Scanner events lane (Sprint 20260530.022)
Decision: Option (ii). SCANNER__EVENTS__ENABLED is the optional scanner-to-Platform analytics lane, not the writer path for the dossier tables (scanner.scans, scanner.scan_findings, concelier.sbom_documents, findings.findings_projection_default, or release_orchestrator.releases). Those tables are populated by their synchronous owning services.
When enabled, Scanner WebService publishes report-ready / scan-completed envelopes to orchestrator:events, the stream consumed by Platform AnalyticsIngestionService. The scanner default, compose fallback, and shipped env samples use orchestrator:events so enabling the flag does not write to an unread stella.events stream. Operators may still override SCANNER_EVENTS_STREAM, but doing so must be paired with a matching Platform analytics Streams.ScannerStream override.
4 · Offline & disaster readiness
- Offline Kit: Packages raw PostgreSQL snapshots (
advisory_raw,vex_raw) plus guard configuration and CLI verifier binaries so air-gapped sites can re-run AOC checks before promotion. - Recovery: Supersedes chains allow rollback to prior revisions without mutating rows. Disaster exercises must rehearse restoring from snapshot, replaying logical replication into Policy Engine, and re-validating guard compliance.
- Migration: Legacy normalised fields are moved to temporary views during cutover; ingestion runtime removes writes once guard-enforced path is live (see Migration playbook). Platform database migrations are forward-only by policy; services apply embedded SQL at startup and rollback is a PostgreSQL snapshot restore, not a reverse migration. See ADR-004 and the Migration Recovery Runbook.
5 · Replay CAS & deterministic bundles
- Replay CAS: Content-addressed storage lives under
cas://replay/<sha256-prefix>/<digest>.tar.zst. Writers must use StellaOps.Replay.Core helpers to ensure lexicographic file ordering, POSIX mode normalisation (0644/0755), LF newlines, zstd level 19 compression, and shard-by-prefix CAS URIs (BuildCasUri). Bundle metadata (size, hash, created) feeds the platform-widereplay_bundlescollection defined indocs/db/replay-schema.md. - Artifacts: Each recorded scan stores three bundles:
manifest.json(canonical JSON, hashed and signed via DSSE).inputbundle.tar.zst(feeds, policies, tools, environment snapshot).outputbundle.tar.zst(SBOM, findings, VEX, logs, Merkle proofs). Every artifact is signed with multi-profile keys (FIPS, GOST, SM, etc.) managed by Authority. Seedocs/modules/replay/guides/DETERMINISTIC_REPLAY.md§2–§5 for the full schema.
- Reachability subtree: When reachability recording is enabled, Scanner uploads graphs & runtime traces under
cas://replay/<scan-id>/reachability/graphs/andcas://replay/<scan-id>/reachability/traces/. Manifest references (StellaOps.Replay.Core) bind these URIs along with analyzer hashes so Replay + Signals can rehydrate explainability evidence deterministically. - Storage tiers: Primary storage is PostgreSQL (
replay_runs,replay_subjects) plus the CAS bucket. Evidence Locker mirrors bundles for long-term retention and legal hold workflows (docs/modules/evidence-locker/architecture.md). Offline kits package bundles underoffline/replay/<scan-id>with detached DSSE envelopes for air-gapped verification. - APIs & ownership: Scanner WebService produces the bundles via
recordmode, Scanner Worker emits Merkle metadata, Signer/Authority provide DSSE signatures, Attestor anchors manifests to Rekor, CLI/Evidence Locker handle retrieval, and Docs Guild maintains runbooks. Responsibilities are tracked indocs/implplan/SPRINT_185_shared_replay_primitives.mdthroughSPRINT_187_evidence_locker_cli_integration.md. - Operational policies: Retention defaults to 180 days for hot CAS storage and 2 years for cold Evidence Locker copies. Rotation and pruning follow the checklist in
docs/operations/runbooks/replay_ops.md.
6 · References
- Aggregation-Only Contract reference
- Concelier architecture
- Excititor architecture
- Policy Engine architecture
- Authority service
- Replay specification
- Replay developer guide
- Replay schema —
docs/db/replay-schema.md - Replay test strategy (draft)
- Observability standards (upcoming) – interim reference for telemetry naming.
7 · Compliance checklist
- [ ] AOC guard enabled for all Concelier and Excititor write paths in production.
- [ ] PostgreSQL schema constraints deployed for
advisory_rawandvex_raw; logical replication scoped per tenant. - [ ] Authority scopes (
advisory:*,vex:*,effective:*) configured in Gateway and validated via integration tests. - [ ]
stella aoc verifywired into CI/CD pipelines with seeded violation fixtures. - [ ] Console AOC dashboard and CLI documentation reference the new ingestion contract.
- [ ] Offline Kit bundles include guard configs, verifier tooling, and documentation updates.
- [ ] Observability dashboards include violation, latency, and supersedes depth metrics with alert thresholds.
- [ ] Test Catalog updated for new modules and gates.
- [ ] CI lanes mapped to the Test Catalog and enforced with deterministic filters.
Last updated: 2026-01-05.
