Request and data flows (detailed)
Audience: engineers and operators debugging or auditing Stella Ops end to end — from a scan submission through verdict, attestation, and notification.
This document describes the canonical end-to-end flows at a level useful for debugging and auditing. Exact endpoints and payloads are defined by each module dossier under docs/modules/; for the layered service map see Platform topology, and for how evidence reaches the Policy Engine see the Policy Engine Data Pipeline.
Note on the ASCII diagrams below. The sequence diagrams use illustrative SQL statements and table names (e.g.
scan_jobs,sbom_documents) to convey intent. The verified live names differ — for example, the Scanner schema’s job/scan tables arejobsandscans, notscan_jobs(seesrc/Scanner/__Libraries/StellaOps.Scanner.Storage/Postgres/Migrations/). Treat the diagram SQL as pseudo-code; the prose steps and the inline call-outs are reconciled against source.
1) Scan execution (happy path)
- Client -> Gateway: submit scan request (authenticated; tenant-scoped).
- Gateway -> Scanner.WebService: route request after auth/rate-limit checks.
- Scanner.WebService -> PostgreSQL: persist scan manifest and initial status.
- Scanner.WebService -> queue/stream: enqueue a scan job (transport is profile/config dependent; for example Valkey streams or NATS).
- Scanner.Worker -> queue/stream: claim job, pull image, extract layers, run analyzers.
- Scanner.Worker -> RustFS/S3: write SBOM fragments, composed SBOMs, and other scan artifacts.
- Scanner.Worker -> Concelier: query linksets / observations needed for evaluation (deployment-dependent).
- Scanner.Worker -> Scanner.WebService: heartbeat and completion callbacks.
- Scanner.WebService -> Policy: request verdict evaluation using SBOM + advisory + VEX + policy inputs.
- Scanner.WebService -> Attestor (optional): create DSSE/in-toto evidence bundles and (optionally) attach transparency receipts. The Signer service (DSSE signing) and Provenance live inside the Attestor module — see
src/Attestor/StellaOps.Signer/andsrc/Attestor/StellaOps.Provenance.Attestation/. - Scanner.WebService -> events stream: publish completion events for notifications and downstream consumers.
- Notification engine -> channels: render and deliver notifications with idempotency tracking.
Offline note: for air-gapped deployments, step 6 writes to local object storage and step 7 relies on offline mirrors/bundles rather than public feeds. See docs/OFFLINE_KIT.md and docs/modules/airgap/guides/overview.md.
Scan execution sequence diagram
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 1. CLIENT REQUEST (CLI or Web UI) │
│ $ stella scan docker://alpine:latest --sbom-format=spdx │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ HTTPS
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 2. GATEWAY (API Router) │
│ - Terminates TLS │
│ - Routes to appropriate backend service │
│ - Load balancing (if multiple instances) │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ HTTP (internal)
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 3. AUTHORITY (Authentication) │
│ - Validates OAuth2 access token (DPoP-bound) │
│ - Checks DPoP proof against Valkey nonce cache │
│ - Returns user identity and scopes │
│ Scope claims are the JWT authorization surface; the canonical catalog is │
│ StellaOpsScopes.cs. Note: services map ASP.NET *policy names* to scope │
│ claims, and the policy name often differs from the scope string (e.g. the │
│ Scanner submit endpoint's policy is "scanner.api"). The Scanner WebService │
│ additionally uses its own scope strings (scanner.scans.enqueue/.read/.write) │
│ rather than the catalog's scanner:scan / scanner:read. │
│ │
│ ┌─────────────┐ │
│ │ Valkey │◀── DPoP nonce validation (GET/SET) │
│ │ (Cache) │ │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── User/client lookup (SELECT) │
│ └─────────────┘ │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ Authenticated request
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 4. SCANNER.WEB (Scan API Controller) │
│ - Validates scan request parameters │
│ - Creates scan job record in PostgreSQL │
│ - Enqueues scan job to Valkey queue (default) or NATS (if configured) │
│ │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── INSERT scans / jobs (id, image_ref, status) [illustr.] │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ Valkey │◀── XADD scanner:jobs (enqueue job; default stream key, │
│ │ (Queue) │ consumer group "scanner-workers"; transport=redis default) │
│ └─────────────┘ │
│ │
│ Returns: HTTP 202 Accepted { "scanId": "...", "status": "...", │
│ "location": "...", "created": true } (ScanSubmitResponse) │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│
│ (Client polls for status)
│
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 5. SCANNER.WORKER (Background Processor) │
│ - Consumes job from Valkey queue (XREADGROUP scanner:jobs) │
│ - Updates job status to 'running' │
│ - Downloads container image from registry │
│ - Executes analyzers (OS packages, language deps, files) │
│ - Generates SBOM (SPDX/CycloneDX) │
│ - Stores artifacts to RustFS │
│ │
│ ┌─────────────┐ (table names illustrative; verified: scans/jobs/artifacts) │
│ │ PostgreSQL │◀── UPDATE scans SET status='running' │
│ │ │◀── INSERT artifacts / images / layers (SBOM + component data) │
│ │ │◀── UPDATE scans SET status='completed' │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ RustFS │◀── PUT /artifacts/scan-abc123/sbom.spdx.json │
│ │ (S3 API) │◀── PUT /artifacts/scan-abc123/image-layers.tar.gz │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ Valkey │◀── XADD stella.events (kind=scanner.event.scan.completed) │
│ │(Event Stream│ (default stream key "stella.events"; events driver=redis) │
│ └─────────────┘ │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ Event published
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ 6. EVENT PROPAGATION (Valkey Streams) │
│ │
│ ┌───────────────────────────────────────────────────────────────────┐ │
│ │ Valkey Event Stream: "stella.events" (default) │ │
│ │ Event: { "kind": "scanner.event.scan.completed", "tenant": ... } │ │
│ └───────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌──────────────┼──────────────┬───────────────┐ │
│ ▼ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Notify │ │Timeline │ │ Policy │ │ Export │ │
│ │ Worker │ │ Indexer │ │ Engine │ │ Center │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ │ │ │ │ │
│ │ (consumers subscribe to stella.events via XREADGROUP; consumer set is │
│ │ deployment-dependent) │
└─────────┼───────────────┼──────────────┼───────────────┼─────────────────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────┐ ┌──────────────┐
│ 7a. NOTIFY │ │ 7b. TIMELINE │ │ 7c.POLICY│ │ 7d. EXPORT │
│ │ │ INDEXER │ │ ENGINE │ │ CENTER │
│ - Query scan │ │ │ │ │ │ │
│ results │ │ - Index event│ │ - Eval │ │ - Generate │
│ - Check user │ │ timeline │ │ policy │ │ SARIF │
│ notif prefs│ │ - Store in │ │ rules │ │ - Export to │
│ - Send Slack │ │ PostgreSQL │ │ - Block/ │ │ external │
│ message │ │ │ │ Allow │ │ systems │
│ │ │ │ │ │ │ │
│ PostgreSQL ◀─┤ │ PostgreSQL ◀─┤ │PostgreSQL│ │ RustFS ◀─┤
│ (user prefs) │ │ (timeline) │ │(policies)│ │ (exports) │
└──────────────┘ └──────────────┘ └──────────┘ └──────────────┘
2) Advisory ingestion (delta-driven)
- Concelier.Worker fetches advisories from configured sources (mirrors first; no hidden outbound calls in air-gap profiles).
- Concelier validates and normalizes advisories, producing canonical observations and linksets.
- Concelier -> PostgreSQL (
vuln) persists immutable raw documents (append-only patterns where required) plus derived linksets. - Concelier -> Scheduler notifies about deltas (new/updated advisories) via webhook/event.
- Scheduler schedules impacted re-scans or evaluations based on the delta.
3) VEX ingestion and consensus
- Excititor.Worker fetches VEX statements from configured sources (mirrors/bundles for offline).
- Excititor verifies signatures where required and normalizes statements into a canonical shape.
- Excititor -> PostgreSQL (
vex) persists immutable raw statements and consensus outcomes. - Excititor -> Scheduler / Policy emits deltas so verdicts can be recomputed deterministically.
4) Policy evaluation (decision trace)
The former Policy.Gateway has been merged into the Policy Engine (see Port Registry, slot 14); the steps below describe the single Policy Engine surface.
- Caller (Scanner/UI/CLI) -> Policy Engine submits an evaluation request.
- Policy Engine loads exception objects and policy snapshots from its own store.
- Policy Engine consumes advisory/VEX observations (by read model, replication, or API depending on deployment) and applies deterministic precedence/lattice rules (K4 lattice + ordered gates — see Policy Engine Data Pipeline).
- Policy Engine -> caller returns a verdict plus a trace/explain payload suitable for audits.
5) Notification delivery
- Notification engine consumes platform events (scan completed, advisory delta, etc.).
- Notification engine -> queue/stream enqueues delivery tasks with idempotency keys (when a worker model is used).
- Delivery workers -> channels deliver (email/chat/webhook), record results, and retry with deterministic backoff rules.
Notification flow diagram (vulnerability alert)
┌──────────────────────────────────────────────────────────────────────────────────┐
│ TRIGGER: New critical CVE detected in existing scan │
│ Source: Concelier advisory ingestion │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ CONCELIER.WORKER (Advisory Processor) │
│ │
│ 1. Ingest new advisory from NVD/OSV/CSAF │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── INSERT INTO vuln.advisories (...) [schema: vuln] │
│ └─────────────┘ │
│ │
│ 2. Match advisory against existing scans (PURL/CPE matching) │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── SELECT scans WHERE package_purl IN (affected_purls) │
│ └─────────────┘ │
│ │
│ 3. Publish drift/delta event to Valkey (illustrative stream name) │
│ ┌─────────────┐ │
│ │ Valkey │◀── XADD <drift stream> (new vulnerability found) │
│ └─────────────┘ │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│ Event published
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ NOTIFY.WORKER (Notification Processor) │
│ │
│ 1. Consume drift event from Valkey stream │
│ ┌─────────────┐ │
│ │ Valkey │◀── XREADGROUP <drift stream> <consumer group> │
│ └─────────────┘ │
│ │
│ 2. Evaluate notify rules / channels (notify.rules, notify.channels) │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── SELECT * FROM notify.rules / notify.channels │
│ │ │ (schema: notify; quiet hours, throttles also consulted) │
│ └─────────────┘ │
│ │
│ 3. Render notification template │
│ Template: "New critical CVE-2024-1234 affects alpine:latest scan" │
│ │
│ 4. Deliver notification via configured channels │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ External APIs │ │
│ │ - POST https://hooks.slack.com/services/T00/B00/xxx │ │
│ │ - POST https://graph.microsoft.com/v1.0/teams/channels │ │
│ │ - SMTP send (email) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ 5. Store delivery receipt in PostgreSQL │
│ ┌─────────────┐ │
│ │ PostgreSQL │◀── INSERT INTO notify.deliveries (status, ...) │
│ └─────────────┘ │
└──────────────────────────────────────────────────────────────────────────────────┘
6) Export flow (SBOM distribution)
┌──────────────────────────────────────────────────────────────────────────────────┐
│ EXPORT REQUEST: GET /api/v1/scans/{scanId}/exports/sbom │
│ (SBOM format selected via Accept content-negotiation: application/spdx+json, │
│ application/vnd.cyclonedx+json, etc. SARIF: .../exports/sarif) │
└───────────────────────────────────┬──────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ SCANNER.WEB or EXPORT CENTER │
│ │
│ 1. Query scan metadata from PostgreSQL │
│ ┌─────────────┐ (table names illustrative; verified: scans/jobs/artifacts) │
│ │ PostgreSQL │◀── SELECT * FROM scans WHERE id = $1 │
│ │ │◀── SELECT * FROM artifacts WHERE scan_id = $1 │
│ └─────────────┘ │
│ │
│ 2. Retrieve SBOM artifact from RustFS │
│ ┌─────────────┐ │
│ │ RustFS │◀── GET /artifacts/scan-abc123/sbom.spdx.json │
│ └─────────────┘ │
│ │
│ 3. Sign SBOM with Signer service (Signer lives inside the Attestor module) │
│ ┌─────────────┐ │
│ │ Signer │◀── POST /api/v1/signer/sign/dsse (SBOM payload) │
│ │ │──▶ Returns: DSSE envelope with signature │
│ └─────────────┘ │
│ │
│ 4. Create in-toto attestation with Attestor │
│ ┌─────────────┐ │
│ │ Attestor │◀── POST /api/v1/attestations (signed SBOM) │
│ │ │──▶ Returns: in-toto attestation bundle │
│ └─────────────┘ │
│ │
│ 5. Store final bundle to RustFS │
│ ┌─────────────┐ │
│ │ RustFS │◀── PUT /artifacts/scan-abc123/bundle.jsonl │
│ └─────────────┘ │
│ │
│ 6. Return signed bundle to client │
│ Returns: HTTP 200 OK (application/vnd.in-toto+json) │
└──────────────────────────────────────────────────────────────────────────────────┘
Related documentation
- Platform topology — layer-by-layer service map.
- Module Matrix — complete module inventory and consolidations.
- Policy Engine Data Pipeline — how evidence becomes a verdict.
- Data flows — datastore-level read/write paths.
- Port Registry — service ports, loopback IPs, and hostnames.
- Step-by-step flows: scan submission, SBOM generation, policy evaluation, notification, export.