Data Schemas & Persistence Contracts

This reference describes the persistence contracts behind Stella Ops: the Valkey keyspace, the PostgreSQL tables, and the on-disk blob shapes that the scanner, policy, scheduler, and notify subsystems read and write.

Audience — backend developers, plug-in authors, and database administrators.

Altitude note. This document captures the shape, contracts, and invariants of each schema. Exhaustive field/enum/key lists live in source and are the authority — when this doc and the code disagree, the code wins. Each section points at the canonical model, JSON Schema, or sample fixture.

0 Document Conventions

• CamelCase for JSON.
• All timestamps are RFC 3339 / ISO 8601 with Z (UTC).
• ⭑ = planned but not shipped yet (kept on Feature Matrix “To Do”).

1 SBOM Wrapper Envelope

Every SBOM blob (regardless of format) is stored on disk or in object storage with a sidecar JSON file that indexes it for the scanners.

1.1 JSON Shape

A wrapper records the digest of the SBOM file (id), the source imageDigest, created, the SBOM format (trivy-json-v2 | spdx-json | cyclonedx-json), the ordered layers[], a partial flag, and a planned provenanceId link to an SLSA attestation (⭑).

{
  "id": "sha256:417f…",          // digest of the SBOM file itself
  "imageDigest": "sha256:e2b9…", // digest of the original container image
  "format": "trivy-json-v2",     // trivy-json-v2 | spdx-json | cyclonedx-json
  "layers": ["sha256:d38b…"],    // ordered layer digests
  "partial": false               // true => delta SBOM (only some layers)
}

formatsupports multiple SBOM formats; partialis true when generated via the delta SBOM flow (§1.3).

1.2 File‑system Layout

blobs/
 └─ 417f…/             # digest prefix
     ├─ sbom.json      # payload (any format)
     └─ sbom.meta.json # wrapper (shape above)

Note – RustFS is the primary object store; S3/MinIO compatibility layer available for legacy deployments; driver plug‑ins support multiple backends.

1.3 Delta SBOM Extension

When partial: true, only the missing layers have been scanned. Merging logic inside the Scanner stitches new data onto the cached full SBOM in Valkey, keyed off layers:* (§2).

2 Valkey Keyspace

Valkey (Redis-compatible) provides cache, DPoP nonces, event streams, and queues for real-time messaging and rate limiting.

Delta SBOM uses layers:* to skip work in <20 ms. Quota enforcement increments quota:<token> atomically; once the counter reaches the {{quota_token}}-scan ceiling the API returns 429. DPoP & Events: Valkey Streams support high-throughput, ordered event delivery for re-evaluation and notification triggers. Alternative: NATS JetStream can replace Valkey for queues (opt-in only; requires explicit configuration).

3 PostgreSQL Tables

PostgreSQL is the canonical persistent store for long-term audit and history.

policy_versions stores the YAML ruleset plus an optional rego field (filled only when Rego is uploaded), the authorId, created, and an import comment.

3.1 Scheduler Artifacts

Tables. schedules, runs, impact_snapshots, audit, run_summaries (module-local). All rows use the canonical JSON emitted by StellaOps.Scheduler.Models so agents and fixtures remain deterministic.

Canonical source — models: src/JobEngine/StellaOps.Scheduler.__Libraries/StellaOps.Scheduler.Models/ (Schedule.cs, Run.cs, ImpactSet.cs, AuditRecord.cs, RunSummary.cs, CanonicalJsonSerializer.cs, Validation.cs). Canonical source — fixtures: samples/api/scheduler/ (schedule.json, run.json, impact-set.json, audit.json, run-summary.json). Use these for the exhaustive field-by-field shape.

The shapes and their load-bearing invariants:

• Schedule (schedules) — id, tenantId, name, enabled, cronExpression, timezone, mode, a selection block (scope / namespaces / repositories / tags / labels), optional onlyIf and notify gates, limits, subscribers, and audit timestamps. Invariants: arrays are alphabetically sorted; selection.tenantId is optional but when present must match tenantId; cron expressions are validated for newline/length; timezones via TimeZoneInfo.

• Run (runs) — scheduleId, trigger, state, a stats counter block, an optional reason, lifecycle timestamps, error, and a deltas[] array of per-image findings (counts, KEV hits, top findings, attestation pointer). Invariants: counters clamped to ≥0; timestamps coerced to UTC; delta arrays sorted by severity precedence (critical → info) then vulnerability id; missing deltas ⇒ “no change”.

• Impact Snapshot (impact_snapshots) — a selector, an images[] set (digest / registry / repository / namespaces / tags / usage / labels), usageOnly, total, and snapshotId. Invariants: images deduplicated and sorted by digest; label keys lowercased to avoid case-sensitive duplicates; snapshotId lets run planners diff snapshots for drift.

• Audit (audit) — category, action, occurredAt, an actor, optional scheduleId/runId, correlationId, a metadata map, and a message. Invariants: metadata keys lowercased, first-writer wins (casing duplicates ignored); empty optional IDs trimmed; emit via the canonical serializer so audit digests stay reproducible.

• Run Summary (run_summaries) — materialized roll-up powering Scheduler UI dashboards (last-run banners, sparklines) without scanning runs. Holds lastRun, a recent[] window, and aggregate counters. Invariants: _id = tenant:schedule; recent keeps the 20 newest runs by createdAt (UTC), replacing the entry per run to honour state transitions; counters aggregate over that 20-run window and are recomputed on every update; the projection service must be called after each run state change. Sample: samples/api/scheduler/run-summary.json.

4 Policy Schema (YAML v1.0)

Minimal viable grammar (subset of OSV‑SCHEMA ideas) — a version plus a list of rules, each with a name, severity[], an action (block | ignore | escalate), and optional environments, sources, and expires.

version: "1.0"
rules:
  - name: Block Critical
    severity: [Critical]
    action: block
  - name: Ignore Low Dev
    severity: [Low, None]
    environments: [dev, staging]
    action: ignore
    expires: "2026-01-01"

Canonical schema: src/Policy/__Libraries/StellaOps.Policy/Schemas/policy-schema@1.json (embedded into StellaOps.Policy). Reusable validator: src/Policy/__Libraries/StellaOps.Policy/PolicyValidationCli.cs (PolicyValidationCli.RunAsync(PolicyValidationCliOptions{ Inputs, Strict })) — the command handler the main CLI wires up.

4.1 Rego Variant (Advanced ⭑)

Accepted but stored as-is in the rego field. Planned for evaluation via an internal OPA side-car once the feature ships.

4.2 Policy Scoring Config (JSON)

Configures how findings are scored: version, a severityWeights map, action penalties (quietPenalty / warnPenalty / ignorePenalty), trustOverrides, reachabilityBuckets (numeric multipliers, default ≤1.0), and an unknownConfidence block (initial / decayPerDay / floor plus ordered confidence bands).

Schema id: https://schemas.stella-ops.org/policy/policy-scoring-schema@1.json Source: src/Policy/__Libraries/StellaOps.Policy/Schemas/policy-scoring-schema@1.json; default fixture …/Schemas/policy-scoring-default.json (both embedded in StellaOps.Policy). The exhaustive default weights/keys live there.

Validation runs alongside policy binding (PolicyScoringConfigBinder), producing deterministic digests via PolicyScoringConfigDigest. Bands are ordered descending by min so consumers resolve confidence tiers deterministically. Reachability bucket keys are case-insensitive.

Runtime usage (contract)

• trustOverrides match finding.tags (trust:<key>) first, then finding.source/finding.vendor; missing keys default to 1.0.
• reachabilityBuckets consume finding.tags prefixed reachability: (fallback usage: or unknown). Missing buckets fall back to the unknown weight when present, else 1.0.
• Policy verdicts expose scoring inputs (severityWeight, trustWeight, reachabilityWeight, baseScore, penalties) plus unknown-state metadata (unknownConfidence, unknownAgeDays, confidenceBand) for auditability. Reference payloads: samples/policy/policy-preview-unknown.json, samples/policy/policy-report-unknown.json.
• Unknown confidence derives from unknown-age-days: (preferred) or unknown-since: + observed-at: tags; with no hints the engine keeps initial confidence, decaying by decayPerDay down to floor, then resolving to the first matching band.

Validate the sample payloads with Ajv (--spec=draft2020 -c ajv-formats) against their published schemas before changing them; see the Docs CI validation loop in §6.

5 SLSA Attestation Schema ⭑

Planned for Q1‑2026 (kept here for early plug‑in authors). A provenance doc carries id, imageDigest, buildType, a builder, build metadata (invocation parameters + timestamps + completeness), materials[], and a rekorLogIndex into the local Rekor mirror — following the SLSA provenance v1 layout.

6 Notify Foundations (Rule · Channel · Template · Event)

Canonically describes the Notify data shapes that UI, workers, and storage consume. JSON Schemas live under docs/modules/notify/resources/schemas/; deterministic fixtures under docs/modules/notify/resources/samples/.

Cross-cutting invariants (all four artifacts):

• Keys are lower-cased camelCase; each document pins a schemaVersion (notify.rule@1, notify.channel@1, notify.template@1). Use StellaOps.Notify.Models.NotifySchemaMigration.Upgrade*(JsonNode) (src/Notify/__Libraries/StellaOps.Notify.Models/NotifySchemaMigration.cs) to backfill the version when deserialising legacy payloads.
• Array selectors are pre-sorted and case-normalised for deterministic serialisation.
• Rules, channels, and templates carry a soft-delete marker (deletedAt) persisted in PostgreSQL; repository/API queries filter them by default while retaining revision history.

Per-artifact notes (shape is in the schema; these are the contracts that aren’t):

• Rule — mandatory match + actions; actions[].throttle serialises as an ISO 8601 duration (PT5M) mirroring worker backoff; optional vex gates exclude accepted/not-affected justifications.
• Channel — type matches plug-in identifiers (slack/teams/email/webhook/custom); config.secretRef stores an external secret handle (Authority/Vault/K8s) — Notify never persists raw credentials; config.limits.timeout uses ISO 8601 durations.
• Template — presentation key (channelType/key/locale) plus raw body; renderMode enumerates engines (aligns with NotifyTemplateRenderMode); format signals the downstream connector.
• Event envelope — platform event contract: eventId UUID, RFC 3339 ts, tenant isolation; enumerated kind (e.g. scanner.report.ready, scheduler.rescan.delta, zastava.admission); scope.labels/scope.attributes plus top-level attributes mirror the worker templating/audit metadata. Workers wrap payloads via the same migration helper so schema additions stay additive.

Validation loop (matches Docs CI): compile every …/schemas/*.json with npx ajv compile -c ajv-formats, then validate each *.sample.json against its sibling schema with npx ajv validate. Integration tests embed the sample fixtures to guarantee deterministic serialisation from the StellaOps.Notify.Models DTOs.

7 Validator Contracts

• SBOM wrapper – ISbomValidator (DLL plug‑in) must return a typed error list.
• YAML policies – JSON‑Schema (§4, policy-schema@1.json).
• Rego – OPA opa eval --fail-defined under the hood.
• Free‑tier quotas – IQuotaService integration tests ensure quota:<token> resets at UTC midnight and produces correct Retry‑After headers.

8 Migration Notes

1. Add format column to existing SBOM wrappers; default to trivy-json-v2.
2. Populate layers & partialvia backfill script (ships with stellopsctl migrate wizard).
3. Policy YAML previously stored in Valkey → copy to PostgreSQL if persistence enabled.
4. Prepare attestations table (empty) – safe to create in advance.

9 Open Questions / Future Work

• How to de‑duplicate identical Rego policies differing only in whitespace?
• Embed GOST 34.11‑2018 digests when users enable Russian crypto suite?
• Should enterprise tiers share the same Valkey quota keys or switch to a JWT claim tier != Free bypass?
• Evaluate sliding‑window quota instead of strict daily reset.
• Consider rate‑limit for /layers/missing to avoid brute‑force enumeration.

10 Change Log