Unknowns Architecture

Scope. Architecture for the Unknowns microservice: a structured registry that tracks unresolved components, symbols, and mappings that Scanner and other analyzers cannot definitively identify. This dossier is for implementers and operators of the Unknowns service and for analyzer authors that feed it.

Source-reconciliation note (2026-05-30). Reconciled directly against src/Unknowns/ and the canonical scope catalog src/Authority/StellaOps.Authority/StellaOps.Auth.Abstractions/StellaOpsScopes.cs. The Unknown registry (/api/unknowns/*) is fully wired and persisted. The Grey Queue surface (/api/grey-queue/*) and the SLA / metrics / lifecycle background services are scaffolded in source but NOT activated at runtime — see §6 Implementation status for the exact gaps.

0) Mission & boundaries

Mission. Provide a structured registry for tracking unknown components, unresolved symbols, and incomplete mappings. Enable visibility into coverage gaps and guide future enhancement priorities.

Boundaries.

• Unknowns is a standalone Minimal-API microservice (StellaOps.Unknowns.WebService) with its own HTTP API surface and schema ownership (unknowns PostgreSQL schema).
• Unknowns is independently deployable and is not consolidated into Policy or any other module.
• Unknowns does not guess identities. It records what cannot be determined and (where evidence exists) attaches provenance hints with hypotheses.
• All unknowns are classified (UnknownKind) and triaged (HOT/WARM/COLD bands) for actionability.
• Library layers within Unknowns are consumed by Scanner, Signals, and Platform via ProjectReference.

Container facts (from src/Unknowns/README.md). Container stellaops-unknowns-web; slot 46; port 8080; Router consumer group unknowns; resource tier light. Storage: PostgreSQL via ConnectionStrings:Default / ConnectionStrings:UnknownsDb (configured through the Postgres options section — see ServiceCollectionExtensions.AddUnknownsServices).

Boundary decision (Sprint 206, 2026-02-25): Unknowns retains its own schema ownership. No source consolidation into Policy and no DbContext merge. See docs-archive/implplan/2026-03-04-completed-sprints/SPRINT_20260225_206_Policy_absorb_unknowns.md (archived) for rationale.

1) Solution & project layout

Actual layout under src/Unknowns/ (verified 2026-05-30):

src/Unknowns/
 ├─ StellaOps.Unknowns.WebService/          # Standalone Minimal API host (Program.cs)
 │   ├─ Endpoints/
 │   │   ├─ UnknownsEndpoints.cs             # /api/unknowns  (registry; WIRED)
 │   │   └─ GreyQueueEndpoints.cs            # /api/grey-queue (scaffolded; NOT wired — see §6)
 │   ├─ Security/UnknownsPolicies.cs         # Unknowns.Read / Unknowns.Write policy names
 │   ├─ Translations/en-US.unknowns.json     # localized endpoint descriptions
 │   └─ ServiceCollectionExtensions.cs       # AddUnknownsServices + DatabaseHealthCheck
 ├─ StellaOps.Unknowns.Services/             # SLA / metrics / lifecycle / watchdog services
 │   │                                       #   (scaffolded; NOT registered in the host — see §6)
 │   ├─ UnknownsLifecycleService.cs
 │   ├─ UnknownsMetricsService.cs
 │   ├─ UnknownsSlaMonitorService.cs
 │   ├─ GreyQueueWatchdogService.cs
 │   ├─ SlaCalculator.cs / UnknownsSlaHealthCheck.cs
 │   └─ Abstractions.cs
 ├─ __Libraries/
 │   ├─ StellaOps.Unknowns.Core/             # Models, enums, provenance hints, analysis services
 │   │   ├─ Models/                          # Unknown, GreyQueueEntry, ProvenanceHint(+Type/Evidence), UnknownRanking, NativeUnknownContext
 │   │   ├─ Hints/                           # IProvenanceHintBuilder, ProvenanceHintBuilder
 │   │   ├─ Services/                        # NativeUnknownClassifier, RuntimeSignalIngester, UnknownProofEmitter, UnknownRanker
 │   │   ├─ Repositories/                    # IUnknownRepository, IGreyQueueRepository
 │   │   └─ Schemas/provenance-hint.schema.json
 │   ├─ StellaOps.Unknowns.Persistence/      # ACTIVE persistence
 │   │   ├─ Postgres/                         # UnknownsDataSource, PostgresUnknownRepository (runtime path used by the host)
 │   │   ├─ EfCore/                           # UnknownsDbContext, UnknownEntity, UnknownEfRepository, CompiledModels
 │   │   ├─ Extensions/UnknownsPersistenceExtensions.cs  # EF Core registration alternative (not used by host)
 │   │   └─ Migrations/                       # 001_v1_unknowns_baseline.sql (collapsed pre-1.0 chain) (+_archived/, excluded from embedding)
 │   └─ StellaOps.Unknowns.Persistence.EfCore/  # DEPRECATED scaffold — referenced by NO csproj (dead; do not use)
 │
 └─ __Tests/
     ├─ StellaOps.Unknowns.Core.Tests/
     ├─ StellaOps.Unknowns.Persistence.Tests/   # PostgresUnknownRepositoryTests
     └─ StellaOps.Unknowns.WebService.Tests/    # endpoint, persistence, startup-migration, health tests

Two persistence trees, one active. The host (ServiceCollectionExtensions.AddUnknownsServices) registers IUnknownRepository → PostgresUnknownRepository backed by UnknownsDataSource (raw Npgsql). The EF Core path (UnknownsPersistenceExtensions.AddUnknownsPersistence / …WithCompiledModel, UnknownsDbContext, UnknownEfRepository) lives in the same project but is not registered by the WebService. The separate top-level StellaOps.Unknowns.Persistence.EfCore project is a deprecated scaffold referenced by no .csproj.

2) HTTP API surface

Two endpoint groups are mapped in Program.cs (MapUnknownsEndpoints() and MapGreyQueueEndpoints()). Both route groups call .RequireTenant() and resolve the tenant exclusively from the envelope-bound stellaops:tenant claim via HttpContext.RequireTenant() (a missing tenant is rejected with 400 tenant_missing); the legacy X-Tenant-Id request header is never honoured as a tenancy source (SPRINT_20260611_001, UNKNOWNS-1). Every endpoint also requires an authorization policy. Health endpoints (/healthz, /readyz, /health) are anonymous.

2.1 Unknowns registry — /api/unknowns (WIRED)

Group default policy: Unknowns.Read (scope unknowns:read). All routes below are GET.

The list endpoint is repository-driven: asOf → AsOfAsync; else kind → GetByKindAsync; else severity → GetBySeverityAsync; else GetOpenUnknownsAsync + CountOpenAsync (see UnknownsEndpoints.ListUnknowns).

2.2 Grey Queue — /api/grey-queue (forward design; returns 501 until a store is wired — see §6)

Group default policy: Unknowns.Read. Mutating routes additionally require Unknowns.Write and emit audit events (.Audited(AuditModules.Unknowns, …)).

Runtime gate (BL-4, SPRINT_20260713_002). MapGreyQueueEndpoints checks at startup whether an IGreyQueueRepository is registered (IServiceProviderIsService). The live host wires none, so the full table below is not mapped; instead the entire /api/grey-queue/* surface answers 501 Not Implemented (title: grey_queue_not_implemented, application/problem+json, anonymous) — a truthful “not yet built” signal rather than the former 500 from an unresolved DI dependency. The real handlers below map automatically the moment a repository is registered (which the WebService test host does, exercising them). Routes are kept mapped so integrators see an unbuilt-but-planned surface.

Audit action constants are defined in StellaOps.Audit.Emission (AuditModules.Unknowns = "unknowns", AuditActions.Unknowns.*).

2.3 Authorization scopes

Defined in the canonical catalog StellaOps.Auth.Abstractions/StellaOpsScopes.cs:

Policies are wired via AddStellaOpsScopePolicy in Program.cs; policy name constants live in Security/UnknownsPolicies.cs (Unknowns.Read, Unknowns.Write).

2.4 Health

/healthz (liveness, Predicate=false → ignores DB), /readyz (readiness, only ready-tagged DatabaseHealthCheck), and /health (legacy) — all anonymous (Sprint 20260512_023, UNKNOWNS_HEALTH-01).

3) Contracts & data model

3.1 Unknown DTO (UnknownDto, from UnknownsEndpoints.cs)

Field names below match the serialized record exactly:

{
  "id": "0f3b…-uuid",
  "tenantId": "default",
  "subjectHash": "<sha256-hex>",
  "subjectType": "Binary",
  "subjectRef": "/usr/lib/libfoo.so",
  "kind": "UnknownBuildId",
  "severity": "High",
  "sourceScanId": "…",
  "sourceGraphId": "…",
  "sourceSbomDigest": "sha256:…",
  "validFrom": "2025-01-15T10:30:00Z",
  "validTo": null,
  "resolvedAt": null,
  "resolutionType": null,
  "resolutionRef": null,
  "compositeScore": 0.72,
  "triageBand": "Hot",
  "isOpen": true,
  "isResolved": false,
  "provenanceHints": [ /* ProvenanceHintDto[] */ ],
  "bestHypothesis": "Binary matches openssl 1.1.1k from debian",
  "combinedConfidence": 0.95,
  "primarySuggestedAction": "verify_build_id",
  "createdAt": "2025-01-15T10:30:00Z",
  "updatedAt": "2025-01-15T10:30:00Z"
}

The domain record Unknown (in Core/Models/Unknown.cs) additionally carries bitemporal sysFrom/sysTo, full scoring inputs (popularityScore, deploymentCount, exploitPotentialScore, uncertaintyScore/uncertaintyFlags, centralityScore/degreeCentrality/betweennessCentrality, stalenessScore/daysSinceAnalysis), rescan scheduling (rescanAttempts, nextScheduledRescan, lastAnalyzedAt), and evidence hashes (evidenceSetHash, graphSliceHash) — these are persisted but not all surfaced on the DTO.

3.2 UnknownKind (classification)

Core/Models/Unknown.cs defines 15 kinds. The first 10 are also declared in the unknowns.unknown_kind PostgreSQL enum (migration 001); the 5 native-binary kinds (Sprint 3500-0013-0001) exist in C# but are not in the DB enum:

Other enums: UnknownSubjectType (Package, Ecosystem, Version, SbomEdge, File, Runtime, Binary — note the DB subject_type enum omits Binary), UnknownSeverity (Critical/High/Medium/Low/Info), ResolutionType (FeedUpdated, SbomProvided, ManualMapping, Superseded, FalsePositive, WontFix), TriageBand (Hot/Warm/Cold).

3.3 Triage scoring (HOT / WARM / COLD)

Composite score and band assignment are computed both in PL/pgSQL helpers (unknowns.calc_composite_score, unknowns.assign_band) and reflected in C#:

• Composite = clamp01(wP·P + wE·E + wU·U + wC·C + wS·S) with default weights wP=0.25, wE=0.25, wU=0.25, wC=0.15, wS=0.10.
• HOT ≥ 0.70 (immediate rescan), WARM 0.40–0.69 (12–72h), COLD < 0.40 (weekly).
• popularity_score = min(1, log10(1+deployments)/log10(101)); staleness_score = min(1, age_days/14).

3.4 Provenance hints

ProvenanceHint (Core/Models/ProvenanceHint.cs) is an immutable record with snake_case JSON: hint_id (content-addressed, format hint:sha256:<hex24>), type, confidence (0–1), confidence_level, summary, hypothesis, evidence, suggested_actions, generated_at, source.

Hint types (ProvenanceHintType, exactly 15): BuildIdMatch, DebugLink, ImportTableFingerprint, ExportTableFingerprint, SectionLayout, StringTableSignature, CompilerSignature, PackageMetadata, DistroPattern, VersionString, SymbolPattern, PathPattern, CorpusMatch, SbomCrossReference, AdvisoryCrossReference.

Confidence levels (HintConfidence): VeryHigh (≥0.9), High (0.7–0.9), Medium (0.5–0.7), Low (0.3–0.5), VeryLow (<0.3).

Suggested action (SuggestedAction): action, priority (1 = highest), effort (low/medium/high), description, optional link. The endpoint DTO (SuggestedActionDto) exposes action, priority, description, and url (mapped from link).

Typed evidence (ProvenanceEvidence): at most one of build_id, debug_link, import_fingerprint, export_fingerprint, section_layout, compiler, distro_pattern, version_string, corpus_match, plus a raw JSON escape hatch.

JSON Schema: src/Unknowns/__Libraries/StellaOps.Unknowns.Core/Schemas/provenance-hint.schema.json.

The hint-combination confidence boost (single → two-agreeing → three-agreeing) is implemented in Core/Hints/ProvenanceHintBuilder.cs / Core/Services/UnknownRanker.cs. Exact numeric examples are not re-stated here to avoid drift from the builder; consult those classes and __Tests/.../Hints/HintCombinationTests.cs for the authoritative formula.

3.5 Grey Queue model (scaffolded)

GreyQueueEntry (Core/Models/GreyQueueEntry.cs) carries a DSSE-style evidence bundle (SBOM slice, advisory snippet, VEX evidence, diff traces, reachability evidence — each with an integrity hash), a deterministic fingerprint, trigger conditions (feeds/tools/CVEs/PURL patterns), and processing metadata (processingAttempts, maxAttempts=10, retry scheduling).

State machine (GreyQueueStateMachine): statuses Pending, Processing, Retrying, UnderReview, Escalated, Resolved, Rejected, Failed, Expired, Dismissed. Resolved/Expired are terminal; Rejected/Failed/Dismissed can return to Pending; UnderReview requires an assignee.

GreyQueueReason (10): InsufficientVex, ConflictingVex, MissingReachability, AmbiguousIdentity, FeedNotAvailable, ToolUnsupported, BinaryAnalysisInconclusive, BackportUncertain, MultipleInterpretations, AwaitingUpstreamAdvisory.

GreyQueueResolution (7): FeedUpdate, ToolUpdate, VexReceived, ManualResolution, Superseded, NotApplicable, Expired.

4) Persistence

The owning schema is unknowns(plus the unknowns_app helper schema). Startup convergence is mandatory: ServiceCollectionExtensions.AddUnknownsServices (the host path) and UnknownsPersistenceExtensions (the EF path) both call AddStartupMigrations(UnknownsDataSource.DefaultSchemaName, "Unknowns.Persistence", typeof(UnknownsDataSource).Assembly), and the persistence project embeds Migrations/**/*.sql (explicitly excluding Migrations/_archived/** per Sprint 20260512_026 to avoid leaf-name collisions in the migration runner).

Migrations (active): a single embedded file, 001_v1_unknowns_baseline.sql — the pre-1.0 chain (001_initial_schema.sql, 002_provenance_hints.sql, 003_unknowns_scoring_drift_repair.sql) was collapsed into it. Those three filenames now exist only under Migrations/_archived/pre_1.0/mig061/ and are excluded from embedding; cite the baseline, not the archived names. Folded content, in preserved source order:

• (from 001_initial_schema.sql) — schemas, enum types, the bitemporal unknowns.unknown table (with full scoring columns), indexes, views (unknowns.current, unknowns.resolved, unknowns.triage_summary), temporal/scoring PL/pgSQL functions (as_of, count_by_kind, count_by_severity, calc_popularity_score, calc_staleness_score, assign_band, calc_composite_score), row-level security (unknown_tenant_isolation, FORCE ROW LEVEL SECURITY, unknowns_admin BYPASSRLS role), and an updated_at trigger.
• (from 002_provenance_hints.sql, baseline line 541 onward) — adds provenance_hints (JSONB), best_hypothesis, combined_confidence, primary_suggested_action; GIN + partial confidence indexes; validate_provenance_hints function + chk_provenance_hints_valid constraint.
• (from 003_unknowns_scoring_drift_repair.sql) — idempotent reconciliation for databases that drifted when the runner previously picked up archived pre-1.0 scripts (Sprint 20260512_026, UNKNOWNS_DRIFT-01). No-op on fresh DBs.

There is no grey_queue table. No active migration creates unknowns.grey_queue (or any grey-queue table). The only table created is unknowns.unknown. See §6.

Tenant isolation relies on the app.tenant_id session GUC, enforced by RLS via unknowns_app.require_current_tenant().

5) Observability

Metrics and tracing are defined in StellaOps.Unknowns.Services/UnknownsMetricsService.cs (Meter and ActivitySource both named StellaOps.Unknowns, version 1.0.0):

• Gauges: unknowns_queue_depth, unknowns_queue_depth_hot, unknowns_queue_depth_warm, unknowns_queue_depth_cold, unknowns_sla_compliance_rate, unknowns_sla_compliance.
• Histograms: unknowns_processing_time_seconds, unknowns_resolution_time_hours (label band).
• Counter: unknowns_state_transitions_total (labels from_state, to_state).
• Structured log messages via UnknownsLogging (LoggerMessage source-gen) and W3C traceparent propagation helpers (TraceContextPropagation).

These meters live in UnknownsMetricsService, which is not registered in the running host (see §6); they emit only if/when that background service is wired.

6) Implementation status (what is and isn’t wired at runtime)

This section is the load-bearing correction versus prior revisions of this dossier.

WIRED (functional in the running stellaops-unknowns-web):

• The /api/unknowns/* registry, backed by IUnknownRepository → PostgresUnknownRepository (registered in AddUnknownsServices).
• AddStartupMigrations for the unknowns schema; the unknowns.unknown table + provenance-hint columns.
• Authentication/authorization (unknowns:read / unknowns:write), tenant middleware, Router integration, localization, audit emission, CORS, health endpoints.
• Provenance-hint persistence (AttachProvenanceHintsAsync, GetWithHighConfidenceHintsAsync in PostgresUnknownRepository), with deterministic ordering and tenant scoping (Sprint 304).

SCAFFOLDED but NOT wired (will not function at runtime as-is):

• Grey Queue endpoints (GreyQueueEndpoints.MapGreyQueueEndpoints, mapped in Program.cs) resolve IGreyQueueRepository via [FromServices], but no concrete IGreyQueueRepository is registered anywhere in the host DI, and no grey_queue table exists in any migration. As of BL-4 (SPRINT_20260713_002) the mapping is gated on repository availability (IServiceProviderIsService): with no store wired, the whole /api/grey-queue/* surface returns a documented 501 Not Implemented (grey_queue_not_implemented) instead of the former runtime 500. The full handler set (§2.2) maps automatically once a repository is registered. Treat the Grey Queue HTTP surface as a forward design that is not yet operational.
• Background services in StellaOps.Unknowns.Services (UnknownsLifecycleService, UnknownsMetricsService, UnknownsSlaMonitorService, GreyQueueWatchdogService, UnknownsSlaHealthCheck) all depend on IGreyQueueRepository and are not registered (Program.cs contains no AddHostedService for them). The README correctly states “Background Workers: None”.
• Core analysis services (NativeUnknownClassifier, RuntimeSignalIngester, UnknownProofEmitter, UnknownRanker, ProvenanceHintBuilder) are not referenced by the WebService host; they are library building blocks awaiting integration.
• The standalone StellaOps.Unknowns.Persistence.EfCoreproject is a deprecated scaffold referenced by no .csproj.

Follow-up flag for maintainers. Option (b) — gate the grey-queue mapping so it fails honestly until backed — is done (BL-4, SPRINT_20260713_002): the surface now returns 501 Not Implemented rather than 500. The remaining forward work is option (a): register a concrete IGreyQueueRepository implementation + add a grey_queue migration + register the background services, at which point the gate lights up the real API with no further mapping changes.

Related Documentation

• Scanner: ../scanner/architecture.md
• Signals: ../signals/architecture.md
• Policy: ../policy/architecture.md(Policy references Unknowns via UnknownsBudgetGate but does not own Unknowns persistence or source)
• Boundary decision (archived): docs-archive/implplan/2026-03-04-completed-sprints/SPRINT_20260225_206_Policy_absorb_unknowns.md

Advisory Gap Status (2026-03-04 Batch)

Status: provenance-hints persistence delivered in Sprint 304.

• AttachProvenanceHintsAsync and GetWithHighConfidenceHintsAsync are implemented in active repositories:
  • src/Unknowns/__Libraries/StellaOps.Unknowns.Persistence/Postgres/Repositories/PostgresUnknownRepository.cs (runtime path used by the host)
  • src/Unknowns/__Libraries/StellaOps.Unknowns.Persistence/EfCore/Repositories/UnknownEfRepository.cs (EF path, not registered by the host)
• High-confidence retrieval applies deterministic ordering (combined_confidence DESC, id ASC) and tenant scoping.
• The provenance-hint columns are added to unknowns.unknown by src/Unknowns/__Libraries/StellaOps.Unknowns.Persistence/Migrations/001_v1_unknowns_baseline.sql (line 562 onward — folded in from the archived 002_provenance_hints.sql).
• The duplicate scaffold project src/Unknowns/__Libraries/StellaOps.Unknowns.Persistence.EfCore/** is non-active/deprecated (referenced by no csproj) — do not extend it.
• Startup convergence is mandatory for the owned unknowns schema (AddStartupMigrations + embedded Migrations/**/*.sql, archived scripts excluded). Fresh PostgreSQL volumes converge through service startup, not manual init scripts.

Closure sprint (archived):

• docs-archive/implplan/2026-03-04-completed-sprints/SPRINT_20260304_304_Unknowns_provenance_hints_persistence_completion.md