Unknowns Queue Management Runbook

Version: 2.1.0 Last Updated: 2026-05-31 Audience: Security analysts and operations / on-call engineers Reconciled against: src/Policy/__Libraries/StellaOps.Policy.Unknowns, src/Policy/StellaOps.Policy.Engine/Endpoints, src/Unknowns, src/Scanner/StellaOps.Scanner.WebService/Endpoints/UnknownsEndpoints.cs, src/Cli/StellaOps.Cli/Commands/UnknownsCommandGroup.cs

This runbook covers the day-to-day operation of Unknowns — items a scan could not fully classify — including triage, escalation, resolution, per-environment budgets, the grey queue, and queue-health monitoring.

Read this first — there are THREE distinct unknowns surfaces in the codebase. Operators must know which one they are talking to, because scores, bands, scopes, and even the host differ:

Surface	Host / route prefix	Scope	Score scale	Write ops?	Source of truth
Scanner Unknowns API (read-only registry)	Scanner WebService, /api/v1/unknowns	scanner.scans.read	0.0–1.0	No — list/get/stats/bands/evidence/history only	src/Scanner/StellaOps.Scanner.WebService/Endpoints/UnknownsEndpoints.cs
Policy Engine Unknowns + Budgets (ranking, escalate/resolve, budgets)	Policy Engine, /api/v1/policy/unknowns and /api/v1/policy/budgets	policy:read / policy:operate	0–100	Yes — escalate, resolve, budget check	src/Policy/StellaOps.Policy.Engine/Endpoints/{UnknownsEndpoints,BudgetEndpoints}.cs
Unknowns WebService (provenance hints + grey queue)	Unknowns WebService, /api/unknowns and /api/grey-queue	unknowns:read / unknowns:write	0.0–1.0 (grey-queue band)	Yes — full grey-queue state machine	src/Unknowns/StellaOps.Unknowns.WebService/Endpoints/*.cs

The stella unknowns CLI talks to the Policy Engine surface (PolicyApi client). Replace example hosts/paths with your gateway routes; the example hostnames below are placeholders.

Table of Contents

1. Overview
2. Queue Operations (CLI)
3. Triage Procedures
4. Escalation Workflows
5. Resolution Procedures
6. Troubleshooting
7. Monitoring & Alerting
8. Unknown Budgets
9. Grey Queue Operations

1. Overview

What are Unknowns?

Unknowns are items that could not be fully classified during scanning. In the ranking model (UnknownReasonCode), each unknown carries a single primary reason code:

Source: src/Policy/__Libraries/StellaOps.Policy.Unknowns/Models/UnknownReasonCode.cs and the ShortCodes map in UnknownsEndpoints.cs.

Unknown ranking (Policy Engine model)

The Policy Engine ranks unknowns with a two-factor model (NOT a blast/scarcity/pressure weighted sum). The implemented formula is:

rawScore        = (UncertaintyFactor × 50) + (ExploitPressure × 50)   # 0–100
decayedScore    = rawScore × decayFactor                              # time decay
finalScore      = max(0, decayedScore × (1 − containmentReduction))   # rounded to 2dp

Source: UnknownRanker.Rank in src/Policy/__Libraries/StellaOps.Policy.Unknowns/Services/UnknownRanker.cs.

Uncertainty factor (capped at 1.0):

Exploit pressure (capped at 1.0):

Time decay (DecayBucket, basis points; applied to the score, not added): age ≤7d → ×1.00, ≤30d → ×0.90, ≤90d → ×0.75, ≤180d → ×0.60, ≤365d → ×0.40, older → ×0.20. Decay can be disabled (EnableDecay).

Containment reduction (multiplicative, capped at MaxContainmentReduction = 0.40): isolated package (dependents=0) −0.15; not network-facing −0.05; non-root privilege −0.05; seccomp enforced −0.10; read-only filesystem −0.10; isolated network policy −0.05. Can be disabled (EnableContainmentReduction).

Band assignment (Policy Engine, 0–100 scale)

Defaults from UnknownRankerOptions (HotThreshold/WarmThreshold/ColdThreshold):

Source: UnknownBand in Models/Unknown.cs and the thresholds in UnknownRanker.cs.

The Scanner read-only registry and the grey queue use a different scale. Both express score on 0.0–1.0 with bands HOT ≥ 0.70, WARM ≥ 0.40, COLD < 0.40 (Scanner: UnknownsEndpoints.DetermineBand; grey queue: GreyQueueEntry.Band / EffectiveSla). When you see a score like 0.62 in a Scanner API response, that is the 0–1 scale — not the 0–100 Policy Engine score.

2. Queue Operations (CLI)

The stella unknowns command tree (built in src/Cli/StellaOps.Cli/Commands/UnknownsCommandGroup.cs) implements exactly these subcommands: list, escalate, resolve, summary, show, proof, export, triage, and budget {check,status}. All commands accept a global --tenant / -t option and --verbose. They call the Policy Engine PolicyApi client.

2.1 View summary

# Summary counts by band (HOT/WARM/COLD/Resolved + Total)
stella unknowns summary

# JSON output
stella unknowns summary --format json

Backed by GET /api/v1/policy/unknowns/summary (policy:read). Returns { hot, warm, cold, resolved, total }.

2.2 List Unknowns

# List (default table; default limit 50)
stella unknowns list

# Filter by band (HOT, WARM, COLD)
stella unknowns list --band HOT

# Sort and paginate
stella unknowns list --sort score --limit 20 --offset 0

# JSON output
stella unknowns list --format json

Options: --band/-b, --limit/-l, --offset, --format/-f {table,json}, --sort/-s {age,band,cve,package}. Backed by GET /api/v1/policy/unknowns (policy:read).

Not implemented: --reason, --artifact, --kev, --since, --sla-breached, --has-conflicts, --conflict-type, --observation-state, --show-fingerprint, and --show-conflicts filters are not CLI options — do not script against them.

2.3 View Unknown Details

# Detailed view (requires --id / -i; id is a GUID)
stella unknowns show --id 12345678-abcd-1234-5678-abcdef123456

# JSON
stella unknowns show --id <guid> --format json

The text view prints package, band, score (2dp), reason code + short code, first-seen / last-evaluated, fingerprint, triggers, next actions, conflicts, and a remediation hint. Backed by GET /api/v1/policy/unknowns/{id:guid} (policy:read).

# Evidence proof (fingerprint, triggers, evidence refs) as JSON or DSSE envelope
stella unknowns proof --id <guid> --format json

The proof command derives the proof object client-side from the same GET /api/v1/policy/unknowns/{id} response (there is no dedicated /proof endpoint on the Policy Engine). On the Scanner surface, evidence is fetched via GET /api/v1/unknowns/{id}/evidence instead.

2.4 Export Queue Data

# Export all unknowns (default JSON envelope to stdout)
stella unknowns export

# Filter by band and choose a format
stella unknowns export --band HOT --format csv --output hot-unknowns.csv

# NDJSON
stella unknowns export --format ndjson --output unknowns.ndjson

Options: --band/-b {HOT,WARM,COLD,all}, --format/-f {json,csv,ndjson}, --schema-version (default unknowns.export.v1), --output/-o (default stdout). Output is deterministically ordered (band priority, then score desc, then package, version, id).

Not implemented: --verbose/--include-proofs/--include are not export options. There is no stella unknowns snapshot command.

3. Triage Procedures

3.1 Daily Triage Workflow

Suggested cadence: daily review of HOT items by a security analyst / on-call engineer.

# 1. Snapshot today's queue (use export, not the non-existent `snapshot` command)
stella unknowns export --output daily-$(date +%Y%m%d).json

# 2. Review HOT items
stella unknowns list --band HOT

# 3. For each item, apply a triage decision (see 3.2)

3.2 Triage decisions (CLI triage)

stella unknowns triage --id <guid> --action <action> --reason "<text>" [--duration-days <n>]

--id/-i, --action/-a, and --reason/-r are required. The valid --action values are validated client-side:

The CLI posts to POST /api/v1/policy/unknowns/{id}/triage.

Implementation gap (verify before relying on it): the triage subcommand exists in the CLI, but the Policy Engine endpoint group (UnknownsEndpoints.cs) currently maps only list, summary, {id}, {id}/escalate, and {id}/resolve — there is no {id}/triage route in source. Treat stella unknowns triage as not-yet-wired server-side until a backing route + audit action are added. Track via a sprint task before documenting it as live.

Not implemented: the old --notes, --priority, --due-date, --assign, --evidence flags and the --action investigate value do not exist. There is no triage “decision matrix” command, no sla-status command, and no --since filter.

4. Escalation Workflows

4.1 Manual Escalation (CLI)

# Escalate an unknown (id and optional reason)
stella unknowns escalate --id <guid> --reason "Customer reported potential exploit"

Options: --id/-i (required), --reason/-r. Backed by POST /api/v1/policy/unknowns/{id}/escalate (policy:operate, audited as Policy.EscalateUnknown).

What escalation does today: the endpoint promotes the unknown to the Hot band and sets its score to the HOT threshold (75.0) if it is not already HOT. Rescan is not yet wired — the Scheduler rescan trigger is a TODO in Escalate (UnknownsEndpoints.cs). Do not expect an escalation to enqueue a rescan job.

Not implemented: --rescan, bulk escalation (escalate --filter ..., escalate --artifact ...), escalation-status, and --force are not CLI options. There is no automatic escalation rules engine and no policy.unknowns.escalation.yaml config file in the repo — do not configure auto-escalation triggers against a schema that does not exist.

4.2 Grey-queue escalation (Unknowns WebService)

The separate grey queue has its own escalate action that notifies the security team:

POST /api/grey-queue/{id}/escalate    (unknowns:write)
Body: { "reason": "Conflicting VEX requires security team decision" }

This transitions the grey-queue entry to Escalated and publishes an escalation notification. See Section 9.

5. Resolution Procedures

5.1 Resolve an Unknown (CLI / Policy Engine)

stella unknowns resolve --id <guid> --resolution <type> [--note "<text>"]

--id/-i and --resolution/-r are required. The CLI documents the resolution values as matched, not_applicable, deferred, with an optional free-text --note/-n. Backed by POST /api/v1/policy/unknowns/{id}/resolve (policy:operate, audited as Policy.ResolveUnknown).

The server requires a non-empty resolution reason: UnknownsEndpoints.Resolve returns 400 when request.Reason is null/blank, 404 when the id is not found, and on success marks the unknown resolved (drops out of active budget / open-unknown counts) and persists the reason for audit.

Contract mismatch (verify before relying on it): the CLI serialises its request as { "resolution": <value>, "note": <text> } (camelCase), but the Policy Engine endpoint binds ResolveUnknownRequest(string Reason) — i.e. it reads a reason field, not resolution or note. Because resolution and reason are different property names (System.Text.Json is case-insensitive on names, not aliasing), the bound Reason is null, so the endpoint takes its 400 “Resolution reason is required.” branch regardless of the --resolution value supplied. As written, stella unknowns resolve does not successfully resolve against the live Policy Engine endpoint. Until the CLI request shape (or the endpoint binding) is aligned, resolve an unknown by POSTing { "reason": "<text>" } directly to /api/v1/policy/unknowns/{id}/resolve. Track via a sprint task before documenting the CLI path as live.

Not implemented: the resolution types not_affected, fixed, mitigated, false_positive, wont_fix and the flags --justification, --evidence, --expires, --approver are not part of the CLI or the Policy Engine resolve endpoint. There is no resolve-batch, no resolve-conflict, no history, no audit-export, no resolution-requirements, and no per-band approver gating. The grey queue has its own resolution enum (GreyQueueResolution: FeedUpdate, ToolUpdate, VexReceived, ManualResolution, Superseded, NotApplicable, Expired) — see Section 9.

6. Troubleshooting

6.1 Score seems wrong

Symptom: Unknown scored higher or lower than expected.

Diagnosis:

# View the full record (includes score, uncertaintyFactor, exploitPressure, reasonCode)
stella unknowns show --id <guid> --format json

# Or query the Scanner read-only registry for its 0–1 score + proofRef
curl -s "$SCANNER_API/api/v1/unknowns/<unk-id>" -H "Authorization: Bearer $TOKEN"

The JSON score, uncertaintyFactor, and exploitPressure fields let you reproduce the formula in Section 1. Common causes: stale EPSS/KEV signals, outdated reachability data, or missing containment signals (which would otherwise reduce the score).

Not implemented: stella unknowns show --score-details, stella unknowns proof --verbose, stella unknowns recalculate, and stella unknowns refresh do not exist. Re-ranking happens on the next analysis cycle or on escalate (which forces HOT).

6.2 Escalation does not trigger a rescan

This is expected today — the rescan integration is a TODO in the Policy Engine escalate handler (see Section 4.1). Escalation only promotes the band/score. To re-acquire signals, trigger a scan through the normal Scanner flow.

Not implemented: stella unknowns escalation-status, stella scheduler queue status rescan, stella scan trigger --priority high, stella unknowns escalate --force, and stella health check --service scheduler are not commands provided by this module.

6.3 Duplicate unknowns

There is no stella unknowns duplicates or stella unknowns merge command. Deduplication in the grey queue is handled by the deterministic SHA-256 Fingerprint on each GreyQueueEntry (GreyQueueEntry.Fingerprint), which is used for dedup and replay. Investigate apparent duplicates by comparing fingerprints rather than merging via CLI.

6.4 Resolution rejected

A resolve call returns 400 when the bound resolution reason is empty (UnknownsEndpoints.Resolve requires a non-empty Reason) and 404 when the unknown id is not found in the tenant.

Note the CLI/endpoint field mismatch documented in Section 5.1: stella unknowns resolve sends { "resolution", "note" }, but the endpoint binds reason, so the CLI always trips the 400 branch. Resolve via the API contract directly until that is fixed:

curl -X POST "$POLICY_API/api/v1/policy/unknowns/<guid>/resolve" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{ "reason": "Code review CRV-123 confirmed function not used" }'

Not implemented: there is no per-band approver requirement or stella unknowns resolution-requirements command.

7. Monitoring & Alerting

Metrics below are emitted by the Unknowns WebService background services (src/Unknowns/StellaOps.Unknowns.Services/). They observe the grey queue (0–1 band model), not the Policy Engine 0–100 unknowns.

7.1 Metrics actually emitted

Not implemented (do not alert on these names): unknowns_sla_breach_total, unknowns_escalated_total, unknowns_demoted_total, unknowns_expired_total, unknowns_enqueued_total, and unknowns_resolved_total are not emitted. SLA breaches are counted in-process via the no-op IUnknownsMetrics.IncrementSlaBreaches implementation (UnknownsMetrics), which does not currently export a Prometheus counter. SLA breach/warning events are surfaced as notifications (SlaBreachNotification / SlaWarningNotification) by UnknownsSlaMonitorService, and via the health check below — not as a counter time series.

7.2 Background services & thresholds

• UnknownsSlaMonitorServicepolls every 5 min (Unknowns:Sla:PollingInterval); SLAs default HOT 24h / WARM 7d / COLD 30d; warns at 80% elapsed (WarningThreshold), breach at 100%.
• GreyQueueWatchdogServicechecks every 5 min (Unknowns:Watchdog:CheckInterval); alerts on entries processing ≥ 1h (ProcessingAlertThreshold), times out at ≥ 4h (ProcessingTimeout), forces retry with exponential backoff (BaseRetryDelayMinutes = 15) or marks Failed after MaxAttempts.
• UnknownsMetricsServicecollects band/SLA gauges every 1 min (Unknowns:Metrics:CollectionInterval).

7.3 Health check

The SLA health check (UnknownsSlaHealthCheck, registered via AddUnknownsSlaCheck) reports Unhealthy when any pending entry has breached SLA, Degraded when entries are approaching breach, otherwise Healthy, with data fields total_pending, hot_count, warm_count, cold_count, breached_count, warning_count, healthy_count, sla_compliance_rate.

Wiring caveat: the Unknowns WebService Program.cs currently registers only the DatabaseHealthCheck (/healthz liveness, /readyz readiness, /health aggregate). The SLA health check is available as an extension (AddUnknownsSlaCheck) but is not wired into the WebService host as of this writing, so there is no live /health/sla endpoint. Add the registration before relying on an SLA health endpoint.

7.4 Grafana dashboard & Prometheus rules

Draft / not present in repo. A pre-built Grafana dashboard JSON and Prometheus alert-rules YAML for the unknowns queue are not committed under devops/observability/. Build dashboards and alerts from the metric names in Section 7.1. The previously documented files (devops/observability/grafana/dashboards/unknowns-queue-dashboard.json, devops/observability/prometheus/rules/unknowns-queue-alerts.yaml) and named alerts (UnknownsSlaBreachCritical, UnknownsHotQueueHigh, etc.) do not exist in the codebase.

7.5 Metric-based troubleshooting

# SLA compliance gauge
curl -s "http://prometheus:9090/api/v1/query?query=unknowns_sla_compliance" | jq

# Currently processing (grey queue)
curl -s "http://prometheus:9090/api/v1/query?query=greyqueue_processing_count" | jq

# Stuck / timeout rates (watchdog)
curl -s "http://prometheus:9090/api/v1/query?query=rate(greyqueue_stuck_total[1h])" | jq
curl -s "http://prometheus:9090/api/v1/query?query=rate(greyqueue_timeout_total[1h])" | jq

# State-transition rates (e.g. into Escalated)
curl -s "http://prometheus:9090/api/v1/query?query=rate(unknowns_state_transitions_total[1h])" | jq

To inspect stuck grey-queue entries, list entries in Processing status (see Section 9). The watchdog auto-retries timed-out entries; there is no HTTP /grey-queue/{id}/retry route — re-driving happens via the watchdog or via POST /api/grey-queue/{id}/process.

8. Unknown Budgets

Unknown budgets enforce per-environment caps on unknowns by reason code, evaluated by the Policy Engine. Budgets can warn or block when exceeded (BudgetAction: Warn, Block, WarnUnlessException).

8.1 CLI

# Check a scan or verdict file against an environment budget (CI gate)
stella unknowns budget check --environment prod --scan-id <scan-id> --fail-on-exceed
stella unknowns budget check --environment prod --verdict ./verdict.json --output sarif

# Show current budget status for an environment
stella unknowns budget status --environment prod

budget check options: --scan-id/-s, --verdict/-v, --environment/-e (default prod), --config/-c, --fail-on-exceed (default true), --output/-o {text,json,sarif}. Exit codes: 0 pass, 1 error, 2 budget exceeded.

8.2 API

Budget endpoints live under /api/v1/policy/budgets in the Policy Engine (BudgetEndpoints.cs):

The CLI budget check/budget status currently target /api/v1/policy/unknowns/budget/{check,status}, which differs from the implemented /api/v1/policy/budgets/{environment}/{check,status} routes above. If a CLI budget call returns 404, fall back to the API paths in this table (the CLI also performs a local default-budget check when the API call fails).

8.3 Default budgets

From DefaultBudgets (Configuration/DefaultBudgets.cs), keyed by environment (prod/production, stage/staging, dev/development, else default):

8.4 Configuration

Budgets bind from the UnknownBudgets configuration section (UnknownBudgetOptions):

# Configuration section: UnknownBudgets
UnknownBudgets:
  EnforceBudgets: true        # false = warn only
  Budgets:
    prod:
      Environment: prod
      TotalLimit: 3
      ReasonLimits:
        Reachability: 0
        Provenance: 0
        VexConflict: 1
      Action: Block
      ExceededMessage: "Production requires zero reachability unknowns"
    stage:
      Environment: stage
      TotalLimit: 10
      ReasonLimits:
        Reachability: 1
      Action: WarnUnlessException
    dev:
      Environment: dev
      TotalLimit: null
      Action: Warn
    default:
      Environment: default
      TotalLimit: 5
      Action: Warn

The section name (UnknownBudgets) and key casing (EnforceBudgets, Budgets, ReasonLimits, Action, ExceededMessage) match the bound options. There is no checked-in etc/policy.unknowns.budgets.yaml file — supply this configuration through your normal Policy Engine config layering. ReasonLimits keys must be valid UnknownReasonCode names (Reachability, Identity, Provenance, VexConflict, FeedGap, ConfigUnknown, AnalyzerLimit).

8.5 Exception coverage

To allow an approved exception to cover specific unknown reason codes, set the exception metadata key unknown_reason_codes to a comma-separated list of reason-code names (consumed by UnknownBudgetService). Example: Reachability, VexConflict. This applies when an environment’s Action is WarnUnlessException.

9. Grey Queue Operations

Sprint: SPRINT_20260118_018_Unknowns_queue_enhancement (UQ-005), SPRINT_20260112_010

The grey queue (Unknowns WebService, /api/grey-queue, scopes unknowns:read / unknowns:write) handles unknowns that cannot be definitively resolved and are queued for reprocessing when feeds/tools update. It persists a DSSE-style evidence bundle (SBOM slice, advisory snippet, VEX evidence, diff traces, reachability evidence) and a deterministic fingerprint for dedup/replay. It is distinct from the Policy Engine unknowns and the Scanner read-only registry.

All grey-queue requests require an X-Tenant-Id header (read directly from the request header by the endpoint handlers).

9.1 State machine

GreyQueueStatus and GreyQueueStateMachine (GreyQueueEntry.cs) define the states and legal transitions:

There is no PendingDeterminization, Disputed, or GuardedPass grey-queue state. Transition to UnderReview requires an assignee.

GreyQueueReason values: InsufficientVex, ConflictingVex, MissingReachability, AmbiguousIdentity, FeedNotAvailable, ToolUnsupported, BinaryAnalysisInconclusive, BackportUncertain, MultipleInterpretations, AwaitingUpstreamAdvisory.

9.2 Read endpoints (unknowns:read)

GET  /api/grey-queue                          # list (skip, take, status?, reason?)
GET  /api/grey-queue/{id}                      # by entry id
GET  /api/grey-queue/by-unknown/{unknownId}    # by unknown id
GET  /api/grey-queue/ready                      # entries ready for processing
GET  /api/grey-queue/triggers/feed/{feedId}    # entries triggered by a feed update
GET  /api/grey-queue/triggers/tool/{toolId}    # entries triggered by a tool update
GET  /api/grey-queue/triggers/cve/{cveId}      # entries triggered by a CVE update
GET  /api/grey-queue/summary                    # counts by status + by reason
GET  /api/grey-queue/{id}/transitions          # valid next states for an entry

The list endpoint filters by status (a GreyQueueStatus) or reason (a GreyQueueReason) query parameter and paginates with skip/take. Example:

# List pending entries (default), or filter by status
curl -s "$UNKNOWNS_API/api/grey-queue?status=Processing&take=50" -H "X-Tenant-Id: $TENANT"

# Summary (Total/Pending/Processing/Retrying/Resolved/Failed/Expired/Dismissed + ByReason)
curl -s "$UNKNOWNS_API/api/grey-queue/summary" -H "X-Tenant-Id: $TENANT"

9.3 Write endpoints (unknowns:write, audited)

POST /api/grey-queue                  # enqueue a new entry
POST /api/grey-queue/{id}/process     # mark as processing
POST /api/grey-queue/{id}/result      # record processing result
POST /api/grey-queue/{id}/resolve     # resolve (body: { resolution, resolutionRef? })
POST /api/grey-queue/{id}/dismiss     # dismiss (body: { dismissedBy, reason? })
POST /api/grey-queue/expire           # expire old entries
POST /api/grey-queue/{id}/assign      # assign for review (body: { assignee, notes? })
POST /api/grey-queue/{id}/escalate    # escalate to security team (body: { reason })
POST /api/grey-queue/{id}/reject      # reject (body: { reason, rejectedBy })
POST /api/grey-queue/{id}/reopen      # reopen (body: { reason })

Resolution uses the GreyQueueResolution enum (FeedUpdate, ToolUpdate, VexReceived, ManualResolution, Superseded, NotApplicable, Expired):

curl -X POST "$UNKNOWNS_API/api/grey-queue/$ID/resolve" \
  -H "X-Tenant-Id: $TENANT" -H "Content-Type: application/json" \
  -d '{ "resolution": "VexReceived", "resolutionRef": "vex-observation-id-123" }'

The stella unknowns CLI targets the Policy Engine unknowns, not the grey queue. There are no stella unknowns ... --state grey, --grey, fingerprint, triggers, defer, or resolve-conflict subcommands; interact with the grey queue over HTTP using the routes above.

Related Documentation

• Unknowns API Reference (Scanner read-only surface; notes which write routes are NOT implemented)
• Unknowns API
• Score Proofs Runbook
• Policy Engine
• Determinization API
• VEX Consensus Guide