Policy Engine Observability

This guide is the operations reference for instrumenting and monitoring the Stella Ops Policy Engine. It catalogs the metrics, logs, traces, dashboards, alerts, and sampling controls the service emits, and the incident workflow operators follow when a policy run misbehaves. Read it before standing up dashboards or wiring alert rules for the Policy Engine.

Audience: Observability Guild, SRE/Platform operators, Policy Guild.
Scope: Metrics, logs, traces, dashboards, alerting, sampling, and incident workflows for the Policy Engine service (Sprint 20).
Prerequisites: Policy Engine v2 deployed with OpenTelemetry exporters enabled (observability:enabled=true in config).

1 · Instrumentation Overview

• Telemetry stack: OpenTelemetry SDK (metrics + traces), Serilog structured logging, OTLP exporters → Collector → Prometheus/Loki/Tempo.
• Namespace conventions: policy.* for metrics/traces/log categories; labels use tenant, policy, mode, runId.
• Sampling: Default 10 % trace sampling, 1 % rule-hit log sampling; incident mode overrides to 100 % (see §6).
• Correlation IDs: Every API request gets traceId + requestId. CLI/UI display IDs to streamline support.

2 · Metrics

2.1 Run Pipeline

2.2 Evaluator Insights

2.3 API Surface

2.4 Queue & Change Streams

3 · Logs

• Format: JSON (Serilog). Core fields: timestamp, level, message, policyId, policyVersion, tenant, runId, rule, traceId, env.sealed, error.code.
• Log categories:
  • policy.run (queue lifecycle, run begin/end, stats)
  • policy.evaluate (batch execution summaries; rule-hit sampling)
  • policy.materialize (Mongo operations, conflicts, retries)
  • policy.simulate (diff results, CLI invocation metadata)
  • policy.lifecycle (submit/review/approve events)
• Sampling: Rule-hit logs sample 1 % by default; toggled to 100 % in incident mode or when --trace flag used in CLI.
• PII: No user secrets recorded; user identities referenced as user:<id> or group:<id> only.

4 · Traces

• Spans emit via OpenTelemetry instrumentation.
• Primary spans:
  • policy.api – wraps HTTP request, records endpoint, status, scope.
  • policy.select – change stream ingestion and batch assembly (attributes: candidateCount, cursor).
  • policy.evaluate – evaluation batch (attributes: batchSize, ruleHits, severityChanges).
  • policy.materialize – Mongo writes (attributes: writes, historyWrites, retryCount).
  • policy.simulate – simulation diff generation (attributes: sbomCount, diffAdded, diffRemoved).
• Trace context propagated to CLI via response headers traceparent; UI surfaces in run detail view.
• Incident mode forces span sampling to 100 % and extends retention via Collector config override.

5 · Dashboards

5.1 Policy Runs Overview

Widgets:

• Run duration histogram (per mode/tenant).
• Queue depth + backlog age line charts.
• Failure rate stacked by error code.
• Incremental backlog heatmap (policy × age).
• Active vs scheduled runs table.

5.2 Rule Impact & VEX

• Top N rules by firings (bar chart).
• VEX overrides by vendor/justification (stacked chart).
• Suppression usage (pie + table with justifications).
• Quieted findings trend (line).

5.3 Simulation & Approval Health

• Simulation diff histogram (added vs removed).
• Pending approvals by age (table with SLA colour coding).
• Compliance checklist status (lint, determinism CI, simulation evidence).

Replace these panel descriptions with Grafana screenshots once the dashboards land (store them under ../assets/policy-observability/*.png).

6 · Alerting

Alerts integrate with Notifier channels (policy.alerts) and Ops on-call rotations.

7 · Incident Mode & Forensics

• Toggle via POST /api/policy/incidents/activate (requires policy:operate scope).
• Effects:
  • Trace sampling → 100 %.
  • Rule-hit log sampling → 100 %.
  • Retention window extended to 30 days for incident duration.
  • policy.incident.activated event emitted (Console + Notifier banners).
• Post-incident tasks:
  • stella policy run replay for affected runs; attach bundles to incident record.
  • Restore sampling defaults with .../deactivate.
  • Update the incident checklist in the Policy lifecycle guide (§8 Incident Response & Rollback) with findings.

8 · Integration Points

• Authority: Exposes metric policy_scope_denied_total for failed authorisation; correlate with policy_api_requests_total.
• Concelier/Excititor: Shared trace IDs propagate via gRPC metadata to help debug upstream latency.
• Scheduler: Future integration will push run queues into shared scheduler dashboards (planned in SCHED-MODELS-20-002).
• Offline Kit: CLI exports logs + metrics snapshots (stella offline bundle metrics) for air-gapped audits.

9 · Compliance Checklist

• [ ] Metrics registered: All metrics listed above exported and documented in Grafana dashboards.
• [ ] Alert policies configured: Ops or Observability Guild created alerts matching table in §6.
• [ ] Sampling overrides tested: Incident mode toggles verified in staging; retention roll-back rehearsed.
• [ ] Trace propagation validated: CLI/UI display trace IDs and allow copy for support.
• [ ] Log scrubbing enforced: Unit tests guarantee no secrets/PII in logs; sampling respects configuration.
• [ ] Offline capture rehearsed: Metrics/log snapshot commands executed in sealed environment.
• [ ] Docs cross-links: Links to architecture, runs, lifecycle, CLI, API docs verified.

10 · Related guides

• Telemetry standards — common envelope, scrubbing policy, and sampling defaults that this guide inherits.
• Tracing standards — span naming and propagation conventions.
• Policy lifecycle & approvals — submit/review/approve flow and incident rollback.

Last updated: 2025-10-26 (Sprint 20).