Timeline Anomaly V2

Sprint: SPRINT_20260518_053.

This dossier is the implementation contract for the Timeline anomaly v2 system. It replaces the hard-coded in-memory anomaly builder with persisted rules, baseline-aware evaluators, dispatchable alerts, and cross-service correlation support.

Schema

The shipped anomaly-v2 DDL belongs to the Timeline owner chain in src/Timeline/__Libraries/StellaOps.Timeline.Persistence/Migrations/001_v1_timeline_consolidated_baseline.sql. EST-9 removed the predecessor SQL from Timeline.Core’s production resources. A byte-pinned historical fixture remains only in the persistence tests for forward upgrades. The pre-1.0 filenames below describe provenance, not additional live migration authorities:

The folded-in 008_anomaly_rules.sql DDL creates timeline.anomaly_rules with tenant, rule kind, JSONB params, enabled state, severity, and a deterministic params_hash used for (tenant_id, rule_kind, params_hash) uniqueness. It also creates the Sprint 053 T2b fallback table timeline.tenant_business_hours because shared.tenants.timezone is not present in the current schema. The fallback defaults are UTC, 06:00-22:00 local business hours, and off_hours weekend treatment.

The folded-in 009_audit_anomaly_alerts.sql DDL creates timeline.audit_anomaly_alerts. Alerts are keyed by UUID, point back to timeline.anomaly_rules(id), carry dispatcher state (pending, dispatched, failed, suppressed), and include dedup_hash so evaluator reruns do not duplicate the same (tenant_id, rule_kind, affected_event_ids, window_start) alert. The dispatcher worklist index is (tenant_id, dispatch_state, detected_at) and the read-path index is (tenant_id, acknowledged, detected_at).

Retention classes (P13, 007_v1_timeline_anomaly_retention_classes.sql): actor_behavior_baseline is derived-rebuildable, because the rollup recomputes every group from timeline.unified_audit_events. anomaly_rules and audit_anomaly_alerts are source-of-truth and nothing reclaims them by age. An alert’s detection can be recomputed, but its acknowledgement and dispatch state cannot: the Notify dispatcher’s only cross-process idempotency is dispatch_state. A rule that has raised an alert cannot be deleted while the alert exists, because rule_id references the rule without ON DELETE.

Rule Kinds

The public contract enum lives in src/Timeline/__Libraries/StellaOps.Timeline.Core/Anomalies/AnomalyContracts.cs.

The canonical Notify channel type literal for anomaly dispatch is audit_anomaly. Dot notation (audit.anomaly) remains an operator-facing label, not the Postgres enum literal.

Default rules are seeded by AnomalyDefaultRuleSeedHostedService for tenants known to Timeline anomaly storage. The same seed path is called lazily by the anomaly read/admin paths for tenant-scoped tests and first-use tenants. The seeded kinds are the five legacy rules plus per_actor_volume_zscore, new_actor_ip, actor_action_novelty, off_hours_policy, and escalation_chain.

Admin CRUD is exposed under /api/v1/audit/anomalies/rules and requires Timeline.Admin. Request bodies use:

{
  "ruleKind": "failed_auth_spike",
  "severity": "error",
  "enabled": true,
  "params": { "min_count": 3, "max_affected_events": 20 }
}

Validation is hand-rolled per rule kind. T2 did not add JsonSchema.Net or any new dependency.

Baseline Rollup

The folded-in 007_actor_behavior_baseline.sql DDL (baseline line 538) creates timeline.actor_behavior_baseline. The logical key is tenant, actor, metric, action, and resource type. PostgreSQL cannot place COALESCE(...) expressions directly in a primary key, so the migration stores generated action_key and resource_type_key columns and uses them for the primary key while preserving nullable public action and resource_type columns.

Baseline rows use the metric name hourly_event_count. The rollup service reads timeline.unified_audit_events over a rolling seven-day window, groups events by (tenant_id, actor_id, action, resource_type), and computes nearest rank p50/p95/p99 over non-empty hourly event-count samples. sample_count is therefore the number of active hours included in the projection, not the number of raw audit events.

ActorBehaviorBaselineRollupHostedService runs once on startup and then on a five-minute cadence. Operators can override the cadence with TIMELINE_BASELINE_ROLLUP_INTERVAL_SECONDS; the bounded source-event page size defaults to 10000 and can be changed with Timeline:BaselineRollup:BatchSize. Source-event scans use deterministic keyset ordering by timestamp, event id, and tenant id.

Each source page is folded before the next page is requested. The accumulator retains at most Timeline:BaselineRollup:MaxDistinctGroups distinct (tenant_id, actor_id, action, resource_type) groups (default 10000, hard clamp 20000); TIMELINE_BASELINE_ROLLUP_MAX_DISTINCT_GROUPS is the direct environment override. When the limit is reached, existing admitted groups continue to aggregate, events belonging to later groups are omitted, and the pass returns DegradedDistinctGroupLimit. A structured warning records reason=distinct_group_limit, the configured limit, admitted groups, source events, and omitted events. This is an explicit partial projection, not a successful-complete claim.

The projector stores at most 50 distinct IP addresses and 50 distinct user agents per row. Both arrays are serialized in ordinal order so identical audit input and identical now produce byte-identical baseline rows. last_updated is set to the evaluated window end by the rollup path for the same reason.

The warm-up gate for baseline-aware anomaly rules is sample_count >= 20, backed by the partial index `idx_actor_behavior_baseline_warm_actor (tenant_id, actor_id) WHERE sample_count

= 20`. T2 evaluators must treat rows below that gate as cold-start baselines and avoid high-confidence actor-baseline alerts from them.

The Timeline meter emits:

Escalation Graph

EscalationChainEvaluator handles rule_kind = escalation_chain. The default rule is enabled at severity critical with:

{
  "window_minutes": 30,
  "max_gap_minutes": 30,
  "max_chains": 100
}

The evaluator reads the current rolling slice ending at the evaluation WindowEnd. It orders audit events by timestamp and event id, then builds link indexes by actor id and by non-empty correlation_id. Actor id is the normal graph key; correlation_id is a strong-link bridge for cross-service events where downstream services may project a service actor instead of the originating operator.

The matched pattern is:

1. root: module = authority and action in grant, role_assign, or scope_grant;
2. impact: follows the root within max_gap_minutes, shares actor id or correlation_id, has module in policy, releaseorchestrator, findings, or evidencelocker, and has severity warning or critical;
3. optional terminal: follows the impact within max_gap_minutes, shares actor id or correlation_id with the impact, and has action in impersonate, assume_role, or sudo.

Each leaf path emits one escalation_chain_detected alert. A two-event chain is emitted when no terminal event extends the root-impact edge; otherwise each terminal leaf gets its own alert. affected_event_ids are ordered by event timestamp and event id before persistence so the T2 dedup_hash path remains deterministic across evaluator reruns.

Alert details include chain_type, event_count, root_event_id, leaf_event_id, actor_ids, modules, actions, window_minutes, max_gap_minutes, linked_via_correlation_id, and confidence. When at least two events in the chain share the same non-empty correlation_id, details also include correlation_id, set linked_via_correlation_id = true, and use confidence 0.95. Actor-only chains use confidence 0.75.

Example:

{
  "ruleKind": "escalation_chain",
  "severity": "critical",
  "params": {
    "window_minutes": 30,
    "max_gap_minutes": 30,
    "max_chains": 100
  }
}

For a correlated sequence authority/scope_grant -> policy/update warning -> authority/sudo, the persisted alert has affected_event_ids ordered as the grant, policy warning, then sudo event, and details set linked_via_correlation_id to true.

Dispatcher Contract

T2 persists evaluator output to timeline.audit_anomaly_alerts with dispatch_state = 'pending'. GET /api/v1/audit/anomalies now evaluates the current seven-day tenant window, upserts any resulting alerts, and returns the legacy response shape from the persisted table. POST /api/v1/audit/anomalies/{id}/acknowledge continues to write the legacy acknowledgement row and also flips audit_anomaly_alerts.acknowledged; pending alerts are suppressed so T5 does not dispatch them.

Sprint 20260518_053 T5 exposes the HTTP dispatch boundary without coupling Notify to Timeline storage:

• GET /api/v1/audit/anomalies/_pending-dispatch?limit=50 is admin-scoped, tenant-scoped, and returns unacknowledged alerts where dispatch_state = 'pending'. Rows are ordered oldest first by detected_at, id so a dispatcher can make forward progress under load.
• POST /api/v1/audit/anomalies/{id}/dispatched is admin-scoped and idempotently marks an unacknowledged pending or already dispatched alert as dispatch_state = 'dispatched', setting dispatched_at when it was not already set.
• POST /api/v1/audit/anomalies/{id}/dispatch-failed is admin-scoped and terminally marks an unacknowledged pending alert as dispatch_state = 'failed'. The Notify dispatcher hosted service calls this after exhausting Notify:AnomalyDispatcher:MaxRetries HTTP retries against /dispatched. Once failed, subsequent polls skip the alert; operators triage via the notify_anomaly_dispatch_failures_total metric and the Notify deliveries table.
• Acknowledged alerts are excluded from pending dispatch. Acknowledgement also suppresses any still-pending alert, so Notify never dispatches an alert the operator has already handled.

Notify owns the routing subscription table notify.anomaly_subscriptions and admin endpoints at /api/v1/notify/anomaly-subscriptions. Subscriptions match Timeline rule_kind, min_severity, tenant_id, and configured Notify channel_ids.

The hosted dispatcher AnomalyAlertDispatcherHostedService (in src/Notify/StellaOps.Notify.WebService/HostedServices/) polls every Notify:AnomalyDispatcher:IntervalSeconds (default 10s) seconds. For each tenant returned by IAnomalySubscriptionRepository.ListEnabledTenantsAsync, it GETs Timeline’s _pending-dispatch, looks up matching subscriptions via ListEnabledForFanOutAsync(tenantId, ruleKind), filters by min_severity (info < warning < error < critical), de-duplicates channels across overlapping subscriptions, and writes one Notify DeliveryEntity per (alert, channel) pair with ExternalId = anomaly-{alertId}-{channelId} (the idempotency key). The existing Notifier worker dispatch pipeline then fans out the delivery via the channel adapter. On success the dispatcher POSTs /dispatched; on failure after MaxRetries (default 3) it POSTs /dispatch-failed. Acknowledged alerts in the response payload are skipped (defence-in-depth — Timeline already filters them server-side).

Metrics:

Operator Runbook

Timezone source: T2 uses timeline.tenant_business_hours until Cluster 1 ships shared.tenants.timezone. The resolver accepts IANA names where the runtime can load them and falls back to UTC for invalid or unavailable timezone ids. When Cluster 1 adds the shared column, replace this fallback resolver with the shared tenant source of truth and retain the table only as an operator override if that is approved in a later sprint.

T7 fills the remaining runbook section with the end-to-end operator verification recipe and the Tier 9 rerun evidence paths.