Timeline Architecture

Audience: developers integrating with the audit/timeline APIs, and operators reasoning about tamper-evident audit, anomaly detection, and event replay. For the module overview and quick links, see ./README.md.

Unified audit sink plus query/presentation service for HLC-ordered cross-service event timelines.

Overview

Timeline is a single deployable Stella Ops service (StellaOps.Timeline.WebService) that fulfils two related responsibilities:

1. Unified audit sink. It owns the platform-wide, tamper-evident audit log in timeline.unified_audit_events. Every Stella Ops service emits audit events here (for example, Authority login/auth events, Policy governance changes, JobEngine/Release operations, EvidenceLocker, Notify). Events are pushed to POST /api/v1/audit/ingest and persisted with a per-tenant SHA-256 hash chain for tamper evidence. The /api/v1/audit/* API serves filtered event lists, statistics, correlation clusters, chain verification, retention policies, GDPR redaction, and an anomaly-detection (anomaly v2) subsystem.
2. HLC event timeline. It provides a REST API for querying, analyzing, and replaying events indexed by the TimelineIndexer (consolidated into this module). All such events carry Hybrid Logical Clock (HLC) timestamps that establish causal ordering across distributed services. The service supports correlation-based queries, critical path analysis for latency diagnosis, and deterministic event replay/export.

The TimelineIndexer ingestion side (NATS/Redis subscribers, the timeline.timeline_events store, and query/evidence endpoints) was merged into StellaOps.Timeline.WebService and is co-hosted in the same process. The separate StellaOps.TimelineIndexer.WebService and StellaOps.TimelineIndexer.Worker projects exist but are dormant — their logic now runs inside the unified WebService (see Components).

Production Timeline hosts do not use process-local replay/export state. The in-memory replay orchestrator and export bundle builder are registered only in Development and Testing (AddTimelineServices); other environments fall back to UnsupportedTimelineReplayOrchestrator / UnsupportedTimelineBundleBuilder, so replay/export operations return 501 problem+json (timeline_replay_unavailable / timeline_export_unavailable) until durable operation storage and worker-backed execution are configured.

Components

src/Timeline/
  StellaOps.Timeline.WebService/              # Unified WebService (audit sink + HLC timeline + indexer)
    Audit/                                    # Unified audit store, providers, aggregation, retention, classifier
      PostgresUnifiedAuditEventStore.cs       # Hash-chained Postgres store (primary source of truth)
      PostgresUnifiedAuditStateStore.cs       # Durable ack/export operation state
      CompositeUnifiedAuditEventProvider.cs   # Merges Postgres store + (neutered) HTTP provider
      HttpUnifiedAuditEventProvider.cs        # Legacy per-service poller (neutered, DEPRECATE-002)
      UnifiedAuditAggregationService.cs       # Stats, correlations, timeline search, export
      AuditDataClassifier.cs                  # GDPR classification (none/personal/sensitive/restricted)
      AuditRetentionPurgeService.cs           # Scheduled per-classification retention purge (hosted)
      TimelineAuthorizationAuditSink.cs       # IAuthEventSink logging auth outcomes
      UnifiedAuditContracts.cs                # Event model + module/action/severity catalogs
    Anomalies/                                # Anomaly v2 rule engine + evaluators + escalation chains
    HostedServices/                           # AnomalyDefaultRuleSeed, ActorBehaviorBaselineRollup
    Authorization/TimelineAuthorizationMiddleware.cs
    Security/TimelinePolicies.cs              # Read/Write/Admin policy names (timeline:read/write/admin)
    Endpoints/
      TimelineEndpoints.cs                    # HLC query endpoints (/api/v1/timeline/hlc/*)
      ExportEndpoints.cs                      # HLC event export endpoints
      ReplayEndpoints.cs                      # Deterministic replay endpoints
      UnifiedAuditEndpoints.cs                # Unified /api/v1/audit ingest + query endpoints
      HealthEndpoints.cs                      # /health, /health/ready, /health/live
    Program.cs                                # Host configuration; also defines /api/v1/timeline indexer routes
  StellaOps.TimelineIndexer.WebService/       # Dormant (logic merged into Timeline.WebService)
  StellaOps.TimelineIndexer.Worker/           # Dormant standalone worker (logic merged; runs in-process)
  __Libraries/
    StellaOps.Timeline.Core/                  # HLC query/replay/export + unified-audit persistence + anomalies
      ITimelineQueryService.cs                # HLC query interface
      TimelineQueryService.cs                 # HLC query implementation (over StellaOps.Eventing store)
      Migrations/                             # timeline schema: ONE live file, 001_v1_timeline_core_baseline.sql
                                              #   (collapsed 002-010: critical-path view, unified_audit_events,
                                              #    GIN/retention/operation-state/baseline/rules/alerts).
                                              #   Pre-collapse files live in _archived/ and are NOT embedded.
      Postgres/                               # TimelineCoreDataSource + anomaly/baseline/alert repositories
      Replay/                                 # ITimelineReplayOrchestrator (+ Unsupported fallback)
      Export/                                 # ITimelineBundleBuilder (+ Unsupported fallback)
      Anomalies/                              # Contracts, default rules, baseline calculator/repository
    StellaOps.TimelineIndexer.Core/           # Ingestion domain logic
      Abstractions/                           # ITimelineEventStore, ITimelineIngestionService, etc.
      Models/                                 # TimelineEventEnvelope, TimelineEventView, etc.
      Services/                               # TimelineIngestionService, TimelineQueryService
    StellaOps.TimelineIndexer.Infrastructure/ # Persistence, EfCore, messaging subscribers
      Db/                                     # Migrations (001_initial_schema.sql), event store, query store
      EfCore/                                 # Compiled models, context, entity models
      Subscriptions/                          # NATS, Redis, Null subscribers, envelope parser, ingestion worker
      Options/TimelineIngestionOptions.cs     # Ingestion:Nats / Ingestion:Redis transport config
  __Tests/
    StellaOps.Timeline.Core.Tests/            # HLC query tests
    StellaOps.Timeline.WebService.Tests/      # API integration, audit store, anomaly, replay tests
    StellaOps.TimelineIndexer.Tests/          # Indexer unit and integration tests

Data Flow

HLC Timeline Flow

1. Events are produced by various Stella Ops services and carry HLC timestamps.
2. The in-process ingestion worker (TimelineIngestionWorker) consumes events from the message bus (NATS or Redis subscriber) and writes indexed events to the timeline.timeline_events store.
3. Timeline WebService receives HLC query requests (Platform, CLI, Web) under /api/v1/timeline/hlc/*.
4. TimelineQueryService (StellaOps.Timeline.Core) executes queries against the StellaOps.Eventing event store (ITimelineEventStore), applying correlation, service, kind, and HLC-range filters.
5. Results are returned in HLC-sorted order, with optional critical path analysis computing latency stages between correlated events. The timeline.critical_path materialized view (Timeline.Core baseline 001_v1_timeline_core_baseline.sql:28) pre-computes stage transitions over timeline.timeline_events for performance.

Unified Audit Ingest + Query Flow

1. Ingest. Any service emits an audit event by POSTing to /api/v1/audit/ingest (wire-compatible with the AuditEventPayload from the shared StellaOps.Audit.Emission library, timeline:write scope). IngestEventAsync normalizes the module/action/severity (UnifiedAuditValueMapper against UnifiedAuditCatalog) and persists via PostgresUnifiedAuditEventStore.AddAsync, which appends to timeline.unified_audit_events under SERIALIZABLE isolation with a per-tenant SHA-256 hash chain (content_hash linked to previous_entry_hash, monotonic sequence_number). PostgreSQL serialization/deadlock aborts retry the whole transaction with bounded deterministic backoff, so sequence allocation and hash computation are repeated against the current chain head. Duplicate event IDs roll back their sequence allocation and return idempotently instead of leaving a gap in the chain head. Returns 202 Accepted.
2. Read. UnifiedAuditEndpoints serves /api/v1/audit/* read requests from Web/CLI clients.
3. UnifiedAuditAggregationService routes event-list requests through the optional IUnifiedAuditPagedEventProvider capability implemented by the production CompositeUnifiedAuditEventProvider. PostgresUnifiedAuditEventStore.GetPageAsync applies the public filters, exact total count, and stable (timestamp DESC, id ASC) cursor paging in PostgreSQL; the event-list path is not bounded by the legacy 10,000-row snapshot.
4. The composite provider merges two sources: the PostgresUnifiedAuditEventStore (primary source of truth) and the HttpUnifiedAuditEventProvider. Note (orphaned/transitional): the HTTP provider is neutered as of DEPRECATE-002 — its GetEventsAsync returns an empty list because the per-service audit endpoints it used to poll (Authority, Policy, Notify, JobEngine, EvidenceLocker) now proxy into Timeline, so polling them would self-loop. In effect, Postgres is the sole live source; the HTTP path is retained dead code pending DEPRECATE-003 removal.
5. The production query capabilities remove the legacy fixed snapshot from the remaining reads: event lookup and export count execute directly in PostgreSQL, statistics are aggregated in PostgreSQL, and timeline search plus seven-day anomaly evaluation receive the complete tenant-filtered PostgreSQL result. Correlation listing uses IUnifiedAuditCorrelationEventProvider: PostgreSQL selects at most the requested 1-200 newest multi-event correlation IDs by root-event time, then returns only the events for those clusters. This preserves complete selected clusters without materializing the tenant/date event set in service memory or silently truncating at the legacy 10,000-event boundary. Anomaly acknowledgements and unified audit export status are persisted per tenant in PostgreSQL (PostgresUnifiedAuditStateStore); they are not process-local service memory.
6. If a source is unavailable, the composite provider logs the failure and continues with whatever source returned data instead of failing the unified endpoint.

Anomaly v2 Flow

1. Audit events feed deterministic per-actor behavior baselines, materialized by ActorBehaviorBaselineRollupHostedService into timeline.actor_behavior_baseline. Source rows are keyset-paged and folded immediately into ActorBehaviorBaselineAccumulator; each page is consumed before the next read, while the accumulator retains only per-group hourly counters plus the deterministic 50-value IP/user-agent caps. One pass retains at most Timeline:BaselineRollup:MaxDistinctGroups distinct (tenant, actor, action, resource type) groups (default 10,000; clamped to 20,000). After saturation, already-admitted groups remain complete, later-group source events are omitted, and the result, warning log, and metrics explicitly report a degraded distinct_group_limit state rather than risking unbounded memory growth or claiming a complete projection.
2. The AnomalyRuleEngine evaluates tenant-scoped, configurable rules (timeline.anomaly_rules, seeded by AnomalyDefaultRuleSeedHostedService) using a set of registered IAnomalyRuleEvaluators (unusual volume, failed-auth spike, privilege escalation, off-hours activity, per-actor z-score, new actor/IP, action novelty, escalation chains, etc.).
3. Detected anomalies are persisted to timeline.audit_anomaly_alerts with a dispatch-state worklist (pending → dispatched → failed). Notify polls pending, unacknowledged alerts via the admin dispatch endpoints and fans out notifications.

Database Schema

Everything lives under the timeline PostgreSQL schema. Two independent migration runners converge it on startup (both wired via AddStartupMigrations in Program.cs, against the Postgres:Timeline connection):

• TimelineIndexerrunner — applies StellaOps.TimelineIndexer.Infrastructure/Db/Migrations/ (001_initial_schema.sql): the indexer event tables and RLS scaffolding. Runs first so base tables exist.
• TimelineAuditrunner — applies StellaOps.Timeline.Core/Migrations/, which is a single live file, 001_v1_timeline_core_baseline.sql: the critical-path view, the unified audit sink, retention, durable operation state, and the anomaly v2 tables.

The library also ships TimelineIndexerMigrationHostedService, but Program.cs removes that duplicate registration so a single serialized runner owns the timeline schema (avoids two competing runners).

Migration-filename note (2026-07-12). The Timeline.Core chain formerly numbered 002–010 was collapsed into 001_v1_timeline_core_baseline.sql; those files now exist only under Migrations/_archived/pre_1.0/mig061/ and are excluded from embedding. The parenthetical “(from NNN_*)” markers below are provenance of the folded-in DDL, not live filenames. The TimelineIndexer side is different: its 001_initial_schema.sql is still a real, live migration.

Unified audit sink: timeline.unified_audit_events (baseline line 87; from 003_unified_audit_events.sql)

The platform-wide tamper-evident audit log. Immutable append-only; primary key (id, tenant_id).

Supporting objects: timeline.unified_audit_sequences (per-tenant chain head + next_unified_audit_sequence / update_unified_audit_sequence_hash functions) and verify_unified_audit_chain(tenant, from_seq, to_seq) which backs GET /api/v1/audit/chain/verify.

Retention + GDPR (baseline line 326; folded in from the archived 005_audit_data_classification_retention.sql)

Durable operation state (folded in from the archived 006_unified_audit_operation_state.sql)

Anomaly v2 (baseline lines 538-640; from 007–010)

Indexer tables: HLC timeline (TimelineIndexer 001_initial_schema.sql — a live migration)

HLC event store (Eventing)

The HLC query path (/api/v1/timeline/hlc/*) reads through the StellaOps.Eventing infrastructure (ITimelineEventStore). Runtime Eventing registration is PostgreSQL-only: AddStellaOpsEventing(IConfiguration) registers PostgresTimelineEventStore plus PostgresHlcStateStore. Test harnesses that need volatile storage must call AddStellaOpsEventingInMemory(...) explicitly; setting Eventing:UseInMemoryStore=true on the runtime configuration path fails closed.

Dependencies

Configuration

Endpoints

Scopes: timeline:read, timeline:write, timeline:admin (mapped to the Timeline.Read/Timeline.Write/Timeline.Admin policies), plus the narrow audit:ingest machine scope accepted by the ingest policy. Interactive callers authenticate with an Authority JWT. Router-forwarded internal callers use a verified, short-lived signed identity envelope. Direct service HTTP callers use an Authority-issued bearer token: ExportCenter’s audit-bundle reader requests only tenant-bound timeline:read, while emitters use tenant-bound audit:ingest. Neither path may derive tenant identity from a raw header.

HLC timeline — /api/v1/timeline/hlc/*

Indexer event queries — /api/v1/timeline/* (and bare /timeline/*)

Unified audit — /api/v1/audit/*

Health

Observability

Metrics and traces are emitted from the StellaOps.Timeline meter / ActivitySource (both version 1.0.0), defined in TimelineMetrics (StellaOps.Timeline.Core/Telemetry/TimelineMetrics.cs).

Metrics

event_count_bucket is bucketed as 1-10 / 11-100 / 101-1000 / 1001-10000 / 10000+.

Traces

Security Considerations

• Authentication: Interactive endpoints accept a valid JWT issued by Authority. Internal audit reads and ingest calls may instead use a verified signed service identity envelope; callers fail closed when the shared envelope key is absent. Authorization outcomes for timeline read/write are logged via TimelineAuthorizationAuditSink (IAuthEventSink).
• Scope model: timeline:read for tenant-scoped reads, including effective retention-policy queries; narrow audit:ingest for service emitters (timeline:write is also accepted by that endpoint); timeline:write for acknowledge, export, and replay/export initiation; timeline:admin for anomaly-rule CRUD, the Notify dispatch worklist, and GDPR PII redaction.
• Tenant isolation: Queries, unified audit rows, anomaly state, and export status are scoped to the authenticated tenant; cross-tenant access is prohibited. Indexer tables additionally enforce PostgreSQL Row-Level Security keyed on app.current_tenant.
• Tamper evidence: timeline.unified_audit_events is an append-only log with a per-tenant SHA-256 hash chain (content_hash → previous_entry_hash, monotonic sequence_number), written under SERIALIZABLE isolation. Retryable PostgreSQL concurrency aborts repeat the complete transaction; duplicate deliveries roll back the allocated sequence. GET /api/v1/audit/chain/verify detects breaks. GDPR redaction preserves actor_id so the chain stays verifiable.
• Data classification + retention: Events are classified (none/personal/sensitive/restricted); a scheduled purge enforces per-classification retention windows and honours compliance_hold (legal hold).
• Ingest surface: POST /api/v1/audit/ingest accepts service-to-service events under narrow audit:ingest or operator timeline:write; the authenticated tenant is checked against the payload tenant, and failures return 503 so emitters can retry without data loss.
• Export controls: Exported event payloads may contain sensitive operational data; exports are tracked in durable per-tenant state.
• Replay determinism: Replay operations produce identical output given identical input sequences, supporting audit and compliance verification. Production replay/export endpoints fail closed (501) when only process-local implementations are available.

TimelineIndexer (Event Ingestion and Indexing)

TimelineIndexer was consolidated into the Timeline module (Sprint 210, 2026-03-04). It provides the write/ingestion side of the CQRS pattern while Timeline provides the read/query side. Both share the same schema domain and live under src/Timeline/.

TimelineIndexer Responsibilities

• Event ingestion: Consumes events from NATS/Redis message bus via configurable subscribers.
• HLC timestamping: Assigns Hybrid Logical Clock timestamps to establish causal ordering.
• Event indexing: Writes indexed events to PostgreSQL via EfCore (compiled model preserved for migration identity).
• Authorization audit: Provides audit sink for authorization events.

Ingestion Transport Contract

• Non-testing Timeline hosts must enable at least one real ingestion transport: Ingestion:Nats:Enabled=true or Ingestion:Redis:Enabled=true.
• NullTimelineEventSubscriber is a testing-only harness and is not registered in live hosts.
• If both transports are disabled outside Testing, startup fails fast with a configuration error instead of exposing an idle ingestion worker.

Deployable Services

• Timeline WebService (StellaOps.Timeline.WebService): the single live deployable. It hosts the unified audit sink, the HLC query/analysis/export/replay API, the indexer query/evidence endpoints, the in-process ingestion worker (NATS/Redis), the retention purge service, and the anomaly v2 hosted services.
• The local Compose topology assigns this unified deployable the resources-medium tier (0.50 CPU, 1 GiB memory). The lighter 512 MiB tier is insufficient for its combined request, ingestion, retention, and anomaly-rollup responsibilities under concurrent audit reads.
• TimelineIndexer WebService (StellaOps.TimelineIndexer.WebService) and TimelineIndexer Worker (StellaOps.TimelineIndexer.Worker): dormant. These standalone projects remain in the tree, but their ingestion/query logic was merged into StellaOps.Timeline.WebService, which runs the ingestion worker in-process (AddTimelineIngestionRuntime). They are not deployed independently in current topologies.

The merged design keeps ingestion and query co-hosted while sharing domain libraries under a single module boundary. (Roadmap: the dormant projects are retained should ingestion ever need to scale out as a separate container again.)