Timeline Architecture

Verified against commit 942a18727c on 2026-09-26 (UTC); the ingest identity, actor binding and tenantless-indexer sections re-verified for PCD-888 on 2026-10-19. Re-verify: git grep -n "AddHostedService\|AddTimelineIngestionRuntime\|AddConsolidatedTimelinePersistence" -- src/Timeline/StellaOps.Timeline.WebService/Program.cs and ls src/Timeline/__Libraries/StellaOps.Timeline.Persistence/Migrations.

Audience: developers integrating with the audit/timeline APIs, and operators reasoning about tamper-evident audit, anomaly detection, and event replay. For the module overview and quick links, see ./README.md.

Unified audit sink plus query/presentation service for HLC-ordered cross-service event timelines.

Overview

Timeline is a single deployable Stella Ops service (StellaOps.Timeline.WebService) that fulfils two related responsibilities:

1. Unified audit sink. It owns the platform-wide, tamper-evident audit log in timeline.unified_audit_events. Owner services send supported audit events here (for example, Authority login/auth events, Policy governance changes, JobEngine/Release operations, EvidenceLocker, Notify). Events are pushed to POST /api/v1/audit/ingest and persisted with a per-tenant SHA-256 hash chain for tamper evidence. The /api/v1/audit/* API serves filtered event lists, statistics, correlation clusters, chain verification, retention policies, GDPR redaction, and an anomaly-detection (anomaly v2) subsystem.
2. HLC event timeline. It provides a REST API for querying, analyzing, and replaying events indexed by the TimelineIndexer (consolidated into this module). All such events carry Hybrid Logical Clock (HLC) timestamps that establish causal ordering across distributed services. The service supports correlation-based queries, critical path analysis for latency diagnosis, and deterministic event replay/export.

The TimelineIndexer ingestion side (NATS/Redis subscribers, the timeline.timeline_events store, and query/evidence endpoints) was merged into StellaOps.Timeline.WebService and is co-hosted in the same process. The two standalone host projects that used to carry it, StellaOps.TimelineIndexer.WebService and StellaOps.TimelineIndexer.Worker, were archived to src/__Obsoleted/Timeline/ on 2026-08-10 (SPRINT_20260722_020 W3-01 host disposition) — they are no longer in any solution and cannot be built. Their shared libraries StellaOps.TimelineIndexer.Core and StellaOps.TimelineIndexer.Infrastructure are unaffected and remain live under __Libraries/ (see Components).

Production Timeline hosts do not use process-local replay/export state. The in-memory replay orchestrator and export bundle builder are registered only in Development and Testing (AddTimelineServices); other environments fall back to UnsupportedTimelineReplayOrchestrator / UnsupportedTimelineBundleBuilder, so replay/export operations return 501 problem+json (timeline_replay_unavailable / timeline_export_unavailable) until durable operation storage and worker-backed execution are configured.

Components

src/Timeline/
  StellaOps.Timeline.WebService/              # Unified WebService (audit sink + HLC timeline + indexer)
    Audit/                                    # Unified audit store, providers, aggregation, retention, classifier
      PostgresUnifiedAuditEventStore.cs       # Hash-chained Postgres store (primary source of truth)
      PostgresUnifiedAuditStateStore.cs       # Durable ack/export operation state
      CompositeUnifiedAuditEventProvider.cs   # Merges Postgres store + (neutered) HTTP provider
      HttpUnifiedAuditEventProvider.cs        # Legacy per-service poller (neutered, DEPRECATE-002)
      UnifiedAuditAggregationService.cs       # Stats, correlations, timeline search, export
      AuditDataClassifier.cs                  # GDPR classification (none/personal/sensitive/restricted)
      AuditRetentionPurgeService.cs           # Scheduled per-classification retention purge (hosted)
      TimelineAuthorizationAuditSink.cs       # IAuthEventSink logging auth outcomes
      UnifiedAuditContracts.cs                # Event model + module/action/severity catalogs
    Anomalies/                                # Anomaly v2 rule engine + evaluators + escalation chains
    HostedServices/                           # AnomalyDefaultRuleSeed, ActorBehaviorBaselineRollup
    Authorization/TimelineAuthorizationMiddleware.cs
    Security/TimelinePolicies.cs              # Read/Write/Admin policy names (timeline:read/write/admin)
    Endpoints/
      TimelineEndpoints.cs                    # HLC query endpoints (/api/v1/timeline/hlc/*)
      ExportEndpoints.cs                      # HLC event export endpoints
      ReplayEndpoints.cs                      # Deterministic replay endpoints
      UnifiedAuditEndpoints.cs                # Unified /api/v1/audit ingest + query endpoints
      HealthEndpoints.cs                      # /health, /health/ready, /health/live
    Program.cs                                # Host configuration; also defines /api/v1/timeline indexer routes
  # (StellaOps.TimelineIndexer.WebService/ and .Worker/ were archived 2026-08-10 to
  #  src/__Obsoleted/Timeline/ — logic had already merged into Timeline.WebService)
  __Libraries/
    StellaOps.Timeline.Persistence/ # Sole host-owned startup migration chain
      Migrations/                             # 000-009: bridge, baseline, maintenance, Eventing, retention, retention classes, audit ingest identity
    StellaOps.Timeline.Core/                  # HLC query/replay/export + unified-audit persistence + anomalies
      ITimelineQueryService.cs                # HLC query interface
      TimelineQueryService.cs                 # HLC query implementation (over StellaOps.Eventing store)
      Migrations/_archived/                   # Legacy lineage only; nothing embedded or run.
      Postgres/                               # TimelineCoreDataSource + anomaly/baseline/alert repositories
      Replay/                                 # ITimelineReplayOrchestrator (+ Unsupported fallback)
      Export/                                 # ITimelineBundleBuilder (+ Unsupported fallback)
      Anomalies/                              # Contracts, default rules, baseline calculator/repository
    StellaOps.TimelineIndexer.Core/           # Ingestion domain logic
      Abstractions/                           # ITimelineEventStore, ITimelineIngestionService, etc.
      Models/                                 # TimelineEventEnvelope, TimelineEventView, etc.
      Services/                               # TimelineIngestionService, TimelineQueryService
    StellaOps.TimelineIndexer.Infrastructure/ # Persistence, EfCore, messaging subscribers
      Db/                                     # Event/query stores + embedded 001_initial_schema.sql;
                                              # timeline-web removes the hosted migrator that would run it.
      EfCore/                                 # Compiled models, context, entity models
      Subscriptions/                          # NATS, Redis, Null subscribers, envelope parser, ingestion worker
      Options/TimelineIngestionOptions.cs     # Ingestion:Nats / Ingestion:Redis transport config
    StellaOps.Timeline.DispositionProjection/ # DC-29 findings.dispositions consumer
  __Tests/
    StellaOps.Timeline.Core.Tests/            # HLC query tests
    StellaOps.Timeline.Persistence.Tests/     # Consolidated migration chain tests
    StellaOps.Timeline.DispositionProjection.Tests/
    StellaOps.Timeline.WebService.Tests/      # API integration, audit store, anomaly, replay tests
    StellaOps.TimelineIndexer.Tests/          # Indexer unit and integration tests

Data Flow

HLC Timeline Flow

1. Events are produced by various Stella Ops services and carry HLC timestamps.
2. The in-process ingestion worker (TimelineIngestionWorker) consumes events from the message bus (NATS or Redis subscriber) and writes indexed events to the timeline.timeline_events store.
3. Timeline WebService receives HLC query requests (Platform, CLI, Web) under /api/v1/timeline/hlc/*.
4. TimelineQueryService (StellaOps.Timeline.Core) executes queries against the StellaOps.Eventing event store (ITimelineEventStore), applying correlation, service, kind, and HLC-range filters.
5. Results are returned in HLC-sorted order, with optional critical path analysis computing latency stages between correlated events. The timeline.critical_path materialized view in the Timeline.Persistence owner baseline pre-computes stage transitions over timeline.timeline_events for performance. Timeline.Core embeds no migration SQL after EST-9; historical heap reconstruction is confined to the persistence test fixture.

Unified Audit Ingest + Query Flow

1. Ingest. Any authorized service emits an audit event by POSTing to /api/v1/audit/ingest (wire-compatible with the AuditEventPayload from the shared StellaOps.Audit.Emission library, audit:ingest or timeline:write scope). Before anything is written, IngestEventAsync decides the event tenant: an audit:ingest producer keeps the payload tenantId, and a payload without one is a platform-level event stored under the structural installation tenant in its own hash chain (owner ruling 2026-10-18); a caller without audit:ingest writes to its own authenticated tenant, a different payload tenant returns 403 tenant_claim_mismatch, the installation tenant returns 403 tenant_reserved, and no tenant returns 400 tenant_missing. No event is written to a default tenant. Next it records who is writing (PCD-888): UnifiedAuditIngestGuard.ResolveIngestIdentity takes the verified subject (sub, else the name identifier, else client_id), the verified client_id, and whether the caller holds audit:ingest; a caller with none of them gets 403 audit_ingest_identity_missing. An audit:ingest producer may record any actor, because the shared emitter files events on behalf of the user its service served. Any other caller records only itself: actor.id must equal its verified subject (it is filled from the credential when omitted, never system), and a supplied actor name or e-mail must match its verified subject or name claim and its email claim; otherwise 403 actor_claim_mismatch. Both refusals return before anything is written. IngestEventAsync then trims/lowercases the raw module name and normalizes action/severity against UnifiedAuditCatalog, then persists via PostgresUnifiedAuditEventStore.AddAsync, which appends to monthly-partitioned timeline.unified_audit_events under SERIALIZABLE isolation with a per-tenant SHA-256 hash chain (content_hash linked to previous_entry_hash, monotonic sequence_number). PostgreSQL serialization/deadlock aborts retry the whole transaction with bounded deterministic backoff, so sequence allocation and hash computation are repeated against the current chain head. The physical key includes the immutable timestamp, while the writer preserves logical (tenant_id, id) identity under the tenant sequence lock: exact-timestamp replay rolls back its sequence allocation and returns idempotently; the same ID with a shifted timestamp fails. The verified writer is stored on the row (ingest_subject, ingest_client_id, ingest_producer, migration 009), is part of the hashed content, and is returned on reads as ingestIdentity; the store refuses an event without it. Returns 202 Accepted.
2. Read. UnifiedAuditEndpoints serves /api/v1/audit/* read requests from Web/CLI clients.
3. UnifiedAuditAggregationService routes event-list requests through the optional IUnifiedAuditPagedEventProvider capability implemented by the production CompositeUnifiedAuditEventProvider. PostgresUnifiedAuditEventStore.GetPageAsync applies the public filters, exact total count, and stable (timestamp DESC, id ASC) cursor paging in PostgreSQL; the event-list path is not bounded by the legacy 10,000-row snapshot.
4. The composite provider merges two sources: the PostgresUnifiedAuditEventStore (primary source of truth) and the HttpUnifiedAuditEventProvider. Note (orphaned/transitional): the HTTP provider is neutered as of DEPRECATE-002 — its GetEventsAsync returns an empty list because the per-service audit endpoints it used to poll (Authority, Policy, Notify, JobEngine, EvidenceLocker) now proxy into Timeline, so polling them would self-loop. In effect, Postgres is the sole live source; the HTTP path is retained dead code pending DEPRECATE-003 removal.
5. The production query capabilities remove the legacy fixed snapshot from the remaining reads: event lookup and export count execute directly in PostgreSQL, statistics are aggregated in PostgreSQL, and timeline search plus seven-day anomaly evaluation receive the complete tenant-filtered PostgreSQL result. Correlation listing uses IUnifiedAuditCorrelationEventProvider: PostgreSQL selects at most the requested 1-200 newest multi-event correlation IDs by root-event time, then returns only the events for those clusters. This preserves complete selected clusters without materializing the tenant/date event set in service memory or silently truncating at the legacy 10,000-event boundary. Anomaly acknowledgements and unified audit export status are persisted per tenant in PostgreSQL (PostgresUnifiedAuditStateStore); they are not process-local service memory.
6. If a source is unavailable, the composite provider logs the failure and continues with whatever source returned data instead of failing the unified endpoint.

Module filtering is a separate contract from ingest. modules= values must be in UnifiedAuditCatalog.Modules; unknown values, including binaryindex, return 400 unknown_modules_filter. A mixed known/unknown query is also rejected rather than silently narrowed. Unfiltered events and module statistics may still carry raw names outside that catalog; their rows remain unchanged. PCD-413 removes the unsupported BinaryIndex Console filter without adding an alias or claiming that BinaryIndex history exists in an estate.

Verified against e8c66e5967 (2026-10-06). Re-verify the real HTTP filter and constant-parity controls from the repository root:

pwsh tools/scripts/test-targeted-xunit.ps1 -Project src/Timeline/__Tests/StellaOps.Timeline.WebService.Tests/StellaOps.Timeline.WebService.Tests.csproj -Class "*UnifiedAuditModuleFilterTests*" -BuildProjectReferences

Anomaly v2 Flow

1. Audit events feed deterministic per-actor behavior baselines, materialized by ActorBehaviorBaselineRollupHostedService into timeline.actor_behavior_baseline. Source rows are keyset-paged and folded immediately into ActorBehaviorBaselineAccumulator; each page is consumed before the next read, while the accumulator retains only per-group hourly counters plus the deterministic 50-value IP/user-agent caps. One pass retains at most Timeline:BaselineRollup:MaxDistinctGroups distinct (tenant, actor, action, resource type) groups (default 10,000; clamped to 20,000). After saturation, already-admitted groups remain complete, later-group source events are omitted, and the result, warning log, and metrics explicitly report a degraded distinct_group_limit state rather than risking unbounded memory growth or claiming a complete projection.
2. The AnomalyRuleEngine evaluates tenant-scoped, configurable rules (timeline.anomaly_rules, seeded by AnomalyDefaultRuleSeedHostedService) using a set of registered IAnomalyRuleEvaluators (unusual volume, failed-auth spike, privilege escalation, off-hours activity, per-actor z-score, new actor/IP, action novelty, escalation chains, etc.).
3. Detected anomalies are persisted to timeline.audit_anomaly_alerts with a dispatch-state worklist (pending → dispatched → failed). Notify polls pending, unacknowledged alerts via the admin dispatch endpoints and fans out notifications.

Anomaly pipeline fixture acceptance

Verified against test source e781d1b52a (2026-10-06). AnomalyV2PipelineE2ETests applies the complete consolidated Timeline Startup chain once through the shared isolated PostgreSQL fixture. Its audit seeds, baseline rollup and alert state are persisted in that owned test database. Notify subscriptions, channels and deliveries remain explicitly in-memory, with the seeded tenant registered through a fixture-only INotifyActiveTenantSource. One existing fake clock drives the rollup, dispatcher and repository-backed simulated HTTP handler. The case retains its seed, warm baseline, spike, alert, delivery identity, dispatch transition, acknowledgement and no-redispatch checks. This is fixture integration proof; it does not verify real HTTP middleware, authentication, a Notify database or an estate deployment.

Re-verify from the repository root:

pwsh tools/scripts/test-targeted-xunit.ps1 -Project src/__Tests/e2e/AnomalyV2Pipeline/StellaOps.E2E.AnomalyV2Pipeline.csproj -Method '*FullPipeline_SeedToBaselineToSpikeToAlertToDeliveryToAckToNoReDispatch*' -BuildProjectReferences

Database Schema

Which database (own database stellaops_timeline)

Timeline owns the physical PostgreSQL database stellaops_timelineand, inside it, the timeline schema (+ the RLS-helper schema timeline_app). It lives on the shared installation server db.stella-ops.local with its own login role timeline; the ownership boundary is that role plus REVOKE CONNECT on every sibling database, not a dedicated server (ADR-039 P1). The canonical connection variable is STELLAOPS_POSTGRES_TIMELINE_CONNECTION, binding to Postgres:Timeline:ConnectionString; names and the declared shape live in TimelineSchemaTopology.

Since 2026-09-15 (SPRINT_20260911_001 TLC-1) the same database also carries a third schema, eventing— the P4 reliability store (inbox, consumer_checkpoints, leases, outbox, stream_state, remote_stream_consumers) that StellaOps.Eventing.Reliability self-migrates from its own embedded SQL under its own eventing.schema_migrations ledger. It is not shared and not foreign: every host that consumes a DC-29 stream keeps these tables in its OWN database, which is what allows inbox admission, the timeline.events insert and the checkpoint advance to commit in one transaction. Reliability migrations belong only in eventing.schema_migrations; recording one in timeline.schema_migrations would mean one runner had claimed both schemas. The current Timeline source chain runs 000 through 009, including the audit and event retention anchors, three comment-only retention-class migrations and the audit ingest-identity columns. Historical installations can also retain their two pre-consolidation ledger rows. EST-8 renamed the assembly to StellaOps.Timeline.Persistence while preserving Timeline.Consolidated as its persistent module identity and preserving every SQL leaf name and byte. Discovery remains prefix-free and bound by type.

Evidence compile edge ruled (Q-5, 2026-08-25), executed repo-side 2026-08-27: retire. Direction verification found the dormant ExportCenter TimelineEvidenceClient only at its own declaration, with no caller, registration, test, reflection lookup, configuration or runtime entrypoint. MBI-5/025 deleted that wrapper and exactly three TimelineIndexer.Core compile edges. Timeline’s /api/v1/timeline/{eventId}/evidence owner endpoint remains; the separate ExportCenter HTTP audit source and timeline event-publication paths also remain. A cross-service evidence-read seam is designed only when a real consumer exists. Receipt: SPRINT_20260730_001 Decisions & Risks and the W3-01/025/026 execution logs.

Three things about the live estate that a reader will otherwise get wrong (re-measured 2026-09-11 at W3-01 close, SPRINT_20260722_020).

1. The M1/M2/M3 code half is deployed. The 2026-08-24 window replaced the 2026-07-24 image with one built from a clean HEAD worktree. stellaops-timeline-web resolves the canonical STELLAOPS_POSTGRES_TIMELINE_CONNECTION and answers GET /doctor/checks with 200 doctor-check/v1— six checks, 6/6 healthy at 2026-09-11.
2. The consolidated forward chain is what converges this database. timeline.schema_migrations holds five rows: the two legacy ones applied 2026-08-14 08:20, plus 000_v1_partition_existing_unified_audit_events.sql (5,190 ms), 001_v1_timeline_consolidated_baseline.sql and 002_v1_timeline_partition_maintenance.sql, all applied 2026-08-24 08:03. timeline.unified_audit_events is relkind='p'with monthly children 2026_06…2026_12 plus DEFAULT and 452,007 rows, so retention is partition maintenance rather than a per-row DELETE sweep. The bridge’s temporary unified_audit_events_heap_rollback (104 MB / 77,650 rows) was dropped 2026-09-11 under W3-01’s recorded approval, after an EXCEPT parity check against the partitioned parent returned zero.
3. The historical corpus is not in this database, and it is no longer recoverable. PTC-4/Window B (2026-08-14) moved Timeline from a dedicated cluster onto the shared server by fresh convergence, and owner ruling C deliberately did not carry the audit events, so the live store holds events from 2026-08-14 onward. The pre-move corpus was ABANDONED IN PLACE under that ruling: stellaops_platform.timeline was snapshotted and dropped 2026-08-24, and the two pre-move storage surfaces are both gone — the stopped cluster’s volume compose_timeline-postgres-data is absent from the Docker volume list (re-checked 2026-09-11) and the window snapshot lived under swept tmp/. Read this as a closed decision, not as an available rollback.

How the schema converges

Everything lives under the timeline PostgreSQL schema. The Timeline host has one startup migration authority: StellaOps.Timeline.Persistence, wired by AddConsolidatedTimelinePersistence against the fail-closed owner connection. Its embedded resources run lexically as one forward chain:

• 000_v1_partition_existing_unified_audit_events.sql— no-op on a fresh database or an already-partitioned parent; transactionally converts an existing legacy heap, retaining the byte/row-identical original as unified_audit_events_heap_rollback. It refuses unsupported dependencies, target-name collisions, and duplicate tenant sequence positions.
• 001_v1_timeline_consolidated_baseline.sql— converges the complete union of the legacy Timeline Core and TimelineIndexer schema, including the partitioned audit parent and DEFAULT child.
• 002_v1_timeline_partition_maintenance.sql— replaces the rolling-window function and adds empty-expired-partition pruning. Existing monthly names are accepted only when attached to the correct parent with the exact expected bounds.
• 003_v1_timeline_eventing_adoption.sql— adopts the Timeline Eventing store.
• 004_v1_timeline_audit_retention_anchor.sql— pins the existing empty-partition reclamation boundary and audit retention classification without changing its window.
• 005_v1_timeline_event_retention.sql— adds the 365-day event/detail window while retaining every event with source-of-truth digest linkage.
• 006_v1_timeline_retention_classes.sql— comment-only; restates four source-of-truth headers with the disposal boundary their writers imply.
• 007_v1_timeline_anomaly_retention_classes.sql— comment-only; classifies the anomaly baseline (derived-rebuildable), rules, alerts and event digests (source-of-truth) from their writers. The alerts were labelled derived-rebuildable in the baseline and are source-of-truth: their acknowledgement and dispatch state cannot be recomputed from events and rules.
• 008_v1_timeline_heap_bridge_copy_retention_class.sql— comment-only and guarded; a no-op unless unified_audit_events_heap_rollback exists. It classifies the heap 000 keeps as source-of-truth: nothing reclaims its rows, and only the operator disposal step described below removes it. 000’s own append-audit label claimed a retention window that no routine enforces. The file name avoids the word “rollback”, because the migration runner reads any such name as a Release migration and startup convergence would stop there.
• 009_v1_timeline_audit_ingest_identity.sql— PCD-888. Adds the nullable ingest_subject, ingest_client_id and ingest_producer columns to the partitioned unified_audit_events parent, which propagates them to every partition, DEFAULT included, as a catalog-only change. Rows written before it stay NULL; nothing is backfilled. The heap that 000 keeps is not altered, so a parity comparison against it uses the heap’s own column list.

On an existing database, the two legacy 001 ledger rows may already exist; they do not collide with this chain’s resource names. The 000 bridge deliberately sorts before consolidated 001, and a second convergence is a no-op. Program.cs no longer registers the two legacy AddStartupMigrations calls and removes TimelineIndexerMigrationHostedService, so the owning host cannot race multiple runners over the same schema. Operational procedure and the bounded rollback-heap disposal gate are in the partition cutover runbook.

Migration-filename note (2026-07-12). The Timeline.Core chain formerly numbered 002–010 was collapsed into 001_v1_timeline_core_baseline.sql; those files now exist only under Migrations/_archived/pre_1.0/mig061/ and are excluded from embedding. The parenthetical “(from NNN_*)” markers below are provenance of the folded-in DDL, not live filenames. The TimelineIndexer side is different: its 001_initial_schema.sql is still embedded in StellaOps.TimelineIndexer.Infrastructure, but timeline-web removes TimelineIndexerMigrationHostedService, so the host never runs it; the consolidated baseline 001_v1_timeline_consolidated_baseline.sql creates the indexer tables.

Unified audit sink: timeline.unified_audit_events

The platform-wide tamper-evident audit log. It is range-partitioned monthly on timestamp, with a DEFAULT partition for timestamps outside the rolling window. The physical primary key is (id, tenant_id, timestamp). The supported writer retains the public logical identity (tenant_id, id) under the per-tenant sequence lock and treats timestamp as immutable; direct SQL is not a supported ingest path.

Supporting objects: timeline.unified_audit_sequences (per-tenant chain head + next_unified_audit_sequence / update_unified_audit_sequence_hash functions) and verify_unified_audit_chain(tenant, from_seq, to_seq) which backs GET /api/v1/audit/chain/verify.

Retention + GDPR (baseline line 326; folded in from the archived 005_audit_data_classification_retention.sql)

Durable operation state (folded in from the archived 006_unified_audit_operation_state.sql)

Anomaly v2 (baseline lines 538-640; from 007–010)

Indexer tables: HLC timeline (created by the consolidated baseline)

HLC event store (Eventing)

The HLC query path (/api/v1/timeline/hlc/*) reads through the StellaOps.Eventing infrastructure (ITimelineEventStore). Runtime Eventing registration is PostgreSQL-only: AddStellaOpsEventing(IConfiguration) registers PostgresTimelineEventStore plus PostgresHlcStateStore. Test harnesses that need volatile storage must call AddStellaOpsEventingInMemory(...) explicitly; setting Eventing:UseInMemoryStore=true on the runtime configuration path fails closed.

Ownership (2026-09-14, SPRINT_20260911_001 TLC-1 owner ruling): the store’s tables — timeline.events and timeline.hlc_state — are timeline-web’s OWN, created by its consolidated baseline’s 003_v1_timeline_eventing_adoption.sql in stellaops_timeline (Startup band). Before that ruling no deployed database held them and the aggregate /health answered 500. Eventing:ConnectionString is derived from the owner connection when unset and the host fails closed if it names another database; timeline.outbox was dead code and exists nowhere (the P6 eventing.outbox is the one outbox). Producers such as findings-web emit finding.disposition.changed onto their own P6 outbox (findings.dispositions stream).

The findings.dispositions consumer (DC-29, 2026-09-15)

StellaOps.Timeline.DispositionProjection folds that stream into this host’s own timeline.events ledger. FindingDispositionProjectionService polls Findings’ retained catch-up feed (GET /api/findings/v1/ledger/dispositions/events) from the durable checkpoint and reports its committed cursor back (POST /api/findings/v1/ledger/dispositions/consumers/{id}) so the producer’s retention floor never advances past what this host has stored. One lease per tenant (timeline.finding-dispositions.projector:<tenant stream>) serializes the writer; the consumer id is timeline.finding-dispositions:<tenant stream>.

FindingDispositionTimelineConsumer.ApplyAsync is ScanCompletedProjectionConsumer.ApplyAsync in a different namespace: the checkpoint is read outside the transaction, then lease fencing, IInboxConsumer.TryAdmitAsync, the timeline.events insert and AdvanceCheckpointAsync commit together. A gap rolls back and leaves the checkpoint alone; an epoch change rolls back and is answered by ResetForEpochAsync, which discards timeline.events WHERE service = 'findings-ledger' AND payload->>'tenant_id' = <tenant> and replays the retained window from sequence 1. An unknown type or a structurally invalid payload is admitted and checkpointed with no projection, so one unprojectable row cannot stall a tenant; a payload whose tenant disagrees with the authenticated feed tenant throws and stops the poll fail-closed.

Each projected row is indistinguishable from one TimelineEventEmitter would have written: correlation_id is the finding id, service is findings-ledger, kind is EMIT, ts_wall is the producer’s occurredAt, the payload is the same snake_case canonical JSON, and the event id is the documented SHA-256(correlation_id || t_hlc || service || kind)[0:32]. The HLC timestamp comes from this host’s own PostgresHlcStateStore. The tick happens inside ApplyAsync but outside the destination transaction, because IHybridLogicalClock.Tick() persists its node row fire-and-forget on the clock’s own connection and cannot enlist in a caller transaction; it is gated on InboxAdmission.Admitted so a redelivery consumes no tick.

Verified live 2026-09-15 (docs/implplan/_evidence/20260915-tlc1-consumer-window/): a VEX trust override written through the gateway produced two eventing.outbox rows in stellaops_findings, which the consumer admitted, projected into timeline.events and checkpointed at sequence 2, and GET /api/v1/timeline/hlc/{findingId} returned both EMIT events with their HLC stamps. The producer’s eventing.remote_stream_consumers row carries the reported cursor, so retention prunes behind a real reader. Twenty-eight minutes of ten-second polling added no duplicate row.

Dependencies

Configuration

Endpoints

Scopes: timeline:read, timeline:write, timeline:admin (mapped to the Timeline.Read/Timeline.Write/Timeline.Admin policies), plus the narrow audit:ingest machine scope accepted by the ingest policy. Interactive callers authenticate with an Authority JWT. Router-forwarded internal callers use a verified, short-lived signed identity envelope. Direct service HTTP callers use an Authority-issued bearer token: ExportCenter’s audit-bundle reader requests only tenant-bound timeline:read, while emitters use tenant-bound audit:ingest. Neither path may derive tenant identity from a raw header.

HLC timeline — /api/v1/timeline/hlc/*

Indexer event queries — /api/v1/timeline/* (and bare /timeline/*)

Unified audit — /api/v1/audit/*

Console views of event history

Verified against source on 2026-10-09 (CCP-034). The Console’s event-history view is Audit Trail (/ops/operations/audit): its event, timeline-search and correlation tabs read the unified audit API above and require timeline:read. The Console’s Event Stream page (/ops/operations/event-stream) holds no event-bus connection and sends no request to this service. It states that live streaming and replay are not available in the Console, and it links to Audit Trail only when the session can open it. A bounded run snapshot on an Operations page is not a stream and not a historical audit.

Health

Aggregate /health therefore reports on the event STORE, not on the DC-29 consumer. A projection that is disabled, unauthorized or simply behind leaves /health at 200, by design: the store is readable either way, and a consumer lag is not a reason to fail a liveness probe. Diagnose the consumer instead from eventing.consumer_checkpoints in stellaops_timeline (its cursor per tenant stream) and from the host’s poll-failure warnings, which name the exception TYPE only — the poll runs on a client-credentials client whose exception graph can carry bearer material (.memory/standards/LOGGING_HYGIENE.md §1), so the type is what an operator gets and the rest is deliberately not logged. A poll that stops on a gap, an epoch mismatch or a retention horizon ahead of the checkpoint leaves the checkpoint untouched and retries on the next interval.

Observability

Metrics and traces are emitted from the StellaOps.Timeline meter / ActivitySource (both version 1.0.0), defined in TimelineMetrics (StellaOps.Timeline.Core/Telemetry/TimelineMetrics.cs).

Metrics

event_count_bucket is bucketed as 1-10 / 11-100 / 101-1000 / 1001-10000 / 10000+.

Traces

Security Considerations

• Authentication: Interactive endpoints accept a valid JWT issued by Authority. Internal audit reads and ingest calls may instead use a verified signed service identity envelope; callers fail closed when the shared envelope key is absent. Authorization outcomes for timeline read/write are logged via TimelineAuthorizationAuditSink (IAuthEventSink).
• Scope model: timeline:read for tenant-scoped reads, including effective retention-policy queries; narrow audit:ingest for service emitters (timeline:write is also accepted by that endpoint); timeline:write for acknowledge, export, and replay/export initiation; timeline:admin for anomaly-rule CRUD, the Notify dispatch worklist, and GDPR PII redaction.
• Tenant isolation: Queries, unified audit rows, anomaly state, and export status are scoped to the authenticated tenant; cross-tenant access is prohibited. The structural installation tenant holds platform-level audit events; every /api/v1/audit route except ingest answers 403 installation_tenant_admin_only for a caller in that tenant who is not a global administrator (stellaops:admin=true), who reaches it by selecting X-StellaOps-TenantId: installation. Indexer tables additionally enforce PostgreSQL Row-Level Security keyed on app.current_tenant.
• Tamper evidence: timeline.unified_audit_events is an append-only, monthly-partitioned log with a per-tenant SHA-256 hash chain (content_hash → previous_entry_hash, monotonic sequence_number), written under SERIALIZABLE isolation. Retryable PostgreSQL concurrency aborts repeat the complete transaction; duplicate deliveries with the exact immutable timestamp roll back the allocated sequence, while a shifted-timestamp duplicate ID fails. GET /api/v1/audit/chain/verify detects breaks. GDPR redaction preserves actor_id and the ingest_* writer columns so the chain stays verifiable.
• Data classification + retention: Events are classified (none/personal/sensitive/restricted); a scheduled purge enforces per-classification retention windows and honours compliance_hold (legal hold). Dry-run issues no schema maintenance. Empty old partitions are reclaimed only after row-level policy makes them empty.
• Ingest surface: POST /api/v1/audit/ingest accepts service-to-service events under narrow audit:ingest or operator timeline:write. Only audit:ingest may name the event tenant: Authority grants it to no human role and no interactive client, and the shared emitter signs the event tenant into its service identity envelope. A timeline:write caller (the Console client holds that scope) is bound to its own authenticated tenant, so it cannot append to another tenant’s hash chain (403 tenant_claim_mismatch). A producer’s event without a tenant joins the reserved installation tenant’s own chain, which only global administrators can read and only producers can append to (403 tenant_reserved for anyone else); any other request without a tenant is refused (400 tenant_missing). Events are never placed in the default tenant’s chain. Every stored event records its verified writer (ingestIdentity), which is part of the hashed content, so an actor a caller supplied is always attributable to the identity that wrote it. Only an audit:ingest producer may record another actor; a caller without it records only itself (403 actor_claim_mismatch). Resource and description stay caller-supplied and are attributable through the same identity. Persistence failures return 503 so emitters can retry without data loss.
• Export controls: Exported event payloads may contain sensitive operational data; exports are tracked in durable per-tenant state.
• Replay determinism: Replay operations produce identical output given identical input sequences, supporting audit and compliance verification. Production replay/export endpoints fail closed (501) when only process-local implementations are available.

TimelineIndexer (Event Ingestion and Indexing)

TimelineIndexer was consolidated into the Timeline module (Sprint 210, 2026-03-04). It provides the write/ingestion side of the CQRS pattern while Timeline provides the read/query side. Both share the same schema domain and live under src/Timeline/.

TimelineIndexer Responsibilities

• Event ingestion: Consumes events from NATS/Redis message bus via configurable subscribers.
• HLC timestamping: Assigns Hybrid Logical Clock timestamps to establish causal ordering.
• Event indexing: Writes indexed events to PostgreSQL via EfCore (compiled model preserved for migration identity).
• Authorization audit: Provides audit sink for authorization events.

Ingestion Transport Contract

• Non-testing Timeline hosts must enable at least one real ingestion transport: Ingestion:Nats:Enabled=true or Ingestion:Redis:Enabled=true.
• NullTimelineEventSubscriber is a testing-only harness and is not registered in live hosts.
• If both transports are disabled outside Testing, startup fails fast with a configuration error instead of exposing an idle ingestion worker.

Deployable Services

• Timeline WebService (StellaOps.Timeline.WebService): the single live deployable. It hosts the unified audit sink, the HLC query/analysis/export/replay API, the indexer query/evidence endpoints, the in-process ingestion worker (NATS/Redis), the retention purge service, and the anomaly v2 hosted services.
• The local Compose topology assigns this unified deployable the resources-medium tier (0.50 CPU, 1 GiB memory). The lighter 512 MiB tier is insufficient for its combined request, ingestion, retention, and anomaly-rollup responsibilities under concurrent audit reads.
• TimelineIndexer WebService (StellaOps.TimelineIndexer.WebService) and TimelineIndexer Worker (StellaOps.TimelineIndexer.Worker): archived 2026-08-10 to src/__Obsoleted/Timeline/ under the W3-01 host disposition. Their ingestion/query logic had already been merged into StellaOps.Timeline.WebService, which runs the ingestion worker in-process (AddTimelineIngestionRuntime); neither host was ever deployed — no publish key, no compose service. Each carries an AGENTS.md naming the ruling and the replacement.

The merged design keeps ingestion and query co-hosted while sharing domain libraries under a single module boundary. Scaling ingestion out again would mean a new host over the same live TimelineIndexer.Core/.Infrastructure libraries, not un-freezing the archived shells.