Identity Watchlist Monitoring Runbook

Evidence host and route currency (source-verified 2026-10-07): Evidence API references in this document target evidence-web under /api/evidence/v1. Attestation policy attestor:write accepts attest:create; attestation read/verify policies accept attest:read or attest:create, with no trusted-network bypass. Source: src/Evidence/StellaOps.Evidence.WebService/Program.cs and its application mappings. Earlier dated operational observations remain historical; verify the deployed build before applying a procedure.

Audience: Security operations and platform teams running the Attestor.

This runbook covers operational procedures for the Stella Ops identity watchlist monitoring system — watchlist authoring, alert triage, performance tuning, deduplication maintenance, air-gap operation, and disaster recovery. Read the implementation-status note below first: the CRUD/test API is mapped in current source, but the background scan-and-alert monitor is defined but not yet wired in a default deployment, and several procedures here describe that target pipeline.

Implementation status (source-rechecked in src/Evidence, 2026-10-07). Two parts of this feature have very different maturity:

• Mapped in current source: watchlist CRUD (create/list/get/update/delete), the POST /api/evidence/v1/watchlist/{id}/test pattern tester, the matching engine, the PostgreSQL schema, the dedup table, and the attestor.watchlist.* meter definitions. CLI builds carrying PCD-449 call the current owner definition and pattern-test contracts with configured bearer/tenant authority. The CLI has no alert-history command. Source/in-process acceptance does not verify the deployed CLI or owner database.
• Defined but NOT wired in production: the background monitor that scans new Attestor entries, matches them, and emits alerts. In the Evidence web composition the monitor’s collaborators default to no-op stubs — IAttestorEntrySource resolves to DisabledAttestorEntrySource (throws if invoked) and IIdentityAlertPublisher resolves to DisabledIdentityAlertPublisher — and AddWatchlistMonitorBackgroundService() is not called at composition time (AttestorWebServiceComposition.cs). As a result, no entries are scanned, no alerts are emitted, and the attestor.watchlist.* counters stay at zero in a default deployment. No alert-history operation is mapped: CCP-036 (2026-10-09) withdrew the former GET /api/evidence/v1/watchlist/alerts handler, which returned a fixed empty list, and the Console Watchlist page states that history and monitoring are unavailable instead of showing zero alerts. WatchlistOwnerHttpContractTests pins the six definition and pattern-test routes and the disabled adapters.

The triage, performance-tuning, and disaster-recovery procedures below describe the target monitoring pipeline. They become operational only once a durable IAttestorEntrySource, a Notify-backed IIdentityAlertPublisher, an alert-history store, and the background hosted service are registered. Sections that depend on the live monitor are flagged inline.

Service Overview

The identity watchlist monitor is designed as a background service that:

1. Monitors new Attestor entries in real-time (ChangeFeed mode via PostgreSQL LISTEN/NOTIFY) or via polling
2. Matches signer identities against configured watchlist patterns
3. Emits alerts through the notification system
4. Applies deduplication to prevent alert storms

The CRUD API and pattern tester are mapped in current source; the scan/match/emit loop above is not yet registered in the Evidence web composition (see the status note above).

Configuration

Bound from the Attestor:Watchlist section into WatchlistMonitorOptions (WatchlistMonitorOptions.cs). Default values shown.

{
  "Attestor": {
    "Watchlist": {
      "Enabled": true,
      "Mode": "ChangeFeed",
      "PollingInterval": "00:00:05",
      "MaxEventsPerSecond": 100,
      "DefaultDedupWindowMinutes": 60,
      "RegexTimeoutMs": 100,
      "MaxWatchlistEntriesPerTenant": 1000,
      "PatternCacheSize": 1000,
      "InitialDelay": "00:00:10",
      "NotifyChannelName": "attestor_entries_inserted"
    }
  }
}

• Mode: ChangeFeed (real-time, PostgreSQL LISTEN/NOTIFY) or Polling (use for air-gap / where LISTEN/NOTIFY is unavailable).
• PollingInterval / InitialDelay are TimeSpan strings (hh:mm:ss); the defaults are 5s and 10s.
• NotifyChannelName is the PostgreSQL channel used in ChangeFeed mode (default attestor_entries_inserted).

Authorization

The mapped watchlist REST API is gated by the canonical trust scope family (AttestorWebServiceComposition.cs). The watchlist:* / watchlist.* forms are accepted as legacy aliases for older clients; new integrations should use the trust scopes.

Notes:

• Creating or changing the scope of a Global or System entry requires admin (trust:admin/admin role); non-admins are returned 403.
• System-scoped entries cannot be deleted by anyone via the API.
• There is no watchlist:* entry in the canonical scope catalog (StellaOpsScopes.cs) — only trust:read, trust:write, and trust:admin exist there. The watchlist:* names are ASP.NET policy names and legacy aliases, not registered scopes.

Current CLI availability

CLI builds carrying PCD-449 register supported definition CRUD and sample-pattern testing against the configured Evidence API. Requests carry operator bearer authorization and the resolved tenant; denial, unavailability and invalid results remain explicit non-zero outcomes. See the current definition guide. The CLI history command is withdrawn. The dated monitoring procedures below do not establish active monitoring, alert history or a backup capability; definition CRUD and pattern testing remain separate from the unregistered monitoring pipeline.

Historical target monitoring procedures (2026-05-30; current CLI guidance withdrawn 2026-10-07)

The following triage, tuning, cleanup, recovery and CLI examples preserve the earlier target pipeline. Their actual prerequisites and unsupported current CLI/monitor boundary are stated above; they are not current operator procedures.

Alert Triage Procedures

The triage procedures below assume the live monitor and notification delivery are wired (see the status note at the top). In a default deployment no alerts are emitted, so these steps are forward-looking.

Critical Severity Alert

Response Time: Immediate (< 15 minutes)

1. Acknowledge the alert in your incident management system
2. Verify the matched identity in Rekor:

   rekor-cli get --uuid <rekor-uuid>
   

3. Determine impact:
   • What artifact was signed?
   • Is this a known/expected signer?
   • What systems consume this artifact?
4. Escalate if malicious activity is confirmed
5. Document findings in incident record

Warning Severity Alert

Response Time: Within 1 hour

1. Review the alert details
2. Check context:
   • Is this a new legitimate workflow?
   • Is the pattern too broad?
3. Adjust watchlist entry if needed:

   stella watchlist update <id> --severity info
   # or
   stella watchlist update <id> --enabled false
   

4. Document decision rationale

Info Severity Alert

Response Time: Next business day

1. Review for patterns or trends
2. Consider if alert should be disabled or tuned
3. Archive after review

Performance Tuning

High Scan Latency

Symptom: attestor.watchlist.scan_latency_seconds > 10ms

Investigation:

1. Check pattern cache hit rate:

   SELECT COUNT(*) FROM attestor.identity_watchlist WHERE enabled = true;
   

2. Review regex patterns for complexity
3. Check tenant watchlist count

Resolution:

• Increase PatternCacheSize if cache misses are high
• Simplify complex regex patterns
• Consider splitting overly broad patterns

High Alert Volume

Symptom: attestor.watchlist.alerts_emitted_total growing rapidly

Investigation:

1. Identify top-triggering entries: Current source does not provide an alert-history operation. The PCD-449 CLI history command and the owner’s fixed-empty history route (CCP-036) are both withdrawn; a request to the former route returns 404. Deduplication rows and zero-valued monitoring counters are not a substitute history source. This target-pipeline investigation remains unavailable until the declared monitoring/history prerequisites are implemented and accepted.
2. Check if pattern is too broad

Resolution:

• Narrow pattern scope
• Increase dedup window
• Reduce severity if appropriate

Database Performance

Symptom: Slow list/match queries

Investigation:

EXPLAIN ANALYZE
SELECT * FROM attestor.identity_watchlist
WHERE enabled = true AND (tenant_id = 'tenant-1' OR scope IN ('Global', 'System'));

Resolution:

• Verify indexes exist:

  SELECT indexname FROM pg_indexes WHERE tablename = 'identity_watchlist';
  

• Run VACUUM ANALYZE if needed
• Consider partitioning for large deployments

Deduplication Table Maintenance

The attestor.identity_alert_dedup table is created by the schema migration (live today) but is written only by the background monitor (IAlertDedupRepository.CheckAndUpdateAsync, called from IdentityMonitorService.ProcessMatchAsync). Until the monitor is wired (see the status note at the top), this table stays empty, so the cleanup and effectiveness queries below are forward-looking.

Cleanup Expired Records

Run periodically (daily recommended):

DELETE FROM attestor.identity_alert_dedup
WHERE last_alert_at < NOW() - INTERVAL '7 days';

Check Dedup Effectiveness

SELECT
  watchlist_id,
  COUNT(*) as suppressed_identities,
  SUM(alert_count) as total_suppressions
FROM attestor.identity_alert_dedup
GROUP BY watchlist_id
ORDER BY total_suppressions DESC
LIMIT 10;

Air-Gap Operation

For environments without network access to PostgreSQL LISTEN/NOTIFY:

1. Set Mode: Polling in configuration
2. Adjust PollingInterval based on acceptable delay (default: 5s)
3. Ensure sufficient database connection pool size
4. Monitor for missed entries during polling gaps

Disaster Recovery

Service Restart

Applies once the background monitor is wired (see status note at the top). Behaviour described here is from IdentityMonitorBackgroundService (Execute.cs/Polling.cs).

1. After startup the monitor waits InitialDelay (default 10s) before processing begins.
2. There is no persisted checkpoint. In Polling mode the cursor (lastPolledAt) is initialized to the current wall-clock time at startup and advances only while the service runs; in ChangeFeed mode the service consumes a live LISTEN/NOTIFY stream with no replay. Each AttestorEntryInfo carries an IntegratedTimeUtc, but it is not used as a resumable cursor.
3. Consequence: entries inserted while the service was down — including during the InitialDelay window — are not caught up and may be missed, not replayed. Duplicate alerts on restart are therefore unlikely; the real recovery risk is dropped entries. To backfill after extended downtime, an operator must re-drive the affected entries through whatever durable IAttestorEntrySource is registered.

Database Failover

Applies once the background monitor is wired (see status note at the top).

1. Retry behaviour is mode-dependent. In Polling mode the monitor loop catches exceptions per cycle, logs a warning, and retries on the next PollingInterval (IdentityMonitorBackgroundService.Polling.cs). In ChangeFeed mode the registered code simply iterates the IAttestorEntrySource stream (Execute.cs, RunChangeFeedModeAsync) with no internal reconnect — durable reconnection on a dropped LISTEN/NOTIFY connection must be provided by the registered IAttestorEntrySource implementation, or the monitor must be restarted.
2. Pattern cache survives in-memory during brief outages (compiled patterns are cached up to PatternCacheSize).
3. Long outages may require service restart. Note the missed-entry caveat in Service Restart: there is no persisted checkpoint, so entries inserted while the service was unavailable are not replayed.

Watchlist Export/Import

Export:

stella watchlist list --include-global --format json > watchlist-backup.json

Import (manual):

# Process each entry and recreate
jq -c '.[]' watchlist-backup.json | while read entry; do
  # Extract fields and call stella watchlist add
done

Metrics Reference

Emitted by the StellaOps.Attestor.Watchlist meter (IdentityMonitorService.cs). All counters are produced only by the background monitor, so they remain at zero until the monitor is wired (see the status note at the top).

Escalation Contacts

Related Documents

• Identity Watchlist User Guide
• Attestor Architecture
• Notification System