Identity Watchlist Monitoring Runbook

Audience: Security operations and platform teams running the Attestor.

This runbook covers operational procedures for the Stella Ops identity watchlist monitoring system — watchlist authoring, alert triage, performance tuning, deduplication maintenance, air-gap operation, and disaster recovery. Read the implementation-status note below first: the CRUD/test API is live today, but the background scan-and-alert monitor is defined but not yet wired in a default deployment, and several procedures here describe that target pipeline.

Implementation status (verified against src/Attestor, 2026-05-30). Two parts of this feature have very different maturity:

• Live and wired: watchlist CRUD (create/list/get/update/delete), the POST /api/v1/watchlist/{id}/test pattern tester, the matching engine, the PostgreSQL schema, the dedup table, and the attestor.watchlist.* meter definitions. The REST API and the stella watchlist CLI exercise these end-to-end.
• Defined but NOT wired in production: the background monitor that scans new Attestor entries, matches them, and emits alerts. In the Attestor WebService the monitor’s collaborators default to no-op stubs — IAttestorEntrySource resolves to DisabledAttestorEntrySource (throws if invoked) and IIdentityAlertPublisher resolves to DisabledIdentityAlertPublisher — and AddWatchlistMonitorBackgroundService() is not called at composition time (AttestorWebServiceComposition.cs). As a result, no entries are scanned, no alerts are emitted, and the attestor.watchlist.* counters stay at zero in a default deployment. The GET /api/v1/watchlist/alerts endpoint is also a stub that always returns an empty list (WatchlistEndpoints.cs, ListWatchlistAlerts carries a // TODO).

The triage, performance-tuning, and disaster-recovery procedures below describe the target monitoring pipeline. They become operational only once a durable IAttestorEntrySource, a Notify-backed IIdentityAlertPublisher, an alert-history store, and the background hosted service are registered. Sections that depend on the live monitor are flagged inline.

Service Overview

The identity watchlist monitor is designed as a background service that:

1. Monitors new Attestor entries in real-time (ChangeFeed mode via PostgreSQL LISTEN/NOTIFY) or via polling
2. Matches signer identities against configured watchlist patterns
3. Emits alerts through the notification system
4. Applies deduplication to prevent alert storms

The CRUD API and pattern tester are live today; the scan/match/emit loop above is not yet registered in the Attestor WebService (see the status note above).

Configuration

Bound from the Attestor:Watchlist section into WatchlistMonitorOptions (WatchlistMonitorOptions.cs). Default values shown.

{
  "Attestor": {
    "Watchlist": {
      "Enabled": true,
      "Mode": "ChangeFeed",
      "PollingInterval": "00:00:05",
      "MaxEventsPerSecond": 100,
      "DefaultDedupWindowMinutes": 60,
      "RegexTimeoutMs": 100,
      "MaxWatchlistEntriesPerTenant": 1000,
      "PatternCacheSize": 1000,
      "InitialDelay": "00:00:10",
      "NotifyChannelName": "attestor_entries_inserted"
    }
  }
}

• Mode: ChangeFeed (real-time, PostgreSQL LISTEN/NOTIFY) or Polling (use for air-gap / where LISTEN/NOTIFY is unavailable).
• PollingInterval / InitialDelay are TimeSpan strings (hh:mm:ss); the defaults are 5s and 10s.
• NotifyChannelName is the PostgreSQL channel used in ChangeFeed mode (default attestor_entries_inserted).

Authorization

The watchlist REST API and stella watchlist CLI are gated by the canonical trust scope family (AttestorWebServiceComposition.cs). The watchlist:* / watchlist.* forms are accepted as legacy aliases for older clients; new integrations should use the trust scopes.

Notes:

• Creating or changing the scope of a Global or System entry requires admin (trust:admin/admin role); non-admins are returned 403.
• System-scoped entries cannot be deleted by anyone via the API.
• There is no watchlist:* entry in the canonical scope catalog (StellaOpsScopes.cs) — only trust:read, trust:write, and trust:admin exist there. The watchlist:* names are ASP.NET policy names and legacy aliases, not registered scopes.

Alert Triage Procedures

The triage procedures below assume the live monitor and notification delivery are wired (see the status note at the top). In a default deployment no alerts are emitted, so these steps are forward-looking.

Critical Severity Alert

Response Time: Immediate (< 15 minutes)

1. Acknowledge the alert in your incident management system
2. Verify the matched identity in Rekor:

   rekor-cli get --uuid <rekor-uuid>
   

3. Determine impact:
   • What artifact was signed?
   • Is this a known/expected signer?
   • What systems consume this artifact?
4. Escalate if malicious activity is confirmed
5. Document findings in incident record

Warning Severity Alert

Response Time: Within 1 hour

1. Review the alert details
2. Check context:
   • Is this a new legitimate workflow?
   • Is the pattern too broad?
3. Adjust watchlist entry if needed:

   stella watchlist update <id> --severity info
   # or
   stella watchlist update <id> --enabled false
   

4. Document decision rationale

Info Severity Alert

Response Time: Next business day

1. Review for patterns or trends
2. Consider if alert should be disabled or tuned
3. Archive after review

Performance Tuning

High Scan Latency

Symptom: attestor.watchlist.scan_latency_seconds > 10ms

Investigation:

1. Check pattern cache hit rate:

   SELECT COUNT(*) FROM attestor.identity_watchlist WHERE enabled = true;
   

2. Review regex patterns for complexity
3. Check tenant watchlist count

Resolution:

• Increase PatternCacheSize if cache misses are high
• Simplify complex regex patterns
• Consider splitting overly broad patterns

High Alert Volume

Symptom: attestor.watchlist.alerts_emitted_total growing rapidly

Investigation:

1. Identify top-triggering entries:

   stella watchlist alerts --since 1h --format json | jq 'group_by(.watchlistEntryId) | map({id: .[0].watchlistEntryId, count: length}) | sort_by(-.count)'
   

   Note: stella watchlist alerts (and the underlying GET /api/v1/watchlist/alerts) currently returns an empty list — alert-history retrieval is a stub (// TODO in WatchlistEndpoints.ListWatchlistAlerts). Until an alert-history store is implemented, identify hot entries from the dedup table instead (see Check Dedup Effectiveness) or from the attestor.watchlist.* metric labels. Both of those workarounds also depend on the live monitor: attestor.identity_alert_dedup is written only by IdentityMonitorService.ProcessMatchAsync (via IAlertDedupRepository.CheckAndUpdateAsync), and the metrics are emitted only by the monitor — so in a default deployment both stay empty/zero.

2. Check if pattern is too broad

Resolution:

• Narrow pattern scope
• Increase dedup window
• Reduce severity if appropriate

Database Performance

Symptom: Slow list/match queries

Investigation:

EXPLAIN ANALYZE
SELECT * FROM attestor.identity_watchlist
WHERE enabled = true AND (tenant_id = 'tenant-1' OR scope IN ('Global', 'System'));

Resolution:

• Verify indexes exist:

  SELECT indexname FROM pg_indexes WHERE tablename = 'identity_watchlist';
  

• Run VACUUM ANALYZE if needed
• Consider partitioning for large deployments

Deduplication Table Maintenance

The attestor.identity_alert_dedup table is created by the schema migration (live today) but is written only by the background monitor (IAlertDedupRepository.CheckAndUpdateAsync, called from IdentityMonitorService.ProcessMatchAsync). Until the monitor is wired (see the status note at the top), this table stays empty, so the cleanup and effectiveness queries below are forward-looking.

Cleanup Expired Records

Run periodically (daily recommended):

DELETE FROM attestor.identity_alert_dedup
WHERE last_alert_at < NOW() - INTERVAL '7 days';

Check Dedup Effectiveness

SELECT
  watchlist_id,
  COUNT(*) as suppressed_identities,
  SUM(alert_count) as total_suppressions
FROM attestor.identity_alert_dedup
GROUP BY watchlist_id
ORDER BY total_suppressions DESC
LIMIT 10;

Air-Gap Operation

For environments without network access to PostgreSQL LISTEN/NOTIFY:

1. Set Mode: Polling in configuration
2. Adjust PollingInterval based on acceptable delay (default: 5s)
3. Ensure sufficient database connection pool size
4. Monitor for missed entries during polling gaps

Disaster Recovery

Service Restart

Applies once the background monitor is wired (see status note at the top). Behaviour described here is from IdentityMonitorBackgroundService (Execute.cs/Polling.cs).

1. After startup the monitor waits InitialDelay (default 10s) before processing begins.
2. There is no persisted checkpoint. In Polling mode the cursor (lastPolledAt) is initialized to the current wall-clock time at startup and advances only while the service runs; in ChangeFeed mode the service consumes a live LISTEN/NOTIFY stream with no replay. Each AttestorEntryInfo carries an IntegratedTimeUtc, but it is not used as a resumable cursor.
3. Consequence: entries inserted while the service was down — including during the InitialDelay window — are not caught up and may be missed, not replayed. Duplicate alerts on restart are therefore unlikely; the real recovery risk is dropped entries. To backfill after extended downtime, an operator must re-drive the affected entries through whatever durable IAttestorEntrySource is registered.

Database Failover

Applies once the background monitor is wired (see status note at the top).

1. Retry behaviour is mode-dependent. In Polling mode the monitor loop catches exceptions per cycle, logs a warning, and retries on the next PollingInterval (IdentityMonitorBackgroundService.Polling.cs). In ChangeFeed mode the registered code simply iterates the IAttestorEntrySource stream (Execute.cs, RunChangeFeedModeAsync) with no internal reconnect — durable reconnection on a dropped LISTEN/NOTIFY connection must be provided by the registered IAttestorEntrySource implementation, or the monitor must be restarted.
2. Pattern cache survives in-memory during brief outages (compiled patterns are cached up to PatternCacheSize).
3. Long outages may require service restart. Note the missed-entry caveat in Service Restart: there is no persisted checkpoint, so entries inserted while the service was unavailable are not replayed.

Watchlist Export/Import

Export:

stella watchlist list --include-global --format json > watchlist-backup.json

Import (manual):

# Process each entry and recreate
jq -c '.[]' watchlist-backup.json | while read entry; do
  # Extract fields and call stella watchlist add
done

Metrics Reference

Emitted by the StellaOps.Attestor.Watchlist meter (IdentityMonitorService.cs). All counters are produced only by the background monitor, so they remain at zero until the monitor is wired (see the status note at the top).

Escalation Contacts

Related Documents

• Identity Watchlist User Guide
• Attestor Architecture
• Notification System