Unified Search Operations Runbook

Audience: Operations and platform teams running AdvisoryAI unified search.

This runbook covers the AdvisoryAI unified search service end to end: setup, key endpoints and scopes, monitoring, performance tuning, feature-flag rollout, troubleshooting, and backup/recovery. Unified search federates findings, VEX, policy, graph, OpsMemory, timeline, and scanner corpora behind a single query API with weighted RRF fusion and optional synthesis.

Setup

  1. Configure AdvisoryAI:KnowledgeSearch:ConnectionString (section AdvisoryAI:KnowledgeSearch, bound to KnowledgeSearchOptions).

  2. Configure AdvisoryAI:UnifiedSearch options (section AdvisoryAI:UnifiedSearch, bound to UnifiedSearchOptions).

  3. For live compose/runtime, set AdvisoryAI:KnowledgeSearch:FindingsAdapterBaseUrl, ...:VexAdapterBaseUrl, and ...:PolicyAdapterBaseUrl together so findings, VEX, and policy ingest from live services instead of partial fallback snapshots. Each adapter also has a ...:FindingsAdapterEnabled / ...:VexAdapterEnabled / ...:PolicyAdapterEnabled toggle (default true). When live adapter base URLs are configured, the background refresh service auto-enables unified indexing even if AdvisoryAI:KnowledgeSearch:UnifiedAutoIndexEnabled is false.

    Point each base URL at the service that actually serves the adapter’s route. A base URL that resolves but does not host the route produces a 404 on every refresh, and the adapter answers with its committed snapshot, so the corpus looks healthy while it is a fixture. The targets:

    AdapterRoute it requestsService that serves itScope the caller needs
    FindingsGET /api/v2/security/findingsfindings-web (alias findings.stella-ops.local)findings:read
    PolicyGET /api/v1/authority/effective-policies?enabledOnly=false&includeExpired=true&limit=100policy-engine (alias policy-engine.stella-ops.local)policy:read
    VEXGET /api/v1/canonicalno service in the current estate — Concelier was decommissioned; the successor is undecidedvex:read

    The VEX row is accurate, not a typo: that adapter’s live path is structurally dead and it serves its snapshot. Treat a Snapshot reading for vex as expected until the successor is chosen; treat one for findings as a defect.

    policy has a second, benign way to read Snapshot, and the reason string is what tells them apart (SPRINT_20260722_013 AAI-17). A reason naming an HTTP status is a defect: the route was unreachable, unauthorised, or rejected the request. A reason of upstream returned zero mappable effective policies is not — the call succeeded and the tenant has no effective-policy bindings. Effective-policy bindings live only in the policy-engine process (EffectivePolicyService holds them in a ConcurrentDictionary; no Policy migration creates a table for them), so an estate that has created none through POST /api/v1/authority/effective-policies, or one whose policy-engine has restarted since, has an empty corpus by construction and the adapter serves its snapshot. Create at least one binding before reading Source=Upstream for policy as achievable.

  4. Ensure the published AdvisoryAI image carries the repo-shaped local corpus under /app:

    • findings, vex, policy snapshots come from AdvisoryAI:KnowledgeSearch:Unified{Findings,Vex,Policy}SnapshotPath (default src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/{findings,vex,policy}.snapshot.json).
    • graph, opsmemory, timeline, scanner snapshots come from AdvisoryAI:UnifiedSearch:Ingestion:{Graph,OpsMemory,Timeline,Scanner}SnapshotPath (default src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/{graph,opsmemory,timeline,scanner}.snapshot.json).

  5. Ensure model artifact path exists when AdvisoryAI:KnowledgeSearch:VectorEncoderType=onnx:

    • default AdvisoryAI:KnowledgeSearch:OnnxModelPath: models/all-MiniLM-L6-v2.onnx
    • When onnx is selected but the model file is missing, the encoder falls back to hash with a warning (no hard failure).

  6. Rebuild indexes when verifying live search quality (both require advisory-ai:admin scope):

    • POST /v1/advisory-ai/index/rebuild (legacy AdvisoryAI Knowledge Search — docs, OpenAPI, Doctor metadata).
    • POST /v1/search/index/rebuild (unified index across all registered ingestion adapters).

  7. Verify query endpoint:

    • POST /v1/search/query with tenant context and advisory-ai:operate scope (operate is also satisfied by advisory-ai:admin).

  8. Verify the corpus is live, not substituted. GET /v1/search/adapters (tenant context, advisory-ai:operate) reports, per adapter, whether the rows it last produced came from Upstream or Snapshot, the cause when it is not upstream, the target that was attempted, and — after a 404 — when the suppressed upstream call will be retried. SnapshotBackedCount above zero for anything but vex means an adapter is serving a fixture. The same list rides on the POST /v1/search/index/rebuild response, so an operator who triggers a rebuild sees the provenance without a second request.

Identity note: tenant, user (sub), and scopes are resolved from gateway-signed envelope claims only. Raw X-User-Id / X-Tenant-Id / X-StellaOps-Actor / X-StellaOps-Scopes / X-StellaOps-TenantId / X-Stella-Scopes / X-Scopes headers are stripped at inbound by InboundIdentityHeaderStripMiddleware (full HeadersToStrip list) and are not honored by the endpoints.

Key Endpoints

Unified search group (/v1/search, all under the advisory-ai rate-limit policy):

Analytics & history group (/v1/advisory-ai/search):

Feedback & quality group (/v1/advisory-ai/search):

Legacy AdvisoryAI Knowledge Search (pre-unified, still mounted under /v1/advisory-ai; group default scope advisory-ai:view):

Scope / policy model

AdvisoryAIPolicies defines three named policies enforced via RequireAuthorization:

advisory-ai:view, advisory-ai:operate, and advisory-ai:admin are in the canonical scope catalog (StellaOpsScopes). The synthesis gate scope search:synthesize is not in StellaOpsScopes — it is checked directly in UnifiedSearchEndpoints.HasSynthesisScope and must be granted to the caller’s token explicitly (or substituted by advisory-ai:admin).

Monitoring

Track per-tenant and global:

Performance Targets

The query-path P50 < 100ms / P95 < 500ms / P99 < 800ms numbers are the only targets backed by assertions in the test suite (UnifiedSearchPerformanceEnvelopeTests). Those tests are [Fact(Skip = …)] and run only in the dedicated perf lane, not in normal CI. The deterministic-synthesis and LLM-synthesis numbers are design targets, not currently asserted; the LLM total budget aligns with AdvisoryAI:KnowledgeSearch:SynthesisTimeoutMs (default 5000ms, which triggers template fallback when exceeded).

SQL Query Tuning and EXPLAIN Evidence

Unified search read paths rely on:

Recommended validation commands:

EXPLAIN (ANALYZE, BUFFERS)
SELECT c.chunk_id
FROM advisoryai.kb_chunk c
WHERE c.body_tsv_en @@ websearch_to_tsquery('english', @query)
ORDER BY ts_rank_cd(c.body_tsv_en, websearch_to_tsquery('english', @query), 32) DESC, c.chunk_id
LIMIT 20;

EXPLAIN (ANALYZE, BUFFERS)
SELECT c.chunk_id
FROM advisoryai.kb_chunk c
WHERE c.embedding_vec IS NOT NULL
ORDER BY c.embedding_vec <=> CAST(@query_vector AS vector), c.chunk_id
LIMIT 20;

Index expectations:

Automated EXPLAIN evidence is captured by:

Load and Capacity Envelope

Test envelope (in-process benchmark harness, UnifiedSearchPerformanceEnvelopeTests, perf-lane only — skipped in normal CI):

Sizing guidance:

Feature Flags and Rollout

Global enables (defaults in parentheses):

Per-tenant overrides — config path AdvisoryAI:UnifiedSearch:TenantFeatureFlags (nullable booleans; null = inherit global):

Ranking bias toggles (AdvisoryAI:KnowledgeSearch):

Background indexing and retention (AdvisoryAI:KnowledgeSearch):

Per-tenant feature-flag example:

{
  "AdvisoryAI": {
    "UnifiedSearch": {
      "TenantFeatureFlags": {
        "tenant-alpha": { "Enabled": true, "FederationEnabled": true, "SynthesisEnabled": false },
        "tenant-beta":  { "Enabled": true, "FederationEnabled": false, "SynthesisEnabled": false }
      }
    }
  }
}

Troubleshooting

Symptom: empty results

Symptom: poor semantic recall

Symptom: synthesis unavailable

Symptom: search feels self-serve weak

Symptom: high latency

Backup and Recovery

Validation Commands

# Fast PR-level quality gate
dotnet test src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj \
  -- --filter-class StellaOps.AdvisoryAI.Tests.UnifiedSearch.UnifiedSearchQualityBenchmarkFastSubsetTests

# Full benchmark + tuning evidence
dotnet test src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj \
  -- --filter-class StellaOps.AdvisoryAI.Tests.UnifiedSearch.UnifiedSearchQualityBenchmarkTests

# Performance envelope
dotnet test src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj \
  -- --filter-class StellaOps.AdvisoryAI.Tests.UnifiedSearch.UnifiedSearchPerformanceEnvelopeTests

# Self-serve telemetry and gap surfacing slice
dotnet build src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj -v minimal
src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/bin/Debug/net10.0/StellaOps.AdvisoryAI.Tests.exe \
  -method "StellaOps.AdvisoryAI.Tests.Integration.UnifiedSearchSprintIntegrationTests.G10_SelfServeMetrics_IncludeFallbackReformulationAndRescueSignals" \
  -method "StellaOps.AdvisoryAI.Tests.Integration.UnifiedSearchSprintIntegrationTests.G10_RecoveredFallbackSessions_DoNotCountAsAbandoned" \
  -reporter verbose -noColor