Unified Search Operations Runbook

Audience: Operations and platform teams running AdvisoryAI unified search.

This runbook covers the AdvisoryAI unified search service end to end: setup, key endpoints and scopes, monitoring, performance tuning, feature-flag rollout, troubleshooting, and backup/recovery. Unified search federates findings, VEX, policy, graph, OpsMemory, timeline, and scanner corpora behind a single query API with weighted RRF fusion and optional synthesis.

Setup

1. Configure AdvisoryAI:KnowledgeSearch:ConnectionString (section AdvisoryAI:KnowledgeSearch, bound to KnowledgeSearchOptions).
2. Configure AdvisoryAI:UnifiedSearch options (section AdvisoryAI:UnifiedSearch, bound to UnifiedSearchOptions).
3. For live compose/runtime, set AdvisoryAI:KnowledgeSearch:FindingsAdapterBaseUrl, ...:VexAdapterBaseUrl, and ...:PolicyAdapterBaseUrl together so findings, VEX, and policy ingest from live services instead of partial fallback snapshots. Each adapter also has a ...:FindingsAdapterEnabled / ...:VexAdapterEnabled / ...:PolicyAdapterEnabled toggle (default true). When live adapter base URLs are configured, the background refresh service auto-enables unified indexing even if AdvisoryAI:KnowledgeSearch:UnifiedAutoIndexEnabled is false.
4. Ensure the published AdvisoryAI image carries the repo-shaped local corpus under /app:
   • findings, vex, policy snapshots come from AdvisoryAI:KnowledgeSearch:Unified{Findings,Vex,Policy}SnapshotPath (default src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/{findings,vex,policy}.snapshot.json).
   • graph, opsmemory, timeline, scanner snapshots come from AdvisoryAI:UnifiedSearch:Ingestion:{Graph,OpsMemory,Timeline,Scanner}SnapshotPath (default src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/{graph,opsmemory,timeline,scanner}.snapshot.json).
5. Ensure model artifact path exists when AdvisoryAI:KnowledgeSearch:VectorEncoderType=onnx:
   • default AdvisoryAI:KnowledgeSearch:OnnxModelPath: models/all-MiniLM-L6-v2.onnx
   • When onnx is selected but the model file is missing, the encoder falls back to hash with a warning (no hard failure).
6. Rebuild indexes when verifying live search quality (both require advisory-ai:admin scope):
   • POST /v1/advisory-ai/index/rebuild (legacy AdvisoryAI Knowledge Search — docs, OpenAPI, Doctor metadata).
   • POST /v1/search/index/rebuild (unified index across all registered ingestion adapters).
7. Verify query endpoint:
   • POST /v1/search/query with tenant context and advisory-ai:operate scope (operate is also satisfied by advisory-ai:admin).

Identity note: tenant, user (sub), and scopes are resolved from gateway-signed envelope claims only. Raw X-User-Id / X-Tenant-Id / X-StellaOps-Actor / X-StellaOps-Scopes / X-StellaOps-TenantId / X-Stella-Scopes / X-Scopes headers are stripped at inbound by InboundIdentityHeaderStripMiddleware (full HeadersToStrip list) and are not honored by the endpoints.

Key Endpoints

Unified search group (/v1/search, all under the advisory-ai rate-limit policy):

• POST /v1/search/query — unified query with weighted RRF fusion and entity-grouped cards. Scope: advisory-ai:operate.
• POST /v1/search/suggestions/evaluate — preflights a bounded list of suggested queries against the active corpus (no analytics recorded). Scope: advisory-ai:operate.
• POST /v1/search/synthesize — SSE stream: deterministic synthesis first, then optional LLM synthesis. Scope: advisory-ai:operate plus the synthesis-specific scope search:synthesize (or advisory-ai:admin); requests without it return 403 { "error": "Missing required scope: search:synthesize" }.
• POST /v1/search/index/rebuild — full unified index rebuild across all ingestion adapters. Scope: advisory-ai:admin.

Analytics & history group (/v1/advisory-ai/search):

• POST /v1/advisory-ai/search/analytics — batch analytics events (max 100/request, fire-and-forget). Scope: advisory-ai:operate.
• GET /v1/advisory-ai/search/history — up to 50 recent queries for the current user. Scope: advisory-ai:view.
• DELETE /v1/advisory-ai/search/history — clear the user’s history. Scope: advisory-ai:operate.
• DELETE /v1/advisory-ai/search/history/{historyId} — delete one history entry (GUID id). Scope: advisory-ai:operate.

Feedback & quality group (/v1/advisory-ai/search):

• POST /v1/advisory-ai/search/feedback — submit helpful / not_helpful signal for a result. Scope: advisory-ai:view.
• GET /v1/advisory-ai/search/quality/metrics — aggregate quality metrics for a period (24h, 7d, 30d; default 7d). Scope: advisory-ai:admin.
• GET /v1/advisory-ai/search/quality/alerts — open quality alerts (filterable by status, alertType). Scope: advisory-ai:admin.
• PATCH /v1/advisory-ai/search/quality/alerts/{alertId} — acknowledge / resolve an alert. Scope: advisory-ai:admin.

Legacy AdvisoryAI Knowledge Search (pre-unified, still mounted under /v1/advisory-ai; group default scope advisory-ai:view):

• POST /v1/advisory-ai/search — legacy AKS query (docs/api/doctor). Scope: advisory-ai:operate. The web UnifiedSearchClient falls back to it when unified search fails (mapped result carries diagnostics.mode = legacy-fallback). Responses carry deprecation signalling headers set by ApplyLegacyKnowledgeSearchDeprecationHeaders: Deprecation: true, Sunset: 2026-04-30T00:00:00Z, Link: </v1/search/query>; rel="successor-version", and a Warning: 299 advising migration to /v1/search/query.
• POST /v1/advisory-ai/index/rebuild — legacy AKS index rebuild (returns documentCount, chunkCount, apiSpecCount, apiOperationCount, doctorProjectionCount, durationMs). Scope: advisory-ai:admin.

Scope / policy model

AdvisoryAIPolicies defines three named policies enforced via RequireAuthorization:

• advisory-ai:view — read-only (history read, feedback submit).
• advisory-ai:operate — query/operate (implies view at the policy level; operate is granted by holding advisory-ai:operate or advisory-ai:admin).
• advisory-ai:admin — index rebuild, quality dashboards, alert management (admin scope only).

advisory-ai:view, advisory-ai:operate, and advisory-ai:admin are in the canonical scope catalog (StellaOpsScopes). The synthesis gate scope search:synthesize is not in StellaOpsScopes — it is checked directly in UnifiedSearchEndpoints.HasSynthesisScope and must be granted to the caller’s token explicitly (or substituted by advisory-ai:admin).

Monitoring

Track per-tenant and global:

• Query throughput (query, click, zero_result, synthesis events)
• Self-serve journey signals (answer_frame, reformulation, rescue_action)
• P50/P95/P99 latency for /v1/search/query
• Zero-result rate
• Fallback answer rate, clarify rate, insufficient-evidence rate
• Reformulation count, rescue-action count, abandoned fallback count
• Synthesis quota denials (per-day cap Synthesis:SynthesisRequestsPerDay, default 200; concurrency cap Synthesis:MaxConcurrentPerTenant, default 10)
• Index size and rebuild duration (rebuild responses report domainCount, chunkCount, durationMs)
• Unified-query retrieval mode (diagnostics.mode is hybrid when the vector lane is used, otherwise fts-only; the web fallback path reports legacy-fallback)
• Active encoder: surfaced on the legacy /v1/advisory-ai/search response as diagnostics.activeEncoder. Possible values (from KnowledgeSearchService.ResolveActiveEncoderName): hash (deterministic SHA-256 selected), onnx (semantic ONNX inference active), onnx-fallback (ONNX encoder constructed but inference not active), hash-fallback (VectorEncoderType=onnx requested but a hash encoder is in use). The unified /v1/search/query diagnostics object does not carry an encoder field — use diagnostics.usedVector plus diagnostics.mode there.

Performance Targets

• Instant results: P50 < 100ms, P95 < 200ms, P99 < 300ms
• Full results (federated): P50 < 200ms, P95 < 500ms, P99 < 800ms
• Deterministic synthesis: P50 < 30ms, P95 < 50ms
• LLM synthesis: TTFB P50 < 1s, total P95 < 5s

The query-path P50 < 100ms / P95 < 500ms / P99 < 800ms numbers are the only targets backed by assertions in the test suite (UnifiedSearchPerformanceEnvelopeTests). Those tests are [Fact(Skip = …)] and run only in the dedicated perf lane, not in normal CI. The deterministic-synthesis and LLM-synthesis numbers are design targets, not currently asserted; the LLM total budget aligns with AdvisoryAI:KnowledgeSearch:SynthesisTimeoutMs (default 5000ms, which triggers template fallback when exceeded).

SQL Query Tuning and EXPLAIN Evidence

Unified search read paths rely on:

• FTS query over advisoryai.kb_chunk.body_tsv*
• Trigram fuzzy fallback (% / similarity())
• Vector nearest-neighbor (embedding_vec <=> query_vector)

Recommended validation commands:

EXPLAIN (ANALYZE, BUFFERS)
SELECT c.chunk_id
FROM advisoryai.kb_chunk c
WHERE c.body_tsv_en @@ websearch_to_tsquery('english', @query)
ORDER BY ts_rank_cd(c.body_tsv_en, websearch_to_tsquery('english', @query), 32) DESC, c.chunk_id
LIMIT 20;

EXPLAIN (ANALYZE, BUFFERS)
SELECT c.chunk_id
FROM advisoryai.kb_chunk c
WHERE c.embedding_vec IS NOT NULL
ORDER BY c.embedding_vec <=> CAST(@query_vector AS vector), c.chunk_id
LIMIT 20;

Index expectations:

• idx_kb_chunk_body_tsv_en (GIN over body_tsv_en)
• idx_kb_chunk_body_trgm (GIN trigram over body)
• idx_kb_chunk_embedding_vec_hnsw (HNSW over embedding_vec)

Automated EXPLAIN evidence is captured by:

• UnifiedSearchLiveAdapterIntegrationTests.PostgresKnowledgeSearchStore_ExplainAnalyze_ShowsIndexedSearchPlans

Load and Capacity Envelope

Test envelope (in-process benchmark harness, UnifiedSearchPerformanceEnvelopeTests, perf-lane only — skipped in normal CI):

• 50 concurrent requests over 300 total requests (IncludeSynthesis: false)
• Asserts P50 < 100ms, P95 < 500ms, P99 < 800ms; a separate test asserts ≤10% P95 regression vs a 120ms phase-1 baseline

Sizing guidance:

• Up to 100k chunks: 2 vCPU / 4 GB RAM
• 100k-500k chunks: 4 vCPU / 8 GB RAM

• 500k chunks or heavy synthesis: 8 vCPU / 16 GB RAM, split synthesis workers

Feature Flags and Rollout

Global enables (defaults in parentheses):

• AdvisoryAI:UnifiedSearch:Enabled (true) — master switch for the unified search service.
• AdvisoryAI:UnifiedSearch:Federation:Enabled (true) — federated console/graph/timeline fan-out.
• AdvisoryAI:UnifiedSearch:Synthesis:Enabled (true) — deterministic + LLM synthesis path.
• AdvisoryAI:UnifiedSearch:GravityBoost:Enabled (true) — graph-neighbor gravity boost.
• AdvisoryAI:KnowledgeSearch:LlmSynthesisEnabled (false) — gate LLM-grounded synthesis; also requires non-empty ...:LlmAdapterBaseUrl and ...:LlmProviderId.

Per-tenant overrides — config path AdvisoryAI:UnifiedSearch:TenantFeatureFlags (nullable booleans; null = inherit global):

• Enabled
• FederationEnabled
• SynthesisEnabled

Ranking bias toggles (AdvisoryAI:KnowledgeSearch):

• RoleBasedBiasEnabled (true) — when on, DomainWeightCalculator applies additive domain-weight bias from the requesting token’s scopes (e.g. scanner:read lifts findings).
• PopularityBoostEnabled (false) — opt-in click-through popularity boost (log2(1 + clickCount) * PopularityBoostWeight, weight default 0.05); off by default to preserve deterministic results.
• UnifiedFreshnessBoostEnabled (false) — opt-in recency boost during unified indexing.

Background indexing and retention (AdvisoryAI:KnowledgeSearch):

• UnifiedAutoIndexOnStartup (true) — rebuild the unified index on service start.
• UnifiedAutoIndexEnabled (false) — enable the periodic refresh loop (auto-enabled regardless when live adapter base URLs are configured).
• UnifiedIndexRefreshIntervalSeconds (300, min 30) — refresh cadence.
• SearchQualityMonitorEnabled (true) / SearchQualityMonitorIntervalSeconds (300) — periodic quality-alert refresh.
• SearchTelemetryEnabled (true) — analytics/feedback telemetry; retrieval, suggestions, and history stay functional when false.
• SearchAnalyticsRetentionEnabled (true) / SearchAnalyticsRetentionDays (90) / SearchAnalyticsRetentionIntervalSeconds (3600) — pruning of analytics/feedback/history.

Per-tenant feature-flag example:

{
  "AdvisoryAI": {
    "UnifiedSearch": {
      "TenantFeatureFlags": {
        "tenant-alpha": { "Enabled": true, "FederationEnabled": true, "SynthesisEnabled": false },
        "tenant-beta":  { "Enabled": true, "FederationEnabled": false, "SynthesisEnabled": false }
      }
    }
  }
}

Troubleshooting

Symptom: empty results

• Confirm the request carries a verified envelope with a tenant claim (stellaops:tenant / tenant / tenant_id); a missing tenant yields 400 with advisoryai.validation.tenant_required, not empty results.
• Verify AdvisoryAI:UnifiedSearch:Enabled and the per-tenant flag Enabled.
• Run an index rebuild (POST /v1/search/index/rebuild, admin scope) and check chunkCount in the response.
• If suggestions also fail, run both rebuild steps from Setup (legacy /v1/advisory-ai/index/rebuild then unified /v1/search/index/rebuild) and re-check with a known live query such as database connectivity.
• If only findings answer lanes work while VEX/policy/graph/OpsMemory remain corpus-unready, verify the published snapshot files exist (findings/vex/policy under the KnowledgeSearch snapshot paths; graph/opsmemory/timeline/scanner under the UnifiedSearch:Ingestion snapshot paths) and confirm the VEX/policy adapter base URLs are configured in runtime env.

Symptom: poor semantic recall

• Verify AdvisoryAI:KnowledgeSearch:VectorEncoderType and (on the legacy /v1/advisory-ai/search response) the diagnostics.activeEncoder value — an onnx-fallback or hash-fallback value means the ONNX path silently degraded to the hash encoder; on /v1/search/query confirm diagnostics.usedVector=true / diagnostics.mode=hybrid.
• Confirm the ONNX model path is accessible and valid — a missing model silently falls back to the hash encoder.
• Rebuild the index after switching encoder.

Symptom: synthesis unavailable

• Confirm the caller holds the search:synthesize scope (or advisory-ai:admin) in addition to advisory-ai:operate; otherwise the endpoint returns 403.
• Check Synthesis:Enabled (global) and the per-tenant SynthesisEnabled flag.
• For LLM synthesis specifically, confirm LlmSynthesisEnabled=true with non-empty LlmAdapterBaseUrl and LlmProviderId; otherwise only deterministic synthesis is emitted (llm_status SSE event reports unavailable).
• Check quota counters: SynthesisRequestsPerDay (per-day) and MaxConcurrentPerTenant (concurrency); exhaustion surfaces as llm_status: quota_exceeded.

Symptom: search feels self-serve weak

• Inspect GET /v1/advisory-ai/search/quality/metrics?period=7d.
• Watch fallbackAnswerRate, clarifyRate, insufficientRate, reformulationCount, rescueActionCount, and abandonedFallbackCount.
• Inspect GET /v1/advisory-ai/search/quality/alerts for fallback_loop and abandoned_fallback.
• Treat repeated fallback loops as ranking/context gaps; treat abandoned fallback sessions as UX/product gaps.

Symptom: high latency

• Check federated backend timeout budget.
• Review EXPLAIN (ANALYZE) plans.
• Verify index health and cardinality growth by tenant.

Backup and Recovery

• Unified index is derivable state.
• Recovery sequence:
  1. Restore primary domain systems (findings/vex/policy/docs sources).
  2. Restore AdvisoryAI DB schema.
  3. Trigger full index rebuild.
  4. Validate with quality benchmark fast subset.

Validation Commands

# Fast PR-level quality gate
dotnet test src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj \
  -- --filter-class StellaOps.AdvisoryAI.Tests.UnifiedSearch.UnifiedSearchQualityBenchmarkFastSubsetTests

# Full benchmark + tuning evidence
dotnet test src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj \
  -- --filter-class StellaOps.AdvisoryAI.Tests.UnifiedSearch.UnifiedSearchQualityBenchmarkTests

# Performance envelope
dotnet test src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj \
  -- --filter-class StellaOps.AdvisoryAI.Tests.UnifiedSearch.UnifiedSearchPerformanceEnvelopeTests

# Self-serve telemetry and gap surfacing slice
dotnet build src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj -v minimal
src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/bin/Debug/net10.0/StellaOps.AdvisoryAI.Tests.exe \
  -method "StellaOps.AdvisoryAI.Tests.Integration.UnifiedSearchSprintIntegrationTests.G10_SelfServeMetrics_IncludeFallbackReformulationAndRescueSignals" \
  -method "StellaOps.AdvisoryAI.Tests.Integration.UnifiedSearchSprintIntegrationTests.G10_RecoveredFallbackSessions_DoNotCountAsAbandoned" \
  -reporter verbose -noColor

Related Documentation

• Unified Search Architecture — ranking, fusion, and ingestion design
• AdvisoryAI Knowledge Search — the legacy AKS surface this service supersedes
• AdvisoryAI Architecture — service overview and module boundaries
• LLM Setup Guide — configuring the optional synthesis provider