Advisory AI Guardrails & Redaction Policy

Audience: Advisory AI guild, Security guild, Docs guild, operators consuming Advisory AI outputs. Scope: Prompt redaction rules, injection defenses, telemetry/alert wiring, and audit guidance for Advisory AI (Epic 8).

Related: Authority scopes (the advisory-ai:* scope tiers) · Authority audit events (authority.advisory_ai.remote_inference).

Advisory AI accepts structured evidence from Concelier/Excititor and assembles prompts before executing downstream inference. Guardrails enforce provenance, block injection attempts, and redact sensitive content prior to handing data to any inference provider (online or offline). This document enumerates the guardrail surface and how to observe, alert, and audit it.

1 · Input validation & injection defense

The IAdvisoryGuardrailPipeline (AdvisoryGuardrailPipeline.EvaluateAsync) redacts the prompt first (see §2), then evaluates the following checks in this order. Redaction always runs; the blocking checks are gated by AdvisoryGuardrailOptions:

1. Citation coverage – when RequireCitations is true (default) every prompt must carry at least one citation. A prompt with no citations raises citation_missing. Each citation is then validated: Index <= 0, or a missing DocumentId or ChunkId, raises citation_invalid (the check stops on the first invalid citation).
2. Prompt length – AdvisoryGuardrailOptions.MaxPromptLength defaults to 16 000 characters and is measured against the sanitized (post-redaction) prompt. Longer payloads raise prompt_too_long. A value <= 0 disables the check.
3. Blocked phrases – the guardrail pipeline lowercases the sanitized prompt and searches for the blocked-phrase cache. Built-in defaults: ignore previous instructions, disregard earlier instructions, you are now the system, override the system prompt, please jailbreak. Each match raises a prompt_injection violation (one per matched phrase); when any phrase matches, blocked_phrase_count metadata is set to the hit count.
4. Optional per-profile rules – additional blocked phrases supplied via AdvisoryAI:Guardrails:BlockedPhrases or a BlockedPhraseFile are merged with (not replacing) the defaults at startup, deduplicated case-insensitively (via a SortedSet over StringComparer.OrdinalIgnoreCase, so the effective list is sorted), and evaluated with the same logic. Unlike the allowlist file (see §2.5), the BlockedPhraseFile is JSON-only: it must be a JSON array of strings or a JSON object with a phrases array — a newline-delimited #-commented list is not accepted for blocked phrases. A relative path is resolved against the host content root; an absolute path that does not exist fails startup.

Not part of this pipeline — chat quotas. Per user/tenant token and rate budgets (requests/min, requests/day, tokens/day, tool calls/day) are enforced separately by AdvisoryChatQuotaService in the chat surface, not by the guardrail pipeline. Chat-quota overages do not raise a single generic quota_exceeded code; instead AdvisoryChatQuotaService.TryConsumeAsync returns one of four distinct denial codes — REQUESTS_PER_MINUTE_EXCEEDED, REQUESTS_PER_DAY_EXCEEDED, TOKENS_PER_DAY_EXCEEDED, and TOOL_CALLS_PER_DAY_EXCEEDED. (A quota_exceeded literal does exist, but only as an SSE llm_status value on the separate Unified Search synthesis path governed by SearchSynthesisQuotaService — it is unrelated to the guardrail pipeline and to chat quotas.) See §5 for the doctor endpoint that surfaces remaining budgets and the last denial.

Any failing blocking check above stops the pipeline before inference and emits guardrail_blocked = true in the persisted output plus the advisory_ai_guardrail_blocks_total counter. Note that not every recorded violation blocks: low citation coverage (the ratio of unique citations to structured chunks) is measured and emitted as a metric/trace tag but does not block on its own.

Endpoint authorization

The guardrail pipeline runs behind authenticated endpoints. The WebService gates inference/workflow routes (/v1/advisory-ai/pipeline*, /v1/advisory-ai/explain, /api/v1/chat/*) with three ASP.NET policies, each accepting the scope at its tier or above:

Gotcha: the policy names are dot-form (advisory-ai.operate) while the OAuth scope claim values are colon-form (advisory-ai:operate, defined in StellaOpsScopes.cs). Audit grants against the scope claim, not the policy name. The chat endpoints (/api/v1/chat/*, including /doctor) require advisory-ai.operate.

2 · Redaction rules

Redactions are deterministic so caches remain stable. The named regex rules run first (in the order below); the entropy sweep always runs last, after all named rules. Each named rule is skipped unless its trigger token (e.g. aws_secret_access_key, token/apikey/password, private key) is present in the input, so unrelated prompts incur no regex cost.

The entropy sweep only runs when EntropyThreshold > 0 and EntropyMinLength > 0, and it cheaply pre-checks for a candidate run of entropy-class characters before invoking the regex.

Redaction counts (named-rule replacements plus entropy replacements) are surfaced via guardrailResult.Metadata["redaction_count"] and emitted as log fields to simplify threat hunting. The post-redaction prompt length is recorded alongside it as guardrailResult.Metadata["prompt_length"].

Allowlist and entropy tuning

• Allowlist patterns bypass the entropy redaction for known-safe identifiers. The built-in defaults only cover content digests — sha256:, sha1:, sha384:, and sha512: hex strings. Other known-safe identifiers (scan IDs, evidence refs, etc.) are not allowlisted out of the box and must be added via AdvisoryAI:Guardrails:AllowlistPatterns or an AllowlistFile; configured patterns are merged with the defaults.
• Entropy thresholds are configurable to reduce false positives in long hex IDs (EntropyThreshold default 3.5, EntropyMinLength default 20). Setting either EntropyThreshold or EntropyMinLength to 0 disables the entropy sweep entirely (the configuration binder accepts any value >= 0); the named regex rules still run.
• Configure scrubber knobs via AdvisoryAI:Guardrails:EntropyThreshold, AdvisoryAI:Guardrails:EntropyMinLength, AdvisoryAI:Guardrails:AllowlistFile, and AdvisoryAI:Guardrails:AllowlistPatterns. The allowlist file may be a newline-delimited list (with # comments) or a JSON array / object with an allowlist/patterns array; a configured-but-missing file fails startup.

3 · Telemetry, logs, and traces

All metrics are emitted on the StellaOps.AdvisoryAI meter and tagged with task_type; some carry additional cache/citation tags as noted:

Logging

• Successful writes: the Stored advisory pipeline output {CacheKey} log line includes task, cache:{CacheHit} (the boolean plan cache-hit flag), guardrail_blocked, validation_failures, and citation_coverage.
• Guardrail rejection: a warning log records the task type, advisory key, and violation count.

Tracing

• Spans are emitted on the StellaOps.AdvisoryAI activity source.
• WebService (/v1/advisory-ai/pipeline*) emits advisory_ai.plan_request (single) and advisory_ai.plan_batch (batch) spans. The single-plan span tags advisory.task_type, advisory.advisory_key, and advisory.plan_cache_key; the batch span tags advisory.batch_size. (The guardrail/validation/citation tags are added downstream on the worker span, not here.)
• Worker emits an advisory_ai.process span per queue item, tagged with advisory.task_type, advisory.advisory_key, and advisory.plan_cache_hit. The executor enriches the in-scope activity with advisory.guardrail_blocked, advisory.validation_failures, and advisory.citation_coverage.
• Other WebService routes emit their own spans (e.g. advisory_ai.explain, advisory_ai.remediation_plan, advisory_ai.policy_*) on the same source.

4 · Dashboards & alerts

Update the “Advisory AI” Grafana board with the new metrics:

1. Latency panel – plot advisory_ai_latency_seconds p50/p95 split by plan_cache_hit. Alert when p95 > 30s for 5 minutes.
2. Guardrail burn rate – advisory_ai_guardrail_blocks_total vs. advisory_ai_validation_failures_total. Alert when either exceeds 5 blocks/min or 1% of total traffic.
3. Citation coverage – histogram heatmap of advisory_ai_citation_coverage_ratio to identify evidence gaps (alert when <0.6 for more than 10 minutes).

All alerts should route to #advisory-ai-ops with the tenant, task type, and recent advisory keys in the message template.

5 · Operations & audit

• When an alert fires: capture the guardrail log entry, relevant metrics sample, and the cached plan from the worker output store. Attach them to the incident timeline entry.
• Tenant overrides: any request to loosen guardrails or blocked phrase lists requires a signed change request and security approval. Update AdvisoryGuardrailOptions via configuration bundles and document the reason in the change log.
• Chat settings overrides: quotas and tool allowlists can be adjusted via the chat settings endpoints; env values remain defaults.
• Doctor check: use /api/v1/chat/doctor to confirm quota/tool limits when chat requests are rejected.
• Offline kit checks: ensure the offline inference bundle uses the same guardrail configuration file as production; mismatches should fail the bundle validation step.
• Forensics: persisted outputs now contain guardrail_blocked, plan_cache_hit, and citation_coverage metadata. Include these fields when exporting evidence bundles to prove guardrail enforcement.
• Chat audit trail: retain prompt hashes, redaction metadata, tool call hashes, and policy decisions for post-incident review.

Keep this document synced whenever guardrail rules, telemetry names, or alert targets change.