LLM Provider Setup Guide

This guide explains how to configure an LLM (Large Language Model) provider for AdvisoryAI features in StellaOps.

Overview

AdvisoryAI uses LLM providers to power AI-assisted vulnerability analysis, advisory recommendations, and conversational assistance. Production-like chat question answering defaults to an in-house OpenAI-compatible local gateway; cloud providers are not fallback paths for chat.

Supported Providers

ProviderDescriptionRequirements
In-house OpenAI-compatibleLocal/private model gateway. Default setup examples pin microsoft/bitnet-b1.58-2B-4T; future custom/larger Stella Ops models must update the release guide before adoption.HTTP service with /health, /v1/models, /v1/chat/completions
OpenAIGPT-4o, GPT-4, GPT-3.5 TurboAPI key
Anthropic ClaudeClaude 4 Sonnet, Claude 3.5 Sonnet, Claude 3 OpusAPI key
Google GeminiGemini 1.5 Flash, Gemini 1.5 ProAPI key
OllamaLocal LLM (Llama 3, Mistral, etc.)Local Ollama instance

The built-in dummy echo/reverse provider is a test harness only. It is not registered in production-like hosts, even when etc/llm-providers/dummy.yaml exists. It may only be exposed when the host environment is Development or Testing and AdvisoryAI:RuntimeBindings:AllowLocalHarnessFakes=true.

The prompt-preview harness is also development/testing only. When explicitly enabled for a local harness, its model id is dev-harness.prompt-preview and generated metadata marks inference.harness=local-dev-test-only plus inference.production_eligible=false. Production-like hosts bind an unavailable inference client instead of this harness.

Production-like chat has no model fallback chain. If the configured local gateway is absent, unhealthy, missing the configured model, or returns invalid/uncited JSON, AdvisoryAI returns controlled runtime unavailability instead of a prompt preview, dummy response, sanitized prompt echo, or cloud result.

Quick Start

Using the Setup Wizard (Recommended)

Run the interactive setup wizard to configure an LLM provider:

stella setup --step llm

The wizard will:

  1. Present available provider options
  2. Prompt for required credentials
  3. Test API connectivity
  4. Save the configuration

Using Environment Variables

You can also configure providers using environment variables:

# In-house OpenAI-compatible local gateway
export ADVISORYAI__AdvisoryAI__Chat__Inference__Provider="inhouse-ternary"
export ADVISORYAI__AdvisoryAI__Chat__Inference__BaseUrl="http://127.0.0.1:8080"
export ADVISORYAI__AdvisoryAI__Chat__Inference__Model="microsoft/bitnet-b1.58-2B-4T"
export ADVISORYAI__AdvisoryAI__Chat__Inference__ModelDigest="sha256:<model-digest>"
export ADVISORYAI__AdvisoryAI__Chat__Inference__CapabilityValidation__RequireModelDigestMatch="true"

# OpenAI
export OPENAI_API_KEY="sk-..."

# Anthropic Claude
export ANTHROPIC_API_KEY="sk-ant-..."

# Google Gemini
export GEMINI_API_KEY="AIza..."
# or
export GOOGLE_API_KEY="AIza..."

Provider Configuration

The legacy provider catalog may still contain model fallback fields for non-chat tooling, but production-like AdvisoryAI question answering does not use a fallback chain. Chat binds one configured provider and fails closed when that provider is unavailable.

In-house OpenAI-compatible Local Gateway

This is the default production-like chat provider. The service boundary is deliberately small:

Default model guidance:

AdvisoryAI:
  Chat:
    Inference:
      Provider: "inhouse-ternary"
      BaseUrl: "http://127.0.0.1:8080"
      Model: "microsoft/bitnet-b1.58-2B-4T"
      ModelDigest: "sha256:<model-digest>"
      MaxTokens: 512
      Temperature: 0
      TopP: 1
      TopK: 1
      SendTopK: true
      Seed: 42
      TimeoutSeconds: 900
      HealthEndpoint: "/health"
      ModelsEndpoint: "/v1/models"
      ChatCompletionsEndpoint: "/v1/chat/completions"
      CapabilityValidation:
        Enabled: true
        RequireModelDigestMatch: true
        RequiredCapabilities:
          - "chat.completions"
          - "response_format.json_object"
          - "citations.required"
          - "seed"
          - "top_k"

The same block is available as a copyable appsettings example at docs/modules/advisory-ai/samples/inhouse-model-gateway.appsettings.json.

The configured model entry in /v1/models must include equivalent capability metadata:

{
  "data": [
    {
      "id": "microsoft/bitnet-b1.58-2B-4T",
      "model_digest": "sha256:<model-digest>",
      "capabilities": [
        "chat.completions",
        "response_format.json_object",
        "citations.required",
        "seed",
        "top_k"
      ]
    }
  ]
}

The AdvisoryAI web host exposes model-aware health endpoints:

The model response must be structured JSON with summary and cited claims. Every factual sentence must include known citation IDs from the assembled context. Unknown citations, invalid JSON, and uncited factual claims fail closed.

Fail-closed responses include a structured diagnostic under details.diagnostic for non-streaming calls and diagnostic on streaming error events. Diagnostics include code, stage, provider, model ID/digest, context digest, prompt-template hash, citation IDs, and citation content hashes. They do not include prompt previews or full evidence text.

OpenAI

Configuration file: etc/llm-providers/openai.yaml

enabled: true
priority: 100

api:
  apiKey: "${OPENAI_API_KEY}"
  baseUrl: "https://api.openai.com/v1"

model:
  name: "gpt-4o"
  fallbacks:
    - "gpt-4-turbo"
    - "gpt-3.5-turbo"

inference:
  temperature: 0.0
  maxTokens: 8192
  seed: 42

Models available:

Anthropic Claude

Configuration file: etc/llm-providers/claude.yaml

enabled: true
priority: 100

api:
  apiKey: "${ANTHROPIC_API_KEY}"
  baseUrl: "https://api.anthropic.com"

model:
  name: "claude-sonnet-4-20250514"
  fallbacks:
    - "claude-3-5-sonnet-20241022"
    - "claude-3-haiku-20240307"

inference:
  temperature: 0.0
  maxTokens: 8192

Models available:

Google Gemini

Configuration file: etc/llm-providers/gemini.yaml

enabled: true
priority: 100

api:
  apiKey: "${GEMINI_API_KEY}"
  baseUrl: "https://generativelanguage.googleapis.com/v1beta"

model:
  name: "gemini-1.5-flash"
  fallbacks:
    - "gemini-1.5-pro"
    - "gemini-1.0-pro"

inference:
  temperature: 0.0
  maxTokens: 8192
  topP: 1.0
  topK: 40

Models available:

Ollama (Local)

Configuration file: etc/llm-providers/ollama.yaml

enabled: true
priority: 50

api:
  endpoint: "http://localhost:11434"

model:
  name: "llama3:8b"
  fallbacks:
    - "mistral:7b"

inference:
  temperature: 0.0
  maxTokens: 4096

Prerequisites:

  1. Install Ollama: https://ollama.ai
  2. Pull a model: ollama pull llama3:8b
  3. Start Ollama: ollama serve

Recommended models:

Checking Configuration

Using Doctor

Run the Doctor checks to validate your LLM configuration:

# Check all AI-related configuration
stella doctor run --category ai

# Check specific provider
stella doctor run --check check.ai.provider.openai
stella doctor run --check check.ai.provider.claude
stella doctor run --check check.ai.provider.gemini

Using the CLI

Check your AdvisoryAI chat configuration:

stella advise chat-doctor

Troubleshooting

“AI/LLM provider not configured”

This error appears when no LLM provider is configured. For production-like chat, configure the in-house OpenAI-compatible gateway first. Solutions:

  1. Set AdvisoryAI:Chat:Inference:Provider to inhouse-ternary or openai-compatible.
  2. Set AdvisoryAI:Chat:Inference:BaseUrl to the local gateway base URL.
  3. Set AdvisoryAI:Chat:Inference:Model to an ID returned by /v1/models.
  4. Ensure /v1/models advertises the required capabilities and, when RequireModelDigestMatch=true, the configured digest.
  5. Check /health/model or /health/ready for the exact fail-closed diagnostic code.

API Key Invalid

If you receive authentication errors:

  1. Verify your API key is correct
  2. Check the API key has not expired
  3. Ensure billing is active on your provider account
  4. For Gemini, ensure the Generative Language API is enabled

Connection Timeout

If connections time out:

  1. Check network connectivity to the provider endpoint
  2. Verify proxy settings if behind a firewall
  3. For Ollama, ensure the service is running locally

Rate Limiting

If you encounter rate limits:

  1. Reduce request frequency
  2. Consider upgrading your API tier
  3. Enable request queueing in configuration

Offline/Air-Gapped Operation

For air-gapped deployments, use the in-house OpenAI-compatible gateway or Ollama with locally-available models:

  1. Transfer model files and the gateway image/binary through the approved offline kit
  2. Start the local model service and confirm /health, /v1/models, and required capability metadata
  3. Configure AdvisoryAI:Chat:Inference:* for the local endpoint, model ID, and model digest or signed manifest
  4. Keep cloud provider credentials unset in sealed environments

Security Considerations

  1. API Key Storage: Never commit API keys to version control. Use environment variables or secure vaults.
  2. Data Privacy: Be aware of data sent to cloud providers. Use the in-house gateway or Ollama for sensitive data.
  3. Rate Limiting: Configure appropriate rate limits to prevent abuse.
  4. Audit Logging: Enable audit logging for all LLM interactions. Chat audit metadata includes model ID, optional model digest, prompt template hash, and context digest.