checkId: check.binaryanalysis.corpus.kpi.baseline plugin: stellaops.doctor.binaryanalysis severity: warn tags: [binaryanalysis, corpus, kpi, baseline, regression, ci, groundtruth, security]

KPI Baseline Configuration

This Stella Ops Doctor check verifies that a binary-analysis KPI baseline file exists so CI gates can detect regressions in matching accuracy. It is part of the binary-analysis diagnostics surfaced by stella doctor; see the Doctor overview for how checks are run and reported.

What It Checks

Verifies that a KPI baseline file exists for regression detection in CI gates. The check:

Looks for a baseline file at the configured directory (default /var/lib/stella/baselines) and filename (default current.json), configurable via BinaryAnalysis:Corpus:BaselineDirectory and BinaryAnalysis:Corpus:BaselineFilename.
If the directory does not exist, warns.
If the default baseline file is missing but other .json files exist in the directory, warns and identifies the latest one.
Validates the baseline file as JSON and checks for expected KPI fields: precision, recall, falseNegativeRate, deterministicReplayRate, ttfrpP95Ms.
Fails if the file exists but is invalid JSON or has no recognized KPI fields.
Warns if some recommended fields are missing.

Why It Matters

Without a KPI baseline, CI gates cannot detect regressions in binary matching accuracy. A regression in precision or recall means vulnerability detection quality has degraded without anyone being alerted. The baseline enables automated quality gates that block releases when binary analysis accuracy drops.

Common Causes

KPI baseline has never been established (first run of corpus validation not yet completed)
Baseline directory path misconfigured
Baseline file was deleted or corrupted
Baseline created with an older tool version missing newer KPI fields

How to Fix

Docker Compose

# Create baseline directory
docker exec <binaryindex-container> mkdir -p /var/lib/stella/baselines

# Run corpus validation to establish baseline
docker exec <binaryindex-container> stella groundtruth validate run \
  --corpus src/__Tests/__Datasets/golden-corpus/seed/ --output-baseline

Bare Metal / systemd

sudo mkdir -p /var/lib/stella/baselines

# Run validation and save baseline
stella groundtruth validate run \
  --corpus src/__Tests/__Datasets/golden-corpus/seed/ \
  --output /var/lib/stella/baselines/current.json

# Or promote latest results
stella groundtruth baseline update --from-latest \
  --output /var/lib/stella/baselines/current.json

Kubernetes / Helm

binaryAnalysis:
  corpus:
    baselineDirectory: "/var/lib/stella/baselines"
    persistence:
      enabled: true

Run a one-time job to establish the baseline:

kubectl exec -it <binaryindex-pod> -- stella groundtruth validate run --output-baseline

Verification

stella doctor run --check check.binaryanalysis.corpus.kpi.baseline

check.binaryanalysis.corpus.mirror.freshness — verifies corpus mirror data is not stale
check.binaryanalysis.symbol.recovery.fallback — meta-check for symbol recovery availability