Golden Corpus Operations Runbook

Audience: BinaryIndex operators and on-call engineers responsible for the golden-corpus mirrors, KPI baselines, and ground-truth validation pipeline.

Overview

This runbook covers day-to-day operation of the golden-corpus infrastructure that underpins Stella Ops binary-analysis ground truth: health checks, troubleshooting, incident response, and routine maintenance.

Status note (doc<->code reconciliation, verified against src/). The only golden-corpus operations that are unconditionally live in a shipped stack today are the Doctor checks (the six check.binaryanalysis.* checks below; they are compiled into StellaOps.Doctor.Plugin.BinaryAnalysis and run via stella doctor --check). The stella groundtruth (alias gt) commands below are split across two different implementations in the tree, and the wiring matters:

• The richer command surface this runbook leans on — bundle export, bundle import --verify, baseline update, baseline show, and validate check (backed by the real IBundleExportService / IBundleImportService / IKpiRegressionService services) — lives in src/Cli/StellaOps.Cli/Commands/GroundTruthCommandGroup.cs. That command group is not wired into the shipped stella root command. BuildGroundTruthCommand there is referenced only by unit tests (GroundTruthCommandTests.cs); no CommandFactory registration and no CLI plugin manifest activates it. Treat bundle/baseline/validate check as library-backed but not runtime-wired (Draft/roadmap) until a manifest or factory call registers the group. The underlying services are DI-registered (AddCorpusBundleExport / AddCorpusBundleImport / AddKpiRegressionGates in StellaOps.BinaryIndex.GroundTruth.Reproducible), so the capability exists at the library layer.
• The groundtruth command that actually self-registers onto the root command is the plugin module src/Cli/__Libraries/StellaOps.Cli.Plugins.GroundTruth/GroundTruthCliCommandModule.cs (loaded via the CLI plugin manifest mechanism). Its surface is different and narrower: sources (list/enable/disable/sync), symbols (lookup/search/stats), pairs (create/list/stats), and validate (run/list/metrics/export/compare, alias val). It has no bundle and no baseline subcommand at all, has no validate check, and every subcommand requires a --postgres/-p connection string. Note: as of this writing there is no GroundTruth manifest staged under src/Cli/plugins/cli/ (only Aoc and Symbols are), so even this plugin module is not guaranteed to be loaded in a given deployment — verify with stella groundtruth --help against your build.
• validate run, validate metrics, and validate export are stubs in both implementations: they print an error and return a non-zero exit, explicitly refusing to synthesize results until a real validation harness and persisted run store are wired (GroundTruthCommandGroup.HandleValidateRunAsync/HandleValidateMetricsAsync/HandleValidateExportAsync; GroundTruthCliCommandModule.ExecuteValidateRunAsync and the run-not-found metrics/export handlers).
• groundtruth sources sync is not a working sync in either implementation: GroundTruthCommandGroup.HandleSourcesSyncAsync is a // TODO: Integrate with actual sync service progress simulation, and GroundTruthCliCommandModule.ExecuteSourcesSyncAsync prints “Sync functionality requires connector services to be configured” and exits 0 without fetching anything. Do not rely on it to refresh mirrors.

Operator-side scaffolding (sync wrapper scripts, log paths, systemd timers) shown in examples is illustrative; Stella Ops does not ship those scripts or paths.

Default paths and configuration keys (verify against your deployment)

The Doctor corpus checks read these locations. They are configurable; the values below are the in-code defaults from src/Doctor/__Plugins/StellaOps.Doctor.Plugin.BinaryAnalysis/Checks/.

The mirror freshness check inspects these subdirectories under the mirrors root: debian/archive, debian/snapshot, ubuntu/usn-index, alpine/secdb, osv.

The logical corpus tree (corpus/<distro>/<package>/<advisory-id>/{pre,post,metadata}, bench/baselines/, bench/results/) used in the validation/regression examples below is described in Golden Corpus Folder Layout. Map those relative paths to your deployment root.

Quick Reference

<id> is a single, exact, case-insensitive check ID. stella doctor --check does not accept globs or wildcards (it matches CheckId via an exact-set lookup; see CheckRegistry.cs). To run the corpus-related checks, invoke each by its full ID — there is no corpus.* shorthand.

The groundtruth command group exposes the alias gt. The flag spelling differs between the two implementations: the (tests-only) GroundTruthCommandGroup uses --source/-s for sources sync, while the self-registering plugin module uses --source-id/-s and requires --postgres. Confirm flags with stella groundtruth sources sync --help against your build before scripting. See the status note above before treating any bundle/baseline/validate check row as runnable in your deployment.

Troubleshooting

Mirror Sync Failures

Symptoms

• Doctor check check.binaryanalysis.corpus.mirror.freshness warns or fails. The check reports Warn when the mirrors root is missing or some mirrors are stale, and Fail when no mirrors exist or all existing mirrors are stale (older than the configured threshold; default 7 days).
• Validation runs fail with “source not found” errors
• Alerts for stale mirrors

Diagnosis

# Run the check directly (exact ID; no wildcards)
stella doctor --check check.binaryanalysis.corpus.mirror.freshness

# Inspect the mirror subdirectories the check looks at, under the
# configured mirrors root (default /var/lib/stella/mirrors; override
# via BinaryAnalysis:Corpus:MirrorsDirectory)
ls -la /var/lib/stella/mirrors/debian/archive
ls -la /var/lib/stella/mirrors/debian/snapshot
ls -la /var/lib/stella/mirrors/ubuntu/usn-index
ls -la /var/lib/stella/mirrors/alpine/secdb
ls -la /var/lib/stella/mirrors/osv

# Freshness is derived from the most-recent file mtime in each subtree —
# there is no .last-sync marker file. Sort by mtime to find staleness:
find /var/lib/stella/mirrors -type f -printf '%T+ %p\n' | sort | tail

# Test connectivity to upstreams the connectors use.
# snapshot.debian.org and the OSV bucket are confirmed in code
# (DebianSnapshotMirrorConnector / OsvDumpMirrorConnector); the others
# are typical upstreams to sanity-check.
curl -I https://snapshot.debian.org/
curl -I https://osv-vulnerabilities.storage.googleapis.com/
curl -I https://buildinfos.debian.net/
curl -I https://ubuntu.com/security/notices.json

If you run a sync wrapper that writes its own logs (e.g. /var/log/corpus/*-sync.log) or sync markers, those are operator-provided conventions — Stella Ops does not create them. There is no working built-in sync today: stella groundtruth sources sync is a placeholder in both CLI implementations (it does not fetch from upstreams — see the status note at the top). Until a connector-backed sync ships, populate the mirrors with your own fetch tooling against the upstreams below.

Resolution

1. Network connectivity issues

   # Check firewall rules
   iptables -L -n | grep -E "80|443"
   
   # Check DNS resolution
   nslookup snapshot.debian.org
   
   # Test with proxy if applicable
   export https_proxy=http://proxy:3128
   curl -I https://snapshot.debian.org/
   

2. Upstream service unavailable

   • Check upstream service status
   • Wait and retry (services may be temporarily unavailable)
   • Switch to backup mirror if available

3. Disk space issues

   # Check disk usage on the mirrors volume
   df -h /var/lib/stella
   
   # Clean up old archives (operator-provided cleanup; Stella Ops does not
   # ship an archive-old-results script — see Disk Space Critical below)
   

4. Re-sync the mirrors

   There is no working built-in mirror sync today (see status note). The sources sync subcommand exists in two forms but neither performs a real fetch:

   # tests-only GroundTruthCommandGroup form (uses --source, not wired into
   # the shipped CLI): progress is simulated, nothing is fetched
   stella groundtruth sources sync
   stella groundtruth sources sync --source debuginfod-ubuntu --full
   
   # self-registering plugin-module form (requires --postgres; prints
   # "Sync functionality requires connector services to be configured")
   stella groundtruth sources sync --source-id debuginfod-ubuntu --full \
       --postgres "Host=...;Database=...;Username=...;Password=..."
   

   Until a connector-backed sync ships, refresh the mirror subtrees with your own fetch tooling.

   Note: some in-code Doctor remediations suggest groundtruth mirror sync --all / --source / --sources, but no groundtruth mirror subcommand exists in the CLI — those remediation strings in CorpusMirrorFreshnessCheck are stale. The groundtruth sources sync subcommand is the closest match (but is itself a stub). The top-level stella mirror command is unrelated — it manages air-gap mirror bundles, not corpus source mirrors.

5. Permission issues

   # Check ownership of the mirrors root
   ls -la /var/lib/stella/mirrors/
   
   # Fix permissions (adjust owner to the account the worker runs as)
   chown -R stella:stella /var/lib/stella/mirrors/
   chmod -R 755 /var/lib/stella/mirrors/
   

Validation Failures

Symptoms

• CI pipeline fails on regression check
• Validation run exits with non-zero code
• Lower than expected KPI metrics

Diagnosis

# Compare a results file with a baseline. The regression service backing this
# is real (IKpiRegressionService.CheckRegression), but the `validate check`
# CLI binding lives in GroundTruthCommandGroup, which is NOT wired into the
# shipped stella root command (see status note). The self-registering plugin
# module has no `validate check` at all. Until the group is registered, drive
# the regression service from a harness/test rather than this command line.
# --verbose prints the active thresholds; exit code is non-zero on regression.
stella groundtruth validate check \
    --results bench/results/latest.json \
    --baseline bench/baselines/current.json \
    --verbose

# Review specific failures in the results file
jq '.failedPairs[]' bench/results/latest.json

stella groundtruth validate metrics --run-id <id> is not implemented in either CLI implementation: the handler returns a non-zero exit (or “Run not found”) and explicitly refuses to fabricate metrics until a persisted GroundTruth validation run store is wired. There is no --detailed flag and no latest pseudo-run-id. Until a run store ships, obtain metrics from the results JSON your harness writes (e.g. jq '.metrics' bench/results/latest.json).

Resolution

1. True regression (algorithm degradation)

   • Review recent code changes
   • Identify the causing commit
   • Either fix the regression or update baseline if intentional

2. False positive (ground truth incorrect)

   # Review ground truth for specific pair
   cat corpus/debian/openssl/DSA-5678-1/metadata/ground-truth.json
   
   # Update ground truth if incorrect
   # (Requires manual review by security team)
   

3. Infrastructure issues

   • Check if build environment is consistent
   • Verify debug symbols are available
   • Check Ghidra/BSim connectivity

4. Baseline drift

   • If corpus was significantly updated, baseline may need refresh
   • Run full validation and update baseline following procedures

Bundle Verification Failures

Wiring caveat: the flags below are an accurate transcription of the bundle import command defined in GroundTruthCommandGroup.cs, and the IBundleImportService it calls is real and DI-registered. But that command group is not wired into the shipped stella root command, and the self-registering CLI plugin module has no bundle subcommand. Until the group is registered (manifest or CommandFactory call), invoke the import service from a harness/test rather than this command line. See the status note at the top.

Symptoms

• stella groundtruth bundle import --verify fails
• Signature verification errors
• Timestamp validation errors

Diagnosis

# Verbose verification, emitting a machine-readable report.
# The report format defaults to "markdown"; pass --format json so jq can read it.
stella groundtruth bundle import \
    --input bundle.tar.gz \
    --verify \
    --verbose \
    --output report.json \
    --format json

# Check specific failures (fields are camelCase in the JSON report)
jq '.signatureResult, .timestampResult, .digestResult' report.json

Individual verification facets can be toggled: --verify-signatures, --verify-timestamps, --verify-digests, and --run-matcher (all default true when --verify is set). Supply --trusted-keys <path> and/or --trust-profile <path> to control signature verification, and --extract / --extract-path <dir> to unpack the bundle contents.

Resolution

1. Signature verification failure

   # Re-run verification pointing at the trusted keys explicitly
   # (--trusted-keys is the option the import command accepts; the path
   # below is an example location, not a shipped default)
   stella groundtruth bundle import \
       --input bundle.tar.gz --verify \
       --trusted-keys /etc/stellaops/trusted-keys.pub
   
   # Inspect the keys / verify the cert hasn't expired
   cat /etc/stellaops/trusted-keys.pub
   openssl x509 -in /etc/stellaops/trusted-keys.pub -noout -dates
   
   # If the bundle was signed with a different key, add that signing key
   # to your trusted keys file (or pass a matching --trust-profile)
   

2. Timestamp verification failure

   • Check TSA certificate validity
   • Verify system clock is accurate
   • Check if timestamp is within validity window

3. Digest mismatch

   • Bundle may be corrupted during transfer
   • Re-download or re-generate the bundle
   • Check for partial transfers

Baseline Not Found

Symptoms

• Doctor check check.binaryanalysis.corpus.kpi.baseline warns (declared DefaultSeverity is Warn; it warns on a missing directory, a missing default file (even if other baselines exist), or a baseline missing some of the expected KPI fields: precision, recall, falseNegativeRate, deterministicReplayRate, ttfrpP95Ms). Note it escalates to Fail when the baseline file exists but is corrupt/invalid JSON or contains no recognized KPI fields.
• Regression check (groundtruth validate check) errors because the baseline could not be loaded (subject to the same CLI-wiring caveat — see below)

Resolution

The Doctor check looks for the baseline at <BaselineDirectory>/<BaselineFilename> — default /var/lib/stella/baselines/current.json (config keys BinaryAnalysis:Corpus:BaselineDirectory / BinaryAnalysis:Corpus:BaselineFilename). Adjust the paths below to wherever your deployment stores baselines.

Wiring caveat: baseline update/baseline show are defined in GroundTruthCommandGroup.cs against the real IKpiRegressionService, but that command group is not wired into the shipped stella CLI, and the self-registering plugin module has no baseline subcommand. The commands below are accurate as written but will only run once the group is registered. Until then, drive IKpiRegressionService.UpdateBaselineAsync / LoadBaselineAsync from a harness, or manage the baseline JSON file directly.

# Check the configured baseline path
ls -la /var/lib/stella/baselines/current.json

# If missing, create one from a results file (--output is required).
# Use --from-results <path> OR --from-latest (one of the two is required).
stella groundtruth baseline update \
    --from-results bench/results/latest.json \
    --output /var/lib/stella/baselines/current.json \
    --description "Initial baseline"

# Inspect an existing baseline (precision/recall/FN-rate/determinism/TTFRP p95)
stella groundtruth baseline show --path /var/lib/stella/baselines/current.json

# Or restore from an archived copy (operator-managed; archive dir is a
# convention, not shipped)
cp /var/lib/stella/baselines/archive/baseline-20260115.json \
   /var/lib/stella/baselines/current.json

--from-latest selects the latest validation run via the regression service’s run store; if no store is configured it will fail the same way validate metrics does. When in doubt, use --from-results <path> against a results file your harness produced.

Debuginfod Connectivity Issues

Symptoms

• Doctor check check.binaryanalysis.debuginfod.available warns (the default severity is Warn, not a hard fail)
• Missing debug symbols during validation

In sealed/air-gap mode the debuginfod check skips rather than warns — it does not attempt outbound HTTPS. Configure a local debuginfod mirror or a pre-populated symbol cache instead (see Ground Truth Corpus Overview).

Diagnosis

# Run the check directly (exact ID; the suffix is .available, not .availability)
stella doctor --check check.binaryanalysis.debuginfod.available

# Check the DEBUGINFOD_URLS environment variable (space-separated list).
# If unset, the check probes the default https://debuginfod.fedoraproject.org
echo $DEBUGINFOD_URLS

# Test debuginfod connectivity
curl -I "https://debuginfod.fedoraproject.org/buildid/xyz/debuginfo"
curl -I "https://debuginfod.ubuntu.com/buildid/xyz/debuginfo"

Resolution

1. Configure DEBUGINFOD_URLS

   export DEBUGINFOD_URLS="https://debuginfod.fedoraproject.org/ https://debuginfod.ubuntu.com/"
   

2. Use local fallback

   • Enable local debug symbol cache
   • Sync ddeb packages for Ubuntu (check.binaryanalysis.ddeb.enabled covers ddeb repo configuration)
   • Download debug packages from archives

Incident Response

KPI Regression Detected in Production

Severity: High Response Time: 4 hours

1. Acknowledge and assess

   # Get current status
   stella groundtruth validate check \
       --results bench/results/latest.json \
       --baseline bench/baselines/current.json
   

2. Identify root cause

   • Check recent code changes
   • Review validation logs
   • Compare with previous runs

3. Mitigate

   • If code regression: revert the change
   • If ground truth issue: fix ground truth
   • If infrastructure issue: fix and re-run

4. Verify fix

   # Produce a fresh results file with your validation harness, then verify
   # the regression is cleared. `validate check` is backed by a real service
   # but its CLI binding is NOT wired into the shipped stella CLI, and
   # `validate run` is a stub in both implementations — see the status note.
   # Until the command group is registered, run the regression check via the
   # IKpiRegressionService from a harness rather than this command line.
   stella groundtruth validate check \
       --results results-fix.json \
       --baseline bench/baselines/current.json
   

5. Post-incident

   • Document in incident log
   • Update runbook if new issue type
   • Consider adding monitoring/alerting

Mirror Corruption Detected

Severity: Medium Response Time: 24 hours

1. Identify corrupted files

   # Check file integrity under the mirrors root
   find /var/lib/stella/mirrors -name "*.deb" -exec dpkg-deb --info {} \; 2>&1 | grep -i error
   

2. Remove corrupted files

   # Move corrupted files to a quarantine directory (operator convention)
   mkdir -p /var/lib/stella/quarantine
   mv /var/lib/stella/mirrors/debian/archive/path/to/corrupted.deb \
      /var/lib/stella/quarantine/
   

3. Re-sync affected mirror

   # NOTE: `groundtruth sources sync` is a stub in both CLI implementations
   # (no real fetch — see status note), and no sync-debian-mirrors.sh script
   # is shipped. Re-fetch the affected subtree with your own tooling, or
   # restore the corrupted files from a known-good mirror copy. The command
   # below will return success without fetching anything.
   stella groundtruth sources sync --source-id buildinfo-debian --full
   

4. Verify fix

   stella doctor --check check.binaryanalysis.corpus.mirror.freshness
   

Disk Space Critical

Severity: High Response Time: 1 hour

1. Check usage

   df -h /var/lib/stella
   du -sh /var/lib/stella/*
   

2. Quick cleanup

   Stella Ops does not ship archive-old-results.sh or prune-baselines.sh; the steps below are example cleanup an operator can script. Keep at least the active baseline (/var/lib/stella/baselines/current.json) and any baselines referenced by open incidents.

   # Remove old evidence bundles (adjust path to your evidence root)
   find /var/lib/stella/evidence -name "*.tar.gz" -mtime +90 -delete
   
   # Prune archived baselines, keeping the most recent N (operator-managed)
   ls -t /var/lib/stella/baselines/archive/*.json | tail -n +11 | xargs -r rm
   

3. Expand storage if needed

   • Request additional storage
   • Mount new volume
   • Migrate data if necessary

Scheduled Maintenance

Weekly Tasks

• [ ] Review Doctor health checks
• [ ] Check mirror freshness alerts
• [ ] Review validation results trends
• [ ] Archive old results

Monthly Tasks

• [ ] Generate compliance evidence bundles
• [ ] Review and update ground truth annotations
• [ ] Prune old baselines (keep last 10)
• [ ] Review storage usage trends

Quarterly Tasks

• [ ] Full corpus validation (not just seed) — run via your validation harness and feed the results into the regression check (IKpiRegressionService.CheckRegression); the CLI validate run driver is a stub and the validate check CLI binding is not runtime-wired in the shipped stella CLI (see status note)
• [ ] Review and update documentation
• [ ] Test disaster recovery procedures
• [ ] Review access permissions

Contact Information

Related Documentation

• Golden Corpus Folder Layout
• Golden Corpus Maintenance
• Ground Truth Corpus Overview