Performance Baselines

Overview

This document defines performance baselines for StellaOps scanner operations. All metrics are measured against reference images and workloads to ensure consistent, reproducible benchmarks.

Last Updated: 2025-12-14 Next Review: 2026-03-14

Reference Images

Standard images used for performance benchmarking:

Scan Performance Targets

Container Image Scanning

Per-Image Targets

Reachability Analysis Targets

By Codebase Size

By Language

SBOM Generation Targets

Combined Operations

VEX Processing Targets

CVSS Scoring Targets

Attestation Targets

Database Operation Targets

Regression Thresholds

Performance regression is detected when metrics exceed these thresholds compared to baseline:

Regression Detection Rules

1. Warning: Alert engineering team, add to release notes
2. Blocking: Cannot merge/release until resolved or waived
3. Waiver: Requires documented justification and SME approval

Measurement Methodology

Environment Setup

# Standard test environment
# - CPU: 8 cores (x86_64)
# - Memory: 16GB RAM
# - Storage: NVMe SSD
# - OS: Ubuntu 22.04 LTS
# - Docker: 24.x

# Clear caches before cold start tests
docker system prune -af
sync && echo 3 > /proc/sys/vm/drop_caches

CLI surface. The binary is stella, not stellaops. scan and sbom are command groups, not leaf commands — stella scan --image … exits 2 with Unrecognized command or argument '--image'. Verify any invocation below against stella <verb> --help before quoting a number produced by it.

Scan Performance

Image scanning is two steps: generate a deterministic SBOM, then match it against the compact vuln DB. Time them separately — they have different cost profiles.

# Cold start measurement (SBOM generation)
time stella sbom generate --image alpine:3.19 --format cyclonedx --output sbom.json --force

# Warm cache measurement (run 3x, take average)
for i in {1..3}; do
  time stella sbom generate --image alpine:3.19 --format cyclonedx --output sbom.json --force
done

# Offline match against the compact vuln DB (no backend required)
time stella sbom check --sbom sbom.json --format json > /dev/null

# Memory profiling
/usr/bin/time -v stella sbom generate --image alpine:3.19 --format cyclonedx \
  --output sbom.json --force 2>&1 | grep "Maximum resident set size"

# CPU profiling
perf stat stella sbom check --sbom sbom.json --format json > /dev/null

--source-date pins the SBOM timestamp, which is what makes repeat runs byte-identical; leave it at the default only when you are not comparing digests across runs.

Reachability Analysis

Reachability is measured over a call graph extracted from source. stella scan graph does the extraction; stella reachability operates on the result.

# Time measurement — call-graph extraction
time stella scan graph --lang dotnet --target ./src --output callgraph.json

# Memory profiling
/usr/bin/time -v stella scan graph --lang dotnet --target ./src --output callgraph.json 2>&1

# Reachability over an existing SBOM, in one pass (offline engine)
time stella sbom check --sbom sbom.json --reachability --call-graph callgraph.json --format json > /dev/null

--lang accepts dotnet, java, node, python, go, rust, ruby, php. There is no csharp value — .NET projects use dotnet. stella sbom check --reachability additionally accepts --source + --reach-lang (go|rust only) to extract a graph in-process instead of taking a pre-extracted one.

SBOM Generation

# Time measurement
time stella sbom generate --image node:20-alpine --format cyclonedx --output sbom.json --force

# Output size
stella sbom generate --image node:20-alpine --format cyclonedx --output sbom.json --force && \
  ls -lh sbom.json

Batch Operations

There is no CLI flag for parallel multi-image scanning. stella sbom generate takes a single --image or --directory; drive concurrency from the shell.

# Process multiple images in parallel (4 at a time)
time xargs -P 4 -I{} sh -c \
  'stella sbom generate --image "$1" --format cyclonedx --output "./results/$(echo "$1" | tr "/:" "__").json" --force' _ {} \
  < images.txt

# Throughput test (images per minute)
START=$(date +%s)
for i in {1..10}; do
  stella sbom generate --image alpine:3.19 --format cyclonedx --output /tmp/sbom.json --force > /dev/null
done
END=$(date +%s)
echo "Throughput: $(( 10 * 60 / (END - START) )) images/minute"

stella scan submit-list --file <ids.txt> --max-concurrent 10 does exist and is throttled, but it takes already-registered SBOM source ids (GUIDs), not image references, and it requires a running backend — it is not a substitute for the local loop above.

CI Integration

Benchmark Workflow

# .gitea/workflows/performance-benchmark.yml
name: Performance Benchmark

on:
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 2 * * 1'  # Weekly Monday 2am

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run benchmarks
        run: python src/Tools/StellaOps.Bench/Determinism/run_bench.py --output results/

      - name: Aggregate results
        run: |
          dotnet run --project src/Tools/Bench/StellaOps.Bench.Compute.Cli -- \
            report results/ --output results/report.json

      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: benchmark-results
          path: results/

The regression gate is not shipped. Earlier revisions of this page ran stellaops benchmark compare --baseline … --fail-on-regression and make benchmark-performance. Neither exists: stella benchmark exits 2 (Unrecognized command or argument 'benchmark') and the repository has no root Makefile. The thresholds in the tables above are targets a reviewer applies by reading the report — nothing enforces them automatically today.

Real harnesses, for anyone wiring this up properly:

• src/Tools/StellaOps.Bench/Determinism/run_bench.py and run_reachability.py — Python, offline-friendly (offline_run.sh).
• src/Tools/StellaOps.Bench/{Graph,ImpactIndex,PolicyCache,PolicyDelta}/ — per-subsystem microbenchmarks.
• src/Tools/Bench/StellaOps.Bench.Compute.Cli — binary name stella-bench-compute, verbs discovery-truth, evidence-score, compare-metrics, report. It is a separate tool from stella; the comparison it does is scanner-vs-scanner discovery coverage, not run-over-run performance regression.

Verify the exact flags with --help before pasting any of this into CI.

Local Testing

# Run full benchmark suite
make benchmark-performance

# Run specific image benchmark
make benchmark-image IMAGE=alpine:3.19

# Generate baseline
make benchmark-baseline

# Compare against baseline
make benchmark-compare

Optimization Guidelines

For Scan Performance

1. Pre-pull images for consistent timing
2. Use layered caching for repeat scans
3. Enable parallel analysis for multi-ecosystem images
4. Consider selective scanning for known-safe layers

For Reachability

1. Incremental analysis for unchanged files
2. Cache resolved dependencies
3. Use language-specific optimizations (e.g., Roslyn for C#)
4. Limit call graph depth for very large codebases

For Memory

1. Stream large SBOMs instead of loading fully
2. Use batched database operations
3. Release intermediate data structures early
4. Configure GC appropriately for workload

Historical Baselines

Version History

Improvement Targets

References

• Accuracy Metrics Framework
• Benchmark Submission Guide (pending)
• Scanner Architecture
• Reachability Module