EPSS Air-Gapped Bundles Guide

Audience: operators who build, transfer, and import EPSS data bundles for sealed Stella Ops deployments.

Overview

This guide describes how to create, distribute, and import EPSS (Exploit Prediction Scoring System) data bundles for air-gapped Stella Ops deployments. EPSS bundles enable offline vulnerability-risk scoring with the same probabilistic threat intelligence available to online deployments.

EPSS scores are one payload within the broader risk bundle (which also carries CISA KEV and advisory data). For the transport format that wraps these payloads, see Mirror Bundles; for staging bundles into a sealed install, see the Air-Gap Importer Guide.

Key concepts:

Bundle Structure

Standard Risk Bundle Layout

risk-bundle-2025-12-17/
├── manifest.json                           # Bundle metadata and checksums
├── epss/
│   ├── epss_scores-2025-12-17.csv.zst      # EPSS data (ZSTD compressed)
│   └── epss_metadata.json                  # EPSS provenance
├── kev/
│   └── kev-catalog.json                    # CISA KEV catalog
├── advisories/
│   ├── nvd-updates.ndjson.zst
│   └── ghsa-updates.ndjson.zst
└── signatures/
    ├── bundle.dsse.json                    # DSSE signature (optional)
    └── bundle.sha256sums                   # File integrity checksums

manifest.json

{
  "bundle_id": "risk-bundle-2025-12-17",
  "created_at": "2025-12-17T00:00:00Z",
  "created_by": "stellaops-bundler-v1.2.3",
  "bundle_type": "risk",
  "schema_version": "v1",
  "contents": {
    "epss": {
      "model_date": "2025-12-17",
      "file": "epss/epss_scores-2025-12-17.csv.zst",
      "sha256": "abc123...",
      "size_bytes": 15728640,
      "row_count": 231417
    },
    "kev": {
      "catalog_version": "2025-12-17",
      "file": "kev/kev-catalog.json",
      "sha256": "def456...",
      "known_exploited_count": 1247
    },
    "advisories": {
      "nvd": {
        "file": "advisories/nvd-updates.ndjson.zst",
        "sha256": "ghi789...",
        "record_count": 1523
      },
      "ghsa": {
        "file": "advisories/ghsa-updates.ndjson.zst",
        "sha256": "jkl012...",
        "record_count": 8734
      }
    }
  },
  "signature": {
    "type": "dsse",
    "file": "signatures/bundle.dsse.json",
    "key_id": "stellaops-bundler-2025",
    "algorithm": "ed25519"
  }
}

epss/epss_metadata.json

{
  "model_date": "2025-12-17",
  "model_version": "v2025.12.17",
  "published_date": "2025-12-17",
  "row_count": 231417,
  "source_uri": "https://epss.empiricalsecurity.com/epss_scores-2025-12-17.csv.gz",
  "retrieved_at": "2025-12-17T00:05:32Z",
  "file_sha256": "abc123...",
  "decompressed_sha256": "xyz789...",
  "compression": "zstd",
  "compression_level": 19
}

Creating EPSS Bundles

Prerequisites

Build System Requirements:

Permissions:

Daily Bundle Creation (Automated)

Recommended Schedule: Daily at 01:00 UTC (after FIRST publishes at ~00:00 UTC)

Script: scripts/create-risk-bundle.sh

#!/bin/bash
set -euo pipefail

BUNDLE_DATE=$(date -u +%Y-%m-%d)
BUNDLE_DIR="risk-bundle-${BUNDLE_DATE}"
STAGING_DIR="/tmp/stellaops-bundles/${BUNDLE_DIR}"

echo "Creating risk bundle for ${BUNDLE_DATE}..."

# 1. Create staging directory
mkdir -p "${STAGING_DIR}"/{epss,kev,advisories,signatures}

# 2. Fetch EPSS data from FIRST.org
echo "Fetching EPSS data..."
curl -sL "https://epss.empiricalsecurity.com/epss_scores-${BUNDLE_DATE}.csv.gz" \
  -o "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv.gz"

# 3. Decompress and re-compress with ZSTD (better compression for offline)
gunzip "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv.gz"
zstd -19 -q "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv" \
  -o "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv.zst"
rm "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv"

# 4. Generate EPSS metadata
stellaops-bundler epss metadata \
  --file "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv.zst" \
  --model-date "${BUNDLE_DATE}" \
  --output "${STAGING_DIR}/epss/epss_metadata.json"

# 5. Fetch KEV catalog
echo "Fetching KEV catalog..."
curl -sL "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json" \
  -o "${STAGING_DIR}/kev/kev-catalog.json"

# 6. Fetch advisory updates (optional, for comprehensive bundles)
# stellaops-bundler advisories fetch ...

# 7. Generate checksums
echo "Generating checksums..."
(cd "${STAGING_DIR}" && find . -type f ! -name "*.sha256sums" -exec sha256sum {} \;) \
  > "${STAGING_DIR}/signatures/bundle.sha256sums"

# 8. Generate manifest
stellaops-bundler manifest create \
  --bundle-dir "${STAGING_DIR}" \
  --bundle-id "${BUNDLE_DIR}" \
  --output "${STAGING_DIR}/manifest.json"

# 9. Sign bundle (if signing key available)
if [ -n "${SIGNING_KEY:-}" ]; then
  echo "Signing bundle..."
  stellaops-bundler sign \
    --manifest "${STAGING_DIR}/manifest.json" \
    --key "${SIGNING_KEY}" \
    --output "${STAGING_DIR}/signatures/bundle.dsse.json"
fi

# 10. Create tarball
echo "Creating tarball..."
tar -C "$(dirname "${STAGING_DIR}")" -czf "/var/stellaops/bundles/${BUNDLE_DIR}.tar.gz" \
  "$(basename "${STAGING_DIR}")"

echo "Bundle created: /var/stellaops/bundles/${BUNDLE_DIR}.tar.gz"
echo "Size: $(du -h /var/stellaops/bundles/${BUNDLE_DIR}.tar.gz | cut -f1)"

# 11. Verify bundle
stellaops-bundler verify "/var/stellaops/bundles/${BUNDLE_DIR}.tar.gz"

Cron Schedule:

# Daily at 01:00 UTC (after FIRST publishes EPSS at ~00:00 UTC)
0 1 * * * /opt/stellaops/scripts/create-risk-bundle.sh >> /var/log/stellaops/bundler.log 2>&1

Distributing Bundles

Transfer Methods

1. Physical Media (Highest Security)

# Copy to USB drive
cp /var/stellaops/bundles/risk-bundle-2025-12-17.tar.gz /media/usb/stellaops/

# Verify checksum
sha256sum /media/usb/stellaops/risk-bundle-2025-12-17.tar.gz

2. Secure File Transfer (Network Isolation)

# SCP over dedicated management network
scp /var/stellaops/bundles/risk-bundle-2025-12-17.tar.gz \
  admin@airgap-gateway.internal:/incoming/

# Verify after transfer
ssh admin@airgap-gateway.internal \
  "sha256sum /incoming/risk-bundle-2025-12-17.tar.gz"

3. Offline Bundle Repository (CD/DVD)

# Burn to CD/DVD (for regulated industries)
growisofs -Z /dev/sr0 \
  -R -J -joliet-long \
  -V "StellaOps Risk Bundle 2025-12-17" \
  /var/stellaops/bundles/risk-bundle-2025-12-17.tar.gz

# Verify disc
md5sum /dev/sr0 > risk-bundle-2025-12-17.md5

Storage Recommendations

Bundle Retention:

Naming Convention:

Directory Structure (air-gapped system):

/opt/stellaops/bundles/
├── incoming/               # Transfer staging area
├── verified/               # Verified, ready to import
├── imported/               # Successfully imported (archive)
└── failed/                 # Failed verification/import (quarantine)

Importing Bundles (Air-Gapped System)

Pre-Import Verification

Step 1: Transfer to Verified Directory

# Transfer from incoming to verified (manual approval gate)
sudo mv /opt/stellaops/bundles/incoming/risk-bundle-2025-12-17.tar.gz \
        /opt/stellaops/bundles/verified/

Step 2: Verify Bundle Integrity

# Extract bundle
cd /opt/stellaops/bundles/verified
tar -xzf risk-bundle-2025-12-17.tar.gz

# Verify checksums
cd risk-bundle-2025-12-17
sha256sum -c signatures/bundle.sha256sums

# Expected output:
# epss/epss_scores-2025-12-17.csv.zst: OK
# epss/epss_metadata.json: OK
# kev/kev-catalog.json: OK
# manifest.json: OK

Step 3: Verify DSSE Signature (if signed)

stellaops-bundler verify-signature \
  --manifest manifest.json \
  --signature signatures/bundle.dsse.json \
  --trusted-keys /etc/stellaops/trusted-keys.json

# Expected output:
# ✓ Signature valid
# ✓ Key ID: stellaops-bundler-2025
# ✓ Signed at: 2025-12-17T01:05:00Z

Import Procedure

Step 4: Import Bundle

# Import using stellaops CLI
stellaops offline import \
  --bundle /opt/stellaops/bundles/verified/risk-bundle-2025-12-17.tar.gz \
  --verify \
  --dry-run

# Review dry-run output, then execute
stellaops offline import \
  --bundle /opt/stellaops/bundles/verified/risk-bundle-2025-12-17.tar.gz \
  --verify

Import Output:

Importing risk bundle: risk-bundle-2025-12-17
  ✓ Manifest validated
  ✓ Checksums verified
  ✓ Signature verified

Importing EPSS data...
  Model Date: 2025-12-17
  Row Count: 231,417
  ✓ epss_import_runs created (import_run_id: 550e8400-...)
  ✓ epss_scores inserted (231,417 rows, 23.4s)
  ✓ epss_changes computed (12,345 changes, 8.1s)
  ✓ epss_current upserted (231,417 rows, 5.2s)
  ✓ Event emitted: epss.updated

Importing KEV catalog...
  Known Exploited Count: 1,247
  ✓ kev_catalog updated

Import completed successfully in 41.2s

Step 5: Verify Import

# Check EPSS status
stellaops epss status

# Expected output:
# EPSS Status:
#   Latest Model Date: 2025-12-17
#   Source: bundle://risk-bundle-2025-12-17
#   CVE Count: 231,417
#   Staleness: FRESH (0 days)
#   Import Time: 2025-12-17T10:30:00Z

# Query specific CVE to verify
stellaops epss get CVE-2024-12345

# Expected output:
# CVE-2024-12345
#   Score: 0.42357
#   Percentile: 88.2th
#   Model Date: 2025-12-17
#   Source: bundle://risk-bundle-2025-12-17

Step 6: Archive Imported Bundle

# Move to imported archive
sudo mv /opt/stellaops/bundles/verified/risk-bundle-2025-12-17.tar.gz \
        /opt/stellaops/bundles/imported/

Automation (Air-Gapped System)

Automated Import on Arrival

Script: /opt/stellaops/scripts/auto-import-bundle.sh

#!/bin/bash
set -euo pipefail

INCOMING_DIR="/opt/stellaops/bundles/incoming"
VERIFIED_DIR="/opt/stellaops/bundles/verified"
IMPORTED_DIR="/opt/stellaops/bundles/imported"
FAILED_DIR="/opt/stellaops/bundles/failed"
LOG_FILE="/var/log/stellaops/auto-import.log"

log() {
  echo "[$(date -Iseconds)] $*" | tee -a "${LOG_FILE}"
}

# Watch for new bundles in incoming/
for bundle in "${INCOMING_DIR}"/risk-bundle-*.tar.gz; do
  [ -f "${bundle}" ] || continue

  BUNDLE_NAME=$(basename "${bundle}")
  log "Detected new bundle: ${BUNDLE_NAME}"

  # Extract
  EXTRACT_DIR="${VERIFIED_DIR}/${BUNDLE_NAME%.tar.gz}"
  mkdir -p "${EXTRACT_DIR}"
  tar -xzf "${bundle}" -C "${VERIFIED_DIR}"

  # Verify checksums
  if ! (cd "${EXTRACT_DIR}" && sha256sum -c signatures/bundle.sha256sums > /dev/null 2>&1); then
    log "ERROR: Checksum verification failed for ${BUNDLE_NAME}"
    mv "${bundle}" "${FAILED_DIR}/"
    rm -rf "${EXTRACT_DIR}"
    continue
  fi

  log "Checksum verification passed"

  # Verify signature (if present)
  if [ -f "${EXTRACT_DIR}/signatures/bundle.dsse.json" ]; then
    if ! stellaops-bundler verify-signature \
      --manifest "${EXTRACT_DIR}/manifest.json" \
      --signature "${EXTRACT_DIR}/signatures/bundle.dsse.json" \
      --trusted-keys /etc/stellaops/trusted-keys.json > /dev/null 2>&1; then
      log "ERROR: Signature verification failed for ${BUNDLE_NAME}"
      mv "${bundle}" "${FAILED_DIR}/"
      rm -rf "${EXTRACT_DIR}"
      continue
    fi
    log "Signature verification passed"
  fi

  # Import
  if stellaops offline import --bundle "${bundle}" --verify >> "${LOG_FILE}" 2>&1; then
    log "Import successful for ${BUNDLE_NAME}"
    mv "${bundle}" "${IMPORTED_DIR}/"
    rm -rf "${EXTRACT_DIR}"
  else
    log "ERROR: Import failed for ${BUNDLE_NAME}"
    mv "${bundle}" "${FAILED_DIR}/"
  fi
done

Systemd Service: /etc/systemd/system/stellaops-bundle-watcher.service

[Unit]
Description=StellaOps Bundle Auto-Import Watcher
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/inotifywait -m -e close_write --format '%w%f' /opt/stellaops/bundles/incoming | \
          while read file; do /opt/stellaops/scripts/auto-import-bundle.sh; done
Restart=always
RestartSec=10
User=stellaops
Group=stellaops

[Install]
WantedBy=multi-user.target

Enable Service:

sudo systemctl enable stellaops-bundle-watcher
sudo systemctl start stellaops-bundle-watcher

Staleness Handling

Staleness Thresholds

Days Since Model DateStatusAction
0-1FRESHNormal operation
2-7ACCEPTABLEContinue, low-priority alert
8-14STALEAlert, plan bundle import
15+VERY_STALEFallback to CVSS-only, urgent alert

Monitoring Staleness

SQL Query:

SELECT * FROM concelier.epss_model_staleness;

-- Output:
-- latest_model_date | latest_import_at       | days_stale | staleness_status
-- 2025-12-10        | 2025-12-10 10:30:00+00 | 7          | ACCEPTABLE

Prometheus Metric:

epss_model_staleness_days{instance="airgap-prod"}

# Alert rule:
- alert: EpssDataStale
  expr: epss_model_staleness_days > 7
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "EPSS data is stale ({{ $value }} days old)"

Fallback Behavior

When EPSS data is VERY_STALE (>14 days):

Automatic Fallback:

Manual Override (force continue using stale data):

# etc/scanner.yaml
scanner:
  epss:
    staleness_policy: continue  # Options: fallback, continue, error
    max_staleness_days: 30      # Override 14-day default

Troubleshooting

Bundle Import Failed: Checksum Mismatch

Symptom:

ERROR: Checksum verification failed
epss/epss_scores-2025-12-17.csv.zst: FAILED

Diagnosis:

  1. Verify bundle was not corrupted during transfer:

    # Compare with original
sha256sum risk-bundle-2025-12-17.tar.gz

  2. Re-transfer bundle from source

Resolution:

Bundle Import Failed: Signature Invalid

Symptom:

ERROR: Signature verification failed
Invalid signature or untrusted key

Diagnosis:

  1. Check trusted keys configured:

    cat /etc/stellaops/trusted-keys.json

  2. Verify key ID in bundle signature matches:

    jq '.signature.key_id' manifest.json

Resolution:

No EPSS Data After Import

Symptom:

Diagnosis:

-- Check import runs
SELECT * FROM concelier.epss_import_runs ORDER BY created_at DESC LIMIT 1;

-- Check epss_current count
SELECT COUNT(*) FROM concelier.epss_current;

Resolution:

  1. If import_runs shows FAILED status:

    • Check error column: SELECT error FROM concelier.epss_import_runs WHERE status = 'FAILED'
    • Re-run import with verbose logging

  2. If epss_current is empty:

    • Manually trigger upsert:
      -- Re-run upsert for latest model_date
-- (This SQL is safe to re-run)
INSERT INTO concelier.epss_current (cve_id, epss_score, percentile, model_date, import_run_id, updated_at)
SELECT s.cve_id, s.epss_score, s.percentile, s.model_date, s.import_run_id, NOW()
FROM concelier.epss_scores s
WHERE s.model_date = (SELECT MAX(model_date) FROM concelier.epss_import_runs WHERE status = 'SUCCEEDED')
ON CONFLICT (cve_id) DO UPDATE SET
  epss_score = EXCLUDED.epss_score,
  percentile = EXCLUDED.percentile,
  model_date = EXCLUDED.model_date,
  import_run_id = EXCLUDED.import_run_id,
  updated_at = NOW();

Best Practices

1. Weekly Bundle Import Cadence

Recommended Schedule:

2. Bundle Verification Checklist

Before importing:

3. Rollback Plan

If new bundle causes issues:

# 1. Identify problematic import_run_id
SELECT import_run_id, model_date, status
FROM concelier.epss_import_runs
ORDER BY created_at DESC LIMIT 5;

# 2. Delete problematic import (cascades to epss_scores, epss_changes)
DELETE FROM concelier.epss_import_runs
WHERE import_run_id = '550e8400-...';

# 3. Restore epss_current from previous day
-- (Upsert from previous model_date as shown in troubleshooting)

# 4. Verify rollback
stellaops epss status

4. Audit Trail

Log all bundle imports for compliance:

Audit Log Format (/var/log/stellaops/bundle-audit.log):

{
  "timestamp": "2025-12-17T10:30:00Z",
  "action": "import",
  "bundle_id": "risk-bundle-2025-12-17",
  "bundle_sha256": "abc123...",
  "imported_by": "admin@example.com",
  "import_run_id": "550e8400-e29b-41d4-a716-446655440000",
  "result": "SUCCESS",
  "row_count": 231417,
  "duration_seconds": 41.2
}

Appendix: Bundle Creation Tools

stellaops-bundler CLI Reference

# Create EPSS metadata
stellaops-bundler epss metadata \
  --file epss_scores-2025-12-17.csv.zst \
  --model-date 2025-12-17 \
  --output epss_metadata.json

# Create manifest
stellaops-bundler manifest create \
  --bundle-dir risk-bundle-2025-12-17 \
  --bundle-id risk-bundle-2025-12-17 \
  --output manifest.json

# Sign bundle
stellaops-bundler sign \
  --manifest manifest.json \
  --key /path/to/signing-key.pem \
  --output bundle.dsse.json

# Verify bundle
stellaops-bundler verify risk-bundle-2025-12-17.tar.gz

Custom Bundle Scripts

Example for creating weekly bundles (7-day snapshots):

#!/bin/bash
# create-weekly-bundle.sh

WEEK_START=$(date -u -d "last monday" +%Y-%m-%d)
WEEK_END=$(date -u +%Y-%m-%d)
BUNDLE_ID="risk-bundle-weekly-${WEEK_START}"

echo "Creating weekly bundle: ${BUNDLE_ID}"

for day in $(seq 0 6); do
  CURRENT_DATE=$(date -u -d "${WEEK_START} + ${day} days" +%Y-%m-%d)
  # Fetch EPSS for each day...
  curl -sL "https://epss.empiricalsecurity.com/epss_scores-${CURRENT_DATE}.csv.gz" \
    -o "epss/epss_scores-${CURRENT_DATE}.csv.gz"
done

# Compress and bundle...
tar -czf "${BUNDLE_ID}.tar.gz" epss/ kev/ manifest.json

See also