Symbol Bundles for Air-Gapped Installations

Audience: build and release engineers producing symbol artifacts, and operators deploying them into air-gapped environments.
Purpose: create, verify, and deploy deterministic, signed symbol bundles for air-gapped Stella Ops installations.

This guide describes how to create, verify, and deploy deterministic symbol bundles for air-gapped Stella Ops installations. Symbol bundles let a sealed deployment resolve debug symbols during reachability and binary analysis without contacting an online symbol server.

Overview

Symbol bundles package debug symbols (PDBs, DWARF, etc.) into a single archive with:

Deterministic ordering for reproducible builds
BLAKE3 hashes for content verification
DSSE signatures for authenticity
Rekor checkpoints for transparency log integration
Merkle inclusion proofs for offline verification

Bundle Structure

bundle-name-1.0.0.symbols.zip
├── manifest.json              # Bundle manifest with all metadata
├── symbols/
│   ├── {debug-id-1}/
│   │   ├── myapp.exe.symbols  # Symbol blob
│   │   └── myapp.exe.symbols.json  # Symbol manifest
│   ├── {debug-id-2}/
│   │   ├── libcrypto.so.symbols
│   │   └── libcrypto.so.symbols.json
│   └── ...

Creating a Bundle

Prerequisites

Collect symbol manifests from CI builds or ingest tools
Ensure all manifests follow the *.symbols.json naming convention
Have signing keys available (if signing is required)

Build Command

# Basic bundle creation
stella symbols bundle \
  --name "product-symbols" \
  --version "1.0.0" \
  --source ./symbols-dir \
  --output ./bundles

# With signing and Rekor submission
stella symbols bundle \
  --name "product-symbols" \
  --version "1.0.0" \
  --source ./symbols-dir \
  --output ./bundles \
  --sign \
  --key ./signing-key.pem \
  --key-id "release-key-2025" \
  --rekor \
  --rekor-url https://rekor.sigstore.dev

# Filter by platform
stella symbols bundle \
  --name "linux-symbols" \
  --version "1.0.0" \
  --source ./symbols-dir \
  --output ./bundles \
  --platform linux-x64

Bundle Options

Option	Description
`--name`	Bundle name (required)
`--version`	Bundle version in SemVer format (required)
`--source`	Source directory containing symbol manifests (required)
`--output`	Output directory for bundle archive (required)
`--platform`	Filter symbols by platform (e.g., linux-x64, win-x64)
`--tenant`	Filter symbols by tenant ID
`--sign`	Sign bundle with DSSE
`--key`	Path to signing key (PEM-encoded private key)
`--key-id`	Key ID for DSSE signature
`--algorithm`	Signing algorithm (ecdsa-p256, ed25519, rsa-pss-sha256)
`--rekor`	Submit to Rekor transparency log
`--rekor-url`	Rekor server URL
`--format`	Archive format: zip (default) or tar.gz
`--compression`	Compression level (0-9, default: 6)

Verifying a Bundle

Online Verification

stella symbols verify --bundle ./product-symbols-1.0.0.symbols.zip

Offline Verification

For air-gapped environments, include the Rekor public key:

stella symbols verify \
  --bundle ./product-symbols-1.0.0.symbols.zip \
  --public-key ./signing-public-key.pem \
  --rekor-offline \
  --rekor-key ./rekor-public-key.pem

Verification Output

Bundle verification successful!
  Bundle ID: a1b2c3d4e5f6g7h8
  Name: product-symbols-1.0.0.symbols
  Version: 1.0.0
  Signature: valid (ecdsa-p256)
  Hash verification: 42/42 valid

Extracting Symbols

Full Extraction

stella symbols extract \
  --bundle ./product-symbols-1.0.0.symbols.zip \
  --output ./extracted-symbols

Platform-Filtered Extraction

stella symbols extract \
  --bundle ./product-symbols-1.0.0.symbols.zip \
  --output ./linux-symbols \
  --platform linux-x64

Manifests Only

stella symbols extract \
  --bundle ./product-symbols-1.0.0.symbols.zip \
  --output ./manifests-only \
  --manifests-only

Inspecting Bundles

# Basic info
stella symbols inspect --bundle ./product-symbols-1.0.0.symbols.zip

# With entry listing
stella symbols inspect --bundle ./product-symbols-1.0.0.symbols.zip --entries

Bundle Manifest Schema

The bundle manifest (manifest.json) follows this schema:

{
  "schemaVersion": "stellaops.symbols.bundle/v1",
  "bundleId": "blake3-hash-of-content",
  "name": "product-symbols",
  "version": "1.0.0",
  "createdAt": "2025-12-14T10:30:00Z",
  "platform": null,
  "tenantId": null,
  "entries": [
    {
      "debugId": "abc123def456",
      "codeId": "...",
      "binaryName": "myapp.exe",
      "platform": "win-x64",
      "format": "pe",
      "manifestHash": "blake3...",
      "blobHash": "blake3...",
      "blobSizeBytes": 102400,
      "archivePath": "symbols/abc123def456/myapp.exe.symbols",
      "symbolCount": 5000
    }
  ],
  "totalSizeBytes": 10485760,
  "signature": {
    "signed": true,
    "algorithm": "ecdsa-p256",
    "keyId": "release-key-2025",
    "dsseDigest": "sha256:...",
    "signedAt": "2025-12-14T10:30:00Z",
    "publicKey": "-----BEGIN PUBLIC KEY-----..."
  },
  "rekorCheckpoint": {
    "rekorUrl": "https://rekor.sigstore.dev",
    "logEntryId": "...",
    "logIndex": 12345678,
    "integratedTime": "2025-12-14T10:30:01Z",
    "rootHash": "sha256:...",
    "treeSize": 987654321,
    "inclusionProof": {
      "logIndex": 12345678,
      "rootHash": "sha256:...",
      "treeSize": 987654321,
      "hashes": ["sha256:...", "sha256:..."]
    },
    "logPublicKey": "-----BEGIN PUBLIC KEY-----..."
  },
  "hashAlgorithm": "blake3"
}

Air-Gap Deployment Workflow

1. Create Bundle (Online Environment)

# On the online build server
stella symbols bundle \
  --name "release-v2.0.0-symbols" \
  --version "2.0.0" \
  --source /build/symbols \
  --output /export \
  --sign --key /keys/release.pem \
  --rekor

2. Transfer to Air-Gapped Environment

Copy the following files to the air-gapped environment:

release-v2.0.0-symbols-2.0.0.symbols.zip
release-v2.0.0-symbols-2.0.0.manifest.json
signing-public-key.pem (if not already present)
rekor-public-key.pem (for Rekor offline verification)

3. Verify (Air-Gapped Environment)

# On the air-gapped server
stella symbols verify \
  --bundle ./release-v2.0.0-symbols-2.0.0.symbols.zip \
  --public-key ./signing-public-key.pem \
  --rekor-offline \
  --rekor-key ./rekor-public-key.pem

4. Extract and Deploy

# Extract to symbols server directory
stella symbols extract \
  --bundle ./release-v2.0.0-symbols-2.0.0.symbols.zip \
  --output /var/stellaops/symbols \
  --verify

Determinism Guarantees

Symbol bundles are deterministic:

Entry ordering: Entries sorted by debug ID, then binary name (lexicographic)
Hash algorithm: BLAKE3 for all content hashes
Timestamps: UTC ISO-8601 format
JSON serialization: Canonical form (no whitespace, sorted keys)
Archive entries: Sorted by path within archive

This ensures that given the same input manifests, the same bundle (excluding signatures) is produced.

CI Integration

GitHub Actions Example

- name: Build symbol bundle
  run: |
    stella symbols bundle \
      --name "${{ github.repository }}-symbols" \
      --version "${{ github.ref_name }}" \
      --source ./build/symbols \
      --output ./dist \
      --sign --key ${{ secrets.SIGNING_KEY }} \
      --rekor

- name: Upload bundle artifact
  uses: actions/upload-artifact@v4
  with:
    name: symbol-bundle
    path: ./dist/*.symbols.zip

Troubleshooting

“No symbol manifests found”

Ensure manifests follow the *.symbols.json naming convention and are not DSSE envelopes (*.dsse.json).

“Signature verification failed”

Check that:

The public key matches the signing key
The bundle has not been modified after signing
The key ID matches what was used during signing

“Rekor inclusion proof invalid”

For offline verification:

Ensure the Rekor public key is current
The checkpoint was created when the log was online
The tree size hasn’t changed since the checkpoint

Offline Update Kit
Scanner Architecture — symbol resolution and binary analysis
Attestor Architecture — DSSE signing and Rekor integration