VEX Signature Verification: Offline Mode

Last updated: 2025-12-28

Audience: operators running Stella Ops in air-gapped (sealed) environments who need VEX documents verified without reaching public trust infrastructure.

This guide describes how to configure VEX signature verification for air-gapped deployments where network access to public trust infrastructure (Sigstore, Fulcio, Rekor) is unavailable. It covers the offline trust model, bundle preparation and transfer, verification behaviour, update cadence, monitoring, and troubleshooting.

Offline Mode Architecture

┌─────────────────────────────────────────────────────────────┐
│                   Air-Gapped Environment                    │
│                                                             │
│  ┌───────────────┐     ┌────────────────────────────────┐  │
│  │ VEX Documents │────▶│ ProductionVexSignatureVerifier │  │
│  └───────────────┘     └────────────────────────────────┘  │
│                                   │                         │
│                    ┌──────────────┴────────────────┐       │
│                    ▼                               ▼       │
│  ┌─────────────────────────┐    ┌─────────────────────┐   │
│  │ Bundled Trust Anchors   │    │ Bundled Issuer Dir  │   │
│  │ /var/stellaops/trust/   │    │ /var/stellaops/     │   │
│  │ ├── fulcio-root.pem     │    │ bundles/issuers.json│   │
│  │ ├── sigstore-root.pem   │    └─────────────────────┘   │
│  │ └── internal-ca.pem     │                               │
│  └─────────────────────────┘                               │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Configuration

1. Enable Offline Mode

File: etc/excititor.yaml

VexSignatureVerification:
  Enabled: true
  DefaultProfile: "world"
  OfflineMode: true  # Critical: Enable offline verification

  # Offline-specific settings
  OfflineBundle:
    Enabled: true
    BundlePath: "/var/stellaops/bundles"
    RefreshOnStartup: false

  # Trust anchors for signature verification
  TrustAnchors:
    Fulcio:
      - "/var/stellaops/trust/fulcio-root.pem"
      - "/var/stellaops/trust/fulcio-intermediate.pem"
    Sigstore:
      - "/var/stellaops/trust/sigstore-root.pem"
    Internal:
      - "/var/stellaops/trust/internal-ca.pem"
      - "/var/stellaops/trust/internal-intermediate.pem"

  # IssuerDirectory in offline mode
  IssuerDirectory:
    OfflineBundle: "/var/stellaops/bundles/issuers.json"
    FallbackToBundle: true
    # ServiceUrl not needed in offline mode

2. Directory Structure

/var/stellaops/
├── bundles/
│   ├── issuers.json           # Issuer directory bundle
│   ├── revocations.json       # Key revocation data
│   └── tuf-metadata/          # TUF metadata for updates
│       ├── root.json
│       ├── targets.json
│       └── snapshot.json
├── trust/
│   ├── fulcio-root.pem        # Sigstore Fulcio root CA
│   ├── fulcio-intermediate.pem
│   ├── sigstore-root.pem      # Sigstore root
│   ├── rekor-pubkey.pem       # Rekor public key
│   ├── internal-ca.pem        # Internal enterprise CA
│   └── internal-intermediate.pem
└── cache/
    └── verification-cache.db  # Local verification cache

Bundle Preparation

1. Download Trust Anchors

Run this on a connected machine to prepare the bundle:

#!/bin/bash
# prepare-offline-bundle.sh

BUNDLE_DIR="./offline-bundle"
mkdir -p "$BUNDLE_DIR/trust" "$BUNDLE_DIR/bundles"

# Download Sigstore trust anchors
echo "Downloading Sigstore trust anchors..."
curl -sSL https://fulcio.sigstore.dev/api/v2/trustBundle \
    -o "$BUNDLE_DIR/trust/fulcio-root.pem"

curl -sSL https://rekor.sigstore.dev/api/v1/log/publicKey \
    -o "$BUNDLE_DIR/trust/rekor-pubkey.pem"

# Download TUF metadata
echo "Downloading TUF metadata..."
cosign initialize --mirror=https://tuf-repo.sigstore.dev \
    --root="$BUNDLE_DIR/bundles/tuf-metadata"

# Export issuer directory
echo "Exporting issuer directory..."
stellaops-cli issuer-directory export \
    --format json \
    --output "$BUNDLE_DIR/bundles/issuers.json"

# Export revocation data
echo "Exporting revocation data..."
stellaops-cli revocations export \
    --format json \
    --output "$BUNDLE_DIR/bundles/revocations.json"

# Create manifest
echo "Creating bundle manifest..."
cat > "$BUNDLE_DIR/manifest.json" <<EOF
{
    "version": "1.0.0",
    "createdAt": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
    "expiresAt": "$(date -u -d '+90 days' +%Y-%m-%dT%H:%M:%SZ)",
    "contents": {
        "trustAnchors": ["fulcio-root.pem", "rekor-pubkey.pem"],
        "bundles": ["issuers.json", "revocations.json"],
        "tufMetadata": true
    },
    "checksum": "$(find $BUNDLE_DIR -type f -exec sha256sum {} \; | sha256sum | cut -d' ' -f1)"
}
EOF

# Package bundle
echo "Creating tarball..."
tar -czvf "stellaops-trust-bundle-$(date +%Y%m%d).tar.gz" -C "$BUNDLE_DIR" .

echo "Bundle ready: stellaops-trust-bundle-$(date +%Y%m%d).tar.gz"

2. Transfer to Air-Gapped Environment

# On air-gapped machine
sudo mkdir -p /var/stellaops/{trust,bundles,cache}
sudo tar -xzvf stellaops-trust-bundle-20250128.tar.gz -C /var/stellaops/

# Verify bundle integrity
stellaops-cli bundle verify /var/stellaops/manifest.json

Issuer Directory Bundle Format

File: /var/stellaops/bundles/issuers.json

{
    "version": "1.0.0",
    "exportedAt": "2025-01-28T10:30:00Z",
    "issuers": [
        {
            "id": "redhat-security",
            "name": "Red Hat Product Security",
            "description": "Official Red Hat security advisories",
            "jurisdiction": "us",
            "trustLevel": "high",
            "keys": [
                {
                    "keyId": "rh-vex-signing-key-2024",
                    "algorithm": "ECDSA-P256",
                    "publicKey": "-----BEGIN PUBLIC KEY-----\nMFkwEwYHKoZIzj0...\n-----END PUBLIC KEY-----",
                    "notBefore": "2024-01-01T00:00:00Z",
                    "notAfter": "2026-01-01T00:00:00Z",
                    "revoked": false
                }
            ],
            "csafPublisher": {
                "providerMetadataUrl": "https://access.redhat.com/.well-known/csaf/provider-metadata.json",
                "tlpWhite": true
            }
        },
        {
            "id": "internal-security",
            "name": "Internal Security Team",
            "description": "Internal VEX attestations",
            "jurisdiction": "internal",
            "trustLevel": "high",
            "keys": [
                {
                    "keyId": "internal-vex-key-001",
                    "algorithm": "Ed25519",
                    "publicKey": "-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VwAyEA...\n-----END PUBLIC KEY-----",
                    "notBefore": "2024-06-01T00:00:00Z",
                    "notAfter": "2025-06-01T00:00:00Z",
                    "revoked": false
                }
            ]
        }
    ],
    "revokedKeys": [
        {
            "keyId": "old-compromised-key",
            "revokedAt": "2024-03-15T00:00:00Z",
            "reason": "key_compromise"
        }
    ]
}

Verification Behavior in Offline Mode

Supported Verification Methods

Fallback Behavior

VexSignatureVerification:
  OfflineFallback:
    # When Rekor is unavailable
    SkipRekorVerification: true
    WarnOnMissingTransparency: true

    # When issuer key not in bundle
    UnknownIssuerAction: "warn"  # warn | block | allow

    # When certificate chain incomplete
    IncompleteChainAction: "warn"

Verification Result Fields

{
    "verified": true,
    "method": "dsse",
    "mode": "offline",
    "warnings": [
        "transparency_log_skipped"
    ],
    "issuerName": "Red Hat Product Security",
    "keyId": "rh-vex-signing-key-2024",
    "bundleVersion": "2025.01.28",
    "bundleAge": "P3D"
}

Bundle Updates

Manual Update Process

1. Export new bundle on connected machine
2. Transfer via secure media (USB, CD)
3. Verify bundle signature on air-gapped machine
4. Deploy with rollback capability

# On air-gapped machine
cd /var/stellaops

# Backup current bundle
sudo cp -r bundles bundles.backup-$(date +%Y%m%d)

# Deploy new bundle
sudo tar -xzvf new-bundle.tar.gz -C /tmp/new-bundle
sudo stellaops-cli bundle verify /tmp/new-bundle/manifest.json

# Apply with verification
sudo stellaops-cli bundle apply /tmp/new-bundle --verify
sudo systemctl restart stellaops-excititor-web

# Rollback if needed
# sudo stellaops-cli bundle rollback --to bundles.backup-20250115

Recommended Update Frequency

Monitoring and Alerts

Bundle Expiration Warning

# prometheus-alerts.yaml
groups:
  - name: stellaops-verification
    rules:
      - alert: TrustBundleExpiringSoon
        expr: stellaops_trust_bundle_expiry_days < 30
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Trust bundle expires in {{ $value }} days"

      - alert: TrustBundleExpired
        expr: stellaops_trust_bundle_expiry_days <= 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Trust bundle has expired - verification may fail"

Metrics

Troubleshooting

Common Issues

1. “Unknown issuer” for known vendor

   • Update issuer directory bundle
   • Add vendor’s keys to bundle

2. “Expired certificate” for recent VEX

   • Certificate may have rotated after bundle export
   • Update trust anchors bundle

3. “Chain validation failed”

   • Missing intermediate certificate
   • Add intermediate to bundle

4. Stale revocation data

   • Key may be compromised but bundle doesn’t know
   • Update revocation bundle urgently

See Also

• Air-Gap Mode Guide — sealed-mode activation and supported deployment scope.
• Bootstrap Pack — priming a sealed environment with trust roots and time-anchor bundles.
• Offline Bundle Format — canonical bundle layout and manifest fields.
• Time-Anchor Trust Roots — managing the trust roots used during offline verification.