Federation Setup and Operations

Per SPRINT_8200_0014_0003.

Related: Bundle Export Format for detailed bundle schema.

Overview

Federation enables secure, cursor-based synchronization of canonical vulnerability advisories between StellaOps sites. It supports:

Delta exports: Only changed records since the last cursor are included
Air-gap transfers: Bundles can be written to files for offline transfer
Multi-site topology: Multiple sites can synchronize independently
Cryptographic verification: DSSE signatures ensure bundle authenticity

Bundle Format

Federation bundles are ZST-compressed TAR archives containing:

File	Description
`MANIFEST.json`	Bundle metadata, cursor, counts, hash
`canonicals.ndjson`	Canonical advisories (one per line)
`edges.ndjson`	Source edges linking advisories to sources
`deletions.ndjson`	Withdrawn/deleted advisory IDs
`SIGNATURE.json`	Optional DSSE signature envelope

Cursor Format

Cursors use ISO-8601 timestamp with sequence number:

{ISO-8601 timestamp}#{sequence number}

Examples:
2025-01-15T10:00:00.000Z#0001
2025-01-15T10:00:00.000Z#0002

Cursors are site-specific (each site maintains independent cursors)
Sequence numbers distinguish concurrent exports
Cursors are monotonically increasing within a site

Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                            Federation Topology                              │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│    ┌──────────────────┐           ┌──────────────────┐                     │
│    │  Site A (HQ)     │           │  Site B (Branch) │                     │
│    │                  │           │                  │                     │
│    │  ┌────────────┐  │  Export   │  ┌────────────┐  │                     │
│    │  │ Concelier  │──┼──────────►│  │ Concelier  │  │                     │
│    │  │            │  │  Bundle   │  │            │  │                     │
│    │  └────────────┘  │           │  └────────────┘  │                     │
│    │        │         │           │        │         │                     │
│    │        ▼         │           │        ▼         │                     │
│    │  ┌────────────┐  │           │  ┌────────────┐  │                     │
│    │  │ PostgreSQL │  │           │  │ PostgreSQL │  │                     │
│    │  └────────────┘  │           │  └────────────┘  │                     │
│    └──────────────────┘           └──────────────────┘                     │
│                                                                             │
│    ┌──────────────────┐                                                     │
│    │  Site C (Air-Gap)│                                                     │
│    │                  │                                                     │
│    │  ┌────────────┐  │   USB/Secure                                       │
│    │  │ Concelier  │◄─┼───Transfer                                         │
│    │  │            │  │                                                     │
│    │  └────────────┘  │                                                     │
│    └──────────────────┘                                                     │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Setup

1. Enable Federation

Configure federation in concelier.yaml:

Federation:
  Enabled: true
  SiteId: "site-us-west-1"  # Unique identifier for this site
  DefaultCompressionLevel: 3
  DefaultMaxItems: 10000
  RequireSignature: true

FederationImport:
  AllowedSites:
    - "site-us-east-1"
    - "site-eu-central-1"
  MaxBundleSizeBytes: 104857600  # 100 MB
  SkipSignatureOnTrustedSites: false

2. Configure Site Policies

Create site policies for each trusted federation partner:

# Add trusted site
stella feedser sites add site-us-east-1 \
  --display-name "US East Production" \
  --enabled

# Configure policy
stella feedser sites policy site-us-east-1 \
  --max-bundle-size 100MB \
  --allowed-sources nvd,ghsa,debian

3. Generate Signing Keys

For signed bundles, configure Authority keys:

# Generate federation signing key
stella authority keys generate \
  --name federation-signer \
  --algorithm ES256 \
  --purpose federation

# Export public key for distribution
stella authority keys export federation-signer --public

Import Operations

API Import

POST /api/v1/federation/import
Content-Type: application/zstd

Query Parameters:

Parameter	Type	Default	Description
`dry_run`	bool	false	Validate without importing
`skip_signature`	bool	false	Skip signature verification (requires trust)
`on_conflict`	enum	prefer_remote	`prefer_remote`, `prefer_local`, `fail`
`force`	bool	false	Import even if cursor is not after current

Response:

{
  "success": true,
  "bundle_hash": "sha256:a1b2c3...",
  "imported_cursor": "2025-01-15T10:30:00.000Z#0042",
  "counts": {
    "canonical_created": 100,
    "canonical_updated": 25,
    "canonical_skipped": 10,
    "edges_added": 200,
    "deletions_processed": 5
  },
  "conflicts": [],
  "duration_ms": 1234
}

CLI Import

# Import from file
stella feedser bundle import ./bundle.zst

# Import with dry run
stella feedser bundle import ./bundle.zst --dry-run

# Import from stdin (for pipes)
cat bundle.zst | stella feedser bundle import -

# Import without signature verification (testing only)
stella feedser bundle import ./bundle.zst --skip-signature

# Force import (override cursor check)
stella feedser bundle import ./bundle.zst --force

Conflict Resolution

When conflicts occur between local and remote values:

Strategy	Behavior
`prefer_remote`	Remote value wins (default)
`prefer_local`	Local value preserved
`fail`	Abort import on first conflict

Conflicts are logged with full details:

{
  "merge_hash": "sha256:abc...",
  "field": "severity",
  "local_value": "high",
  "remote_value": "critical",
  "resolution": "prefer_remote"
}

Site Management

List Sites

stella feedser sites list

Output:

SITE ID                    STATUS    LAST SYNC            CURSOR
─────────────────────────  ────────  ───────────────────  ──────────────────────────
site-us-east-1             enabled   2025-01-15 10:30     2025-01-15T10:30:00Z#0042
site-eu-central-1          enabled   2025-01-15 09:15     2025-01-15T09:15:00Z#0038
site-asia-pacific-1        disabled  never                -

View Site History

stella feedser sites history site-us-east-1 --limit 10

Update Site Policy

stella feedser sites policy site-us-east-1 \
  --enabled false  # Disable imports from this site

Air-Gap Operations

For sites without network connectivity:

Export for Transfer

# On connected site
stella feedser bundle export \
  -c "2025-01-14T00:00:00Z#0000" \
  -o ./delta-2025-01-15.zst

# Transfer via USB/secure media

Import on Air-Gap Site

# On air-gapped site
stella feedser bundle import ./delta-2025-01-15.zst

# Verify import
stella feedser sites list

Full Sync Workflow

Initial Sync:

# Export full dataset
stella feedser bundle export -o ./full-sync.zst

Transfer to air-gap site

Import on air-gap:

stella feedser bundle import ./full-sync.zst

Subsequent Delta Syncs:

# Get current cursor from air-gap site
stella feedser sites list  # Note the cursor

# On connected site, export delta
stella feedser bundle export -c "{cursor}" -o ./delta.zst

# Transfer and import on air-gap

Verification

Validate Bundle Without Import

stella feedser bundle validate ./bundle.zst

Output:

Bundle: bundle.zst
  Version: feedser-bundle/1.0
  Site: site-us-east-1
  Cursor: 2025-01-15T10:30:00.000Z#0042

Counts:
  Canonicals: 1,234
  Edges: 3,456
  Deletions: 12
  Total: 4,702

Verification:
  Hash: ✓ Valid
  Signature: ✓ Valid (key: sha256:abc...)
  Format: ✓ Valid

Ready for import.

Preview Import Impact

stella feedser bundle import ./bundle.zst --dry-run --json

Monitoring

Sync Status Endpoint

GET /api/v1/federation/sync/status

Response:

{
  "sites": [
    {
      "site_id": "site-us-east-1",
      "enabled": true,
      "last_sync_at": "2025-01-15T10:30:00Z",
      "last_cursor": "2025-01-15T10:30:00.000Z#0042",
      "bundles_imported": 156,
      "total_items_imported": 45678
    }
  ],
  "local_cursor": "2025-01-15T10:35:00.000Z#0044"
}

Event Stream

Import events are published to the canonical-imported stream:

{
  "canonical_id": "uuid",
  "cve": "CVE-2024-1234",
  "affects_key": "pkg:npm/express@4.0.0",
  "merge_hash": "sha256:...",
  "action": "Created",
  "bundle_hash": "sha256:...",
  "site_id": "site-us-east-1",
  "imported_at": "2025-01-15T10:30:15Z"
}

Cache Invalidation

After import, cache indexes are automatically updated:

PURL index updated for affected packages
CVE index updated for vulnerability lookups
Existing cache entries invalidated for refresh

Troubleshooting

Common Issues

Issue	Cause	Solution
“Cursor not after current”	Bundle is stale	Use `--force` or export newer bundle
“Signature verification failed”	Key mismatch	Verify signing key is trusted
“Site not allowed”	Policy restriction	Add site to `AllowedSites` config
“Bundle too large”	Size limit exceeded	Increase `MaxBundleSizeBytes` or export smaller delta

Debug Logging

Enable verbose logging for federation operations:

Logging:
  LogLevel:
    StellaOps.Concelier.Federation: Debug

Verify Sync State

# Check local vs remote cursor
stella feedser sites status site-us-east-1

# List recent imports
stella feedser sites history site-us-east-1 --limit 5

# Verify specific canonical was imported
stella feedser canonical get sha256:mergehash...

Best Practices

Regular Sync Schedule: Configure automated delta exports/imports on a schedule (e.g., hourly)
Monitor Cursor Drift: Alert if cursor falls too far behind
Verify Signatures: Only disable signature verification in development
Size Bundles Appropriately: For large deltas, split into multiple bundles
Test Import Before Production: Use --dry-run to validate bundles
Maintain Key Trust: Regularly rotate and verify federation signing keys
Document Site Policies: Keep a registry of trusted sites and their policies

Multi-Site Topologies

Hub-and-Spoke Topology

           ┌─────────────┐
           │   Hub Site  │
           │  (Primary)  │
           └──────┬──────┘
                  │
       ┌──────────┼──────────┐
       ▼          ▼          ▼
 ┌──────────┐ ┌──────────┐ ┌──────────┐
 │  Site A  │ │  Site B  │ │  Site C  │
 │ (Spoke)  │ │ (Spoke)  │ │ (Spoke)  │
 └──────────┘ └──────────┘ └──────────┘

Mesh Topology

Each site can import from multiple sources for redundancy:

federation:
  import:
    allowed_sites:
      - "hub-primary"
      - "hub-secondary"  # Redundancy

Verification Details

Hash Verification

Bundle hash is computed over compressed content:

SHA256(compressed bundle content)

DSSE Signature Format

DSSE envelope contains:

{
  "payloadType": "application/stellaops.federation.bundle+json",
  "payload": "base64(bundle_hash + site_id + cursor)",
  "signatures": [
    {
      "keyId": "signing-key-001",
      "algorithm": "ES256",
      "signature": "base64(signature)"
    }
  ]
}

Monitoring Metrics

Key Prometheus Metrics

federation_export_duration_seconds - Export time
federation_import_duration_seconds - Import time
federation_bundle_size_bytes - Bundle sizes
federation_items_processed_total - Items processed by type
federation_conflicts_total - Merge conflicts encountered

Security Considerations

Never skip signature verification in production
Validate allowed_sites whitelist
Use TLS for API endpoints
Rotate signing keys periodically
Audit import events
Monitor for duplicate bundle imports

Snapshot Pinning and Rollback

Sprint: SPRINT_20260208_035_Concelier_feed_snapshot_coordinator

Overview

Snapshot pinning provides cross-instance coordination for federated deployments. It ensures that:

All federated sites can synchronize to a common snapshot version
Failed imports are automatically rolled back to the previous stable state
Concurrent snapshot operations are detected and prevented

Services

The following services are registered by AddConcelierFederationServices():

Service	Description
`IFeedSnapshotPinningService`	Low-level snapshot pinning using SyncLedgerRepository
`ISnapshotIngestionOrchestrator`	High-level orchestration with automatic rollback

Automatic Rollback on Import Failure

When importing a snapshot bundle, the ISnapshotIngestionOrchestrator provides:

Lock acquisition - Prevents concurrent operations on the same source
Conflict detection - Checks for cursor conflicts before proceeding
Pin-before-import - Pins the snapshot ID before import begins
Automatic rollback - On import failure, automatically reverts to previous state

// Example usage in application code
var result = await orchestrator.ImportWithRollbackAsync(
    inputStream,
    importOptions,
    sourceId,
    cancellationToken);

if (!result.Success)
{
    if (result.WasRolledBack)
    {
        _logger.LogWarning(
            "Import failed but rolled back to {SnapshotId}",
            result.RolledBackToSnapshotId);
    }
}

API Endpoints

The snapshot pinning service is available through the existing feed snapshot endpoints:

POST /api/v1/feeds/snapshot/import

When the orchestrator is used, the response includes rollback information:

{
  "success": false,
  "error": "Import failed: invalid bundle format",
  "was_rolled_back": true,
  "rolled_back_to_snapshot_id": "snapshot-2024-001"
}

Configuration

Snapshot pinning uses the same FederationOptions as other federation features:

Federation:
  Enabled: true
  SiteId: "site-us-west-1"  # Required for pinning coordination

Monitoring

Key metrics for snapshot pinning:

snapshot_pin_success_total - Successful pin operations
snapshot_pin_failure_total - Failed pin operations
snapshot_rollback_total - Rollback operations triggered
snapshot_conflict_total - Conflict detections