Surface.FS Workflow Guide

Currency note (2026-09-26, commit 942a18727c). The Scheduler service is retired (its scheduling moved to JobEngine; hosts frozen under src/__Obsoleted/JobEngine/) and the Zastava observer/webhook is retired (runtime evidence now comes from RuntimeInstrumentation agents through POST /api/v1/runtime/events). Sections that name Scheduler or Zastava as consumers describe the retired design. The Surface libraries themselves are live under src/Scanner/__Libraries/StellaOps.Scanner.Surface*; Scanner-side paths were checked, deep content was not re-verified.

Version: 1.0 (2025-11-28) · Audience: Scanner Worker/WebService integrators, Zastava operators, Offline Kit builders

Overview

Surface.FS is the content-addressable storage layer that holds Scanner-derived artefacts (layer fragments, EntryTrace graphs, SBOM fragments) in Stella Ops. This guide walks the end-to-end workflow from artefact generation through storage to consumption, and covers how the same artefacts move through an air-gapped Offline Kit.

Workflow Stages

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Scanner Worker │───▶│  Surface.FS     │───▶│  Consumers      │
│  - Scan image   │    │  - Store manifest│   │  - WebService   │
│  - Generate     │    │  - Store payload │   │  - Zastava      │
│    artefacts    │    │  - Local cache   │   │  - CLI          │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                      │                      │
         ▼                      ▼                      ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Generate:      │    │  Store:         │    │  Consume:       │
│  - Layer frags  │    │  - RustFS/S3    │    │  - Report API   │
│  - EntryTrace   │    │  - Local disk   │    │  - Drift detect │
│  - SBOM frags   │    │  - Offline kit  │    │  - Rescan plan  │
└─────────────────┘    └─────────────────┘    └─────────────────┘

Stage 1: Artefact Generation (Scanner Worker)

1.1 Configure Surface.FS

// In Scanner Worker startup
builder.Services.AddSurfaceFileCache();
builder.Services.AddSurfaceManifestStore();

Environment variables (see Surface.Env guide):

SCANNER_SURFACE_FS_ENDPOINT=http://rustfs:8080
SCANNER_SURFACE_FS_BUCKET=surface-cache
SCANNER_SURFACE_CACHE_ROOT=/var/lib/stellaops/surface
SCANNER_SURFACE_TENANT=default

1.2 Generate and Publish Artefacts

public async Task<ScanResult> ExecuteScanAsync(ScanJob job, CancellationToken ct)
{
    // 1. Run analyzers to generate artefacts
    var layerFragments = await AnalyzeLayersAsync(job.Image, ct);
    var entryTrace = await AnalyzeEntryPointsAsync(job.Image, ct);
    var sbomFragments = await GenerateSbomAsync(job.Image, ct);

    // 2. Create manifest document
    var manifest = new SurfaceManifestDocument
    {
        Schema = "stellaops.surface.manifest@1",
        Tenant = _environment.Settings.Tenant,
        ImageDigest = job.Image.Digest,
        ScanId = job.Id,
        GeneratedAt = DateTimeOffset.UtcNow,
        Source = new SurfaceManifestSource
        {
            Component = "scanner.worker",
            Version = _version,
            WorkerInstance = Environment.MachineName,
            Attempt = job.Attempt
        },
        Artifacts = new List<SurfaceManifestArtifact>()
    };

    // 3. Add artefacts to manifest
    foreach (var fragment in layerFragments)
    {
        var payloadUri = await _manifestWriter.StorePayloadAsync(
            fragment.Content,
            "layer.fragments",
            ct);

        manifest.Artifacts.Add(new SurfaceManifestArtifact
        {
            Kind = "layer.fragments",
            Uri = payloadUri,
            Digest = fragment.Digest,
            MediaType = "application/vnd.stellaops.layer-fragments+json",
            Format = "json",
            SizeBytes = fragment.Content.Length
        });
    }

    // 4. Publish manifest
    var result = await _manifestWriter.PublishAsync(manifest, ct);

    _logger.LogInformation(
        "Published manifest {Digest} with {Count} artefacts",
        result.Digest,
        manifest.Artifacts.Count);

    return new ScanResult
    {
        ManifestUri = result.Uri,
        ManifestDigest = result.Digest
    };
}

1.3 Cache EntryTrace Results

public async Task<EntryTraceGraph?> GetOrComputeEntryTraceAsync(
    ImageReference image,
    EntryTraceOptions options,
    CancellationToken ct)
{
    // Create deterministic cache key
    var cacheKey = new SurfaceCacheKey(
        @namespace: "entrytrace.graph",
        tenant: _environment.Settings.Tenant,
        digest: ComputeOptionsHash(options, image.Digest));

    // Try cache first
    var cached = await _cache.TryGetAsync<EntryTraceGraph>(cacheKey, ct);
    if (cached is not null)
    {
        _logger.LogDebug("EntryTrace cache hit for {Key}", cacheKey);
        return cached;
    }

    // Compute and cache
    var graph = await ComputeEntryTraceAsync(image, options, ct);
    await _cache.SetAsync(cacheKey, graph, ct);

    return graph;
}

Stage 2: Storage (Surface.FS)

2.1 Manifest Storage Layout

<bucket>/
├── manifests/
│   └── <tenant>/
│       └── <digest[0..1]>/
│           └── <digest[2..3]>/
│               └── <digest>.json
└── payloads/
    └── <tenant>/
        └── <kind>/
            └── sha256/
                └── <digest[0..1]>/
                    └── <digest[2..3]>/
                        └── <digest>.json.zst

2.2 Local Cache Layout

<cache_root>/
├── manifests/          # Manifest JSON files
│   └── <tenant>/...
├── cache/              # Hot artefacts
│   └── <namespace>/
│       └── <tenant>/
│           └── <digest>
└── temp/               # In-progress writes

2.3 Manifest URI Format

cas://<bucket>/<prefix>/<tenant>/<digest[0..1]>/<digest[2..3]>/<digest>.json

Example:

cas://surface-cache/manifests/acme/ab/cd/abcdef0123456789...json

Stage 3: Consumption

3.1 WebService API

GET /api/v1/scans/{id}

Response includes Surface manifest pointer:

{
  "id": "scan-1234",
  "status": "completed",
  "surface": {
    "manifestUri": "cas://surface-cache/manifests/acme/ab/cd/...",
    "manifestDigest": "sha256:abcdef...",
    "artifacts": [
      {
        "kind": "layer.fragments",
        "uri": "cas://surface-cache/payloads/acme/layer.fragments/...",
        "digest": "sha256:123456...",
        "mediaType": "application/vnd.stellaops.layer-fragments+json"
      }
    ]
  }
}

3.2 Zastava Drift Detection

public async Task<DriftResult> DetectDriftAsync(
    string imageDigest,
    CancellationToken ct)
{
    // 1. Fetch baseline manifest
    var manifestUri = await _surfacePointerService.GetManifestUriAsync(imageDigest, ct);
    var manifest = await _manifestReader.TryGetByUriAsync(manifestUri, ct);

    if (manifest is null)
    {
        return DriftResult.NoBaseline();
    }

    // 2. Get EntryTrace artefact
    var entryTraceArtifact = manifest.Artifacts
        .FirstOrDefault(a => a.Kind == "entrytrace.graph");

    if (entryTraceArtifact is null)
    {
        return DriftResult.NoEntryTrace();
    }

    // 3. Compare with runtime
    var baseline = await _payloadStore.GetAsync<EntryTraceGraph>(
        entryTraceArtifact.Uri, ct);

    var runtime = await _runtimeCollector.CollectAsync(ct);

    return CompareGraphs(baseline, runtime);
}

3.3 Scheduler Rescan Planning

public async Task<RescanPlan> CreateRescanPlanAsync(
    string imageDigest,
    CancellationToken ct)
{
    // 1. Read manifest to understand what was scanned
    var manifest = await _manifestReader.TryGetByDigestAsync(imageDigest, ct);

    if (manifest is null || IsExpired(manifest))
    {
        return RescanPlan.FullRescan();
    }

    // 2. Check for layer changes
    var layerArtifact = manifest.Artifacts
        .FirstOrDefault(a => a.Kind == "layer.fragments");

    if (layerArtifact is not null)
    {
        var layers = await _payloadStore.GetAsync<LayerFragments>(
            layerArtifact.Uri, ct);

        var changedLayers = await DetectChangedLayersAsync(layers, ct);

        if (changedLayers.Any())
        {
            return RescanPlan.IncrementalRescan(changedLayers);
        }
    }

    return RescanPlan.NoRescanNeeded();
}

Offline Kit Workflow

Export (Online Environment)

# 1. Build offline kit with Surface manifests
python ops/offline-kit/build_offline_kit.py \
  --version 2025.10.0 \
  --include-surface-manifests \
  --output-dir out/offline-kit

# 2. Kit structure includes:
# offline/
#   surface/
#     manifests/
#       <tenant>/<digest[0..1]>/<digest[2..3]>/<digest>.json
#     payloads/
#       <tenant>/<kind>/sha256/<digest[0..1]>/<digest[2..3]>/<digest>.json.zst
#     manifest-index.json

Import (Air-Gapped Environment)

public async Task ImportOfflineKitAsync(
    string kitPath,
    CancellationToken ct)
{
    var surfacePath = Path.Combine(kitPath, "surface");
    var indexPath = Path.Combine(surfacePath, "manifest-index.json");

    var index = await LoadIndexAsync(indexPath, ct);

    foreach (var entry in index.Manifests)
    {
        // 1. Load and verify manifest
        var manifestPath = Path.Combine(surfacePath, entry.RelativePath);
        var manifest = await LoadManifestAsync(manifestPath, ct);

        // 2. Verify digest
        var computedDigest = ComputeDigest(manifest);
        if (computedDigest != entry.Digest)
        {
            throw new InvalidOperationException(
                $"Manifest digest mismatch: expected {entry.Digest}, got {computedDigest}");
        }

        // 3. Import via Surface.FS API
        await _manifestWriter.PublishAsync(manifest, ct);

        _logger.LogInformation(
            "Imported manifest {Digest} for image {Image}",
            entry.Digest,
            manifest.ImageDigest);
    }

    // 4. Import payloads
    foreach (var payload in index.Payloads)
    {
        var payloadPath = Path.Combine(surfacePath, payload.RelativePath);
        await _payloadStore.ImportAsync(payloadPath, payload.Uri, ct);
    }
}

Offline Operation

Once imported, Surface.FS consumers operate normally:

// Same code works online and offline
var manifest = await _manifestReader.TryGetByUriAsync(manifestUri, ct);
var payload = await _payloadStore.GetAsync(artifact.Uri, ct);

Configuration Reference

SurfaceManifestStoreOptions

SurfaceCacheOptions

The shipped roles select /var/lib/stellaops/surface and mount the shared scanner-surface-cache volume there. SCANNER_SURFACE_CACHE_QUOTA_MB belongs to surface environment validation; it does not configure automatic file eviction. See the physical measurement procedure.

Metrics

The worker’s ScannerWorkerMetrics emits scanner_worker_surface_manifests_published_total, scanner_worker_surface_manifests_skipped_total, scanner_worker_surface_manifests_failed_total, scanner_worker_surface_manifest_publish_duration_ms and scanner_worker_surface_payload_persisted_total. FileSurfaceCache reports hits through trace logs; it emits no surface_manifest_cache_hit_total metric.

Troubleshooting

Manifest Not Found

1. Check tenant matches between writer and reader
2. Verify the configured manifest directory exists and is readable
3. Check file permissions under that directory
4. Review scanner_worker_surface_manifests_published_total and the worker publication logs

Cache Miss Despite Expected Hit

1. Verify cache key components match (namespace, tenant, digest)
2. Check the resolved cache directory and any operator retention actions
3. Review FileSurfaceCache hit/miss trace logs and payload deserialization errors

Offline Import Failures

1. Verify manifest digest matches index
2. Check file permissions on import path
3. Ensure the configured local destination directory is writable
4. Review import logs for specific errors

References

• Surface.FS Design
• Surface.Env Design
• Surface.Validation Extensibility Guide
• Offline Kit Documentation