Registry Referrer Discovery Troubleshooting

Audience: platform operators and SREs running Stella Ops mirror-bundle exports. Modules: Export Center, Air-Gap.

This runbook helps you diagnose and resolve OCI referrer discovery issues during mirror-bundle exports — when referrers (SBOMs, attestations, signatures) attached to a container image are not discovered, are only partially discovered, or fail validation on import. Start at Quick Reference to map a symptom to a likely cause, then drop into Diagnosing Issues for the detailed checks.

Quick Reference

Registry Compatibility Quick Reference

See Registry Compatibility Matrix for detailed information.

Diagnosing Issues

1. Check Export Logs

Look for capability probing and discovery logs:

# Look for probing logs
grep "Probing.*registries for OCI referrer" /var/log/stellaops/export-center.log

# Check individual registry results
grep "Registry.*OCI 1" /var/log/stellaops/export-center.log

# Example output:
# [INFO] Probing 2 registries for OCI referrer capabilities before export
# [INFO] Registry gcr.io: OCI 1.1 (referrers API supported, version=OCI-Distribution/2.1, probe_ms=42)
# [WARN] Registry ghcr.io: OCI 1.0 (using fallback tag discovery, version=registry/2.0, probe_ms=85)

2. Check Telemetry Metrics

Query Prometheus for referrer discovery metrics:

# Capability probes by registry and support status
# api_supported is the lowercased bool string "true" | "false"
sum by (registry, api_supported) (
  rate(export_registry_capabilities_probed_total[5m])
)

# Discovery method breakdown (method = "native" | "fallback")
sum by (registry, method) (
  rate(export_referrer_discovery_method_total[5m])
)

# Referrers discovered by artifact type (artifact_type = sbom|vex|attestation|other|unknown)
sum by (registry, artifact_type) (
  rate(export_referrers_discovered_total[5m])
)

# Failure rate by registry (tagged with error_type)
sum by (registry, error_type) (
  rate(export_referrer_discovery_failures_total[5m])
)

All four counters are emitted by OciReferrerDiscoveryService in StellaOps.ExportCenter.WebService (meter StellaOps.ExportCenter). The probe itself is delegated to OciReferrerFallback.ProbeCapabilitiesAsync, but the metric .Add() calls all live in OciReferrerDiscoveryService — OciReferrerFallback emits no telemetry of its own.

3. Test Registry Connectivity

Manually probe registry capabilities:

# Test OCI referrers API (OCI 1.1)
curl -H "Accept: application/vnd.oci.image.index.v1+json" \
  "https://registry.example.com/v2/myrepo/referrers/sha256:abc123..."

# Expected responses (how Stella Ops interprets them):
# - 200 OK with manifest list: Registry supports referrers API (native discovery)
# - 404 Not Found with an OCI image-index body: API supported, just no referrers for this digest
# - 404 Not Found without an OCI index body: API NOT supported -> tag-based fallback
# - 405 Method Not Allowed: API NOT supported -> tag-based fallback
# - 501 Not Implemented: API NOT supported -> tag-based fallback
#
# NB: OciReferrerFallback.ProbeReferrersApiAsync treats only 501 / 405 as
# "not implemented" during the lightweight capability probe (it probes with a
# zero-digest against a synthetic repo). The doctor check below additionally
# inspects 404 bodies to distinguish "no referrers" from "no API".

# Check distribution version
curl -I "https://registry.example.com/v2/"
# Look for: OCI-Distribution-API-Version (or legacy Docker-Distribution-API-Version) header
# OCI 1.1+ in the version string => SupportsReferrersApi is short-circuited to true

4. Test Fallback Tag Discovery

If native API is not supported:

# List tags matching fallback pattern
curl "https://registry.example.com/v2/myrepo/tags/list" | \
  jq '.tags | map(select(startswith("sha256-")))'

# Expected: Tags like "sha256-abc123.sbom", "sha256-abc123.att"

Common Issues and Solutions

Issue: “Failed to probe capabilities for registry”

Symptoms:

• Warning logs about probe failures
• Referrer discovery using fallback or skipped

Causes:

1. Network connectivity issues
2. Authentication failures
3. Registry rate limiting
4. TLS certificate issues

Solutions:

# Check network connectivity
curl -v "https://registry.example.com/v2/"

# Verify authentication
docker login registry.example.com

# Check TLS certificates
openssl s_client -connect registry.example.com:443 -servername registry.example.com

These probe/auth failures are about the registry credentials used by the export worker (configured via OCI:* / Registry:* settings), not Stella’s own Authority scopes. For reference, the ExportCenter API scopes are export.viewer (read runs/bundles), export.operator (run exports), and export.admin; mirror import while sealed requires airgap:import. (Source: StellaOps.Auth.Abstractions/StellaOpsScopes.cs.)

Issue: “No referrers found for image”

Symptoms:

• Discovery succeeds but returns empty list
• Bundle missing expected SBOMs/attestations

Causes:

1. No referrers actually attached to image
2. Referrers attached to different digest (tag vs digest mismatch)
3. Referrers pruned by registry retention policy

Solutions:

# Verify referrers exist for the specific digest
crane manifest registry.example.com/repo@sha256:abc123 | \
  jq '.subject.digest'

# List referrers using oras
oras discover registry.example.com/repo@sha256:abc123

# Check if referrers exist with different artifact types
curl "https://registry.example.com/v2/repo/referrers/sha256:abc123?artifactType=application/vnd.cyclonedx%2Bjson"

Issue: “Referrer checksum mismatch during import”

Symptoms:

• The import fails and is quarantined with reason referrer-validation-failed:checksum_mismatch=N
• ImportValidator logs offlinekit.import.validation failed ... reason_code=REFERRER_VALIDATION_FAILED ... checksum_mismatch=N
• Bundle verification fails

How it is detected:

• ReferrerValidator.Validate (called by ImportValidator only for mirror-bundle and offline-kit bundle types) recomputes SHA-256 over each declared referrer artifact and compares it to the sha256 field in the manifest referrers section.
• A mismatch produces a ReferrerChecksumMismatch issue (Error severity); any Error (ReferrerMissing, ReferrerChecksumMismatch, or ReferrerSizeMismatch) fails the import. OrphanedReferrer is a Warning only and does not fail the import.

Causes:

1. Referrer artifact modified after export
2. Registry replaced artifact
3. Bundle corruption during transfer

Solutions:

1. Re-export the bundle to get fresh referrer content
2. Verify bundle integrity: sha256sum bundle.tgz
3. Check if referrer was intentionally updated upstream
4. Inspect the quarantine record (the failed bundle is moved to quarantine with the full verification log) to see the expected vs. actual digest

Issue: Harbor UI shows referrers as “UNKNOWN” artifact type

Symptoms:

• Referrer artifacts (cosign signatures, SBOMs) appear as “UNKNOWN” in Harbor UI
• API-level discovery works correctly

Causes:

1. Harbor UI mediaType classification lags API capabilities (especially around v2.15+)
2. Custom artifact types not recognized by Harbor’s UI layer

Solutions:

• This is a Harbor-side UI classification issue; it does not affect Stella Ops referrer discovery or functionality
• Verify API-level discovery works: curl -H "Accept: application/vnd.oci.image.index.v1+json" "https://harbor.example.com/v2/repo/referrers/sha256:..."
• If needed, check Harbor release notes for mediaType classification updates

Issue: Quay referrers API returns inconsistent results

Symptoms:

• Referrer discovery works on Quay.io but not on self-hosted Quay
• Intermittent 404 or empty results from referrers endpoint

Causes:

1. OCI Referrers API feature not enabled in self-hosted Quay deployment
2. Quay admin toggles or deployment flags controlling the feature

Solutions:

• Verify the OCI Referrers API feature is enabled in Quay’s deployment configuration
• Check Quay admin console for referrers-related feature flags
• If the feature is disabled, Stella Ops automatically uses tag-based fallback; no action required
• Contact Quay administrator to enable the feature if native referrers discovery is preferred

Issue: Slow referrer discovery

Symptoms:

• Export takes much longer than expected
• Timeout warnings in logs

Causes:

1. Large number of referrers per image
2. Slow registry responses
3. No capability caching (cache miss)

Solutions:

# Increase timeout in export config
export:
  referrer_discovery:
    timeout_seconds: 120
    max_concurrent_discoveries: 4

Validation Commands

Inspect Bundle Referrers

The bundle manifest is JSON (manifest.json) with a referrers section (referrers.subjects[].artifacts[], each carrying digest, path, sha256, size, and optional artifactType). Referrer payloads live under referrers/.

# List referrer payloads in the bundle
tar -tzf bundle.tgz | grep "^referrers/"

# Dump the referrers section of the manifest
tar -xzf bundle.tgz -O manifest.json | jq '.referrers'

# Recompute a specific referrer checksum and compare to the declared sha256
tar -xzf bundle.tgz -O referrers/sha256-abc123/sha256-def456.json | sha256sum

CLI Validation

There is no dedicated --check-referrers flag; referrer validation runs automatically during import for mirror-bundle and offline-kit bundle types (see the checksum-mismatch issue above). The operator-facing commands are:

# Probe whether the configured registry supports the OCI 1.1 referrers API.
# Requires OCI:RegistryUrl (or Registry:Url) to be configured; optional
# OCI:TestRepository (default library/alpine) and OCI:TestTag (default latest).
# (Source: RegistryReferrersApiCheck, CheckId "check.integration.oci.referrers".)
stella doctor --check check.integration.oci.referrers

# Broader registry capability matrix (referrers API + chunked upload, cross-repo
# mount, manifest/blob delete) reported as a capability_score like "3/5".
# (Source: RegistryCapabilityProbeCheck, CheckId "check.integration.oci.capabilities".)
stella doctor --check check.integration.oci.capabilities

# Verify an exported evidence bundle. Checks the manifest, artifact checksums,
# DSSE signatures, Rekor inclusion proofs, expected payload types, and (with
# --replay) large-blob content. NB: DSSE signatures are only *cryptographically*
# verified when --trust-root is supplied; without it they are presence-checked
# and reported as a warning. --strict fails on any warning (e.g. missing optional
# artifacts); --offline runs with no network access.
# (Source: BundleVerifyCommand, the `verify` subcommand of `stella bundle`.)
stella bundle verify --bundle bundle.tar.gz --strict
stella bundle verify --bundle bundle.tar.gz --offline

Note: the CLI binary is invoked as stella. There is no stella bundle validate or stella bundle import subcommand; the only stella bundle subcommand is verify (registered by BundleCommandGroup). An evidence-bundle verify is also reachable as stella verify bundle, a separate command that does not run referrer validation. Mirror-bundle import (which triggers referrer validation) is performed by the AirGap importer, not a bundle import CLI verb.

Escalation

If issues persist after following this runbook:

1. Run the built-in registry diagnostics and capture their evidence blocks:

   • stella doctor --check check.integration.oci.referrers (reports registry_url, api_endpoint, http_status, oci_version, referrers_supported, and fallback_required; emits a Warn result with a fallback_pattern of sha256-{digest}.{artifactType} when the API is absent)
   • stella doctor --check check.integration.oci.capabilities (broader matrix: supports_referrers_api, supports_chunked_upload, supports_cross_repo_mount, supports_manifest_delete, supports_blob_delete, and an aggregate capability_score)

2. Collect diagnostic information:

   • Export logs with DEBUG level enabled
   • Telemetry metrics for the affected time window
   • Registry type and version
   • Network trace if applicable

3. Search the issue tracker for known issues tagged referrer-discovery.

4. Open a support ticket with:

   • Environment details (Stella Ops version, registry type)
   • Error messages and logs
   • Steps to reproduce

Related Documentation

• Export Center Architecture
• Registry Compatibility Matrix
• Offline Bundle Format