Runbook: Scanner — Registry Authentication Failures

Use this runbook when the Stella Ops scanner worker cannot authenticate to a container registry — for example, private images failing with 401 Unauthorized or producing zero non-empty layers while public images scan fine. It is written for the Platform and Security teams.

The essential mental model: the scanner worker resolves registry credentials from one of two places — the Integrations module (per-tenant registry integrations) or static worker pipeline config (environment variables) — and supports only Anonymous, Basic, and Bearer auth. There is no cloud-IAM/OIDC federation and no operator CLI for registry credentials; see How credential resolution actually works before diagnosing.

Metadata

FieldValue
ComponentScanner (scanner-worker)
SeverityHigh
On-call scopePlatform team, Security team
Last updated2026-05-30
Doctor checkNone dedicated. The Scanner Doctor plugin ships check.scanner.queue, check.scanner.sbom, check.scanner.vuln, check.scanner.reachability, check.scanner.resources, check.scanner.slice.cache, and check.scanner.witness.graph — there is no check.scanner.registry-auth check. (Source: src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Scanner/Checks/.)

Reconciliation note (2026-05-30): This runbook was reconciled against src/Scanner and the CLI. The original draft described a stella registry ... credential-management workflow and cloud-IAM/OIDC federation flows that do not exist in the codebase. The real model is: the scanner-worker resolves registry credentials either from the Integrations module (integrations-web HTTP catalog) or from static worker pipeline config (env vars). Supported auth types are Anonymous, Basic, and Bearer only. Sections that describe unimplemented surfaces are flagged inline.

Symptoms

Flag — alert name unverified. The original draft referenced an alert ScannerRegistryAuthFailed. No such alert rule was found in src/Scanner or the telemetry assets during reconciliation; treat it as unverified rather than authoritative.

Impact

Impact TypeDescription
User-facingCannot scan private images; release pipeline blocked
Data integrityNo data loss; authentication issue only
SLA impactAll scans for affected registry blocked

How credential resolution actually works

Before diagnosing, understand the resolution chain the scanner-worker runs for every distinct registry hostname it encounters during a scan (source: src/Scanner/StellaOps.Scanner.Worker/Processing/IntegrationAwareRegistryCredentialProvider.cs):

  1. If the Integrations path is enabled (Scanner:Worker:Integrations:Enabled=true and a BaseAddress is set), the worker calls integrations-web:
    • GET /api/v1/integrations?type=Registry (paged) to list registry integrations.
    • Matches the requested registry hostname against each integration’s endpoint (case-insensitive, scheme-agnostic, via NormalizeHostname).
    • On a match that is not flagged requiresCredentialsResolution=true, it calls POST /api/v1/integrations/{id}/resolve-credentials to fetch the plaintext secret.
    • The secret is translated to a credential: user:password → Basic; an opaque value → Bearer (TranslateSecret).
  2. If no integration matches the hostname, the worker falls back to PipelineRegistryCredentialProvider, which uses static env-var credentials (Scanner:Worker:Pipeline:RegistryUsername / RegistryPassword) when set, otherwise Anonymous.
  3. The chosen credential is applied by RegistryClient (source: src/Scanner/__Libraries/StellaOps.Scanner.Registry/RegistryClient.cs): Basic/Bearer header on the first request, and if the registry replies 401 with a WWW-Authenticate: Bearer challenge, the client performs the Docker v2 token handshake (fetches a token from the challenge realm, caches it, retries).

Supported auth types are only Anonymous, Basic, and Bearer (RegistryAuthType in IRegistryCredentialProvider.cs). There is no AWS ECR IAM-role, GCP workload-identity/OIDC-federation, or Azure ACR managed-identity flow anywhere in Scanner — see the flagged section under “Root cause fix”.

Fail-closed behaviour (default)

When a registry integration matches the hostname but credential resolution returns nothing (resolve-credentials errored, returned empty, or the integration is flagged requiresCredentialsResolution=true), the worker returns null instead of falling back to anonymous, because the operator intent is that registries must be authenticated. This is controlled by Scanner:Worker:Integrations:FailClosed (default true). The downstream registry client then surfaces this as an auth failure / zero non-empty layers rather than silently scanning nothing. Note: a lookup failure (transport/parse/config error before any match) still falls back to the static provider so a transient integrations-web outage does not take the worker down.

Diagnosis

Quick checks

  1. Run the Scanner Doctor checks (there is no registry-auth-specific check; use the general Scanner queue/vuln/sbom checks plus the worker logs below):

    stella doctor run

    Flag — NOT IMPLEMENTED. The original draft invoked stella doctor --check check.scanner.registry-auth. No such check exists (see Metadata). Use stella doctor without that filter and inspect scanner-worker logs.

  2. Confirm whether the Integrations credential path is enabled. Inspect the scanner-worker config (Scanner:Worker:Integrations): Enabled, BaseAddress, Tenant, FailClosed. If Enabled=false, only the static pipeline env-var credentials are in play (resolution step 2 above).

  3. List configured registry integrations (this is the data the worker matches against). Use the Integrations module surface — the scanner-worker itself calls GET /api/v1/integrations?type=Registry. There is no stella registry list --show-status command.

    Flag — NOT IMPLEMENTED. stella registry list --show-status and stella registry test <url> do not exist as working commands. A stella registry group exists only as dead, un-wired stub code (RegistryCommandGroup.cs — prints simulated output, never wired into the CLI root). The only wired registry CLI surface is stella config registry list / stella config registry configure, and both are mock stubs that print hardcoded output and do not persist or test anything.

Deep diagnosis

  1. Read scanner-worker logs for the resolution-stage warnings. Filter the scanner-worker container/service logs for the literal messages emitted by IntegrationAwareRegistryCredentialProvider:

    • Skipping registry probe for {Registry}: integration {IntegrationId} is flagged requires_credentials_resolution=true → the integration is a legacy/anonymous row that needs credentials added.
    • IntegrationAwareRegistryCredentialProvider: resolve-credentials returned {StatusCode} / ... resolve-credentials failed ... → integrations-web reachable but credential resolution failed.
    • IntegrationAwareRegistryCredentialProvider: list-integrations returned {StatusCode} → the worker could not list integrations (often auth/tenant: the resolver HttpClient must carry a valid Authority client-credentials JWT whose stellaops:tenant claim matches the integration’s tenant).
    • Bearer token request failed for {Registry}: {StatusCode} (from RegistryClient) → the Docker v2 token endpoint rejected the basic credentials during the challenge handshake.

    Flag — NOT IMPLEMENTED. stella scanner logs --filter ... --last 30m does not exist. Read the scanner-worker service/container logs directly (e.g. docker logs).

  2. Verify the resolver’s auth + tenant binding. The HttpClient named scanner-worker-integrations-resolver carries an Authority client-credentials token; integrations-web enforces a claim-only RequireTenant() filter and strips raw X-Tenant-Id-style headers at ingress. A 401/403 on list-integrations or resolve-credentials almost always means the worker’s token tenant claim does not match the integration’s tenant, or the token lacks the configured scope (Scanner:Worker:Integrations:Scope, default integration:read).

  3. Confirm the hostname actually matches an integration. Matching is on the normalized host (lowercased authority, port-aware). A mismatch (e.g. registry.example.test:5000 vs an integration endpoint of https://registry.example.test) causes the worker to fall through to static/anonymous instead of using the integration credential.

    Flag — NOT IMPLEMENTED. stella registry credentials show, stella registry test --verbose, and stella registry iam-status do not exist. Diagnose via integrations-web data + worker logs instead.

Resolution

There is no operator CLI for managing scanner registry credentials. Remediation happens in one of two places: the Integrations module (the per-tenant registry integration + its stored secret) or the scanner-worker configuration (static env vars). Choose the path that matches how the failing hostname is being resolved (see “How credential resolution actually works”).

Immediate mitigation

  1. Integrations path — update the registry integration secret. Update the credential stored on the matching registry integration so the next resolve-credentials call returns valid material. The secret must be in one of the shapes the worker understands: user:password (→ Basic) or an opaque token (→ Bearer). Use the Integrations module’s own management surface to update the secret; the scanner-worker caches the hostname→integration mapping for the lifetime of a worker process, so a worker restart guarantees the new secret is picked up.

  2. Static path — set worker pipeline credentials. If the hostname does not match any integration, set the static env vars on the scanner-worker and restart it:

    SCANNER__SCANNER__WORKER__PIPELINE__REGISTRYUSERNAME=<user>
SCANNER__SCANNER__WORKER__PIPELINE__REGISTRYPASSWORD=<token-or-password>

    These map to ScannerWorkerOptions.ScanPipelineOptions.RegistryUsername / RegistryPassword. When set, the worker sends HTTP Basic; when unset it falls back to Anonymous.

  3. Clear a stuck fail-closed state. If the integration is flagged requiresCredentialsResolution=true (legacy anonymous row), the worker fail-closes and logs Skipping registry probe .... Add real credentials to that integration (clearing the flag), or — only as a temporary, non-production workaround — set Scanner:Worker:Integrations:FailClosed=false so a missing credential falls back to anonymous/static. Document any such change.

    Flag — NOT IMPLEMENTED. stella registry refresh-credentials, stella registry update-credentials, and stella registry configure docker-hub --access-token do not exist. Docker Hub rate-limiting is not specially handled by Scanner; provide an authenticated Docker Hub credential via the integration or the static env vars above to raise the rate limit.

Root cause fix

If credentials expired: Rotate the secret at the source registry, then update it on the matching registry integration (Integrations path) or in the scanner-worker env vars (static path) and restart the worker. There is no auto-refresh/refresh_interval setting for scanner registry credentials; the only token refresh that happens automatically is the in-memory Docker v2 bearer-token cache inside RegistryClient (refreshed expires_in - 30s before expiry, default 5 min when the token endpoint omits expires_in). That cache is per worker process and is not operator-configurable beyond RegistryClientOptions.TokenExpirationBufferSeconds.

Flag — NOT IMPLEMENTED (cloud IAM/OIDC). The original draft documented AWS ECR IAM-role assumption, GCP Artifact Registry workload-identity/OIDC federation, and Azure ACR flows (stella registry iam verify, stella registry oidc verify, stella registry configure ecr --role-arn, ... gcr --workload-identity-provider). None of these exist. Scanner authenticates to registries only with static Basic/Bearer credentials (resolved from an integration or env vars) plus the standard Docker v2 bearer-token challenge handshake. The --type ecr|gcr|acr option on stella config registry configure is a label on a mock stub that persists nothing and triggers no cloud-IAM behaviour. For cloud registries, supply a username/password or token credential (e.g. an ECR docker login password, a GCR JSON-key or access token) as a normal Basic/Bearer secret. This also aligns with the platform’s on-prem-first / no-cloud-managed-defaults posture.

If a self-hosted registry uses a private CA or non-TLS: TLS handling is configuration on the registry client, not a per-registry CLI flag:

Verification

There is no stella registry test / stella scan image --dry-run verification command for this flow. Verify by re-running a real scan of a private image and confirming the worker logs no longer emit the resolution-stage warnings listed under “Deep diagnosis”, and that the scan produces non-empty layers / components.

  1. Restart the scanner-worker (clears the per-process hostname→integration cache and the bearer-token cache).
  2. Trigger a scan of a private image on the affected registry.
  3. Tail the scanner-worker logs and confirm: no Skipping registry probe, no resolve-credentials failed, no Bearer token request failed, and the scan completes with layers/components emitted.

Flag — NOT IMPLEMENTED. The original verification block (stella registry test, stella scan image --dry-run, stella scanner logs) referenced commands that do not exist; replaced with the log-based real-scan verification above.

Prevention