Deployment-observation agent authentication

The deployment-observation agent calls Signals directly. Signals requires an Authority token with signals:write and derives the observation’s tenant from that token. A body tenant that differs is refused before storage. The gateway does not publish this internal operation.

Source contract verified against fe48a471274a4ed6474ee452c940818e1160ff81 (PCD-865). The owning implementations are RuntimeEvidencePublisherRegistration in Agent.Core, ResolveObservationTenantAsync in the Signals host, and SignalsTenantsReplica. This source verification does not substitute for the live acceptance steps below.

Configure the identities

1. Register a confidential Authority client for the agent’s actual tenant through POST /console/admin/clients. Grant client_credentials, signals:write and aoc:verify, with audience stellaops. Send the selected tenant in the canonical tenant header. Use the operator’s current approved authentication; do not put credentials in a committed request.

2. Store the generated secret in the operator’s secret store or a private mounted file. The client used by the local profile below reads .local-secrets/agent-core-runtime-default.secret.

3. Set these entries in the enrolled agent’s private devops/compose/env/agent-core.local.env:

   Agent__RuntimeEvidence__Authority__ClientId=<registered-client-id>
   Agent__RuntimeEvidence__Authority__Tenant=<registered-tenant-slug>
   Agent__RuntimeEvidence__Authority__ClientSecretFile=/run/secrets/agent-runtime-authority
   

   ClientSecretFile wins over an environment secret. The agent refuses to start when its Signals address is configured without a complete client. Keep the enrollment ID and mTLS material.

4. Enable Signals’ tenant replica with Catalog:Replication:Tenants:Enabled=true, an Authority tenants-feed base address, AuthMode=client-credentials, and an explicit AuthTenant. Reuse Signals’ existing Doctor service identity with catalog:replicate. Do not read Authority’s database from Signals or seed a replica by hand.

Apply the local profile

The committed devops/compose/docker-compose.runtime-observation-auth.override.yml owns the local image pins, the private secret bind, and the tenant-replica settings. It is applied after ok5/docker-compose.signals.override.yml, which retains the existing S3, event, Doctor and environment-state settings. The runtime-auth profile preserves Signals’ Hybrid bridge setting when adopting the canonical chain. Check the image source and retained rollback images before using the pins on a different checkout or installation.

Set AGENT_CORE_RUNTIME_STATE_VOLUME in the private Compose .env to the physical volume name currently mounted at /var/lib/stella-agent in the enrolled agent. The profile declares that volume external and reuses it. It must not create an empty replacement for existing agent state.

From the primary checkout, set the profiles for the canonical Compose helper:

$env:COMPOSE_EXTRA_FILES = 'ok5/docker-compose.signals.override.yml;docker-compose.runtime-observation-auth.override.yml'

Before service recreation, render each selected service and compare its environment, mounts, network and physical volume names with the running container. Preserve unrelated settings. Acquire the shared deployment window. Apply agent-core first; the preceding Signals image ignores the additional bearer token. Then apply Signals. Use a service name and --no-deps for each force-recreate; never recreate the entire rendered stack.

The tenant replica drains Authority’s authenticated feed into Signals’ own database. Wait for its checkpoint before the ingest positive control. A disabled or unavailable replica returns retryable 503; an unknown or inactive tenant returns 403. A healthy process does not prove a converged feed.

Verify the boundary

• The agent’s token request succeeds with both scopes. signals:write alone is refused.
• A real runtime.observe.container task finishes with Signals status 202 and a row for the enrolled tenant in Signals’ database.
• A direct anonymous observation returns 401. A tenant-A token naming tenant B returns 403. Neither request creates an observation or agent row.
• A body without a tenant is accepted for the token’s tenant. Retrying its event ID does not increment the observation twice.
• An observation naming an agent ID owned by another tenant returns 403 with error: agent_not_authorized. Check that the complete agent row and its existing observations remain unchanged. Repeat with and without the optional body tenant. An own-tenant agent ID must still accept new observations and retries.
• A runtime-facts request for the caller’s tenant emits runtime.updated for that tenant. A differing metadata tenant is refused without storing a fact or emitting that event.
• The gateway returns 404 for the internal observation operation. Its aggregated OpenAPI can still list the service’s internal path; that listing is an independently tracked limitation.

For an agent-authentication rollout, run the native PostgreSQL tenant controls and agent regression project before closing the task:

pwsh ./tools/scripts/test-targeted-xunit.ps1 -Project src/Signals/__Tests/StellaOps.Signals.Tests/StellaOps.Signals.Tests.csproj -Class '*SignalsRuntime*' -BuildProjectReferences
pwsh ./tools/scripts/test-targeted-xunit.ps1 -Project src/ReleaseOrchestrator/__Tests/StellaOps.Agent.Core.Tests/StellaOps.Agent.Core.Tests.csproj -All -BuildProjectReferences

The native controls also cover a tenantless runtime-facts request emitting no runtime.updated event. Record each executed test count and distinguish native-host proof from deployed proof. For a Signals-only ownership fix, run the observation, gateway-publication and runtime-facts tenant classes; retain the accepted agent image and verify a real enrolled-agent task against the new Signals image. Re-run the agent project when its publisher or authentication code changes. Remove only uniquely marked probe rows after recording their identities and results.

Roll back Signals first, then agent-core, using the retained image IDs and the same reviewed configuration. Keep forward-only schema changes. Never remove or reset unrelated data or volumes.

Reconcile a historical agent ownership mismatch

Observation ingestion never changes the owner of an existing agent ID. A historical mismatch therefore needs a separately authorized operator repair before that agent can ingest again. Use the tenant-scoped ReleaseOrchestrator agent read as the ownership authority. Match the enrolled certificate’s tenant UUID and thumbprint to that registration and the tenant catalog. Do not treat a submitted observation body as ownership evidence.

Capture the complete Signals agent row and all observations naming that agent before repair. Under the shared deployment window, lock that one agent row and refuse the repair if its complete snapshot changed. Update only tenant_id to the verified owner UUID and require exactly one affected row. Preserve existing observations, registration time, metadata and volume state. Record the authorization, canonical identity checks, guarded statement and before/after snapshots in the task’s evidence. Then run a real enrolled-agent observation and verify its accepted fact.