Operator-managed trust anchors

Sprint 20260505_042 — generalises Sprint 033’s per-connector embedded Apple Root CA into a first-class operator-managed asset.

Audience: operators running Stella Ops connectors against upstreams whose TLS chains are not rooted in the standard system CA bundle — internal mirrors, government CERT portals, or anything reached through a TLS-inspecting proxy.

What this is

Stella Ops connectors fetch advisory data from a wide spectrum of upstreams. Some of those upstreams (Apple’s gdmf.apple.com / support.apple.com, regional/government CERT portals, internal stella-mirror replicas) present TLS chains that are NOT rooted in the standard ca-certificates bundle that ships in our slim Ubuntu runtime image. Operators in offline / air-gap deployments cannot fix this by apt-get update-ing the system trust store, and operators behind a TLS-inspecting corporate proxy must trust the proxy’s interception CA explicitly.

OperatorTrustAnchorStore (in StellaOps.Concelier.Connector.Common.Http) lets operators drop PEM-encoded certificates into a known directory. Every named SourceHttpClient registered through AddSourceHttpClient automatically receives the loaded roots through the existing custom-trust chain-build callback in Connector.Common.Http.ServiceCollectionExtensions — no per-connector code change is required.

Where to put PEMs

The compose stack (devops/compose/docker-compose.stella-services.yml) mounts the host directory read-only into:

• concelier
• excititor-web
• excititor-worker
• advisory-ai-web

Add the same mount + STELLAOPS_TRUST_ANCHORS_PATH env var to any other service that goes through Concelier’s AddSourceHttpClient and is downstream of AddSourceCommon.

File format

• *.pem glob, top-level only (no recursion).
• Each PEM file may contain one or more -----BEGIN CERTIFICATE----- blocks.
• Duplicate certificates are deduplicated across files by certificate thumbprint (the SHA-1 cert hash; note the startup log prints the SHA-256 fingerprint for identification).
• Non-PEM files in the directory are ignored (for example, README.md).

Startup logging

At first AddSourceHttpClient resolution the loader emits one log line per discovered certificate plus a summary:

Loaded operator trust anchor 'CN=Apple Root CA, OU=Apple Certification Authority, O=Apple Inc., C=US' (issuer 'CN=Apple Root CA, OU=Apple Certification Authority, O=Apple Inc., C=US', SHA-256 B0B1730ECBC7FF4505142C49F1295E6EDA6BCAED7E2C68C5BE91B5A11001F024, valid until 2035-02-09T21:40:36Z) from '/etc/stellaops/trust-anchors/apple-root.pem'.
Operator trust-anchor scan complete: 1 unique certificate(s) loaded from '/etc/stellaops/trust-anchors'.

Logs are at Information level under the StellaOps.Concelier.Connector.Common.Http.OperatorTrustAnchorStore category. When the directory is missing or empty the loader logs a single Information line stating that connectors will rely on the system trust store + connector-bundled fallbacks.

Relationship to embedded fallbacks (Sprint 033)

The Apple connector still ships Internal/AppleRootCertificates.pem as an embedded resource fallback so dev-mode setups (where the operator has not mounted etc/trust-anchors) keep working out of the box. In production deployments operators drop the same PEM into /etc/stellaops/trust-anchors/apple-root.pem — both copies trust the same root, so the deduplication logic in AddSourceHttpClient keeps the collection clean.

When converting a future vendor connector that previously needed an embedded PEM, prefer to ship the operator-managed path documentation instead of embedding. Embedding remains acceptable only for vendors whose public root ships under explicit license that allows redistribution AND where the “works without operator action” property is part of the connector’s acceptance criteria (Apple is the only such case today).

Relationship to the legacy etc/certificates/trust-roots/ mount

There is a separate, older mount that maps etc/certificates/trust-roots/ to /etc/ssl/certs/stellaops/. That target directory is consumed by update-ca-certificates paths in the compliance overlay images and the scanner offline kit. It is orthogonal to this loader:

Use both if you want a CA trusted both at the .NET HttpClient layer (this one) and at the system OpenSSL layer (the legacy one).

Setup-wizard hook

The platform setup wizard exposes a “Cryptographic Provider” step (SetupStepId crypto, see docs/INSTALL_GUIDE.md lines 226-229 + src/Web/StellaOps.Web setup-wizard models). A future sprint may add a “Trust anchors” sub-step that lists the loaded CAs read-only (operator drops files, wizard reports). The hook lives in OperatorTrustAnchorStore.Records (already public + immutable per process). The platform service can expose this through a setup-wizard probe endpoint without re-reading the directory.

For now operators verify the loaded set via the startup log lines above, or by running:

docker compose -f devops/compose/docker-compose.stella-ops.yml \
  logs concelier 2>&1 | grep "operator trust anchor"

Security posture

• Mounts are read-only (:ro). Operators control what enters; services cannot mutate the trust store at runtime.
• Adding a CA here does NOT bypass the per-client AllowedHosts allowlist.
• Certificates are loaded once per process at first AddSourceHttpClient resolution (lazy). Rotate by replacing files and recreating the affected services.
• RemoteCertificateValidationCallback uses X509ChainTrustMode.CustomRootTrust so the loaded anchors REPLACE (not augment) the default OS root set for the named client — adding a CA here scopes its trust to that one client, not to the rest of the process.

Related

• Advisory source credential entry — configuring the connectors that consume these trust anchors (including internal-mirror URIs).