checkId: check.observability.logs.shipping plugin: stellaops.doctor.observability severity: warn tags: [observability, logging, log-shipping, forwarder]

Log Shipping Health

What It Checks

Reads live forwarder drain state through the read-only ILogShippingStateProvider seam. Evidence includes forwarder name, configured state, drain status, backlog count, and oldest backlog age.

ConditionResult
Log shipping disabled, unconfigured, or no state provider registeredskip
Forwarder is configured but not drainingfail
Forwarder is draining with backlogwarn
Forwarder is configured, alive, draining, and backlog is zeropass

Why It Matters

Local logs are only useful during incidents if they reach the configured backend. Directory writability and JSON formatting do not prove the forwarder is alive or that buffers are draining.

Common Causes

How to Fix

Docker Compose

docker compose -f devops/compose/docker-compose.stella-ops.yml ps vector fluent-bit
docker compose -f devops/compose/docker-compose.stella-ops.yml logs --tail 100 vector
docker compose -f devops/compose/docker-compose.stella-ops.yml logs --tail 100 fluent-bit

Restart only the configured local forwarder after confirming the backend is reachable. Do not delete buffer directories unless an operator has explicitly accepted log loss.

Bare Metal / systemd

systemctl status vector fluent-bit stellaops-journald-forwarder
journalctl -u vector -n 100 --no-pager
journalctl -u fluent-bit -n 100 --no-pager

Check output errors, retry counters, and buffer usage before restarting the forwarder.

Kubernetes / Helm

kubectl -n stellaops get pods -l app.kubernetes.io/component=log-forwarder
kubectl -n stellaops logs -l app.kubernetes.io/component=log-forwarder --tail=100

Verify the DaemonSet or sidecar has permission to read application logs and reach the local observability backend.

Verification

stella doctor run --check check.observability.logs.shipping