checkId: check.db.migrations.failed plugin: stellaops.doctor.database severity: fail tags: [database, migrations, postgres, schema]

Failed Migrations

This Doctor check surfaces platform migrations that cannot be integrity-verified. A migration recorded without a verifiable checksum leaves the schema in an untrustworthy state and is a leading cause of subtle drift and runtime 500 errors after a Stella Ops upgrade.

What It Checks

Reads the real per-schema <schema>.schema_migrations ledger that the platform migration runner writes — NOT the legacy stella_migration_history table (which the runner never creates) and NOT EF Core’s __EFMigrationsHistory. The platform runner fails closed at startup on a genuine checksum mismatch, so the observable “failed/incomplete” condition Doctor can verify against the live database is an applied row whose checksum is blank (cannot be integrity-verified). Those rows are reported as a failure.

If no schema has a schema_migrations ledger, the check skips honestly (there is nothing to verify) rather than reporting a green “no failed migrations”.

Why It Matters

A blank-checksum row means a migration was recorded out-of-band, or by a pre-checksum runner build, and was never reconciled. The schema’s integrity can no longer be confirmed from the ledger, which is a common precursor to startup failures and runtime 500 errors after upgrades.

Common Causes

How to Fix

Docker Compose

docker compose -f devops/compose/docker-compose.stella-ops.yml logs --tail 200 doctor-web
docker compose -f devops/compose/docker-compose.stella-ops.yml exec postgres psql -U stellaops -d stellaops -c "SELECT migration_name, category, checksum, applied_at FROM <schema>.schema_migrations WHERE checksum IS NULL OR checksum = '' ORDER BY applied_at DESC;"

Review the owning service logs for Checksum mismatch / migration errors emitted by StartupMigrationHost, then re-run the owning service so it re-records normalized checksums, or restore from a known-good PostgreSQL snapshot.

Bare Metal / systemd

journalctl -u <service-name> -n 200
psql -h <db-host> -U <db-user> -d <db-name> -c "SELECT migration_name FROM <schema>.schema_migrations WHERE checksum IS NULL OR checksum = '';"

Kubernetes / Helm

kubectl logs deploy/<service-name> -n <namespace> --tail=200
kubectl exec -n <namespace> <postgres-pod> -- psql -U <db-user> -d <db-name> -c "SELECT migration_name, checksum FROM <schema>.schema_migrations;"

Verification

stella doctor --check check.db.migrations.failed