Artifact Store Migration Runbook

For platform operators migrating existing evidence — SBOMs, DSSE envelopes, VEX documents — out of legacy per-module stores into the unified Stella Ops ArtifactStore. Read the status banner first: most of the operator surface below is intended-but-not-yet-wired, and is annotated inline.

Sprint: SPRINT_20260118_017_Evidence_artifact_store_unification (AS-006)

Status: DRAFT / partially implemented. The unified ArtifactStore library (StellaOps.Artifact.Core / StellaOps.Artifact.Infrastructure) and the ArtifactMigrationService are present in source, but the end-to-end migration workflow is not yet wired:
The stella artifacts migrate CLI command (MigrateArtifactsCommand) is defined but is not registered in the CLI command tree (CommandFactory), so it cannot currently be invoked.
The CLI resolves IArtifactMigrationService from DI, but no implementation of that interface is registered — the registered service is the concrete ArtifactMigrationService (via ServiceCollectionExtensions.AddArtifactMigration), which has a different shape. Running the command as written would fail at DI resolution.
No concrete ILegacyArtifactSource is implemented in src/, so the migration has no real legacy store to enumerate yet.
The unified store is not hosted by any running service. AddUnifiedArtifactStore (ServiceCollectionExtensions) has no caller anywhere in src/, and the retrieval API (ArtifactController, api/v1/artifacts) is never added as an MVC application part by any WebService. Only the PostgreSQL schema migration is wired: the Platform evidence migration module (MigrationModulePlugins.EvidenceMigrationModulePlugin) includes ArtifactDataSource’s assembly, so evidence.artifact_index is created on Platform startup — but the S3 store, the index repository, and the query endpoint are not registered or served. The retrieval curl below therefore has no live endpoint to hit today.
Treat the CLI invocations below as the intended operator surface. Note also that all invocations use stella artifacts (plural); the only artifact-related command actually registered is stella artifact (singular) — an unrelated artifact-link association group (ArtifactCommandGroup, wired at CommandFactory line 130). There is no artifacts command group at all. Verify against src/ before relying on any of this in production. Items that do not yet exist in code are flagged inline.

Overview

This runbook covers the migration of existing evidence from legacy artifact stores to the unified ArtifactStore.

Migration Sources

The migration enumerates legacy artifacts through an ILegacyArtifactSource abstraction and classifies each artifact’s type from its content type and path (ArtifactMigrationService.InferArtifactType). The CLI --source flag maps to the MigrationSource enum: evidence → EvidenceLocker, attestor → Attestor, vex → Vex, all → All.

NOT IMPLEMENTED (legacy sources): no concrete ILegacyArtifactSource exists in src/ for any of these sources. The legacy path shapes below are illustrative. The only shape grounded in code is the EvidenceLocker tenants/.../bundles/... layout, which the fallback bom-ref generator strips (ArtifactMigrationService.GenerateFallbackBomRef removes the tenants/ and bundles/ segments). The Attestor / VEX path shapes are not yet realised by any source implementation.

Source (`MigrationSource`)	Illustrative legacy path	Description
`EvidenceLocker`	`tenants/{tenantId}/bundles/{bundleId}/{sha256}-{name}`	Evidence bundles
`Attestor`	`attest/dsse/{bundleSha256}.json`	DSSE envelopes
`Vex`	`{prefix}/{format}/{digest}.{ext}`	VEX documents

Target Path Convention

Artifacts are written to S3 using the layout produced by BomRefEncoder.BuildPath (StellaOps.Artifact.Core/BomRefEncoder.cs):

artifacts/{encoded-bom-ref}/{encoded-serial-number}/{artifactId}.json

Notes (verified against BomRefEncoder and S3UnifiedArtifactStore.BuildFullKey):

There is no leading slash; the key is relative to the configured store prefix.
Both the bom-ref and the serial number are encoded — /, :, @, ?, #, and % are replaced (e.g. @ → _at_). / and : both collapse to _ and are not reversible (BomRefEncoder.Decode only restores @/?/#/%).
A configurable store Prefix (S3UnifiedArtifactStoreOptions.Prefix) is prepended to the key by S3UnifiedArtifactStore.BuildFullKey. The default Prefix is "artifacts", not empty. Because BomRefEncoder.BuildPath already emits a leading artifacts/ segment, the default full S3 key is doubled — artifacts/artifacts/{encoded-bom-ref}/{encoded-serial}/{artifactId}.json. Set ArtifactStore:S3:Prefix to an empty string (or a different bucket prefix) to avoid the duplicated segment.

Pre-Migration Checklist

[ ] Backup existing S3 buckets
[ ] Verify PostgreSQL backup is current
[ ] Ensure sufficient storage for duplicated data (migration defaults to copy mode, ArtifactMigrationOptions.CopyMode = true, preserving originals)
[ ] Review migration in dry-run mode first
[ ] Notify stakeholders of potential service impact

Running the Migration

NOT WIRED: the artifacts migrate command is implemented in MigrateArtifactsCommand.BuildCommand but is not attached to the CLI command tree (CommandFactory never calls it — MigrateArtifactsCommand is referenced nowhere outside its own file), and IArtifactMigrationService has no registered implementation (AddArtifactMigration registers the concrete ArtifactMigrationService, whose MigrateAsync shape does not match the interface). The command word below is also artifacts (plural), but the only registered top-level group is artifact (singular), an unrelated artifact-link group. The invocations below describe the intended interface (flags --source, --dry-run, --parallelism/-p, --batch-size/-b, --resume-from, --tenant, --output/-o). Confirm wiring in src/Cli before use.

Dry Run (Recommended First Step)

stella artifacts migrate --source all --dry-run --output migration-preview.json

Full Migration

# Migrate all sources with default settings (parallelism 4, batch size 100)
stella artifacts migrate --source all

# Migrate with increased parallelism
stella artifacts migrate --source all --parallelism 8 --batch-size 200

# Migrate specific source
stella artifacts migrate --source evidence --output migration-report.json

# Migrate specific tenant (value must be a tenant GUID; parsed via Guid.Parse)
stella artifacts migrate --source all --tenant <tenant-uuid>

Resuming Failed Migration

# Use checkpoint ID from previous run
stella artifacts migrate --source all --resume-from <checkpoint-id>

PARTIAL (checkpoint/resume): the --resume-from flag and the CheckpointId reported on completion exist only on the CLI-local MigrationOptions / MigrationResult types (MigrateArtifactsCommand.cs). The implemented ArtifactMigrationService (StellaOps.Artifact.Infrastructure) does not persist or honour checkpoints — there is no checkpoint store, and ArtifactMigrationOptions exposes no resume option. Resume is not yet functional.

Progress Monitoring

The CLI displays real-time progress. The format below is emitted by MigrateArtifactsCommand from the CLI-local MigrationProgress (Processed/Total/PercentComplete/Succeeded/Failed/Skipped):

  Progress: 1500/10000 (15.0%) - Success: 1495, Failed: 3, Skipped: 2

DIVERGENT SHAPE: the implemented service reports a different MigrationProgress type. The infrastructure StellaOps.Artifact.Infrastructure.MigrationProgress exposes ProcessedItems/TotalItems/SuccessCount/FailureCount/SkippedCount (plus StartedAt/LastUpdateAt/CurrentItem/EstimatedRemaining) and has no PercentComplete. Its MigrateAsync also has a different signature — IAsyncEnumerable<ArtifactMigrationResult> MigrateAsync(IProgress<MigrationProgress>?, CancellationToken), taking no options argument — whereas the CLI calls Task<MigrationResult> MigrateAsync(MigrationOptions, IProgress<MigrationProgress>?, CancellationToken). The two are incompatible, which is the concrete reason the command cannot bind to the registered service. The format above reflects the CLI’s intent only.

Rollback Procedure

When to Rollback

Migration corrupted data
Performance degradation after migration
Business-critical bug discovered

Rollback Steps

1. Stop New Writes to Unified Store

NOT IMPLEMENTED (config keys): the toggles below are placeholders. No ARTIFACT_STORE_UNIFIED_ENABLED environment variable and no artifactStore.useUnifiedStore / legacyMode configuration keys exist in src/. The actual options are registered under the ArtifactStore configuration section (ServiceCollectionExtensions.AddUnifiedArtifactStore, binding ArtifactStore:S3 and ArtifactStore:Postgres), but there is no built-in disable/legacy-fallback switch. Stopping writes today means stopping the writing service.

# Placeholder — not honoured by current code.
ARTIFACT_STORE_UNIFIED_ENABLED=false

# Placeholder — not honoured by current code.
artifactStore:
  useUnifiedStore: false
  legacyMode: true

2. Clear Unified Store Index

-- Clear the PostgreSQL artifact index (preserves S3 data).
-- Schema/table: evidence.artifact_index (NOT artifact_store.artifacts).
TRUNCATE TABLE evidence.artifact_index;

3. (Optional) Remove Migrated S3 Objects

# Only if disk space is critical and you're certain about rollback.
# WARNING: This is destructive! With the default store Prefix ("artifacts"),
# keys live under the doubled prefix "artifacts/artifacts/"; with an empty
# Prefix they live under "artifacts/". Adjust the path for your configured
# ArtifactStore:S3:Prefix before running.
aws s3 rm s3://<artifacts-bucket>/artifacts/artifacts/ --recursive

4. Restart Services

# Restart through Docker Compose or the approved host service manager.
docker compose restart evidence-locker attestor

Post-Migration Validation

Verify Artifact Counts

Classify migrated artifacts by their artifact_type column. The schema comment documents the type values as Sbom, Vex, DsseEnvelope, RekorProof, Verdict, etc. (ArtifactType enum). Do not classify by storage_key: that column holds the full S3 key (artifacts/{encoded-bom-ref}/...), which does not contain the substrings evidence/dsse/vex.

-- Count migrated artifacts by type.
SELECT artifact_type, COUNT(*) AS count
FROM evidence.artifact_index
WHERE NOT is_deleted
GROUP BY artifact_type
ORDER BY count DESC;

Verify bom-ref Extraction

When CycloneDX bom-ref extraction fails, the migration assigns a synthetic fallback bom-ref of the form pkg:stella/legacy/<escaped-path> (ArtifactMigrationService.GenerateFallbackBomRef) and a synthetic serial of the form urn:uuid:<sha256-derived-guid>. Surface fallback rows with:

-- Count artifacts that fell back to a synthetic (path-derived) bom-ref.
SELECT COUNT(*) AS synthetic_count
FROM evidence.artifact_index
WHERE bom_ref LIKE 'pkg:stella/legacy/%';

Test Retrieval

NOT HOSTED: ArtifactController ([Route("api/v1/artifacts")], class-level [Authorize]) is defined in StellaOps.Artifact.Core but no WebService adds it as an MVC application part, and AddUnifiedArtifactStore has no caller. The endpoint below is therefore unreachable in any current deployment; query the evidence.artifact_index table directly (see the SQL above) to validate retrieval until the API is wired into a host. The list endpoint requires the bom_ref query parameter (it returns 400 without it) and, once hosted, a valid bearer token (no specific scope policy beyond authentication).

# Intended once the controller is hosted — currently not mounted.
# Required query parameter: bom_ref. Optional: serial_number, from, to, limit,
# continuation_token (see ArtifactController.ListArtifactsAsync).
curl -H "Authorization: Bearer <token>" \
  "https://api.example.com/api/v1/artifacts?bom_ref=pkg:docker/acme/api@sha256:abc123"

NOT IMPLEMENTED (artifacts compare): there is no stella artifacts compare CLI command. The CLI has compare subcommands for VEX, SBOM, scores, and ground-truth runs, but none under artifacts. Compare original vs. migrated content out-of-band (e.g. fetch both objects from S3 and diff their SHA-256). The index stores sha256 per row for this purpose.

Troubleshooting

Migration Stuck

# Check for stuck workers
ps aux | grep migrate

NOTE: there is no on-disk checkpoint file. The /var/lib/stella/migration-checkpoint.json path referenced in earlier drafts does not exist — the implemented service holds progress in memory (ArtifactMigrationState) only.

High Failure Rate

Check migration report (--output) for common errors; failed items are listed in FailedItems with SourceKey and Error.
Verify source store connectivity
Check for corrupted source artifacts
Reduce batch size if memory pressure is the cause

Slow Migration

Increase parallelism (--parallelism, up to CPU count; default 4)
Run during off-peak hours
Consider migrating by tenant in parallel (--tenant)
Verify network bandwidth to S3

Representative Dataset Testing

Before production migration, test with a representative dataset on a non-prod tenant.

NOTE: the stella evidence list / stella attestor list and stella evidence get / stella attestor get commands referenced in earlier drafts are not present in the CLI. The evidence command group exposes holds list and audit list (and holds create/release, card export/verify, export/verify); evidence holds are gated by the evidence:hold / evidence:read scopes (StellaOpsScopes). Bundle export/retrieval goes through the EvidenceLocker REST API as an async job, not a single GET (ExportEndpoints):
POST /api/v1/bundles/{bundleId}/export — enqueues the job, returns 202 Accepted with an export job ID and a status URL. Requires the export.operator scope (StellaOpsResourceServerPolicies.ExportOperator).
GET /api/v1/bundles/{bundleId}/export/{exportId} — returns 202 while in progress or 200 with the manifest when complete. Requires export.viewer.
GET /api/v1/bundles/{bundleId}/export/{exportId}/download — streams the completed bundle as a gzip archive (409 while still running). Requires export.viewer.
Build a sample dataset using the available evidence/export surface, then run the migration against a test tenant once the CLI is wired:

# Run migration against a test tenant and capture the report.
stella artifacts migrate --source all --tenant <test-tenant-uuid> --output test-report.json

Artifact Store API
IArtifactStore Interface
PostgreSQL Index Schema
Migration Service
CLI Command (not yet wired)