E2E Analyzer-Coverage — local Gitea harness runbook

This runbook covers the local Gitea harness for the analyzer-coverage end-to-end (e2e) flow, introduced in Sprint 20260514_003 (now archived as completed). It is Sprint S3 of the SCM-integration + analyzer-coverage e2e effort:

• Sprint S1 shipped the bulk SCM import primitive (POST /api/v1/scm/{integrationId}/bulk-import, stella scm import).
• Sprint S2 shipped the deterministic synthetic-repo corpus factory (StellaOps.E2E.SyntheticRepos).
• Sprint S3 (this doc) stands up a standalone, reproducible local SCM environment: Gitea pre-seeded with the S2 corpus, a registered SCM integration row pointing at it, and idempotent provisioning scripts.
• Sprint S4 adds the StellaOps.E2E.AnalyzerCoverage harness that drives this stack.
• Sprint S5 adds the GitHub/GitLab live targets + the nightly CI workflow — see §9.

Everything here is offline-first and BUSL-clean: no real third-party repositories are cloned. The corpus is generated by the S2 factory from in-repo fixtures and pushed into a local Gitea. No new third-party container images — Gitea and Vault are the stock images already referenced by devops/compose/docker-compose.integrations.yml.

The committed SCM corpus seed is kept under src/__Tests/e2e/scanner-analyzer-corpus/: distribution-matrix.json defines the scanner mix and fixtures/<scanner>/<scenario>/ contains the source repo outlines. Generated git repositories and provisioner outputs are written beside that seed under the same E2E root and are ignored by git.

1. What the stack contains

devops/compose/docker-compose.e2e-analyzer-coverage.yml include:s two existing compose stacks unchanged and adds two one-shot provisioning services:

Why integrations.yml, not dev.yml? The Sprint S3 plan text said “extends docker-compose.dev.yml”, but the dev compose has no Gitea service — it is the minimal Postgres/Valkey/S3/Rekor/Zot/Mailpit lane. Gitea (and the Vault used for authref://vault/dev/scm/gitea) live in docker-compose.integrations.yml. The e2e compose therefore includes the stella-ops stack + the integrations stack. No Gitea service is duplicated.

Shared named volume stellaops-e2e-corpus-data carries the generated artefacts between the provisioner and (Sprint S4) the harness container:

<volume>/corpus/                  the materialised synthetic repos
<volume>/corpus/corpus-manifest.json   the S4 e2e contract (from the S2 factory)
<volume>/corpus/repos.txt          TSV (name<TAB>cloneUrl) for `stella scm import`
<volume>/integration-id.txt        the registered SCM integration GUID

2. Prerequisites

Hosts file entries

Add to /etc/hosts (Linux/macOS) or C:\Windows\System32\drivers\etc\hosts (Windows):

127.1.2.1  gitea.stella-ops.local
127.1.2.4  vault.stella-ops.local
127.1.0.8  scanner.stella-ops.local

No new third-party dependencies; no external network access is required at runtime (the corpus is synthesised locally).

3. Bring the stack up

From devops/compose/:

# Parse-check first (always cheap, no containers started):
docker compose -f docker-compose.e2e-analyzer-coverage.yml config

# Full stack — infra + Scanner + Gitea + Vault + both provisioning steps:
docker compose -f docker-compose.e2e-analyzer-coverage.yml up -d

The two one-shot services run in dependency order:

1. gitea-e2e-bootstrap waits for Gitea to be healthy, then creates the e2e-bot user, rotates its PAT, and seeds Vault.
2. corpus-provisioner waits for gitea-e2e-bootstrap to complete and scanner-web to be healthy, then runs provision-corpus.sh (generate → push → emit repos.txt → register the SCM integration row).

Watch the provisioning progress:

docker compose -f docker-compose.e2e-analyzer-coverage.yml logs -f \
    gitea-e2e-bootstrap corpus-provisioner

Both exit 0 on success (restart: "no" — they are not long-running).

Provision-only (skip the full platform)

If you only need Gitea + the corpus (e.g. iterating on the corpus):

docker compose -f docker-compose.e2e-analyzer-coverage.yml up \
    gitea gitea-e2e-bootstrap corpus-provisioner

(Drop --register-integration by setting E2E_REGISTER_INTEGRATION=0 if the Scanner stack is not up.)

4. Provision the corpus manually (host-run)

The provisioning scripts are runnable directly on a host — useful for debugging or for re-pushing after a corpus tweak. They are idempotent.

POSIX / macOS / Git-Bash:

devops/scripts/e2e/provision-corpus.sh \
    --gitea-url http://gitea.stella-ops.local:3000 \
    --gitea-user e2e-bot \
    --gitea-token <pat> \
    --seed 42 --count 2400 \
    --profile comprehensive \
    --min-per-scanner 80 \
    --min-cases-per-scanner 6 \
    --corpus-dir src/__Tests/e2e/scanner-analyzer-corpus \
    --register-integration \
    --integrations-api http://scanner.stella-ops.local:80

Windows PowerShell:

powershell -File devops/scripts/e2e/provision-corpus.ps1 `
    -GiteaUrl http://gitea.stella-ops.local:3000 `
    -GiteaUser e2e-bot `
    -GiteaToken <pat> `
    -Seed 42 -Count 2400 `
    -Profile comprehensive `
    -MinPerScanner 80 `
    -MinCasesPerScanner 6 `
    -CorpusDir src\__Tests\e2e\scanner-analyzer-corpus `
    -RegisterIntegration `
    -IntegrationsApi http://scanner.stella-ops.local:80

The PAT is resolved in this precedence order:

1. --gitea-token / -GiteaToken
2. $E2E_GITEA_BOT_TOKEN environment variable
3. Vault secret/dev/scm/gitea (when E2E_VAULT_SEED=1, the default)

--profile comprehensive --min-per-scanner 80 keeps the weighted distribution but prevents low-weight scanners from collapsing to token coverage. The current comprehensive local corpus path is src/__Tests/e2e/scanner-analyzer-corpus/corpus/; it contains actual git repositories and is suitable for dedicated SCM ingestion.

Use --min-cases-per-scanner / -MinCasesPerScanner when the run must enforce distinct scanner input breadth, not just repo count. The floor is evaluated against coverageCaseId values, so language cases include dependency depth, pinning shape, mixed-stack, and synthetic-secret variations. This can also make comprehensive runs exceed --count. OS/native families are bounded by their fixture scenario count; an impossible floor fails generation explicitly.

The current seed-42 audit corpus was generated with count 2400, profile comprehensive, --min-per-scanner 80, and --min-cases-per-scanner 6. It materializes 2,585 repos, 1,821 distinct coverage cases, 392 source scenarios, 870 call-path reachability repos, and 319 secret-tagged repos under src/__Tests/e2e/scanner-analyzer-corpus/corpus/.

--skip-generate reuses an existing corpus directory and only re-pushes + re-emits repos.txt — handy when only the Gitea side changed.

Outputs land under --corpus-dir (default <repo>/src/__Tests/e2e/scanner-analyzer-corpus/): corpus/, corpus-manifest.json, repos.txt, and (with --register-integration) integration-id.txt.

5. Verify Gitea has the repos

Count the bot’s repositories via the Gitea API:

curl -s -H "Authorization: token <pat>" \
    "http://gitea.stella-ops.local:3000/api/v1/repos/search?uid=$(\
      curl -s -H 'Authorization: token <pat>' \
        http://gitea.stella-ops.local:3000/api/v1/user | \
      sed -n 's/.*\"id\":\\([0-9]*\\).*/\\1/p')&limit=1" \
    | sed -n 's/.*\"total_count\":\\([0-9]*\\).*/repos: \\1/p'

Or simply browse to http://gitea.stella-ops.local:3000/e2e-bot — you should see the generated repository set. Weighted runs match --count; comprehensive runs may exceed --count when the per-scanner repo or coverage-case floor requires it.

The repos.txt emitted into the shared volume / --corpus-dir is the authoritative list:

docker compose -f docker-compose.e2e-analyzer-coverage.yml run --rm \
    --entrypoint sh corpus-provisioner -c \
    'grep -cv "^#" /var/lib/stellaops/e2e/corpus/repos.txt'

6. Point stella scm import at the corpus

The corpus-provisioner registers the SCM integration row and writes its GUID to integration-id.txt in the shared volume. Read it, then drive the bulk import (Sprint S1 CLI verb):

INTEGRATION_ID=$(docker compose -f docker-compose.e2e-analyzer-coverage.yml \
    run --rm --entrypoint cat corpus-provisioner \
    /var/lib/stellaops/e2e/integration-id.txt)

stella scm import \
    --integration "$INTEGRATION_ID" \
    --file src/__Tests/e2e/scanner-analyzer-corpus/repos.txt \
    --max-concurrent 20

stella scm import reads the TSV (name<TAB>cloneUrl[<TAB>branch], # comments and blank lines skipped), POSTs the bulk-import endpoint, and polls every 5 s until the batch is terminal. It exits 0 only when every repository succeeded.

7. Determinism guarantee

The harness is built so that the same seed always produces the same Gitea state:

• Corpus: the S2 factory is byte-deterministic — --seed 42 --count 600 always materialises a byte-identical corpus + corpus-manifest.json.
• Push: provision-corpus creates-or-reuses each Gitea repo and git push --forcees the deterministic single commit. Re-running is a no-op because the commit is byte-identical.
• repos.txt: emitted sorted under LC_ALL=C with LF line endings, so the file is byte-stable across runs and across the .sh / .ps1 variants.
• No timestamps in emitted artefacts: the scripts never write $(date) / DateTime.UtcNow into repos.txt, corpus-manifest.json, or integration-id.txt. The only timestamps are in human-readable log lines.

One intentional exception: the Gitea PAT is server-generated and rotates on every gitea-e2e-bootstrap run. The Vault path (secret/dev/scm/gitea) is stable; only the credential value rotates. This is correct credential hygiene and does not affect the e2e contract — the corpus and the Gitea repo set remain byte-deterministic.

8. Troubleshooting

9. GitHub / GitLab live targets (Sprint S5)

Sprint 20260514_005 (S5) adds opt-in, credential-gated live SCM targets — GitHub and GitLab — to the same StellaOps.E2E.AnalyzerCoverage harness, plus the nightly Gitea CI workflow. Live targets are not on the default CI lane: the scheduled nightly run only ever drives the local Gitea stack.

9.1 The two credential entry points (read this first)

There are two distinct credential paths, by design — do not conflate them:

This split mirrors the Gitea path: provision-corpus.sh reads E2E_GITEA_BOT_TOKEN (or Vault) for the push; the harness drives the bulk import by the registered integration id whose authref://vault/dev/scm/gitea the Scanner resolves. No new credential plumbing was added in S5 — the harness change is purely per-target defaults (see §9.4).

9.2 Provisioning the SCM credential through RFC-0001

The harness needs a registered SCM integration row whose authRefUri resolves to a valid token. RFC-0001 (docs/rfcs/RFC-0001-connector-credentials-storage.md) is the credential store; the authref:// URI scheme is opaque to the consumer.

GitHub — mint the PAT:

1. In the GitHub test org, create a fine-grained PAT (or a classic PAT) scoped to the test org only, with Contents: readand Metadata: readrepository permissions (classic equivalent: the repo scope). For the provisioning step the PAT additionally needs Administration: write(classic: repo) so it can create repos. You can use one PAT for both, or two — the provisioning PAT can be the broader one and the harness PAT the narrower read-only one.
2. Store the harness PAT in the credential backend:
   • Vault backend (the dev/CI default — same as the Gitea path): write it to a Vault KV path, e.g. vault kv put secret/dev/scm/github token=<pat>, and use authRefUri = authref://vault/dev/scm/github on the integration row.
   • Postgres backend (RFC-0001 §3.1 default for production single-node): create it through the Console at /admin/connectors/credentials or the Platform endpoint POST /platform/api/v1/connector-credentials; the row’s authRefUri is then the locator the integration references.
3. Register the SCM integration row (POST /api/v1/integrations) with provider = GitHubApp, endpoint = https://github.com (or the GHES /api/v3 base), and authRefUri pointing at the credential from step 2. Capture the returned integration GUID.

GitLab — mint the token:

1. In the GitLab test group, create a group access token (or a personal access token owned by a bot account) with the apiscope (needed to create projects + push). For a read-only harness token, read_api + read_repository suffice once the corpus is pushed.
2. Store it the same way — Vault KV (secret/dev/scm/gitlab) or the RFC-0001 Postgres backend — and note the authRefUri.
3. Register the SCM integration row with provider = GitLabServer, endpoint = https://gitlab.com (or the self-managed base), and the authRefUri from step 2. Capture the integration GUID.

The connector plugins (StellaOps.Integrations.Plugin.GitHubApp, .GitLab) resolve the credential at request time via IntegrationConfig.ResolvedSecret — populated server-side from the authRefUri. The harness never sees the secret.

9.3 Test-org / test-group setup + corpus push

The synthetic corpus is the same S2 corpus as the Gitea path — generated locally from in-repo fixtures, no real third-party repos. Push it to a dedicated test org/group (never a production namespace):

GitHub:

export GITHUB_E2E_TOKEN=ghp_xxx          # provisioning PAT (repo / Administration:write)
devops/scripts/e2e/provision-corpus-github.sh \
    --github-org stellaops-e2e \
    --seed 42 --count 600
# Windows: powershell -File devops/scripts/e2e/provision-corpus-github.ps1 -GitHubOrg stellaops-e2e

GitLab:

export GITLAB_E2E_TOKEN=glpat-xxx        # provisioning token (api scope)
devops/scripts/e2e/provision-corpus-gitlab.sh \
    --gitlab-group stellaops-e2e \
    --seed 42 --count 600
# Windows: powershell -File devops/scripts/e2e/provision-corpus-gitlab.ps1 -GitLabGroup stellaops-e2e

Both scripts are idempotent (create-or-reuse repos/projects, force-push the byte-identical deterministic commit) and emit a byte-stable repos.txt (name<TAB>cloneUrl) under out/e2e-corpus-{github,gitlab}/. Same --seed → same corpus → same repo set → same repos.txt. The repos are created private — keep the test corpus off the public internet.

9.4 The --target github|gitlab harness flags + throttle defaults

StellaOps.E2E.AnalyzerCoverage accepts --target gitea|github|gitlab. As of S5 all three are fully functional. Per-target defaults live in TargetProfile:

• When --integration-id is omitted, the harness resolves it from the per-target env var above — so CI can wire it via a secret without a CLI flag.
• When --max-concurrent is omitted, the per-target cap applies. An explicit --max-concurrent always wins — an operator who has warmed the rate limit can raise it.
• Phase 3 can optionally assert corpus breadth with --min-corpus-repos-per-plugin <n> and/or --min-corpus-cases-per-plugin <n>. These floors apply only to analyzer plugins with a corpus contract and produce warnings by default; add --enforce-corpus-breadth to make shortfalls fail Phase 3.
• The harness still drives Phase 2 by hitting the Scanner bulk-import API with --integration-id; only the defaults differ per target.

9.5 First-run rate-limit warm-up

The first live run is the riskiest for rate limits — the corpus push + per-repo bulk import + Phase 3 SBOM queries all hit the live API. Guidance:

• Provision first, run later. The corpus push (provision-corpus-*.sh) and the harness run are separate steps. Push the corpus when the rate budget is fresh; the harness run that follows only does the bulk import + SBOM queries.
• Start with --phases 1(the no-stack test-runner gate) to confirm the build is green before spending any rate budget.
• Then --phases 2alone against a small corpus (--count 50) to confirm the integration id + credential resolve, before the full 600-repo run.
• GitHub: check remaining budget via the connector’s health check (CheckHealthAsync surfaces remaining/limit) — keep headroom above ~20%.
• GitLab: the 600 req/min ceiling refills every minute; a stalled Phase 2 usually means the cap is too high — drop --max-concurrent and re-run (the bulk import is idempotent).
• A 429/403 rate limited from the live service surfaces as a Phase 2 failure with the upstream error in the per-repo outcome — wait for the window to refill and re-run.

9.6 Reading e2e-summary.json

Phase 4 writes e2e-summary.json (+ per-plugin sub-reports) under docs/qa/feature-checks/runs/scanner/analyzer-coverage-scm-import/<runId>/. Phase 3 is the contract gate. It always reads corpus-manifest.json for the per-repo corpus contract, but language analyzer purl-prefix authority is now manifest-first:

• For language analyzers, Phase 3 resolves plugins/scanner/analyzers/<language>/manifest.json through the DSSE-verifying ScannerPluginLoader and uses the verified signed manifest’s purlPrefixes when the manifest id matches the analyzer plugin. If those prefixes differ from the corpus seed table, the harness records a note and still treats the signed manifest as authoritative.
• The corpus manifest’s expectedPurlPrefixes remains the fallback for older or no-manifest paths: missing manifest, invalid manifest JSON, unverified or unsigned manifest, id mismatch, no purlPrefixes, missing trust store, or an unresolved repository root. It also remains the authority for non-language families such as OS, Homebrew, native, and Secrets coverage.
• Phase 3 is strict when scan evidence is available: every effective expected purl prefix must be observed in the scan-scoped /api/v1/sbom/{scanId}/evidence componentPurls for the selected corpus repos. The global hot-lookup component search is only a fallback for manifest-only runs with no Phase 2 repo-to-scan map. Every required evidence kind from each selected corpus repo must be observed on that repo’s scan. metadata, file, reachability, drift, and call-graph evidence are excluded from the required-evidence gate; reachability/drift remains a non-gating warning sub-assertion for the reachability-wired language analyzers.
• Dedicated call-path repos add an expectedReachability contract to the corpus manifest. When Phase 2 scan ids are available, Phase 3 requires a matching scan-scoped reachabilityFindings[] entry with the expected vulnerability id, component PURL, reachable/unreachable verdict, entrypoint, and ordered path symbols. Reachable contracts also require the vulnerable symbol to match; explicit unreachable controls instead require the ordered safe path to match so the stored evidence is not forced to pretend there is a path into vulnerable code. Missing or mismatched call-path evidence fails the plugin. Ordered path symbols are asserted for both reachable and explicit unreachable contracts. Manifest-only runs warn because there is no scan scope to query. Scanner WebService populates that evidence either from the scan evidence projection or, for GUID scan/source-run ids with persisted stack frames, by reconstructing ordered reachability paths from reachability_stack_frames.
• Secret coverage is also strict: every secret-tagged repo must have a Phase 2 scan id, meet its expectedSecretFindings.minCount, and include all expected detector classes, currently ast, regex, and provider. A missing class fails Phase 3 even if the total finding count is high enough.

Field order is stable (diff-friendly). The shape:

{
  "runId": "e2e-github-1234",
  "target": "GitHub",                  // which SCM target this run drove
  "generatedAt": "2026-05-14T03:31:02Z",
  "phases": {
    "testRunner":        { "status": "passed",  "detail": { ... } },
    "bulkImport":        { "status": "passed",  "detail": { "batchId": "...", "succeeded": 600, ... } },
    "contractAssertion": { "status": "passed",  "detail": { ... } }
  },
  "plugins": [
    {
      "id": "...", "language": "...",
      "phase1Status": "passed",        // the analyzer's test project (Phase 1)
      "phase3Status": "passed",        // the per-plugin purl/evidence contract (Phase 3)
      "corpusRepoCount": 35,            // selected corpus repos for this analyzer
      "corpusCaseCount": 31,            // distinct corpus coverageCaseId values
      "minimumCorpusRepoCount": null,   // configured --min-corpus-repos-per-plugin, or null
      "minimumCorpusCaseCount": null,   // configured --min-corpus-cases-per-plugin, or null
      "corpusBreadthStatus": "skipped", // skipped unless a minimum is configured; warning/fail on shortfall
      "expectedReachabilityRepoCount": 0,
      "observedReachabilityFindingCount": 0,
      "observedPurlPrefixes":  [ ... ],
      "observedEvidenceKinds": [ ... ]
    }
    // ... one row per analyzer plugin
  ],
  "overallStatus": "passed"            // the gating verdict — drives the exit code
}

• overallStatusis the gate: passed → exit 0; failed → exit 1. warning (a non-gating sub-assertion, e.g. reachability evidence) still exits 0.
• Triage order: overallStatus → the failing phases.*.status → for a Phase 3 failure, the plugins[] row whose phase3Status is not passed, then the per-plugin phase3-contract.json sub-report. Compare observedPurlPrefixes with the effective expected prefix set there: signed-manifest purlPrefixes for language analyzers when present, otherwise the corpus manifest’s expectedPurlPrefixes fallback. Also inspect corpusCaseCount, corpusBreadthStatus, the per-plugin corpusCoverageCaseIds, expectedEvidenceKinds vs observedEvidenceKinds, reachability finding counts, and any secret-detector class notes. corpusCaseCount is based on coverageCaseId, not just sourceScenario, so dependency depth, version-pin shape, mixed-stack, call-path reachable/unreachable state, and synthetic-secret variations count as distinct language scanner cases.
• Phase 1 failures are almost always a missing build — run dotnet build on the named analyzer test project first.

9.7 The CI workflow

.gitea/workflows/e2e-analyzer-coverage.yml (Sprint S5):

• schedule: nightly 03:30 UTC — runs the harness against the Gitea target only (the local S3 compose stack; no credentials, deterministic). Brings up docker-compose.e2e-analyzer-coverage.yml, waits for the corpus-provisioner one-shot, extracts repos.txt + integration-id.txt from the shared volume, runs --phases 1,2,3,4, uploads e2e-summary.json
  • the run-dir as artifacts, and fails the job on a non-zero harness exit.
• workflow_dispatch:— a target input (gitea|github|gitlab) picks the target and a phases input picks the phase subset. The github/gitlab targets are gated on the E2E_GITHUB_INTEGRATION_ID / E2E_GITLAB_INTEGRATION_ID secret being present; absent the secret the run skips with a ::notice:: and the job still succeeds (a missing optional credential is not a failure). The integration id from the secret is passed to the harness via the per-target env var (§9.4).
• pull_request:— smoke-tests the no-stack --phases 1 gate when the workflow, the harness, the provisioning scripts, the e2e compose, or this runbook is touched.

Live GitHub/GitLab runs assume the test org/group is already provisioned (§9.3) and the SCM integration row is already registered (§9.2) — the workflow only runs the harness against the registered integration id; it does not push the corpus.

10. Related documents

• Sprint files (archived as completed under docs-archive/implplan/2026-05-17-completed-sprints/): SPRINT_20260514_003_Scanner_local_gitea_e2e_harness.md (S3), SPRINT_20260514_004_Scanner_e2e_analyzer_coverage_harness.md (S4), SPRINT_20260514_005_Scanner_scm_live_targets_nightly_ci.md (S5).
• Plan: SCM-integration + analyzer-coverage e2e (Sprints S1–S5).
• Sprint S1 bulk import: src/Scanner/StellaOps.Scanner.WebService/Endpoints/ScmImportEndpoints.cs, src/Cli/StellaOps.Cli/Commands/Scm/ScmCommandGroup.cs.
• Sprint S2 corpus factory: src/E2E/StellaOps.E2E.SyntheticRepos/.
• Sprint S4 harness: src/E2E/StellaOps.E2E.AnalyzerCoverage/.
• Compose: devops/compose/docker-compose.e2e-analyzer-coverage.yml.
• Scripts: devops/scripts/e2e/provision-corpus.{sh,ps1} (Gitea), devops/scripts/e2e/provision-corpus-github.{sh,ps1}, devops/scripts/e2e/provision-corpus-gitlab.{sh,ps1} (S5 live targets), devops/scripts/e2e/gitea-e2e-bootstrap.sh.
• CI workflow: .gitea/workflows/e2e-analyzer-coverage.yml (S5 — nightly Gitea + workflow_dispatch-gated live targets).
• Credential store: RFC-0001 connector-credentials (RFC-0001-connector-credentials-storage.md).
• Determinism discipline precedent: offline-kit-assembly.md.