Workflow Architecture

Scope. Architecture and operational contract for the Workflow Engine: its HTTP API, authorization model, PostgreSQL persistence, signal pump, and graph renderer. This dossier is for service implementers and for operators running or troubleshooting workflow instances (including the signal dead-letter runbooks).

Implementation status (2026-07-06): source retained, service dormant. The Workflow engine, storage backends, WebService project, and workflow PostgreSQL schema remain in source for possible future use. The canonical compose stack does not deploy StellaOps.Workflow.WebService, Router has no /api/workflow route, and ReleaseOrchestrator/Scheduler callers were removed. The gateway route ^/api/v1/workflows(.*) is configured to translate to ReleaseOrchestrator (Router .../appsettings.json:130), not to this service — though note that RO does not currently register any /api/v1/workflows handler (a grep of src/ReleaseOrchestrator/** finds the route only in the gateway config and a Doctor probe, not in RO’s endpoint map), so that path is a configured-but-unserved gateway route today, not a live surface either place. Reactivation requires an explicit product decision plus canonical scopes, client grants, authenticated callers, compose wiring, and a gateway route; none of those are implied by the retained source.

The retained Workflow Engine is a general-purpose runtime for definition deployment, instance lifecycle, signal-driven task transitions, dead-letter handling, and authorization-gated task progression. StellaOps.Workflow.WebService/Program.cs still contains Router integration and HTTP host composition so the source remains buildable, but that composition is not current deployment topology.

Retained HTTP API Surface

If the dormant WebService is hosted, its business endpoints are mapped under the /api/workflow route group (WorkflowEndpoints.MapWorkflowEndpoints). Every endpoint requires a named authorization policy (see “Authorization & Scopes”); state-changing endpoints additionally emit an audit record via .Audited(AuditModules.Workflow, …).

The retained host also maps operational/diagnostic endpoints outside the group: anonymous GET /healthz (liveness, Predicate => false), anonymous GET /readyz (readiness), the build-info endpoint (BuildInfoEndpointExtensions.MapBuildInfoEndpoint, fallback module workflow), and the OpenAPI document (MapOpenApi().AllowAnonymous()).

The POST /start request body is StartWorkflowRequest (WorkflowName required; WorkflowVersion, BusinessReference, and a Payload dictionary optional) and returns StartWorkflowResponse (WorkflowInstanceId, resolved WorkflowName/WorkflowVersion, NextTasks, WorkflowState). GET /instances and GET /tasks accept their filters as query parameters (workflowName, workflowVersion, workflowInstanceId, businessReferenceKey, status, assignee, includeDetails, actorId). RunRetention is the only endpoint that reads the wall clock (DateTime.UtcNow, explicitly annotated Determinism:Allowed) when no ReferenceUtc override is supplied.

Authorization & Scopes

The dormant WebService source uses a three-tier scope hierarchy defined locally in StellaOps.Workflow.WebService.Authorization.WorkflowPolicies. These scope strings are deliberately not registered in the canonical StellaOps.Auth.Abstractions.StellaOpsScopes catalog and no current client is granted them. They describe the retained endpoint policy shape, not a grantable production contract:

Named policies registered via AddWorkflowPolicies:

• View (workflow.view) — satisfied by any of workflow:view, workflow:operate, workflow:admin.
• Operate (workflow.operate) — satisfied by workflow:operate or workflow:admin.
• Admin (workflow.admin) — satisfied only by workflow:admin.

HasAnyScope reads both space-delimited scope claims (OAuth standard form, StellaOpsClaimTypes.Scope) and individual scp item claims (StellaOpsClaimTypes.ScopeItem), case-insensitively.

On any future reactivation, the gateway must remain the primary trust anchor: it validates bearer tokens, strips them, and forwards a signed identity envelope. The retained service verifies that envelope via UseIdentityEnvelopeAuthentication() and the WorkflowEnvelopeAuthenticationHandler authentication scheme, then runs the standard UseAuthentication()/UseAuthorization() pipeline. The defense-in-depth RequireAuthenticatedUser gates reject any direct in-cluster access that bypasses the gateway with 401. Do not restore callers using anonymous direct HTTP or network bypasses; provision a scoped service identity and signed gateway envelope.

PostgreSQL Persistence Schema

The Postgres backend’s tables are created by the forward-only migration baseline embedded in StellaOps.Workflow.DataStore.PostgreSQL (<EmbeddedResource Include="Migrations\**\*.sql" />) and applied on startup via AddStartupMigrations (module name Workflow.DataStore.PostgreSQL).

• 001_v1_workflow_datastore_postgresql_baseline.sql — the active pre-1.0 collapsed baseline. It creates wf_instances, wf_tasks, wf_task_events, wf_runtime_states, wf_host_locks, wf_signal_queue, wf_signal_dead_letters, wf_signal_wake_outbox, and wf_definitions.
• The former 001_initial_schema.sql and 002_wf_definitions.sql are preserved under Migrations/_archived/pre_1.0/mig061/ as historical inputs and are explicitly excluded from embedded resources.

The canonical Stella PostgreSQL schema is workflow. PostgresWorkflowBackendOptions.DefaultSchemaName is the single default used by both runtime qualification and migration registration, matching the workflow.-qualified baseline. SRD_WFKLW is a separate legacy Oracle backend schema and must not be used as the PostgreSQL default. Because the shipped baseline uses qualified DDL, arbitrary PostgreSQL schema overrides are not a supported migration contract.

The configurable table names (defaults) live in PostgresWorkflowBackendOptions: RuntimeStatesTableName = wf_runtime_states, HostedJobLocksTableName = wf_host_locks, InstancesTableName = wf_instances, TasksTableName = wf_tasks, TaskEventsTableName = wf_task_events, SignalQueueTableName = wf_signal_queue, DeadLetterTableName = wf_signal_dead_letters, WakeOutboxTableName = wf_signal_wake_outbox. LISTEN/NOTIFY channels default to workflow_signal and workflow_signal_wake_outbox.

Configuration Keys

WorkflowRuntimeProviderNames exposes the two runtime-provider identifiers used in persisted state: Stella.InProcess and Stella.Engine.

Runtime Persistence Contract

If reactivated, Workflow WebService runs against PostgreSQL-backed runtime stores outside Development and Testing. WorkflowBackend:Provider must be Postgres, and ConnectionStrings:WorkflowPostgres or ConnectionStrings:Default must be configured so startup migrations can converge the workflow schema before request handling. No current canonical host starts this WebService.

Production startup rejects authoritative local fallbacks for runtime state, hosted-job locks, definition storage, signal stores, signal drivers, wake outbox, and signal dead letters. The engine-level null and in-memory store implementations remain available only for explicit local/test harnesses.

Workflow transport defaults are not mocks. When no HTTP, Rabbit, legacy Rabbit, GraphQL, or microservice transport plugin is registered, the engine binds explicit unconfigured transports that return failed execution responses naming the missing plugin. They do not enqueue, call, or acknowledge work, so production workflows fail closed until the required transport plugin is registered by the host.

Definition imports fail closed when no backend-specific definition store is registered. The null definition store never acknowledges writes or activations because that would make workflow deployment look successful without persisting deployable definitions.

PostgreSQL Runtime Bindings

When the Postgres backend is active, the WebService uses PostgreSQL implementations for runtime state, hosted-job locks, projections, definition storage, signal store/claims/scheduler, wake outbox sender/receiver, and signal dead-letter storage. The native signal driver is enabled by WorkflowSignalDriver:Provider = Native.

Renderer Layout Engines

Workflow definitions are visualised through IWorkflowRenderLayoutEngine providers. Three implementations are wired in this repo:

• StellaOps.Workflow.Renderer.ElkSharp — pure C# port of the Eclipse Layout Kernel (StellaOps.ElkSharp). Runs in-process; no JVM, no IPC, no subprocess.
• StellaOps.Workflow.Renderer.ElkJs — shells out to a Node.js script bundled under tools/elk-layout/ that drives the upstream elkjs package. The wrapper performs an idempotent npm ci on first use guarded by a SemaphoreSlim (no per-call install). Each layout call writes one JSON input file, spawns one node process per call, and reads one JSON output file.
• StellaOps.Workflow.Renderer.Msagl — Microsoft Automatic Graph Layout, in-process .NET.

Lifecycle decision (sprint SPRINT_20260429_005_Workflow_renderer_jvm_lifecycle.md)

The sprint was created on a hypothesis that StellaOps.ElkSharp wraps a JVM child process (the upstream Eclipse project ELK is implemented in Java) and that the renderer test host crashed because of JVM forking, deadlocked stdin/stdout buffers, or unhandled JVM aborts. The hypothesis is incorrect: StellaOps.ElkSharp is a pure C# port of the algorithm, not a wrapper. There is no JVM, no Java process, no IPC, and no subprocess in the ElkSharp call path.

The renderer test host did not crash. It was hanging because a single Best-effort layout of the canonical 24-node DocumentProcessingWorkflow takes ~60 s in Release on a typical dev box (the iterative-routing path explores many candidate strategies). The renderer test suite contained several fixtures that performed dozens of such layouts back-to-back, far exceeding VSTest’s default --blame-hang-timeout (3 min). VSTest then collected a hangdump of the alive-but-busy worker process and reported “test host crashed.”

Decisions (no JVM-worker pattern was needed):

• ElkSharp remains pure in-process. No long-lived worker, no IPC, no JVM bring-up. Each LayoutAsync call is a regular method invocation against an ElkSharpLayeredLayoutEngine instance.
• The renderer test fixtures that drive the 24-node Best-effort layout (DocumentProcessingWorkflowRender, DocumentProcessingWorkflowRenderBenchmark, DocumentProcessingWorkflowRenderingTests, RenderReviewHarness, RenderReviewAnalyzer) are marked [Explicit] so they are excluded from the default suite. They remain runnable on demand via NUnit class/name filters.
• A standalone reproducer at src/Workflow/__Libraries/StellaOps.Workflow.Renderer.ElkJs/tools/elk-stress exercises ElkSharp’s full call path outside the test runner. It serves both as the diagnostic harness for the original sprint and as a performance regression guard for future ElkSharp changes.
• The renderer CI lane (.gitea/workflows/workflow-renderer.yml) runs the default test suite on every relevant PR with --blame --blame-hang-timeout=180000 and fails the build if any hangdump is captured. A separate nightly job runs the [Explicit] fixtures and the elk-stress harness.

There is no Java runtime requirement on the production host for ElkSharp rendering. The ElkJs provider needs Node.js available on the host that runs the renderer. ElkSharp is the default provider used by the engine and tests.

ElkSharp edge-correctness contracts

Sprints SPRINT_20260430_081_Workflow_renderer_elksharp_correctness.md and SPRINT_20260501_053_Workflow_elksharp_aspirational_helpers_followup.md define the current edge-routing contracts for the in-process ElkSharp provider.

• Boundary-slot assignments are the canonical lattice for source exits and target entries. Late cleanup phases must preserve already-assigned slot points when they are still on the owning node boundary, and singleton edges fall back to the owning face’s discrete slot instead of keeping arbitrary off-lattice coordinates.
• Gateway source exits and gateway target entries intentionally use different validity rules. Source exits must leave from a face interior and are repaired away from left/right diamond tip vertices. Target entries may converge at left/right tip vertices only when the target-specific boundary-angle validator accepts the approach. Generic gateway boundary-angle checks still reject tip vertices.
• Mixed-face, source-departure, target-approach, and under-node repair phases are stabilization passes over the same boundary-slot contract. A repair is accepted only when it reduces the targeted violation family without reintroducing source-exit, boundary-angle, crossing, or under-node defects.
• Protected under-node candidates are still eligible for final detour closure after the protected edge no longer has an under-node violation. The deterministic renderer tests use synthetic fixtures for that contract; slow document-processing render reviews remain explicit/on-demand harnesses.
• ElkEdgeNodeAvoidance.ShiftSection runs a post-shift visual-clearance nudge after the free- interval shift. The free-interval shift only considers peers that are primary-adjacent (within PrimaryAdjacencyMargin = 8 px along the segment’s primary axis); peers outside that window can still have a cross boundary close to the shifted coord. The nudge loop walks the same coord outward in the shift direction until it clears every peer’s cross boundary by at least the visual margin (DefaultMargin = 12 px). Source-exit and target-entry endpoints stay anchored to their owning node boundary regardless of the nudge.

Operational notes

• Hangdumps from the renderer CI lane are uploaded as artifacts under renderer-tests-hangdumps. The build fails when one is captured so a perf regression in ElkSharp’s iterative-routing budget cannot silently push the suite past the blame timeout again.
• For incident response on a renderer hang locally, repro with: dotnet run --project src/Workflow/__Libraries/StellaOps.Workflow.Renderer.ElkJs/tools/elk-stress -- --iterations 5 --effort Best --direction TopToBottom. If a single iteration takes >2 min in Release, ElkSharp’s iterative-routing has regressed and needs profiling against the canonical document-processing graph.

Signal pump retry budget (WorkflowSignalDeliveryAttemptTracker)

The signal pump worker (StellaOps.Workflow.Engine.HostedServices.WorkflowSignalPumpWorker) owns the authoritative retry budget for workflow signal envelopes. Sprint SPRINT_20260430_080_Workflow_oracle_aq_retry_budget reworked the budget semantics after the original implementation relied on the transport-reported IWorkflowSignalLease.DeliveryCount, which is not authoritative for Oracle AQ under the worker’s Visibility = OnCommit rollback flow.

Semantics:

• The worker tracks attempts per WorkflowSignalEnvelope.SignalId in a per-process WorkflowSignalDeliveryAttemptTracker (DI-singleton). On each RunOnceAsync dispatch the tracker increments the count for the envelope’s SignalId and returns the new value.
• When the count reaches WorkflowAq:MaxDeliveryAttempts and the processor throws, the worker calls IWorkflowSignalLease.DeadLetterAsync(), which forwards the message to the configured user dead-letter queue (WorkflowAq:DeadLetterQueueName) and forgets the tracker entry.
• On successful processing or a WorkflowRuntimeStateConcurrencyException skip, the tracker entry is forgotten so a future re-publish of the same signal id starts again from attempt 1.
• lease.DeliveryCount is now informational only; transports report their best-effort view (Oracle AQ surfaces DequeueAttempts, Redis surfaces a driver-defined value), but the budget no longer depends on monotonic agreement between transport and worker.

Cross-process safety net:

• The tracker is per-process; counts reset on host restart. Oracle AQ’s queue-side max_retries (default 5) acts as the cross-host safety net so a poison message that survives a worker restart is eventually moved to Oracle’s internal exception queue (AQ$_<table>_E) regardless of in-process state. The user-defined dead-letter queue (WorkflowAq:DeadLetterQueueName) is populated only when the worker reaches its budget; Oracle’s automatic max_retries handoff targets the AQ-internal exception queue, not the user DLQ. Operators should monitor BOTH the user DLQ and the AQ-internal exception queue when investigating stuck signals.

Configuration binding:

• Both StellaOps.Workflow.Engine.Hosting.WorkflowAqOptions (consumed by the pump and hosted service) and StellaOps.Workflow.Signaling.OracleAq.WorkflowAqOptions (consumed by the transport, signal bus, schedule bus, and dead-letter store) bind from the WorkflowAq configuration section. Prior to sprint SPRINT_20260430_080 only the signaling-side copy was bound, so host-supplied values for MaxDeliveryAttempts, BlockingDequeueSeconds, and ConsumerName silently fell back to type-defaults at the worker layer. The duplicate option class is documented as a TODO at the top of Engine/Hosting/WorkflowAqOptions.cs; consolidation is deferred.

Operator runbook for inspecting stuck signals:

• User DLQ inspection (worker-driven dead-letters):

  SELECT msgid, corrid, enq_time, retry_count
    FROM <schema>.<DeadLetterQueueTableName>
   ORDER BY enq_time DESC;
  

• Oracle AQ-internal exception queue inspection (max_retries handoff):

  -- Oracle stores the queue's exception queue under AQ$_<table>_E by default.
  -- Confirm via DBA_QUEUE_TABLES.
  SELECT msgid, corrid, msg_state, retry_count, enq_time
    FROM <schema>.AQ$_<SignalQueueTableName>_E
   ORDER BY enq_time DESC;
  

• Cross-reference corrid (which the engine sets to WorkflowSignalEnvelope.SignalId) with wf_signal_dead_letters runtime store rows to identify the workflow instance that owns the stuck signal. Replay via the POST /api/workflow/signals/dead-letters/replay endpoint (Admin scope), which calls WorkflowSignalDeadLetterService.ReplayAsync(WorkflowSignalDeadLetterReplayRequest), once the underlying processor failure is resolved.

Oracle/Redis integration test fixture

The Workflow Oracle data store and Oracle AQ signaling are exercised end-to-end by StellaOps.Workflow.DataStore.Oracle.Tests via WorkflowOracleRedisFixture (sprint SPRINT_20260429_004_Workflow_oracle_test_fixture.md, D-009). The fixture spawns one Oracle Free 23 slim-faststart container and one Redis 7-alpine container per test run (NUnit [SetUpFixture] lifetime), and exposes a BuildServiceProvider(...) helper that mirrors the production host’s DI wiring with the Oracle backend selected.

Key wiring decisions encoded in the fixture, recorded here so they persist beyond the originating sprint:

• OracleManagedAqTransport reads ConnectionStrings:DefaultConnection directly while AddWorkflowOracleDataStore reads ConnectionStrings:WorkflowOracle (then Default). Production hosts must seed all three keys with the Oracle connection string.
• OracleManagedAqTransport injects the base Microsoft.EntityFrameworkCore.DbContext type, but AddDbContext<WorkflowDbContext> only registers the concrete subclass. Hosts wiring Oracle AQ must alias the base type: services.AddScoped<DbContext>(sp => sp.GetRequiredService<WorkflowDbContext>()).
• AddWorkflowRedisSignaling consumes IConnectionMultiplexer but does not own its lifetime. Hosts must register one, typically as a singleton bound to the configured Redis endpoint.

CI lane: .gitea/workflows/workflow-oracle-tests.yml runs the suite on PRs that touch the Oracle/Redis libraries or tests, with cached docker save archives keyed on the pinned image tags so cold-cache pulls happen once per tag bump rather than per PR.

Test-pattern reference and per-test cleanup discipline: see “Oracle / Redis test fixtures” in docs/code-of-conduct/TESTING_PRACTICES.md.