Operations
This page is for the people who run a host of the engine: retention, dead letters, health, metrics, logs, scaling and backups.
Retention
The retention job (AddWorkflowEngineHostedServices) runs once a day on one node at a time:
- It marks an open instance
Stale, with its open tasks, when nothing changed it forWorkflowRetention:OpenStaleAfterDays(30). A stale instance stays in the store. - It deletes a completed instance, with its tasks, task events and runtime state,
WorkflowRetention:CompletedPurgeAfterDays(180) after completion. A completed task of an open instance is deleted on its own date.
The scheduled job never deletes an open instance; a run on demand with runningOlderThanDays does (below). An instance under legal hold, under the HOLD_ALL retention policy, or with an outbox row not yet delivered (Durable execution) is kept. A parent and its child workflows are deleted together, when every member is past its date. Stored definitions, the definition audit, the dead letters and the Designated Workflow ledger are never deleted by age.
{
"WorkflowRetention": { "OpenStaleAfterDays": 60, "CompletedPurgeAfterDays": 365 },
"WorkflowRetentionHostedJob": { "Enabled": true, "Interval": "1.00:00:00", "InitialDelay": "00:05:00" }
}
Run it once on demand (scope workflow:admin):
curl -s -X POST http://localhost:5090/api/workflow/retention/run \
-H "Authorization: Bearer $ADMIN_TOKEN" -H "Content-Type: application/json" -d '{}'
The response counts what it did: staleInstancesMarked, staleTasksMarked, purgedInstances, purgedTasks, purgedTaskEvents, purgedRuntimeStates. A referenceUtc in the body evaluates the dates against another time than now. runningOlderThanDays and completedOlderThanDays purge instances by age, whatever their stored dates; runningOlderThanDays deletes open instances. Use them with care (Human tasks and the HTTP API).
Cases:
- Audit requirements. Raise
CompletedPurgeAfterDaysto your retention period. - Long-running processes. Raise
OpenStaleAfterDaysabove the longest normal pause of a workflow, so that a waiting instance is not marked stale. - A separate retention node. Disable the job on the request-serving nodes (
WorkflowRetentionHostedJob:Enabled = false) and leave it on one worker node. - Changing the retention periods. The stale and purge dates of a row are fixed when the row is written. A new
OpenStaleAfterDaysorCompletedPurgeAfterDaysapplies to rows written after the change, not to the rows already stored.
Dead letters
A signal whose processing fails WorkflowAq:MaxDeliveryAttempts times (10) in total, on any node, is moved to the dead letters. Its instance waits until the signal is replayed.
curl -s "http://localhost:5090/api/workflow/signals/dead-letters?workflowInstanceId=$INSTANCE_ID&maxMessages=50" \
-H "Authorization: Bearer $TOKEN"
Filters: signalId, workflowInstanceId, signalType, maxMessages, includeRawPayload.
Replay one signal after you fixed the cause (scope workflow:admin):
curl -s -X POST http://localhost:5090/api/workflow/signals/dead-letters/replay \
-H "Authorization: Bearer $ADMIN_TOKEN" -H "Content-Type: application/json" \
-d '{"signalId":"<signal id>"}'
Procedure:
- List the dead letters of the instance and read the error.
- Fix the cause: a service that was down, a missing transport configuration, a definition error.
- Replay the signal. Expected result:
"replayed": true, and the instance continues. - If the same signal fails again, it returns to the dead letters after the delivery attempts.
Signal pump statistics
GET /api/workflow/signals/pump/stats returns the pump counters of the node that answers: processedCount, failureCount, deadLetterCount, emptyPollCount, concurrencySkipCount, the counts by signal type, and the time and detail of the last success and the last failure. The counters are kept in memory per node and start at zero with the process. concurrencySkipCount counts the signals returned to the queue because they kept losing the instance version check to concurrent changes (Persistence and signals).
Health
| Endpoint | Content |
|---|---|
/healthz (liveness) | The process answers; no checks |
/readyz (readiness) | Every registered check, with the Workflow response writer |
The engine registers one check, licence, which always reports Healthy and carries the licence status as data. Add the checks that decide readiness for your host, for example the database, with the ASP.NET Core health-check packages. (Hosting)
Metrics and traces
| Meter or source | Instrument | Content |
|---|---|---|
Meter StellaOps.Workflow.Licence | Gauge stellaops.workflow.licence.designated_workflows | The Designated Workflow count, tagged licence.status |
Meter StellaOps.Workflow | Counter workflow.execution.terminal | One count for each instance that finishes, on both runtime providers, after the transaction that finishes it commits: at start, at a durable start, when its last task is completed, or when a signal, a timer or a cancellation ends it. Labels: definition (the workflow name) and outcome (the runtime status the provider reports, Completed for a normal finish). A transition that rolls back is not counted |
Meter StellaOps.Workflow | Counter workflow.console.requests, histogram workflow.console.duration.ms | Execution console requests, labelled operation and profile; only a host that calls AddWorkflowEngineDurableServices has the execution console |
Activity source StellaOps.Workflow | Activities workflow.console.* | Traces of the execution console |
Every host has the StellaOps.Workflow meter: AddWorkflowEngineCoreServices registers it.
Export them with OpenTelemetry: add the meters StellaOps.Workflow and StellaOps.Workflow.Licence and the activity source StellaOps.Workflow to your meter and tracer providers. To read a counter in your own code without OpenTelemetry, use a MeterListener (lesson 9).
Logs
The engine logs through Microsoft.Extensions.Logging under the category names of its classes (StellaOps.Workflow.*). Useful levels:
| Category | Level | Shows |
|---|---|---|
StellaOps.Workflow.Engine.Licensing | Information | The licence status at start and on change; warnings at start and every hour while a status warns |
StellaOps.Workflow.Transport | Warning | Failed calls, with the service, the status and the path |
StellaOps.Workflow.Engine.HostedServices | Warning | Signal pump failures and dead letters |
Scaling
- More throughput per node. Raise
WorkflowEngine:MaxConcurrentSignalHandlersand the store’s pool size together. - More nodes. Add nodes on the same store; they share the signals by lease and the retention by host lock (Persistence and signals).
- The store. Every node works on the same database, so the database bounds the total throughput: size it, and its connection limit, for all nodes together.
- Separate roles. Serve the HTTP API from some nodes and run the signal pump on others: call
AddWorkflowEngineHostedServicesonly on the pump nodes. The API nodes then need the health checks service registered themselves (AddHealthChecks()). Two consequences on an API node: a request whose next steps run through the signal pump (for example a child workflow) waits the fullWorkflowEngine:MaxInlineWaitSecondsand returns without the next tasks, because only a pump on the same node ends that wait; and the node has no licence monitor and nolicencehealth entry. - Large maps. Bound them with
WorkflowRuntime:MaxMapConcurrencyandMaxMapItems.
Backups
Back up the engine’s database or MongoDB database with your usual tools. A restore returns every instance, task, signal and stored definition to the backup time; signals that were due in between run after the restore. The Designated Workflow ledger is part of the database and keeps its entries.
