Operations

This page is for the people who run a host of the engine: retention, dead letters, health, metrics, logs, scaling and backups.

Retention

The retention job (AddWorkflowEngineHostedServices) runs once a day on one node at a time:

  1. It marks an open instance Stale, with its open tasks, when nothing changed it for WorkflowRetention:OpenStaleAfterDays (30). A stale instance stays in the store.
  2. It deletes a completed instance, with its tasks, task events and runtime state, WorkflowRetention:CompletedPurgeAfterDays (180) after completion. A completed task of an open instance is deleted on its own date.

The scheduled job never deletes an open instance; a run on demand with runningOlderThanDays does (below). An instance under legal hold, under the HOLD_ALL retention policy, or with an outbox row not yet delivered (Durable execution) is kept. A parent and its child workflows are deleted together, when every member is past its date. Stored definitions, the definition audit, the dead letters and the Designated Workflow ledger are never deleted by age.

{
  "WorkflowRetention": { "OpenStaleAfterDays": 60, "CompletedPurgeAfterDays": 365 },
  "WorkflowRetentionHostedJob": { "Enabled": true, "Interval": "1.00:00:00", "InitialDelay": "00:05:00" }
}

Run it once on demand (scope workflow:admin):

curl -s -X POST http://localhost:5090/api/workflow/retention/run \
  -H "Authorization: Bearer $ADMIN_TOKEN" -H "Content-Type: application/json" -d '{}'

The response counts what it did: staleInstancesMarked, staleTasksMarked, purgedInstances, purgedTasks, purgedTaskEvents, purgedRuntimeStates. A referenceUtc in the body evaluates the dates against another time than now. runningOlderThanDays and completedOlderThanDays purge instances by age, whatever their stored dates; runningOlderThanDays deletes open instances. Use them with care (Human tasks and the HTTP API).

Cases:

Dead letters

A signal whose processing fails WorkflowAq:MaxDeliveryAttempts times (10) in total, on any node, is moved to the dead letters. Its instance waits until the signal is replayed.

curl -s "http://localhost:5090/api/workflow/signals/dead-letters?workflowInstanceId=$INSTANCE_ID&maxMessages=50" \
  -H "Authorization: Bearer $TOKEN"

Filters: signalId, workflowInstanceId, signalType, maxMessages, includeRawPayload.

Replay one signal after you fixed the cause (scope workflow:admin):

curl -s -X POST http://localhost:5090/api/workflow/signals/dead-letters/replay \
  -H "Authorization: Bearer $ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"signalId":"<signal id>"}'

Procedure:

  1. List the dead letters of the instance and read the error.
  2. Fix the cause: a service that was down, a missing transport configuration, a definition error.
  3. Replay the signal. Expected result: "replayed": true, and the instance continues.
  4. If the same signal fails again, it returns to the dead letters after the delivery attempts.

Signal pump statistics

GET /api/workflow/signals/pump/stats returns the pump counters of the node that answers: processedCount, failureCount, deadLetterCount, emptyPollCount, concurrencySkipCount, the counts by signal type, and the time and detail of the last success and the last failure. The counters are kept in memory per node and start at zero with the process. concurrencySkipCount counts the signals returned to the queue because they kept losing the instance version check to concurrent changes (Persistence and signals).

Health

EndpointContent
/healthz (liveness)The process answers; no checks
/readyz (readiness)Every registered check, with the Workflow response writer

The engine registers one check, licence, which always reports Healthy and carries the licence status as data. Add the checks that decide readiness for your host, for example the database, with the ASP.NET Core health-check packages. (Hosting)

Metrics and traces

Meter or sourceInstrumentContent
Meter StellaOps.Workflow.LicenceGauge stellaops.workflow.licence.designated_workflowsThe Designated Workflow count, tagged licence.status
Meter StellaOps.WorkflowCounter workflow.execution.terminalOne count for each instance that finishes, on both runtime providers, after the transaction that finishes it commits: at start, at a durable start, when its last task is completed, or when a signal, a timer or a cancellation ends it. Labels: definition (the workflow name) and outcome (the runtime status the provider reports, Completed for a normal finish). A transition that rolls back is not counted
Meter StellaOps.WorkflowCounter workflow.console.requests, histogram workflow.console.duration.msExecution console requests, labelled operation and profile; only a host that calls AddWorkflowEngineDurableServices has the execution console
Activity source StellaOps.WorkflowActivities workflow.console.*Traces of the execution console

Every host has the StellaOps.Workflow meter: AddWorkflowEngineCoreServices registers it.

Export them with OpenTelemetry: add the meters StellaOps.Workflow and StellaOps.Workflow.Licence and the activity source StellaOps.Workflow to your meter and tracer providers. To read a counter in your own code without OpenTelemetry, use a MeterListener (lesson 9).

Logs

The engine logs through Microsoft.Extensions.Logging under the category names of its classes (StellaOps.Workflow.*). Useful levels:

CategoryLevelShows
StellaOps.Workflow.Engine.LicensingInformationThe licence status at start and on change; warnings at start and every hour while a status warns
StellaOps.Workflow.TransportWarningFailed calls, with the service, the status and the path
StellaOps.Workflow.Engine.HostedServicesWarningSignal pump failures and dead letters

Scaling

Backups

Back up the engine’s database or MongoDB database with your usual tools. A restore returns every instance, task, signal and stored definition to the backup time; signals that were due in between run after the restore. The Designated Workflow ledger is part of the database and keeps its entries.