Skip to content

Workers & recovery

Implemented

Shared executor coordination and domain fences have integration coverage. Deployed alerts and operational restore acceptance are separate.

Code and tests

The shared worker runtime embeds DBOS and coordinates executor leases in PostgreSQL. A new process gets a new executor identity. Recovery selects compatible work from expired owners and preserves durable invocation identity.

sequenceDiagram
  participant A as Worker A
  participant DB as PostgreSQL
  participant B as Worker B
  A->>DB: Renew lease and checkpoint
  Note over A: Process pauses or dies
  B->>DB: Revoke expired owner and re-enqueue compatible work
  B->>DB: Claim and replay checkpoints
  A->>DB: Resumed process attempts a write
  DB-->>A: Reject revoked executor

A paused process can resume after a peer has taken over. Lease expiry alone does not stop its code, so every publication path needs an authority check at the database write boundary.

An executor holds its lock through an admitted transaction, so revocation cannot race a commit already in progress. Product publication also checks task generation, cancellation and permission: a live worker lease does not authorize an obsolete task. Document jobs check the base revision before writing, and legal indexing checks source freshness and attempt ownership before publishing. These checks reject late output after a newer version or worker has taken over.

A successful private probe does not establish that the business queue is dequeuing. The runtime separately monitors eligible backlog and claims. This distinction is exercised by TestExecutorBusinessQueueWatchdog in packages/workflows-go/queue_watchdog_integration_test.go.

Lease, renewal, watchdog and recovery limits live in packages/workflows-go/runtime.go, recovery.go and the queue-watchdog implementation. The pinned DBOS dependency lives in go.mod; the checked catalog version lives in recovery.go. These values must be reviewed with the adapter and its tests when changing the SDK, rather than copied into this page.

Poison workflows eventually exhaust recovery. Keeping them terminal avoids an endless resource-consuming replay loop. TestExecutorPoisonRecovery, TestExecutorExtendedRecovery and TestAIXRecoveryExhaustionSettlement cover runtime and business-state reconciliation. Fixing an exhausted workflow requires identifying its cause; resetting its retry count is not a recovery proof.

The runtime uses SDK shutdown to leave unfinished work recoverable. Its readiness method checks launch state, coordination progress and lease validity. runtime.go and the process recovery tests exercise shutdown, paused processes and watchdog termination.

The owner page for already-dispatched provider calls is Model gateway; process recovery cannot establish whether a provider billed a call. The owner page for cleanup and restore limitations is Storage & indexes.

Run ./bin/hermit test tests/suites/data-foundation/worker-coordination.yaml for shared-runtime recovery. The suite file owns selected test names and timeouts. Product and corpus suites additionally prove their domain publication checks. Deployed alert delivery and operational restore readiness remain environment work.