Worker coordination and recovery

Kesita runs DBOS inside the product and corpus workers. There is no Conductor service. Each worker renews a PostgreSQL lease, checks its own queue runner, and scans for expired peers. PostgreSQL decides who may still write; DBOS supplies durable queues and checkpoints.

This is a small application-owned coordination layer, not a replacement for all Conductor features. We own its failure detection, recovery adapter, testing, and operations.

Open the interactive recovery sequence.

Components and ownership

flowchart TB
    subgraph WorkerA[Worker A: unique process ID]
        RuntimeA[Lease renewal and recovery scanner]
        QueueA[DBOS business queues]
        ProbeA[Private diagnostic queue]
        WatchA[Local watchdog]
    end
    subgraph WorkerB[Worker B: different process ID]
        RuntimeB[Lease renewal and recovery scanner]
        QueueB[DBOS business queues]
    end
    Registry[(workflow_coordination.executor)]
    Journal[(DBOS queues and checkpoints)]
    Business[(Application tables)]
    RuntimeA --> Registry
    RuntimeB --> Registry
    RuntimeA -->|re-enqueue expired peers| Journal
    RuntimeB -->|re-enqueue expired peers| Journal
    QueueA --> Journal
    QueueB --> Journal
    WatchA -->|checks progress| ProbeA
    QueueA -->|lease-guarded writes| Business
    QueueB -->|lease-guarded writes| Business

Product and corpus use separate owning databases. Executor IDs are fresh UUIDs on every process start, including container restarts. Compatibility versions are application-scoped (kesita-product-v1, kesita-corpus-v1); recovery never crosses application or version boundaries. Do not set SDK identity/version overrides.

Migrations own workflow_coordination, its lease table, and application-table write triggers. The shared schema source is packages/workflows-go/coordination.sql. The product baseline builder includes it; the legal baseline includes the same definition. DBOS initializes its own catalog. Before launching queues, the runtime checks that catalog and installs lease guards on DBOS mutation tables. This narrow adapter is pinned to DBOS Go 1.4.0, catalog 113; review and test it whenever upgrading DBOS.

Failure detection

Mechanism	Setting	Purpose
Lease renewal	Every 10 seconds	Prove that an executor can still coordinate with PostgreSQL
Lease validity	75 seconds	Permit peer takeover after renewal stops
Recovery scan	Every 5 seconds, up to 100 executors	Recover compatible expired or revoked owners
Diagnostic workflow	Every 15 seconds, one outstanding	Exercise the private diagnostic queue independently of business capacity
Business queue monitor	Every 5 seconds	Detect eligible backlog with spare capacity but no queue claims for 136 seconds; increases for configured polling intervals above 120 seconds
Local watchdog	45 seconds without heartbeat, scan, or probe progress	Stop a running process whose coordination has stalled
Forced exit	5 seconds after watchdog failure	Bound shutdown when application or SDK code will not stop
Application step budget	Execute: 30 minutes; settle: 1 minute	Bound a stuck individual step; durable waits are outside this budget

A step that ignores its cancellation gets five seconds to cooperate, then triggers the watchdog’s five-second forced-exit deadline. Normal provider calls retain their shorter HTTP timeouts. No finite timeout distinguishes all slow work from hung work: these budgets deliberately favor recovery over indefinite execution.

The heartbeat watchdog starts at the successful renewal request’s send time, not its response time. Database response latency cannot extend the safety window. PostgreSQL only renews a lease that is still active and unexpired; a delayed or resumed executor cannot resurrect an expired lease.

The private diagnostic queue has concurrency one. Business queues at capacity do not consume that slot. DBOS Go 1.4.0 runs each queue in a separate goroutine: a successful private probe does not prove that business dequeuing works. The business monitor separately observes claims and eligible backlog, accounting for global, worker, partition, and rate limits and application versions. Delayed workflows do not count as eligible. Its 136-second budget allows the SDK’s 120-second maximum contention backoff, 5% jitter, and two observation intervals; monitoring aggregates active and recent rows rather than all completed history.

This detects queue-wide starvation; a stalled replica can remain undetected while peers keep that queue progressing. It also cannot distinguish genuinely busy PENDING slots from claims that never reached workflow dispatch: the SDK does not expose that per-queue in-memory execution state. Investigate persistently pending work even when probes pass. Individual executing steps have the separate cancellation budget above.

Successful probe records are deleted immediately; peer scans remove abandoned probes and private queues belonging to revoked executors. This cleanup is required: each replica produces about 5,760 probes per day. Executor tombstones are retained to reject late incarnations.

Transfer of work

sequenceDiagram
    participant A as Worker A
    participant DB as PostgreSQL
    participant B as Worker B
    A->>DB: Renew lease; checkpoint work
    Note over A: Crashes or stops making progress
    B->>DB: Scan expired peers
    Note over DB: Lease expires using database time
    B->>DB: Transaction: revoke A and re-enqueue its PENDING work
    DB-->>B: Commit together
    B->>DB: DBOS claims work and reads checkpoints
    Note over B: Completed steps replay; unfinished steps may run again
    A->>DB: A resumes and tries to write
    DB-->>A: Reject expired or revoked executor

Scanners use row locks and SKIP LOCKED, without electing a leader. Revocation and recovery are one transaction: an error or scanner crash cannot commit just one half. Recovery changes only compatible PENDING workflows. It preserves IDs, inputs, queue assignment, deduplication IDs, priorities, partition keys, durable deadlines, checkpoint history, and recovery counters. Cancelled, completed, errored, and already-enqueued workflows are not restarted by this scanner.

Direct workflows, including children, are assigned to _dbos_internal_queue during recovery, matching the pinned SDK’s behavior. Every runtime listens to that queue. They do not remain stranded because their original queue name was empty.

Poison workflows and capacity

Business workflows allow three recovery retries: four executions total, including the original execution. DBOS increments recovery_attempts when claiming an execution, not when re-enqueuing. The adapter must not increment it a second time. After the fourth worker loss, recovery atomically marks the workflow MAX_RECOVERY_ATTEMPTS_EXCEEDED instead of re-enqueuing it. The terminal row is retained and a workflow_recovery_exhausted error signal identifies the workflow without logging its input.

Independent reconciliation settles exhausted product tasks/runs and corpus jobs as failed without executing the poison workflow again. Product purge intents remain quarantined for operator repair because they have no failed business state. Settlement failures are retried and emit workflow_recovery_settlement; do not delete the terminal DBOS record before reconciliation completes. Inspect the offending workflow and fix its cause before explicitly retrying it—automatic resume would reset the protection.

Recovered work enters normal SDK queues. Product and corpus queues both set their global and per-worker concurrency to the configured concurrency value, so adding replicas or recovering many workflows does not multiply the workflow concurrency budget. This is not a universal provider request/rate limit: parallel calls inside one workflow still require provider-specific admission controls.

Revoked owners are checked again if they have pending work. This handles a late claim from a previously paused runner. Database guards additionally prevent that runner’s normal SDK pool from making the claim in the first place.

With an available compatible peer and PostgreSQL, expect takeover after lease expiry plus scan and queue scheduling delay—roughly 75–80 seconds from the last successful renewal, not a hard end-to-end completion SLA. An orderly shutdown can release its lease earlier. If every replica is down, work waits for one to start; PostgreSQL cannot execute workflows itself.

Why a stale worker cannot publish

flowchart LR
    Write[Application or DBOS write] --> Guard{Executor lease active and unexpired?}
    Guard -->|yes: shared row lock| Commit[Transaction may commit]
    Guard -->|no| Reject[Reject mutation]
    Revoke[Recovery revokes lease] -->|exclusive update waits for admitted transactions| Guard
    Provider[Outbound provider or blob call] --> Check[Check lease before sending]
    Check --> External[External system]

Runtime-owned application and DBOS pools set kesita.executor_id when opening connections. Statement triggers check the lease and hold a shared executor-row lock through the transaction. Revocation must wait for already-admitted transactions, so takeover cannot race their commit. Database statement and idle-transaction timeouts keep these transactions bounded.

The guard supplements—not replaces—existing tenant authorization, task generations, document revisions, corpus attempt fences, cancellation, and recovery epochs. API/migrator pools and direct domain tests have no executor setting and retain their existing authorization. Always use Runtime.ConfigurePool for a worker’s business pool and handle its error: it rejects a different database, host, port, or failover endpoint from the registry’s configuration. Credentials may differ; endpoint aliases intentionally fail closed. A newly added table needs the migration-owned trigger. The separate control ledger is read for admission checks, not used for worker result publication. This is cooperative runtime fencing, not a security boundary against a malicious holder of worker database credentials.

Before outbound HTTP and blob operations, application code also verifies its lease. Already-sent external calls cannot be recalled atomically. A provider may finish after ownership is lost, and a crash between an external effect and its checkpoint may cause re-execution. Preserve idempotency keys, provider receipts, and unknown-outcome handling; never equate lease expiry with proof that a provider did nothing or that billing is safe to retry blindly.

Shutdown and health

An infrastructure shutdown stops application work and calls the SDK’s Shutdown API. The SDK root is intentionally detached from ordinary parent cancellation: plain context cancellation can durably cancel workflows, whereas SDK shutdown leaves them recoverable. The runtime releases its lease only after SDK shutdown reports success; otherwise it leaves recovery to expiry. Heartbeats stop during this bounded shutdown.

/health/live checks HTTP responsiveness. /health/ready also checks launch state, recent heartbeat/scan/probe progress, and the current lease. The product worker additionally checks its business database and configured control ledger. A local watchdog exits on detected failure; health probes are not themselves the takeover protocol.

Terraform configures at least two replicas for each enabled worker service, including staging, and rejects a maximum below two. Local Compose normally has one worker per service: it can recover after restart but has no surviving peer during a complete process pause. Compose health status alone does not restart an unhealthy container; restart: unless-stopped restarts processes that exit. A process frozen so thoroughly that its watchdog cannot run still needs an external supervisor to terminate it, while a healthy peer can recover its work after expiry.

Operations and verification

Database outages

Loss of PostgreSQL stops progress across the pool. A lease check before a step or outbound call fails closed immediately; otherwise the coordination watchdog stops the worker after its 45-second budget. Replicas can therefore exit together. While PostgreSQL is unavailable, replacement workers retry startup with cancellable exponential jitter: initially 0.5–1 second, capped at 15–30 seconds between attempts. Each attempt uses a fresh executor identity and SDK context. Invalid configuration, authentication, permissions, or incompatible schema fail fast rather than retrying indefinitely.

Startup remains unready until initialization succeeds; infrastructure may still restart a container during its startup-probe budget. Backoff reduces database pressure, not the need to restore PostgreSQL. Once the database returns, expired owners are revoked and pending work may be recovered in a burst. Queue concurrency still applies. Database outages consume execution attempts for interrupted work too; repeated outages can exhaust the same safety budget. Expect restart, recovery, and potentially quarantine alerts—do not reset counters in bulk.

Signals and tests

Watch structured operations executor_watchdog, executor_coordination, executor_recovery, executor_startup, executor_shutdown, and dbos_readiness. Recovery logs contain counts; watchdog logs contain an executor ID and a bounded reason, not workflow payloads. Terraform’s operations module defines a severity-1 alert for workflow_recovery_exhausted and workflow_recovery_settlement, evaluated every five minutes over ten minutes of Container Apps console logs and routed to the existing on-call action group. It takes effect when the foundation configuration is applied; local tests do not deploy it. Other watchdog/readiness signals still need operational alert policies. Log ingestion delays or quota limits can delay notifications; the terminal DBOS row remains the durable evidence.

Inspect leases in the owning database:

SELECT application_name, application_version, executor_id, state,
       heartbeat_at, expires_at, revoked_at
FROM workflow_coordination.executor
ORDER BY heartbeat_at DESC;

For pending workflows that remain stuck, compare their executor_id, application_name, and application_version with the registry. Missing registry rows and incompatible versions are deliberately not guessed or stolen. Investigate configuration and run the matching worker; do not blindly resume cancelled workflows or reset deadlines. A cross-store restore remains a separate recovery-epoch/quarantine procedure.

Run the dedicated Hermit suite against disposable PostgreSQL:

./bin/hermit test tests/suites/data-foundation/worker-coordination.yaml
./bin/hermit run tests/plans/platform/aix.yaml

The coordination suite executes real compiled DBOS worker subprocesses under the worker database role. It kills a worker, pauses one with SIGSTOP through the production lease window, resumes the stale process, verifies checkpoint replay and fencing, and tests transactional rollback, concurrent scanners, version isolation, and preserved terminal states/deadlines. A poison workflow kills four replacement processes, is quarantined, and leaves a survivor able to process healthy work. The suite also verifies direct/child recovery, metadata preservation, two-replica mass recovery under a global concurrency cap, mandatory probe cleanup, and delayed renewal timing. It stalls business dequeuing while the private probe and heartbeat remain healthy, checks capacity/rate/delay exclusions, separately stalls the private diagnostic queue, and runs a step that ignores cancellation. It belongs to both the AIX plan and the CI platform integration plan. All seven required top-level tests must execute without skipped fixtures; the suite has a 12-minute timeout.

Keep service-level Hermit coverage for authorization, provider uncertainty, cancellation, clarification, publication, documents, and corpus indexing. Coordination tests do not replace those domain contracts or prove production availability during a database outage.