Distributed Workflow Engine
API + workers + queue for durable async jobs with retries, idempotency, and dead-letter handling.
Problem
Synchronous APIs buckle under long-running or unreliable third-party work. Teams need visible job state and safe retries.
Overview
A reference design for a production-style job/workflow engine. Clients enqueue work; workers process asynchronously with retries and a dead-letter path.
Solution
Separate the request path from execution. Persist job state in PostgreSQL, coordinate work via a queue, and expose status for clients and operators.
Architecture
API service accepts jobs → persists row → enqueues message → worker claims with lease → executes → updates status or schedules retry → poison messages go to DLQ.
- API
- PostgreSQL
- Queue
- Workers
- DLQ
- Metrics
Implementation
- Job table with status, attempts, payload, and lease timestamps.
- Idempotency keys on enqueue and handler side-effects.
- Exponential backoff with jitter.
- Admin/status endpoints for operators.
Data model
jobs(id, type, status, attempts, idempotency_key, payload jsonb, available_at, leased_until, last_error, created_at, updated_at). Unique idempotency_key where set.
API design
POST /jobs, GET /jobs/:id, POST /jobs/:id/cancel. Internal worker heartbeat/lease renewal.
Tradeoffs
- Redis queue speed vs Postgres SKIP LOCKED simplicity.
- At-least-once delivery requires idempotent handlers.
Challenges
- Avoiding double-processing under retries.
- Fairness when one job type dominates the queue.
- Clear operator UX for DLQ replay.
Reliability
- Lease timeouts reclaim stuck work.
- DLQ isolates poison payloads.
- Structured logs per attempt.
Security
- Auth on public enqueue APIs.
- Payload validation before persistence.
Performance
Benchmarks belong in a controlled load test once implemented — not invented here.
What I learned
- Idempotency is a product requirement, not an afterthought.
- Status visibility prevents “is it stuck?” support noise.
Highlights
- Idempotent handlers
- Dead-letter queue
- Job status API
- Horizontal workers