Resilient: Every step is saved
Each step's result is saved before the next one starts. A deploy, a crash or a restart can't lose work that's already done.
step :charge, fn ctx -> ... endDurable is an open-source workflow engine for Elixir. Write workflows as plain steps. Every step is saved, so runs resume after crashes, deploys and week-long waits.
Durable execution for AI agents in Elixir. Every LLM and tool call is saved, so crashes, timeouts and deploys never lose progress or re-bill completed calls.
Build an AI workflow, step by step, in the guide.
planwrite_patch run_tests await_review mergeworkflow "fix_issue", timeout: days(14) do
step :plan, [retry: [max_attempts: 3, backoff: :exponential]], fn ctx ->
plan = ReqLLM.generate_text!(@model, "Plan a fix for: #{ctx["issue"]}")
{:ok, assign(ctx, :plan, plan)}
end
step :write_patch, [retry: [max_attempts: 3]], fn ctx ->
patch = ReqLLM.generate_text!(@model, ctx.plan)
{:ok, assign(ctx, :patch, patch)}
end
step :run_tests, [retry: [max_attempts: 5]], fn ctx ->
{:ok, run} = CI.run(ctx["repo"], ctx.patch)
{:ok, assign(ctx, :ci_url, run.url)}
end
step :await_review, fn ctx ->
decision = wait_for_approval("maintainer", timeout: days(7))
{:ok, assign(ctx, :decision, decision)}
end
step :merge, fn ctx ->
GitHub.merge(ctx["repo"], ctx.patch)
{:ok, ctx}
end
endEach step's result is saved before the next one starts. A deploy, a crash or a restart can't lose work that's already done.
step :charge, fn ctx -> ... endSteps retry with exponential, linear or constant backoff. Workflows have timeouts. When a step fails for good, compensations undo what already happened, in reverse.
retry: [max_attempts: 3, backoff: :exponential]
compensate: :cancel_hotelSleep for days, wait for a webhook or a person's approval. A waiting run holds no process, and when it wakes up it carries on with everything it already knew.
sleep(days(3))All of it is plain Elixir in one module, and every step it runs is saved, retried and visible in the dashboard.
parallel do ... endretry: [max_attempts: 3]branch on: fn ctx -> ... endwait_for_event("paid")wait_for_approval("lead")step :book, [compensate: :cancel]sleep(days(3))@schedule cron: "0 9 * * *"call_workflow(Child, input)timeout: hours(2)step :ocr, [queue: :ocr]{:snooze, seconds(15)}{:fail, "the file is gone"}Durable.start_many(items)Durable.start(Article, input, key: "article:42")durable_dashboard is a Phoenix LiveView dashboard: live runs, step traces, an inbox for pending approvals, queues and schedules. Retry, cancel or pause right from the browser.
| Status | Workflow | Run | Took | Started |
|---|---|---|---|---|
| Running | issue_agent | a91c3e | 12s | now |
| Failed | process_order | 8f3c2a | 1.4s | 2m ago |
| Waiting | expense_approval | 4b07d1 | 2d 3h | 2d ago |
| Completed | process_order | 77e0b9 | 2.1s | 5m ago |
| Compensated | book_trip | c2d4a0 | 3.8s | 9m ago |
| Completed | daily_sales_report | 1fa9e2 | 41s | 9:00 |
process_order run 8f3c2a Failedvalidate12mscharge #11.1scharge #20.9scharge #31.0ssend_confirmationnot run(MatchError) no match of right hand side value: {:error, :card_declined}(my_app 0.1.0) lib/my_app/orders.ex:28: MyApp.OrderWorkflow.charge/1expense_approvalApprove the $420 expense from Dana?
waiting for manager, 2hissue_agentReview the patch for acme/shop#1042
waiting for maintainer, 1drefund_requestRefund or store credit?
waiting for support, 3ddefault7 of 10 busyPausepriority2 of 5 busyPauseemailsPausedResumeMount it in your Phoenix router with one line. Dashboard docs
use DurableDashboard.Router, mount: "/dashboard", durable: MyApp.DurableKnow how your workflows behave in production with the tools you already use.
Runs, steps, compensations, queues and schedules emit :telemetry spans shaped like Phoenix, Ecto and Oban, so your existing tooling reads them as-is.
Durable.Telemetry.metrics/0 returns Telemetry.Metrics definitions for Prometheus, StatsD or LiveDashboard.
Durable.Stats gives you overviews, timelines, step performance, queue history and grouped errors.
Recovered runs, fenced workers and queue pauses are recorded, so you can see what the system did on its own.
# lib/my_app_web/telemetry.ex
def metrics, do: my_metrics() ++ Durable.Telemetry.metrics()[:durable, :workflow, :stop][:durable, :step, :retry][:durable, :queue, :job_claimed][:durable, :schedule, :trigger]Retry a failed run from the step that failed, rerun a finished one, cancel, pause a queue on every node, or prune old runs.
Durable.start(Module, input)
Durable.start(Module, input, queue: :priority, scheduled_at: datetime)
Durable.start(Module, input, key: "article:42") # at most one run per key
Durable.start(Module, input, tags: ["account:7"]) # labels, to find runs across workflows
Durable.start_many([{Module, input, key: "row:1"}, {Module, input2}]) # one transaction, keys checked
Durable.get_execution(id)
Durable.list_executions(workflow: Module, status: :running)
Durable.list_executions(tag: "account:7")
Durable.cancel(id, "reason")
Durable.cancel_all([tag: "account:7"], "account closed") # every open run that matches
Durable.send_event(id, "event", payload)
Durable.send_event(id, "event", payload, buffer: true) # keep it until the run waits
Durable.provide_input(id, "input_name", data)
Durable.list_children(parent_id)
# Operating
Durable.retry(id) # failed run: resume from the failed step
Durable.rerun(id, input: new_input) # finished run: start a fresh copy
Durable.release_key(id, by: user) # finished run: let its key start again
Durable.pause_queue(:default, by: user) # on every node, until resume_queue/2
Durable.drain() # this node: stop claiming, finish running steps
Durable.list_nodes() # heartbeating instances and what they run
Durable.prune() # apply the :prune retention now
# Observing
Durable.Stats.overview(since: dt) # also timeline/1, workflows/1, step_performance/1,
# queues/1, queue_history/1, error_groups/1
Durable.Activity.list() # recovered runs, fenced workers, queue pauses…# Show queue status and workflow summary
mix durable.status
# List workflow executions (with filters)
mix durable.list # all executions
mix durable.list --status running # filter by status
mix durable.list --workflow MyApp.OrderWorkflow # filter by workflow
mix durable.list --limit 20 --format json # limit results, JSON output
# Start a workflow
mix durable.run MyApp.OrderWorkflow # no input
mix durable.run MyApp.OrderWorkflow --input '{"id": 123}' # with JSON input
mix durable.run MyApp.OrderWorkflow --queue high_priority # specific queue
# Cancel a workflow
mix durable.cancel <execution_id>
mix durable.cancel <execution_id> --reason "no longer needed"
# Clean up old executions
mix durable.cleanup --older-than 30d # completed/failed older than 30 days
mix durable.cleanup --older-than 7d --status completed # only completed, older than 7 days
mix durable.cleanup --older-than 24h --dry-run # preview what would be deletedDurable is a library. Workflows are plain Elixir modules: no YAML, no new language. It starts in your supervision tree on every node and uses your Ecto repo. Runs are stored through a pluggable queue adapter, PostgreSQL by default.
FOR UPDATE SKIP LOCKEDdef deps do
[{:durable, "~> 1.5"}]
enddefmodule MyApp.Repo.Migrations.AddDurable do
use Ecto.Migration
def up, do: Durable.Migration.up()
def down, do: Durable.Migration.down()
endchildren = [
MyApp.Repo,
{Durable, repo: MyApp.Repo, queues: %{default: [concurrency: 10]}}
]Durable is an open-source workflow engine for Elixir. You write a workflow as a series of steps; Durable saves each step's result in your database, retries failures, and resumes runs after crashes, deploys, sleeps and waits for events or human input.
Oban runs background jobs. Durable runs multi-step workflows: context flows from step to step, every step is persisted, and a workflow can sleep, wait for events or approvals, branch, run steps in parallel, call child workflows and roll back with compensations. Durable has its own queue, so it does not need Oban.
It follows the same durable execution idea, but it's a library: it runs inside your Elixir application, under your supervision tree, and uses your Ecto repo.
An Elixir app and an Ecto repo. Out of the box, runs are stored in PostgreSQL 13 or newer and workers claim them with FOR UPDATE SKIP LOCKED. The queue is a pluggable adapter, so other backends can be added.
Yes. Durable works on direct connections and behind PgBouncer in transaction mode, because it uses no session features such as LISTEN/NOTIFY. A PgBouncer older than 1.24 needs prepare: :unnamed on your repo.
Workers send heartbeats while a step runs. If a node dies, its lock goes stale and another node recovers the run, re-running only the step that was interrupted. Steps that finished don't run again, so keep each step idempotent.
Yes. Put each LLM or tool call in a step: its result is saved, failed calls retry with backoff, and an agent can wait days for a human approval without holding a process. After a crash or deploy it resumes at the next step instead of starting over.
Yes. durable_dashboard is a Phoenix LiveView dashboard you mount in your router with one line. It shows runs, step timelines, pending inputs, queues and schedules.