Here is a scenario that plays out in almost every organization that moves agents from demo to production.

An agent workflow starts. It is a complex task — research, synthesis, multiple tool calls, several LLM reasoning steps. It runs for twelve minutes. At minute nine, a Kubernetes pod is evicted. The worker dies. The agent loses everything — all the work done, all the state accumulated, all the tool call results retrieved.

The workflow restarts from the beginning. It runs the same tool calls again. It makes the same LLM calls again. It burns the same tokens again. If the eviction happens a second time, it starts over a third time. The task never completes.

This is not a hypothetical. It is the default behavior of every agent system built on a stateless execution model. And the astonishing thing is how many production agent deployments are built exactly this way — trusting that the worker will stay alive for the full duration of a task that might run for hours, with no recovery path if it doesn't.

Stateless execution is fine for workloads that run in milliseconds. For agent workflows that run for minutes, hours, or days — across tool calls that have real-world side effects, LLM inference that costs real money, and state that represents real work — stateless execution is a liability. A durable execution model is not a performance optimization. It is the foundational requirement for running agents reliably at any meaningful scale.

What Durable Execution Actually Means

Durable execution is a deceptively simple concept. Every meaningful step in a workflow is recorded to a persistent store before the workflow proceeds. If the worker executing the workflow dies, a new worker reads the record, replays to the point of failure, and continues. No work is repeated. No state is lost. No task restarts from the beginning.

The key word is persistent. Not in-memory. Not cached. Written to a durable store before the step is considered complete — the same write-ahead-log principle that makes databases reliable applied to the workflow execution layer.

The other key word is replay. When a new worker picks up a failed workflow, it doesn't re-execute the steps that already completed. It replays the recorded outputs of those steps — feeding them back into the workflow as if they just happened — and then resumes executing from where the failure occurred. The worker that died is irrelevant. The work it did lives in the durable store, permanent and accessible to any worker that picks up the task.

This is what "your agent survives a server crash" actually looks like under the hood. Not resilient code. Not careful exception handling. Durable state and deterministic replay.

The Stateful Case for Long-Running Agents

Short-lived agents — complete a task in one or two LLM calls, return a result, exit — can get away with stateless execution. The cost of a restart is low. The probability of a failure in a two-second window is low. Stateless is fine.

Long-running agents are a fundamentally different category. An agent doing overnight financial research. A continuous monitoring agent running for days. An autonomous workflow executing a multi-stage business process with human approval gates, external API dependencies, and sequential tool calls where each step's output feeds the next. These agents cannot afford to restart from scratch on failure.

The economics alone make this clear. A research agent that makes forty LLM calls over twenty minutes of wall-clock time has accumulated real cost — tokens, API calls, retrieved documents. If it fails at step thirty-eight and restarts from step one, you've just doubled the cost of that task and guaranteed a longer wait for the result. At scale — hundreds of these workflows running in parallel — the cost of failed restarts compounds fast.

But cost is the benign version of the problem. The more serious version is correctness. Many agent tool calls have real-world side effects: emails sent, database records created, external APIs updated, files written. In a stateless system that restarts from the beginning, these side effects happen again. The email gets sent twice. The database record gets created twice. The external API receives duplicate writes. Without durable state and idempotency enforcement, the agent that fails and restarts doesn't just waste compute — it causes data corruption.

Durable execution solves both. Once a step is recorded as complete, it is complete. Replay feeds its output back into the workflow without re-executing it. The side effects of completed steps are not repeated. The cost of completed steps is not re-incurred. The workflow resumes from exactly where it stopped.

Graceful Degradation: What Failure Actually Looks Like in a Durable System

In a stateless agent system, a worker failure is a workflow failure. The task is lost and must be restarted.

In a durable execution system, a worker failure is a routing event. The durable engine detects the failure — via heartbeat timeout, via explicit signal, via infrastructure health check — and reassigns the workflow to a healthy worker. The healthy worker replays to the last checkpoint and continues. From the perspective of the task, nothing happened. From the perspective of the user, the task is running. From the perspective of the infrastructure, a failure occurred and was handled transparently.

This is graceful degradation in its truest form. Not degrading to a degraded state — continuing at full capability despite a component failure. The system's reliability guarantee is not "the workflow will complete if the worker stays healthy." It is "the workflow will complete, period, regardless of what happens to any individual worker."

This changes the infrastructure conversation entirely. In a stateless world, worker reliability is a first-class concern — you invest heavily in keeping workers healthy because a failed worker means a failed task. In a durable world, worker reliability is a secondary concern. Workers are cattle, not pets. They are interchangeable compute units that pull work from a queue, execute it, and can be killed and replaced without consequence. The durable engine is the reliability layer. The workers are just execution capacity.

This also changes the deployment story. Rolling updates, blue-green deployments, spot instance preemptions, node drains — all of these become routine operations rather than carefully choreographed events that require ensuring no in-flight tasks are on the affected workers. Tasks simply migrate to healthy workers when their current worker is removed. Deployments become operationally simple in a way they can never be with stateless execution.

Horizontal Scaling: Why Stateless Workers Are the Key

The scaling story for durable execution systems is unusually clean, because the architecture has a fundamental property that most distributed systems have to engineer their way toward: workers are inherently stateless.

In a typical distributed system, the challenge of horizontal scaling is state. Adding a new server is easy. Ensuring that new server has the right state, or can route users to the right existing server that holds their state, or can share state with other servers — this is where the complexity lives. Session affinity, sticky routing, distributed caches, state migration — these are all engineering investments required because the compute and the state are coupled.

In a durable execution system, they are not coupled. The workflow's state lives in the durable store, not in the worker. Any worker can execute any workflow at any time by reading the durable store. Adding a new worker is literally just adding compute capacity — no state to migrate, no routing to configure, no warm-up required. The new worker pulls tasks from the task queue and starts executing.

This means horizontal scaling is operationally trivial. Queue depth rising? Add workers. Queue depth falling? Remove workers. The scale-up and scale-down events have no effect on in-flight workflows. Tasks in progress continue on their current workers. New tasks are distributed across the expanded pool. The system self-balances automatically.

For agent workloads, which are characteristically spiky — burst traffic from external events, overnight batch jobs, sudden spikes from product features driving user engagement — this elasticity is critical. A system that can scale from ten workers to one thousand workers in minutes, without any state coordination overhead, is a fundamentally different cost and reliability profile than one that requires careful capacity planning and warm-up time.

The Temporal architecture makes this concrete. The Temporal server is the durable execution engine — it holds the event histories, manages the task queues, tracks workflow state, and dispatches work. Workers are stateless processes that connect to the Temporal server, pull tasks, execute them, and return results. You can run ten workers or ten thousand. You can kill any worker at any time. You can deploy new workers without telling the server anything. The system adapts continuously, automatically, and without coordination overhead.

Auto Compaction: The Long-Running Session Problem

Every durable execution engine stores the history of every workflow — every step executed, every tool call made, every LLM response received, every signal processed. This history is the source of truth. It enables replay, debugging, compliance reporting, and the deterministic reconstruction of any workflow's state at any point in time.

For short workflows, this history is lightweight. A workflow that runs twenty steps produces twenty history events. Trivial to store, trivial to replay.

For long-running agent workflows, the history grows. An agent session that runs for eight hours, executing a step every thirty seconds, produces nearly a thousand history events. An agent that runs continuously for days produces tens of thousands. Storing this history is cheap per event but expensive in aggregate at scale. Replaying from the beginning of a long history is slow — the longer the history, the longer the replay time on failure recovery.

Auto compaction solves this. At defined thresholds — event count, storage size, age — the compaction process takes a snapshot of the workflow's current state and truncates the event log to that snapshot point. The snapshot is semantically equivalent to the full history — it represents everything the workflow knows and everything it has done — but it is a fraction of the size. Old events are archived rather than deleted, but the active execution path replays from the snapshot rather than from the beginning of time.

The practical effect is dramatic. A workflow that has been running for twenty-four hours with five thousand history events, after compaction, replays from a snapshot that captures its current state in a few hundred records. Recovery time on failure drops from minutes to seconds. Storage cost scales sublinearly with session length rather than linearly. The performance characteristics of a twenty-four-hour session look like the performance characteristics of a fresh session.

For the specific workload of long-running agentic sessions — research agents that run overnight, monitoring agents that run indefinitely, multi-day autonomous workflows — compaction is not optional. Without it, the operational cost and recovery time of long sessions grows unboundedly. With it, long sessions are first-class, well-performing citizens of the production system.

Triggers and Event-Based Invocation

Durable execution systems treat workflow activation as a first-class concern — not an afterthought bolted onto the execution layer.

In Temporal's model, a workflow can be started by any of the following:

Scheduled triggers using cron expressions natively understood by the engine. The scheduling is durable — if the scheduled time passes while the engine is handling a peak load event, the workflow is queued and executed when capacity is available rather than silently dropped. Missed executions are tracked and handled according to configurable overlap policies.

External signals sent to running workflows mid-execution. A workflow waiting for human approval doesn't poll — it sleeps durably until a signal arrives. The signal can carry a payload — the approval decision, the approver's identity, any additional context — which the workflow incorporates into its subsequent execution. The workflow resumes immediately on signal receipt, regardless of how long it waited.

Parent workflow triggers where one durable workflow spawns child workflows. Child workflows are independent execution units — they can run on different workers, at different times, and can outlive their parent. The parent can wait for child completion, fire and forget, or receive child signals. Multi-agent systems map naturally to this model: the orchestrator workflow spawns specialist agent workflows, coordinates their execution, and synthesizes their results.

API-driven starts from any external system that can make an HTTP call. Webhooks, event bus integrations, human-initiated tasks — all of these result in a durable workflow record being created before the first line of application code runs. If the workflow starts but the initial worker dies in the first second, it simply replays from the beginning on a new worker. Nothing is lost.

The unifying property across all of these trigger types: the trigger creates a durable record. The execution is decoupled from the triggering event. The reliability guarantee doesn't start after the workflow gets running — it starts at the moment the trigger fires.

Cost Efficiency: The Economics of Durability

Durable execution is sometimes framed as adding complexity and overhead. The reality — especially for agent workloads — is the opposite. Durability is a cost optimization.

The cost model for stateless agents at failure: full task re-execution. Every token, every tool call, every API request — repeated from the beginning. At low failure rates, this is tolerable. As task complexity and duration grow, failure becomes more likely (more things that can go wrong over a longer window) and more expensive (more work to repeat). The expected cost of a stateless task includes the expected cost of all its restarts — which grows with both task length and system failure rate.

The cost model for durable agents at failure: resume from last checkpoint. Only the work after the checkpoint is repeated. For a workflow with frequent checkpointing, the cost of a failure is bounded by the work done since the last checkpoint — minutes rather than hours of re-execution. The expected cost of a durable task is close to the cost of a successful execution, regardless of how many times it fails and recovers.

At scale, this difference is significant. A system running a thousand concurrent long-running agent workflows on spot instances — cheap compute that can be preempted at any time — is economically viable with durable execution and operationally untenable without it. Spot preemptions become routine handled events rather than task-killing failures. The cost savings from spot pricing far exceed the overhead of the durable execution engine. The economics favor durability decisively.

There is also the cost of correctness. A stateless system that restarts workflows from the beginning, re-executing tool calls that already succeeded, may duplicate real-world side effects. Cleaning up those duplicates — the extra emails sent, the duplicate database records created, the redundant external API calls — is operational overhead that doesn't appear in the compute cost model but is very real. Durable execution prevents these duplicates by design, eliminating a category of operational cost that stateless systems must address through other means.

Why Temporal Is the Right Reference Point

Temporal is not the only durable execution engine, but it is the most mature, most production-proven, and most directly applicable to agent workloads.

It was originally built at Uber to orchestrate microservices — specifically to solve the problem of long-running workflows that needed to survive infrastructure failures at massive scale. The core insight from that work is the same insight that applies directly to agents: make the execution layer durable, make workers stateless, and the reliability of individual components becomes irrelevant to the reliability of the system.

Temporal's model maps cleanly to agent workloads. Workflows are agent sessions. Activities are individual steps — tool calls, LLM inferences, state reads and writes. Workers are the compute processes that execute agent logic. Signals are human approvals, external events, and mid-session updates. Timers are the "sleep until" constructs that let agents wait for conditions without holding compute resources. Namespaces are the tenant isolation boundaries that let different teams or different customers operate independently.

The primitives are right. The operational characteristics — horizontal scalability, exactly-once activity execution, configurable retry policies, durable timers, signal-based mid-execution updates — are exactly what production agent workloads require.

Teams building serious agentic infrastructure should evaluate Temporal before building custom orchestration. The custom orchestration path leads, reliably, to a partial reimplementation of what Temporal already provides — but with less test coverage, less operational tooling, and a smaller community of practitioners who've debugged it in production.

What Production Readiness Looks Like

A durable execution environment for agents has the following properties — all of them, not some of them:

Every workflow step writes to the durable store before proceeding. No step is considered complete until its output is persisted. Workers can die at any moment without data loss.

Workers are stateless. Any worker can execute any workflow. Scaling is horizontal and does not require state coordination.

Failure recovery is automatic. The engine detects worker failure, reassigns the workflow, and resumes from the last checkpoint without human intervention.

Long-running timers are durable. An agent that needs to wait twelve hours for an external condition does not hold a thread or a connection during that wait. The timer persists across worker restarts and infrastructure events.

Signals enable external interaction with running workflows. Human approvals, external events, and mid-session updates are delivered reliably to running workflows without polling.

Compaction prevents unbounded history growth. Long-running sessions maintain consistent performance characteristics regardless of session age.

Cost is attributable per workflow execution. Every token, every tool call, every API request is recorded in the workflow history. Cost attribution is exact and queryable.

The full execution history of any workflow is queryable and replayable. Incident debugging is possible without log spelunking. Compliance reporting is exact. Any workflow's state at any point in time can be reconstructed deterministically.

⚡ The practitioner take

The hardest conversation to have with teams shipping their first production agent system is about infrastructure they don't think they need yet.

They have a working prototype. It runs reliably in demos. The tasks are short enough that restarts are rare and cheap. Why invest in a durable execution engine before you have to?

The answer is that by the time you have to, you already have a production system that has trained users to expect a certain reliability and latency — and rebuilding its execution layer without downtime, data loss, or behavioral regression is one of the hardest migrations in distributed systems. The architectural debt of a stateless execution layer compounds with every month of production usage.

The right time to build on a durable execution foundation is before the first production deployment, not after the first production incident. The overhead of integrating Temporal or a comparable engine into a greenfield agent system is measured in days. The overhead of migrating a production system to durability after the fact is measured in months — and the migration window doesn't cleanly exist, because production is always running.

Build durable from the start. The reliability property it gives you — every workflow completes, regardless of what happens to the infrastructure executing it — is not a property you can easily add later. It is the foundation that everything else is built on.

— Santosh

👀 Also Watching