Crash, Checkpoint, Resume: Durable Agent State in LangGraph, Agent Framework and ADK

Recovery outcomes with plain tools for LangGraph, Agent Framework and ADK at three crash points
On this page

An agent that calls tools is a long-running process with side effects, and long-running processes die. A container is rescheduled, a deployment rolls, a node runs out of memory. When that happens in the middle of a tool call, somebody has to decide what to do with the half-finished work, and the usual answer is to run it again. The question that matters is what gets done twice.

This article answers it by experiment for three agent frameworks as they stand this month: LangGraph 1.1.3, Microsoft Agent Framework 1.0.0rc5 and Google ADK 1.27.2. One agent places an order with three tools: reserve stock, charge a card, send a confirmation. The agent process is killed with SIGKILL at three points around the charge, recovered in the way each framework allows, and a ledger written by the tools is read to count how often each thing really happened.

With tools written the ordinary way, six of fifteen recoveries finished with every effect done once. Six did something twice, a second charge among them, and three did not finish at all. With the same tools made idempotent, twelve finished cleanly and none duplicated anything. No framework setting produced that improvement. The tools did.

The problem

Every durable system has the same gap. A tool does something in the outside world, and then the framework records that it was done. Those are two steps, and a crash can fall between them. On recovery the record says the tool has not run, so it runs again. Checkpointing makes the gap smaller. It does not close it.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','signalColor':'#0073BB','signalTextColor':'#0F172A','actorBkg':'#FFF3E0','actorBorder':'#FF9900','actorTextColor':'#232F3E','noteBkgColor':'#FFF8E7','noteBorderColor':'#FF9900','noteTextColor':'#232F3E'}}}%%
sequenceDiagram
    participant A as Agent process
    participant T as Payment service
    participant S as State store
    A->>S: Checkpoint: model asked for charge_card
    A->>T: Charge 37.50
    T-->>A: Captured
    Note over A: Process killed here
    Note over S: Store still says: charge_card pending
    A->>S: Restart: load checkpoint
    A->>T: Charge 37.50 again
    T-->>A: Captured
    A->>S: Checkpoint: charge_card done

The guarantee on offer is therefore at-least-once: a tool whose call was recorded will run, and may run more than once. Google’s ADK documentation says so in as many words, that tools “are run at least once, and may run more than once when resuming”. What differs between frameworks is how much is replayed, and whether recovery works at all.

Test setup

The model is a scripted stand-in on localhost, so that every run is identical. It keeps no memory between requests. It reads the conversation it is sent and asks for the first of the three tools that has no result in it, which is what a sensible model would do. After a crash, what it asks for depends only on what the framework chose to send, and that makes it a probe for the framework’s state:

Python13 lines
def read_chat(body: dict) -> tuple[set[str], list[str], int]:
    """Tools with a result, tool calls left unanswered, and how many user messages there are."""
    called: dict[str, str] = {}
    answered: set[str] = set()
    users = 0
    for message in body.get("messages", []):
        if message.get("role") == "user":
            users += 1
        for call in message.get("tool_calls") or []:
            called[call["id"]] = call["function"]["name"]
        if message.get("role") == "tool":
            answered.add(message.get("tool_call_id", ""))
    return {called[i] for i in answered if i in called}, [name for i, name in called.items() if i not in answered], users

Like the real Chat Completions API, the stand-in rejects a conversation in which a tool call has no tool message answering it. It also gives each tool call a fresh random id, as a real model does.

The tools write to an append-only ledger that stands in for the warehouse, the payment provider and the mail service. A line of kind effect is something that happened and cannot be taken back. There are two versions of each tool, selected by an environment variable, and a crash point that the lab sets:

Python26 lines
def act(tool: str, key: str, result: dict) -> dict:
    """Do the work of a tool exactly as the selected version would."""
    crash_point(tool, "before")
    if IDEMPOTENT:
        earlier = recorded(key)
        if earlier is not None:
            record("replayed", tool, key, earlier)  # asked again, nothing done again
            return earlier
    record("effect", tool, key, result)
    crash_point(tool, "after")  # the effect exists, and the framework has not been told
    return result


def reserve_stock(order_id: str, sku: str, quantity: int) -> dict:
    """Reserve stock for an order."""
    return act("reserve_stock", f"reserve:{order_id}:{sku}", {"order_id": order_id, "sku": sku, "reserved": quantity})


def charge_card(order_id: str, amount_cents: int) -> dict:
    """Charge the customer's card for an order."""
    return act("charge_card", f"charge:{order_id}", {"order_id": order_id, "charged_cents": amount_cents, "status": "captured"})


def send_confirmation(order_id: str, email: str) -> dict:
    """Send the order confirmation email."""
    return act("send_confirmation", f"confirm:{order_id}", {"order_id": order_id, "sent_to": email})
Crash pointWhere the process is killedWhat a perfect recovery would do
Before the chargeInside charge_card, before anything is recordedCharge once, then send the email
After the chargeInside charge_card, after the charge exists and before the tool returnsNot charge again, then send the email
Before the emailInside send_confirmation, after charge_card has returnedSend the email only

The kill is SIGKILL to the process group: no handlers run and nothing is flushed. Recovery happens in a new process with the same state directory. Two recovery actions are tried where the framework has both. Resume uses the framework’s own call for continuing an interrupted run. Resend sends the original request again on the same session, which is what a caller does when all it knows is that the request did not finish.

ℹ️
TEST SCOPE

The stand-in follows one plan and never deviates from it. A real model that sees an interrupted step may repeat it, skip it, or ask the user. The results show what each framework sends to the model and re-executes by itself after a crash, which is the part a framework controls.

Persistence in each framework

Each agent is given the most ordinary durable store its framework offers on local disk. Each framework is installed into its own virtual environment, so that one set of dependency pins cannot affect another.

LangGraph saves a checkpoint of the whole graph state to SQLite after each step, keyed by a thread id. Resuming is an invocation with no input on the same thread:

Python11 lines
async def main(action: str, state: str, url: str) -> None:
    from langchain.agents import create_agent
    from langchain_openai import ChatOpenAI
    from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver

    model = ChatOpenAI(model="stand-in", api_key="not-a-real-key", base_url=f"{url}/v1", max_retries=0)
    async with AsyncSqliteSaver.from_conn_string(os.path.join(state, "langgraph.sqlite")) as checkpointer:
        agent = create_agent(model, TOOLS, system_prompt=INSTRUCTIONS, checkpointer=checkpointer)
        given = None if action == "resume" else {"messages": [("user", TASK)]}
        result = await agent.ainvoke(given, THREAD)
        print(result["messages"][-1].content)

Agent Framework, in this release candidate, stores conversation history through a history provider that is called when a run ends. The provider below writes it to a file. There is no call that continues an interrupted run of a single agent, so the only recovery is to send the request again:

Python30 lines
class FileHistory(BaseHistoryProvider):
    """Conversation history in one JSON file per session, written atomically."""

    def __init__(self, directory: str) -> None:
        super().__init__("file-history")
        self.directory = directory

    def path(self, session_id: str | None) -> str:
        return os.path.join(self.directory, f"agent-framework-{session_id}.json")

    async def get_messages(self, session_id: str | None, *, state: dict[str, Any] | None = None, **kwargs: Any) -> list[Message]:
        if not os.path.exists(self.path(session_id)):
            return []
        with open(self.path(session_id), encoding="utf-8") as handle:
            return [Message.from_dict(item) for item in json.load(handle)]

    async def save_messages(self, session_id: str | None, messages: Sequence[Message], *, state: dict[str, Any] | None = None, **kwargs: Any) -> None:
        stored = [m.to_dict() for m in await self.get_messages(session_id)] + [m.to_dict() for m in messages]
        temporary = self.path(session_id) + ".tmp"
        with open(temporary, "w", encoding="utf-8") as handle:
            json.dump(stored, handle)
        os.replace(temporary, self.path(session_id))


async def main(action: str, state: str, url: str) -> None:
    client = OpenAIChatClient(model_id="stand-in", api_key="not-a-real-key", base_url=f"{url}/v1")
    agent = Agent(client, INSTRUCTIONS, name="orders", tools=TOOLS, context_providers=[FileHistory(state)])
    session = agent.create_session(session_id="order-O-1001")
    response = await agent.run(TASK, session=session)
    print(response.text)

ADK appends each event to a session service as it happens, here a SQLite one, and has a resume feature that is switched on per application. Resuming passes the id of the interrupted invocation and no new message. The library prints a warning that the feature is experimental:

Python25 lines
async def main(action: str, state: str, url: str) -> None:
    from google.adk.agents import LlmAgent
    from google.adk.apps import App, ResumabilityConfig
    from google.adk.models import Gemini
    from google.adk.runners import Runner
    from google.adk.sessions.sqlite_session_service import SqliteSessionService
    from google.genai import types

    os.environ.setdefault("GOOGLE_API_KEY", "not-a-real-key")
    agent = LlmAgent(name="orders", model=Gemini(model="gemini-stand-in", base_url=url), instruction=INSTRUCTIONS, tools=TOOLS)
    sessions = SqliteSessionService(os.path.join(state, "adk.sqlite"))
    runner = Runner(app=App(name=APP, root_agent=agent, resumability_config=ResumabilityConfig(is_resumable=True)), session_service=sessions)
    if action == "start":
        await sessions.create_session(app_name=APP, user_id=USER, session_id=SESSION)
    arguments: dict = {"user_id": USER, "session_id": SESSION}
    if action == "resume":
        stored = await sessions.get_session(app_name=APP, user_id=USER, session_id=SESSION)
        arguments["invocation_id"] = stored.events[-1].invocation_id
    else:
        arguments["new_message"] = types.Content(role="user", parts=[types.Part(text=TASK)])
    text = ""
    async for event in runner.run_async(**arguments):
        if event.is_final_response() and event.content and event.content.parts:
            text = "".join(part.text or "" for part in event.content.parts)
    print(text)

Test results

With the plain tools, counting effects in the ledger after recovery (reserve, charge, email):

FrameworkRecoveryKilled before the chargeKilled after the chargeKilled before the email
LangGraphResume1, 1, 11, 2, 1: charged twice1, 1, 1
LangGraphResendDid not finishDid not finishDid not finish
Agent FrameworkResend2, 1, 1: reserved twice2, 2, 1: reserved and charged twice2, 2, 1: reserved and charged twice
ADKResume1, 1, 11, 2, 1: charged twice1, 1, 1
ADKResend1, 1, 11, 2, 1: charged twice1, 1, 1
Recovery outcomes with plain tools for LangGraph, Agent Framework and ADK at three crash points: LangGraph and ADK resume repeat only the interrupted tool and charge twice when killed after the charge; Agent Framework replays the whole run and repeats every completed tool; LangGraph resend fails with HTTP 400
What each framework repeated after a kill, with tools that act every time they are called.

LangGraph and ADK resume the same way. Both had recorded that the model asked for charge_card, and both ran that call again themselves on recovery, without asking the model. The first request the stand-in received after the crash already contained a result for the charge. That is the best a framework can do: only the interrupted tool is repeated, and nothing earlier. It is also exactly the case where the charge had already been made, so the card was charged twice.

Agent Framework replays the run. Nothing had been stored when the process died, because history is saved at the end of a run. The recovered agent started from an empty conversation, the stand-in asked for all three tools again, and every tool that had completed before the crash was done a second time. A crash after the charge duplicated both the reservation and the charge. This is the behaviour of a single agent with a history provider in a release candidate; the framework’s workflows have their own checkpoint storage, which this lab did not test.

Resending is not a safe default. On LangGraph it left the thread unusable. The checkpoint held an assistant message with a tool call and no answer; adding a new user message after it produced a conversation the model API rejects with HTTP 400, on every attempt. The thread has to be resumed or repaired before it will accept input again. On ADK the resend went through, because the Gemini-style conversation it sent was accepted by the stand-in, which does not check that protocol for unanswered calls. Whether the hosted API would accept it was not tested.

Idempotent tools

The second version of each tool derives a key from the business identifiers in its arguments, looks the key up before acting, and returns the recorded result if the work was already done. It is the IDEMPOTENT branch of the function shown earlier, a handful of lines. The same fifteen crashes, repeated with those tools:

ToolsEvery effect exactly onceAn effect duplicatedDid not finish
Plain6 of 156 of 153 of 15
Idempotent12 of 150 of 153 of 15
Fifteen recoveries with plain tools and fifteen with idempotent tools: plain tools gave six clean recoveries, six with a duplicated effect and three unfinished; idempotent tools gave twelve clean, none duplicated and the same three unfinished
The same crashes with tools that check a business key before acting.

Every duplicate disappeared, in every framework, including the full replay in Agent Framework: the tools were called again and did nothing again. The three that still did not finish are the LangGraph resends, which fail before any tool is reached. Idempotent tools fix repeated effects. They do not fix a recovery procedure that cannot proceed.

The choice of key is the design decision. It is tempting to use the tool call id, since frameworks expose it and it is unique. It works when a framework re-executes a recorded call, as LangGraph and ADK do on resume, because the id is the same. It fails on a replay, because the model is asked again and issues a new id for the same intent. A key built from what the operation means, here the order number, is the same in both cases.

⚠️
WHERE THE KEY MUST BE CHECKED

In this lab the effect and its record are one append to one file, so the check is airtight. In a real system the effect is a call to another service, and a local “already done” table has the same gap as a checkpoint. The key has to travel with the request and be enforced by the service that performs the effect. Payment APIs support this directly: Stripe, for one, saves the result of the first request made with a given idempotency key and returns it for retries.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','clusterBkg':'#F8FAFC','clusterBorder':'#CBD5E1','edgeLabelBackground':'#FFFFFF','nodeTextColor':'#232F3E'}}}%%
flowchart TD
    C["Tool called with arguments"] --> K["Build key from business identifiers:
operation and order number"] K --> Q{"Has the downstream
service seen this key?"} Q -- "yes" --> R["Return the stored result,
do nothing"] Q -- "no" --> D["Perform the effect,
passing the key"] D --> ST["Service stores key and result
as one operation"] ST --> OUT["Return the result"] R --> OUT style K fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74 style R fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202 style ST fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95

Recovery procedure

The results suggest an order of preference for whatever supervises the agent. Use the framework’s resume call where one exists, because it repeats the least. Fall back to sending the request again only on a framework that has no resume, and only with idempotent tools. And never mix the two on one thread without checking its state first.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','clusterBkg':'#F8FAFC','clusterBorder':'#CBD5E1','edgeLabelBackground':'#FFFFFF','nodeTextColor':'#232F3E'}}}%%
flowchart TD
    X["Agent process died
during a run"] --> F{"Does the framework
have a resume call?"} F -- "yes" --> RS["Resume the same thread
or invocation"] RS --> ONE["Interrupted tool runs again,
earlier tools do not"] F -- "no" --> RE["Send the request again
on the same session"] RE --> ALL["Every tool runs again"] ONE --> I{"Are the tools
idempotent?"} ALL --> I I -- "yes" --> OK["Effects happen once"] I -- "no" --> BAD["Duplicated effects:
reconcile by hand"] style OK fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202 style BAD fill:#FEF2F2,stroke:#D13212,stroke-width:2px,color:#991B1B style RS fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74

The supervisor needs one more piece of information that none of these frameworks hands it: that a run was in flight when the process died. In the lab the test knows. In production that is a record written before the run starts and cleared when it ends, in the same store as the checkpoints, and scanned on start-up.

Cost

Durable state is not free, and this lab did not measure its price. Saving state as a run proceeds means a write to the store for every model call and every tool call, and the latency of that store is added to each step. At the scale of one order with three tools it is invisible. For long sessions the size of what is written and the time it takes should be measured against the store you intend to use, not assumed.

The cost of not having it is easier to state. Without mid-run persistence, recovery means running the whole task again: every model call is paid for twice, and every tool must tolerate being called twice. With idempotent tools that is slow and correct. Without them it is a refund.

Production considerations

  • Classify tools. Reads can repeat freely. Writes need a key. Sends, payments and anything a person will notice need a key that the downstream service enforces. Write the class next to the tool.
  • Derive keys from the request. Order number, invoice number, a hash of the recipient and template. Not the tool call id, not a timestamp, not a random value generated inside the tool.
  • Return the original result on a repeat. A repeated call must give the model the same answer as the first, not an error saying it was already done, or the model will try to fix a problem that does not exist.
  • Use a durable store. An in-memory checkpointer or session service survives nothing. LangGraph’s documentation says as much of its in-memory saver and points to PostgreSQL for production.
  • Test the kill. A recovery path that has never run will not work the first time. The lab’s method, a marker file and SIGKILL, takes a few seconds per case and fits in a pipeline.
  • Repair before resending. A thread left with an unanswered tool call must be resumed, or have the call answered with an explicit failure, before it is given new input.
  • Watch pre-release behaviour. Agent Framework is a release candidate and ADK flags its resume feature as experimental. Pin versions and rerun the kill tests on every upgrade.
⚖️
AUDIT AND REGULATED ACTIONS

Where a tool performs a regulated action, such as a payment, a prescription or a record update, a duplicate is an incident and not a retry. The ledger of effects, with keys, is the audit evidence that an action happened once. Keep it outside the agent framework, in the system of record that performs the action.

Decision framework

Your situationWhat to doWhy
Tools only readAny persistence, resend on failureNothing can be duplicated; only model calls are repeated
Tools write, framework has resumeDurable checkpoints, resume, idempotent writesResume repeated one tool in this run, and that one was the charge
Tools write, no resume for your agent typeIdempotent tools first, then resendA replay repeated every completed tool
Payments, messages to people, regulated actionsKey enforced by the downstream serviceA local check has the same gap as a checkpoint
Long sessions with many stepsMeasure checkpoint size and store latencyState is written on every step
A caller that retries on timeoutCheck for an in-flight run before sending againResending onto an interrupted thread failed outright on LangGraph

Key takeaways

  • Agent frameworks give at-least-once execution of tools. A kill between a tool’s effect and the framework’s record of it leads to the tool running again.
  • LangGraph and ADK resumed by re-executing only the interrupted tool call. Agent Framework 1.0.0rc5, as a single agent with a history provider, replayed the whole run and repeated every completed tool.
  • With plain tools, six of fifteen recoveries duplicated an effect, including a second card charge in every framework. With idempotent tools, none did.
  • Build the idempotency key from business identifiers, not from the tool call id, which changes when the model is asked again. Have the downstream service enforce it.
  • Sending the request again is not a universal recovery. On LangGraph it left the thread rejecting every request until it was resumed or repaired.

If you have run kill tests against an agent in production, or against a persistence back end not covered here, leave a comment with what repeated.

📝
HOW THIS WAS CHECKED

Everything above ran in GitHub Actions on a hosted Linux runner with Python 3.13.18, in three virtual environments: LangGraph 1.1.3 with langchain 1.2.13, langchain-openai 1.1.11 and langgraph-checkpoint-sqlite 3.0.3; Agent Framework 1.0.0rc5 (agent-framework-core); and Google ADK 1.27.2 with google-genai 1.68.0. Other dependencies were resolved as they stood on 23 March 2026. The lab ran three uninterrupted baselines and thirty crash cases: three crash points, the recovery actions each framework supports, and two tool versions. Each case had a fresh stand-in and a fresh state directory, the agent was killed with SIGKILL when a tool wrote its marker file, and effects were counted from the ledger. 10 unit tests cover the stand-in and the tools. The model was a scripted stand-in speaking Chat Completions to LangGraph and Agent Framework and the Gemini API to ADK; it rejects unanswered tool calls on the first protocol and does not check them on the second. Not run: any hosted model; other persistence back ends; Agent Framework workflows and their checkpoint storage; concurrent recoveries; a crash during a checkpoint write. Checkpoint size and latency were not measured. The run is here.

Glossary

  • Checkpoint— A saved copy of an agent’s state at a point in a run, from which the run can continue.
  • Thread— LangGraph’s name for one conversation’s sequence of checkpoints, identified by a thread id.
  • Session— The stored history of one conversation in Agent Framework and ADK.
  • Invocation— In ADK, one run of an agent in response to a message. An interrupted invocation can be resumed by its id.
  • At-least-once— A guarantee that an operation will happen, with the possibility that it happens more than once.
  • Idempotent— An operation that has the same effect whether it is performed once or several times.
  • Idempotency key— A value sent with a request so that the receiver can recognise a repeat and return the first result.
  • Replay— Recovering by running the whole task again from the start.
  • Resume— Recovering by continuing from saved state, repeating only what was in progress.
  • SIGKILL— The signal that ends a process immediately, with no chance to clean up.

References & Further Reading


Discover more from C4: Container, Code, Cloud & Context

Subscribe to get the latest posts sent to your email.

Written by

Nithin Mohan TK

Hello Namaste! 👋🏻 I'm an accomplished Enterprise Solution Architect & Engineering Leader with 20+ years of progressive experience architecting, building, and scaling transformative enterprise solutions. ⚜️ Solution Architect / AI & Cloud 📌 Galway, Ireland

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.