On this page
An agent that calls tools is a long-running process with side effects, and long-running processes die. A container is rescheduled, a deployment rolls, a node runs out of memory. When that happens in the middle of a tool call, somebody has to decide what to do with the half-finished work, and the usual answer is to run it again. The question that matters is what gets done twice.
This article answers it by experiment for three agent frameworks as they stand this month: LangGraph 1.1.3, Microsoft Agent Framework 1.0.0rc5 and Google ADK 1.27.2. One agent places an order with three tools: reserve stock, charge a card, send a confirmation. The agent process is killed with SIGKILL at three points around the charge, recovered in the way each framework allows, and a ledger written by the tools is read to count how often each thing really happened.
With tools written the ordinary way, six of fifteen recoveries finished with every effect done once. Six did something twice, a second charge among them, and three did not finish at all. With the same tools made idempotent, twelve finished cleanly and none duplicated anything. No framework setting produced that improvement. The tools did.
The problem
Every durable system has the same gap. A tool does something in the outside world, and then the framework records that it was done. Those are two steps, and a crash can fall between them. On recovery the record says the tool has not run, so it runs again. Checkpointing makes the gap smaller. It does not close it.
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','signalColor':'#0073BB','signalTextColor':'#0F172A','actorBkg':'#FFF3E0','actorBorder':'#FF9900','actorTextColor':'#232F3E','noteBkgColor':'#FFF8E7','noteBorderColor':'#FF9900','noteTextColor':'#232F3E'}}}%%
sequenceDiagram
participant A as Agent process
participant T as Payment service
participant S as State store
A->>S: Checkpoint: model asked for charge_card
A->>T: Charge 37.50
T-->>A: Captured
Note over A: Process killed here
Note over S: Store still says: charge_card pending
A->>S: Restart: load checkpoint
A->>T: Charge 37.50 again
T-->>A: Captured
A->>S: Checkpoint: charge_card doneThe guarantee on offer is therefore at-least-once: a tool whose call was recorded will run, and may run more than once. Google’s ADK documentation says so in as many words, that tools “are run at least once, and may run more than once when resuming”. What differs between frameworks is how much is replayed, and whether recovery works at all.
Test setup
The model is a scripted stand-in on localhost, so that every run is identical. It keeps no memory between requests. It reads the conversation it is sent and asks for the first of the three tools that has no result in it, which is what a sensible model would do. After a crash, what it asks for depends only on what the framework chose to send, and that makes it a probe for the framework’s state:
def read_chat(body: dict) -> tuple[set[str], list[str], int]:
"""Tools with a result, tool calls left unanswered, and how many user messages there are."""
called: dict[str, str] = {}
answered: set[str] = set()
users = 0
for message in body.get("messages", []):
if message.get("role") == "user":
users += 1
for call in message.get("tool_calls") or []:
called[call["id"]] = call["function"]["name"]
if message.get("role") == "tool":
answered.add(message.get("tool_call_id", ""))
return {called[i] for i in answered if i in called}, [name for i, name in called.items() if i not in answered], usersLike the real Chat Completions API, the stand-in rejects a conversation in which a tool call has no tool message answering it. It also gives each tool call a fresh random id, as a real model does.
The tools write to an append-only ledger that stands in for the warehouse, the payment provider and the mail service. A line of kind effect is something that happened and cannot be taken back. There are two versions of each tool, selected by an environment variable, and a crash point that the lab sets:
def act(tool: str, key: str, result: dict) -> dict:
"""Do the work of a tool exactly as the selected version would."""
crash_point(tool, "before")
if IDEMPOTENT:
earlier = recorded(key)
if earlier is not None:
record("replayed", tool, key, earlier) # asked again, nothing done again
return earlier
record("effect", tool, key, result)
crash_point(tool, "after") # the effect exists, and the framework has not been told
return result
def reserve_stock(order_id: str, sku: str, quantity: int) -> dict:
"""Reserve stock for an order."""
return act("reserve_stock", f"reserve:{order_id}:{sku}", {"order_id": order_id, "sku": sku, "reserved": quantity})
def charge_card(order_id: str, amount_cents: int) -> dict:
"""Charge the customer's card for an order."""
return act("charge_card", f"charge:{order_id}", {"order_id": order_id, "charged_cents": amount_cents, "status": "captured"})
def send_confirmation(order_id: str, email: str) -> dict:
"""Send the order confirmation email."""
return act("send_confirmation", f"confirm:{order_id}", {"order_id": order_id, "sent_to": email})| Crash point | Where the process is killed | What a perfect recovery would do |
|---|---|---|
| Before the charge | Inside charge_card, before anything is recorded | Charge once, then send the email |
| After the charge | Inside charge_card, after the charge exists and before the tool returns | Not charge again, then send the email |
| Before the email | Inside send_confirmation, after charge_card has returned | Send the email only |
The kill is SIGKILL to the process group: no handlers run and nothing is flushed. Recovery happens in a new process with the same state directory. Two recovery actions are tried where the framework has both. Resume uses the framework’s own call for continuing an interrupted run. Resend sends the original request again on the same session, which is what a caller does when all it knows is that the request did not finish.
The stand-in follows one plan and never deviates from it. A real model that sees an interrupted step may repeat it, skip it, or ask the user. The results show what each framework sends to the model and re-executes by itself after a crash, which is the part a framework controls.
Persistence in each framework
Each agent is given the most ordinary durable store its framework offers on local disk. Each framework is installed into its own virtual environment, so that one set of dependency pins cannot affect another.
LangGraph saves a checkpoint of the whole graph state to SQLite after each step, keyed by a thread id. Resuming is an invocation with no input on the same thread:
async def main(action: str, state: str, url: str) -> None:
from langchain.agents import create_agent
from langchain_openai import ChatOpenAI
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
model = ChatOpenAI(model="stand-in", api_key="not-a-real-key", base_url=f"{url}/v1", max_retries=0)
async with AsyncSqliteSaver.from_conn_string(os.path.join(state, "langgraph.sqlite")) as checkpointer:
agent = create_agent(model, TOOLS, system_prompt=INSTRUCTIONS, checkpointer=checkpointer)
given = None if action == "resume" else {"messages": [("user", TASK)]}
result = await agent.ainvoke(given, THREAD)
print(result["messages"][-1].content)Agent Framework, in this release candidate, stores conversation history through a history provider that is called when a run ends. The provider below writes it to a file. There is no call that continues an interrupted run of a single agent, so the only recovery is to send the request again:
class FileHistory(BaseHistoryProvider):
"""Conversation history in one JSON file per session, written atomically."""
def __init__(self, directory: str) -> None:
super().__init__("file-history")
self.directory = directory
def path(self, session_id: str | None) -> str:
return os.path.join(self.directory, f"agent-framework-{session_id}.json")
async def get_messages(self, session_id: str | None, *, state: dict[str, Any] | None = None, **kwargs: Any) -> list[Message]:
if not os.path.exists(self.path(session_id)):
return []
with open(self.path(session_id), encoding="utf-8") as handle:
return [Message.from_dict(item) for item in json.load(handle)]
async def save_messages(self, session_id: str | None, messages: Sequence[Message], *, state: dict[str, Any] | None = None, **kwargs: Any) -> None:
stored = [m.to_dict() for m in await self.get_messages(session_id)] + [m.to_dict() for m in messages]
temporary = self.path(session_id) + ".tmp"
with open(temporary, "w", encoding="utf-8") as handle:
json.dump(stored, handle)
os.replace(temporary, self.path(session_id))
async def main(action: str, state: str, url: str) -> None:
client = OpenAIChatClient(model_id="stand-in", api_key="not-a-real-key", base_url=f"{url}/v1")
agent = Agent(client, INSTRUCTIONS, name="orders", tools=TOOLS, context_providers=[FileHistory(state)])
session = agent.create_session(session_id="order-O-1001")
response = await agent.run(TASK, session=session)
print(response.text)ADK appends each event to a session service as it happens, here a SQLite one, and has a resume feature that is switched on per application. Resuming passes the id of the interrupted invocation and no new message. The library prints a warning that the feature is experimental:
async def main(action: str, state: str, url: str) -> None:
from google.adk.agents import LlmAgent
from google.adk.apps import App, ResumabilityConfig
from google.adk.models import Gemini
from google.adk.runners import Runner
from google.adk.sessions.sqlite_session_service import SqliteSessionService
from google.genai import types
os.environ.setdefault("GOOGLE_API_KEY", "not-a-real-key")
agent = LlmAgent(name="orders", model=Gemini(model="gemini-stand-in", base_url=url), instruction=INSTRUCTIONS, tools=TOOLS)
sessions = SqliteSessionService(os.path.join(state, "adk.sqlite"))
runner = Runner(app=App(name=APP, root_agent=agent, resumability_config=ResumabilityConfig(is_resumable=True)), session_service=sessions)
if action == "start":
await sessions.create_session(app_name=APP, user_id=USER, session_id=SESSION)
arguments: dict = {"user_id": USER, "session_id": SESSION}
if action == "resume":
stored = await sessions.get_session(app_name=APP, user_id=USER, session_id=SESSION)
arguments["invocation_id"] = stored.events[-1].invocation_id
else:
arguments["new_message"] = types.Content(role="user", parts=[types.Part(text=TASK)])
text = ""
async for event in runner.run_async(**arguments):
if event.is_final_response() and event.content and event.content.parts:
text = "".join(part.text or "" for part in event.content.parts)
print(text)Test results
With the plain tools, counting effects in the ledger after recovery (reserve, charge, email):
| Framework | Recovery | Killed before the charge | Killed after the charge | Killed before the email |
|---|---|---|---|---|
| LangGraph | Resume | 1, 1, 1 | 1, 2, 1: charged twice | 1, 1, 1 |
| LangGraph | Resend | Did not finish | Did not finish | Did not finish |
| Agent Framework | Resend | 2, 1, 1: reserved twice | 2, 2, 1: reserved and charged twice | 2, 2, 1: reserved and charged twice |
| ADK | Resume | 1, 1, 1 | 1, 2, 1: charged twice | 1, 1, 1 |
| ADK | Resend | 1, 1, 1 | 1, 2, 1: charged twice | 1, 1, 1 |

LangGraph and ADK resume the same way. Both had recorded that the model asked for charge_card, and both ran that call again themselves on recovery, without asking the model. The first request the stand-in received after the crash already contained a result for the charge. That is the best a framework can do: only the interrupted tool is repeated, and nothing earlier. It is also exactly the case where the charge had already been made, so the card was charged twice.
Agent Framework replays the run. Nothing had been stored when the process died, because history is saved at the end of a run. The recovered agent started from an empty conversation, the stand-in asked for all three tools again, and every tool that had completed before the crash was done a second time. A crash after the charge duplicated both the reservation and the charge. This is the behaviour of a single agent with a history provider in a release candidate; the framework’s workflows have their own checkpoint storage, which this lab did not test.
Resending is not a safe default. On LangGraph it left the thread unusable. The checkpoint held an assistant message with a tool call and no answer; adding a new user message after it produced a conversation the model API rejects with HTTP 400, on every attempt. The thread has to be resumed or repaired before it will accept input again. On ADK the resend went through, because the Gemini-style conversation it sent was accepted by the stand-in, which does not check that protocol for unanswered calls. Whether the hosted API would accept it was not tested.
Idempotent tools
The second version of each tool derives a key from the business identifiers in its arguments, looks the key up before acting, and returns the recorded result if the work was already done. It is the IDEMPOTENT branch of the function shown earlier, a handful of lines. The same fifteen crashes, repeated with those tools:
| Tools | Every effect exactly once | An effect duplicated | Did not finish |
|---|---|---|---|
| Plain | 6 of 15 | 6 of 15 | 3 of 15 |
| Idempotent | 12 of 15 | 0 of 15 | 3 of 15 |

Every duplicate disappeared, in every framework, including the full replay in Agent Framework: the tools were called again and did nothing again. The three that still did not finish are the LangGraph resends, which fail before any tool is reached. Idempotent tools fix repeated effects. They do not fix a recovery procedure that cannot proceed.
The choice of key is the design decision. It is tempting to use the tool call id, since frameworks expose it and it is unique. It works when a framework re-executes a recorded call, as LangGraph and ADK do on resume, because the id is the same. It fails on a replay, because the model is asked again and issues a new id for the same intent. A key built from what the operation means, here the order number, is the same in both cases.
In this lab the effect and its record are one append to one file, so the check is airtight. In a real system the effect is a call to another service, and a local “already done” table has the same gap as a checkpoint. The key has to travel with the request and be enforced by the service that performs the effect. Payment APIs support this directly: Stripe, for one, saves the result of the first request made with a given idempotency key and returns it for retries.
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','clusterBkg':'#F8FAFC','clusterBorder':'#CBD5E1','edgeLabelBackground':'#FFFFFF','nodeTextColor':'#232F3E'}}}%%
flowchart TD
C["Tool called with arguments"] --> K["Build key from business identifiers:
operation and order number"]
K --> Q{"Has the downstream
service seen this key?"}
Q -- "yes" --> R["Return the stored result,
do nothing"]
Q -- "no" --> D["Perform the effect,
passing the key"]
D --> ST["Service stores key and result
as one operation"]
ST --> OUT["Return the result"]
R --> OUT
style K fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74
style R fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202
style ST fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95Recovery procedure
The results suggest an order of preference for whatever supervises the agent. Use the framework’s resume call where one exists, because it repeats the least. Fall back to sending the request again only on a framework that has no resume, and only with idempotent tools. And never mix the two on one thread without checking its state first.
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','clusterBkg':'#F8FAFC','clusterBorder':'#CBD5E1','edgeLabelBackground':'#FFFFFF','nodeTextColor':'#232F3E'}}}%%
flowchart TD
X["Agent process died
during a run"] --> F{"Does the framework
have a resume call?"}
F -- "yes" --> RS["Resume the same thread
or invocation"]
RS --> ONE["Interrupted tool runs again,
earlier tools do not"]
F -- "no" --> RE["Send the request again
on the same session"]
RE --> ALL["Every tool runs again"]
ONE --> I{"Are the tools
idempotent?"}
ALL --> I
I -- "yes" --> OK["Effects happen once"]
I -- "no" --> BAD["Duplicated effects:
reconcile by hand"]
style OK fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202
style BAD fill:#FEF2F2,stroke:#D13212,stroke-width:2px,color:#991B1B
style RS fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74The supervisor needs one more piece of information that none of these frameworks hands it: that a run was in flight when the process died. In the lab the test knows. In production that is a record written before the run starts and cleared when it ends, in the same store as the checkpoints, and scanned on start-up.
Cost
Durable state is not free, and this lab did not measure its price. Saving state as a run proceeds means a write to the store for every model call and every tool call, and the latency of that store is added to each step. At the scale of one order with three tools it is invisible. For long sessions the size of what is written and the time it takes should be measured against the store you intend to use, not assumed.
The cost of not having it is easier to state. Without mid-run persistence, recovery means running the whole task again: every model call is paid for twice, and every tool must tolerate being called twice. With idempotent tools that is slow and correct. Without them it is a refund.
Production considerations
- Classify tools. Reads can repeat freely. Writes need a key. Sends, payments and anything a person will notice need a key that the downstream service enforces. Write the class next to the tool.
- Derive keys from the request. Order number, invoice number, a hash of the recipient and template. Not the tool call id, not a timestamp, not a random value generated inside the tool.
- Return the original result on a repeat. A repeated call must give the model the same answer as the first, not an error saying it was already done, or the model will try to fix a problem that does not exist.
- Use a durable store. An in-memory checkpointer or session service survives nothing. LangGraph’s documentation says as much of its in-memory saver and points to PostgreSQL for production.
- Test the kill. A recovery path that has never run will not work the first time. The lab’s method, a marker file and
SIGKILL, takes a few seconds per case and fits in a pipeline. - Repair before resending. A thread left with an unanswered tool call must be resumed, or have the call answered with an explicit failure, before it is given new input.
- Watch pre-release behaviour. Agent Framework is a release candidate and ADK flags its resume feature as experimental. Pin versions and rerun the kill tests on every upgrade.
Where a tool performs a regulated action, such as a payment, a prescription or a record update, a duplicate is an incident and not a retry. The ledger of effects, with keys, is the audit evidence that an action happened once. Keep it outside the agent framework, in the system of record that performs the action.
Decision framework
| Your situation | What to do | Why |
|---|---|---|
| Tools only read | Any persistence, resend on failure | Nothing can be duplicated; only model calls are repeated |
| Tools write, framework has resume | Durable checkpoints, resume, idempotent writes | Resume repeated one tool in this run, and that one was the charge |
| Tools write, no resume for your agent type | Idempotent tools first, then resend | A replay repeated every completed tool |
| Payments, messages to people, regulated actions | Key enforced by the downstream service | A local check has the same gap as a checkpoint |
| Long sessions with many steps | Measure checkpoint size and store latency | State is written on every step |
| A caller that retries on timeout | Check for an in-flight run before sending again | Resending onto an interrupted thread failed outright on LangGraph |
Key takeaways
- Agent frameworks give at-least-once execution of tools. A kill between a tool’s effect and the framework’s record of it leads to the tool running again.
- LangGraph and ADK resumed by re-executing only the interrupted tool call. Agent Framework 1.0.0rc5, as a single agent with a history provider, replayed the whole run and repeated every completed tool.
- With plain tools, six of fifteen recoveries duplicated an effect, including a second card charge in every framework. With idempotent tools, none did.
- Build the idempotency key from business identifiers, not from the tool call id, which changes when the model is asked again. Have the downstream service enforce it.
- Sending the request again is not a universal recovery. On LangGraph it left the thread rejecting every request until it was resumed or repaired.
If you have run kill tests against an agent in production, or against a persistence back end not covered here, leave a comment with what repeated.
Everything above ran in GitHub Actions on a hosted Linux runner with Python 3.13.18, in three virtual environments: LangGraph 1.1.3 with langchain 1.2.13, langchain-openai 1.1.11 and langgraph-checkpoint-sqlite 3.0.3; Agent Framework 1.0.0rc5 (agent-framework-core); and Google ADK 1.27.2 with google-genai 1.68.0. Other dependencies were resolved as they stood on 23 March 2026. The lab ran three uninterrupted baselines and thirty crash cases: three crash points, the recovery actions each framework supports, and two tool versions. Each case had a fresh stand-in and a fresh state directory, the agent was killed with SIGKILL when a tool wrote its marker file, and effects were counted from the ledger. 10 unit tests cover the stand-in and the tools. The model was a scripted stand-in speaking Chat Completions to LangGraph and Agent Framework and the Gemini API to ADK; it rejects unanswered tool calls on the first protocol and does not check them on the second. Not run: any hosted model; other persistence back ends; Agent Framework workflows and their checkpoint storage; concurrent recoveries; a crash during a checkpoint write. Checkpoint size and latency were not measured. The run is here.
The complete sample for this article is on GitHub: nithinmohantk/blog-samples-working-repo/agent-crash-resume. Clone it with git clone https://github.com/nithinmohantk/blog-samples-working-repo.git and open the agent-crash-resume folder. Its README says how to run it and what has been verified.
Glossary
- Checkpoint— A saved copy of an agent’s state at a point in a run, from which the run can continue.
- Thread— LangGraph’s name for one conversation’s sequence of checkpoints, identified by a thread id.
- Session— The stored history of one conversation in Agent Framework and ADK.
- Invocation— In ADK, one run of an agent in response to a message. An interrupted invocation can be resumed by its id.
- At-least-once— A guarantee that an operation will happen, with the possibility that it happens more than once.
- Idempotent— An operation that has the same effect whether it is performed once or several times.
- Idempotency key— A value sent with a request so that the receiver can recognise a repeat and return the first result.
- Replay— Recovering by running the whole task again from the start.
- Resume— Recovering by continuing from saved state, repeating only what was in progress.
- SIGKILL— The signal that ends a process immediately, with no chance to clean up.
References & Further Reading
- → LangGraph documentation: persistence— Checkpointers, threads, and which savers survive a restart
- → ADK documentation: Resume stopped agents— How to enable and use resume, and the statement that tools may run more than once
- → Stripe API: Idempotent requests— How a payment API stores the first result for a key and returns it on retries
- → agent-framework-core on PyPI— Release history; 1.0.0rc5 is the version tested
- → langgraph-checkpoint-sqlite on PyPI— The SQLite checkpointer used for the LangGraph agent
- → google-adk on PyPI— Release history; 1.27.2 is the version tested
- → Sample code for this article— The stand-in, the tools and ledger, the three agents and the crash harness
Discover more from C4: Container, Code, Cloud & Context
Subscribe to get the latest posts sent to your email.