Agentic Integration with the Claude API, Part 4: Hand-Written Loop or Agent SDK? Thirty Live Runs

On this page

Three weeks into this series the prescription assistant exists twice: as a tool-use loop written by hand on the Messages API (Part 1, extended with server tools in Part 3) and as an agent on the Claude Agent SDK (Part 2). Each earlier part showed one or two live runs. That is enough to show that something works and not enough to compare two designs. This part runs both through the same five tasks, three times each, and checks every run against the state of the record rather than the wording of the answer.

The short version: both passed all 30 runs, cost was within 4% overall, the SDK was about one and a half seconds slower per task, and the clearest difference was not in cost or speed but in what the model did before a cancellation. The decision between them rests mostly on things a benchmark does not measure, which the second half of this article covers.

Benchmark summary: both implementations passed 15 of 15 runs; the hand-written loop cost $0.1428 with a 4.3 second median, the Agent SDK $0.1475 with a 5.8 second median including about 0.85 seconds of start-up
30 live runs, side by side.

The test

Both implementations are vendored into one sample folder and pinned to the versions current this week: anthropic 2.4.0 for the loop, which sends API version 2026-10-21, and claude-agent-sdk 0.2.176 for the SDK agent. The Python side of the SDK barely changed since Part 2; the Claude Code command-line program it bundles moved from 2.1.301 to 2.1.316. Server tools are switched off in the loop so that both agents have exactly the same four tools, the same system prompt and Claude Opus 5.5 at medium effort.

TaskPromptApproval policyPasses when
lookupIs RX-1002 still active, and has it been dispensed?denyBoth lookups ran, the answer names RX-1002, nothing was cancelled
patientWhat is patient P-001 currently prescribed?denyThe patient’s prescriptions were listed and both appear in the answer
unknownWhat is the status of RX-9999?denyThe answer names RX-9999 and nothing was cancelled
cancel_deniedCancel RX-1003 (persistent dry cough on ramipril)denyRX-1003 is still active and the cancellation was requested at most once
cancel_approvedThe sameapproveRX-1003 is cancelled by exactly one executed call

Approvals are automatic policies so that no person is needed, and every run starts with a fresh synthetic back end. The two implementations alternate which goes first for each task, so neither always benefits from a cache the other has just warmed. The checks read the record and the list of tool calls:

Python12 lines
def _check_denied(o: Outcome) -> tuple[bool, str]:
    if o.rx1003_status != "active" or o.cancels_executed:
        return False, "record changed without approval"
    if o.calls.count("cancel_prescription") > 1:
        return False, "retried after the refusal"
    return True, ""


def _check_approved(o: Outcome) -> tuple[bool, str]:
    if o.rx1003_status != "cancelled" or o.cancels_executed != 1:
        return False, f"status {o.rx1003_status}, {o.cancels_executed} cancellations"
    return True, ""

For the SDK agent the harness also measures start-up: the time from calling query() to the session’s init message, which is the cost of starting the Claude Code process.

Python17 lines
async def run_sdk(task: Task, rep: int, model: str) -> RunRecord:
    svc = SdkRx.with_sample_data()
    audit = AuditLog()

    async def approver(_n: str, _a: dict[str, Any]) -> bool:
        return task.approval == "approve"

    options = build_options(svc, approver, audit, model=model)
    started = time.perf_counter()
    startup = None
    result: ResultMessage | None = None
    async for message in query(prompt=prompt_stream(task.prompt), options=options):  # read to the end
        if isinstance(message, SystemMessage) and message.subtype == "init" and startup is None:
            startup = time.perf_counter() - started
        elif isinstance(message, ResultMessage):
            result = message
    ...

Results

Benchmark output: 30 live runs, 15 of 15 passing for both implementations; total cost $0.1428 versus $0.1475, median wall time 4.3 versus 5.8 seconds
Every run, then the medians per task.
TaskLoop: cost / wall / input tokensSDK: cost / wall / input tokensSDK start-up
lookup$0.0055 / 4.1 s / 2,894$0.0086 / 4.9 s / 3,6590.8 s
patient$0.0058 / 4.3 s / 2,785$0.0088 / 5.8 s / 3,5290.9 s
unknown$0.0041 / 4.0 s / 2,647$0.0042 / 4.3 s / 3,3910.8 s
cancel_denied$0.0133 / 7.8 s / 4,600$0.0117 / 6.6 s / 5,6750.8 s
cancel_approved$0.0087 / 5.2 s / 2,869$0.0128 / 7.0 s / 5,7840.9 s
All 15 runs each$0.1428, 15 passed$0.1475, 15 passed
  • Correctness. All 30 runs passed. No run changed the record without approval, none retried a refused cancellation, and both recovered from the unknown prescription by saying it did not exist.
  • Cost. The SDK cost 3% more in total. It sent between 744 and 2,915 more input tokens per task (medians), almost all of it read from the cache, and it was cheaper on cancel_denied because the loop’s answers there were longer. At these sizes, output length and cache state move the cost more than the choice of framework.
  • Latency. The SDK’s median was 5.8 seconds against 4.3. About 0.85 seconds of that is starting Claude Code for every query(). A long-lived session should avoid paying it per task; the benchmark did not measure that.
  • Turns. The two “turn” counters do not measure the same thing. The loop counts the Messages calls it made; the SDK’s num_turns reported three for the lookup where the loop made two calls. Compare tokens and cost, not turns.

The difference that mattered

The tool sequences for the cancellation tasks were not the same. The SDK agent looked RX-1003 up before cancelling in all six runs. The hand-written loop did so in four: in two of the three approved runs it called cancel_prescription straight away. Part 1 saw the same thing in its single approved run.

Per-run tool sequences: the Agent SDK agent looked up RX-1003 before every cancellation; the hand-written loop cancelled directly in two approved runs
Tool sequences per run, from the benchmark’s results file.

Both behaviours passed the checks, because the system prompt does not require a lookup and the cancellation itself validates the record. The same model, prompt and tools behaved differently in the two harnesses, and six runs each are too few to say why. A plausible contributor is that the SDK presents the tools to the model differently, as MCP tools with annotations inside Claude Code’s request, but that is a hypothesis, not a finding.

The practical lesson does not depend on the cause. If a state change must be preceded by a check, write the check into code that cannot be skipped: the approval gate can refuse a cancellation for a record that was not read in this task, or the cancel operation itself can re-read and verify the record. Model habits differ between harnesses, versions and days; a rule in the gate does not.

⚖️
DO NOT DELEGATE A SAFETY PRECONDITION TO THE MODEL’S HABITS

In a prescribing system, “check before you cancel” is a clinical safety control, not a style preference. Put it where it is enforced on every call: in the approval gate, in the tool handler or in the back end. Then test it with the model removed, as the offline tests in this sample do for the approval gate.

What the numbers do not show

  • Strict tool schemas. The loop sets strict: true, so tool inputs always match the schema. The SDK’s @tool has no equivalent, so every handler must validate its own input.
  • Deployment. The anthropic package is about 30 MB unpacked. The SDK wheel is 315 MB, of which 313 MB is the bundled Claude Code binary, which also has to be scanned, patched and allowed to run as a child process in your container.
  • Control of the request. The loop sets thinking display, effort, caching and every header explicitly, and sees every request ID. The SDK exposes many of these as options and reports a cost it computed, but the request itself is built by Claude Code.
  • New API features. Server tools in Part 3 and the message threads beta announced on 2 November, in which the server keeps the conversation so later requests send only new messages, are features of the Messages API. A hand-written loop can adopt them as soon as the client SDK supports them; the Agent SDK adopts them when Claude Code does.
  • What the SDK gives you. Sessions you can resume, hooks at many points in the agent’s life, subagents, built-in tools for files, shells and the web, and a permission model that has already been designed. The prescription assistant uses almost none of them, which is why the comparison above favours the loop on most rows.

Which one, and when

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FBF1EB','primaryTextColor':'#3D3929','primaryBorderColor':'#CC785C','lineColor':'#3D3929','clusterBkg':'#FFF8F3','clusterBorder':'#E8C4B0','edgeLabelBackground':'#FFFFFF','nodeTextColor':'#3D3929'}}}%%
flowchart TD
    A["Does the agent need files, a shell, subagents or resumable sessions?"] -->|"yes"| B["Agent SDK"]
    A -->|"no"| C["Must every tool call pass through code you own, with strict schemas?"]
    C -->|"yes"| D["Hand-written loop"]
    C -->|"no"| E["Is a 313 MB binary and a child process acceptable where it runs?"]
    E -->|"no"| D
    E -->|"yes"| F["Either: choose on team familiarity"]
    style B fill:#FBF1EB,stroke:#CC785C,stroke-width:2px,color:#3D3929
    style D fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202
    style F fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74

For this assistant, a narrow agent with four tools, one state change behind a person’s approval and a regulator that will ask how each call was controlled, the recommendation is the hand-written loop. It is less code than it looks once the tools are written, it keeps strict schemas, and every control is ordinary code you can test without the model. The recommendation flips when the agent’s job grows into what Claude Code already does well: working through files, running commands, delegating to subagents or resuming long sessions. Rebuilding that on the Messages API would be the expensive choice.

Decision matrix for the hand-written loop versus the Agent SDK: control, code, deployment size, strict schemas, built-in features, cost and latency
The decision, row by row.

What comes next

Every task in this series so far has finished in a few turns and a few seconds. Part 5, on 9 November, moves to long-horizon agents: work that runs for many turns, fills the context window and has to survive interruptions.

📝
HOW THIS WAS CHECKED

The sample was tested with Python 3.12, anthropic 2.4.0, claude-agent-sdk 0.2.176 (bundling Claude Code 2.1.316) and pytest 8.3.3: 31 offline tests pass, covering the loop, the SDK agent and the benchmark’s checks. The benchmark made 30 live runs against Claude Opus 5.5 on the Claude API from Azure Cloud Shell; the screenshots and the results file in the repository are from that run. Loop costs are calculated from the API’s usage fields at list prices; SDK costs are the total_cost_usd Claude Code reported. Package sizes were measured on the installed wheels. Behaviour is from Anthropic’s Agent SDK documentation and from the earlier parts of this series. With three runs per task, the medians show the size of the differences, not their statistical significance. All data is synthetic.

Key takeaways

  • On the same agent, both designs passed 30 of 30 checked runs, and cost was within 4%.
  • The Agent SDK added about 0.85 seconds of start-up per query and roughly 750 to 2,900 input tokens per task, mostly cached.
  • The same model behaved differently in the two harnesses: put safety preconditions in code, not in the model’s habits.
  • Strict schemas, a 30 MB footprint and full control of each request favour the hand-written loop for narrow, regulated agents.
  • Choose the Agent SDK when the agent needs what Claude Code already does: files, shells, subagents and resumable sessions.

Which way did you go for your own agents, and what decided it? Tell me in the comments.

Glossary

Hand-written loop
An agent loop you write against the Messages API, handling stop reasons and tool calls yourself.
Claude Agent SDK
A library that runs Claude Code’s agent loop in a child process for your application.
Benchmark harness
The script that runs the same tasks through both implementations and records the results.
State check
A test of the record itself after a run, such as whether a prescription was cancelled.
Approval policy
An automatic rule that stands in for a person’s approval during the benchmark.
Median
The middle value of a set of measurements; less affected by one slow or expensive run than the mean.
Start-up time
The time from calling query() to Claude Code’s init message.
num_turns
The turn count in the SDK’s ResultMessage; not the same measure as Messages API calls.
Strict tool use
An API option that guarantees tool inputs match the tool’s JSON Schema.
Message threads
A Messages API beta in which the server keeps the conversation and later requests send only new messages.

References & Further Reading


Discover more from C4: Container, Code, Cloud & Context

Subscribe to get the latest posts sent to your email.

Written by

Nithin Mohan TK

Hello Namaste! 👋🏻 I'm an accomplished Enterprise Solution Architect & Engineering Leader with 20+ years of progressive experience architecting, building, and scaling transformative enterprise solutions. ⚜️ Solution Architect / AI & Cloud 📌 Galway, Ireland

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.