Tool Result Payloads Need Size Budgets
By wGrow Project Team ·
Refined Article
A SQL tool returns 50,000 rows. The orchestrator tries to push them through a streaming connection, and the connection drops. The agent sees a timeout, retries, and gets the same 50,000 rows. Three days of prompt tuning has turned into a retry loop.
That scenario is illustrative, not a benchmark. The failure in it is real, though, and it has little to do with reasoning. It comes from transport limits.
The Illusion of Infinite Context
Developers treat tool calls like local function calls. The function returns, the result appears, the model reads it. The abstraction is so clean that the cost looks free.
It isn’t. Every tool result crosses a process boundary, then a network boundary, then a provider boundary, and each one caps memory, frame size, and time. Agents often hit those caps long before they hit any limit of model capability. The symptoms look like model problems: stalled runs, truncated answers, loops. The cause is a payload nobody sized.
What the 2014 Sync Crash Taught Us
In 2014 we built a JSON synchronisation service for a legacy integration project. A client system dumped a 2 GB error log into a single text field. The transport layer failed immediately, and our parsers choked on whatever was left.
Better parsing was not the fix. We enforced payload limits and converted oversized text into attached files, so the record carried a pointer instead of the bytes.
Agent builders are now repeating this mistake. A 40 MB JSON blob from a vector database query can fail the same way. The format is newer, but the failure is identical.
Streaming Frames Are a Product Requirement
Most agent stacks stream tool output over server-sent events or websockets. Those channels have frame and buffer limits, and an oversized result can break the stream mid-flight.
The lesson from these streaming failures is that when large tool results share a channel with model output, the transport stops being an implementation detail. It becomes part of the product surface.
So treat the limit as a product requirement. If you pass a raw query result straight into the model, you are betting on someone else’s transport protocol. The tool wrapper has to protect the transport from the tool, and it can’t assume the tool will behave.
Set a Hard Byte Budget
| 1 | const result = await executeTool(args); | |
| 2 | const bytes = Buffer.byteLength(JSON.stringify(result)); | ← ① |
| 3 | if (bytes > 102400) { | |
| 4 | return handleOversized(result, bytes); | ← ② |
| 5 | } | |
| 6 | return result; |
- ① Measure serialized payload size
- ② Trip the 100 KB budget and fallback
Pick a byte ceiling for every tool response before the first deployment. We use 100 KB. That figure is our house policy, not a provider limit, so choose yours from your own frame limits and context costs.
Enforce it in an interceptor at the tool wrapper level. The logic is short:
- Run the tool.
- Measure the serialised size.
- Under budget: return it unchanged.
- Over budget: do not forward the payload. Call the fallback handler.
Put the check in the wrapper, not in each tool. A tool author won’t remember it at 11pm. A wrapper applies it to every tool, including the ones added next quarter.
Summary First, Artifact Behind It
When a result trips the budget, stop the transfer and return a summary instead. Include:
- Total row count
- Key schema definitions
- Statistical aggregates the agent can reason over
Write the raw blob to object storage. S3 or any equivalent blob store works. Then append a scoped, expiring artifact link to the summary.
It’s the 2014 fix in new clothes: a pointer instead of the bytes. The agent keeps the shape of the data and can still reach the data itself.
If it needs specific rows, make it ask for them. Give it a pagination tool or a targeted ID lookup, and don’t let it re-run the wildcard select. The budget then shapes behaviour. Broad queries come back as summaries, and the agent learns to query narrowly.
None of this is free. A summary discards detail, so a poorly chosen one can hide the one row that matters. Follow-up lookups add round trips and latency. Artifact links expose raw data outside the tool call, so they need real access control. Tune the summary format against your actual queries, and review the storage permissions.
Fail Loudly, Not Quietly
The worst fallback is silent truncation. Cut a 10,000-row array down to 10 rows without saying so, and the agent will reasonably conclude the table holds 10 records. It will then answer confidently from that false premise. No error is raised, and nothing in the logs looks wrong.
Put the truncation in the payload itself. Say that the result is partial, how many rows exist, and how many came back. The model can then reason about incomplete data instead of mistaking it for the whole.
Define retries the same way, before launch:
- Which failures are retryable (timeouts, dropped frames) and which aren’t. A result that is over budget will be over budget again.
- A maximum attempt count.
- A log line an operator can actually see, not a quiet loop.
The 50,000-row loop at the top of this piece is a retry-rule failure. Retrying a deterministic oversize result cannot succeed unless the underlying data changes. One rule, “oversize is not retryable, summarise instead”, would have ended it on the first attempt.
Budget Before You Prompt
Infrastructure limits cap how reliable an agent can be, and no amount of prompt tuning lifts that cap.
Before your next agent ships, write down four things: the payload budget, the summary format, where artifacts go, and which failures retry. For a single agent that’s usually an afternoon of work. Skip it and you may pay with a worse afternoon later, spent tuning prompts against a network timeout.
As agents take on more data-heavy tools, the builders who hold up will be the ones who sized the pipe first. The model matters less if the result never arrives.
Editor’s notes for the fact-checker (not part of the article):
- AWS claim: Cut. The draft gave no service or announcement behind it. The paragraph now states the transport lesson without a vendor claim.
- Step 4 of the budget logic: Kept as “do not forward the payload”. The earlier “do not serialise the whole object” contradicted step 2, which measures serialised size.
- Unsourced figures: The 2 GB log, the 40 MB blob, and the 100 KB budget are presented as first-hand or illustrative. Confirm the 2014 project details are accurate and approved for publication.
- No [Sn] tokens: The draft contained none, so there was nothing to preserve and I added none.