Tool Result Payloads Need Size Budgets
By wGrow Project Team ·
A SQL tool returns 50,000 rows. The orchestrator tries to push them through a streaming connection, and the connection drops. The agent sees a timeout, retries, and gets the same 50,000 rows. Prompt tuning has turned into a retry loop.
That scenario is illustrative, not a benchmark. The failure in it is real, though, and it has little to do with reasoning. It comes from transport limits.
The Illusion of Infinite Context
Developers treat tool calls like local function calls. The function returns, the result appears, the model reads it. The abstraction is so clean that the cost looks free.
It isn’t. Every tool result crosses a process boundary, then a network boundary, then a provider boundary, and each one caps memory, frame size, and time. An agent can hit those caps long before it hits any limit of model capability. The symptoms look like model problems: stalled runs, truncated answers, loops. The cause is a payload nobody sized.
What an Oversized Sync Payload Shows
A JSON synchronisation service for a legacy integration can receive a record in which a source system has dumped a multi-gigabyte error log into a single text field. The transport layer can fail immediately, and the parsers can choke on whatever is left.
Better parsing is not the fix. The fix is to enforce payload limits and convert oversized text into attached files, so the record carries a pointer instead of the bytes.
Agent builders are now repeating this mistake. A very large JSON blob from a vector database query can fail the same way. The format is newer, but the failure is identical.
Streaming Frames Are a Product Requirement
Most agent stacks stream tool output over server-sent events or websockets. Those channels have frame and buffer limits, and an oversized result can break the stream mid-flight.
When large tool results share a channel with model output, the transport stops being an implementation detail. It becomes part of the product surface.
So treat the limit as a product requirement. If you pass a raw query result straight into the model, you are betting on someone else’s transport protocol. The tool wrapper has to protect the transport from the tool, and it can’t assume the tool will behave.
Set a Hard Byte Budget
| 1 | const result = await executeTool(args); | |
| 2 | const bytes = Buffer.byteLength(JSON.stringify(result)); | ← ① |
| 3 | if (bytes > 102400) { | |
| 4 | return handleOversized(result, bytes); | ← ② |
| 5 | } | |
| 6 | return result; |
- ① Measure serialized payload size
- ② Trip the 100 KB budget and fallback
Pick a byte ceiling for every tool response before the first deployment. A figure like 100 KB is a reasonable starting point for illustration. It is a policy choice, not a provider limit, so choose yours from your own frame limits and context costs.
Enforce it in an interceptor at the tool wrapper level. The logic is short:
- Run the tool.
- Measure the serialised size.
- Under budget: return it unchanged.
- Over budget: do not forward the payload. Call the fallback handler.
Put the check in the wrapper, not in each tool. A tool author won’t remember it at 11pm. A wrapper applies it to every tool, including the ones added next quarter.
Summary First, Artifact Behind It
When a result trips the budget, stop the transfer and return a summary instead. Include:
- Total row count
- Key schema definitions
- Statistical aggregates the agent can reason over
Write the raw blob to object storage. S3 or any equivalent blob store works. Then append a scoped, expiring artifact link to the summary.
It’s the familiar attachment fix in new clothes: a pointer instead of the bytes. The agent keeps the shape of the data and can still reach the data itself.
If it needs specific rows, make it ask for them. Give it a pagination tool or a targeted ID lookup, and don’t let it re-run the wildcard select. The budget then shapes behaviour. Broad queries come back as summaries, and the agent learns to query narrowly.
None of this is free. A summary discards detail, so a poorly chosen one can hide the one row that matters. Follow-up lookups add round trips and latency. Artifact links expose raw data outside the tool call, so they need real access control. Tune the summary format against your actual queries, and review the storage permissions.
Fail Loudly, Not Quietly
The worst fallback is silent truncation. If a large array is cut down to 10 rows (an example figure) without saying so, and the agent will reasonably conclude the table holds 10 records. It will then answer confidently from that false premise. No error is raised, and nothing in the logs looks wrong.
Put the truncation in the payload itself. State that the result is partial, how many rows exist, and how many came back. The model can then reason about incomplete data instead of mistaking it for the whole.
Define retries the same way, before launch:
- Which failures are retryable (timeouts, dropped frames) and which aren’t. A result that is over budget will be over budget again.
- A maximum attempt count.
- A log line an operator can actually see, not a quiet loop.
The 50,000-row loop at the top of this piece is a retry-rule failure. Retrying a deterministic oversize result cannot succeed unless the underlying data changes. One rule, “oversize is not retryable, summarise instead”, would have ended it on the first attempt.
Budget Before You Prompt
Infrastructure limits cap how reliable an agent can be, and no amount of prompt tuning lifts that cap.
Before your next agent ships, write down four things: the payload budget, the summary format, where artifacts go, and which failures retry. Skip it and you may pay later with an afternoon spent tuning prompts against a network timeout.
As agents take on more data-heavy tools, the builders who hold up will be the ones who sized the pipe first. The model matters less if the result never arrives.