Agent Memory Needs Garbage Collection Jobs

The Bill for Remembering Everything

At Google Cloud’s published rate card as of this writing, a standard Vertex AI Vector Search deployed index runs $1.08 per node hour — about$777 for a 30-day month just to keep an agent’s memory hot and queryable [S1]. Every retrieval to build context for Gemini 1.5 Pro adds a separate charge: $1.25 per million input tokens at the standard context tier [S2], stacked on top of a storage bill that’s already accruing whether you use it or not.

Agent memory isn’t a toy vector array running on someone’s laptop anymore. It’s structured, managed state with a storage line item and a read-write latency profile — the same category of thing as a production database. Google’s own framing of Agent Memory Bank as a managed service makes this explicit: memory now carries an operational cost, not just a conceptual one.

Most developers I’ve worked with — including earlier versions of my own team, if I’m honest — treat agent memory as an infinite append-only log. Every user interaction, every extracted rule, every session note gets embedded and dumped into a vector store on the theory that more context can only help. It doesn’t. In practice, that habit produces two outcomes: an escalating cloud bill, and an agent that hallucinates more often because retrieval keeps surfacing contradictory versions of the same fact. Once memory carries a bill, it needs the lifecycle discipline any database needs — retention classes, deletion jobs, scheduled reviews for facts that have gone stale. Few teams would run production Postgres without a retention policy. Agent memory doesn’t get a pass just because it’s newer.

Failure Modes of Append-Only Memory

We learned this the expensive way, running multi-agent crews in production and cleaning up after them. Treating agent state as an infinite ledger fails in two specific, repeatable ways.

Vector retrieval bloat. We built a multi-agent crew for automating compliance document reviews. In the early version, every rule interpretation the agent produced got extracted and written to long-term memory as a fact — nothing ever removed old entries when a rule changed. Within weeks, the vector database had filled with obsolete regulatory interpretations sitting right alongside current ones. The agent started hallucinating, and not because the model was weak. Retrieval was surfacing near-duplicate embeddings of an old rule and its replacement at almost identical semantic distance, and the agent had no way to know which one was current. So it blended them into an answer that was confidently wrong.

Context window token burn. A customer service agent prototype we built tracked user dietary preferences in long-term memory. A user who changed their preference three times over a few weeks didn’t get their record updated — they got three new entries, each true when written and false the moment after. The agent retrieved the entire preference history on every prompt, because the retrieval layer had no concept of “supersedes.” That meant paying token costs to read stale, contradictory data on every single turn, and occasionally acting on the wrong preference because the ranking between conflicting facts was close enough to be arbitrary.

Both failures trace back to the same design mistake: writing memory like a log instead of a record. The append-only architecture eventually forced us to write bespoke purge scripts just to stop the system collapsing under its own accumulated history. That’s not a scalable position — it’s a symptom of missing a retention model from the start.

Defining Memory Retention Classes

We stopped treating agent memory as one undifferentiated pool of text. Now retention classes are enforced before anything gets written to disk, full stop, and the class determines storage tier, write behavior, and deletion rule.

Ephemeral state covers variables needed only for the current task execution — the working scratchpad of a single run. It stays in local runtime memory and gets discarded the moment the task ends, at zero cloud storage cost.

Session memory is context needed for an active conversation or a multi-step workflow spanning several agent calls. It lives in an in-memory datastore like Redis with a hard 24-hour time-to-live. If a session hasn’t been touched in a day, it’s gone. No exceptions carved out for “might be useful later” — that instinct is exactly what created the bloat in the first place.

Core entity facts are the verified, current-state data: an active compliance rule, a user’s current dietary preference, an account’s live configuration. These live in the managed vector database under stable entity keys, and updates here are destructive by design — the system uses UPSERT-or-delete-and-reinsert semantics so the new fact replaces the old one instead of appending to it. When a user changes their dietary preference, the old preference stops existing as a retrievable fact the instant the new one is written. There’s exactly one current answer to “what does this user prefer,” not three competing answers ranked by embedding similarity.

Audit log is the historical record kept for debugging or regulatory purposes — every version of every fact, timestamped, written to cold object storage outside the agent’s retrieval path [S4]. Critically, the active agent can’t read this bucket. It exists for a human or a compliance process to query after the fact, not for the agent to reason over during a live task. Separating the audit trail from working memory is what actually solved the hallucination problem in the compliance crew — the agent no longer has access to superseded rules at inference time.

Building the Garbage Collection Pipeline

Retention classes are policy. They don’t enforce themselves. The lifecycle jobs have to be built into the agent memory architecture from day one, and this can’t be delegated to the LLM — an agent won’t reliably decide to delete its own memory, and asking it to try just adds another unpredictable step to an already probabilistic system.

Time-to-live sweeps run as scheduled jobs independent of the agent framework — a cron job, not a prompt. These scripts query the datastore for anything past its TTL, mostly expired session memory, and execute hard deletes. It’s plain database maintenance, and it should look as unremarkable as a nightly job pruning expired sessions from any other web application.

Stale-fact consolidation runs as a background semantic deduplication job — weekly, in our compliance crew. When the script detects two highly similar vectors tied to the same entity but different timestamps, it doesn’t ask the LLM to adjudicate. It triggers a deterministic function: keep the newer, delete the older. No model call, no ambiguity, no added cost.

Right-to-forget workflows are the one that catches people off guard. When a user deletes an account, or a client revokes a document from a compliance review, the system needs to cascade that deletion through every memory tier the entity touched. Managed vector services have no inherent concept of which embeddings belong to which entity beyond whatever metadata was attached at write time — so the entity-to-vector mapping, and the deletion pipeline that walks it, has to be built by hand. Skip this step and you’re not just leaving stale data lying around. You’re leaving data that legally shouldn’t exist anymore sitting in a bucket you’re still paying to store.

Treat Agent Memory Like a Database

Infinite agent memory isn’t the feature it’s sometimes made out to be. In production, it’s a liability with a monthly invoice attached. An agent that remembers everything forever isn’t smarter for it — it’s carrying an ever-growing set of contradictions it has no principled way to resolve, and someone is paying to keep those contradictions retrievable.

If you store state, you manage it. Define retention classes before writing the first vector. Schedule the garbage collection jobs before the first stale fact accumulates. Build the deletion pipeline before someone asks you to prove it exists. The alternative isn’t a smarter agent. It’s a more expensive one — occasionally wrong for reasons nobody can trace back to a root cause.