Context Is a Budget, Not a Bucket: Agent Memory Architecture That Survives Production

Every agent I’ve shipped that broke in production broke the same way. Not a hallucination, not a bad tool call - the context window quietly filled with garbage until the model started ignoring instructions, and the invoice for the month arrived looking like a hardware purchase. Prompt quality gets all the attention. Context architecture is what decides whether your agent works on step 40 the same way it worked on step 3.

This is the part of agent engineering that feels like backend work, because it is. You’re designing a memory hierarchy under a hard byte limit with a per-byte price tag. Let me show you how I actually do it.

The failure mode nobody budgets for

A naive agent loop appends everything: the user message, the model’s reasoning, every tool call, every tool result. Then it sends the whole thing again next step. Cost per step grows linearly, so total cost for an N-step task grows quadratically.

Real numbers from a data-enrichment agent I refactored last year:

• System prompt + tool schemas: ~6k tokens
• Average tool result (an HTML fetch, truncated): ~4k tokens
• Steps per task: 25–40

Naive loop, 35 steps: the final call carries ~145k tokens, and the sum across the task is roughly 2.6M input tokens. After refactoring - same model, same task success rate - it was 310k input tokens. An 8x cost reduction with zero prompt-engineering cleverness. And accuracy went up, because the model stopped drowning in stale HTML from step 4.

That last part is the one people underestimate. Long context isn’t just expensive, it’s actively harmful. Instructions in a 6k-token prompt get followed. The same instructions buried under 140k tokens of tool noise get “sort of” followed. Models degrade with irrelevant context long before they hit the window limit.

Four kinds of memory, four different lifetimes

Stop thinking about “the context.” Think about four stores with different lifetimes and different rules for entering the prompt.

1. Constant memory - the system prompt, role, hard rules, tool schemas, output format. Written once, never mutated during a run. Goes into every call.

2. Working memory - the current task state: goal, plan, what’s done, what’s pending, key facts discovered. Small, structured, rewritten each step. Goes into every call.

3. Episodic memory - the raw transcript of what happened: messages, tool calls, results. Grows without limit. Does not go into every call. Lives in your database or on disk, referenced by ID.

4. External memory - durable artifacts: files, DB rows, scraped pages, generated reports, user profile data. Retrieved on demand, never pushed.

The entire discipline is: only 1 and 2 are always present. Everything else enters the prompt by explicit, budgeted, on-demand retrieval.

Give every slot a byte budget

I write the budget down before I write the loop. For a mid-size agent on a 200k-window model, my default:

Slot	Budget	Behaviour on overflow
System + rules	3k	Hard fail - refactor the prompt
Tool schemas	4k	Prune tools by phase
Working state (JSON)	2k	Compact oldest fields
Last N tool results	8k	Truncate + store to disk
Retrieved documents	10k	Drop lowest-scoring chunks
Conversation tail	6k	Roll into working state
Total per call	~33k	Kill switch at 60k

The kill switch matters more than the budget. If a single call exceeds the hard ceiling, the run aborts and logs a diagnostic dump. That turns a silent $400 weekend into a Slack alert on Friday afternoon. I’ve had a scraper agent get stuck in a fetch-retry loop; the ceiling caught it in 90 seconds.

Also: prune tool schemas by phase. An agent with 18 tools burns 6–8k tokens on definitions every single call, and choice quality drops. Split the run into phases (research → decide → write) and expose 4–6 tools per phase. Cheaper and more accurate.

Compact into structured state, not chat summaries

The most common compaction approach is “summarize the conversation when it gets long.” It’s lossy in exactly the wrong way - it keeps the vibes and loses the identifiers. I’ve seen a summary preserve “the user asked about pricing tiers” and drop the actual account ID the agent needed three steps later.

Instead, maintain a single explicit state object that the agent rewrites (or that you update deterministically after each step):

{
  "goal": "Enrich 500 companies with founder email + headcount",
  "phase": "enrichment",
  "progress": { "done": 312, "failed": 9, "remaining": 179 },
  "decisions": [
    "LinkedIn blocked at step 14 -> using Apollo fallback",
    "Headcount from Crunchbase preferred over site footer"
  ],
  "open_questions": [],
  "artifacts": [
    { "id": "a_71", "kind": "csv", "path": "/run/out/enriched_p1.csv", "rows": 312 }
  ],
  "next_action": "resume batch at offset 312"
}

This is ~400 tokens and it replaces 60k of transcript. The model reads it and knows where it stands. Critically, it survives process restarts - this object is your resume point. Persist it to Postgres or a JSON file after every step. An agent that can’t resume from a crash isn’t a production system, it’s a demo.

My rule: the transcript is a log, the state is the memory. Logs are for humans debugging. State is for the model.

Return handles, not payloads

Tool output is where 80% of bloat lives. A tool that returns 40k characters of HTML or a 2,000-row query result is a design bug.

What tools should return:

BAD:  fetch_page(url) -> "<html>...38,000 chars...</html>"
GOOD: fetch_page(url) -> { "artifact": "pg_44",
                           "path": "/run/cache/pg_44.html",
                           "bytes": 38214,
                           "title": "Pricing - Acme",
                           "preview": "first 600 chars..." }

Then give the agent narrow readers over artifacts: grep_artifact(id, pattern), read_lines(id, start, end), extract_fields(id, schema). The model pulls the 200 tokens it needs instead of paying for 10k it doesn’t. This is exactly how a competent human works with a big file - nobody reads the whole log, they grep it.

Same for SQL: return row count plus the first 5 rows plus the artifact path to the full CSV. Same for API calls: strip response envelopes, drop nulls, drop fields not in your schema. I keep a slim() helper that projects every API response down to a whitelist of fields before it ever touches the prompt. It routinely cuts payloads by 70%.

Order for the prefix cache

All major providers cache repeated prefixes and charge a fraction for cache hits. That’s a 5–10x discount on your biggest, most stable block - but only if the prefix is byte-identical.

So: stable first, volatile last.

1. System prompt and rules (never changes)
2. Tool schemas (per phase, stable within phase)
3. Long-lived reference context (style guide, client config)
4. Working state (changes each step)
5. Recent tool results (changes each step)

And stop putting Current time: 2026-02-11T14:03:22Z at the top of your system prompt. One volatile token at position 12 invalidates the cache for everything after it. If the agent needs the time, give it a get_time() tool or inject it in the volatile tail. I’ve seen this single change cut a client’s bill by more than half.

Instrument it or you’re guessing

Per step, log: step_index, input_tokens, cached_tokens, output_tokens, tool_name, tool_result_bytes, state_bytes, latency_ms. Per run, log: total cost, steps, high-water context, outcome.

The metric I actually watch is cost per completed task, not cost per token. A model that’s 3x more expensive per token but finishes in 6 steps instead of 30 is cheaper. Cheap models that loop are the most expensive thing in AI automation.

Second metric: cache hit ratio. Below 60% on a multi-step agent means your prefix is unstable - go find the timestamp.

Third: context high-water mark per run, plotted over time. When it starts creeping up week over week, some tool started returning more data than it used to. That’s your early warning before the invoice is.

The loop, roughly

state = load_state(run_id) or init_state(goal)

while not state["done"] and state["step"] < MAX_STEPS:
    ctx = build_context(
        constant=SYSTEM,                        # stable prefix
        tools=tools_for_phase(state["phase"]),  # 4-6 tools
        state=state,                            # ~400 tokens
        tail=recent_results(limit=3, max_tokens=8000),
    )
    assert count_tokens(ctx) < HARD_CEILING

    step = model.call(ctx)
    result = run_tool(step.tool, step.args)     # returns a handle
    store_artifact(result)
    state = update_state(state, step, result)   # deterministic where possible
    save_state(run_id, state)                   # crash-safe

Note update_state is deterministic where it can be - counters, artifact lists, phase transitions are Python, not LLM output. Only the fuzzy parts (decisions, open questions) come from the model. Every field you can compute in code is a field the model can’t corrupt.

What I skip

• A vector DB by default. For most agents, a state object plus grep over files plus one SQL query beats semantic search. Add embeddings when you have thousands of documents and a genuine recall problem, not because it’s the expected architecture.
• Memory frameworks with opinions. I’ve replaced two of them with ~200 lines of state + artifact code and got better observability.
• Infinite conversation history. Nobody needs step 4’s HTML at step 40. If they do, that’s what the artifact store is for.

Context engineering is unglamorous - it’s byte counting, projection, and cache discipline. But it’s the difference between an agent demo and an agent that runs 500 tasks a day at a price you can put in a proposal.