The reflex to dump everything
First week with 1M context, every engineer on the team started packing more into the prompt. Full repos, full logs, full docs. Response quality did not match the enthusiasm. A few patterns emerged once we settled down.
Where 1M context pays off
Whole-repo reasoning. Handing the model every relevant file at once is qualitatively different from hoping RAG surfaces the right chunks. For cross-cutting refactors — renaming a domain concept, updating an interface across 40 call sites, auditing a feature end-to-end — there is no substitute for seeing everything at once.
Long transcripts and documents. Legal contracts, incident post-mortems across a week of Slack, a full design doc with 50 comments — these benefit from being read as one piece rather than chunked.
Agent sessions with deep history. Coding agents that accumulate context over a long task maintain more coherence when they do not have to re-summarize. Prompt caching makes this economically sane.
Where we still keep it short
Needle-in-haystack questions. Asking "where is the bug in these 800k tokens of code" gets worse answers than narrowing to the relevant subsystem first. The model can find a specific fact in a million tokens, but complex reasoning across a million tokens still benefits from focus.
Hot-path APIs. A 200k-token system prompt on every request, cached or not, slows first-byte latency and increases cost. For a latency-sensitive production endpoint, we still trim aggressively.
Prompt caching is the economic enabler
Without caching, repeated 500k-token prompts would be untenable. With cache breakpoints placed at the end of stable content, the repeat-read cost drops by roughly 10x. Cache TTL is five minutes — design the session cadence around it. A query every four minutes keeps the cache warm; a query every six minutes recomputes it cold.
The practical rule: put stable context first (codebase, docs, instructions), put the variable task last. Cache breaks at the boundary.
What changed in how we build
Two tools we rewrote on top of 1M context:
-
Code review agent. Old version summarized the diff and relied on embeddings to find related files. New version loads the full relevant package and the diff. Review quality improved dramatically — it now catches regressions that required understanding of distant code.
-
Incident triage. Old version took a stack trace and a few log lines. New version ingests the full alert, a window of logs, relevant service code, and recent deploys. The first guess is useful in a way the old version's was not.
What did not change
For most chat turns, a tight prompt with the minimum relevant context still wins on latency and cost. The 1M window is a capability for specific workloads, not a default to turn on. Treat it like you treat a powerful but expensive tool: reach for it when the problem shape matches.