Your Context Budget

TL;DR

Every piece of information costs attention. From AI context windows to the humans who have to read and maintain it, context (or cognitive) overload is real.

Documentation can really help, it can distill helpful context, but it is a double edged sword. Poor quality or stale documentation defeats its purpose and just adds to the noise. Too much context and the cost of keeping it updated outwights the benifit. But not enough and you end up relearning the same things again and again. Its a tricky budget to balance.

This post lays out a framework for deciding what to document, where to put it, and when to delete it. 5-10 minute read. For developers and engineering leads tired of documentation that nobody trusts.

The problem

Teams can end up documenting implementation details instead of intent. Ticket details  get copied into README.md (or even worse Claude.md) files. Specs get committed alongside code. Planning notes get preserved "for posterity".

Then the code changes, but the docs don't. And now you've got documentation that actively misleads anyone who reads it, human or AI. Eating up precious cognitive/context budget for a negative impact. You would be better off with no documentation at all.

Outdated documentation damages developer trust in ALL documentation. When docs go stale, developers stop trusting them. Now you've spent context budget on docs that harm rather than help.

"When documentation is outdated, it's worse than no documentation at all because people trust it and act on that trust" — Trevor Lasn

The foundational assumption: docs should live with the code

Store documentation about the code, with the code. That kind of documentation should be committed into the repository, not in external tools.

Why?

• AI-first: The primary consumers of documentation are increasingly AI agents. Docs in the repo are already in scope where the AI operates. (For more on this shift, see AI is going to improve your documentation... but not the way you expect)
• Developer-friendly: Developers find docs where they expect them—next to the code—without separate logins or editing systems.
• Single source of truth: Code and docs evolve together in the same commits, PRs, and review cycles.

Format: Markdown files with internal linking. Human-readable AND machine-readable. That's the whole point.

Four layers of documentation

Documentation exists at four distinct layers. Each has different purposes, lifecycles, and budget implications. Two you keep, two you throw away.

Layer 1: Process & standards (repo wide, loaded every time)

• What it is: How work gets done, not what the code does
• Where it lives: Root level—CLAUDE.md, CONTRIBUTING.md, .cursor/rules, core docs/ folder etc.
• Content: Coding standards, team norms, tooling setup, CI/CD processes, agentic personas
• Budget impact: High leverage, high cost. This gets loaded on every edit, every commit, every push. Less is more.

Layer 2: Implementation documentation (code specific, loaded sometimes)

• What it is: Current truth about the code you are interested in. What it does and is meant to do
• Where it lives: In the codebase, hierarchical, next to the code it describes
• Content: Module purpose, boundaries, key interfaces, non-obvious behaviors
• Budget impact: Medium ongoing cost. Only loaded when needed, but must be maintained. Worth the spend if it prevents misunderstanding, but over-documentation is a big risk. Less is still more.

Anti-pattern: Don't document future plans or previous implementation plans (layer 3) in code. They become stale and create mistrust.

Layer 3: Planning (team wide, loaded only once)

• What it is: Future plans for execution, tickets, epics etc
• Where it lives: External tools. Notion, Jira, GitHub Issues, Google Docs
• Content: What you're going to build, ideas on how you might
• Budget impact: One-time cost. It's gone in the next task. But expensive if you keep re-reading the same things / miss critical detail

Key rule: Don't have this in code. Plans change, often by people who don't want to use git. It's ephemeral from the code's point of view, an input, not an artifact.

Modern AI agent systems distinguish between ephemeral and persistent context. Google's ADK framework describes "working context" as "ephemeral (thrown away after the call)" while "sessions" serve as the "durable log of the interaction." — Google Developers Blog

Layer 4: Notes and Thoughts (task specific, todo tracking)

• What it is: Your plan or the AI's implementation plan
• Where it lives: In your head, in the ai's todo list, in plan.md or in micro ticketing like beads
• Content: How you plan to build, steps
• Budget impact: One-time cost. But can expensive (due to re-reading) if you need to onboard new agents sessions (including compactions) or teamates to the task. Doubly so if its not well structured, clean and up to date.

Key rule: Don't commit this to the repo. Make it easy to edit and keep up-to date. Don't over think it. Try beads. It can make working with sub agents easier.

Structure like C4, not like a spec doc

Think wiki, not Word.

The C4 model gives us a mental framework for hierarchical abstraction:

• Context → High-level, stable, rarely changes
• Container → System boundaries, still fairly stable
• Component → More detail, changes more often
• Code → Implementation level, changes frequently

The rule: Documentation detail should match the layer. Implementation details belong at the implementation layer, not at Context or Container level.

If you put implementation details too high, you're risking overspending. Every code change may require cascading documentation updates at multiple levels. That's paying interest on documentation debt.

The principle: High-level docs live closer to root. Implementation docs live at leaf nodes, next to the code.

For cross-cutting concerns: Place docs at the lowest common ancestor in the folder tree—the point where all affected modules are children.

This naturally scopes what gets loaded into context. An AI working on a specific feature only needs the docs at that level and above, not sibling implementation details. You're spending budget only on what's relevant.

"The challenge isn't just crafting the perfect prompt—it's thoughtfully curating what information enters the model's limited attention budget at each step" — Anthropic Engineering

Document intent, not implementation

When you do document decisions, focus on intent and outcomes rather than implementation details.

• Why did we choose this approach for the customer onboarding flow?
• What outcome were we optimizing for when we designed this feature?
• Why does this module exist and what problem does it solve?

Capture the why and the intended outcome, not the how. The how is already in the code and comments.

Our approach (which differs from traditional guidance):

• Co-locate with code: Decision docs live next to the code they relate to. Cross-cutting decisions go at the appropriate higher level.
• Consider deleting when superseded: Old decisions that no longer apply could be deleted. Git history preserves them if anyone needs to spelunk.
• Less is more: Only document decisions that would be non-obvious to a future developer (or AI) reading the code.

On deleting superseded ADRs: Traditional guidance recommends keeping them and marking as "Superseded." I take a different view. Git history serves the archival purpose, and stale docs add cognitive load.

The tradeoff: git history requires deliberate searching. You have to know to look for it. Current docs are loaded automatically. If your team frequently references historical decisions, by all means keep superseded ADRs/docs, but perhaps in an `/archive/` folder excluded from AI context?

But my default is to treat documentation like code: we delete old code, we don't comment it out. Same principle applies to docs.

Match persistence to relevance time-frame

The principle: Select tooling and persistence based on expected relevance timeframe. Notes on how you implemented a bug fix become irrelevant quickly. Your vision document needs to persist much longer.

On task-level documentation: Most developers don't write down every small task in a formal system. They jot notes in a scratch file, on paper, or just keep it in their heads. This is natural and appropriate—these are stream-of-consciousness notes that help the developer but would be noise for anyone else.

The output that matters is the code that gets committed, not the transient thoughts that led to it. Recording every micro-decision clutters your context window and creates a painful archaeology project for future developers.

Tests vs. documentation?

BDD and TDD can be used to move some documentation burden from prose files into runnable code. Its documentation we can prove works.

With LLMs, we can now use specs to check code against specifications, but due to AI's limitations, somewhat unreliably. Even still, the line between tests and documentation is getting blurry.

What I've found: it comes down to "what" versus "why."

If the documentation is describing a what (what its expected to do), there's a good chance you should use a test. Tests run automatically on commits and CI/CD pipelines. Repeatable, predictable results. Some interesting developments in the space, including using AI to help make E2E less brittle e.g Stagehand.

The why is harder. That's where prose documentation and help explain and be useful context for the huma/AI to make future choices, if it can be leveraged effectively.

I don't have a hard and fast rule here. We're still working this one out as we go.

The AI productivity reality check

Despite widespread adoption, the productivity impact of AI tools is nuanced:

• METR's 2025 study found experienced developers take 19% longer on tasks when using AI tools, and those developers believed they were 20% faster. The self-assessment was inverted from reality. — METR Study
• 46% of developers don't trust AI-generated output accuracy (up from 31% in 2024) — Stack Overflow 2025 Survey
• The #1 frustration: "solutions that are almost right, but not quite" (66% of developers)

I think, some, but not all, of this can be aided by better context (documentation).

This is because ambiguity is the enemy. When AI or humans interprets unclear or bloated documentation, it can produces those "almost right" solutions based on out of date information, or result in overly defensive code, adding bloat and cost to future changes.

Clean, well-structured docs reduce ambiguity, gives AI less room to hallucinate and humans more confidence to create concise solutions.

You're not optimizing for AI, you're optimizing for accuracy. The goal isn't "AI-first" for its own sake; it's reducing friction for whoever (or whatever) is reading your docs.

Migrating to this model

If you're on Confluence, Notion, Google docs, or another external wiki:

1. Export to markdown: Most tools have markdown export. Use it. Clean it up, slice it up.
2. Split using C4 layers: Categorize each doc's sections as Context, Container, Component, or Code level.
3. Atomize by location: Move each doc to the appropriate place. High-level at root, implementation details next to the code (same folders).
4. Delete the transient: Planning docs, old specs, meeting notes should be in external systems, not code. Don't migrate these.
5. Prune aggressively: If a doc hasn't been updated in a year and the code has changed, delete it. Git history preserves it and no information is better than bad information.

When this doesn't apply

This framework is optimized for:

• Small to medium teams
• Greenfield or actively developed codebases
• Teams using AI coding assistants

It would probably need adaptation for:

• Regulated industries: Compliance may mandate historical documentation regardless of relevance
• External-facing docs: API docs, user guides have different audiences and lifecycles
• Large organizations: Cross-cutting decisions may need more formal governance
• Legacy systems: Teams spelunking 10-year-old code may need those archived records

Questions to ask before writing (or keeping) any doc

1. Is this worth the context budget?
   Will it be read often enough to justify its cost? Could this be inferred from code or tests instead?

2. What is this for?
   Providing context to AI? Optimize for that. Describing expected behavior? Maybe you need more tests instead.

3. Am I using docs as a surrogate for tests?
   If you're relying on docs to communicate what can/can't be done, consider: less documentation, more tests.

4. At what abstraction level am I documenting this?
   Match detail to scope. Implementation details go at the implementation layer only.

5. Who needs this and for how long?
   Vision → years → persistent collaborative docs.
   Epic → weeks → ticketing.
   Task → hours → disposable notes.

Context is a budget

Every token you write costs attention to read and energy to maintain.

Spend it wisely and deliberately.

Start with one question: look at your last PR. Did you update any documentation? Should you have?

That's your signal for where to start pruning or investing.

Every token you save is attention you can spend on what matters!