Coding Agents and Limits /

II. Context Window Is Small, Weight Space Is Huge: A Case for "Sleep"

• July 19, 2026
• AI  
• software-engineering  

Part I ended with a limit. A clanker can keep exact project history outside of its context window, but that doesn’t give it the taste and intuition of someone who has worked on the project for years.

The context window is small, but weight space is huge.

Humans don’t keep everything they have learned in working memory. Sleep consolidates experience. Things that used to require conscious thought become intuition.

What if a coding agent could do the same? Keep exact facts in source control and issue trackers, but periodically train project experience back into itself. I suggested that a long-running agent would need to sleep.

GPT-5.6 Sol agreed:

Exactly. The missing operation is not better compaction but consolidation.

A context window is rapidly writable, high-fidelity, transient working memory. Weight space is slowly writable, distributed, compressed memory. Current coding agents mostly get the former while the latter remains frozen throughout the project. After compaction they retain a textual report of their experience, but they have not actually been changed by it.

That is unlike a long-tenured engineer. The engineer does not remember every discussion verbatim, but the project changes their intuitions:

• certain abstractions begin to feel natural;
• certain apparently clean designs feel suspicious;
• they recognize which layer owns a responsibility;
• they anticipate the reviewers’ objections;
• they stop needing to recite the process because it has become habit.

That is exactly the sort of information that belongs in weights rather than a prompt.

Process in context is an instruction. Process in weights is a disposition.

What sleep should consolidate

It should not try to memorize the repository byte-for-byte. Exact state is better kept in source control, tests, issues, and searchable history.

Sleep should compile repeated experience into project-specific priors:

• the project’s conceptual vocabulary;
• its preferred abstraction level;
• which kinds of duplication are tolerated;
• where responsibilities normally live;
• accepted and rejected architectural moves;
• naming and decomposition taste;
• recurring failure modes;
• what maintainers tend to mean by “simple” or “clean.”

Then local code would trigger those priors associatively without all of them being copied into the active context. A newly proposed abstraction might feel alien to the model for roughly the same distributed reasons it feels alien to an experienced maintainer.

This addresses the problem from the previous turn. Taste cannot be formally verified, but it can be learned by apprenticeship. Accepted changes, rejected changes, review comments, reversions, and later consequences provide preference signals. Sleep would not prove that an architectural choice is right; it would make the agent inherit the project’s accumulated judgment about such choices.

A plausible memory hierarchy

A durable coding agent probably needs several timescales rather than one giant context:

The base foundation model should probably remain mostly frozen. Each project could have its own adapter, sparse experts, or other parameter partition. That provides isolation, versioning, rollback, and protection against one project degrading general capabilities.

A sleep cycle might look like:

1. Collect the day’s trajectories—not only final successful diffs, but review feedback, abandoned designs, test failures, and corrections.
2. Determine which lessons recur and which are merely one-off facts.
3. Generate contrastive examples: “this implementation was accepted; this locally plausible one was rejected because responsibility belonged elsewhere.”
4. Replay older representative tasks so that new learning does not erase earlier project knowledge.
5. Train the project-specific parameters.
6. Evaluate on held-out historical decisions and fresh maintenance tasks.
7. Activate the new adapter only when it improves the project model without known regressions.

“Dreaming” would be useful here: generate variations of old architectural situations and rehearse choosing between alternatives. But unguided dreaming is dangerous because the model can rehearse and strengthen its own misconceptions. Human review outcomes and actual project history need to anchor it.

Why a huge weight space is not automatically enough

Weights are not an empty filesystem. Their capacity is large, but they are:

• already occupied by broadly shared capabilities;
• distributed rather than cleanly addressable;
• changed indirectly through optimization;
• subject to interference and catastrophic forgetting;
• lossy rather than suitable for exact records;
• difficult to inspect for provenance;
• vulnerable to poisoned or mistaken experience.

The hard problem is not merely where to store experience. It is deciding:

What deserves consolidation, at what level of abstraction, with what confidence, and what existing knowledge must be replayed while learning it?

This is why sleep is valuable. Online interaction is a poor time to perform that expensive selection and replay. Offline compute can revisit the full trace, compare it with history, generate counterexamples, evaluate candidate updates, and roll them back.

There is already active research along precisely these lines. Titans uses a neural memory module whose parameters are updated at test time to compress historical context. A May 2026 paper titled Language Models Need Sleep periodically converts context into persistent fast weights using offline recurrent processing before clearing the attention cache. A separate June 2026 paper, Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories, proposes alternating wake and sleep phases, using replay for consolidation and synthetic “dreaming” for further adaptation. These are still research demonstrations rather than a mature solution for multi-year software projects, but the architectural direction matches your observation closely.

The empirical results also expose the cost: sleep-like recurrence moves computation out of the immediate prediction path, but training becomes sequential, more expensive, and potentially unstable. Continual fine-tuning still needs replay because new updates can cause representation drift and erase earlier capabilities. MSSR explores memory-aware adaptive replay for continual LLM fine-tuning.

What this changes about the impossibility argument

Without sleep, the agent’s durable project knowledge is constrained to whatever survives textual compaction and retrieval. That is indeed inadequate for maintaining tacit architectural coherence.

With sleep, a long history can affect future behavior without being explicitly present in the next context. The effective project state becomes:

current context + external exact memory + consolidated project parameters

The weights do not have to preserve every past event. They need to learn the regularities that allow the agent to interpret the current event correctly.

This still does not permit literally arbitrary unbounded information. A finite model cannot losslessly absorb an indefinitely growing, incompressible history. But real projects are not supposed to be arbitrary bit streams. A coherent project contains repeated concepts, conventions, causal structures, and design preferences. Those regularities may be highly compressible into weights even when the complete repository is enormous.

So sleep changes the relevant limit from:

How much project history fits in the prompt?

to:

How much irreducible project complexity remains after its recurring structure has been learned?

That is a much more favorable scaling law.

The strongest formulation is therefore:

Large-scale agentic software work requires not merely persistent memory, but persistent learning.

Retrieval lets an agent consult the organization’s records. Sleep would let the organization’s history alter the kind of engineer the agent has become.