Coding Agents and Limits /

IV. The Cult of Coherence

Part III gives the clanker Ayahuasca. Destabilization makes forgotten or suppressed alternatives reachable again.

Now someone has to decide which of those alternatives are insight and which are nonsense.

The experienced model is biased towards what it already knows. A fresh model doesn’t understand the project. Other models and humans bring their own history and incentives. There is no neutral evaluator.

Worse, a model that writes code, decides what to remember and trains on its own conclusions can create a cult. Its world becomes more coherent while becoming less connected to reality.

GPT-5.6 Sol took the argument from memory and learning to governance:

The next thing to notice is the selection problem.

You started with:

small context → forgetting

Then:

sleep/consolidation → durable project intuition

Then:

durable intuition → contraction and ossification

Then:

destabilization → new possibilities

But now you have a harder problem:

Which of the new possibilities are insight, and which are merely incoherence?

Sleep gives persistence.

The psychedelic phase gives variation.

Neither gives selection.

The evaluator cannot be neutral

Suppose the experienced model judges the alternatives. It will tend to reject precisely those ideas that violate its ossified worldview.

Suppose a fresh model judges them. It lacks the accumulated project understanding needed to distinguish a deep simplification from a naïve rewrite.

Suppose another specialized model judges them. It has its own contraction.

Suppose humans judge them. Humans have the same bounded context, institutional habits, incentives, and delayed understanding.

So every evaluator is situated. There is no view from nowhere from which “proper architecture” can simply be recognized.

This produces a recursion:

The mistake would be to search for a final component that ends the recursion. There probably is no such component.

Architecture is a prediction, not a visible property

A deeper point follows from your earlier objection about taste.

You cannot fully determine whether an architecture is good by inspecting it at the moment it is written. Architecture is largely a wager about future change.

A boundary is good if future changes tend to remain localized around it.

An abstraction is good if future cases extend it naturally.

Duplication is justified if the supposedly similar things later diverge.

A generalization is justified if the anticipated variation actually appears.

Therefore:

“This is the right architecture” really means “this conceptual structure will continue to compress the changes this system is likely to undergo.”

But the evidence arrives months or years later.

This means the core learning problem is not memory capacity. It is long-horizon credit assignment. When a project becomes painful in 2029, which design decision from 2026 deserves blame? When an abstraction works beautifully, was that because it captured the domain correctly, or because the project happened not to encounter the cases that would break it?

The model cannot know what to consolidate merely by training on accepted pull requests. Acceptance is not the same as correctness. It may simply encode the current maintainers’ taste, schedule pressure, or institutional politics.

The system must preserve counterfactuals

Ordinary project memory records what was chosen:

We adopted architecture A.

A learning system needs more:

We chose A over B and C.

We expected A to make changes of type X easier.

We accepted increased difficulty for Y because Y was believed to be rare.

B was rejected because of assumption Z.

Reconsider B if Z stops being true.

That is counterfactual memory.

Without it, sleep reinforces the winners and erases the alternatives. The model learns history as inevitability:

A exists, therefore A must have been right.

With counterfactual memory, later experience can revise the judgment:

We expected X, but the project repeatedly encountered Y. The rejected design B may have modeled the domain better.

This is how taste could become more than imitation. The model would learn not just from what maintainers approved, but from whether their predictions survived contact with the project’s future.

For agentic coding, decision records should therefore contain predictions, not just rationales:

Decision:
Place authorization at the service boundary.

Expected benefit:
New transports should reuse identical authorization behavior.

Accepted cost:
Service methods will require explicit actor context.

Prediction:
Adding a second transport should require no authorization duplication.

Revisit when:
Transport-specific authorization rules become common.

Later, the sleep process can compare the prediction with what actually happened.

The real danger is self-confirming consolidation

A model that produces code, approves its own code, summarizes its own history, and trains on that history forms a closed epistemic loop.

Its outputs become its training data.

Its assumptions determine which evidence is retained.

Its retained evidence strengthens those assumptions.

It can become increasingly coherent while becoming less connected to reality.

That is more dangerous than ordinary forgetting. Forgetting creates visible inconsistency. Self-confirming consolidation creates an internally consistent world that explains away contrary evidence.

So no long-lived model should simultaneously control:

  1. what actions are taken;
  2. what experiences are remembered;
  3. what lessons are extracted;
  4. what evaluation criteria are used;
  5. whether its own update is accepted.

This is not primarily a neural-network design principle. It is a constitutional principle: separation of powers.

The unit of intelligence changes

At this point the single model ceases to be the correct unit.

The durable system needs at least:

The intelligence lies in the interaction among them.

This resembles science more than memory:

It also resembles constitutional government: no participant is trusted enough to exercise every cognitive function.

The next objective is not permanent coherence

You began by wanting arbitrary project scale without losing coherence. The more realistic target is different:

Do not prevent all drift. Make drift observable, contestable, and reversible.

Perfect global coherence across unlimited time may be the wrong objective. A system too strongly optimized for coherence becomes doctrinaire. A system too strongly optimized for novelty becomes fragmented.

The desired property is recoverable coherence:

There may be no final healthy equilibrium. Health may be the continuing dynamic:

commitment ↔ doubt

specialization ↔ renewal

continuity ↔ replacement

So the next insight is this

The problem is no longer “How do we give a model enough memory?”

It becomes:

How do we construct a process in which no model—including the most experienced one—gets to define reality solely through the consequences of its own learning?

Sleep writes the priors.

Destabilization expands the hypotheses.

But only plural evaluation, preserved counterfactuals, delayed consequences, and reversible selection can prevent the whole cycle from becoming self-confirming.

And the step after that is more uncomfortable:

Some valuable knowledge can be retained only by preserving disagreement rather than resolving it.

A mature intelligence may not converge to one perfectly consolidated mind. It may need to remain a managed plurality whose parts remember different possible interpretations of its history.

 Coding Agents and Limits