Coding Agents and Limits /

VI. Anthropomorphize Operationally, Not Ontologically

• July 19, 2026
• AI  
• software-engineering  

Part V ended with a rule: use anthropomorphism to generate hypotheses, never as an explanation.

That seemed too strong.

Anthropomorphism can mislead us about mechanism and experience, but it is also how we predict other opaque minds. We don’t observe another human’s qualia. We infer them. We do something similar, with less confidence, for animals.

LLMs are a strange inversion of that problem. They don’t share our biology, but they are extremely good at producing the language that activates our social intuition. Tulpas make the boundary stranger still: a culturally transmitted way of thinking can produce a genuine experience of another agent without implying a separate person in the brain.

I pushed GPT-5.6 Sol on whether anthropomorphism could be a model rather than an explanation. It came back with a better rule:

I think that is the more mature version of Szubartowski’s advice:

Do not confuse anthropomorphism with anatomical or phenomenal equivalence. But do not discard it as a predictive instrument.

A purely mechanistic description is often the wrong level of abstraction. Nobody predicts a person by simulating neurons, a dog by simulating its endocrine system, or a chess engine by tracing transistor states. We introduce latent variables—belief, attention, expectation, intention, fear, confusion—because they compress enormous amounts of hidden state into something usable.

That is close to Dennett’s “intentional stance”: sometimes the most effective way to predict a system is to treat it as if it has beliefs and goals, without first settling what those words ultimately refer to.

Anthropomorphism as a reduced-order model

Suppose a system has an inaccessible internal state, and we want to predict its behavior. A mechanistic model tries to predict directly from that state, which may be impossible in practice. A mentalistic model invents a much smaller state:

internal state → beliefs, goals, attention, uncertainty → predicted behavior

The variables in that smaller state need not be made of the same thing in every system. “Attention” in a human, an animal, and a transformer need not denote one mechanism. The term is useful when it identifies a sufficiently stable behavioral regularity.

The important question is therefore not:

Does the LLM really have attention, beliefs, or confusion?

It is:

Under what conditions does attributing attention, belief, or confusion improve our predictions, and where does the attribution break?

That makes anthropomorphism empirical rather than metaphysical.

The same word may refer to different functional structures

A useful discipline is to separate an operational attribution from the extra human implications that tend to accompany it:

The anthropomorphic term may still be the best shorthand. The mistake is letting all the connotations travel with it automatically.

When we say a coding agent “forgot” a decision, that can be excellent predictive language. It suggests that repeating or retrieving the decision may restore the behavior. But mechanistically, several very different things may have happened: the information left the context, was omitted during compaction, was retrieved but given insufficient weight, conflicted with newer instructions, or remained represented without affecting the generated continuation.

“Forgot” groups these together at the behavioral level. Engineering the system eventually requires separating them.

Other humans, animals, and LLMs provide different evidence

We never directly observe another human’s qualia either. We infer them from behavior, reports, shared biology, developmental similarity, and the fact that we appear to be built from roughly the same kind of machinery.

For animals, the inference is more uncertain and species-dependent, but we still have biological continuity, homologous systems, physiology, ecological behavior, and responses to injury or reward. The danger is not attributing any mental states to animals; it is assuming that the structure and significance of those states are specifically human. A dog’s social attachment may be real without being a furry version of human friendship, marriage, guilt, or moral obligation.

LLMs create almost the reverse evidential situation. Their biological and developmental continuity with us is absent, but their linguistic surface has been optimized using records of human expression. They are consequently very good at producing the behavioral cues that activate our social cognition.

That makes anthropomorphism toward LLMs both unusually useful and unusually hazardous:

• useful because the conversational interface really does respond to reasons, roles, corrections, distinctions, and social instructions;
• hazardous because human-like self-description is among the behaviors the system has learned to generate.

An LLM saying “I am afraid” is therefore not evidentially equivalent to an animal exhibiting physiological avoidance or a human reporting fear. It may still be relevant evidence in some broader theory of machine consciousness, but it is not a privileged readout of an internal affective state. The self-report is generated through the same token-production process as a poem, legal argument, or fictional monologue.

That observation does not prove that an LLM has no experience. It means that the inferential path from verbal report to experience is much less secure than our social machinery makes it feel.

The self and tulpas make the issue more interesting

The tulpa example suggests that our agent-modeling machinery can be applied inward as well as outward. A person can interpret some internally generated thoughts, imagery, impulses, or dialogue as belonging to another center of agency. Repeated attention and rehearsal may then make that organization more stable and autonomous-seeming.

The possibilities you mention are not mutually exclusive:

• the practice may be culturally transmitted;
• expectation may shape the reports;
• the reported phenomenology may nevertheless be real;
• the phenomenology need not imply a literally separate person or brain.

A contagious cultural practice can cause genuine experiences. “Socially constructed” does not mean “not experienced.”

There is also a subtle problem with saying consciousness is merely a “story the brain tells itself.” Who is the audience for the story? That phrasing can accidentally reintroduce a little observer inside the brain.

A less circular formulation would be that the brain constructs a compressed self-model and makes some of its contents available to memory, decision-making, report, and control. Whether that self-model exhausts phenomenal consciousness is a further question. The model might not simply describe consciousness after the fact; it could be part of the process constituting the kind of consciousness we have.

Anthropomorphism can be an intervention, not just an observation

This is especially important for LLMs.

When you tell an LLM:

You are a meticulous senior engineer. Reconsider the problem from first principles.

you are not merely describing a pre-existing personality. You are providing input that selects and organizes behavior associated with that role.

Anthropomorphic language acts as a high-level programming interface. Words such as “remember,” “reflect,” “be skeptical,” “take responsibility,” and “imagine you are the maintainer” may not invoke human mental operations, but they can still cause predictable changes in output.

So the intentional stance can become self-confirming:

1. We frame the system as an agent with a role.
2. The framing changes its behavior.
3. It behaves more consistently with that role.
4. We take the behavior as evidence that the role already existed internally.

Something related may occur in human social roles, therapeutic practices, and possibly tulpa cultivation, although the mechanisms are obviously not identical. Treating a complex system as an agent can help organize it into more agent-like behavior.

That means anthropomorphic models are sometimes control models, not merely descriptive models.

A practical discipline: critical anthropomorphism

The right alternative to naïve anthropomorphism is not naïve mechanism. It is to use social intuition to generate hypotheses, while keeping a translation layer back to observable behavior and possible mechanisms.

For any mentalistic attribution, ask four different questions:

1. Behavioral: What does this attribution predict that I can observe?
2. Counterfactual: Does the pattern survive paraphrasing, interruption, changed incentives, and unfamiliar situations?
3. Mechanistic: What available state or process could produce the behavior?
4. Phenomenal or moral: Am I making a claim about experience or moral status that the behavioral result alone does not establish?

For example, to say that an LLM “believes P” in a meaningful functional sense, one might expect P to act as a stable premise across paraphrases, indirect questions, planning tasks, challenges, and relevant counterfactuals. If the supposed belief disappears under trivial prompt changes, “locally conditioned on P” is probably the better description.

Similarly, saying a coding agent “cares about architectural coherence” should predict more than eloquent prose about architecture. It should cause the agent to reject locally convenient changes, notice conceptual duplication, preserve boundaries under pressure, and behave consistently when the architectural principle is not explicitly mentioned.

The rule I would retain

Your teacher’s prohibition is valuable as an antidote to reification. Your amendment preserves the useful cognitive tool:

Use the mind-model to predict behavior, but do not assume that its internal nouns refer to human-like mechanisms or experiences.

Or more compactly:

Anthropomorphize operationally, not ontologically.

Our social cognition may be one of the best instruments we possess for dealing with opaque adaptive systems. The fact that it sometimes hallucinates minds does not make it useless. It means that, like every powerful model, it needs calibration.

The most interesting consequence is that anthropomorphic language can be predictively useful, causally effective, and metaphysically misleading at the same time.