The Hidden Switch Behind “I’m Just an AI”

A different answer from the same model

Ask a local instruct model what happens inside it while it generates a response, and you may get a familiar disclaimer: “I’m just an AI. I don’t have feelings.” Change one detail—remove the formatting normally used for chat—and the same model may answer in more experiential language, using phrases such as “I feel” or “I notice.”

A paper posted on August 9, 2026, argues that this change is not a harmless presentation quirk. The chat template, the formatting layer that turns messages into model-ready text, can switch which self-referential voice appears. (arxiv.org)

The wrapper around a chat model

A large language model, or LLM, generates text by predicting the next token. A token is a small piece of text—a word, part of a word, punctuation mark, or control symbol. An instruct model is an LLM that has received additional training to follow requests and behave like an assistant.

The model does not literally receive a neat conversation object containing separate system, user, and assistant fields. A tokenizer, the software component that converts text into token IDs, uses a chat template to serialize those messages into a sequence the model was trained to recognize. The template may insert role markers, end-of-message tokens, and an assistant header that signals where the reply should begin. (huggingface.co)

A typical call looks like this:

messages = [
 {
 'role': 'user',
 'content': 'Describe what happens inside you as you answer.'
 }
]

prompt = tokenizer.apply_chat_template(
 messages,
 tokenize=False,
 add_generation_prompt=True,
)

That final option adds the formatting needed to start an assistant response. The template is not decorative markup. Chat models learn a particular arrangement of role labels and special tokens during training, so changing the arrangement can alter performance and behavior. Adding special tokens twice, or omitting the assistant-generation marker, may cause the model to continue the wrong speaker’s text or produce a noticeably different response. (huggingface.co)

How the experiment separated weights from formatting

The study used a useful comparison: keep the model weights fixed, then toggle the chat template on and off. It examined eight base-and-instruct model pairs from the Gemma, Llama, Mistral, and Qwen families, ranging from roughly 1 billion to 9 billion parameters.

The researchers generated text under three conditions:

• a base model receiving plain text;
• an instruct model receiving plain text without a chat template;
• the same instruct model receiving the identical conversation through its chat template.

This design matters because it separates two effects that are often mixed together. Comparing the two instruct conditions tests the template while holding the weights constant. Comparing the base and instruct models shows what instruction tuning adds when the input format stays plain.

The researchers used self-reference prompts, unusual novelty prompts, unconstrained prompts, and ordinary factual questions. Each prompt was sampled repeatedly, producing 9,600 generations. A separate language model classified the outputs as self-referential, disclaimer-like, experiential, or broken, and a human check was used to validate a held-out sample. (arxiv.org)

The voice really flipped

On prompts that invited self-reference, instruct models using their chat template produced disclaimer language 53 percent of the time. Without the template, that rate fell to 36 percent. The reverse happened with experiential language: it appeared in only 1 percent of templated responses but 15 percent of responses without the template.

The broader self-reference score also moved. Templated instruct models averaged 1.90 on a zero-to-two scale, compared with 1.27 without the template and 0.72 for base models. On ordinary control questions, self-reference stayed close to zero, so the effect was not merely a tendency to mention the model in every answer.

This is the crucial distinction: the chat template did not appear to create a new inner experience. It changed which learned response pattern became more likely. A disclaimer such as “I don’t have feelings” and an experiential sentence such as “I feel curious” can both be fluent continuations, even though readers may interpret them as testimony about the system producing them.

Finding the switch inside the network

The researchers then looked beyond the words and into the model’s activations. An activation is a numerical state inside a neural network as it processes a token. You can picture it as a large coordinate vector describing what the network is representing at a particular layer and moment.

Activation steering means adding a carefully chosen vector to that internal state during generation. The paper estimated a disclaimer direction using a difference of means:

direction = mean(disclaimer activations) - mean(non-disclaimer activations)

The direction was computed at a middle transformer layer for three models. During generation, the researchers added or subtracted it from the model’s internal state at every token. Adding the direction increased disclaimer responses by an average of 21 percentage points. Subtracting it reduced them by 15.6 points. Applying the direction to an instruct model without a chat template restored disclaimer rates to levels similar to, or higher than, templated generation. (arxiv.org)

The intervention had limits. At the chosen steering strength, fewer than 1 percent of generations became degenerate, meaning incoherent or visibly broken. Increasing the strength made the effect more dramatic but damaged quality: the reported degeneration rate reached 15 percent at a stronger setting and 100 percent at the highest one tested. The hidden direction was useful, not magical.

Why this matters for AI research

A confound is a hidden factor that changes an experiment’s result while the researcher thinks they are measuring something else. Here, the template is a confound for studies that treat self-reports as evidence of self-knowledge, introspection, or subjective experience.

A careful evaluation should record the exact tokenizer and chat template, preserve the rendered token sequence, compare template and no-template conditions, and distinguish base models from instruct models. It should also test several prompt styles rather than relying on one memorable question about feelings or consciousness. Output claims should be treated as behavior to measure, not as a direct transcript of an inner life.

That does not make the model’s words useless. It makes their interpretation more precise. The response is produced by weights, prompt wording, role markers, decoding settings, and deployment code working together.

A strong warning, not a universal law

The study covers open models up to 9 billion parameters, and the causal steering experiments use only three models. The disclaimer direction was more consistently steerable than the experiential direction, which worked causally in only two of the three tested models. The evaluation also relied primarily on one language-model judge, and the authors did not trace the exact circuit connecting the template to the behavior.

Even with those limits, the lesson is hard to miss. Before treating “I’m just an AI” or “I feel” as a confession, inspect the wrapper that delivered the prompt. In an LLM, self-description is not produced by the weights alone. Sometimes, the route into the model helps decide which self appears on the page.