How Neural Networks Hide Symbolic Structure Inside Vectors

Two sentences can contain exactly the same words and still describe opposite events: cats chase dogs and dogs chase cats. A bag of words—the collection of words without their positions—would see the same ingredients in both. Human reasoning depends on the arrangement. Yet a neural network, a learned system of layers that transforms numbers into other numbers, stores its working information as continuous vectors rather than explicit trees, rules, or database entries. The question researchers are chasing is: how can a neural network represent symbolic structure without storing literal symbols? (arxiv.org)

The problem hidden inside a vector

An internal vector, also called a hidden state, is a long list of numbers produced while a model processes an input. Those numbers can encode useful concepts, but the numbers themselves rarely look meaningful to us. A coordinate such as 0.42 does not announce that it represents a subject, a noun, or the third item in a list.

Adding concept-like information together creates another problem. If a model represented cats chase dogs as the sum of vectors for cats, chase, and dogs, then changing the word order would not change the sum. The representation would know what appeared, but not who did what to whom. The same difficulty appears in arithmetic, computer code, and logic, where swapping two elements can change the answer completely.

Give every item a role

Tensor Product Representations, or TPRs, offer a way to preserve that structure. A filler is the thing being represented, such as cats or dogs. A role describes where that thing belongs, such as subject, verb, or object. The representation binds each filler to its role instead of storing the filler alone.

A simplified TPR-style equation looks like this:

E = W(
 f(cats) ⊗ r(subject)
 + f(chase) ⊗ r(verb)
 + f(dogs) ⊗ r(object)
 ) + b

Here, f produces a filler vector and r produces a role vector. The symbol ⊗ is a tensor product, an operation that combines two vectors into a structured table of pairwise interactions. The matrices for all role-filler pairs are added together, then W reshapes the result and b adds a learned offset.

The important detail is that cats can appear with different roles without requiring a separate atomic concept for every possible combination. The same filler can be placed in the subject role, object role, or another position while preserving the relationship between identity and position. TPRs therefore make a continuous vector behave more like a structured record. (arxiv.org)

DISCOVER tests more than visual resemblance

The study uses a method called DISCOVER, short for Dissecting Compositionality in Vector Representations. Its strategy is easier to understand as a replacement test.

First, researchers run a target neural network and collect its internal representations. Then they train a separate analysis model whose architecture is explicitly built from TPRs. This analysis model learns filler vectors, role vectors, and the transformation parameters needed to imitate the target representations. Training minimizes mean squared error, the average squared difference between corresponding numbers.

The decisive step comes afterward. The researchers feed the TPR-based representations into the original network’s decoder, the component that turns an internal representation into an output. If the decoder still produces the correct answer, the TPR approximation has captured something functionally important, not merely drawn a similar-looking cloud of points. DISCOVER produces a compact, interpretable equation for the representation, but it does not claim that the original network literally computes tensor products during its forward pass.

Small networks reveal the pattern first

The experiments began with manageable sequence tasks: copy a list of letters, reverse it, or interleave letters from its two ends. The researchers trained several architectures, including a multi-layer perceptron, a feed-forward network; a gated recurrent unit, or GRU, which maintains information across a sequence; and Transformer models, which use attention to let one token weigh information from other tokens.

DISCOVER tried several possible role schemes. Some roles described positions from left to right, others from right to left, and a bidirectional scheme recorded both. A bag-of-words scheme gave every letter the same role, acting as a test for structure-free representations.

The results were striking. For GRUs trained to reverse lists, a bidirectional TPR approximation reached at least 99.98% output accuracy across ten reruns, while the bag-of-words version performed poorly. Across the broader set of architectures and tasks, the lowest average accuracy for the bidirectional approximation was 0.973; every other combination averaged above 0.99. The role scheme changed with the task, but structured role-filler representations kept appearing.

What happens inside a language model?

The researchers then examined seven open-weight large language models, or LLMs. Open-weight means the learned parameters are publicly available, which is essential when the goal is to inspect internal vectors. Because a left-to-right Transformer creates one vector for every token, the study focused on the vector associated with the period at the end of a list or sentence. By that position, the model has seen the entire preceding context.

A small decoder was trained to reconstruct the list or sentence from the period vector. It recovered subject-verb-object sentences perfectly in the reported conditions, usually reconstructed lists with accuracy above 85%, and handled more complex sentences above 50% under an exact-match metric. DISCOVER then showed that bidirectional role schemes could reproduce much of this information. In complex sentences, linear position often explained the vectors better than syntactic roles, suggesting that the representations preserved order more reliably than a complete human-style parse.

A deeper test used GPT-OSS on arithmetic, syllogisms, Python list manipulation, passivization, tense changes, and question formation. Here, DISCOVER approximated representations across all input tokens and layers rather than studying one punctuation vector. A task-specific role scheme that allowed each token to encode itself and earlier context came close to the model’s original performance, with the largest reported gap only 2.36 percentage points.

Editing a thought by editing a vector

A good representation should support controlled changes. In a TPR, a role-filler component can be removed and replaced:

edited = original
 - TPR(old_role, filler)
 + TPR(new_role, filler)

This is a causal intervention: researchers change an internal state and observe whether the model’s behavior changes in the predicted direction. Moving an adjective from the object of a sentence to the subject, or swapping two numbers in an arithmetic expression, should alter the answer accordingly.

Across 31 intervention types on GPT-OSS, the average accuracy was 0.903. Edits to a word’s identity often worked when applied locally, but edits to structural roles worked much better when changes were propagated across several token representations. That pattern suggests token identity can be relatively localized, while relationships and structure are distributed throughout the model. (arxiv.org)

A promising result, not a final verdict

The evidence is strongest when DISCOVER generalizes to role-filler combinations it never saw during its own training. For example, the analysis can be trained without seeing a particular letter in a particular position, then tested on that missing combination. In almost all of the reported settings, it performed well above a baseline that would be expected from guessing. That behavior is difficult to explain unless the representation combines roles and fillers systematically rather than memorizing every pairing independently.

There are limits. DISCOVER uses human-designed hypotheses about which roles matter, so it is not a completely unsupervised discovery process. A successful approximation also does not prove that the target network contains an exact symbolic program, and the method explains the geometry of the representations more clearly than it explains how training created that geometry. The reported accuracies are high but not perfect.

That middle ground is the interesting part. Neural networks may not contain neat symbolic data structures waiting to be extracted, yet their vector spaces can organize information in symbol-like ways: an item occupies a role, relationships survive across transformations, and carefully chosen edits produce predictable behavior. Symbolic structure does not have to arrive as a separate parser bolted onto a neural model. It may emerge as the model learns how to arrange numbers so that structure can live inside them.