iiinova — where your digital self keeps evolving

You're welcome, and thanks for the question - it's an important one!

010

But that's not the case here: Once we fix a structural hypothesis ("role scheme"), our TPR approximation model is highly constrained, as it's bilinear. There's no guarantee that it can approximate anything, so when it *does* succeed, it's strong evidence that the hypothesized TPR structure is there

100

One more point: Suppose our approximation model were a universal function approximator, like a standard neural net. Then, our approximation wouldn't mean much, since we could approximate anything, even something that didn't use the structure we hypothesized

120

Second, once we have a TPR approximation to an LLM, we can take the LLM's representations & edit them as if they are a symbolic structure, and the edits change the LLM's behavior as intended. This editability gives more fine-grained evidence that the symbolic structure is there in the LLM

120

Why do I feel comfortable concluding Point 3? I.e., how can we conclude "equivalence" from data that merely show "approximation"? First, the way we evaluate an approximation is by seeing if it yields the same behavior as the original. This establishes a type of equivalence (behavioral equivalence)

110

Yeah, good q! The argument is: 1. A TPR is a symbolic structure 2. Thus, if a TPR is equivalent to the LLM rep, then the LLM rep has symbolic structure 3. Our results show that the TPR is equivalent to the LLM rep Point 3 is certainly debatable, but I do feel comfortable concluding it from our data

100

That said, it might well be the case that it's easier for the more recent architectures to discover TPR structure than GPT-2. It could be interesting to look at the training trajectory to see if TPR structure arises faster in some models than in others!

010

Great question! GPT-2-XL approximated about as well as the architectures in that figure. So, my intuition is that the TPR structure arises more out of top-down pressures (i.e., what the representations are used for) rather than bottom-up pressures (i.e., architectural structure).

120

Thank you for the kind words!

010

But I do think there could be something there - will keep thinking about it!

020

E.g., the embedding dimensionality is a free parameter, and your choice for that dimensionality might really influence whether you get linear independence. So the linear independence route might be too dependent on properties of the analysis, rather than the model being analyzed.

130

That said, that intuition might be hard to actually implement, because the representations of roles that we extract in our approach depend somewhat on hyperparameters.

130

Ooh I love this idea! The TPR formalism definitely makes predictions in this direction: TPRs guarantee perfect decoding only if the embeddings of roles are linearly independent, but if they're not linearly independent, you'll get intrusions.

100

Ooh, this looks excellent - will check it out! Thank you!

020

That sounds like an excellent course of action!

020

These results provide a potential explanation for how neural networks can be so successful in seemingly symbolic domains, potentially reconciling longstanding symbolic conceptions of intelligence with the vector-based nature of modern AI. 11/n

1272

In most cases, DISCOVER also generalizes out-of-distribution to filler-role combinations that it did not encounter during training. E.g., if the word "doctor" never occurred as the subject of a sentence, it can generalize to sentences with "doctor" as the subject. 10/n

Results plot showing 7 LLMs (Gemma-3, GPT-2-XL, GPT-OSS, Llama-3.1, OLMo-2, Pythia, and Qwen-3) in two domains (encoding lists or encoding sentences). The accuracy of approximating these encodings stays strong even when there are multiple unseen role-filler pairs.

1220

DISCOVER enables us to edit an LLM's internal representations in targeted ways that are then appropriately reflected in changes in the output. The edits that we make would typically be challenging to do in a vector encoding, but they're straightforward w/ a symbolic encoding 9/n

Edits to a coding prompt. The original prompt says to repeat the list [Q, E, V] two times, producing [Q, E, V, Q, E, V]. We can change the number of repetitions to 3, changing the output to [Q, E, V, Q, E, V, Q, E, V]. We can change the first letter from Q to H, changing the output to [H, E, V, H, E, V]. We can change the function from repeating the list 3 times to adding a B in front of the list, changing the output to [B, Q, E, V]. We can change the variable that is the function’s input to a list equal to [J, R, M], and the output changes to [J, R, M, J, R, M].

1252

We refer to this method as DISCOVER (short for DISsecting COmpositionality in VEctor Representations). The official mascot of DISCOVER is the tapir (pictured below), because the word TAPIR is what results from interleaving "TPR" with "AI" 8/n

An image of a tapir, which is a mammal with a long snout.

1321

Given appropriate choices for the type of symbols used by the TPR, the approximation produces high accuracy in all networks we consider: The network continues to produce the right answer when fed the TPR rather than its own actual encoding. 7/n

A results plot showing the results of approximating GPT-OSS’s representations with DISCOVER. Across all 6 tasks (arithmetic, syllogisms, code execution, passivization, tense reinflection, and question formation), GPT-OSS gets a similar accuracy when fed the TPR approximation as it gets when fed its own encodings.

1220

In the case of LLMs, this means replacing every vector that encodes part of the input, across tokens and layers. 6/n

Top: Image of normal LLM processing. The LLM receives an input and generates one vector representation for each input token at each layer. It then generates the output one word at a time, based on these representations.
Bottom: Approximating an LLM with DISCOVER. We replace every vector that encodes part of the input with a TPR approximation, and then have the LLM generate text based on these TPR vectors.

1260

We analyze a network by approximating its representations w/ a TPR. We then feed the TPR approximation into the network & see if it still produces the right answer - replacing the network's entire representation-generating process with a simple, closed-form TPR equation. 5/n

Visualization of the DISCOVER process. First we train the target model, which takes in an input, encodes it as a vector E, and then decodes from E to produce the output. We then train a DISCOVER model, which aims to approximate E with a TPR encoding, E_TPR. We then feed E_TPR into the decoder of the target model to see if it produces the correct answer when given E_TPR instead of the original E.

2251

The TPR formalism was introduced by Smolensky (1987) at the 1st NeurIPS. Shortly after, in 1990, he hypothesized that, if the field could ever create neural networks that can process language, they might do so via implicit TPR structure. Our work tests that hypothesis! 4/n

Screenshot of text from Smolensky (1990). It reads: “In the short term, at least, our learning rules and network simulators do not seem powerful enough to make network learning of linguistic representation feasible. (2) Even if such learning is feasible at some future point, we will still need to explain how the representation is done. There are two empirical reasons to believe that such explanation will require the kind of analysis begun in this paper: explanation of the computation of real neural networks has turned out to require much analysis, as mere observation has proved woefully inadequate; the same has turned out to be true even of the self-organized connectionist networks that perform computations vastly simpler than most of natural language processing

1392

Our hypothesis: The networks succeed on these tasks by implicitly building symbolic structure in vector space. What makes this hypothesis testable is the Tensor Product Representation (TPR) formalism, a proposal for how symbolic structure could be realized in vectors. 3/n

The structure of a Tensor Product Representation (TPR), which realizes symbolic structure in vector space. In this example, the symbolic structure is the sentence “poets help chefs.” TPRs work by pairing each element of the structure (called a filler - here, each word is a filler) with its role (here, “subject”, “verb”, or “object”) and translating that role-filler structure into a vector.

2478

The Emergent Symbolic Structure of Artificial Neural Networks

Paper: arxiv.org/abs/2608.29530 We consider a variety of neural networks that perform seemingly-symbolic tasks, from small-scale models trained on simple list-manipulation tasks to LLMs performing tasks in math, logic, coding, and language. 2/n

arxiv.org

Modern systems in artificial intelligence (AI) somehow excel in domains for which they seem poorly suited. Intelligence has traditionally been modeled as operating over structured combinations of symb...

2504