My interpretability research focuses on using advanced mathematics to understand how transformer neural networks, which power large language models, process information. The goal of interpretability is to de-mystify the mathematical steps inside a neural network, so we can see why it produces the answers that it does.
Here I investigated a fascinating problem with major impact on understanding and controlling AI models, called subliminal knowledge transfer, or subliminal learning. This is the well-documented phenomenon where a model can be trained using another model’s answers, an somehow inherit behaviors from the ‘teacher’ model that have no relation to the training material.
For example, a ‘student’ model could be trained using the ‘teacher’ model’s answers to math problems, but afterwards the student has also learned the teacher’s favorite animal.
I hypothesize that the explanation may lie in a fundamental property of the mathematical operations that are used in every step of the network powering a large language model. Specifically, the vectors that the network produces at each layer are often viewed as encoding representations of many concepts and features that can slightly overlap, which is called the phenomenon of superposition. The overlap between seemingly unrelated features could explain why an LLM has connections between distant concepts.
I set up an experiment to see if I could show this effect at a much simpler scale than a full LLM. In the experiment I trained two small models with only one layer, with one model having the features to exploit superposition and the other only being a linear model. I showed that the model which can use superposition in it’s one layer learned features of words completely unrelated to the training inputs, while the simpler model did not.
This contribution is a promising start to undertstanding how the basic math of LLM networks leads to surprising phenomena like subliminal learning. While certainly not a definitive proof, my contribution shows that there is a promising direction to pursue for this particular problem.