Learning without training: The implicit dynamics of in-context learning

In this episode, we discuss Learning without training: The implicit dynamics of in-context learning by Benoit Dherin, Michael Munn, Hanna Mazzawi, Michael Wunder, Javier Gonzalvo. The paper investigates how Large Language Models (LLMs) can learn new patterns during inference without weight updates, a phenomenon called in-context learning. It proposes that the interaction between self-attention and MLP layers in transformer blocks enables implicit, context-dependent weight modifications. Through theoretical analysis and experiments, the authors show that this mechanism effectively produces low-rank weight updates, explaining the model's ability to learn from prompts alone.