On the generalization of language models from in-context learning and finetuning: a controlled study

In this episode, we discuss On the generalization of language models from in-context learning and finetuning: a controlled study by Andrew K. Lampinen, Arslan Chaudhry, Stephanie C. Y. Chan, Cody Wild, Diane Wan, Alex Ku, Jörg Bornschein, Razvan Pascanu, Murray Shanahan, James L. McClelland. The paper compares the generalization and deductive reasoning abilities of large language models when learning through fine-tuning versus in-context learning, finding that in-context learning generally enables more flexible generalization. It introduces novel datasets to rigorously test these differences by isolating new factual information from pretraining knowledge. Additionally, the authors propose enhancing fine-tuning by including in-context reasoning traces, which improves the models' reasoning and generalization performance across multiple benchmarks.