Order Matters : Sequence to Sequence for Sets

This research paper examines the importance of data ordering in sequence-to-sequence (seq2seq) models, specifically for tasks involving sets as inputs or outputs. The authors demonstrate that, despite the flexibility of the chain rule in modelling joint probabilities, the order in which data is presented to the model can significantly affect performance. They propose two key contributions: an architecture called “Read-Process-and-Write” to handle input sets and a training algorithm that explores various output orderings during training to find the optimal one. Through a series of experiments on tasks such as sorting, language modelling, and parsing, the authors provide compelling evidence for the impact of ordering on the effectiveness of seq2seq models.

Audio : (Spotify) https://open.spotify.com/episode/3DAkHJxQ204jYvG89dO7sm?si=jhugL6y5RSmwgqJxeTstWg

Paper: https://arxiv.org/pdf/1511.06391