On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
This paper argues that the gating mechanism in modern recurrent networks explains both the successes and failures of State Space Models (SSMs) on in-context learning and retrieval tasks: theory and experiments show gating causes SSMs to first learn an in-weights "memorization" solution while delaying or preventing convergence to a correct in-context learning solution, even when the architecture and memory capacity impose no fundamental limits. The authors also find that gating is often beneficial for improving generalization to long sequence lengths.