I really enjoy finding gaps in our understanding of text-book deep learning.
The RNN Homunculus: Long-range input vanishes RNN gradients. Gates (GRU, LSTM) help, but gates are themselves an RNN. Ie: gates are not sufficient. We need short-range data to learn gates first.
arxiv.org/abs/2610.14279
