5/6 We traced this to layers 5-11, which integrate sequence weights blindly, regardless of whether they help or hurt. When they hurt prediction, we could progressively rescue performance by shuffling weights at earlier layers, saving the model from its own bad intuition.
