Happy AISTATS, to those who celebrate! We're celebrating a long-coming paper in gradient-based optimization that we call “Moonwalk🕺: Inverse-Forward Differentiation”.
indylab.org/pub/Krylov20...
🧵/5
indylab.org
Moonwalk: Inverse-Forward Differentiation
Backpropagation’s main limitation is its need to store intermediate activations, or residuals, during the forward pass, which restricts the depth of trainable networks. This raises a fundamental quest...