dan @danabra.mov · 24/06/2025what is the main reason that gradient descent (at least the vanilla version) uses a proportional step to the gradient value? i understand why it needs to use the sign (for direction) but why is being proportional a good idea? 6131
nic 🔹 @nmk.wtf · 24/06/2025The reason the vanilla version uses it is because it's the simplest version to write down / derive / analyze. 010
Pyotr @pyotr.bsky.social · 24/06/2025I think its cause we dont know the scale of our space, so we adjust the step size in proportion to the gradient 120
dan @danabra.mov · 24/06/2025this video is looking pretty great as an overview of the area. love the visualsyoutube.comWho's Adam and What's He Optimizing? | Deep Dive into Optimizers for Machine Learning!YouTube video by Sourish Kundu 1121
absolute horses @jcoglan.com · 24/06/2025I think it's so that it converges, c.f. the newton raphson method 020
Kris 🧙♂️ Kowal @kriskowal.com · 24/06/2025I’m guessing from this behavior emerges a tendency to approach the local minimum with an instantaneous speed that resembles a logistic function, which is to say logarithmic steps to the bottom rather than linear at some arbitrary resolution that might be too small or too great. 230
