iiinova — where your digital self keeps evolving

Ethan

@ethansmith2000.com

a boy and his gpu vs the world. cofounder/directing research at @leonardoai_. (now at @canva) trying to feel the magic. www.ethansmith2000.com

Posts Replies Media

Ethan @ethansmith2000.com · 01/12/2024

this is so cool

130

Ethan @ethansmith2000.com · 28/11/2024

it's crazy to me that RoPE's issue with BF16 wasn't noticed earlier. For a reasonable N of 2048, these are the computed frequencies prior to cos(x) & sin(x) for fp32 above and bf16 below. Given how short the period is of simple trig functions, this difference is catastrophic for large values.

181

Ethan @ethansmith2000.com · 28/11/2024

140

Ethan @ethansmith2000.com · 25/11/2024

Just added FSDP2 support for MARS and Muon!

182

Ethan @ethansmith2000.com · 25/11/2024

130

Ethan @ethansmith2000.com · 24/11/2024

ADAM's been tuned but SOAP and PSGD just using default params, you love to see it.

191

Ethan @ethansmith2000.com · 23/11/2024

Here, have PSGD-Kron and SOAP with FSDP2 support. Please go wild with it, let's see something finally replace ADAM. github.com/ethansmith20...

3135

Ethan @ethansmith2000.com · 23/11/2024

from a 2012 lecture referencing a 1981 paper (and it all really goes further back than that) If ML experimentation software was as widespreadly accessible as it is now, I'd be generative diffusion models would have been discovered a while ago.

280

Ethan @ethansmith2000.com · 23/11/2024

probably the best in-depth explanation i've seen on FSDP at the most granular levels, props to the authors dev-discuss.pytorch.org/t/fsdp-cudac...

193