iiinova — where your digital self keeps evolving

Ethan @ethansmith2000.com · 28/11/2024

it's crazy to me that RoPE's issue with BF16 wasn't noticed earlier. For a reasonable N of 2048, these are the computed frequencies prior to cos(x) & sin(x) for fp32 above and bf16 below. Given how short the period is of simple trig functions, this difference is catastrophic for large values.

181

Ethan @ethansmith2000.com · 28/11/2024

This is the big one and I can’t stress this enough. All of your data everywhere is being gathered and used anyway by private actors. The only fire you can fight back with is to play on that same field and democratize it. This anger is way mistargeted

160

Ethan @ethansmith2000.com · 25/11/2024

Just added FSDP2 support for MARS and Muon!

182

Reposted by Ethan

TimDarcet @timdarcet.bsky.social · 24/11/2024

Excellent writeup on GPU streams / CUDA memory dev-discuss.pytorch.org/t/fsdp-cudac... TLDR by default mem is proper to a stream, to share it:: - `Tensor.record_stream` -> automatic, but can be suboptimal and nondeterministic - `Stream.wait` -> manual, but precise control

2291

Ethan @ethansmith2000.com · 25/11/2024

Incredible to see what is likely SOTA results coming out of open source with full reproducibility! Happy to have helped provide the compute for this and hoping to support more awesome research like this!

0110

Reposted by Ethan

Fern @fernbear.bsky.social · 25/11/2024

First, my sincerest thanks to @leonardoai.bsky.social with the help of @ethansmith2000.com for generously providing H100s to support this research to enable this release. Y'all rock, thanks so much! <3

121

Reposted by Ethan

Fern @fernbear.bsky.social · 25/11/2024

New NanoGPT training speed record: 3.28 FineWeb val loss in 4.66 minutes Previous record: 5.03 minutes Changelog: - FlexAttention blocksize warmup - hyperparameter tweaks

2333

Reposted by Ethan

xjdr @xjdr.bsky.social · 24/11/2024

i trying to follow as many of my old moots as possible and new people as i find them. some of y'all changing your pfp is just mean spirited (im lazy and learned people's pfps not names)

8361

Reposted by Ethan

Rosmine @rosmineb.bsky.social · 25/11/2024

Untuned SOAP beats tuned adamw at ever single step

061

Ethan @ethansmith2000.com · 24/11/2024

ADAM's been tuned but SOAP and PSGD just using default params, you love to see it.

191

Ethan @ethansmith2000.com · 24/11/2024

I goofed and never tested distributed saving, but now it works! It was a little annoying as both SOAP and psgd maintain preconds as lists of varying size, which fail to be pickled. To fix this I hardcoded there to be a max of 4 (based on conv layers being 4d tensors).

060

Ethan @ethansmith2000.com · 24/11/2024

I’ve generally preferred research to software engineering but I am growing a liking for building the tools used for research

1120

Reposted by Ethan

Ethan @ethansmith2000.com · 23/11/2024

Here, have PSGD-Kron and SOAP with FSDP2 support. Please go wild with it, let's see something finally replace ADAM. github.com/ethansmith20...

3135

Reposted by Ethan

Lucas Beyer (bl16) @giffmana.ai · 23/11/2024

PSA: force-pushing to GitHub does NOT actually delete/overwrite anything. Everything you squash and force-push over is still there:

2301

Ethan @ethansmith2000.com · 23/11/2024

Here, have PSGD-Kron and SOAP with FSDP2 support. Please go wild with it, let's see something finally replace ADAM. github.com/ethansmith20...

3135

Ethan @ethansmith2000.com · 23/11/2024

from a 2012 lecture referencing a 1981 paper (and it all really goes further back than that) If ML experimentation software was as widespreadly accessible as it is now, I'd be generative diffusion models would have been discovered a while ago.

280

Ethan @ethansmith2000.com · 23/11/2024

probably the best in-depth explanation i've seen on FSDP at the most granular levels, props to the authors dev-discuss.pytorch.org/t/fsdp-cudac...

193

Reposted by Ethan

Juan Lam @juanlam.com · 22/11/2024

🥳 You can wishlist my #MR game Galactic Traffic Control NOW on Meta Quest! 🚀☄️🕹️ It's coming soon :D GTC is about guiding spaceships from your walls to their portals by drawing paths and destroying obstacles meta.com/experiences/... Let's talk about the features... 🧵 #gamedev #vr #xr #MetaQuest

meta.com

Galactic Traffic Control on Meta Quest

Galactic Traffic Control is a mixed-reality arcade game where you must guide the hyper-jumping spaceships that need to cross the space highway (your room) in order to get to their destination!

5164

Ethan @ethansmith2000.com · 21/11/2024

Besides X having a longer existing community, I can’t imagine any reason you wouldn’t choose Bluesky over X. Your experience being at the mercy of algo-makers is so tired.

350