Sign in

Ethan

@ethansmith2000.com
179 followers 61 following 35 posts

a boy and his gpu vs the world. cofounder/directing research at @leonardoai_. (now at @canva) trying to feel the magic. www.ethansmith2000.com

PostsRepliesMedia
Ethan @ethansmith2000.com · 02/12/2024
Reminds me of the SolidGoldMagikarp
020
Ethan @ethansmith2000.com · 01/12/2024
distill.pub/2018/differe...
distill.pub
Differentiable Image Parameterizations
A powerful, under-explored tool for neural network visualizations and art.
020
Ethan @ethansmith2000.com · 01/12/2024
this is so cool
130
Ethan @ethansmith2000.com · 28/11/2024
it's crazy to me that RoPE's issue with BF16 wasn't noticed earlier. For a reasonable N of 2048, these are the computed frequencies prior to cos(x) & sin(x) for fp32 above and bf16 below. Given how short the period is of simple trig functions, this difference is catastrophic for large values.
181
Ethan @ethansmith2000.com · 28/11/2024
It’s an unstoppable force and all I can say is don’t hate the player (especially not the underdog) hate the game. Whether or not HF released this dataset your data is being used, you may as well also have access to its collection.
020
Ethan @ethansmith2000.com · 28/11/2024
This is the big one and I can’t stress this enough. All of your data everywhere is being gathered and used anyway by private actors. The only fire you can fight back with is to play on that same field and democratize it. This anger is way mistargeted
160
Ethan @ethansmith2000.com · 28/11/2024
140
Ethan @ethansmith2000.com · 27/11/2024
Aren’t there datasets just like this for twitter and everything else imaginable? Idk why this is suddenly taboo, most making these datasets also aren’t sharing them publicly
390
Ethan @ethansmith2000.com · 26/11/2024
🙄
000
Ethan @ethansmith2000.com · 25/11/2024
github.com/ethansmith20...
github.com
GitHub - ethansmith2000/fsdp_optimizers: supporting pytorch FSDP for optimizers
supporting pytorch FSDP for optimizers. Contribute to ethansmith2000/fsdp_optimizers development by creating an account on GitHub.
010
Ethan @ethansmith2000.com · 25/11/2024
Just added FSDP2 support for MARS and Muon!
182
Ethan @ethansmith2000.com · 25/11/2024
that's what they all say
000
Ethan @ethansmith2000.com · 25/11/2024
130
Ethan @ethansmith2000.com · 25/11/2024
Awesome list, thanks!
010
Reposted by Ethan
TimDarcet @timdarcet.bsky.social · 24/11/2024
Excellent writeup on GPU streams / CUDA memory dev-discuss.pytorch.org/t/fsdp-cudac... TLDR by default mem is proper to a stream, to share it:: - `Tensor.record_stream` -> automatic, but can be suboptimal and nondeterministic - `Stream.wait` -> manual, but precise control
2291
Ethan @ethansmith2000.com · 25/11/2024
Incredible to see what is likely SOTA results coming out of open source with full reproducibility! Happy to have helped provide the compute for this and hoping to support more awesome research like this!
0110
Reposted by Ethan
Fern @fernbear.bsky.social · 25/11/2024
First, my sincerest thanks to @leonardoai.bsky.social with the help of @ethansmith2000.com for generously providing H100s to support this research to enable this release. Y'all rock, thanks so much! <3
121
Ethan @ethansmith2000.com · 25/11/2024
Absolutely sick!
020
Reposted by Ethan
Fern @fernbear.bsky.social · 25/11/2024
New NanoGPT training speed record: 3.28 FineWeb val loss in 4.66 minutes Previous record: 5.03 minutes Changelog: - FlexAttention blocksize warmup - hyperparameter tweaks
2333
Reposted by Ethan
xjdr @xjdr.bsky.social · 24/11/2024
i trying to follow as many of my old moots as possible and new people as i find them. some of y'all changing your pfp is just mean spirited (im lazy and learned people's pfps not names)
8361
Ethan @ethansmith2000.com · 25/11/2024
Greetings xjdr
040
Reposted by Ethan
Rosmine @rosmineb.bsky.social · 25/11/2024
Untuned SOAP beats tuned adamw at ever single step
061
Ethan @ethansmith2000.com · 25/11/2024
Yes @hessianfree.bsky.social can speak more to this
020
Ethan @ethansmith2000.com · 24/11/2024
ADAM's been tuned but SOAP and PSGD just using default params, you love to see it.
191
Ethan @ethansmith2000.com · 24/11/2024
There’s a void of PSGD hype that needs to be filled here
010
Ethan @ethansmith2000.com · 24/11/2024
I goofed and never tested distributed saving, but now it works! It was a little annoying as both SOAP and psgd maintain preconds as lists of varying size, which fail to be pickled. To fix this I hardcoded there to be a max of 4 (based on conv layers being 4d tensors).
060
Ethan @ethansmith2000.com · 24/11/2024
I’ve generally preferred research to software engineering but I am growing a liking for building the tools used for research
1120
Ethan @ethansmith2000.com · 24/11/2024
Fancy seeing you here 👋
010
Ethan @ethansmith2000.com · 24/11/2024
Self-proclaimed hessianfree guy going back on his word
100
Ethan @ethansmith2000.com · 24/11/2024
Haven’t tested, but should be typical FSDP experience.
000
Reposted by Ethan
Ethan @ethansmith2000.com · 23/11/2024
Here, have PSGD-Kron and SOAP with FSDP2 support. Please go wild with it, let's see something finally replace ADAM. github.com/ethansmith20...
3135
Reposted by Ethan
Lucas Beyer (bl16) @giffmana.ai · 23/11/2024
PSA: force-pushing to GitHub does NOT actually delete/overwrite anything. Everything you squash and force-push over is still there:
2301
Ethan @ethansmith2000.com · 23/11/2024
Yes!
110
Ethan @ethansmith2000.com · 23/11/2024
credit to @_clashluke for the original PSGD-kron and SOAP implementations i used and Heavyball which i've learned a lot from
010
Ethan @ethansmith2000.com · 23/11/2024
naive implementation is okay, much needs to be done with full tensor but a few pointwise ops we can benefit from remaining in shards. future improvements likely lie in pre-fetching and grabbing chunks of params, rather than one at a time. this is fragile though and DTensor still needs more support.
110
Ethan @ethansmith2000.com · 23/11/2024
Here, have PSGD-Kron and SOAP with FSDP2 support. Please go wild with it, let's see something finally replace ADAM. github.com/ethansmith20...
3135
Ethan @ethansmith2000.com · 23/11/2024
though really you could argue quite a bunch of older score-based models, hopfield networks and others kinda fit the bill of "the diffusion before diffusion" to some degree
020
Ethan @ethansmith2000.com · 23/11/2024
from a 2012 lecture referencing a 1981 paper (and it all really goes further back than that) If ML experimentation software was as widespreadly accessible as it is now, I'd be generative diffusion models would have been discovered a while ago.
280
Ethan @ethansmith2000.com · 23/11/2024
probably the best in-depth explanation i've seen on FSDP at the most granular levels, props to the authors dev-discuss.pytorch.org/t/fsdp-cudac...
193
Ethan @ethansmith2000.com · 23/11/2024
im a simple guy i see juanposting i rt
110
Reposted by Ethan
Juan Lam @juanlam.com · 22/11/2024
🥳 You can wishlist my #MR game Galactic Traffic Control NOW on Meta Quest! 🚀☄️🕹️ It's coming soon :D GTC is about guiding spaceships from your walls to their portals by drawing paths and destroying obstacles meta.com/experiences/... Let's talk about the features... 🧵 #gamedev #vr #xr #MetaQuest
meta.com
Galactic Traffic Control on Meta Quest
Galactic Traffic Control is a mixed-reality arcade game where you must guide the hyper-jumping spaceships that need to cross the space highway (your room) in order to get to their destination!
5164
Ethan @ethansmith2000.com · 21/11/2024
lol weird I feel the opposite. The wall of text is usually more daunting yap than bite sized chunks of key points
110
Ethan @ethansmith2000.com · 21/11/2024
Besides X having a longer existing community, I can’t imagine any reason you wouldn’t choose Bluesky over X. Your experience being at the mercy of algo-makers is so tired.
350