Ethan @ethansmith2000.com · 01/12/2024distill.pub/2018/differe...distill.pubDifferentiable Image ParameterizationsA powerful, under-explored tool for neural network visualizations and art. 020
Ethan @ethansmith2000.com · 28/11/2024it's crazy to me that RoPE's issue with BF16 wasn't noticed earlier. For a reasonable N of 2048, these are the computed frequencies prior to cos(x) & sin(x) for fp32 above and bf16 below. Given how short the period is of simple trig functions, this difference is catastrophic for large values. 181
Ethan @ethansmith2000.com · 28/11/2024 It’s an unstoppable force and all I can say is don’t hate the player (especially not the underdog) hate the game. Whether or not HF released this dataset your data is being used, you may as well also have access to its collection. 020
Ethan @ethansmith2000.com · 28/11/2024This is the big one and I can’t stress this enough. All of your data everywhere is being gathered and used anyway by private actors. The only fire you can fight back with is to play on that same field and democratize it. This anger is way mistargeted 160
Ethan @ethansmith2000.com · 27/11/2024Aren’t there datasets just like this for twitter and everything else imaginable? Idk why this is suddenly taboo, most making these datasets also aren’t sharing them publicly 390
Ethan @ethansmith2000.com · 25/11/2024github.com/ethansmith20...github.comGitHub - ethansmith2000/fsdp_optimizers: supporting pytorch FSDP for optimizerssupporting pytorch FSDP for optimizers. Contribute to ethansmith2000/fsdp_optimizers development by creating an account on GitHub. 010
Reposted by EthanTimDarcet @timdarcet.bsky.social · 24/11/2024Excellent writeup on GPU streams / CUDA memory dev-discuss.pytorch.org/t/fsdp-cudac... TLDR by default mem is proper to a stream, to share it:: - `Tensor.record_stream` -> automatic, but can be suboptimal and nondeterministic - `Stream.wait` -> manual, but precise control 2291
Ethan @ethansmith2000.com · 25/11/2024Incredible to see what is likely SOTA results coming out of open source with full reproducibility! Happy to have helped provide the compute for this and hoping to support more awesome research like this! 0110
Reposted by EthanFern @fernbear.bsky.social · 25/11/2024First, my sincerest thanks to @leonardoai.bsky.social with the help of @ethansmith2000.com for generously providing H100s to support this research to enable this release. Y'all rock, thanks so much! <3 121
Reposted by EthanFern @fernbear.bsky.social · 25/11/2024New NanoGPT training speed record: 3.28 FineWeb val loss in 4.66 minutes Previous record: 5.03 minutes Changelog: - FlexAttention blocksize warmup - hyperparameter tweaks 2333
Reposted by Ethanxjdr @xjdr.bsky.social · 24/11/2024i trying to follow as many of my old moots as possible and new people as i find them. some of y'all changing your pfp is just mean spirited (im lazy and learned people's pfps not names) 8361
Reposted by EthanRosmine @rosmineb.bsky.social · 25/11/2024Untuned SOAP beats tuned adamw at ever single step 061
Ethan @ethansmith2000.com · 24/11/2024ADAM's been tuned but SOAP and PSGD just using default params, you love to see it. 191
Ethan @ethansmith2000.com · 24/11/2024I goofed and never tested distributed saving, but now it works! It was a little annoying as both SOAP and psgd maintain preconds as lists of varying size, which fail to be pickled. To fix this I hardcoded there to be a max of 4 (based on conv layers being 4d tensors). 060
Ethan @ethansmith2000.com · 24/11/2024I’ve generally preferred research to software engineering but I am growing a liking for building the tools used for research 1120
Reposted by EthanEthan @ethansmith2000.com · 23/11/2024Here, have PSGD-Kron and SOAP with FSDP2 support. Please go wild with it, let's see something finally replace ADAM. github.com/ethansmith20... 3135
Reposted by EthanLucas Beyer (bl16) @giffmana.ai · 23/11/2024PSA: force-pushing to GitHub does NOT actually delete/overwrite anything. Everything you squash and force-push over is still there: 2301
Ethan @ethansmith2000.com · 23/11/2024credit to @_clashluke for the original PSGD-kron and SOAP implementations i used and Heavyball which i've learned a lot from 010
Ethan @ethansmith2000.com · 23/11/2024naive implementation is okay, much needs to be done with full tensor but a few pointwise ops we can benefit from remaining in shards. future improvements likely lie in pre-fetching and grabbing chunks of params, rather than one at a time. this is fragile though and DTensor still needs more support. 110
Ethan @ethansmith2000.com · 23/11/2024Here, have PSGD-Kron and SOAP with FSDP2 support. Please go wild with it, let's see something finally replace ADAM. github.com/ethansmith20... 3135
Ethan @ethansmith2000.com · 23/11/2024though really you could argue quite a bunch of older score-based models, hopfield networks and others kinda fit the bill of "the diffusion before diffusion" to some degree 020
Ethan @ethansmith2000.com · 23/11/2024from a 2012 lecture referencing a 1981 paper (and it all really goes further back than that) If ML experimentation software was as widespreadly accessible as it is now, I'd be generative diffusion models would have been discovered a while ago. 280
Ethan @ethansmith2000.com · 23/11/2024probably the best in-depth explanation i've seen on FSDP at the most granular levels, props to the authors dev-discuss.pytorch.org/t/fsdp-cudac... 193
Reposted by EthanJuan Lam @juanlam.com · 22/11/2024🥳 You can wishlist my #MR game Galactic Traffic Control NOW on Meta Quest! 🚀☄️🕹️ It's coming soon :D GTC is about guiding spaceships from your walls to their portals by drawing paths and destroying obstacles meta.com/experiences/... Let's talk about the features... 🧵 #gamedev #vr #xr #MetaQuestmeta.comGalactic Traffic Control on Meta QuestGalactic Traffic Control is a mixed-reality arcade game where you must guide the hyper-jumping spaceships that need to cross the space highway (your room) in order to get to their destination! 5164
Ethan @ethansmith2000.com · 21/11/2024lol weird I feel the opposite. The wall of text is usually more daunting yap than bite sized chunks of key points 110
Ethan @ethansmith2000.com · 21/11/2024Besides X having a longer existing community, I can’t imagine any reason you wouldn’t choose Bluesky over X. Your experience being at the mercy of algo-makers is so tired. 350