Sign in

Fern

@fernbear.bsky.social
201 followers 507 following 37 posts

Neural network speedrunner and community-funded open source researcher. Set the CIFAR-10 record several times. Send me consulting/contracting work! she/they❤️

PostsRepliesMedia
Fern @fernbear.bsky.social · 03/02/2025
This is a classic example of _why_ choose-one-of-n datasets need to have large-scale, crowd-sourced statistics and should use the KL-divergence instead of cross-entropy. Reviewers will be more biased than a crowd, it's a high variance+bias estimator, it can harm research.
A multiple choice question with an apparently incorrect answer chosen as the correct answer.

Img source: @whybyfire.bsky.social
120
Reposted by Fern
Ted Underwood @tedunderwood.com · 01/12/2024
Did you know that attention across the whole input span was inspired by the time-negating alien language in Arrival? Crazy anecdote from the latest Hard Fork podcast (by @kevinroose.com and @caseynewton.bsky.social). HT nwbrownboi on Threads for the lead.
Transcript of Hard Fork ep 111: Yeah. And I could talk for an hour about transformers and why they are so important.
But I think it's important to say that they were inspired by the alien language in the film Arrival, which had just recently come out.
And a group of researchers at Google, one researcher in particular, who was part of that original team, was inspired by watching Arrival and seeing that the aliens in the movie had this language which represented entire sentences with a single symbol. And they thought, hey, what if we did that inside of a neural network? So rather than processing all of the inputs that you would give to one of these systems one word at a time, you could have this thing called an attention mechanism, which paid attention to all of it simultaneously.
That would allow you to process much more information much faster. And that insight sparked the creation of the transformer, which led to all the stuff we see in Al today.
1924653
Reposted by Fern
Ethan @ethansmith2000.com · 28/11/2024
it's crazy to me that RoPE's issue with BF16 wasn't noticed earlier. For a reasonable N of 2048, these are the computed frequencies prior to cos(x) & sin(x) for fp32 above and bf16 below. Given how short the period is of simple trig functions, this difference is catastrophic for large values.
181
Reposted by Fern
Ethan @ethansmith2000.com · 25/11/2024
Just added FSDP2 support for MARS and Muon!
182
Fern @fernbear.bsky.social · 25/11/2024
Thanks for 100 followers, y'all! Happened so fast and can't wait to put out more research on here! 😊❤️
170
Fern @fernbear.bsky.social · 25/11/2024
New NanoGPT training speed record: 3.28 FineWeb val loss in 4.66 minutes Previous record: 5.03 minutes Changelog: - FlexAttention blocksize warmup - hyperparameter tweaks
2333
Reposted by Fern
Birchlabs @birchlabs.co.uk · 19/11/2024
NATTEN just added fused support for self-cross attention! so you can attend to local neighbourhood and registers or text condition. it lets you reduce partial attention results (e.g. logsumexp provided by xformers APIs) into its LSE. github.com/SHI-Labs/NAT...
github.com
Support for fused cross-NA by alihassanijr · Pull Request #182 · SHI-Labs/NATTEN
Adds experimental support for additional context tokens to Fused NA. Any number of partial attention results can be reduced into a final one as if their contexts were merged, which is just the same...
091
Reposted by Fern
Kosta Derpanis @csprofkgd.bsky.social · 24/11/2024
❤️ my MNIST socks
6944
Reposted by Fern
Peyman Milanfar @docmilanfar.bsky.social · 24/11/2024
Radon Transform (RT) was formulated in 1917 but remained useless in practice until CT scanners were invented in the 60s But RT isn't just for CTs. It's a sort of generalization of marginals in probability RT g(p,θ): Shoot rays at θ+90 & offset p, measure line integrals of f(x,y) along the ray 1/n
28512
Reposted by Fern
Ethan @ethansmith2000.com · 23/11/2024
Here, have PSGD-Kron and SOAP with FSDP2 support. Please go wild with it, let's see something finally replace ADAM. github.com/ethansmith20...
3135
Reposted by Fern
Ethan @ethansmith2000.com · 23/11/2024
probably the best in-depth explanation i've seen on FSDP at the most granular levels, props to the authors dev-discuss.pytorch.org/t/fsdp-cudac...
193