iiinova — where your digital self keeps evolving

Philip Bontrager @pbontrager.bsky.social · 08/06/2025

This thread is a bit long, but I thought it’d be interesting to share just one of the mundane parts of the deep learning stack that break and have to be rethought as models and training scale.

020

Philip Bontrager @pbontrager.bsky.social · 08/06/2025

To save, you need to let each GPU save their own partial safetensors, because communication is slow, and then line up the memory blocks and merge into one file.

120

Philip Bontrager @pbontrager.bsky.social · 08/06/2025

Safetensors are great for hosting checkpoints and make no assumptions about if your model is distributed by saving full unshared parameters. To work natively with safetensors, DCP needs to tell each GPU the exact slice of data to read without loading the full parameter.

100

Philip Bontrager @pbontrager.bsky.social · 08/06/2025

On startup, DCP has to map your old GPU layout to your new one so each GPU knows which file to read from and only read the data they need. But there’s one last problem; when you’re ready to take your model to another tool (serving, eval, etc), it expects safetenors checkpoints.

110

Philip Bontrager @pbontrager.bsky.social · 08/06/2025

Distributed Checkpoints (DCP) solve this by having every GPU save their own checkpoint asynchronously so you can save a checkpoint in less than a second. But this creates a new problem, the next time you want to use the model, you might have a different number of GPUs.

100

Philip Bontrager @pbontrager.bsky.social · 08/06/2025

What goes into saving checkpoints is not something that many people think about, but as models get bigger this becomes a challenge. The biggest open models now have checkpoints over 700gb that can take tens of minutes every time you want to consolidate into a checkpoint. pytorch.org/blog/hugging...

162

Philip Bontrager @pbontrager.bsky.social · 26/02/2025

I’m enjoying it while it lasts before everything fully homogenizes again

010

Reposted by Philip Bontrager

Eugene Vinitsky 🍒 @eugenevinitsky.bsky.social · 06/02/2025

We've built a simulated driving agent that we trained on 1.6 billion km of driving with no human data. It is SOTA on every planning benchmark we tried. In self-play, it goes 20 years between collisions.

2229855

Philip Bontrager @pbontrager.bsky.social · 27/01/2025

Aren’t these two paradoxes functionally the same? en.m.wikipedia.org/wiki/Braess%...

en.m.wikipedia.org

Braess's paradox - Wikipedia

040

Philip Bontrager @pbontrager.bsky.social · 25/01/2025

Original post here: x.com/jjitsev/stat...

x.com

050

Philip Bontrager @pbontrager.bsky.social · 25/01/2025

In the Alice In Wonderland (github.com/LAION-AI/AIW) reasoning and generalization benchmark, DeepSeek R1 appears to perform much more like o1 mini than o1 -preview. (Plot from laion-ai)

240

Philip Bontrager @pbontrager.bsky.social · 20/01/2025

What are the best benchmarks for reasoning models?

010

Philip Bontrager @pbontrager.bsky.social · 14/01/2025

Can we just study LLM activations/behavior because it’s interesting and it can tell us things about language and AI without imbuing artificial importance or meaning on top of it?

020

Philip Bontrager @pbontrager.bsky.social · 13/01/2025

Haha, that wasn’t lost on me. Facebook’s still going strong, but it’s a different site and users from when I was in HS.

020

Philip Bontrager @pbontrager.bsky.social · 13/01/2025

If you can choose who follows you, that sounds more like “friends” from the old Facebook days.

120

Philip Bontrager @pbontrager.bsky.social · 07/01/2025

I found out about Warp because I was on jury duty with one of their devs 😂 It’s been great compared to the Mac’s default terminal.

040

Philip Bontrager @pbontrager.bsky.social · 07/01/2025

Maybe let’s go the other direction and include blog posts in CVs too.

120

Philip Bontrager @pbontrager.bsky.social · 07/01/2025

That would imply that we solved self-driving (image recognition) and search (language understanding), among other things.

020

Philip Bontrager @pbontrager.bsky.social · 04/01/2025

This could be a good case for mixed models. The model parsing the text could likely be smaller or be fairly cheap like DeepSeek

110

Philip Bontrager @pbontrager.bsky.social · 03/01/2025

Thankfully in a small startup you only have to sell an idea to a couple of people and you can get going.

000

Philip Bontrager @pbontrager.bsky.social · 03/01/2025

One startup I joined had a model getting 95% on benchmarks but terrible in practice. Spent the first 6 months developing new benchmarks instead of a new model.

110

Philip Bontrager @pbontrager.bsky.social · 03/01/2025

I always set out to propose a new idea and end up having to proposing a new benchmark instead

140

Philip Bontrager @pbontrager.bsky.social · 03/01/2025

What if humanity knows X and wants to understand Z. If a computer can give us Y so that we can understand Z, that would be useful for science. Though I’d say that we still didn’t know Y ourselves yet.

000

Philip Bontrager @pbontrager.bsky.social · 03/01/2025

Imagine if under the hood O1 is just calling “write better code” over and over again 😂

050

Philip Bontrager @pbontrager.bsky.social · 02/01/2025

I posted about this recently. Benchmarks show what models can’t do, not what they can do.

010

Philip Bontrager @pbontrager.bsky.social · 01/01/2025

Plagiarize other people’s research

010

Philip Bontrager @pbontrager.bsky.social · 30/12/2024

Imagine being an editor for an LLM, so much work with low confidence that you’ll have something interesting in the end.

030

Philip Bontrager @pbontrager.bsky.social · 30/12/2024

I remember a lot of focus being on the loss function. My impression was that we thought we had models that would work well if only we had a good perceptual loss to train them with. In comes the GAN

000

Philip Bontrager @pbontrager.bsky.social · 28/12/2024

Base models are closer, but they’re still affected by the company’s decisions on which data to filter out and more indirectly on what data is given free hosting on the internet.

000

Philip Bontrager @pbontrager.bsky.social · 28/12/2024

Jermey’s analysis makes sense but it’s difficult to compare GPU and CPU data centers. Looking at Meta’s data centers, most of them cost ~1B but the new AI focused Richland Paris costs 10B. I’m not sure how much power it draws compared to the other data centers. datacenters.atmeta.com/all-locations/

datacenters.atmeta.com

Global data centers (Asia, Europe, U.S.) - Meta Data Centers

000

Philip Bontrager @pbontrager.bsky.social · 28/12/2024

10% is still much higher than I would have guessed given the scale of the internet and how recent really large scale training is.

000

Philip Bontrager @pbontrager.bsky.social · 28/12/2024

Similar to nightshade, I’m very skeptical on how effective these approaches are against scaled up models and future architectures. They may just be helping make the model training more robust in the long term.

070

Philip Bontrager @pbontrager.bsky.social · 26/12/2024

Did you run QwQ locally? I wonder if the massive variance can just be from temperature? I guess it could be a great model if you have a verifier.

120

Philip Bontrager @pbontrager.bsky.social · 26/12/2024

Yeah, I imagine the meta for Bluesky right now is to create a starter pack of a lot of influential people with a catchy name like "BayesIsAllYouNeed" or something that will get them to share it around. Meanwhile you get added to the pack for free.

010

Philip Bontrager @pbontrager.bsky.social · 26/12/2024

In your setup, do you have control over the model sampling? I find it very interesting to see the spread between pass@5 and 5/5 for the different models and wonder if that’s connected to how they’re sampled.

110

Philip Bontrager @pbontrager.bsky.social · 25/12/2024

I think starter packs are great if you make the jump early, but they bake in a massive first mover advantage. I think as much as people here love follow feeds, follow feeds give tremendous advantage to people who are already well known and are hard for newcomers to break into.

140

Philip Bontrager @pbontrager.bsky.social · 25/12/2024

Every time I’m by Cold Spring Harbor the campus looks so beautiful I wish I had an excuse to visit.

010

Philip Bontrager @pbontrager.bsky.social · 25/12/2024

Thanks! That makes sense. We need these models now to know when to go between chain of thought and direct answers.

000

Reposted by Philip Bontrager

mr. TIM @timkellogg.me · 23/12/2024

this seems like a decent LLM test. 3 sequential game states of Qwirkle. Sonnet 3.5 gets the first play but not the second o1 is very bad at this. first play it takes 59 seconds and it’s answer isn’t even a play that’s on the board. same with second play, but only 36 seconds

3122

Philip Bontrager @pbontrager.bsky.social · 25/12/2024

How’s it compare to other VLMs you’ve tried?

100

Philip Bontrager @pbontrager.bsky.social · 24/12/2024

Now you just find one benchmark where AI is inferior to humans and you have a contradiction! While this approach is great at showing what AIs can’t do, you can never use it to prove the positive statement, there could always exist more benchmarks

000

Philip Bontrager @pbontrager.bsky.social · 24/12/2024

It is a proof by contradiction, where we want to prove the statement “AI has not reached human level”. We first assume “AI has reached human level” and therefore AI should be able to be human level or above on every possible benchmark.

100

Philip Bontrager @pbontrager.bsky.social · 24/12/2024

Contrary to what I see in a lot of online discussions, AI benchmarks are not meant to show how capable an AI system is, but instead they show what they can’t do.

100

Philip Bontrager @pbontrager.bsky.social · 23/12/2024

It looks like Pixtral-12B is the current leading “small” model if you’re using vllm. But if you want to run with a tool like ollama, llama 3.2 is probably your best bet. lmarena.ai

lmarena.ai

Chatbot Arena (formerly LMSYS): Free AI Chat to Compare & Test Best AI Chatbots

020

Philip Bontrager @pbontrager.bsky.social · 23/12/2024

So you do a lot of work prompt engineering the style you want, and then put that in a dataset so you can Lora finetune to get that style every time?

000

Philip Bontrager @pbontrager.bsky.social · 23/12/2024

Really cool! How are these made?

000

Philip Bontrager @pbontrager.bsky.social · 23/12/2024

We’re going to be adding diffusion too soon!

110

Philip Bontrager @pbontrager.bsky.social · 22/12/2024

I’ve had pretty good luck following a lot of academics but I use this app in a pretty narrow way. I have noticed a drop off in activity the last week.

010