iiinova — where your digital self keeps evolving

Andrei Mircea @mirandrom.bsky.social · 01/08/2025

not really sure what that implies with respect to their results, but it's a surprising contrast with no obvious explanation

010

Andrei Mircea @mirandrom.bsky.social · 01/08/2025

we also found interesting differences in optimizers with respect to loss deceleration (www.arxiv.org/abs/2506.05447). Surprisingly, Muon had worse post-deceleration convergence. That suggests it exacerbates rather than reduces interference in language modeling despite being a second order optimizer.

220

Andrei Mircea @mirandrom.bsky.social · 12/07/2025

Thanks to my collaborators and mentors @katelobacheva.bsky.social, Irina Rish, Supriyo Chakraborty, and Nima Chitsazan. Also Ashwinee Panda for coining "zero-sum learning", which is honestly a pretty great name.

020

Andrei Mircea @mirandrom.bsky.social · 12/07/2025

All of our code and artefacts are also open, which hopefully will help. Code: github.com/mirandrom/zsl Checkpoints: huggingface.co/mirandrom/zs... Wandb logs: wandb.ai/amr-amr/zsl/...

100

Andrei Mircea @mirandrom.bsky.social · 12/07/2025

TL;DR We find two new phenomena (loss deceleration + zero-sum learning) and show quantifiably how scaling improves LLMs by mitigating these. What’s cool is that these could potentially be mitigated independent of scaling (Step 2). Exactly how to do this remains an open question.

110

Andrei Mircea @mirandrom.bsky.social · 12/07/2025

Step 1: Understand how scaling improves LLMs. Step 2: Directly target underlying mechanism. Step 3: Improve LLMs independent of scale. Profit. In our ACL 2025 paper we look at Step 1 in terms of training dynamics. Project: mirandrom.github.io/zsl Paper: arxiv.org/pdf/2506.05447

241

Andrei Mircea @mirandrom.bsky.social · 10/06/2025

Mechanistic understanding of systematic failures in language models is something more research should strive for IMO. This is really interesting work in that vein by @ziling-cheng.bsky.social, highly recommend you check it out.

010

Reposted by Andrei Mircea

Ziling Cheng @ziling-cheng.bsky.social · 06/06/2025

Do LLMs hallucinate randomly? Not quite. Our #ACL2025 (Main) paper shows that hallucinations under irrelevant contexts follow a systematic failure mode — revealing how LLMs generalize using abstract classes + context cues, albeit unreliably. 📎 Paper: arxiv.org/abs/2505.22630 1/n

14517

Andrei Mircea @mirandrom.bsky.social · 15/12/2024

Special thanks to @katelobacheva.bsky.social and Irina Rish from @mila-quebec.bsky.social for their supervision; and to Nima Chitsazan and Supriyo Chakraborty from CapitalOne for their support on this project during my summer internship there!

000

Andrei Mircea @mirandrom.bsky.social · 15/12/2024

🧵 (12/N) If you’re still reading, here are some neat plots to express my gratitude. These are per-token loss landscape cross-sections, taken along weight update directions at different train steps. Also equivalent cross-sections of overall losses extruded in 3D because why not.

000

Andrei Mircea @mirandrom.bsky.social · 15/12/2024

🧵 (11/N) While our hypothesis and results confirm that there exists mechanisms underlying scaling improvements that can be targeted directly and independent of scale, they do not fully account for the effect of scaling on loss deceleration. This is something we’re working on!

000

Andrei Mircea @mirandrom.bsky.social · 15/12/2024

🧵 (10/N) We also observe that scaling decreases gradient opposition before deceleration, effectively contributing to greater loss improvements before deceleration. While SGO converges to ~1 across scales, its relative effect on ZSL appears to be mitigated by scale.

000

Andrei Mircea @mirandrom.bsky.social · 15/12/2024

🧵 (9/N) Explaining ZSL with systematic gradient opposition (SGO) In our paper, we show how SGO (when destructive interference in per-example gradients approaches 1) fundamentally results in ZSL, and confirm it occurs with and explains deceleration.

100

Andrei Mircea @mirandrom.bsky.social · 15/12/2024

🧵 (8/N) To go beyond co-occurrence, we disentangle the relative contribution of ZSL to slowing loss improvements and show that it is indeed the principal contributor to loss deceleration across scales.

000

Andrei Mircea @mirandrom.bsky.social · 15/12/2024

🧵 (7/N) To quantify ZSL, we define destructive interference as the rate with which elements in a sum cancel-out, and measure it for per-example loss improvements throughout training. Consistent with our hypothesis, ZSL occurs with deceleration and is decreased by scale.

000

Andrei Mircea @mirandrom.bsky.social · 15/12/2024

🧵 (6/N) In ZSL, systematic gradient opposition between tokens leads to degenerate training dynamics where improvements in one set of tokens are offset by degradation in another, bottlenecking the overall rate of improvement and leading to deceleration.

000

Andrei Mircea @mirandrom.bsky.social · 15/12/2024

🧵 (5/N) Explaining loss deceleration with zero-sum learning In other words, by explaining loss deceleration (and the mitigating effect of scale) we can explain scaling improvements. We propose the zero-sum learning (ZSL) hypothesis as an explanation for deceleration.

010

Andrei Mircea @mirandrom.bsky.social · 15/12/2024

🧵 (4/N) Specifically, scaling seems to improve loss by improving 1) the loss at which deceleration occurs; and 2) the log-log rate of loss improvement after deceleration. Using BNSL, we can measure these quantities and tie them to final loss (i.e. to scaling improvements).

000

Andrei Mircea @mirandrom.bsky.social · 15/12/2024

🧵 (3/N) Explaining scaling improvements with loss deceleration Scaling improvements can be expressed in terms of mitigating “loss deceleration”, an abrupt slowdown in the rate of loss improvement; characterized by piecewise linear log-log loss curves.

000

Andrei Mircea @mirandrom.bsky.social · 15/12/2024

🧵 (2/12) Motivation LLM scaling laws predict but do not explain *how* scaling model size improves loss. By identifying a mechanism underlying scaling improvements, we could target it directly and potentially improve LLMs independent of scale.

110

Andrei Mircea @mirandrom.bsky.social · 15/12/2024

📢 New paper “Language model scaling laws and zero-sum learning” at Sci4DL #neurips2024. ℹ️ openreview.net/forum?id=yBq2g832Go TL;DR: scaling improves LMs by mitigating zero-sum learning, a mechanism that could be targeted directly and independent of scale. West 205-207 4:30-5:30 PM 🧵 (1/12)

1292

Reposted by Andrei Mircea

Muawiz Chaudhary @muawizc.bsky.social · 15/12/2024

My collaborators (Vivian White, @kamdh.bsky.social) will be presenting our work at the #Sci4DL workshop at #NeurIPS2024 today. Location: West Meeting Room 205-207 Time: 4:30-5:30 PM We present a principled probability distribution model of pre-trained deep neural networks. Check it out!

143

Reposted by Andrei Mircea

Vagrant Gautam @dippedrusk.com · 29/11/2024

For those of you attending #NeurIPS2024 in person: I'm from Vancouver and I made an extensive list of restaurants, bars, bookstores, etc., that I used to frequent when I still lived there. Enjoy! dippedrusk.com/posts/2024-0...

dippedrusk.com

Vagrant's Vancouver | Vagrant Gautam

A non-comprehensive list of places to go and things to do in the Greater Vancouver Area as curated by yours truly over 6 years. Might be outdated so please double-check!

810012

Reposted by Andrei Mircea

Abhilasha Ravichander @lasha.bsky.social · 11/11/2024

✨I am on the faculty job market in the 2024-2025 cycle!✨ My research centers on advancing Responsible AI, specifically enhancing factuality, robustness, and transparency in AI systems. If you have relevant positions, let me know! lasharavichander.github.io Please share/RT!

lasharavichander.github.io

Abhilasha Ravichander - Home

25122

Reposted by Andrei Mircea

Ben Newman @benn9.bsky.social · 11/11/2024

✨EMNLP Paper! ✨ Have you ever constructed a table to organize your literature review process? Can we use LMs to generate these automatically? We are excited to present ArxivDIGESTables 🍽️ a study of collecting, generating, and evaluating 🎓 scientific literature review tables 📃!

A screenshot of the first page of the paper discussed in the thread. Figure 1 contains a set of three cartoon papers with related text highlighted in three different colors. To its left, there's an arrow pointing to a cartoon table with a column corresponding to each color and a row corresponding to each paper.

2292