iiinova — where your digital self keeps evolving

Wanyun Xie

@wanyunxie.bsky.social

Wanyun Xie @wanyunxie.bsky.social · 16/07/2025

We'll present our work, "CHAMELEON: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning," at #ICML2025! This is joint work with Francesco Tonin and @CevherLIONS. 📍 Find us at Poster E-2807 from 11 AM today. Excited to connect and discuss!

300

Reposted by Wanyun Xie

Volkan Cevher @cevherlions.bsky.social · 13/02/2025

arxiv.org/abs/2502.07529 🚀 Key results: - Based on conditional gradient method - Beats Muon+Adam on NanoGPT (tested up to 3B params) - Zero-shot learning rate transfer across model size - Uses WAY less memory (just one set of params + half-precision grads) - Provides explicit norm control

141

Reposted by Wanyun Xie

Volkan Cevher @cevherlions.bsky.social · 13/02/2025

🔥 Want to train large neural networks WITHOUT Adam while using less memory and getting better results? ⚡ Check out SCION: a new optimizer that adapts to the geometry of your problem using norm-constrained linear minimization oracles (LMOs): 🧵👇

3186

Wanyun Xie @wanyunxie.bsky.social · 11/12/2024

We'll present "SAMPa: Sharpness-Aware Minimization Parallelized" at #NeurIPS24 on Thursday! This is joint work with Thomas Pethick and Volkan Cevher. 📍 Find us at Poster #5904 from 16:30 in the West Ballroom.

111