iiinova — where your digital self keeps evolving

Skander Moalla

@skandermoalla.bsky.social

PhD @Caglar Gulcehre Lab for AI Research (CLAIRE) @EPFL. Deep Reinforcement Learning, RLHF, foundation models. ML Research Template (github.com/CLAIRE-Labo/python-ml-re…)

Posts Replies Media

Skander Moalla @skandermoalla.bsky.social · 15/07/2025

QRPO is a framework. You can shape the optimal policy! 🎛️ We derive a framework around QRPO for using transformations on top of the quantile reward. Each transformation reshapes the reward distribution and affects the properties of the optimal policy, while having a tractable partition function.

110

Skander Moalla @skandermoalla.bsky.social · 15/07/2025

And we show that for relatively high beta, with good data, the probabilities increase as predicted 💯

110

Skander Moalla @skandermoalla.bsky.social · 15/07/2025

For QRPO, this is not a mystery anymore; we know exactly where the probabilities should move, and we explain how it's normal for them to decrease when the regularization (beta) is very low. This is simply because the target policy is much further away from the training support 🎯

110

Skander Moalla @skandermoalla.bsky.social · 15/07/2025

💬 The reward model we use has been trained to be robust to length bias, and we see that this is preserved in QRPO and REBEL, which use rewards. But when compressed to preferences for DPO and SimPO, it leads to the typical length bias trend, despite the reduction in mean length.

110

Skander Moalla @skandermoalla.bsky.social · 15/07/2025

🥇 QRPO achieves top performance in chat and coding compared to DPO, REBEL, and SimPO, each capturing a different way to learn from the reward signal (preference, reward difference, length normalization).

110

Skander Moalla @skandermoalla.bsky.social · 15/07/2025

Obviously, nothing comes for free, but we give you a great deal! 🤝 * QRPO does not need many reference rewards to estimate quantiles: for high-quality offline datasets, 1-3 are enough! * And you can scale this number for off-policy data generated from the reference model! 📈

110

Skander Moalla @skandermoalla.bsky.social · 15/07/2025

3️⃣ We can transform the reward distribution to make it known. It's uniform for reward quantiles! 🔑 🚀 The result: Quantile Reward Policy Optimization! QRPO transforms rewards to quantile rewards for which we derive Z, and can then fit the closed-form optimal RL solution with a simple regression! 📉

110

Skander Moalla @skandermoalla.bsky.social · 15/07/2025

1️⃣ The “infinite sum over all possible LLM generations” argument is a myth. We rewrite the partition function Z in terms of rewards, revealing that Z is given by the moment generating function (MGF) of the reward distribution! 2️⃣ Knowing the reward distribution => knowing the MGF => knowing Z 🔐

110

Skander Moalla @skandermoalla.bsky.social · 15/07/2025

🚀 Big time! We can finally do simple LLM RL fine-tuning with rewards and leverage offline/off-policy data! ❌ You want rewards, but GRPO only works online? ❌ You want offline, but DPO is limited to preferences? ✅ QRPO can do both! 🧵Here's how we do it:

1124

Skander Moalla @skandermoalla.bsky.social · 10/12/2024

Also, check out our ML project template—it’s a game-changer!🚀🚀 @caglarai.bsky.social 🧑‍💻 github.com/CLAIRE-Labo/...

063

Skander Moalla @skandermoalla.bsky.social · 10/12/2024

Ever been puzzled by your PPO agent collapsing out of nowhere? 📈🤯📉 Come check out our poster tomorrow! Wed 11 Dec 11 am - 2 pm PST West Ballroom A-D #6403 @caglarai.bsky.social @andreamiele.bsky.social @razvan-pascanu.bsky.social

171