Tired of KL penalties constraining your model?
But don't want your policy to just hack the reward?
Try Gradient Regularization!
We show it beats a KL penalty in RLHF, RLVR and LLM-as-a-Judge!
🧵1/7
Johannes Ackermann
@johannesack.bsky.social
38 followers 174 following 18 posts
Reinforcement Learning PhD Student at the University of Tokyo, Prev: Intern at Sakana AI, PFN, M.Sc/B.Sc. from TU Munich johannesack.github.io
