iiinova — where your digital self keeps evolving

Rosmine @rosmineb.bsky.social · 10/01/2025

[1] arxiv.org/pdf/2402.03300 [2] hijkzzz.notion.site/unraveling-r...

Rosmine @rosmineb.bsky.social · 10/01/2025

One callout is [2] found similar performance between GRPO, RLOO, and REINFORCE++, but that GRPO was more prone to reward hacking, and that critic pretraining with PPO outperforms GRPO.

100

Rosmine @rosmineb.bsky.social · 10/01/2025

The drawback of GRPO is it requires you generate many responses for the same prompt, so if you were previously generating few responses per prompt, GRPO may increases computation time.

100

Rosmine @rosmineb.bsky.social · 10/01/2025

Additionally, it moves the KL penalty into the loss function (RLHF usually adds the KL penalty to the rewards), which simplifies the computation of the advantage.

100

Rosmine @rosmineb.bsky.social · 10/01/2025

To do this, it starts by generating several responses for each query. Then when computing the advantage, it replaces the value function by the reward of the sample normalized by the mean and std across all responses for the same query.

100

Rosmine @rosmineb.bsky.social · 10/01/2025

Overview of GRPO (Group Relative Policy Optimization) GRPO is an improvement on PPO introduced in the DeepSeekMath paper The motivation is that PPO requires 4 large models, a policy, value function, reward model, and reference model. GRPO removes the need for the value model.

171

Rosmine @rosmineb.bsky.social · 09/12/2024

I want to train a transformer model to be a random number generator At first it sounds dumb, but you could leverage GPU non-determinism to make it truly random, not just pseudo random There are better ways to do rng so I still think it's a bad idea, but a cool bad idea

010

Rosmine @rosmineb.bsky.social · 02/12/2024

- Performance improvement from RLAIF vs. SFT depends on the base model. E.g. For Llama models, SFT is much more effective than RLAIF (see graph)

000

Rosmine @rosmineb.bsky.social · 02/12/2024

- usually people use SFT generated by gpt 3.5, and rlaif from gpt4. If you use a higher quality model to generate SFT data, then usually RLAIF is less effective than SFT

100

Rosmine @rosmineb.bsky.social · 02/12/2024

- for RLAIF to be valuable, you need 1. sufficiently strong pretrained base model 2. capability mismatch between the teacher used for the SFT data collection and the critic used for collecting ai feedback

100

Rosmine @rosmineb.bsky.social · 02/12/2024

A Critical Evaluation of AI Feedback for Aligning Large Language Models Investigated paradigm of modifying model behavior by first doing SFT training using data from teacher model, then following with RLAIF training by teacher reward model They found:

110

Rosmine @rosmineb.bsky.social · 29/11/2024

ha ha nothing that funny, best was "AI/LLM Nerds (derogatory)"

010

Rosmine @rosmineb.bsky.social · 28/11/2024

All I did was post paper summaries, I guess people don’t like my taste in papers

130

Rosmine @rosmineb.bsky.social · 27/11/2024

(most other approaches just do reasoning through prompting, or require the dataset to have reasoning included) - Improvement varies a lot on the dataset. There's huge improvement on GSM8K, but ARC improvement is at most 1.6% arxiv.org/abs/2411.04282

arxiv.org

Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding

Large language models (LLMs) have shown impressive capabilities, but still struggle with complex reasoning tasks requiring multiple steps. While prompt-based methods like Chain-of-Thought (CoT) can im...

010

Rosmine @rosmineb.bsky.social · 27/11/2024

- The paper shows how to use variational lower bound that explicitly includes the probability of the reasoning, then uses RL (REINFORCE Leave One Out) to optimize the reasoner. Basically, this gives a good way to train the model on reasoning without specialized training data

100

Rosmine @rosmineb.bsky.social · 27/11/2024

Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding This paper introduces LaTent Reasoning Optimization (LaTRO), a training framework - Improves zero shot accuracy by +12.5% on GSM8K over base models. - Doesn't use external feedback or reward models ...

110

Rosmine @rosmineb.bsky.social · 27/11/2024

Nice to meet you, Mr. Cow 😉

010

Rosmine @rosmineb.bsky.social · 27/11/2024

I'm suspicious he's real, the pinned tweet makes me think it's a parody (but mostly I'm suspicious because he followed me lol)

020

Rosmine @rosmineb.bsky.social · 26/11/2024

Fingers crossed this means they're reallocating all compute towards a shiny new model

000

Rosmine @rosmineb.bsky.social · 26/11/2024

Trying a new ML research area is tough. Every time I think I have a good new idea, I google and find there are already 50,000 papers covering it. I cannot read 50,000 papers per day

000

Rosmine @rosmineb.bsky.social · 25/11/2024

(I'm doing a lit review of reasoning for LLMs. Aiming to post 1-2 paper summaries per day. This is the first in the series) arxiv.org/pdf/2110.14168

arxiv.org

000

Rosmine @rosmineb.bsky.social · 25/11/2024

- Found that training for more than 2 epochs on GSM8K caused performance degradation, because the verifiers require high diversity of samples, and too many epochs causes diversity to collapse

100

Rosmine @rosmineb.bsky.social · 25/11/2024

- using a verifier and generating too many solution and picking the best actually hurts performance: it increases the probability of finding adversarial solutions that fool the verifier

100

Rosmine @rosmineb.bsky.social · 25/11/2024

- To use verifiers, they train a model to score if a solution is correct or not. They then generate many high temperature solutions, and use the verifier to select the solution most likely to be correct.

100

Rosmine @rosmineb.bsky.social · 25/11/2024

- GSM8K has 8.5K high quality grade school math word problems. Problems take 2-8 steps to solve, using +,-,x,/, with a focus on diversity of problems.

100

Rosmine @rosmineb.bsky.social · 25/11/2024

Training Verifiers to Solve Math Word Problems (2021) - This paper introduced GSM8K, and showed how using verifiers can significantly improve performance (up to 20+ percentage points compared to finetuning, see graph below)

110

Rosmine @rosmineb.bsky.social · 25/11/2024

Got it, will try out PSGD, thanks!

020

Rosmine @rosmineb.bsky.social · 25/11/2024

Untuned SOAP beats tuned adamw at ever single step

061

Rosmine @rosmineb.bsky.social · 25/11/2024

What's up with PSGD at the beginning? Does it need a while to warm up?

210

Rosmine @rosmineb.bsky.social · 24/11/2024

Adding reminder to never switch between bf16 and fp16 (especially watch out if you’re using old gpus)

040

Rosmine @rosmineb.bsky.social · 24/11/2024

This is both very cool and very confusing lol For the blog post, can we get an analysis of examples where R2L worked and L2R didn't? Hopefully that will give better insight into why R2L is better.

000

Rosmine @rosmineb.bsky.social · 23/11/2024

I made a project to play dance dance revolution without a dance pad, instead using 2 high speed cameras, and running each frame through a shallow convnet to classify the steps. The tape on the floor is so I know where to step. The stools are so I don't accidentally step on cameras

070

Rosmine @rosmineb.bsky.social · 21/11/2024

I quickly hacked this together. There will be bugs and missing features. If there is enough interest, I will make improvements Code + more detailed instructions here github.com/rosmineb/lit...

github.com

010

Rosmine @rosmineb.bsky.social · 21/11/2024

You can also input a list of papers that you've read. If you do, it'll give you an "expertise score" which describes how well you know the subfield based on citation counts of the papers you've read.

110

Rosmine @rosmineb.bsky.social · 21/11/2024

3. Run the script which traverses the reference/citation graph to find the most relevant papers matching your description This will give you a short list of the most important papers you need to read to understand the subfield.

100

Rosmine @rosmineb.bsky.social · 21/11/2024

I made a script to help you quickly become an expert in an ML subfield How works: 1. Input a few papers 2. Input a description of the subfield. Could be generic like "optimizers" or highly specific like "improvements on LoRA" 🧵

110

Rosmine @rosmineb.bsky.social · 20/11/2024

So much learning in re-reading old banger papers e.g. Adafactor includes - new low rank matrix approximation algorithm (used for second moment) - detecting when Adam second moment is out of date - better beta_2 schedules - analysis of model training stability arxiv.org/pdf/1804.04235

arxiv.org

010

Rosmine @rosmineb.bsky.social · 08/11/2024

- If your disk is full, that can sometimes mess with ssh. Set up automated warnings when your disk starts to get too full (very important if you're regularly saving model checkpoints)

010

Rosmine @rosmineb.bsky.social · 08/11/2024

- If you'll be away for extended periods of time, it's important to have backups/failsafes so if something goes wrong you're not stuck without access. Make sure to have a health check running regularly on your computer, so if something goes wrong, it'll restart ssh and/or the computer.

110

Rosmine @rosmineb.bsky.social · 08/11/2024

- For solutions that don't require changing ISP settings, Tailscale worked well for me without any issues. Zerotier seemed to work well at first, but sometimes breaks temporarily when switching wifi networks

100

Rosmine @rosmineb.bsky.social · 08/11/2024

- Have multiple ssh options in case you have issues with one. I wanted something that was easy and fast to set up. Some solutions require setting up port forwarding with your ISP, which was extremely annoying for my use case, but might work better for you.

100

Rosmine @rosmineb.bsky.social · 08/11/2024

I recently spent set up ssh to my computer from anywhere, here are some tips:

110

Rosmine @rosmineb.bsky.social · 07/11/2024

Also, current MIA approaches are sensitive to small changes in the sample, so it's easy to avoid detection with a rephrasing of the test data. arxiv.org/pdf/2402.07841 Not saying that it's impossible to tell if a model trained on test, just agreeing that it's difficult

arxiv.org

010

Rosmine @rosmineb.bsky.social · 07/11/2024

1. LLMs are usually trained with few epochs so don't see a single sample many times 2. language fuzziness, there can be overlap between samples even if there isn't an exact match.

110

Rosmine @rosmineb.bsky.social · 07/11/2024

Is it possible to tell if a model was trained on test? I thought it would be easy to tell if a model was trained on test by using Membership Inference Attacks I looked into it more and turns out it's a lot harder than I thought because...

110

Rosmine @rosmineb.bsky.social · 06/11/2024

Will he gut Biden’s CHIPs act? We need chips manufacturing as well

000

Rosmine @rosmineb.bsky.social · 06/11/2024

Other merging techniques like vector math on the weights are even more ridiculous to me. I get how it works, but it still seems like magic

000