Reposted by Alisa LiuAi2 @ai2.bsky.social · 07/05/2025📢We’re taking your questions now on Reddit for tomorrow’s AMA! Ask us anything about OLMo, our family of fully-open language models. Our researchers will be on hand to answer them Thursday, May 8 at 8am PST. 132
Reposted by Alisa LiuValentin Hofmann @valentinhofmann.bsky.social · 21/03/2025Humans store thousands of multi-word expressions like "of course" in their mental lexicon, but current tokenizers don't support multi-word tokens. Enter SuperBPE, a tokenizer that lifts this restriction and brings substantial gains in efficiency and performance! 🚀 Details 👇 151
Reposted by Alisa LiuAlton B.H. Worthington @abhw.bsky.social · 21/03/2025Hell yeah superwords. (I wanna call em supertokens, but I didn't develop them.) 021
Reposted by Alisa LiuJonathan Hayase @jon.jon.ke · 21/03/2025Tokenizers govern the allocation of computation. It's a waste to spend a whole token of compute predicting the "way" in "By the way". SuperBPE redirects that compute to predict more difficult tokens, leading to wins on downstream tasks! 041
Reposted by Alisa LiuNoah A. Smith @nlpnoah.bsky.social · 21/03/2025a small change to building your BPE tokenizer gets your pretrained LM 8 MMLU points (for example) and 27% inference-time efficiency boost ... 1111
Alisa Liu @alisawuffles.bsky.social · 21/03/2025We created SuperBPE🚀, a *superword* tokenizer that includes tokens spanning multiple words. When pretraining at 8B scale, SuperBPE models consistently outperform the BPE baseline on 30 downstream tasks (+8% MMLU), while also being 27% more efficient at inference time.🧵 38316
Reposted by Alisa LiuChristopher Akiki @cakiki.bsky.social · 28/02/2025This is also addressed in the appendix of @alisawuffles.bsky.social and colleagues' paper on BPE mixture inference. I think it might have been discovered by @soldaini.net if I'm not mistaken. arxiv.org/abs/2407.16607 131
Alisa Liu @alisawuffles.bsky.social · 11/12/2024excited to be at #NeurIPS2024! I'll be presenting our data mixture inference attack 🗓️ Thu 4:30pm w/ @jon.jon.ke — stop by to learn what trained tokenizers reveal about LLM development (‼️) and chat about all things tokenizers. 🔗 arxiv.org/abs/2407.16607 0134
Reposted by Alisa LiuJiacheng Liu @liujch1998.bsky.social · 09/12/2024Want to predict the task performance of LMs before pretraining them? We develop task scaling laws and model ladders, which predict the accuracy on individual tasks by OLMo 2 7B & 13B models within 2 points of absolute error. The cost is 1% of the compute used to pretrain them. 23314
Reposted by Alisa LiuMichael Saxon @saxon.me · 06/12/2024🚨I too am on the job market‼️🤯 I'm searching for faculty positions/postdocs in multilingual/multicultural NLP, vision+language models, and eval for genAI! I'll be at #NeurIPS2024 presenting our work on meta-evaluation for text-to-image faithfulness! Let's chat there! Papers in🧵, see more: saxon.me 1488
Reposted by Alisa LiuMechanical Dirk @mechanicaldirk.bsky.social · 02/12/2024We just updated the OLMo repo at github.com/allenai/OLMo! There are now several training configs that together reproduce the training runs that lead to the final OLMo 2 models. In particular, all the training data is available, tokenized and shuffled exactly as we trained on it!github.comGitHub - allenai/OLMo: Modeling, training, eval, and inference code for OLMoModeling, training, eval, and inference code for OLMo - allenai/OLMo 05411
Reposted by Alisa LiuAi2 @ai2.bsky.social · 26/11/2024Meet OLMo 2, the best fully open language model to date, including a family of 7B and 13B models trained up to 5T tokens. OLMo 2 outperforms other fully open models and competes with open-weight models like Llama 3.1 8B — As always, we released our data, code, recipes and more 🎁 515235
Reposted by Alisa LiuLuca Soldaini 🎀 @soldaini.net · 26/11/2024OLMo 2 is out 🥳 7B and 13B trained on 5T tokens, and meticulousy instruction tuned using Tulu 3 recipe. Simply the best fully open models yet. Really proud of the work & the amazing team at @ai2.bsky.social 926044
Reposted by Alisa LiuChristoph Molnar @christophmolnar.bsky.social · 24/11/2024No one can explain stochastic gradient descent better than this panda.media.tenor.coma panda bear is rolling around in the grass in a zoo enclosure .Alt: a panda bear is rolling around in the grass in a zoo enclosure . 1021632
Reposted by Alisa LiuVivek Kalyan @vivekkalyan.com · 24/11/2024Reading the TÜLU 3 paper from @ai2.bsky.social. It's refreshing to see a research lab treating AI as a real science with full reports, data, code, logs, evals. Paper: allenai.org/papers/tulu-... Demo: playground.allenai.org Code: github.com/allenai/open... Eval: github.com/allenai/olmes Notesallenai.org 1255
Reposted by Alisa LiuAi2 @ai2.bsky.social · 21/11/2024Meet Tülu 3, a set of state-of-the-art instruct models with fully open data, eval code, and training algorithms. We invented new methods for fine-tuning language models with RL and built upon best practices to scale synthetic instruction and preference data. Demo, GitHub, paper, and models 👇 211131
Reposted by Alisa LiuNathan Lambert @natolambert.bsky.social · 21/11/2024I've spent the last two years scouring all available resources on RLHF specifically and post training broadly. Today, with the help of a totally cracked team, we bring you the fruits of that labor — Tülu 3, an entirely open frontier model post training recipe. We beat Llama 3.1 Instruct. Thread. 821343
Reposted by Alisa LiuAkari Asai @akariasai.bsky.social · 19/11/20241/ Introducing ᴏᴘᴇɴꜱᴄʜᴏʟᴀʀ: a retrieval-augmented LM to help scientists synthesize knowledge 📚 @uwnlp.bsky.social & Ai2 With open models & 45M-paper datastores, it outperforms proprietary systems & match human experts. Try out our demo! openscholar.allen.ai 616339
