iiinova — where your digital self keeps evolving

Alex Wettig

@awettig.bsky.social

PhD@Princeton trying to make sense of language models and their training data

Alex Wettig @awettig.bsky.social · 16/07/2025

Presenting two posters at ICML over the next two days: - Both at 11am - 1:30pm - Both about how to improve pre-training with domains - Both at stall # E-2600 in East Exhibition Hall A-B (!) Tomorrow: WebOrganizer w/ @soldaini.net & @kylelo.bsky.social Thursday: MeCo by @gaotianyu1350.bsky.social

041

Alex Wettig @awettig.bsky.social · 18/02/2025

🤔 Ever wondered how prevalent some type of web content is during LM pre-training? In our new paper, we propose WebOrganizer which *constructs domains* based on the topic and format of CommonCrawl web pages 🌐 Key takeaway: domains help us curate better pre-training data! 🧵/N

1268

Reposted by Alex Wettig

Jiacheng Liu @liujch1998.bsky.social · 09/12/2024

Want to predict the task performance of LMs before pretraining them? We develop task scaling laws and model ladders, which predict the accuracy on individual tasks by OLMo 2 7B & 13B models within 2 points of absolute error. The cost is 1% of the compute used to pretrain them.

23314