Azal Ahmad Khan @azalakhan.bsky.social · 09/10/2026Excited to share this work with amazing collaborators from IBM Research and UMN: @keshavramji.bsky.social, @tahiranaseem.bsky.social, Ali Anwar, and @ramon-astudillo.bsky.social! 021
Azal Ahmad Khan @azalakhan.bsky.social · 09/10/2026Our takeaway: test-time scaling need not be limited to generating more tokens. We can also explore different versions of the model itself! Check out the paper for more findings on the keys to making hypernets work better! 📄 Paper: arxiv.org/abs/2610.01934 120
Azal Ahmad Khan @azalakhan.bsky.social · 09/10/2026And perhaps the most surprising finding: these updates are not entirely query-specific! We find that updates generated for one set of questions can be combined and reused on unseen questions. This suggests that the hypernetwork learns shared structure in how LLMs should adapt. 110
Azal Ahmad Khan @azalakhan.bsky.social · 09/10/2026Here's where things get exciting! Instead of sampling K answers from the same model, we sample K LoRA updates, giving us K different adapted models. As we increase K, performance improves, outperforming corresponding token-sampling baselines. Test-time scaling in weight space! 120
Azal Ahmad Khan @azalakhan.bsky.social · 09/10/2026But why predict just one LoRA? There might be many useful ways to adapt a model for the same question. So we train it to predict a distribution instead. Interestingly, even the mean of this distribution can outperform a directly trained deterministic hypernetwork! 110
Azal Ahmad Khan @azalakhan.bsky.social · 09/10/2026Can a model figure out how it should adapt, just by looking at the question? Surprisingly, yes! We train a hypernetwork that takes a query and directly predicts a LoRA update for a frozen LM. No demonstrations, task descriptions, or gradient steps at test time. 110
Azal Ahmad Khan @azalakhan.bsky.social · 09/10/2026🚀 What if we could scale LLM inference by sampling different model weights, not just more tokens? We show that a single query can be enough to generate useful LoRA updates, and that predicting a distribution over them is key to making hypernets work! Here's what we found 🧵 180
