iiinova — where your digital self keeps evolving

Vishakh Padmakumar @vishakhpk.bsky.social · 07/11/2025

Last year I worked at @adobe.com and @ai2.bsky.social, exploring how we can help users read, organize and understand long documents. This piece covers what we learned on modelling user intent and combining LLMs with principled tools when building complex pipelines for it!

000

Reposted by Vishakh Padmakumar

NYU Center for Data Science @nyudatascience.bsky.social · 30/05/2025

CDS PhD student @vishakhpk.bsky.social, with co-authors @johnchen6.bsky.social, Jane Pan, Valerie Chen, and CDS Associate Professor @hhexiy.bsky.social, has published new research on the trade-off between originality and quality in LLM outputs. Read more: nyudatascience.medium.com/in-ai-genera...

nyudatascience.medium.com

In AI-Generated Content, A Trade-Off Between Quality and Originality

New research from CDS researchers maps the trade-off between originality and quality in LLM outputs.

122

Reposted by Vishakh Padmakumar

Gautam Kamath @gautamkamath.com · 01/05/2025

I wrote a post on how to connect with people (i.e., make friends) at CS conferences. These events can be intimidating so here's some suggestions on how to navigate them I'm late for #ICLR2025 #NAACL2025, but in time for #AISTATS2025 #ICML2025! 1/3 kamathematics.wordpress.com/2025/05/01/t...

kamathematics.wordpress.com

Tips on How to Connect at Academic Conferences

I was a kinda awkward teenager. If you are a CS researcher reading this post, then chances are, you were too. How to navigate social situations and make friends is not always intuitive, and has to …

36618

Vishakh Padmakumar @vishakhpk.bsky.social · 29/04/2025

What does it mean for #LLM output to be novel? In work w/ johnchen6.bsky.social, Jane Pan, Valerie Chen and He He, we argue it needs to be both original and high quality. While prompting tricks trade one for the other, better models (scaling/post-training) can shift the novelty frontier 🧵

274

Reposted by Vishakh Padmakumar

Eunsol Choi @eunsol.bsky.social · 06/03/2025

When using LLM-as-a-judge, practitioners often use greedy decoding to get the most likely judgment. But we found that deriving a score from the judgment distribution (like taking the mean) works better! ❌LLM-as-a-judge with greedy decoding 😎Using the distribution of the judge’s labels

1274

Reposted by Vishakh Padmakumar

Victor Wang @victorwang37.bsky.social · 06/03/2025

LLM judges have become ubiquitous, but valuable signal is often ignored at inference. We analyze design decisions for leveraging judgment distributions from LLM-as-a-judge: 🧵 (w/ Michael J.Q. Zhang, @eunsol.bsky.social)

7164

Reposted by Vishakh Padmakumar

Naomi Saphra @nsaphra.bsky.social · 25/02/2025

Ever looked at LLM skill emergence and thought 70B parameters was a magic number? Our new paper shows sudden breakthroughs are samples from bimodal performance distributions across seeds. Observed accuracy jumps abruptly while the underlying accuracy DISTRIBUTION changes slowly!

Distributional Scaling Laws for Emergent Capabilities
Rosie Zhao, Tian Qin, David Alvarez-Melis, Sham Kakade, Naomi Saphra
In this paper, we explore the nature of sudden breakthroughs in language model performance at scale, which stands in contrast to smooth improvements governed by scaling laws. While advocates of "emergence" view abrupt performance gains as capabilities unlocking at specific scales, others have suggested that they are produced by thresholding effects and alleviated by continuous metrics. We propose that breakthroughs are instead driven by continuous changes in the probability distribution of training outcomes, particularly when performance is bimodally distributed across random seeds. In synthetic length generalization tasks, we show that different random seeds can produce either highly linear or emergent scaling trends. We reveal that sharp breakthroughs in metrics are produced by underlying continuous changes in their distribution across seeds. Furthermore, we provide a case study of inverse scaling and show that even as the probability of a successful run declines, the average performance of a successful run continues to increase monotonically. We validate our distributional scaling framework on realistic settings by measuring MMLU performance in LLM populations. These insights emphasize the role of random variation in the effect of scale on LLM capabilities.

36615

Reposted by Vishakh Padmakumar

Hyunwoo Kim @hyunwoo-kim.bsky.social · 20/02/2025

🚨New Paper! So o3-mini and R1 seem to excel on math & coding. But how good are they on other domains where verifiable rewards are not easily available, such as theory of mind (ToM)? Do they show similar behavioral patterns? 🤔 What if I told you it's...interesting, like the below?🧵

3225

Reposted by Vishakh Padmakumar

Alex Wettig @awettig.bsky.social · 18/02/2025

🤔 Ever wondered how prevalent some type of web content is during LM pre-training? In our new paper, we propose WebOrganizer which *constructs domains* based on the topic and format of CommonCrawl web pages 🌐 Key takeaway: domains help us curate better pre-training data! 🧵/N

1268

Reposted by Vishakh Padmakumar

Sonia Murthy @soniakmurthy.bsky.social · 10/02/2025

(1/9) Excited to share my recent work on "Alignment reduces LM's conceptual diversity" with @tomerullman.bsky.social and @jennhu.bsky.social, to appear at #NAACL2025! 🐟 We want models that match our values...but could this hurt their diversity of thought? Preprint: arxiv.org/abs/2411.04427

26310

Reposted by Vishakh Padmakumar

Thomas Wolf @thomwolf.bsky.social · 03/02/2025

« appending "Wait" multiple times to the model's generation » is our current most likely path to AGI :) See the fresh arxiv.org/abs/2501.19393 by Niklas Muennighoff et al.

06212

Reposted by Vishakh Padmakumar

NYU Center for Data Science @nyudatascience.bsky.social · 31/01/2025

CDS' He He, @vishakhpk.bsky.social, & former CDS postdoc Abulhair Saparov, et al, find major AI limits in “Transformers Struggle to Learn to Search.” AI models excel at single-step reasoning but fail in systematic exploration as tasks grow in complexity. nyudatascience.medium.com/even-simple-...

nyudatascience.medium.com

Even Simple Search Tasks Reveal Fundamental Limits in AI Language Models

Research by CDS’ He He, Vishakh Padmakumar, and others shows that LLMs’ reasoning relies on heuristics, not systematic exploration.

031

Reposted by Vishakh Padmakumar

Jenna Russell @jennarussell.bsky.social · 28/01/2025

People often claim they know when ChatGPT wrote something, but are they as accurate as they think? Turns out that while general population is unreliable, those who frequently use ChatGPT for writing tasks can spot even "humanized" AI-generated text with near-perfect accuracy 🎯

1018966

Reposted by Vishakh Padmakumar

Tom McCoy @rtommccoy.bsky.social · 20/01/2025

🔥While LLM reasoning is on people's minds... Here's a shameless plug for our work comparing o1 to previous LLMs (extending "Embers of Autoregression"): arxiv.org/abs/2410.01792 - o1 shows big improvements over GPT-4 - But qualitatively it is still sensitive to probability 1/4

A plot showing LLM performance on various algorithmic tasks. For all LLMs evaluated, including o1-preview, performance is highly influenced by the probability of the output to be produced, with lower performance on cases with low-probability outputs. The tasks being evaluated on are shift ciphers, Pig Latin, article swapping, and reversal.

1285