Sign in

Kenny Peng

@kennypeng.bsky.social
252 followers 261 following 123 posts

CS PhD student at Cornell Tech. Interested in interactions between algorithms and society. Princeton math '22. kennypeng.me

PostsRepliesMedia
Kenny Peng @kennypeng.bsky.social · 06/10/2026
This is a collaboration with @nkgarg.bsky.social and Jon Kleinberg. See the paper here: arxiv.org/abs/2610.05794 I’m also on the academic job market this year. If you find this work exciting, would love to chat more!
arxiv.org
A Testable Theory of Atomic Features
We develop and test a theory of language model representations in which there exist atomic features. Our main theoretical insight is that in such a model, sparse dictionaries (e.g., SAEs) of increasin...
010
Kenny Peng @kennypeng.bsky.social · 06/10/2026
I’m excited for the role of theory in understanding AI (playing a similar role as in theoretical physics). Here, we were able to take a simple postulate, mathematically derive implications that give testable predictions, and then show that these predictions hold in experiments.
100
Kenny Peng @kennypeng.bsky.social · 06/10/2026
Our results suggest the promise of atomic theory as a scientific theory (one where we start with postulates, derive predictions, and then conduct experiments). It also suggests the promise of scaling SAEs: there is no trade-off between coarse and granular features!
100
Kenny Peng @kennypeng.bsky.social · 06/10/2026
There is a lot more in the paper, including a molecular view of SAE features. We can think of non-atomic learned SAE directions as molecules: mixtures of atomic directions. This seems to help explain phenomena like polysemanticity, splitting, and semantic clusters of directions.
100
Kenny Peng @kennypeng.bsky.social · 06/10/2026
How is this consistent with feature splitting results? We find instances where features split at intermediate widths, but return at sufficiently large widths (with sufficiently large sparsity parameter k).
100
Kenny Peng @kennypeng.bsky.social · 06/10/2026
Testing P3, at all levels of ground-truth hierarchies, feature recovery is best SAEs at width 65K or 131K (averaging over features and models, here).
100
Kenny Peng @kennypeng.bsky.social · 06/10/2026
Testing P2, SAEs trained on different distributions share many features. This can be compared with PCA, where very few features are shared across training data.
100
Kenny Peng @kennypeng.bsky.social · 06/10/2026
Testing P1, many features in small SAEs persist in *all larger* SAEs. ~70% of feature directions at size-4K also recovered in sizes 8K, 16K, 32K, 65K, and 131K (cosine sim. 0.7). This holds when looking at activations, and also when using an exclusive matching criteria.
110
Kenny Peng @kennypeng.bsky.social · 06/10/2026
We empirically test these on SAEs ranging from size 512 to 131K trained on two large embedding models. We find that the predictions hold!
100
Kenny Peng @kennypeng.bsky.social · 06/10/2026
In particular, for a fixed distribution, SAEs of increasing size learn a growing prefix of the most prevalent atoms in the distribution (this holds even for hierarchical correlation!) This yields *testable predictions*:
100
Kenny Peng @kennypeng.bsky.social · 06/10/2026
There is a lot of reason to be skeptical of atomic features: SAEs vary in what features they recover (depending on size, training data, random seed). Our core insight is the recovery principle: different SAEs learn different features, but they do learn subsets of atomic features.
100
Kenny Peng @kennypeng.bsky.social · 06/10/2026
Our new paper introduces a scientific theory of atomic features. We mathematically derive testable predictions. Our experiments challenge conventional wisdom. SAEs of different size and training data share many features. Large SAEs recover both parent and child features. 🧵
1207
Reposted by Kenny Peng
Wesley voted in Vancouver's Municipal Election @wesleyfinck.org · 26/08/2026
Some preliminary product insights from our recent onboarding launch: It's no coincidence that the most popular suggested link is "From Feeds to Trails". People are ready for a new kind of social media experience! leaflet.pub/p/did:plc:ld...
leaflet.pub
From Feeds to Trails
To design the future of social media, rethink the interface before the algorithm.
3255
Reposted by Kenny Peng
Erica Chiang @ericachiang.bsky.social · 19/08/2026
Thanks to Grace Stanley for covering our work :) Our paper on personalized recommendations for students applying to NYC high schools (best student paper at EC ’26(!!)) is coming soon! Here's a quick preview... (1/5) news.cornell.edu/stories/2026...
t.co
https://news.cornell.edu/stories/2026/08/research-helps-nyc-students-aim-higher-public-high-school-applications
1114
Kenny Peng @kennypeng.bsky.social · 19/08/2026
Before deploying the nudges, the first step was to carefully quantify the problem and assess what the best intervention would be (the focus of our newly-published work)
nature.com
Connecting application behavior to undermatching in New York City school choice - Nature Cities
Cities are challenged to allocate educational resources efficiently and equitably. This study of the 2022–2023 New York City High School Match program finds that Black and Hispanic students undermatch...
020
Kenny Peng @kennypeng.bsky.social · 19/08/2026
See Erica's thread for more. One key challenge was to nudge students without causing congestion (if everyone applies to the same school, they won't all get in).
130
Kenny Peng @kennypeng.bsky.social · 19/08/2026
Here's an article about our broader effort to help students navigate the NYC school match. The big payoff has been to deploy real nudges (led by @ericachiang.bsky.social!). As a result, actual students this year matched to high-quality schools they wouldn't have applied to otherwise!
news.cornell.edu
Research helps NYC students aim higher in public high school applications | Cornell Chronicle
Many New York City eighth-graders – particularly those from underserved communities – aren’t applying to academically competitive high schools where they could succeed, according to new Cornell resear...
180
Kenny Peng @kennypeng.bsky.social · 18/08/2026
This work is also joint with Jon Kleinberg, Eva Tardos, and @nkgarg.bsky.social. We benefited enormously from close collaboration with NYC Public Schools, as well as from help by the Cornell Center for Social Sciences, who helped us securely work with highly sensitive student data. Open-access pdf:
nature.com
090
Kenny Peng @kennypeng.bsky.social · 18/08/2026
We started this work more than three years ago. Seeing the paper published is bittersweet. My wonderful partner on this project, Emily Ryu, passed away last year. The paper is dedicated in her memory.
180
Kenny Peng @kennypeng.bsky.social · 18/08/2026
In follow-up work led by @ericachiang.bsky.social , we successfully deployed these interventions in collaboration with NYC Public Schools. A preliminary version of the paper won best student paper at EC this year.
150
Kenny Peng @kennypeng.bsky.social · 18/08/2026
Overall, our results suggest that for NYC schools to equitably serve students, we need to focus on application side behavior alongside admissions policies. The strongest interventions nudge competitive students to reach higher in early positions.
150
Kenny Peng @kennypeng.bsky.social · 18/08/2026
There is heterogeneity in what the model recommends. Black and Hispanic students would benefit more relatively from reaching higher in earlier positions. White and Asian students would benefit more relatively from listing a safer option later. There is substantial heterogeneity within groups.
Recommended changes to portfolios differ substantiallyacross demographic groups. For example, the dark blue rectangle in therecommendations for Hispanic students indicates that about 3,000 suchstudents would be most helped (concerning match offer rate in expectation)by changing their first-ranked program to one that has about 50 percentagepoints lower offer rate. Asian and white students are relatively more likely tobenefit from applying to less selective programs later in their list, whereas Blackand Hispanic students would primarily benefit from applying to more selective programs early in their lists.
161
Kenny Peng @kennypeng.bsky.social · 18/08/2026
Recommendations made by the model, if adopted, would reduce individual undermatching and close disparities.
Undermatching and potential intervention effects for offer rate acrossrace/ethnicity groups. The bar chart plots the mean. The error bars show 95%nonparametric bootstrapped confidence interval.
161
Kenny Peng @kennypeng.bsky.social · 18/08/2026
We use a model of optimal application behavior to identify the sources of undermatching. For example, a student can undermatch by not listing a safety school, or by not listing a reach school.
Example ranked lists and recommended changes, as an illustration ofour personalized feedback approach. In lists 1 and 4, the model recommendsthe student apply to a less selective last choice, as a ‘safety program’. The x axisindicates position in the ranked list. In list 2, the model recommends a moreselective second choice, as a ‘reach’ or ‘match’ program. In list 3, the modelrecommends a less selective second choice. The optimal change is a function ofthe applicant’s entire ranked list. These changes are calculated using an algorithmthat finds the optimal perturbation to a portfolio using a theoretical model.
170
Kenny Peng @kennypeng.bsky.social · 18/08/2026
Undermatching is larger for Black, Hispanic, and lower income students. The gap is largest among the most competitive students.
Mean undermatching by lottery number (lower is better) and offerrate of the most selective program they counterfactually would have beenadmitted to (that is as close as their actual match), by race and ethnicity. Themost competitive (by grades, lottery numbers or other admission criteria) Blackand Hispanic students are especially undermatching compared to their Asian andwhite counterparts, indicating that they are not applying as aggressively as theycould have. Similar trends hold when measuring competitiveness by applicantgrade tier (Extended Data Fig. 9). Error bars show 95% nonparametric bootstrapconfidence interval.
160
Kenny Peng @kennypeng.bsky.social · 18/08/2026
We measure undermatching: the difference in school quality between where a student matched and where a student could have matched counterfactually. To do this, we simulate exact counterfactuals using student data and admissions criteria.
Map of NYC by zip code, showing the geographic distribution ofmatch offer rate and undermatching. Left: the average match offer rate forstudents who live in that zip code. Right: the average amount of undermatching.Undermatching varies substantially by geography. Geographic patterns ofmatch offer rates and undermatching distinguish between geographic accessto selective programs and application behavior; some zip codes have 2× higherundermatching than a neighboring one. For example, students in Riverdale(northwest corner of the map) match to relatively high offer rate (less selective)programs (around 0.65), but have little undermatching (around 0.1 difference from counterfactual). This may reflect its relatively remote but affluent status.In the rest of the Bronx (north end of map), students not only match to higheroffer rate programs on average but also experience much more undermatching.Meanwhile, in downtown Manhattan (bottom tip of the island, middle left),students on average match to selective programs (left plot is largely blue), butthey could have matched to even more selective programs, given the high densityof such programs. For privacy reasons, zip codes with 20 or fewer applicants areomitted.
170
Kenny Peng @kennypeng.bsky.social · 18/08/2026
Some background: school choice in NYC in theory allows students to attend any school, overcoming geographic segregation. In practice, inequity remains a major issue. One reason is admissions policy (see below), but we focus on another: information complexity.
nytimes.com
170
Kenny Peng @kennypeng.bsky.social · 18/08/2026
New York City 8th graders choose from 900+ high schools to apply to, in a process that’s spawned Facebook groups and dozens of expensive consulting services. Our new paper shows how application behavior leads to disparities, and how to effectively intervene. 🧵 www.nature.com/articles/s44...
1329
Kenny Peng @kennypeng.bsky.social · 18/08/2026
I've been using the embedding model, which seems solid. Can't really speak to the LLMs though
010
Reposted by Kenny Peng
Benjamin Laufer @laufer.bsky.social · 04/08/2026
I am delighted to share that I'll be joining the University of Washington Information School as an Assistant Professor!
University of Washington campus
2442
Reposted by Kenny Peng
Maria Antoniak @mariaa.bsky.social · 30/07/2026
Wonderful talk by @sjgreenwood.bsky.social about @paper-feed.bsky.social! Custom feeds on Bluesky are unique resources for researchers interested in recommendation algorithms #IC2S2 Featuring many Bluesky friends like @graze.social @devingaffney.com @kissane.myatproto.social @aendra.com
24412
Kenny Peng @kennypeng.bsky.social · 30/07/2026
🫨
000
Reposted by Kenny Peng
Benjamin Laufer @laufer.bsky.social · 21/07/2026
One intuition behind many AI policy proposals is that downstream AI applications -- the companies deploying AI in healthcare, finance, education, customer service, etc. -- should bear responsibility for ensuring safety. Our paper asks: What incentives does that create for the firms building AI?
Illustrative example of our game-theoretic model. This numeric instance of the game consists of one general-purpose producer and three domain-specialists. Each player has a different utility in performance-safety space which dictates the path of development. Within this setting, the no-regulation game (Upper Left) reveals the players’ investment efforts when no floor is imposed on safety. Regulating the domain-specialist alone (Upper Right) exhibits backfiring for all three domains, meaning the regulated safety level is lower than it would be without regulation. In this particular example, the same floor is assumed for all three domain-specialists. Regulating the generalist alone (Lower Left) improves the safety level slightly across all three domains, compared to no-regulation. Finally, a regime that targets both generalists and specialists with regulation (Lower Right) is able to 1) retain the improved safety performance from regulating the generalist, 2) improve the safety level of least-safe domain-specialist, while 3) avoiding backfiring. The purpose of this figure is to visualize the model’s incentive mechanisms; none of these panels represent real regulations.
193
Kenny Peng @kennypeng.bsky.social · 07/07/2026
ah, thanks, arXiv version now shows the most updated version so you can refer to that: arxiv.org/abs/2506.23845
arxiv.org
Position: Use Sparse Autoencoders to Discover Unknowns
While sparse autoencoders (SAEs) have generated significant excitement, a series of negative results have added to skepticism about their usefulness. Here, we establish a conceptual distinction that r...
020
Kenny Peng @kennypeng.bsky.social · 06/07/2026
Here's the arXiv link: arxiv.org/abs/2506.23845
arxiv.org
Use Sparse Autoencoders to Discover Unknown Concepts, Not to Act on Known Concepts
While sparse autoencoders (SAEs) have generated significant excitement, a series of negative results have added to skepticism about their usefulness. Here, we establish a conceptual distinction that r...
010
Kenny Peng @kennypeng.bsky.social · 06/07/2026
openreview.net/pdf?id=x1Px7... (the arXiv draft is slightly outdated but the update should be posted tomorrow)
openreview.net
Verifying your browser | OpenReview
Please complete the verification above.
330
Kenny Peng @kennypeng.bsky.social · 06/07/2026
This work was co-led by @rajmovva.bsky.social , and with Jon Kleinberg, @emmapierson.bsky.social , and @nkgarg.bsky.social Stop by our poster to chat!
020
Kenny Peng @kennypeng.bsky.social · 06/07/2026
Check our @rajmovva.bsky.social 's earlier thread for details! One difference from our previous draft: we leave open the possibility that SAEs can also be a useful tool for acting on knowns (there is new evidence for this) bsky.app/profile/rajm...
110
Kenny Peng @kennypeng.bsky.social · 06/07/2026
🧵Can we reconcile excitement for SAEs with negative results? Our #ICML2026 position paper argues that even if SAEs underperform baselines when acting on knowns (e.g. probing, steering), they're a powerful tool for ~discovering unknowns~ Poster: Tue 2pm, HALL A #1715
2125
Reposted by Kenny Peng
Ira Globus-Harris @iraglobusharris.bsky.social · 03/07/2026
Are you at ICML next week? Feel like your decision-making for which sessions to attend might not be risk minimizing? Don't incur (swap) regret and come to my, @aaroth.bsky.social, and @ncollina.bsky.social's tutorial Monday on multicalibration, decision-making, and collaborative learning!
22011
Kenny Peng @kennypeng.bsky.social · 30/06/2026
Presenting this at #COLT2026! Let me know if you want to chat about (1) the linear representation hypothesis, (2) theoretical foundations for interpretability / science of AI. - Talk: Thursday 2:33 pm - Deep Learning Theory / Optimization - Poster: Thursday Lunch)
081
Kenny Peng @kennypeng.bsky.social · 22/06/2026
I enjoyed making some ruler-and-compass constructions of some of the mentioned triangle centers. CCW from bottom left: the centroid, 1st Napoleon point, Feuerbach point, circumcenter, Exeter point, 2nd Napoleon point, and incenter.
020
Kenny Peng @kennypeng.bsky.social · 22/06/2026
I wrote the essay in 2020, for a creative non-fiction class taught by John McPhee, whose editing I benefited enormously from. I've published it here essentially unchanged (with a short addendum including an update from Clark Kimberling).
110
Kenny Peng @kennypeng.bsky.social · 22/06/2026
In college, I wrote this essay about Clark Kimberling, who maintains the Encyclopedia of Triangle Centers. The essay covers triangle geometry's rise (which includes Napoleon), its fall (from "brute force" methods), and its continued following ("the melody lingers on")
leaflet.pub
Coincidence of Lines
The mathematician Clark Kimberling estimates that he spends between two and five hours each day, seven days a week, maintaining his online encyclopedia of triangle centers.
161
Reposted by Kenny Peng
Nikhil Garg @nkgarg.bsky.social · 11/05/2026
We are very excited to announce our first workshop on From Theory to Practice: behind the scenes on research deployments at EC’26 (July 6 in Rome)! Call for posters and submissions now open! Organized by myself, @ericachiang.bsky.social , Bailey Flanigan, @brwilder.bsky.social
sites.google.com
Home
About This workshop will focus on the practical realities of deploying algorithmic and economic systems from academic research, especially with government and non-profit partners. While economics and ...
0216
Kenny Peng @kennypeng.bsky.social · 30/04/2026
Thanks, that’s very nice! And glad you like the idea.
010
Kenny Peng @kennypeng.bsky.social · 27/04/2026
Better than I 🤷‍♂️ I got from SPEAKER to ZERO in 15 steps in today's Traversle puzzle! Play here: traversle.io
traversle.io
Traversle | Find the path between words
Start from one word, choose a connected neighbor each turn, and try to reach the target before you run out of steps.
020
Reposted by Kenny Peng
spacecowboy @spacecowboy17.bsky.social · 26/04/2026
Two Bluesky client apps have integrated the "also liked" functionality: - @boostblue.bsky.social - iOS and Android - @skywalker.thereforeiam.eu - Android The app integration makes it a much smoother experience than the stand-alone website foryou.club/also-liked Here is how it looks in Boost Blue:
3647
Kenny Peng @kennypeng.bsky.social · 24/04/2026
More broadly, we've been interested in using the internal representations of language models to build information structures that allow human navigation and agency.
040
Kenny Peng @kennypeng.bsky.social · 24/04/2026
This was led and built by Nicholas Goguen-Compagnoni, an MEng student at @cornelltech.bsky.social. This is part of some ongoing work we're doing with Jon Kleinberg and @nkgarg.bsky.social
130