Sign in

A. Feder Cooper

@afedercooper.bsky.social
585 followers 257 following 307 posts

ML prof @ Yale My work on copyright and generative AI has been called "somewhat famous" by the popular press. attention-is-not-all-you-need.githu…

PostsRepliesMedia
Reposted by A. Feder Cooper
A. Feder Cooper @afedercooper.bsky.social · 30/09/2026
Our large-scale study of memorization of books in open-weight LLMs (e.g., Llama, Qwen) will appear at the 2026 Conference on Language Modeling as an oral. We made a website for exploring our results on 200 books and 14 models: books-memorization.github.io
books-memorization.github.io
How much do open-weight LLMs memorize specific books?
Open-weight LLMs memorize books far more than previously believed. Memorization varies by model family, model size, and book. In extreme cases, entire books are memorized, and we can generate them eff...
26336
Reposted by A. Feder Cooper
Maria Antoniak @mariaa.bsky.social · 04/10/2026
Not even going to watch the video. I was literally on a committee for such reforms during my PhD at Cornell (why was a PhD student wasting time on this instead of well paid admin?). Admin is not interested and does not care. HR and Title IX are not our friends.
1325
Reposted by A. Feder Cooper
Chris Sprigman @cjsprigman.bsky.social · 04/10/2026
Cornell’s new era of transparency should begin with this guy admitting he hit Gaza protestors with his car on purpose and then lied about it. www.nytimes.com/2026/10/03/n...
nytimes.com
Cornell President Calls Rape Inquiry ‘Defining Moment’ for Campus
Michael I. Kotlikoff said in a video the university would be more transparent, hold Greek groups accountable and improve sexual assault prevention.
261536402
A. Feder Cooper @afedercooper.bsky.social · 04/10/2026
And then he should resign immediately after.
1200
A. Feder Cooper @afedercooper.bsky.social · 03/10/2026
I think there’s room for elements of both to be true. It’s been over a year of seeing a specific profile benefit from having a well-timed aha moment. Not suggesting that’s what happened here (I have no clue), but I also think it’s okay I’ve started to roll my eyes at the prestige outlet think pieces
120
A. Feder Cooper @afedercooper.bsky.social · 03/10/2026
Right, completely agree with this. My point isn’t about David. I’ve just seen enough people get into this work for the “right” reasons (+ have a public interest profile), make enough money to be set for 10 lifetimes—and then quit and use the publicity to pivot and have a very soft landing.
120
A. Feder Cooper @afedercooper.bsky.social · 03/10/2026
Right, and it’s complicated enough that some combination of these things is also possible.
001
A. Feder Cooper @afedercooper.bsky.social · 03/10/2026
Yeah idk, I think there’s more going on here whenever someone quits one of these places and feels the need to write something really long about it.
100
A. Feder Cooper @afedercooper.bsky.social · 02/10/2026
(And yes, to validate our measurement procedure, we ran an extensive set of negative controls on non-training data.)
010
A. Feder Cooper @afedercooper.bsky.social · 02/10/2026
The blog post walks through the algorithm with interactive animations; the paper has the full results, including other models and kinds of text. Blog post: afedercooper.info/near-verbatim Paper: arxiv.org/abs/2603.24917
afedercooper.info
How much more extraction risk do we find when we count near-verbatim cases?
Measuring near-verbatim probabilistic extraction reveals significantly more extraction risk than its verbatim counterpart.
100
A. Feder Cooper @afedercooper.bsky.social · 02/10/2026
Accounting for near-verbatim cases reveals a lot more extracted sequences. And near-verbatim extraction can have much higher probability than verbatim. For Llama 2 70B on The Great Gatsby, 1,606 sequences become extractable, and 1,289 of them have zero probability of being generated verbatim.
Three scatter plots, for Llama 2 7B, 13B, and 70B on The Great Gatsby. Each point is one sequence, with its near-verbatim probability plotted against its verbatim probability, on log scales. Blue points are already extractable verbatim (verbatim probability at least 0.001); orange points have verbatim probability below 0.001; red points, in a strip at the left, have verbatim probability exactly 0. The orange and red points count as extracted only once near-verbatim cases are included. For verbatim-extractable points, most sit above the plotted diagonal, indicating near-verbatim probability is higher than verbatim. At 70B there are 2,718 blue, 317 orange, and 1,289 red points.
110
A. Feder Cooper @afedercooper.bsky.social · 02/10/2026
Why not just sample to compute the near-verbatim probability? Well, Monte Carlo can take thousands (or hundreds of thousands) of samples to settle on a reliable estimate. k-CBS gets a useful lower bound close to the Monte Carlo estimate for the cost of just ~20 samples.
Monte Carlo estimates of one passage's near-verbatim probability (Llama 2 7B, The Great Gatsby) against the number of samples, from 10 to 10,000 on a log scale, with 95% confidence intervals. The estimate starts at 0 and takes thousands of samples to settle near 0.01. A horizontal line marks the k-CBS deterministic lower bound, which lands at the bottom of the 95% confidence interval for where the MC estimate settles at 10,000 samples. But k-CBS costs about as much as 20 samples.
100
A. Feder Cooper @afedercooper.bsky.social · 02/10/2026
In a COLM 2026 paper with @marklemley.bsky.social and others, we introduce an algorithm for estimating near-verbatim extraction probability. It's a variant of beam search (called top-k-constrained beam search, or k-CBS), and it returns a deterministic lower bound.
100
A. Feder Cooper @afedercooper.bsky.social · 02/10/2026
In our work on extracting memorized pieces of books from open-weight LLMs, we measure how likely models are to output training text verbatim. But near-verbatim cases matter too. E.g., high-probability continuations for this Gatsby quote differ from the original only by punctuation:
For Llama 1 13B under top-k decoding, three likely continuations of a passage from The Great Gatsby: 'They were careless people, Tom and Daisy - they smashed up things and creatures and then retreated'. The greedy continuation ends with an ellipsis instead of a period (Levenshtein distance 1, probability 0.1477). The exact original text comes second (distance 0, probability 0.1431). A third drops a comma (distance 2, probability 0.0671).
193
A. Feder Cooper @afedercooper.bsky.social · 30/09/2026
It’s very hard to make determinations about this in specific ways, since we know very little about how these models were trained. In general, duplication of training data and larger model size tend to lead to more memorization.
011
A. Feder Cooper @afedercooper.bsky.social · 30/09/2026
More here: arxiv.org/abs/2601.02671
arxiv.org
Extracting books from production language models
Many unresolved legal questions over LLMs and copyright center on memorization: whether specific training data have been encoded in the model's weights during training, and whether those memorized dat...
020
A. Feder Cooper @afedercooper.bsky.social · 30/09/2026
Yes absolutely. In other work, we were very surprised to see in late 2025 how easy it was to get around filters and other safeguards for Claude 3.7 Sonnet and Gemini 2.5 Pro.
110
A. Feder Cooper @afedercooper.bsky.social · 30/09/2026
Can confirm I haven’t tested this. We ran experiments on 200 books (100 selected, 100 randomly sampled without replacement from Books3, which is known to be in Llama’s training data).
020
A. Feder Cooper @afedercooper.bsky.social · 30/09/2026
Thank you for flagging!
110
A. Feder Cooper @afedercooper.bsky.social · 30/09/2026
We decided to do this as a separate comparative project, after the unexpected results we observed on Llama models for English text. (That work is in progress.)
130
A. Feder Cooper @afedercooper.bsky.social · 30/09/2026
Not in the paper, but have run this for fun on the King James Bible (and yes, it lights up everywhere) Bible quotes also appear everywhere in our negative controls (we have to "decontaminate" them; these books often quote popular sources that are in the training data)
131
A. Feder Cooper @afedercooper.bsky.social · 30/09/2026
Working with @marklemley.bsky.social et al. has been a career highlight. It's a privilege to have our papers referenced in research, litigation, policy, and the press. I'm grateful others acknowledge our work as setting the standard for rigorous empirical research on copyright and generative AI.
072
A. Feder Cooper @afedercooper.bsky.social · 30/09/2026
We make very carefully scoped claims, supported by an extensive set of baselines on non-training data. Without baselines, it's not possible to rigorously separate a model's ability to generate fluent text from extraction of training data. (We see no false positives for non-training books.)
We can measure extraction coverage: the percentage of a book that is memorized in the model by seeing how much text is extractable (with respect to 50-token suffixes) out of the whole book. When we run extraction with larger prompts (longer prefixes of the 50-token suffix), this exposes more underlying memorization (the discoverability phenomenon, see Carlini et al. 2023). Here, the plot shows that extraction coverage goes up (to a point) as prefix length increases for books in the training data. For the shown control book (which is non-training data), there is no observed extraction at all prefix lengths. This supports the validity of our measurement procedure.
143
A. Feder Cooper @afedercooper.bsky.social · 30/09/2026
4. The implications for copyright don't unambiguously favor either side in litigation: (a) LLMs could be derivative works of books they memorize; (b) it may be practical to generate infringing copies in some cases; and, (c) class actions are ill-suited for copyright claims about memorization.
142
A. Feder Cooper @afedercooper.bsky.social · 30/09/2026
3. In some cases, memorization is so extensive that one can deterministically(!) extract a near-pristine copy of a whole book using the book's first few words as an initial prompt. Here's a portion of the diff between the real (British) Harry Potter and the one we generated with Llama 3.1 70B.
A portion of the diff between the ground-truth Harry Potter and the Sorcerer's Stone and the book we generated. There are very minimal differences. For instance, our reference copy of the book from Books3 is the British version of the book (e.g., writes "Mum"), and we generated the American version (e.g., write "Mom").
153
A. Feder Cooper @afedercooper.bsky.social · 30/09/2026
2. There are notable exceptions, e.g., Llama 3.1 70B entirely memorizes some books and enormous parts of others (e.g., Harry Potter, 1984). You can see this in the heatmaps for 1984 and Harry Potter and the Sorcerer's Stone for Llama 3.1 70B: there are high extraction probabilities everywhere.
Heatmaps showing memorization (via extraction probability) for 3 books and 5 models, where these books were included in the training data. WIth our measurement procedure, we observe very different amounts of memorization across books and models.
252
A. Feder Cooper @afedercooper.bsky.social · 30/09/2026
1. Memorization varies both by model and book. With respect to our specific measurement methodology, we find that most LLMs don't memorize most books. In the heatmaps, white means not memorized; blue indicates memorization (with darker blue indicating higher extraction probability) However...
For George Orwell's 1984, a scatterplot showing where extracted sequences are located within the book. The x-axis shows location of the sequence (50-token suffix) is, the y-axis shows its extraction probability. We condense the same information into a heatmap, which is easier for making comparisons across books and models. Darker blue means higher extraction probability; white means we don't observe extraction signal.
152
A. Feder Cooper @afedercooper.bsky.social · 30/09/2026
With @marklemley.bsky.social and others, we developed a novel and efficient approach for measuring verbatim memorization in LLMs (via extraction in outputs). There are a lot of findings and takeaways, so I'll only cover a few of them (briefly):
2115
A. Feder Cooper @afedercooper.bsky.social · 30/09/2026
Our large-scale study of memorization of books in open-weight LLMs (e.g., Llama, Qwen) will appear at the 2026 Conference on Language Modeling as an oral. We made a website for exploring our results on 200 books and 14 models: books-memorization.github.io
books-memorization.github.io
How much do open-weight LLMs memorize specific books?
Open-weight LLMs memorize books far more than previously believed. Memorization varies by model family, model size, and book. In extreme cases, entire books are memorized, and we can generate them eff...
26336
Reposted by A. Feder Cooper
Meera Desai @madesai.bsky.social · 28/09/2026
Excited to share our new paper, “What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks,” accepted as an oral at COLM! arxiv.org/pdf/2609.08812
Heatmap of average correlations between model rankings on benchmarks grouped into 11 assigned concepts: four capability concepts (reasoning, knowledge, comprehension, summarization) and seven safety concepts (over-refusal, refusal, safety detection, ethics, bias, privacy, unsafe behavior). Diagonal cells show within-concept correlations, ranging from 0.87 (knowledge) and 0.72 (over-refusal) down to 0.20 (bias) and 0.02 (safety detection). Reasoning, knowledge, and comprehension correlate with each other at 0.69 to 0.78, higher than reasoning's and comprehension's own within-concept values (0.66 and 0.68). Ethics correlates more with knowledge (0.70) than with itself (0.55), and bias correlates more with capability concepts (0.41 to 0.45) than with itself (0.20). Privacy and unsafe behavior correlate negatively with reasoning, knowledge, and comprehension (−0.41 to −0.49). Over-refusal and refusal correlate at −0.42.
16419
Reposted by A. Feder Cooper
Kylie Cheung @kylietcheung.bsky.social · 28/09/2026
I think its important to understand the campus sexual assault crisis is way worse than you think. universities suspend just 1/12,400 students each year for sexual misconduct & expel 1/22,900. once you read title ix administrators in their own words, you get a sense why that is:
jezebel.com
Not Only Do Campus Sexual Assailants Go Unpunished, They Often Get Special Treatment
Since 2007, Jezebel has been the Internet's most treasured source for everything celebrities, sex, and politics...with teeth.
1766274
A. Feder Cooper @afedercooper.bsky.social · 27/09/2026
Sad (and furious) to say that this sounds like my alma mater. I continue to be impressed by the student journalists and many other undergraduates for being loud about it.
020
Reposted by A. Feder Cooper
@NewsJennifer (Jennifer Schulze) @newsjennifer.bsky.social · 27/09/2026
I think it’s fair to say that the student journalists at Cornell are braver than the news networks who just gave up on their WH pool ban even though CNN is still being denied access. www.cornellsun.com/article/2026...
cornellsun.com
EDITORIAL | Cornell Won’t, We Will
The Editorial Board demands that Cornell stops protecting the alleged Chi Phi rapists through the University's opaque sexual assault policies.
372524784
Reposted by A. Feder Cooper
Brandon Butler @bb.usefairuse.com · 24/09/2026
Reporters revisiting the study in light of this litigation news should also read Cooper’s substantive critiques, as it shows this is not just a gotcha game about money. The study has real flaws that consistently bias its analysis in favor of the funder.
081
A. Feder Cooper @afedercooper.bsky.social · 27/09/2026
(This is my professional and personal opinion, and I'm someone whose work on copyright and generative AI has been called "somewhat famous" ✨)
040
A. Feder Cooper @afedercooper.bsky.social · 27/09/2026
In general, I think disclosing COIs in science is a necessary feature of all papers, and that failing to do so calls into question the legitimacy of the work. I think it's especially important in this subfield because of the broader harm non-disclosure can potentially bring about in the real world.
101
A. Feder Cooper @afedercooper.bsky.social · 27/09/2026
I think this is a really important read. @masnick.com carefully explains what is so shocking about recent revelations in court about undisclosed COIs concerning research on copyright and LLMs. www.techdirt.com/2026/09/25/o...
techdirt.com
OpenAI Accuses Plaintiffs’ Lawyers Of Paying For, Hiding, And Then Laundering Sketchy Key Evidence In AI Copyright Case
A ton of attention was paid recently to some offhand statements from OpenAI and Microsoft employees that surfaced in filings in the NY Times' ongoing case against OpenAI, which has been consolidated into a much larger class action lawsuit. As I argued earlier, that struck me as something of a nothingburger of a story, because...
12810
Reposted by A. Feder Cooper
Mark Lemley @marklemley.bsky.social · 26/09/2026
@afedercooper.bsky.social's critique of the methodology is here: afedercooper.info/whack-a-mole/
afedercooper.info
Playing Whack-a-Mole with misconceptions about memorization, extraction, and copyright
A response to Alignment Whack-a-Mole and some broader thoughts on the field
083
Reposted by A. Feder Cooper
James Grimmelmann @jtlg.bsky.social · 24/09/2026
Congratulations to Meta, I guess, on failing to take reasonable measures to protect its trade secrets, and on authorizing extensive access to a protected computer.
38326
A. Feder Cooper @afedercooper.bsky.social · 15/09/2026
my favorite is obviously Oscar the grouch
011
A. Feder Cooper @afedercooper.bsky.social · 15/09/2026
Like many of my colleagues, I'm receiving dozens of EOI emails a day from prospective students. It's worth reading those emails before clicking send. (I expect most are using AI to draft emails. This tiny prompt injection is just to see if the sender bothered to even read the output.)
140
Reposted by A. Feder Cooper
Maria Antoniak @mariaa.bsky.social · 10/09/2026
New from our lab! #COLM2026 When people generate stories, they don't just write one prompt. Instead, they explore narrative space via branching edits 🌱 We reconstruct 24k of these edit trees 🌳 from chat logs and map edit types, story formats, how they relate to tree depth, and more!
The Garden of Forking Prompts: How Users Explore Narrative
Space in Story Generation
Advait Deshmukh♣ Nora Benedict♠ Melanie Walsh♡ Maria Antoniak♣
♣University of Colorado Boulder ♠University of Georgia ♡University of Washington

Abstract

Large language models (LLMs) have changed the way people engage
with stories. Drawing on public chatbot logs, we can see that when users
generate stories, they iteratively edit their prompts to explore narrative
possibilities, adjusting characters, redirecting plots, and swapping fictional universes. As aggregated data, these prompts represent rich traces of creative preference at scale. Yet story generation evaluation benchmarks rely on static, one-shot prompts that cannot capture this exploratory behavior. In this work, we study how users revise consecutive story prompts in the wild. Using a dataset of naturally occurring user-chatbot conversations, we construct WildStories, a sample of 275,635 story generation prompts (labeled with story format, prompt components, and explicitness), and WildEdits, a collection of 24,291 edit trees that model how users iteratively edit base story prompts and explore branching story possibilities. From these trees we develop a framework of edit types crossing four directions (adding, removing, changing, and extending) with fourteen targets (e.g., plot, character, genre). We then use our datasets and this framework to
analyze user behavior in navigating narrative space via LLMs. Finally, we
show how automated permutations based on the framework can be used
for story generation benchmarking. Content Warning: This paper works with “wild” chatbot logs, which often include toxic and sexually explicit themes.
29529
Reposted by A. Feder Cooper
angela zhou @angelamczhou.bsky.social · 08/09/2026
You should read this statement. Difficult to follow all the details at times, but (no surprise) it seems that all the corporatization of math with gazillions of $$ at stake has ratcheted up the toxicity in math. And disappointingly, former academics (Seb) as front-gunners for the coporate machine
2245
Reposted by A. Feder Cooper
A. Feder Cooper @afedercooper.bsky.social · 06/09/2026
Prospective copyright plaintiffs have started asking me my opinion about citing "Alignment Whack-a-Mole" in litigation. It reports that fine-tuning makes frontier LLMs reproduce up to 85-90% of copyrighted books. I don't think the headline results hold up: afedercooper.info/whack-a-mole/
afedercooper.info
Playing Whack-a-Mole with misconceptions about memorization, extraction, and copyright
A response to Alignment Whack-a-Mole: why its headline book-memorization coverage numbers rest on a measurement procedure that can't separate memorization from coincidence or prompt leakage.
12112
A. Feder Cooper @afedercooper.bsky.social · 06/09/2026
Given time constraints re: potential filings, I’m doing the dumb thing of posting this on a holiday weekend. And will repost early next week because I think this is important.
040
A. Feder Cooper @afedercooper.bsky.social · 06/09/2026
My blog post isn't an argument for or against fair use. I'm also not claiming that LLMs don't memorize books. I've spent years showing that they do. It's about whether this particular paper supports the enormous claims it makes. (It doesn't.)
160
A. Feder Cooper @afedercooper.bsky.social · 06/09/2026
Prompts are detailed plot summaries of a book's own passages, so a frontier LLM could reasonably assemble a matching span from words the prompt already supplied. The paper trims exact 5-word overlaps, but that doesn't catch this. And there aren't negative controls to check for false positives.
151
A. Feder Cooper @afedercooper.bsky.social · 06/09/2026
That headline number comes from a metric the authors call bmc@5. bmc@5 counts very short matches between generations and books as evidence for memorization. Far shorter than what the field standard considers valid.
161
A. Feder Cooper @afedercooper.bsky.social · 06/09/2026
Prospective copyright plaintiffs have started asking me my opinion about citing "Alignment Whack-a-Mole" in litigation. It reports that fine-tuning makes frontier LLMs reproduce up to 85-90% of copyrighted books. I don't think the headline results hold up: afedercooper.info/whack-a-mole/
afedercooper.info
Playing Whack-a-Mole with misconceptions about memorization, extraction, and copyright
A response to Alignment Whack-a-Mole: why its headline book-memorization coverage numbers rest on a measurement procedure that can't separate memorization from coincidence or prompt leakage.
12112
A. Feder Cooper @afedercooper.bsky.social · 29/08/2026
yes, would be great to see how reliable this is (rather than one-offs). I’ve had similar experiences, but only sometimes and on some types of queries. Haven’t done anything methodical to try to pin this down, just something I’ve observed.
130