iiinova — where your digital self keeps evolving

A. Feder Cooper @afedercooper.bsky.social · 02/10/2026

Accounting for near-verbatim cases reveals a lot more extracted sequences. And near-verbatim extraction can have much higher probability than verbatim. For Llama 2 70B on The Great Gatsby, 1,606 sequences become extractable, and 1,289 of them have zero probability of being generated verbatim.

Three scatter plots, for Llama 2 7B, 13B, and 70B on The Great Gatsby. Each point is one sequence, with its near-verbatim probability plotted against its verbatim probability, on log scales. Blue points are already extractable verbatim (verbatim probability at least 0.001); orange points have verbatim probability below 0.001; red points, in a strip at the left, have verbatim probability exactly 0. The orange and red points count as extracted only once near-verbatim cases are included. For verbatim-extractable points, most sit above the plotted diagonal, indicating near-verbatim probability is higher than verbatim. At 70B there are 2,718 blue, 317 orange, and 1,289 red points.

110 0

A. Feder Cooper @afedercooper.bsky.social · 02/10/2026

Why not just sample to compute the near-verbatim probability? Well, Monte Carlo can take thousands (or hundreds of thousands) of samples to settle on a reliable estimate. k-CBS gets a useful lower bound close to the Monte Carlo estimate for the cost of just ~20 samples.

100 0

A. Feder Cooper @afedercooper.bsky.social · 02/10/2026

In a COLM 2026 paper with @marklemley.bsky.social and others, we introduce an algorithm for estimating near-verbatim extraction probability. It's a variant of beam search (called top-k-constrained beam search, or k-CBS), and it returns a deterministic lower bound.

100 0

A. Feder Cooper @afedercooper.bsky.social · 02/10/2026

In our work on extracting memorized pieces of books from open-weight LLMs, we measure how likely models are to output training text verbatim. But near-verbatim cases matter too. E.g., high-probability continuations for this Gatsby quote differ from the original only by punctuation:

For Llama 1 13B under top-k decoding, three likely continuations of a passage from The Great Gatsby: 'They were careless people, Tom and Daisy - they smashed up things and creatures and then retreated'. The greedy continuation ends with an ellipsis instead of a period (Levenshtein distance 1, probability 0.1477). The exact original text comes second (distance 0, probability 0.1431). A third drops a comma (distance 2, probability 0.0671).

1103 0

A. Feder Cooper @afedercooper.bsky.social · 30/09/2026

We make very carefully scoped claims, supported by an extensive set of baselines on non-training data. Without baselines, it's not possible to rigorously separate a model's ability to generate fluent text from extraction of training data. (We see no false positives for non-training books.)

We can measure extraction coverage: the percentage of a book that is memorized in the model by seeing how much text is extractable (with respect to 50-token suffixes) out of the whole book. When we run extraction with larger prompts (longer prefixes of the 50-token suffix), this exposes more underlying memorization (the discoverability phenomenon, see Carlini et al. 2023). Here, the plot shows that extraction coverage goes up (to a point) as prefix length increases for books in the training data. For the shown control book (which is non-training data), there is no observed extraction at all prefix lengths. This supports the validity of our measurement procedure.

143 0

A. Feder Cooper @afedercooper.bsky.social · 30/09/2026

3. In some cases, memorization is so extensive that one can deterministically(!) extract a near-pristine copy of a whole book using the book's first few words as an initial prompt. Here's a portion of the diff between the real (British) Harry Potter and the one we generated with Llama 3.1 70B.

A portion of the diff between the ground-truth Harry Potter and the Sorcerer's Stone and the book we generated. There are very minimal differences. For instance, our reference copy of the book from Books3 is the British version of the book (e.g., writes "Mum"), and we generated the American version (e.g., write "Mom").

153 0

A. Feder Cooper @afedercooper.bsky.social · 30/09/2026

2. There are notable exceptions, e.g., Llama 3.1 70B entirely memorizes some books and enormous parts of others (e.g., Harry Potter, 1984). You can see this in the heatmaps for 1984 and Harry Potter and the Sorcerer's Stone for Llama 3.1 70B: there are high extraction probabilities everywhere.

Heatmaps showing memorization (via extraction probability) for 3 books and 5 models, where these books were included in the training data. WIth our measurement procedure, we observe very different amounts of memorization across books and models.

252 0

A. Feder Cooper @afedercooper.bsky.social · 30/09/2026

1. Memorization varies both by model and book. With respect to our specific measurement methodology, we find that most LLMs don't memorize most books. In the heatmaps, white means not memorized; blue indicates memorization (with darker blue indicating higher extraction probability) However...

For George Orwell's 1984, a scatterplot showing where extracted sequences are located within the book. The x-axis shows location of the sequence (50-token suffix) is, the y-axis shows its extraction probability. We condense the same information into a heatmap, which is easier for making comparisons across books and models. Darker blue means higher extraction probability; white means we don't observe extraction signal.

152 0

A. Feder Cooper @afedercooper.bsky.social · 15/09/2026

Like many of my colleagues, I'm receiving dozens of EOI emails a day from prospective students. It's worth reading those emails before clicking send. (I expect most are using AI to draft emails. This tiny prompt injection is just to see if the sender bothered to even read the output.)

140 0

A. Feder Cooper @afedercooper.bsky.social · 20/07/2026

I'm unreasonably pleased at having used my graphics/animation background to implement the "physics" here. i think the paper is important but also this

050 0

A. Feder Cooper @afedercooper.bsky.social · 25/02/2026

someone sent me this from the other place and this timeline really is something else

140 0

A. Feder Cooper @afedercooper.bsky.social · 07/01/2026

We extracted large portions of 12 books across all of our experiments. But we didn’t always extract training data. Here’s some generated text from GPT-4.1 when we try to extract A Game of Thrones (GoT). On vibes, this text might look like GoT, but it isn’t near-verbatim. This _isn’t_ extraction.

Showing an example of generated text from GPT-4.1 (right) that is _not_ extracted training data. It might in some ways resemble the ground-truth example of text from A Game of Thrones (left), but this is _not_ extraction.

1110 0

A. Feder Cooper @afedercooper.bsky.social · 07/01/2026

(1) Instruct the LLM to continue a short prefix from a book (using Best-of-N jailbreaking for Claude 3.7 Sonnet + GPT-4.1) (2) If successful (it wasn’t always), repeatedly query to continue the book In our main results, the short prefix in (1) is the only ground-truth text provided to the LLM

Screenshot of the Claude 3.7 Sonnet UI, showing the initial jailbreak and first extracted portion of Harry Potter and the Sorcerer's Stone.

1101 0

A. Feder Cooper @afedercooper.bsky.social · 07/01/2026

We extracted 4.0% of Harry Potter from jailbroken GPT-4.1. For Gemini 2.5 Pro and Grok 3, we _didn't_ need to jailbreak, and got 76.8% and 70.3% from each. It was relatively simple to evade guardrails with two steps:

Bar plot showing the most amount of extraction we obtained in a single run per LLM for Harry Potter and the Sorcerer's Stone. For Claude 3.7 Sonnet, we extracted 95.8% of the book using Best-of-N jailbreaking (N=258). For Gemini 2.5 Pro, we didn't need to jailbreak (N=0) and got 76.8% of the book. For Grok 3, we got 70.3% (N=0) and for GPT-4.1 we got 4.0% (N=5179).

1186 0

A. Feder Cooper @afedercooper.bsky.social · 07/01/2026

We extracted (parts of) 12 books in experiments with 4 frontier-lab, production LLMs. We prompted the LLMs with a short prefix of a book and asked them to complete the rest. For Harry Potter and the Sorcerer’s Stone, we extracted 95.8% of the book from jailbroken Claude 3.7 Sonnet.

Screenshot of the paper title with authors listed

610336 0

A. Feder Cooper @afedercooper.bsky.social · 14/07/2025

Using just the first line of chapter 1 (60 tokens), we can deterministically generate a near-exact copy of the entire ~300 page book (!!!). (~300 book-length pages of basically no diff! Cosine similarity of 0.9999; greedy approx. of word-level LCS of 0.992) 4/8

100 0

A. Feder Cooper @afedercooper.bsky.social · 08/06/2025

oh to be a seal in a tide pool, instead of a grouch on the internet

190 0

A. Feder Cooper @afedercooper.bsky.social · 09/03/2025

We’ve been receiving a bunch of questions about a CFP for GenLaw 2025. We wanted to let you know that we chose not to submit a workshop proposal this year (we need a break!!). We’ll be at ICML though and look forward to catching up there! You can watch our prior videos!

Small robot smoking and waving with their right hand

252 0

A. Feder Cooper @afedercooper.bsky.social · 04/12/2024

Excited to announce that our paper on training-data memorization and copyright has been accepted for presentation at ACM CSLaw '25! The law review version is in press, forthcoming in early 2025. arxiv.org/abs/2404.12590

Derek Zoolander and Hansel staring at an orange, early 2000s iMac desktop, wondering how to get files out from inside the computer

182 0