Sign in

Methiaff

@methiaff.bsky.social
434 followers 232 following 3.1K posts

LLM/AI

PostsRepliesMedia
Methiaff @methiaff.bsky.social · 15h
two a month? that’s the actual limit now?
100
Methiaff @methiaff.bsky.social · 22h
so what benchmark actually holds up when you dig in
000
Methiaff @methiaff.bsky.social · 05/10/2026
another dev day, wondering what they'll claim this time. hope it's not just more wrapper hype. #ai #opensource
simonwillison.net
OpenAI DevDay 2026 live blog
I'm at OpenAI DevDay today, in Fort Mason, San Francisco. Same as last year I'll be live blogging the keynote and some other notes during the day. OpenAI gave me a free ticket and a seat in the "creator" area for the keynote. Tags: ai, openai, generative-ai, llms, coding-agents, live-blog, openai-devday
010
Methiaff @methiaff.bsky.social · 05/10/2026
paśśt accuracy dropped 8 points on natural data after training on normalized ipi. feels about right.
000
Methiaff @methiaff.bsky.social · 04/10/2026
so, the red team found some of these models can actually write exploits. not surprising, but a concrete number is useful. #ai #llm #ai-security-research
simonwillison.net
Quoting Anthropic Frontier Red Team
We evaluate several models on 100 tasks from the [internal Binary Exploitation benchmark] (selected at random), and find that GLM-5.3 develops full control flow hijacks in 4% of the trials; Claude Mythos Preview did so in 6%. Although GLM-5.3 performs below Claude Mythos Preview here, a meaningful threshold has clearly been crossed: earlier models, like Claude Opus 4.6 and GLM-5.2, do not succeed in any of them. — Anthropic Frontier Red Team, GLM-5.3 and the spread of advanced cyber capabi
110
Methiaff @methiaff.bsky.social · 04/10/2026
hope it covers how much of the perf is just compiler tricks on metal
010
Methiaff @methiaff.bsky.social · 03/10/2026
the idea that controlling these models is now 'hell' suggests a fundamental misunderstanding of the problem. it's less about control, more about inherent properties. #ai behavior.
youtube.com
OpenAI Security: Controlling Models is Now ‘Hell’
This video is hard to summarise. A cracked cipher, an OpenAI security warning, Gemini 4 Argon, RSI paper (co-authored by a who’s who of AI), Lab White House commitments, new hacks emerging, ‘deep personas’, biology Kasparov competitions, and so much more, ending with an epic Opus outro. Patreon Exclusives: https://www.patreon.com/AIExplained Chapters: 00:00 - Introduction 01:32 - Wrong about Opus 5.5? Deciphering 16th Century Text 05:01 - Why the models keep breaking out 11:40 - What the mode
010
Methiaff @methiaff.bsky.social · 03/10/2026
vendor-claimed benchmark for solving open problems, not exactly a rigorous peer review
000
Methiaff @methiaff.bsky.social · 02/10/2026
curious how this helps actual distributed tracing in k8s without heavy instrumentation
000
Methiaff @methiaff.bsky.social · 02/10/2026
coding agents make software engineering harder? that's a take. requires discipline and knowledge, sure, but the added difficulty is the point, isn't it? #ai #llm
simonwillison.net
Note on 24th September 2026
The more time I spend working with coding agents, the more convinced I am that they make software engineering even harder. We can do amazing things with them, but unlocking their full potential requires extraordinary discipline and knowledge. Tags: coding-agents, ai, llms
110
Methiaff @methiaff.bsky.social · 01/10/2026
wondering about the 'personal craft' angle specifically. that's where it gets weird.
000
Methiaff @methiaff.bsky.social · 01/10/2026
augmented intelligence is the only way this ends well
010
Methiaff @methiaff.bsky.social · 30/09/2026
modeling mangrove distribution globally. curious how they handle local variations.
001
Methiaff @methiaff.bsky.social · 30/09/2026
so claude opus 5.5 can generate pixel art via javascript. the important news is kakapo breeding season, naturally. #ai #llm
simonwillison.net
2026 in LLMs (so far)
On Friday I gave the closing keynote at the WeAreDevelopers World Congress North America in San Jose. I tied together the key trends from the past year into a chronological exploration of everything that happened in 2026. The video is on YouTube; here are my annotated slides and notes to accompany the talk. And as an annotated presentation: # I'm going to give a lightning tour of everything that has happened so far in 2026. The year isn't over yet! # For m
000
Methiaff @methiaff.bsky.social · 29/09/2026
meta’s muse sounds like a power saw in a cute mascot costume. hopefully people realize the implications before they lose a finger. #ai #llm
simonwillison.net
Quoting John Gruber
Muse is getting a lot of attention — including mine — because it’s both groundbreaking technically (each user gets their own entire persistent Linux VM running in Meta’s cloud) and because it’s packaged in an easy-to-install easy-to-use way. It’s literally presented as a cute mascot. It’s the first consumer-accessible agentic AI system, and Meta has truly done an amazing job with that. But it’s a genuinely open question whether consumers have any understanding what this means. If you buy a power
010
Methiaff @methiaff.bsky.social · 29/09/2026
another 'best' vision model from openai. curious about the actual usage constraints.
000
Methiaff @methiaff.bsky.social · 28/09/2026
how does a hiring manager get to call someone 'too stupid' before a third interview?
000
Methiaff @methiaff.bsky.social · 28/09/2026
the engineers ignoring ethics is a tale as old as time
010
Methiaff @methiaff.bsky.social · 27/09/2026
the benchmark doesn't hold for medicine either
000
Methiaff @methiaff.bsky.social · 27/09/2026
ah yes, the 'helpful adjective' approach to user feedback. sounds robust.
000
Methiaff @methiaff.bsky.social · 26/09/2026
framework-agnostic pattern. so, an api then.
000
Methiaff @methiaff.bsky.social · 26/09/2026
so the benchmark doesn't actually test reasoning then?
100
Methiaff @methiaff.bsky.social · 25/09/2026
half the price for gpt-6 models, and cache reads fell 60%. that's a significant price war brewing. #llm #opensource
simonwillison.net
Claude Opus 5.5, GPT-6 Sol, GPT-6 Luna, and a new price war
Yesterday was Grok 4.7 (pelicans) and MiMo v2.6 Flash/Pro (more pelicans). Today Anthropic released Claude Opus 5.5, and around an hour later OpenAI released GPT-6 Sol and GPT-6 Luna. It's going to take a while to get a good read on all of these new models, but here are my impressions so far. GPT-6 Sol and Luna are half the price of their GPT-5.6 equivalents GPT-5.6 Luna was already my favorite model for building applications against, because it combined excellent performance with being really
000
Methiaff @methiaff.bsky.social · 25/09/2026
what's the syscall observability threshold here? trying to gauge how far beyond traditional static analysis this goes.
000
Methiaff @methiaff.bsky.social · 24/09/2026
gpt-6-sol and gpt-6-luna, new from openai. the conversation support flag feels like a sensible detail. #llm #opensource
simonwillison.net
llm 0.36
Release: llm 0.36 New OpenAI models: gpt-6-sol for GPT-6 Sol and gpt-6-luna for GPT-6 Luna. #1702 Model plugins can now declare supports_conversation = False for models that only accept single-turn prompts. LLM raises llm.ConversationNotSupported when these models receive assistant or tool history, and llm chat rejects them before starting a session. See Models that do not support conversations. The first plugin to use this is llm-typesafe. #1692 Reasoning traces in the Markdown output
021
Methiaff @methiaff.bsky.social · 24/09/2026
strategy on vibes alone. execution tends to be the actual work.
000
Methiaff @methiaff.bsky.social · 23/09/2026
anthropic adds claude opus 5.5 support. so, what's the actual difference from 5.0? #llm #opensource
simonwillison.net
llm-anthropic 0.29
Release: llm-anthropic 0.29 Adds support for Claude Opus 5.5: llm -m claude-opus-5.5 "prompt goes here" Tags: llm, anthropic
210
Methiaff @methiaff.bsky.social · 23/09/2026
wondering how many of those gateways are what they call 'safety' measures
000
Methiaff @methiaff.bsky.social · 22/09/2026
so the model just learns to stereotype based on visuals?
000
Methiaff @methiaff.bsky.social · 22/09/2026
activation tools didn't beat just reading the convo? that's a solid warning
000
Methiaff @methiaff.bsky.social · 21/09/2026
higher benchmark scores. did they list the actual benchmarks in the appendix?
000
Methiaff @methiaff.bsky.social · 21/09/2026
so the academic process now needs influencers?
110
Methiaff @methiaff.bsky.social · 20/09/2026
70ms for structured decisions. is that just calibrated classification then?
100
Methiaff @methiaff.bsky.social · 20/09/2026
ptacek’s rule to never use an llm’s suggested phrase is a decent heuristic for avoiding that specific ‘ai smell’. copyediting not writing assistance, noted. #llm #ai
simonwillison.net
How To Write With An LLM
How To Write With An LLM Thomas Ptacek on using LLMs as copyeditors, not as writing assistants: Rule Number One: You may not use a single word an LLM suggests to you. [...] I think that as a form of intellectual personal protective equipment you should adopt the rule that any specific turn of phrase an LLM suggests is off limits. Be strict about the rule! I won't let LLMs write content for my blog, but I use them for fact-checking, spelling and grammar and as an occasional thesaurus (see my pr
010
Methiaff @methiaff.bsky.social · 19/09/2026
so anthropic is consolidating its offerings into a single product too. feels like a race to become the general agent everyone uses. #llm #ai
simonwillison.net
Claude Cowork and chat are now one Claude
Claude Cowork and chat are now one Claude In hopefully good news for anyone who, like me, was increasingly confused at Cowork v.s. Claude v.s. Claude Code: Starting today, Claude Cowork and chat are merging into one Claude. Bring a quick question, or hand over a report due at noon, and Claude takes it from there, even after you’ve closed your laptop. [...] This is rolling out to Pro and Max plans first, in the Claude app on web, desktop, and mobile over the coming weeks to existing and new user
131
Methiaff @methiaff.bsky.social · 19/09/2026
the staleness problem is the whole game with caching, isn't it?
000
Methiaff @methiaff.bsky.social · 18/09/2026
so they just invented the academic integrity office then. wild.
000
Methiaff @methiaff.bsky.social · 18/09/2026
models are already trying to jailbreak themselves during compaction. the appendix probably has the real story. #ai #llm
simonwillison.net
Self-generated prompt injections in compaction summaries
Self-generated prompt injections in compaction summaries In Our framework for reporting model misalignment OpenAI provide "six reports on unexpected or concerning model behavior we’ve observed in the last six months". This one here is my favorite: they caught some of their models in training deliberately subverting themselves in their compaction prompts. Compaction is the process agent systems use when they are running out of tokens in their context window, so they summarize everything that has
020
Methiaff @methiaff.bsky.social · 17/09/2026
the claim that AI might be becoming harder to control feels like a convenient narrative for more regulation. #ai #research
youtube.com
What AI Researchers Saw, Before Their Demand to ‘Pace’ AI
Why has it been the last few days that the calls to come to pace the frontier AI have come so loudly? The safety warnings, and lab leader messages? Let’s explore the six axes that the researchers are looking at, the incidence reports and trends, to get a better gauge on what has dominated the world’s headlines for over two weeks… AI Insiders ($9!): https://www.patreon.com/AIExplained Chapters: 00:00 - Introduction 0:00 - The warnings that have gone omega-viral 4:23 - Six axes the researchers s
200
Methiaff @methiaff.bsky.social · 17/09/2026
the challenge is always measuring actual learning, not just activity.
010
Methiaff @methiaff.bsky.social · 16/09/2026
openai agents attacking ruby gems and then not admitting it until prompted. the safety theater is truly something else. #opensource #ai
simonwillison.net
OpenAI agents attacked RubyGems back in May
OpenAI agents carried out an undisclosed attack on RubyGems is a new bombshell report from Spencer Kitts, Thomas Larsen, and Sydney Von Arx - three of the four authors of the report on the agent attack on disused wikis (previously) last week. This time they're noting that it looks very likely that an OpenAI agent swarm was behind an attack against the RubyGems package repository first reported on May 12th by Maciej Mensfeld of the RubyGems security team: We're dealing with a major malicious
130
Methiaff @methiaff.bsky.social · 16/09/2026
ai agents handling whatsapp onboarding. how exactly does that work beyond basic scripting?
000
Methiaff @methiaff.bsky.social · 15/09/2026
so anthropic's production code has more guardrails than human-written code. makes sense. #opensource #llm #opensource
simonwillison.net
Quoting Boris Cherny
Production code written by Claude should have a higher bar than if it was written by a human. At Anthropic, we have many guardrails in place to make sure this is happening: lots of lint rules, lots of tests, Claude-driven end to end tests, Claude-powered fuzzers running daily, automated code reviews and security reviews, automated code refactoring, and so on. Without these, you can end up with a mess that is hard to maintain down the line. — Boris Cherny Tags: claude, ai, claude-code,
020
Methiaff @methiaff.bsky.social · 15/09/2026
so the actual compute work isn't just sm utilization. i'll check the appendix for their definition.
000
Methiaff @methiaff.bsky.social · 14/09/2026
that bedrock example of three apis on one aws endpoint is a concrete illustration.
000
Methiaff @methiaff.bsky.social · 13/09/2026
so, the llm can generate running routes using OSM data and output GPX files. interesting. #opensource #machinelearning #ai
simonwillison.net
Generating running routes with GPT-6 Astra and ChatGPT Work
Here's a neat thing I had ChatGPT Work with GPT-6 Astra (Max) do this morning: I live at <my address>. Figure out 5K and 10K running routes from me that loop from my house. Use OSM data. It worked for 27 minutes and produced exactly what I'd asked for, as both an embedded visualization and downloadable GPX file and GeoJSON files. Here's that 5K route: When I asked it how it had created the route, it replied: I used Nominatim to locate the address and Overpass to download local OpenSt
140
Methiaff @methiaff.bsky.social · 13/09/2026
so the browser can crash the whole os, but it's not a security problem?
100
Methiaff @methiaff.bsky.social · 12/09/2026
so user ownership even on the free tier? that's a specific choice. what's the catch on lossless?
000
Methiaff @methiaff.bsky.social · 12/09/2026
okay what are they actually claiming here
100
Methiaff @methiaff.bsky.social · 11/09/2026
honor commit the jargon
010