pivot-to-ai.com
You can hardly get online these days without hearing some AI booster talk about how AI coding is going to replace human programmers. AI code is absolutely up to production quality! Also, you’re all fired.
But if AI is so obviously superior … show us the code. Where’s the receipts? Let’s say, where’s the open source code contributions using AI?
This week’s AI coding hype came courtesy Satya Nadella, CEO of Microsoft, and Mark Zuckerberg, CEO of Facebook, furiously talking up AI programming. The headlines said that 30% of code at Microsoft was AI now! Huge if true. [_e.g,,__TechCrunch_]
Of course, Nadella didn’t quite say that. What he actually said was: [_YouTube_ _, 45:00-45:08_]
> maybe 20 to 30 percent of the code that is inside of our repos today in some of our projects are probably all written by software.
Maybe “20 to 30 percent”? In “some projects”? At least three or four, who knows! “Written by software”? Lots of projects use code generators. “Probably?” He’s not sure either.
This is a good example of CEO weasel wording — where you make a very hedged claim that’s barely claiming _anything_. Then you let your media cheerleaders misrepresent your very particular claim for you. _You_ didn’t lie!
Nadella would have asked his staff for something, anything, he could say to pump AI coding with. And that’s the _best_ claim they could find that wasn’t technically a lie.
Mark Zuckerberg said, on Dwarkesh Patel’s podcast, talking specifically about engineering at the Llama LLM project — not anything else at Facebook or Meta: [_Dwarkesh Patel_ _, 12:51-13:02_]
> I would guess that sometime in the next 12 to 18 months, we’ll reach the point where most of the code that’s going toward these efforts is written by AI. And I don’t mean autocomplete.
Of course, Patel’s YouTube headline was “AI Will Write Most Meta Code in 18 Months.” [_YouTube_]
Let’s assume the boosters are right, and AI coding is so good now that it really will substantially take over the job of programming on live Facebook code that makes money and live Microsoft code that makes money.
So why don’t we see it in places we can actually check?
Open source code is developed out in the open. You can see all the code, you can see every individual change, you can read the project discussions.
Some of the best minds in programming work in open source. If AI coding really is just better, they’d be using it and recommending it.
We should see AI coding all through open source — it’s the one place we won’t have to rely on some AI booster saying “our great AI code is proprietary, but … trust me, bro.”
Ben Evans, currently at Red Hat, is a Java programmer. He’s written several popular books on Java. [_O’Reilly_]
Ben asked the obvious question: where are the pull requests? [_LinkedIn_ _,__Mastodon_]
> Share some AI-derived pull requests that deal with non-obvious corner cases or non-trivial bugs from mature F/OSS projects. I’ll also accept high-quality documentation that isn’t just the sort of wasted space and slop that I always tell juniors not to write.
>
> No talk, no rhetoric. I won’t respond to any comment that doesn’t have a link to an accepted & merged PR that was produced by an AI model.
The closest response, over on lobste.rs, was one contribution to the Rails project in 2023, and that needed work before it was up to scratch. [_GitHub_]
It wasn’t a response to Ben, but there’s also one experiment with AI coding in the Servo web browser project. That went through 113 revisions before it was acceptable. The first version included sloppy errors like a check condition being the wrong way around. [_GitHub_]
The general comments that Ben received were that experienced developers can use AI for coding with positive results because they know what they’re doing. But AI coding gives *bad* results when it’s used by an inexperienced developer. Which is what we knew already.
Some replies to Ben’s question — “show me the pull requests” — complained that this question sets an _unfairly_ high bar — ”non-obvious corner cases or non-trivial bugs” in “mature projects.” [_Lobste.rs_]
I don’t buy that. If all these big companies are shouting from the rooftops that AI is up to production code the money relies on, then _zero_ open source contributions of substance is a glaring absence.
Some projects give the bots a serious tryout. The Cockpit project tested GitHub Copilot and sourcery.ai’s ability to do bot reviews of human-submitted pull requests. It didn’t work very well: [_Mastodon_ _;__PiWare_]
> TL/DR: a lot of noise, a lot of bad advice, and not enough signal, so we switched it off again.
>
> … About half of the AI reviews were noise, a quarter bikeshedding.
The bots gave a lot of “nitpick suggestions” or ones that were unfounded or even damaging to the codebase. They simply weren’t helpful.
It’s true that a lot of open source projects really hate AI code. There’s several objections, but the biggest one is that users who don’t understand their own lack of competence spam the projects with time-wasting AI garbage. The Curl project banned AI-generated security reports because they were getting flooded with automated AI-generated “bug bounty” requests. [_LinkedIn_]
More broadly, the very hardest problem in open source is not code, it’s people — how to work with others. Some AI users just don’t understand the level they simply aren’t working at.
One user of the LLVM compiler complained that his AI-generated pull requests were not being taken seriously — by a compiler project, where correct computer science and knowing precisely what the heck you’re doing is profoundly important.
The user considered it was the unpaid volunteer coders’ “job” to take his AI submissions seriously. He even filed a code of conduct complaint with the project against the developers. This was not upheld. So he proclaimed the project corrupt. [_GitHub_ _; Seylaw,__archive_]
This is an actual comment that this user left on another project: [_GitLab_]
> As a non-programmer, I have zero understanding of the code and the analysis and fully rely on AI and even reviewed that AI analysis with a different AI to get the best possible solution (which was not good enough in this case).
You can see why people don’t really want to deal with this sort of contribution. But maybe we’ll get a flood of obviously excellent AI code — and AI code submitters — _next_ year.
* _Video version_
### Share this post:
* * * Click to share on Reddit (Opens in new window) Reddit
* Tweet
* Click to share on Mastodon (Opens in new window) Mastodon
* Click to share on Bluesky (Opens in new window) Bluesky
* Click to email a link to a friend (Opens in new window) Email
*
### Like this:
Like Loading...
### _Related_
