iiinova — where your digital self keeps evolving

Oh sorry, I was attempting to mock the typical retort but I see that this was quite poorly executed on my part

140

Ah, the old "not all men" debate

100

People want less than we (programmers) sometimes imagine, but definitely more than they're offered by today's mainstream computing platforms

010

Not going to speculate on US politics either. I also think "it's a tool" is also a bit too simplistic. But I'm a software engineer, so I prefer explanations of *how things work*, because that tells you more accurately where you can intervene in the system, rather than "machine god kill us all".

Rob Knight @root.guru · 25/09/2026

Yes, but all of those practical problems have mechanical explanations. There are real things we can do to mitigate them, up to and including shutting down systems which can’t be safely operated. Leading with “10% chance we all die” just provokes resistance to the whole idea, even if it might be true

100

I don’t know why we don’t call them “agencies”, the word is right there and it’s a better analogy with the human organisation we use for software outsourcing!

020

Rob Knight @root.guru · 25/09/2026

I must admit that excerpt doesn’t make much sense to me. I’ll admit that I haven’t read a lot of Yudkowsky (life is too short) but I did spend a while debating this stuff with various MIRI folks since early 2023. Possibly they were adapting their story to what was emerging at the time, though.

030

Rob Knight @root.guru · 25/09/2026

This does not seem right to me at all. The claim was that we have no means of being certain that the AI has internalised the morality, which is to say that we can't be sure that the AI aligns its outputs to that morality (hence "AI alignment").

Rob Knight @root.guru · 23/09/2026

Not that I would prefer the ethnic variety, but I do think civic Englishness is relatively poorly defined. But maybe everyone feels that way about their own civic nationality?

150

But even there, I think the aim has to be to win consensus among people who do not share your world-view, or your science fiction reading list. A mechanical explanation of the problems and why they're hard to fix still feels like a better rhetorical device than emotive fictions.

100

I can just about see the argument for the emotional appeal if your aim is to ban AI entirely. In that sense, I have some sympathy with the genuine doomers, who do - so far as I can tell from personal conversations - authentically believe in some variant of a doom hypothesis.

130

The full-frontal emotional assault produces the opposite result: it encourages resistance to the idea that there is any problem at all, because people will deny nightmare scenarios in order to avoid thinking about them (cf. climate change).

100

But, on the other hand, I think the "machine god" and "P(doom)" rhetoric is totally unhelpful, because it obscures more than it reveals. It doesn't explain the mechanism, and if you want to hook smart people on solving a problem, you want to explain the mechanism.

100

I have complicated feelings about this. On the one hand, AI risks are real, AI alignment is a real problem with no certain solution. We should be careful in how we use it. We should also be extremely curious about alignment and control of AI systems - this is important work!

Even our smartest SIs couldn't have anticipated this

Rob Knight @root.guru · 21/09/2026

I keep having to remind myself that this really happened

A graph showing the trend of UK real GDP per capita, with a negative £10,900 gap between 2023 and the continuation of the pre-2008 trend line, roughly 25% of 2023's real GDP per capita.

050

Reposted by Rob Knight

Grace @gracekind.net · 19/09/2026

Frog built a wet lab for the AI model. "There," he said. "Now it can do its own experiments." "What the fuck?" said Toad

13817129

2. Control therefore has some trade-off with capability. It has to fail closed, disallowing anything that cannot be proven safe. This is a much smaller category than things which are practically safe, and excludes a lot of economically valuable stuff.

There are broadly two reasons for focusing on alignment over control, though: 1. Controlling a misaligned AI has a variety problem, in that the model has a lot more of it than the control system. It's very hard to control for "the model emitted persuasive messages to the human operator".

220

Worth noting that there *are* people within the AI safety community who talk about "AI control", e.g. www.redwoodresearch.org/research/ai-... ARIA's "Safeguarded AI" is explicitly a "control, not alignment" agenda and might be the largest effort in this direction aria.org.uk/opportunity-...

171

Rob Knight @root.guru · 15/09/2026

The final series of Picard was one in which everyone under 25 was infected with mind control that causes them to form a hive mind, which could only be defeated by the boomer Enterprise crew in their network-free vintage spaceship. And you want to make it *more* anti-woke?

1314

This is a bit nit-picky, because people often conflate the model, the harness, the “agent”. The latter term is confusing because it has different meanings in computer science, economics, law, moral philosophy, and so on.

150

Rob Knight @root.guru · 15/09/2026

The model can produce source code as an output, but the harness is the part which, for instance, saves it to disk and runs the compiler. In isolation the model produces almost no effects directly.

342

Rob Knight @root.guru · 14/09/2026

There’s a point where he has to change the agent’s setting to make it less obsequious!

000

Rob Knight @root.guru · 14/09/2026

True, but he definitely saw something very much like this coming anyway. “My name is Tom, and I’m your agent. May I ask if you know what that means?” “You want 15%?”

youtu.be

Douglas Adams - Hyperland

YouTube video by UJ ObscureMedia

Shiladitya Banerjee @shilabanerjee.bsky.social · 15/05/2026

Reposted by Rob Knight

1/ Can a single bacterium learn? No brain, no neurons, just one cell deciding what to do next. Our new paper says yes, and the cell's internal machinery turns out to compute exactly like a recurrent neural network. 🔗 journals.aps.org/prxlife/abstract/10.1103/5zbg-8vll

2426

Rob Knight @root.guru · 13/09/2026

The most obvious choke point is requiring ID/licenses to use the internet. Agents can have their own IDs, subject to approved use of a model of that class for a specific purpose. Use which exceeds the purpose leads to loss of license? (see, you need a British person to think of these things)

knowyourmeme.com

Oi, You Got A Loicense For That, Mate? | Know Your Meme

Oi, You Got A Loicense For That, Mate? refers to a catchphrase and slang term parodying Britain and the greater UK's reputation for overregulation. The mem

041

Rob Knight @root.guru · 13/09/2026

To be fair, computers are much more locked-down now than they were in the late 90s/early 00s, and so if you wanted to make it impossible to transfer model weights over non-approved channels, perhaps you could do so, but to really *prevent* it would be highly invasive.

Rob Knight @root.guru · 13/09/2026

Oh, indeed. I am a regular lurker on a forum in which he is a ghostly presence, banned but not forgotten.

000

But there is something different which I overlooked: if the LLM was trained on these kinds of scenarios, then coordination between the agents is not totally dependent on communication, and the logs or even the context will not (necessarily) reveal the LLM's model of the situation.

000

Whereas this describes something quite different - an illusion of independent processes, whereas in reality they are being coordinated by a central system *at runtime* rather than simply sharing the same repertoire.

I suppose I'm thinking like someone trying to debug a distributed system, in which case I'd want to start by looking at the logs, and would think in terms of the interactions between path-dependent processes.

Rob Knight @root.guru · 11/09/2026

Hmm. This seems wrong to me. I'd make the model analogous to the program, the agent analogous to the process, and the context analogous to the state of that process. In such a metaphor there are clearly multiple processes, each with different state, even if the program is the same.

100

See, what we need here is fully automated drone defences for the data centres, controlled by an AI inside the data centre

010

I think if you just mean “SF tech people” or “tech people involved in the latest hype cycle somehow” I can see what you mean but I think that lumps a bunch of people together in a way that leads to being wrong about the specific question of whether these people truly believe in AI risk.

100

I think we have a different sense of who “these people” are. The AI safety people, in my experience, don’t have much overlap with crypto people. I know some of both and, by and large, they don’t have much in common. There are exceptions but surprisingly few.

100

I simply don’t agree, in this case. You can say they’re wrong in their conclusions and I’d respect that, but there’s just no justification for assuming that people have nothing in their lives beyond merely responding to financial incentives. That’s just bleak.

100

jon ben-menachem @jbenmenachem.com · 09/09/2026

They might be wrong but they're not lying about what they believe.

100

Reposted by Rob Knight

It may be under-discussed that the advent of LLMs has surfaced two thorny philosophical problems and combined them in the same tangle: the chicken and egg question of language and thought, and the fact that theories of human cognition have historically mirrored technological advances.

1387

Not sure I'd go full Stafford Beer here but I am definitely reminded of this

"Forgive my audacity, please, but I have been “in” computers right from the start. I can tell you flatly that they do not make mistakes. People make mistakes. People who program computers make mistakes; systems analysts who organize the programming make mistakes; but these men and women are professionals, and they soon clear up their mistakes. We need to look for the people hiding behind all this mess; the people who are responsible for the system itself being the way it is, the people who
don’t understand what the computer is really for, and the people who have turned computers into one of the biggest businesses of our age, regardless of the societary consequences. These are the people who
make the mistakes, and they do not even know it. As to the ordinary citizen, he is in a fix—and this is why I wax so furious. It is bad enough that folk should be misled into blaming their undoubted troubles
onto machines that cannot answer back while the real culprits go scot free. Where the wickedness lies—and wickedness is not too strong a word—is that ordinary folk are led to think that the computer is an
expensive and dangerous failure, a threat to their freedom and their individuality, whereas it is really their only hope"

- Stafford Beer, Designing Freedom

171

Distributed cognition - Wikipedia

But we do think with external feedback loops! This is what we invented logic *for*! We rely on distributed cognition to help us to reason, and we're not capable of much without it. I agree that we learn from feedback in a way that LLMs do not. Post-training is a form of learning but not the same.

en.wikipedia.org

000

I think option B is pretty much why the guy in the quoted post quit his job, and why the person quoting him is supporting his reasons for doing so.

010

I'm not sure who you think is lying to whom here, or which part is the lie