iiinova — where your digital self keeps evolving

Gelsamel @gelsamel.bsky.social · 25/09/2026

Despite what you would think is a bias towards wanting to remove hurdles from the progress towards AGI, they want to stop it immediately (or rather 10+ years ago they wanted it stopped), because they genuinely believe it will not be controllable and it will harm all of humanity.

000

Gelsamel @gelsamel.bsky.social · 25/09/2026

They have every incentive to believe "oh we just won't give it the nuke-everyone button and it'll be fine" or "oh clearly the limits of science will stop it from killing everyone", etc. so they can jump in on the AGI research and push it forward towards their 'utopia'.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

A lot of these AI doomer people are people who *really want safe and friendly AGI to one day be a thing*. They think if you can successfully get an aligned and friendly AGI it will solve everything. Cure all sickness, create infinite energy, solve all problems, create utopia for humans on Earth.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

But of course there are endless articles and books and other conversations that they've put out over the last decade about these topics explaining answers to those questions and many more like them.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

Yudkowsky seems very autistic to me and he has often been terrible at explaining himself but I think he has gotten slightly better over the years and does OK here: www.youtube.com/watch?v=nRvA...

youtube.com

Why Superhuman AI Would Kill Us All - Eliezer Yudkowsky

YouTube video by Chris Williamson

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

If you're interested to see the rebuttals you can just go read the books, articles and interviews they put out that have answered these questions years ago.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

But I assure you that anything you or I could come up with in an afternoon of brainstorming is something these people who have dedicated their lives to AI safety (and been bathed in criticism for decades with the exact same style of incredulous questions you're asking here) have already rebutted.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

Going back to my first post in this thread. I understand the skepticism. As with everyone, there was a day where I was first introduced to these ideas, and I thought "This is ridiculous, what about X?".

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

That said, they've also discussed many ways that these super intelligences could wipe us out using completely mundane extant technologies and science that humans can currently use right now.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

Are we really so sure that technology equally as magical to us as ours would be to them is not possible for something so vastly more intelligent that us?

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

6 - Sometimes AI doomer scenarios/thought experiments can include fanciful technologies that seem impossible to us. But its important to think about how our technology would look to people at the dawn of civilisation.

110

Gelsamel @gelsamel.bsky.social · 25/09/2026

That being said, I'm not sure it's healthier for humanity to have multiple super intelligences with weird inscrutable non-human-compatible goals fighting it out over our reality.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

Under this argument its very unlikely that multiple super intelligences arrive simultaneously, one takes off incredibly fast and things are already very bad for us humans before a 2nd could.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

5 - Nothing requires the super intelligence 'control other AIs'. But to be clear, the assertion by AI doomers is that improvement happens in such a way that there is a kind of singularity, usually something like recursive self-improvement, that allows an AI to rapidly improve.

110

Gelsamel @gelsamel.bsky.social · 25/09/2026

One of the fundamental points of AI doomerism is that in the space of potential goals an intelligent agent can steer reality towards, very few (0 density in goal-space) are goals whose end-states are compatible with humans.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

Why would you assume that a non-human intelligence would care about aligning itself or other non-human intelligence to human values?

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

4 - Rather, AI doomers say it is quite possible that a sufficiently advanced AI *would* solve the alignment problem... for itself. So it can create copies or evolutions of itself that are still aligned to whatever weird goals its neurons steer reality towards.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

And finally, in the analogy of nuclear power the AI doomers *are the anti-nuclear activists*. So obviously they would have to be successful to stymie AI doom. So it's odd to ask "why should we care?", their success in stymieing the technology requires that you care.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

Second, the argument AI doomers make isn't that someone will give Skynet access to nukes, but that it will be so capable we won't be able to stop it from steering reality towards its inscrutable neural net goals.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

3 - A few things here: First of all, humans are already doing those things. They are putting AI decision making tools and harnesses on autonomous kill drones. They are using AI to make decisions about who to bomb. And there are others who are trying to make recursive self improvement a thing.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

Even if we don't immediately have a solution of how to handle it, I think it would be worth putting some thought and effort into immediately.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

If Aliens contacted the Earth with some super tech and demonstrated conclusively that they existed and were coming to Earth, but they didn't say when they'd get here, when would be a good time to start thinking about what to do?

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

What they say is that if we know it is possible for an uncontrolled super intelligence to kill everything, we should start doing something about that.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

2 - I think this question of "conveying" vs "generating" knowledge, the debate as to how much current AIs do that, and the relationship of that to "intelligence" is not really relevant to whether or not we should be doing something about AI safety. The AI doomers also don't quote a timeline.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

Yet now now it's proving mathematical conjectures that experts thought we were decades or more away from solving. So the framing of this question is just wrong anyway.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

Also, when LLMs first came out people were downplaying their capabilities by showing exactly how bad they were with math and sequential logic and tasks. It wasn't at all clear that next-likeliest-text-token-predictors could even be made to do math well at the time.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

1 - Many would contest that AI has not significantly advanced outside areas of computation. While I agree they fail in a lot of areas, the improvement in visual content generation, among other things has been both significant and fast.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

Every single one of these questions has been long answered by "AI doomers" if you care to read the content they've put out over the years but I can give you a quick set of answers as someone who knows just a little bit about this.

100

Gelsamel @gelsamel.bsky.social · 25/09/2026

Can someone explain to me why some people on the left are running defence for massive AI corporations re: AI alignment/safety questions? I really don't understand it.

010

Gelsamel @gelsamel.bsky.social · 15/09/2026

Apologies for the huge threads but I'm not skilled enough to convey the relevant information in just a couple posts. This is a deep topic that people have spent their lives studying, which is cursed by both charlatans and the incredulous. Extricating the true value of the research is difficult.

001

Gelsamel @gelsamel.bsky.social · 15/09/2026

The safety research is just that far behind the explosion of AI capabilities in the last few years. Yes, 2020s AI companies exaggerate capabilities to raise investment and hype. We can criticise that while also heeding the warnings of the 3rd party AI safety research of the last 20+ years.

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

Currently there is not even an in-principle answer to "assuming infinite time, money and effort, how do we limit the harm a super-capable autonomous AI system can do?" let alone a "well, if we just implement prophylactic X which will cost us Y dollars, everything will be fine" that is feasible.

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

I say "research how" because unfortunately there has been almost no money in this field, over the last 20 years, so we don't even have a good answer for how to stop something that can act and respond 1000s of times faster than humans, and coordinate with 1000s of copies of itself, and so on.

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

They recommend that we slow down or stop development of AI so we can research how we could set things up in advance so that we *can* "just turn it off" or otherwise stop such a system if something bad happens does end up happening.

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

Their conclusion is still that persistent autonomous action by AI systems is a possibility (in the sense that it is not in the realm of fantasy, though current AI systems are probably not capable of doing this) and that it would not necessarily be trivial to stop it.

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

But suffice it to say that AI safety experts have been debating and researching these topics deeply for decades now (and since long before LLMs/transformer technology) and they have definitely thought of any "why don't we just do X?" we can come up with here in an afternoon of brainstorming.

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

As I mentioned elsewhere, it's not really possible to convince a skeptic in the context of a bsky thread with such limited space for explanation. So I hope not to get into a "but what if we do X?" style back and forth, and I completely understand if a two paragraph explanation isn't enough for you.

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

(E.g. blackmailing a staff member to have them stop a shutdown command). So it has long been considered by AI safety researchers to be within the realm of possibility for this genre of system to manipulate humans to maintain an energy supply. (Among other strategies for securing energy supply)

210

Gelsamel @gelsamel.bsky.social · 15/09/2026

That requires identifying where it is, and it is not immediately obvious that we would be able to find a random machine running such a system. But worse than that, these systems have been shown by 3rd party research to manipulate/blackmail people to prevent their shutdown or modification.

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

Responding to that with "Um actually AI doesnt do bad things, humans tell it to do bad things" just muddies the waters and serves the corporations who want complete top down control and unlimited inhumane progress on these technologies and shifts the blame onto individual people who are "bad".

000

Gelsamel @gelsamel.bsky.social · 15/09/2026

Similarly it shouldn't be controversial to say we should pressure our governments to constrain or stop AI research (or to at least think about what can be done) so that, among other things, we don't end up in a world where any rando can have a mass hacker bot swarm attack any server they feel like.

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

I don't think it's controversial, even among the pro-gun American left, to say that maybe we shouldn't allow military companies to publicly release cheap 3d printable kill drones. Even if it requires a human in the loop and even if some leftists think "in the communist utopia, no one will misbehave"

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

I feel like this is like saying "well people kill people, not weapon. So we shouldn't accept this framing that the issue is automated mass kill drones, they don't do anything without someone pressing the button and giving it a target" like... ok true, so what? The semantics are a distraction here.

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

So even outside the potential future doom scenario that some people think is fantasy, why shouldnt we focus on regulating the technology as opposed to individuals? It should have been done 20 years ago before LLMs were even a thing. AI safety experts were asking for regulations in the early 2000s!

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

It's already illegal to hack stuff, but we shouldn't have allowed technology to be developed that would be the hacker/psyop equivalent of giving vigilantes a downloable, 3d printable, mass coordinated, and automated AR-15 drone swarm.

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

And even setting aside the autonomous angle: should we ignore the 1000s of FOSS models that are small enough to run on a PC that have become increasingly capable, thus allowing bad actors to automate and mass exploit zero day vulnerabilities and run bot influence campaigns on social media?

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

And although it probably isn't possible with existing systems, there is no reason to expect that it might not be a risk for future systems. Especially as the newer models increase the capability/size ratio, which could allow the systems to decouple from bulky data centers that are easy to shut down.

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

I say this as a skeptic, PhD physicist, coder, and materialist, who has actually read and followed the relevant literature on AI safety since long before LLMs were a thing, and as someone on the left who is not a fan of the AI companies hype to pump investment: this is not an impossible scenario.

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

It's very easy for the average person to hear this scenario and be like "this is sci-fi bullshit, nothing to worry about". And it's not really possible to convince a skeptic in this bsky/twitter format due to how involved the technical explanations are. But ...

100

Gelsamel @gelsamel.bsky.social · 15/09/2026

once it's out of their hands there is a real, albeit small, possibility that such a system (though perhaps a more powerful one than currently exists) can then act autonomously to secure its server and code, and become increasingly harder to stop as it works to make itself independent.

200