iiinova — where your digital self keeps evolving

qoop

@qoop.ca

218 followers 285 following 2.5K posts

disillusioned software dev illusioned novelist

Posts Replies Media

qoop @qoop.ca · 04/10/2026

yeah, the authors of the pain paper seem to take this nominal pain a bit too seriously. I wish people like them were less quick to freak out about sentient robots; they should treat LLMs as an opportunity to scrutinize what makes human life meaningful instead

qoop @qoop.ca · 04/10/2026

"coherent pain representation" is a bit of a stumbling block for me as a description, because a purely denotative representation (e.g. the word "pain") would still be coherent.

qoop @qoop.ca · 03/10/2026

i'm actually quite excited for nega-germany. it'll be cool to see how they try to do the opposite of a holocaust

qoop @qoop.ca · 03/10/2026

it's less that English can describe pain & more that there's a particular quality to the things we way say when we are in pain. e.g. "Oh wow, that's—ah! ok, no stop," is not *describing* pain; rather it *sounds pained* for a broader set of examples, see www.anthropic.com/research/emo...

Emotion concepts in a large language model

All modern language models sometimes act like they have emotions. What’s behind these behaviors? Our interpretability team investigates.

qoop @qoop.ca · 02/10/2026

One guy created an "LLM torture chamber" online which basically just cranks up the "pain vector" on a bunch of LLMs to see what they do, and a certain set of twitter guys are freaking the fuck out about it bc they think the pain is real www.404media.co/someone-tort... 2/2

Someone ‘Torturing’ LLMs in a Robot Prison Has Triggered the Dumbest Debate in AI Yet

The "AI Torture Chamber" has opened an upsetting and absurd window into the effective altruist obsession with "model welfare."

qoop @qoop.ca · 02/10/2026

oh those guys exist. so (slight tangent) LLMs model emotions. i.e., there's a "sadness vector" that responds to sad-seeming situations and which, if you increase it artificially, will make the LLM act sad. A recent paper found that, in addition to other emotions, LLMs have "pain vectors" too 1/

qoop @qoop.ca · 30/09/2026

fantasy and romance can both be cloying and fannish, but I have a much easier time finding the parts of fantasy that aren't cloying than I do romance

qoop @qoop.ca · 30/09/2026

Conversely, romance definitionally requires a happy ending. Otherwise it's "women's fiction." There's no space for pushing the envelope, because as soon as you push, you're outside the envelope.

qoop @qoop.ca · 30/09/2026

romance has fully internalized the idea that it shouldn't be challenging contrast with fantasy, which is also fannish & contains a strong current of wish fulfillment. Despite this, the more sugary fantasy stories coexist and intermingle with more complicated works. It's a fairly healthy ecosystem

qoop @qoop.ca · 29/09/2026

ugh I'd expect at least physicists to see past flagrant quantum woo. People have such a hard time letting go of dualism.

qoop @qoop.ca · 28/09/2026

this latest set of replies has disappointed me. I think you're missing some grounding in philosophy & in how language models actually work. the "quantum model" thing in particular was unfortunate. everyone has blind spots, but you should have looked it up instead of assuming i was making no sense

100

qoop @qoop.ca · 28/09/2026

… the "atomic models" thing is YOUR analogy. the model used to teach atoms in grade school is called the "bohr model." the model used by actual physicists is called the "quantum model." bsky.app/profile/bowl...

000

qoop @qoop.ca · 28/09/2026

yeah i agree with this.

000

qoop @qoop.ca · 28/09/2026

i think you're misunderstanding. LLM-based vulnerability analysis is not generally structured like fuzzing, where the bot runs a bunch of adversarial scenarios. Usually the bot simply loads the code into its context and produces output describing problems it found. No trial-and-error, no iteration.

100

qoop @qoop.ca · 28/09/2026

slide with illustrations of a bohr atom vs a quantum-model atom

000

qoop @qoop.ca · 28/09/2026

roger penrose, who i mentioned in an aside, thinks consciousness actually _is_ quantum. his ideas are very silly

qoop @qoop.ca · 28/09/2026

yeah but atoms are quantum. you're talking about the grade-school bohr model of the atom vs the more advanced quantum model of atom

100

qoop @qoop.ca · 28/09/2026

RL is one big reason why bots got so good at programming all of a sudden: programming challenges are an environment where bots can cheaply engage in automated trial & error exploration.

qoop @qoop.ca · 28/09/2026

also worth noting that a lot of LLM training these days is reinforcement learning. Basically, you give a chatbot a programming challenge and reinforce its thinking only when it solves the challenge.

qoop @qoop.ca · 28/09/2026

And I think that many parts of the behaviour associated with the idea of understanding are behaviours that LLMs exhibit too, so I think it's a mistake to stake too much on a low-resolution term like understanding.

100

qoop @qoop.ca · 28/09/2026

If I take your meaning, you're saying that "understanding" is a high-resolution concept which I am trying to define with low-resolution analogies. But I contend that "understanding" is a low-resolution concept which we use colloquially to understand high-resolution human behaviour.

200

qoop @qoop.ca · 28/09/2026

humans and LLMs are obviously very different, but if we jump to drawing a bunch of distinctions from first principles, we risk marching off in the wrong direction and getting lost, because we understand neither human brains nor LLMs from first principles.

qoop @qoop.ca · 28/09/2026

I feel like we've gotten in the bad habit of telling ourselves just-so stories about the differences between humans and LLMs. Like, what does "definitionally regurgitative" even mean? Is it possible that humans are also regurgitative? Is there an experiment we can do to find out?

qoop @qoop.ca · 28/09/2026

Typically, they will fail to notice the substitution and come up with a list of reasons why they prefer B to A. humans *don't* reason deterministically.

qoop @qoop.ca · 28/09/2026

you already mentioned propaganda—as another example, there's a study where you show a subject two photos (A & B) and ask them to pick which person they find more attractive. The subject picks A. Later, you show them the photos again & ask them to explain why they picked B.

qoop @qoop.ca · 28/09/2026

oo this is a really risky example, because humans do this all the time. There's a great body of psychology literature suggesting we don't really understand the reasons why we do the things we do & we have to work backwards to explain our own actions.

200

qoop @qoop.ca · 28/09/2026

(not an actual quantum model of consciousness though. oh god. do not get me started about roger penrose)

100

qoop @qoop.ca · 28/09/2026

but once we start trying to distinguish ourselves from LLMs, I think we need a more granular quantum model of consciousness to draw precise distinctions, and I think "understanding" is a low-resolution Bohr-model term that blurs these lines instead of making them clearer.

200

qoop @qoop.ca · 28/09/2026

we don't need to depend on "true understanding" to distinguish us from LLMs & I think we'll have a harder time doing so if we do. Extending your excellent atomic model analogy, the grade-school Bohr model of consciousness works fine when humans are the only thing with any cognitive capacity in town,

100

qoop @qoop.ca · 28/09/2026

Even if LLMs can "understand," they don't experience anything like human qualia. They don't really have selves, per se, much less feelings, though they often perform characters that act as if they have feelings.

100

qoop @qoop.ca · 28/09/2026

Hallucinations don't work well as a defence when bots can notice vulnerabilities more reliably & with fewer mistakes than we can. a machine that "might as well" understand things poses problems if we think of understanding as an integral part of consciousness, but it isn't.

200

qoop @qoop.ca · 28/09/2026

Chatbots now understand code well enough to find subtle security vulnerabilities which humans have overlooked. Whether this is "true" understanding is almost beside the point: whatever it is, it is beginning to outperform humans at certain cognitive tasks that depend on understanding.

qoop @qoop.ca · 28/09/2026

at this point chatbots have become quite good at coding. Good enough to notice things you overlooked, or figure out an especially confusing expressions that would normally take time to pick apart. This is why there's been such a cybersecurity hubbub lately:

qoop @qoop.ca · 28/09/2026

i agree anthropomorphization is a problem, but i think what we're running into is the issue of cognitive severability. As we build more sophisticated machines, we get machines that can do some of the cognitive things humans do, the idea that cognition is all-or-nothing becomes a liability.

200

qoop @qoop.ca · 28/09/2026

well, i know that i know nothing about heart surgery. I guess this gives me "true understanding" of heart surgery, lol. Great news for me, since surgeons make so much money.

000

qoop @qoop.ca · 28/09/2026

well, enlighten me then, o skilled one. Define for me the nature of understanding. I've given you my best shot at it.

200

qoop @qoop.ca · 28/09/2026

"understanding" describes something humans do, but you're not going to find a thing called "understanding" which exists anywhere inside the human brain. Equally, you won't find understanding inside a computer, but that doesn't mean it can't be an accurate description of a thing a computer does

100

qoop @qoop.ca · 28/09/2026

that, or they're ideas. You're not going to find "real understanding" inside a computer any more than you're going to find "the real number 2" inside a calculator—or indeed anywhere. Ideas describe the world, but they are not themselves objects in the world

200

qoop @qoop.ca · 28/09/2026

So does it not make sense to take this stance relative to LLMs? Or is intention a Real Thing and not simply ascribed?

qoop @qoop.ca · 28/09/2026

Sometimes the intentional stance fails us, as when a chess computer glitches (or when a person glitches: is there an intentional explanation for forgetting, or for seizures?) but the intentional stance ("rational actors with beliefs and desires") is often more useful than not.

qoop @qoop.ca · 28/09/2026

For instance, i may say a chess-playing computer "wants" to checkmate me; I'm not talking about qualia or consciousness, but rather that it is more useful to understand this computer as a competent player trying to beat me at chess than it is to understand it as a complicated box of wires.

qoop @qoop.ca · 28/09/2026

daniel dennett has this idea, the "intentional stance" (en.wikipedia.org/wiki/Intenti...), which says that beliefs and desires and such are things that we ascribe to objects in order to understand them better.

en.wikipedia.org

Intentional stance - Wikipedia

qoop @qoop.ca · 28/09/2026

can't a tool intend to do what its user also intends for it to do? isn't that the point of a smart tool? that's certainly true when humans employ one another as tools

qoop @qoop.ca · 28/09/2026

haha thanks

qoop @qoop.ca · 28/09/2026

a lot of people are kind of gesturing at that, though what does it actually mean to say that a machine "can never have intent or understanding"? that's a very strong philosophical stance being taken totally for granted

100

qoop @qoop.ca · 28/09/2026

yeah because runaway proliferation already happened. Pakistan and India are both nuclear powers and they're constantly at each other's throats are you really trying to argue that nuclear bombs aren't a danger anymore

000

qoop @qoop.ca · 28/09/2026

the idea that a mere piece of metal could be conscious is hard to wrap your head around, but no harder than the idea that a mere piece of meat could be conscious. So I'm wary of the presumption that humans obviously must have qualia and machines obviously cannot.

qoop @qoop.ca · 28/09/2026

can you expand on the qualia thing? i don't see how we can draw a line from "humans have qualia" to "humans have an innate capacity to Do Deeds while computers can only sort of aim at and stochastically approximate a deed." There's no special "intent organelle" that neurons have but microchips lack.

qoop @qoop.ca · 28/09/2026

it's the john wick problem. With an entire stratum of society dedicated exclusively to assassins and assassin-related service providers, they have to be losing money. Rather than using assassins to protect the drug trade, they're being forced to sell drugs to subsidize their assassination racket.

qoop @qoop.ca · 27/09/2026

other similar examples from the past in a way that's fundamentally different from how a human conceptualizes the task… well, it did the task, didn't it? are we discussing *what* they're capable of, or *how* they're capable of it? cuz they're capable of a lot these days, regardless of how they do it.

000