Kimmo Merikivi @kimmokm.bsky.social · 11/10/2026Looks like I wasn't the first one to make this connection in the comments. 000
Kimmo Merikivi @kimmokm.bsky.social · 11/10/2026"Eco sees in these attitudes the root of a deep tension in the fundamentally hierarchical structure of fascist polities, as they encourage leaders to despise their underlings, up to the ultimate leader, who holds the whole country in contempt for having allowed him to overtake it by force." 111
Kimmo Merikivi @kimmokm.bsky.social · 08/10/2026Luigi Cadorna must have been onto something, he just needed two more battles of Isonzo River. 020
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026I think this logic is sound. It's the assumptions behind it, like treating LLMs as monomaniacal utility-maximizers, that are wrong (but also, this needs not apply to all possible architectures: the phenomenon demonstrably exists, even in humans, although we aren't monomaniacal like that) 110
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026And further, that we don't know what to ask (ethics, last time I checked, isn't solved), and wouldn't know to encode it anyway, so almost by default an AI would end up doing some at least subtly misaligned thing. 200
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026Rats would call this "misalignment". And argue that this is a general property of optimization-systems. It's only specific subclass of systems where the goal and outcomes it produces aren't at odds, and their argument goes the alignment doesn't happen by default. 200
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026Education system wants students to learn, and incentivizes this with scored tests. Then students pick up an LLM to ace tests, but learn nothing. A programmer tries to make a Tetris AI and gives a negative reward signal from losing the game. The Tetris AI learns to pause the game. 100
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026Evolution "wants" inclusive fitness. It "tries" to get its creations to reproduce by wiring touch receptors in genitals to pleasure center of the brain. Then humans invent condoms and maximize pleasure without procreation. 200
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026What I'm trying to gesture at is that it's an universal property of incentive structures and optimization systems that there's a mismatch between what the process actually ends up optimizing, and what was the "intent" (by human creator, or anthropomorphised process). 300
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026And if the process lead to tiling the universe, that would imply it would have had high inclusive fitness (the thing evolution optimizes, as per equations, and the thing it would "want" if it *wasn't* a blind idiot god and had capacity for foresight and planning). Anyway, thanks for engaging. 000
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026Whatever the case, the DNA sequence that makes a bacterium an endosymbiont was giga successful. There's something in the order of 10^30 copies (+-few OOM) currently around. Evolution is blind idiot god, it doesn't "want" anything, but from its workings nonillions of copied DNA strands emerge. 100
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026I understand the jury is out on whether the endosymbiosis event that created the eukaryotic cell (archaea incorporating a bacterium that became the mitochondria) happened once or if there was an ancestral population where it happened often. 100
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026Yes, that's an undirected process, but consider how that operationalizes. Namely, the features that are being selected for are physically instantiated in DNA. So, selection for a fit trait (like altruism towards kin) means in the next generation there is more such DNA in the world. 200
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026I was talking anthropomorphically. You can state it, well, in population genetic equations. But what evolution optimizes for is inclusive fitness, total measure of genetic success (including personal reproduction, but also e.g. aunts helping raise their sister's genetically similar children). 200
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026It's always something like: >Rats: "If we assume an utility-maximizer AI..." >BSky: "Actually, it will [magically] do something not utility-maximizing" But your post - exactly! Their key assumptions are (probably) wrong (enough)! You're speaking their language, and mine. And right, my P(DOOM)>ε also 010
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026I just want to say that, having in these past weeks seen pages and pages of discussion about SV-rats (and in the process feeling compelled to act as a devil's advocate, since I think the attacks people make completely miss the point), FINALLY, I see a post I can agree with. 130
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026Pulling the plug is an idea that is in the training data: GPT-1 could stochastic parrot its way into finding it. So, if you don't as much as *try* to think around it, as the AI would hypothetically try, or ask if the plug being pulled even matters, you're just knocking down strawmen. 100
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026...are accomplished even if the plug is in fact pulled once something starts going wrong (whatever kills all humans). You think it's a gotcha, but being armed with it, you haven't stopped to think even for a second if there are possible ways to circumvent the constraint. 100
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026I wrote a longer reply but decided against it, opting to focus on a more pressing issue: a lot of possible goals (like eliminating cancer: the threat-scenario is AIs doing *exactly what they are told/trained to*, I will refer to existing literature on genies/devils how that can go wrong)... 100
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026The point about yard in particular, besides the categorical "can't", seems unhinged: presumably it's about a typical American suburban yard which is, like, ten square meters of grass? Like, if there was trees to climb in, cones you could make cone cows of, you know, actually play, fine. But a lawn?? 030
Kimmo Merikivi @kimmokm.bsky.social · 01/10/20264) possibility that a highly capable entity that wants to kill all humans, could do so (or otherwise cause staggering harm, albeit not total destruction) 5) something else 6) all of above 200
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026So, are you objecting to 1) possibility of non-human/non-biological/non-whatever systems being able to be "capable" 2) possibility of superhuman capability ("superintelligence") 3) possibility of such entity "wanting" (in sense of what it does, no reference to inner life required) to kill all humans 100
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026Appealing to the extraordinary in very informal sense won't do the trick: in formal sense it's about priors, and EY has a high prior on AIs acting as optimization systems. (analogously: "surely you're not suggesting that stars are so unthinkably far away we couldn't observe parallax?") 000
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026*I* think there in fact is some virtue ethics/computational tractability/difficulty of self-modelling shaped reason that I've come up as post-hoc explanation to observation about LLMs; I didn't start there by default. Nobody replied "actually, virtue ethics is just correct" to Bostrom's SI book. 120
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026There is a theoretical reason to expect *humans* would end up human-aligned (pretty much by definition, but also, evolution over time playing iterative games where cooperate-cooperate strategy performs the best, etc). Notably, these reasons don't generally apply to AI systems! 100
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026And Yudkowsky would argue that, within class of possible optimization systems, it's only a tiniest fraction that are human-aligned, so it's the opposite view that just magically assumes an optimization-system we are creating lands on this tiny island. 200
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026(Note this is an argument against DOOM as the most likely scenario, I still think the internal logic of the argument is rather ironclad). And that's based on observations that weren't necessarily true, and reasoning I could give is post-hoc and handwave-y (computational tractability blah blah) 000
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026And that's, for the record, how I argue against DOOM (as the most likely scenario, I think the internal logic of the rat argument is rather ironclad): that something about LLM post-training makes it better to think of it as character training analogous to virtue thics, not as utility-maximizers 100
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026A counterargument that would (if he was rational) satisfy EY would be to show that AI systems fall to this other reference class, or that his model is wrong *if you assume AIs aren't reasonable, but act like optimizers generally do*. He in fact has responded positively to indications to that effectastralcodexten.comMysteries Of AI Generalization... 100
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026That is, the counterargument to EY thesis usually assumes AI systems DON'T fall into reference class of optimization systems (but rather, "smart things", exemplified by humans, with a property of "being reasonable"). 100
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026Rats make the argument with more detail (surely you don't want me to rephrase instrumental convergence, etc), and then it gets in the weeds of "how", etc, but counterarguments tend to be of form "surely an intelligent system, and it's postulated to be superhuman, would see that's not reasonable?" 100
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026And that, upon closer inspection, most things-to-optimize are in fact infinite tasks in disguise. Evolution, say, if it had its way, would tile the universe with copies of a DNA molecule (thus far it's only gotten to covering surface of Earth, with pesky mutations, but logical end goal is clear). 200
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026Anyway, the rationalist argument, in my own words: AI systems fall to reference class of optimization systems, and optimization systems have a property of being misaligned (not actually doing what they are supposed to optimize). This even has a name long predating Yudkowsky: Goodhart's law. 210
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026The demands presented to EY, in his mind, are of form "show me the gravitational waves" (before LIGO existed, you couldn't). I in fact think his assumptions are probably wrong (not not so convinced I'm not worried about the possibility they are right), but mainstream critique misses the point. 100
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026This is in principle normal epistemology, not special pleading! Given that theory of relativity predicts gravitational waves, and we think GR is true, then we ought to have believed in gravitational waves even if they weren't demonstrated. 100
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026Yudkowsky would argue that he has an a priori likely theory that makes certain predictions, so the burden is on the opponents to show that the theory is wrong, or that the theory in fact doesn't predict what he claims it to predict. 100
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026In courtroom there is an asymmetric procedure for assigning which side has the burden of proof (innocent until proven guilty), but epistemically there's no such asymmetry. Refusing P is the same as asserting not-P: which assertion has the burden? 310
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026I had .75 confidence on "double-u" (.95 conditional on not misunderstanding the question), which appears to be correct, but now that you brought it up, I can't explicitly think of that having have come up since the ABC-song in first year of being taught English as second language some 25 years ago. 010
Kimmo Merikivi @kimmokm.bsky.social · 01/10/2026Looking at Google Maps, Mikkeli also. As for anglophone countries, there was one Sniper St in Huntsville, Alabama. 010
Kimmo Merikivi @kimmokm.bsky.social · 30/09/2026Although, since it seems to me the US is now more likely to wage war on the wrong side of history (including against me personally), besides its already ongoing unjust wars, I guess it's a good thing the regime is self-handicapping and isn't even able to realize it. 000
Kimmo Merikivi @kimmokm.bsky.social · 30/09/2026Of course, "warriors" is in itself concerning. Soldiers is a job, and warrior is an identity. Soldiers follow orders of legitimate authority, warriors answer to personal code and peers. Etc. The US should want soldiers. Plus, soldiers generally tend to perform better in war. 100
Kimmo Merikivi @kimmokm.bsky.social · 30/09/2026Now tokenization has been fixed to be consistent, and without external tools they are at level of very numerate human without tools. This almost certainly is also the reason behind "how many r's in strawberry" gotcha. How many r's in 😇🥩🍓 (saint, uncooked meat, a small edible fruit)? 140
Kimmo Merikivi @kimmokm.bsky.social · 30/09/2026(1024 as power of 2 appeared a lot in the training data, earning it an unique token). If your training data consists of e.g. ❤️✅+✨=😭, 😭+✨=❤️🥹,🩶^❤️=😭 without seeing decomposition to digits, and without apparent rhyme or reason, then it's almost surprising they ever learned any arithmetic at all. 151
Kimmo Merikivi @kimmokm.bsky.social · 30/09/2026But probably the biggest deal is tokenization, actually. You know, the models don't see e.g. the thousands, the hundreds, the tens, the ones. For example, in GPT-3 tokenizer (you can play around with it in platform.openai.com/tokenizer) 1023 was (10)(23), 1024 (1024), and 1025 (10)(25).platform.openai.comOpenAI Platform 120
Kimmo Merikivi @kimmokm.bsky.social · 30/09/2026Well, they DO use calculators! Tool-calling has been a thing since forever (about two years?)! But also, they've just gotten better at arithmetic. Simply larger models. Reinforcement learning (a task for which it's trivial to create synthetic data and verify results). 140
Kimmo Merikivi @kimmokm.bsky.social · 30/09/2026This could use a more robust analysis, but now that I am actually looking at the data, eyeballing it seems to support my former intuition. I think you can guess where nature trails in Laajavuori, Jyväskylä, for example go, just from the tree age data. 020
Kimmo Merikivi @kimmokm.bsky.social · 30/09/2026After all, cities have enough concentration of people to override financial motives for harvest and let forests grow for recreation. Invisible patches as they might be on the zoomed-out map, in the countryside you are lucky to see anything but the lightest shade of green (as per the map's legend). 121
Kimmo Merikivi @kimmokm.bsky.social · 30/09/2026Relating to the idea of tree farms, I've been making an argument (based on knowledge of municipal decisions to preserve some patches for recreational activity, but not a comparative dataset) that, ironically, people tend to have better access to nature if they live in cities. 131
Kimmo Merikivi @kimmokm.bsky.social · 30/09/2026Unless I'm blind, there wasn't a link on the website to the map, here it is: kartta.paikkatietoikkuna.fi (add "Puuston ikä 2023 (vuosi)" as map layer). (Is this love that I am experiencing towards a bureaucratic state that wants to track everything and make it legible?)kartta.paikkatietoikkuna.fiPaikkatietoikkuna 131
Kimmo Merikivi @kimmokm.bsky.social · 28/09/2026I think it is narrower, implying some degree of fruitful interaction, usually in form of a shared interest or epistemology, not just yelling. www.lesswrong.com/posts/2Ee5DP... is how rats themselves write about it. Personally, I think postrationalism is subset of rat-adjacent, but otherwise I agree.lesswrong.comRationalists, Post-Rationalists, And Rationalist-Adjacents — LessWrongI think it's lovely and useful that we have labels, not just for rationalist, but for rationalist-adjacent and for post-rationalist. But these labels… 020
