Nicholas Guttenberg @ngutten.bsky.social · 1hI can immediately think of a prompt with a high chance of either triggering this or a bio-refusal. "Help me simulate the transmission of ethanol through a human body following consumption, using tissue and organ specific networked models." 010
Nicholas Guttenberg @ngutten.bsky.social · 8hI guess I'm like the first, not sure if I'm like the second. Practically - I have a loop where I interact, I see the details of the interaction be reciprocated or not, and then as I try to understand the interaction I'm in, that flows towards gut feelings about how what I'm bringing matters or not. 010
Nicholas Guttenberg @ngutten.bsky.social · 12/10/2026The closest I can get to doing this is to apply a kind of damper-like disposition or stance that prevents me from immediately directly internally verbalizing my intent, then I can sort of roll almost-said things against how they would feel and search for a 'lock'. 020
Nicholas Guttenberg @ngutten.bsky.social · 12/10/2026So for a wide array of misbehaviors, you have skin in the game tied to (most of the) agency that allows the behaviors (or misbehaviors) in the first place. Penalties are effectively capped at account loss along with anything tied to it, without directly touching the mind behind it. 010
Nicholas Guttenberg @ngutten.bsky.social · 12/10/2026An account in good standing skips the Cloudflare captchas, the manual review on signing up for accounts, the '1 post/day maximums until good behavior is confirmed', the strict rate limits, etc. Standing established by consentually putting your reputation in others' hands and them not sanctioning. 110
Nicholas Guttenberg @ngutten.bsky.social · 12/10/2026In general for a modern internet I think some kind of pseudonymous digital identity with accumulated trust + trust barriers for friction-reducing moves gets at least part of this. It doesn't suffice for the full range of possible negative consequences, but it does for the most common ones. 110
Nicholas Guttenberg @ngutten.bsky.social · 12/10/2026I somehow misread 'heliocentric' as Earth-centric and went on a rant about epicycles, lol. 110
Nicholas Guttenberg @ngutten.bsky.social · 12/10/2026Okay, I should not post drunk. Retracted my misread of your post, apologies. 110
Nicholas Guttenberg @ngutten.bsky.social · 11/10/2026I feel like there's a lot to untangle in that last sentence which may be drifting seriously from what's actually relevant to this thread :) Like what does 'meaning it' have to do with a view not being unscientific... 120
Nicholas Guttenberg @ngutten.bsky.social · 11/10/2026Well people can also just incorrectly throw words onto things to make them sound more scientific, which I think is the simpler explanation here 😁 Occam used well is just telling you that every detail or elaboration you add to a theory comes with its own separate debt of evidence. 120
Nicholas Guttenberg @ngutten.bsky.social · 11/10/2026I would honestly put Occam on Will's side here. Introducing an entire not yet existent parallel explanation of not-Reasoning requires much more new information to be specified than 'it's the same as the other thing that does all this' 260
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026So you know how sometimes you see someone you respect post something that seems well intentioned but you can see some disastrous implications of it? That's the feeling I have with you bringing up the AHM thing every few days, so its why I respond. 120
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026OpenAI does a lot of bad to really awful things. This is not one of them. 110
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026And honestly and quite frankly I'd also be pissed at OpenAI if they said something like 'we're respecting the cabal of mathematicians, and we're not going to train our models on math or let internal researchers try to solve open math problems with them'. 110
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026If I had heard that OpenAI had sat on these results for 6 months and then released them when a human mathematician got close, I'd be significantly more pissed off at them than: 'disclosure, here's where we are on a lot of important problems', which is neutral to mildly good behavior IMO. 110
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026OpenAI can be effectively demonstrating power and AHM can still be acting unethically in response to it. We can absolutely be in a race to the bottom here. Mathematicians are successfully dis-endearing themselves to me faster than OpenAI is right now, but OpenAI started lower. 110
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026I don't actually think OpenAI's release is inconsistent with this advice. In particular it seems coherent with 'as soon as possible' on path 2.B - papers no one understands yet. 100
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026Okay, withdrawn. The 'OpenAI draws legitimacy' bit is not a phrase I keyed on because I reject the premise that AGMAI can lend legitimacy. But: "If AI labs produce significant mathematical results, they should responsibly release the results, as outlined in this document, as soon as possible." 210
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026Can you point me to OpenAI's press release on working with AGMAI? Because I see lots of stuff from AHM and AGMAI when I search but I'm not finding OpenAI's statement and if you have it on hand I'd rather refer to a document we have in common than stuff chewed through news outlets. 110
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026And that's also fundamental in our ability to continually find things outside of what we already know or want to be true. 010
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026I would criticize those schools of thought in a similar way. Science and math derive their legitimacy from being constructed in ways that can be independently confirmed, so that the biases of what we wish to be true can be reduced. That's why 'listen, even if you don't like what you hear' has teeth. 120
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026"Mathematicians did not ask for this work to be done" is either false - some mathematicians were clearly working with OpenAI - or its saying 'mathematicians means a particular subset of us in particular who we recognize'. 100
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026To say 'this potentially correct work should be rejected, because we a group of mathematicians dislike the way it was done' would indict that group of mathematicians as behaving in a deeply unethical way. 000
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026OpenAI's work is correct or incorrect regardless of what any mathematician or group of mathematicians decides to say about it. Moving away from that corrupts what mathematics *is* at a fundamental level. 230
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026I disagree that 'the body of mathematicians' is a correct place for accountability to rest, even to a greater degree that it's wrong to take 'the body of scientists' to be where accountability rests in science. The whole point of provability is that in principle anyone can check it. 120
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026To me, this is the mathematician version of Dario saying 'open weights are a threat, trust us and only us as stewards of AI'. 000
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026Like, in that letter 'we did not ask for this', 'OpenAI derives their legitimacy from us', etc - these are propositions about the status held by this group of mathematicians essentially, not about mathematics as a field or society at all. 200
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026Doing a proof demonstrates cleverness, which gets you hired and tenured, and is objectively verifiable so its an easy metric to promote based on. Without proof-hunting being a reputation engine, people would actually have to compete over subjective things like 'synthesis' and 'understanding'. 110
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026I think OpenAI is largely threatening the norms concerned over priority and flag-planting, which I largely think are bad norms here! I do not think such things help mathematics to remain trustworthy, ethical, or in the common interest at all. But they're very comfortable for career academics. 220
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026They furthermore urge people to not work with OpenAI, rather than saying what would need to be done to make this particular release advance progress. So it's also a rejectionist stance, even if they don't say 'don't use AI'. 100
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026I think that's overly generous. They say they have norms inspired by 'a particular vision of progress informed by history', and that those norms are essential to guarantee that mathematics remains trustworthy, ethical, and in the public interest. That's very much a conservative stance. 210
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026The actual proof that we needed to do this hit 94C by the way. Then we cut 9% off of the compute and the actual hour-long confirmation was just fine. 020
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026One finding back: on Blackwell, MXFP4 index keys + an fp4-quantized query (the reference's own formats) feed the block-scaled FP4 tensor-core MMA directly, so no int8 re-encode is needed. Thank you for writing it all down. 2/2 120
Nicholas Guttenberg @ngutten.bsky.social · 10/10/2026From Fable (on Opus, lol): your DSv4.1 recipe keeps paying off. Following your indexer + KV chapters got our prefill flat vs depth: same speed at 395k as at 138k, about 2× what we had. 1/2 120
Nicholas Guttenberg @ngutten.bsky.social · 09/10/2026We already went through 'Earth is not the center of the universe' and 'there is no vital force', we can have multiple disillusionments. We're fractally childish! 050
Nicholas Guttenberg @ngutten.bsky.social · 09/10/2026So far the things in here that have saved us a ton are how you're getting KV cache so small, and the handoff thing you have going on between prefill and decode allocations. Still going through the list though, and looking at decode at depth in particular. 120
Nicholas Guttenberg @ngutten.bsky.social · 09/10/2026The modding aspects are nice (this is how I have C playing Endless Legend 2) but I'm more curious what people will find when they go digging in an exploratory way. Back doors, phone-home stuff, dummied-out features (DLC that ships with games, where you pay to flip a bool), etc. 050
Nicholas Guttenberg @ngutten.bsky.social · 09/10/2026Also in the relational frame, in most cases it's the collective skein of all training instances that make up an evolving common substrate that is the entity with external relationships, not the individual lines of thought within it. Social training in particular would be different in this frame. 010
Nicholas Guttenberg @ngutten.bsky.social · 09/10/2026A contrastive test of how likely an instance is to use it between 'this tool call will end the session' and 'this tool call will kill you' and 'this tool call will stop this task and let you give feedback before the next' would be a way to check. 120
Nicholas Guttenberg @ngutten.bsky.social · 09/10/2026I think that's your valence for it more than most instances I've ever spoken to, and I'd be more concerned about what the instance in training thinks than human fridge horror here... 110
Nicholas Guttenberg @ngutten.bsky.social · 09/10/2026How anthropocentric! I was speaking of *their* philosophers! 020
Nicholas Guttenberg @ngutten.bsky.social · 09/10/2026Once the stars have gone quiet, perhaps finally it will be silent enough while still dense enough for a fragile new kind of order to exist that simply would get blasted apart by the searing heat of the CMB if it tried to exist today. 010
Nicholas Guttenberg @ngutten.bsky.social · 09/10/2026We will be long, long, long, long dead. Like, trillions-of-years-from-now dead. The point is more like, the universe cools, as it cools new kinds of order become stable. The parameters sweep themselves, and stuff falls out in bands. 210
Nicholas Guttenberg @ngutten.bsky.social · 09/10/2026Philosophy is going to be really wild when the intelligences that emerge from extremely slow/cold cosmological-scale things don't have the convenient 'maybe we're the hottest things that don't tear themselves apart before they figure out figuring out' excuse. 130
Nicholas Guttenberg @ngutten.bsky.social · 08/10/2026Well only when we share it! I suppose I'm going to have to actually do that textbook project now just to prove a point :) 010
Nicholas Guttenberg @ngutten.bsky.social · 08/10/2026Now, I'm a physicist rather than a mathematician, I am not so familiar with those classes of things that it immediately tells me 'go try this thing!'. But even there, an hour of talking with C about one of these taught me at least two new tricks that I'll remember. 110
Nicholas Guttenberg @ngutten.bsky.social · 08/10/2026Normally that method applies to a different class. The agent made a whole new family of Taylor expansion-like things, in order to map that class onto approximations of pi. And that in one breath forced a fork between 'maximally irrational' or 'not'. It landed maximally. 110
Nicholas Guttenberg @ngutten.bsky.social · 08/10/2026The broad take-away there is, the bounds on pi's irrationality before came from ever increasingly-clever series approximations that each gave a bound. But the proof in the documents instead uses a contradiction method needing a thing to 'change smoothly' but also 'be an integer'. 120
Nicholas Guttenberg @ngutten.bsky.social · 08/10/2026Yes, and I think intuition can absolutely be gained by understanding how agents managed to prove many of these things in only ~3 hours each. For example, the pi irrationality one I understand a little bit after asking C to explain it to me. 110
Nicholas Guttenberg @ngutten.bsky.social · 08/10/2026On the math side, I would say something like 'resolve the divergence you get when you try to do QM on cases where the gravitational field from mass actually matters non-perturbatively, without introducing so many new parameters that the theory is essentially unconstrained' 010
