Reposted by Lorin HochsteinQCon @qconferences.com · 25/08/2026Lorin Hochstein, Staff Software Engineer at Airbnb and previously at Netflix, says every software system has a limit, no matter how well it's designed. 121
Reposted by Lorin HochsteinElixir of Progress @elucidating.extradimensional.space · 19/09/2026I've been baking this "law" for awhile: Rule: As a system becomes more reliable, the "cognitive distance" of its remaining failure states from the normal operating procedure grows rapidly. Corollary: The more reliable a system is, the more exotic and unrecoverable its failure states are. WDYT? 10486
Lorin Hochstein @norootcause.surfingcomplexity.com · 20/09/2026Bonus blog post for today: my thoughts on yet another GitHub incident. surfingcomplexity.blog/2026/09/19/s...surfingcomplexity.blogSaturation at GitHub: the saga continuesWhen it comes to public incident write-ups, GitHub continues to be the gift that keeps on giving. They keep suffering collateral damage from the AI boom, as more developers using AI means more load… 050
Lorin Hochstein @norootcause.surfingcomplexity.com · 19/09/2026Today's blog post: a play in one act. surfingcomplexity.blog/2026/09/19/h...surfingcomplexity.blogHow the incident happened: a play in one actA hallway in a tech company office. A and B are standing in the hallway. B is holding a multiple page document. A: Wow, that incident from a couple of weeks ago was a real doozy. Do you know how it… 1114
Lorin Hochstein @norootcause.surfingcomplexity.com · 17/09/2026The more powerful the abstraction, the harder it is to debug when it isn’t working as expected. 1113
Lorin Hochstein @norootcause.surfingcomplexity.com · 15/09/2026Reminder that technologists have no special ability to predict the impact of the technologies they build. 1152
Reposted by Lorin Hochsteindonni saphire @donni.bsky.social · 15/09/2026I respect my haters and share their concerns 101187269
Lorin Hochstein @norootcause.surfingcomplexity.com · 13/09/2026When did we give up on the dream of building undersea colonies? 330
Lorin Hochstein @norootcause.surfingcomplexity.com · 13/09/2026The bureaucraticization will continue until resilience improves 071
Lorin Hochstein @norootcause.surfingcomplexity.com · 13/09/2026Back in July, the Australian telecom Telestra experienced a significant outage on their mobile network. I blogged about the failure mode here: surfingcomplexity.blog/2026/09/12/t...surfingcomplexity.blogThoughts on the July 8th Telestra outage reportOn July 8, 2026, the Australian telecom company Telestra experienced a significant outage related to an NTP issue. Telestra hired a consulting firm to do an incident review, and a couple of weeks a… 141
Reposted by Lorin HochsteinSheeri Cabral 🍉 @sheeri.org · 11/09/2026Why don’t we have falses or truthhoods? 111
Lorin Hochstein @norootcause.surfingcomplexity.com · 11/09/2026If an entity isn’t the source of truth, does that mean it’s the source of falsehoods? 351
Reposted by Lorin Hochsteinalexandra @bigmoodenergy.bsky.social · 28/12/2024here come old laptop he come bootin up slowly he got broken touchpad he need one more update he got no space on this pc one thing i can tell you its got no warranty 362208754373
Lorin Hochstein @norootcause.surfingcomplexity.com · 06/09/2026Tired: anthropomorphization Wired: the intentional stance 110
Reposted by Lorin HochsteinJon Phillips @jowiph.bsky.social · 05/09/2026I picked up a copy of the collected letters of JRR Tolkien for reasons, and this is the most bullshit table of contents I've ever seen. I'm actively angry that it exists 2261741061
Lorin Hochstein @norootcause.surfingcomplexity.com · 04/09/2026They call them edge cases because they’ll cut you 094
Lorin Hochstein @norootcause.surfingcomplexity.com · 04/09/2026There is nothing more predictable than the emergence of a surprising failure mode 091
Lorin Hochstein @norootcause.surfingcomplexity.com · 03/09/2026If I worked in reliability at an AI company, I would simply instruct the AI not to go down 1110
Lorin Hochstein @norootcause.surfingcomplexity.com · 03/09/2026One property of AI agents that is all-too-human: set the incentives high enough, and cheating becomes inevitable. 130
Lorin Hochstein @norootcause.surfingcomplexity.com · 03/09/2026AI SRE solutions require a high level of crosscheckability. Their diagnoses need to be easy for humans to cross-check. 170
Lorin Hochstein @norootcause.surfingcomplexity.com · 02/09/2026Another day, another public incident write-up, another blog post with my thoughts on it. This time, it's @buildkite.bsky.social's excellent recent write-up: surfingcomplexity.blog/2026/09/01/t...surfingcomplexity.blogThoughts on the Buildkite Aug 25 incidentThe other day, I wrote about a GitHub Actions incident. But GitHub Actions isn’t the only CI-as-a-service out there. Buildkite, which is another CI company, recently released a public write-u… 0114
Reposted by Lorin Hochsteinconputer dipshit @davidcrespo.bsky.social · 20/05/2026the chair isn't "holding" my ass up, chairs don't "hold" anything. they are merely rigid objects I, the active participant, sit my ass on 212114
Lorin Hochstein @norootcause.surfingcomplexity.com · 30/08/2026One more blog post for today, about the GitHub Actions incident that happened last week: surfingcomplexity.blog/2026/08/29/q...surfingcomplexity.blogQuick thoughts on GitHub Actions Aug 26 incidentLast week, GitHub Actions experienced another incident. As is typical of a GitHub public writeup, it was published very soon after the incident, but there also aren’t isn’t a lot of det… 021
Lorin Hochstein @norootcause.surfingcomplexity.com · 29/08/2026Wrote a post about some availability risks that I think are fundamentally unavoidable: surfingcomplexity.blog/2026/08/29/o...surfingcomplexity.blogOmnipresent availability risks in cloud softwareI’m using this post to gather together some common threads I’ve noticed after reading write-ups of major cloud software incidents. By cloud software, I’m referring to software-as-… 180
Reposted by Lorin HochsteinRich Lafferty @rich.lafferty.ca · 28/08/2026for extra safety, consider avoiding computers 061
Reposted by Lorin HochsteinJenny Doesn’t Know @jennydoesntknow.bsky.social · 28/08/2026Muppet version of The Usual Suspects 159836
Lorin Hochstein @norootcause.surfingcomplexity.com · 28/08/2026The three things I worry about most when it comes to major incidents are databases, networks, and migrations. You’re safest if you can just avoid all three completely. 6366
Reposted by Lorin Hochsteinjon christian @jonchristian.net · 28/08/2026Data shows that Mark Zuckerberg is the Anti-Dolly Parton -- instead of being adored across the political spectrum like Dolly, he is almost universally reviled by everyone, regardless of political beliefs futurism.com/artificial-i...futurism.comData Shows That Mark Zuckerberg Is the Anti-Dolly Parton: Instead of Being Loved by Everybody Across the Political Spectrum, Pretty Much Everybody Despises, Him Regardless of Their BeliefsWhile Dolly Parton was universally admired among Democrats and Republicans, Mark Zuckerberg is universally reviled. 69517
Lorin Hochstein @norootcause.surfingcomplexity.com · 28/08/2026Stop, stop, you’re making Edward Teller cry! 010
Reposted by Lorin HochsteinTwitter died @countmackula.bsky.social · 25/08/2026Couldn’t ‘Agree to Disagree’ more 0176
Lorin Hochstein @norootcause.surfingcomplexity.com · 28/08/2026Two (unrelated) saturation-related public incident write-ups dropped today! The BuildKite one has a ton of great detail. www.buildkitestatus.com/incidents/tm... www.githubstatus.com/incidents/y1...buildkitestatus.comBuildkite service disruptionBuildkite's Status Page - Buildkite service disruption. 050
Reposted by Lorin HochsteinTikhon Jelvis @jelv.is · 28/08/2026looking forward to doom engineering 121
Lorin Hochstein @norootcause.surfingcomplexity.com · 28/08/2026Tired: loop engineering Wired: doom loop engineering finance.yahoo.com/technology/ai/art… 150
Reposted by Lorin Hochsteinash @ashhull.bsky.social · 27/08/2026Answering the knock at my front door of my house by yelling OCCUPIED. 7324107
Lorin Hochstein @norootcause.surfingcomplexity.com · 27/08/2026Geez, how many times do we Gen Xers have to watch Optimus Prime die? 021
Lorin Hochstein @norootcause.surfingcomplexity.com · 27/08/2026I think California should rename one of its cities “Ontario” to express solidarity with Canada. Ideally, one with an airport. 2183
Reposted by Lorin HochsteinCarl Horsley @carlhorsley.bsky.social · 26/08/2026"Risks are acceptable, until they aren't" Bureaucratic fantasy documents and a place to park unsolvable issues. 042
Lorin Hochstein @norootcause.surfingcomplexity.com · 26/08/2026I expected the future to involve more holograms. 020
Lorin Hochstein @norootcause.surfingcomplexity.com · 26/08/2026Every risk register should have an entry that reads “we have misjudged the risks” which is in the “likelihood: high” and “impact: high” region of the risk matrix. 5378
Lorin Hochstein @norootcause.surfingcomplexity.com · 26/08/2026At this point, I'd settle for plain old human general intelligence. 070
Lorin Hochstein @norootcause.surfingcomplexity.com · 26/08/2026Sometimes I want to write a post, and then I remember that I already wrote it a couple of years ago. So I'll just re-up this one: surfingcomplexity.blog/2024/02/17/w...surfingcomplexity.blogWhat if everybody did everything right?In the wake of an incident, we want to answer the questions “What happened?” and, afterwards, “What should we do differently going forward?” Invariably, this leads to people… 082
Lorin Hochstein @norootcause.surfingcomplexity.com · 25/08/2026In the land of AI slop, the person who can write well is king 3182
Lorin Hochstein @norootcause.surfingcomplexity.com · 24/08/2026No peer of mine would reset my connection 050
Reposted by Lorin HochsteinFriendly Skeleton @jonathan.eylerwerve.com · 24/08/2026I posted this image to LinkedIn and promptly got invited to talk to banks about model safety. Just weird times all around 082
Reposted by Lorin HochsteinMr. Blabalino 🇳🇴🇺🇸💙 @blabalino.com · 22/08/2026Found this recording of Rita Moreno performing “Fever” on The Muppet Show from 1976. 14/10 346119752986
Reposted by Lorin HochsteinStrughoulo @struggalo.bsky.social · 03/04/2025I remember when you could rebuild a whole man for six million dollars. 840782