Sign in

Lorin Hochstein

@norootcause.surfingcomplexity.com
2.1K followers 1K following 1.6K posts

Student of complex systems failures, resilience engineering, cognitive systems engineering. Will talk your ear off about @resilienceinsoftware.org

PostsRepliesMedia
Lorin Hochstein @norootcause.surfingcomplexity.com · 18h
I wrote up a quick blog post about another GitHub incident: surfingcomplexity.blog/2026/10/03/q...
surfingcomplexity.blog
Quick thoughts on GitHub’s Sept 23 incident
On Mastodon, Ole Peder Brandtzæg pointed me to (yet) another GitHub incident that happened back on Sept 23. The write-up is very short, so I’ll just excerpt the whole thing here (emphasis min…
071
Lorin Hochstein @norootcause.surfingcomplexity.com · 03/10/2026
Tragically, I feel like we don’t use “lo and behold” as much as we used to.
050
Lorin Hochstein @norootcause.surfingcomplexity.com · 03/10/2026
New post on a recent GCP outage and the "human error" that "caused" it: surfingcomplexity.blog/2026/10/02/t...
surfingcomplexity.blog
There was a light that never went out
Lex Neva (of SRE Weekly fame) pointed me to this recent GCP incident report, which describes networking issues they experienced in their us-central1 region. This incident is yet another reminder of…
260
Reposted by Lorin Hochstein
QCon @qconferences.com · 25/08/2026
Lorin Hochstein, Staff Software Engineer at Airbnb and previously at Netflix, says every software system has a limit, no matter how well it's designed.
121
Reposted by Lorin Hochstein
Chris Oldwood @chrisoldwood.bsky.social · 25/09/2026
Abstract Data
An abstract picture of Commander Data.
0165
Reposted by Lorin Hochstein
Elixir of Progress @elucidating.extradimensional.space · 19/09/2026
I've been baking this "law" for awhile: Rule: As a system becomes more reliable, the "cognitive distance" of its remaining failure states from the normal operating procedure grows rapidly. Corollary: The more reliable a system is, the more exotic and unrecoverable its failure states are. WDYT?
10486
Lorin Hochstein @norootcause.surfingcomplexity.com · 20/09/2026
Bonus blog post for today: my thoughts on yet another GitHub incident. surfingcomplexity.blog/2026/09/19/s...
surfingcomplexity.blog
Saturation at GitHub: the saga continues
When it comes to public incident write-ups, GitHub continues to be the gift that keeps on giving. They keep suffering collateral damage from the AI boom, as more developers using AI means more load…
050
Lorin Hochstein @norootcause.surfingcomplexity.com · 19/09/2026
Today's blog post: a play in one act. surfingcomplexity.blog/2026/09/19/h...
surfingcomplexity.blog
How the incident happened: a play in one act
A hallway in a tech company office. A and B are standing in the hallway. B is holding a multiple page document. A: Wow, that incident from a couple of weeks ago was a real doozy. Do you know how it…
1114
Lorin Hochstein @norootcause.surfingcomplexity.com · 17/09/2026
The more powerful the abstraction, the harder it is to debug when it isn’t working as expected.
1113
Lorin Hochstein @norootcause.surfingcomplexity.com · 16/09/2026
Here’s a prediction I’m very confident in making: you will continue to experience surprising incidents.
010
Lorin Hochstein @norootcause.surfingcomplexity.com · 15/09/2026
Reminder that technologists have no special ability to predict the impact of the technologies they build.
1152
Lorin Hochstein @norootcause.surfingcomplexity.com · 15/09/2026
What was the idea???
100
Reposted by Lorin Hochstein
donni saphire @donni.bsky.social · 15/09/2026
I respect my haters and share their concerns
101187270
Lorin Hochstein @norootcause.surfingcomplexity.com · 14/09/2026
I was only on the tenure track for two years (it was at a non-Ivy), but I was certainly worried about not actually getting tenure when it was time for me to come up for it!
020
Lorin Hochstein @norootcause.surfingcomplexity.com · 14/09/2026
This is one of the costs of the tenure system, no? On the one hand, it protects tenured professors. On the other hand, it protects tenured professors.
110
Lorin Hochstein @norootcause.surfingcomplexity.com · 13/09/2026
When did we give up on the dream of building undersea colonies?
330
Lorin Hochstein @norootcause.surfingcomplexity.com · 13/09/2026
The bureaucraticization will continue until resilience improves
071
Lorin Hochstein @norootcause.surfingcomplexity.com · 13/09/2026
*Telstra
110
Lorin Hochstein @norootcause.surfingcomplexity.com · 13/09/2026
Back in July, the Australian telecom Telestra experienced a significant outage on their mobile network. I blogged about the failure mode here: surfingcomplexity.blog/2026/09/12/t...
surfingcomplexity.blog
Thoughts on the July 8th Telestra outage report
On July 8, 2026, the Australian telecom company Telestra experienced a significant outage related to an NTP issue. Telestra hired a consulting firm to do an incident review, and a couple of weeks a…
141
Lorin Hochstein @norootcause.surfingcomplexity.com · 11/09/2026
At least we have neighborhoods.
111
Reposted by Lorin Hochstein
Sheeri Cabral 🍉 @sheeri.org · 11/09/2026
Why don’t we have falses or truthhoods?
111
Lorin Hochstein @norootcause.surfingcomplexity.com · 11/09/2026
If an entity isn’t the source of truth, does that mean it’s the source of falsehoods?
351
Reposted by Lorin Hochstein
alexandra @bigmoodenergy.bsky.social · 28/12/2024
here come old laptop he come bootin up slowly he got broken touchpad he need one more update he got no space on this pc one thing i can tell you its got no warranty
362208744371
Lorin Hochstein @norootcause.surfingcomplexity.com · 06/09/2026
Confession: I had to look that one up. (And my brother is a philosophy professor who studies philosophy of science!)
110
Lorin Hochstein @norootcause.surfingcomplexity.com · 06/09/2026
Tired: anthropomorphization Wired: the intentional stance
110
Reposted by Lorin Hochstein
Jon Phillips @jowiph.bsky.social · 05/09/2026
I picked up a copy of the collected letters of JRR Tolkien for reasons, and this is the most bullshit table of contents I've ever seen. I'm actively angry that it exists
Contents

Introduction - vii

Foreword to the Revised and Expanded Edition - xi

Letters of J.R.R. Tolkien - 1

Notes - 607

Index - 637
2261751062
Lorin Hochstein @norootcause.surfingcomplexity.com · 04/09/2026
They call them edge cases because they’ll cut you
094
Lorin Hochstein @norootcause.surfingcomplexity.com · 04/09/2026
There is nothing more predictable than the emergence of a surprising failure mode
091
Lorin Hochstein @norootcause.surfingcomplexity.com · 04/09/2026
Did you try telling it “Make no mistakes”?
000
Lorin Hochstein @norootcause.surfingcomplexity.com · 03/09/2026
If I worked in reliability at an AI company, I would simply instruct the AI not to go down
1110
Lorin Hochstein @norootcause.surfingcomplexity.com · 03/09/2026
One property of AI agents that is all-too-human: set the incentives high enough, and cheating becomes inevitable.
130
Lorin Hochstein @norootcause.surfingcomplexity.com · 03/09/2026
AI SRE solutions require a high level of crosscheckability. Their diagnoses need to be easy for humans to cross-check.
170
Lorin Hochstein @norootcause.surfingcomplexity.com · 02/09/2026
Another day, another public incident write-up, another blog post with my thoughts on it. This time, it's @buildkite.bsky.social's excellent recent write-up: surfingcomplexity.blog/2026/09/01/t...
surfingcomplexity.blog
Thoughts on the Buildkite Aug 25 incident
The other day, I wrote about a GitHub Actions incident. But GitHub Actions isn’t the only CI-as-a-service out there. Buildkite, which is another CI company, recently released a public write-u…
0114
Reposted by Lorin Hochstein
conputer dipshit @davidcrespo.bsky.social · 20/05/2026
the chair isn't "holding" my ass up, chairs don't "hold" anything. they are merely rigid objects I, the active participant, sit my ass on
212114
Lorin Hochstein @norootcause.surfingcomplexity.com · 30/08/2026
One more blog post for today, about the GitHub Actions incident that happened last week: surfingcomplexity.blog/2026/08/29/q...
surfingcomplexity.blog
Quick thoughts on GitHub Actions Aug 26 incident
Last week, GitHub Actions experienced another incident. As is typical of a GitHub public writeup, it was published very soon after the incident, but there also aren’t isn’t a lot of det…
021
Lorin Hochstein @norootcause.surfingcomplexity.com · 29/08/2026
Wrote a post about some availability risks that I think are fundamentally unavoidable: surfingcomplexity.blog/2026/08/29/o...
surfingcomplexity.blog
Omnipresent availability risks in cloud software
I’m using this post to gather together some common threads I’ve noticed after reading write-ups of major cloud software incidents. By cloud software, I’m referring to software-as-…
180
Reposted by Lorin Hochstein
HL @haitchel.bsky.social · 29/08/2026
Text saying how we used to take selfies back in the 90's, and its a guy standing front of a mirror with an old fashioned payphone
219241
Reposted by Lorin Hochstein
Rich Lafferty @rich.lafferty.ca · 28/08/2026
for extra safety, consider avoiding computers
061
Lorin Hochstein @norootcause.surfingcomplexity.com · 28/08/2026
How about the Metalhead episode of Black Mirror?
000
Reposted by Lorin Hochstein
Jenny Doesn’t Know @jennydoesntknow.bsky.social · 28/08/2026
Muppet version of The Usual Suspects
159836
Lorin Hochstein @norootcause.surfingcomplexity.com · 28/08/2026
Those are the three demons you must slay to achieve reliability
040
Lorin Hochstein @norootcause.surfingcomplexity.com · 28/08/2026
The question is: will he run for president?
020
Lorin Hochstein @norootcause.surfingcomplexity.com · 28/08/2026
This is why the late Robert Novak was underrated as a columnist. He regularly reported on internal conflicts going on inside the GOP.
070
Lorin Hochstein @norootcause.surfingcomplexity.com · 28/08/2026
The three things I worry about most when it comes to major incidents are databases, networks, and migrations. You’re safest if you can just avoid all three completely.
6366
Reposted by Lorin Hochstein
jon christian @jonchristian.net · 28/08/2026
Data shows that Mark Zuckerberg is the Anti-Dolly Parton -- instead of being adored across the political spectrum like Dolly, he is almost universally reviled by everyone, regardless of political beliefs futurism.com/artificial-i...
futurism.com
Data Shows That Mark Zuckerberg Is the Anti-Dolly Parton: Instead of Being Loved by Everybody Across the Political Spectrum, Pretty Much Everybody Despises, Him Regardless of Their Beliefs
While Dolly Parton was universally admired among Democrats and Republicans, Mark Zuckerberg is universally reviled.
69517
Lorin Hochstein @norootcause.surfingcomplexity.com · 28/08/2026
Stop, stop, you’re making Edward Teller cry!
010
Reposted by Lorin Hochstein
Twitter died @countmackula.bsky.social · 25/08/2026
Couldn’t ‘Agree to Disagree’ more
0176
Lorin Hochstein @norootcause.surfingcomplexity.com · 28/08/2026
Two (unrelated) saturation-related public incident write-ups dropped today! The BuildKite one has a ton of great detail. www.buildkitestatus.com/incidents/tm... www.githubstatus.com/incidents/y1...
buildkitestatus.com
Buildkite service disruption
Buildkite's Status Page - Buildkite service disruption.
050
Reposted by Lorin Hochstein
Tikhon Jelvis @jelv.is · 28/08/2026
looking forward to doom engineering
121
Lorin Hochstein @norootcause.surfingcomplexity.com · 28/08/2026
Tired: loop engineering Wired: doom loop engineering finance.yahoo.com/technology/ai/art…
150