A peer-to-peer style guide for writing effective alerts

Every summer, I load up my RV and head out to a dry lakebed in the Nevada desert, but it’s not exactly for a vacation. For two weeks, I’m part of the Emergency Services Department at Burning Man, working in emergency dispatch for a temporary city of more than 60,000 people. I started there as a dispatcher in 2010, and I’ve been a Dispatch Supervisor since 2012. The ESD dispatch center looks more like a high-tech operations center than you might expect. It runs a modern computer-aided dispatch (CAD) system and a municipal-grade two-way radio system, and it operates around the clock. Cell coverage is spotty when that many people descend on an empty stretch of desert, so nearly all of our emergency calls come in by radio, from event staff, participants, and partner agencies. We coordinate emergency response across ESD and its partners: the Black Rock Rangers, the on-site advanced medical care facility, and federal, state, and local law enforcement. The stakes are immediate and physical in a way that a software outage rarely is. As a Dispatch Supervisor, my job is to keep track of the overall picture: every active incident, conditions throughout the city (from white-out dust storms to roads made impassable by rain), and the status of the emergency response system as a whole (from the function of the radio system to the availability and location of emergency units of various types). I direct the dispatchers’ work, handle the escalations that don’t fit the routine, and serve as the liaison to our partner agencies. I take a radio channel myself only in a few situations: when we’re short-staffed, while a dispatcher takes a break, or when I urgently need to relay critical information directly to a responding crew, such as something I’ve just learned from our law enforcement partners. Those are deliberate, temporary exceptions. It’s similar to the division of labor I recommend between an incident commander and the engineers doing the technical work: the coordinator can step in briefly when it helps, but by default stays hands-off so that someone is always watching the whole picture. Back in the tech world, I help engineering leaders strengthen their companies’ incident response. Many of the practices the public safety world takes for granted are unfamiliar to their teams, even though those practices adapt readily to software. I’ve been working across both of these worlds for more than thirty years. I started as a volunteer search and rescue pilot while building a tech career in Silicon Valley, and the two tracks have run in parallel ever since. On the public safety side, that’s meant search and rescue missions, CERT instruction, and emergency dispatch. On the tech side, it’s meant building Google’s incident management system, leading incident response at Slack, helping companies from startups to large enterprises strengthen their incident management capabilities as a consultant, and now writing a book, _Incident Management for DevOps and SRE_. Each side makes me better at the other. ## What you see from both sides When you’ve worked across both fields long enough, certain things stand out that are hard to see from within either one alone. **Incidents are normal work, not failures.** For a fire department, emergencies are routine business. Nobody blames the department when it has a busy week; the questions are how well the crews handled the calls, and how to prevent similar ones in the future. Software is different in one important way: the people who respond to incidents are usually the same people who build and run the systems, and incident response competes with everything else on their plate. That makes it easy to treat each incident as an interruption, or as evidence that someone failed. But anyone operating complex systems will have incidents, and responding to them is part of the work of building and running those systems. Many tech companies’ cultures haven’t yet embraced that. They still treat incidents as something between an embarrassment and an accusation, which makes people reluctant to declare them, reluctant to escalate, and reluctant to be forthcoming afterward about what happened. **The hard part isn’t technical.** In both worlds, the technical work matters enormously, whether that’s debugging a cascading failure or treating a patient with a heat emergency. But whether the response goes well or badly usually comes down to coordination. How do you organize people who arrive at different times with different information, different skills, and different concerns? How do you communicate clearly when the situation keeps changing? How do you make decisions when you can’t wait for all the facts? These questions are the same in both fields. The Incident Command System, developed by fire departments in the 1970s and refined by emergency agencies ever since, exists to answer them. It was built to solve coordination problems in emergencies, which is why its principles transfer to software incidents even though the technical work underneath is completely different. **The initial response is decided before the call comes in.** Public safety agencies think through in advance what kinds of calls they’re likely to get and what the initial response to each will be. At Burning Man, for example, a small fire gets one engine, while a large fire gets three engines and a water tender (a large tanker truck). A minor medical call such as a twisted ankle gets a QRV (Quick Response Vehicle: a small, nimble utility vehicle with a medical crew and minimal equipment, which can more easily move through the crowded roads and camps than a full-size ambulance), while a major medical call such as a heart attack gets a QRV, an advanced life support ambulance (which carries more equipment and emergency drugs for patient treatment), and a supervisor. With that dispatch matrix, nobody has to work out, in the middle of an emergency, how much help to send: the dispatcher classifies the call, and the matrix provides a starting point that responders can adjust once they’re on scene, scaling up, scaling down, or adding a different kind of help, such as a crisis intervention team when a call turns out to involve a mental health issue. The tech equivalent is the severity matrix, which many companies aren’t using to its full potential. **Good incident response looks boring from the outside.** When a fire department arrives at a structure fire, the coordination is almost invisible. The incident commander is tracking the situation, allocating resources, and communicating with incoming units, while the crews fight the fire. Everyone knows their role and what information they need to share, and outsiders often marvel at how calm and focused the responders are. The smooth responses are unremarkable; the dramatic ones are the exceptions. Contrast that with many tech companies, where every incident is handled ad hoc, the most senior engineer is simultaneously debugging and trying to coordinate, and nobody quite knows who’s doing what. ## What this means for tech companies That contrast between structured coordination and improvising from scratch is one of the most common gaps I see when I assess companies’ incident management programs. The teams aren’t short on technical skills or motivation. They’ve never seen what emergency coordination looks like when it’s a routine capability rather than something that gets reinvented every time. I don’t want to oversell the analogy. Software incidents and physical emergencies differ in important ways: reversibility, speed of communication, the ability to respond from anywhere in the world, and the physical danger responders face. The Incident Command System as practiced by a fire department doesn’t transfer directly to a software team, and anyone who tells you it does hasn’t spent enough time in both worlds to see where the parallels break down. What transfers are the underlying principles. For example: coordination needs explicit structure; the person coordinating generally shouldn’t also be doing the technical work; communication patterns should be designed in advance; and incidents are a normal part of operating complex systems. Those principles are robust across domains. They’ve been tested under pressures that make a software outage look gentle by comparison. And the companies I’ve worked with that have adopted them, adapted for their own context, handle incidents with noticeably less chaos, less stress, and less customer impact than the ones still improvising from scratch. That’s the perspective I carry back and forth between the dispatch center and the consulting engagement: the conviction that the coordination problems companies struggle with during incidents are neither new nor unsolvable, and that the path forward doesn’t require inventing anything from scratch. It requires studying what’s already been learned in fields where the stakes have always been high, and asking which of those lessons fit. I’m writing **Incident Management for DevOps and SRE**, a practitioner’s book for incident responders, incident commanders, and people building incident management programs. It distills what I’ve learned leading incident management at Google and Slack and consulting for companies worldwide. Sign up at im4ds.com to hear when it’s available. If your company needs help with incident management, my consulting practice is all about engineering better incident management. ### Related Posts * Routine Emergencies * Was your last outage like Apollo 13, or more like Titanic? * Why is it important to separate the Incident Commander and Tech Lead roles?

I’ve been coding with AI (specifically Kiro) assistance at work lately. There are a lot of things I like about it, but it took some tuning to make it more useful and less annoying. With kiro, these should be placed in .kiro/steering, either in your home directory for global settings or if you only want them to apply to a single repository, in $REPO_ROOT/.kiro/steering. With VS Code and Copilot, they belong in $REPO_ROOT/.github/prompts.

Making installation easier and putting a new wrapper on the interface while leaving most of the security to users is a recipe for more trouble with the popular agent harness

The world is currently experiencing what is expected to become the strongest El Niño on record – dubbed a “super El Niño” by many.

A list of IOT devices that are garbage. Contribute to unixorn/internet-of-trash development by creating an account on GitHub.

Differences between “essential vocabulary” lists, 1953 vs. 2023

Is it coddling when an on-call engineer takes the next morning off to recover after handling a production incident at 3 a.m., or is it a smart company managing a reliability risk? Here’s what that night actually looks like. The engineer gets paged at 3 a.m., then spends two hours diagnosing the problem, coordinating with fellow responders, and restoring service. By 5 a.m., the incident is resolved and they get back to bed, but it takes them a while to settle down and get back to sleep. Four hours later, they’re at standup. That afternoon, they’re in a planning meeting. That night, they’re still primary on the pager. This is the default at most companies. Nobody made a deliberate decision that it should work this way; it’s just what happens when there’s no explicit policy for post-incident recovery. And it carries more risk than most leaders realize. ## Incident response is more fatiguing than regular work Responding to an incident isn’t like a normal day of developing features and chasing bug reports. The cognitive demands are qualitatively different: rapid context-switching under time pressure, high-stakes decisions with incomplete information, coordinating across multiple people and systems, all while knowing that users are affected and stakeholders are watching. And there’s a physiological dimension that regular engineering work rarely triggers: adrenaline. Incident response activates the body’s stress response in a way that writing code or reviewing a design doesn’t. That heightened state feels productive in the moment, but it depletes reserves fast, and the crash afterward is steeper than the apparent effort would justify. This, incidentally, is one of the reasons that training and practice matter so much. Responders who’ve rehearsed the process and trust the framework around them experience a less intense stress response when real incidents hit. Turning incident response into a “routine emergency” doesn’t just improve efficiency; it reduces the physiological toll. An engineer who’s been actively responding for a few hours isn’t just tired in the way that a long day makes you tired. They’re measurably less effective at exactly the skills incident response demands: integrating new information, evaluating competing hypotheses, making decisions under ambiguity, and recognizing when a current approach isn’t working. ## The degradation is predictable Responder fatigue follows a recognizable pattern. As it sets in, people stop processing new information as effectively. They agonize over decisions they’d normally make quickly, or they stop making decisions altogether. They develop tunnel vision, fixating on the one theory they’re already pursuing instead of stepping back to consider alternatives. They fall into ruts, essentially pursuing “Plan A, again, with more feeling this time” instead of asking whether Plan A is still the right plan. They get less creative, more rigid, and more prone to mistakes. Fire departments study this, because it’s exactly the scenario firefighters face: interrupted sleep from overnight emergency calls, then back on duty the next day. The research consistently shows that the kind of fragmented, insufficient sleep they get around overnight calls degrades next-day cognitive performance to levels comparable to having had a couple of drinks. That’s why a growing number of fire departments have reconsidered their traditional 48-hour shifts; the performance degradation on day two is bad enough that departments are restructuring around it. ## Self-reporting isn’t enough The insidious part is that fatigue undermines exactly the capacity you need to recognize it. A fatigued responder genuinely believes they’re performing normally. That’s simply how fatigue works. “I’m fine, I can keep going” isn’t evidence of fitness. It’s one of the symptoms. This is the critical organizational point. If a company leaves fatigue management to individual judgment (“take it easy if you need to”), it’s built a system that depends on impaired people accurately assessing their own impairment. Aviation learned this the hard way. The FAA doesn’t ask pilots whether they feel too tired to fly. It sets hard limits on duty time and required rest periods, because decades of accident investigation proved that self-assessment under fatigue is unreliable. Pilots who’d been awake for 20 hours consistently reported feeling capable. The data said otherwise. If you’ve ever had a flight delayed while the airline sought a new crew because the original crew had “timed out,” you’ve seen these rules in action. The tech industry hasn’t had its equivalent reckoning yet, but the same cognitive science applies. An engineer who handled a two-hour incident at 3 a.m. and says they’re fine at 9 a.m. may well believe it. That doesn’t mean they’re right, and building your next day around that assumption is a gamble most companies don’t realize they’re taking. ## What active fatigue management looks like Companies that take responder fatigue seriously don’t rely on individual heroism or self-assessment. They build a few specific practices into their incident management capability. _Explicit rest expectations._ Not “take it easy if you need to,” but clear guidelines: an engineer who responds to a significant incident overnight is expected to start late or take the morning off, depending on duration and severity. The default is rest; working the next morning is the exception that requires a conscious choice, not the other way around. _The incident commander (IC) monitors for fatigue._ During extended incidents, it’s the IC’s responsibility to watch for fatigue signals in responders: slowed decision-making, tunnel vision, repeated questions, irritability, loss of situational awareness. This is the same responsibility a fire officer has for monitoring crew fatigue on a fireground. A fatigued responder who stays on the line isn’t being dedicated; they’re becoming a risk to the response, their teammates, and themselves. On incidents that stretch beyond a few hours, this includes planning responder reliefs early rather than waiting for someone to admit they’re spent. _Promote the backup._ After a significant overnight incident, consider moving the backup on-call engineer to primary for the next 12 to 24 hours. The person who spent two hours at 3 a.m. restoring service is not the person you want as your first line of defense if something else breaks that afternoon. _Rethink shift length._ Most teams seem to default to week-long on-call shifts with several weeks between shifts, but there’s a strong case for shorter, more frequent shifts. The same logic driving fire departments away from 48-hour shifts applies: shorter shifts mean less accumulated fatigue per shift, even if each person’s total on-call hours per quarter are similar. Shift design is a fatigue management decision, whether your company treats it as one or not. Some incident management platforms are starting to build this awareness into their tooling. incident.io, for example, detects overnight pages and proactively asks the responder the next day whether they’d like someone to cover their next shift. That’s the right instinct: making fatigue management a system-level concern rather than leaving it to the judgment of the person who’s least equipped to assess it. ## It’s a reliability decision Most companies aren’t actively choosing to ignore a fatigue problem. Rather, they have a fatigue problem that they haven’t noticed yet, because nobody has framed it as an operational risk. When a leader says “we trust our engineers to manage their own energy,” what they’re actually saying is: we have no organizational mechanism for ensuring that the people responding to our next incident are cognitively fit to do so. Respecting fatigue isn’t coddling. It’s protecting the quality of everything your engineers do the next day, including the next incident response. I’m writing **Incident Management for DevOps and SRE**, a practitioner’s book for incident responders, incident commanders, and people building incident management programs. It distills what I’ve learned leading incident management at Google and Slack and consulting for companies worldwide. Sign up at im4ds.com to hear when it’s available. If your company needs help with incident management, my consulting practice is all about engineering better incident management. ### Related Posts * Heroic saves are near misses * How often should your engineers be on call? * Routine Emergencies

A collection of git utilities, useful extra git scripts, tutorials and other useful articles. - unixorn/git-extra-commands

Most companies that invest in improving their incident management see something counterintuitive in the first few months: their incident count goes up. And someone in the leadership chain, looking at the dashboard, asks with concern: “Why are things getting worse?” The good news is, they’re probably not. What you’re really seeing is evidence that your incident culture is getting better and stronger. Incident count doesn’t measure system health; it measures how willing people are to declare an incident. When you invest in your incident management program by giving people better training, tools, and processes for handling incidents, they put those things to work, and more situations get treated as incidents. Problems that used to be handled informally (a “spicy bug” that someone handled without declaring an incident, a degradation that the on-call engineer white-knuckled through without telling anyone) now enter your incident process. The company is getting visibility into problems that used to go unnoticed, and using incident management practices and tools to address problems it would have struggled with before. And it’s a self-reinforcing cycle; the more people use these skills and tools, the more comfortable they get with them, and the more they’re inclined to use them. That’s a good thing, as the way to build the skills and confidence to handle big incidents is by handling lots of little ones; the small incidents are an invaluable training ground. In most companies, you’re dealing with increases in multiple dimensions simultaneously: number of users, number of products, number of features in those products, usage of those features, number of engineers, level of training and experience of those engineers, and many more. With all those factors increasing, why would you expect incident count to decrease? In a very real sense, rising incident counts can be a sign of success, not failure. The question to ask isn’t “why are we having more incidents?”, it’s “why aren’t we?” And from there, other good questions follow: are we handling our incidents well, do we have the tools and training that we need, are we learning everything we can from every incident, are we preventing future incidents, are we better prepared to handle those we can’t prevent? But what happens if leadership decides that rising incident count is a problem and sets a target to bring it down? People get the message: fewer incidents is better. So marginal incidents stop getting declared. The “spicy bugs” go back to being handled quietly. The degradations get white-knuckled through again. The number on the dashboard goes down, but the company has lost both the visibility its incident process was providing and the benefits of handling those situations with proper coordination, communication, and prioritization. The problems didn’t go away; you just stopped applying your best tools to them. # The Goodhart’s Law problem This pattern has a name: Goodhart’s Law. When a metric becomes a target, it ceases to be a good metric. And it’s not just an incident count problem; it recurs across every incident metric companies reach for. The logic is straightforward. You focus on a particular metric because you think it captures something you care about. You set a target for the metric because you want to improve. And then smart, well-intentioned people find ways to hit the target. Some of those ways involve actually improving the thing you care about. But some involve optimizing the number without improving the underlying reality, and over time, the second category tends to dominate. Make incident count a target for reduction, and marginal incidents stop getting declared. Make MTTR a target, and people close incidents prematurely. Make action item completion rate a target, and people write easy action items instead of hard ones. In every case, the metric looks better while the thing you actually care about (learning, reliability, preparedness) stays the same or gets worse. This is human nature. Smart people optimize for what gets measured; that’s how incentives work. You’re not going to prevent it by writing sternly worded memos about gaming the system. You can only manage it by choosing metrics carefully and then paying attention to the behaviors driven by your focus on those particular metrics. Adjust when those behaviors aren’t what you intended, and be willing to retire the metric when it’s doing more harm than good. # MTTR: the metric everybody loves and nobody should trust Mean Time to Recovery is the most misleading incident metric in the industry. Leadership loves it because it’s a single number that appears to capture “how fast we fix things.” But it’s deeply flawed, both mathematically and in the incentives it creates. Incident durations follow a power-law distribution: most incidents resolve quickly, while a small number take much longer. When you average power-law data, you get a number that describes nobody’s actual experience. Let’s imagine that last month you had ten incidents, nine of which resolved in about ten minutes each, and one that took six hours. That gives you an MTTR of 45 minutes, but that’s nowhere close to what any of those incidents actually took; it’s way off of both ten minutes and six hours. Google SRE Štěpán Davidovič, in _Incident Metrics in SRE: Critically Evaluating MTTR and Friends_ (O’Reilly, 2023), used Monte Carlo simulations to demonstrate that even with a substantial dataset, MTTR can’t reliably tell you whether your incident response is actually improving. The math doesn’t just give you a misleading number; it can’t even detect real improvement when it’s happening. There’s also a flattening problem: MTTR treats all incidents as interchangeable, as if the only thing that matters about an incident is how long it took. A six-hour incident where page load times were degraded but the service was still usable somehow scores worse than a one-hour total outage. The incentive problems are even worse than the mathematical ones. MTTR incentivizes speed over understanding. Thorough incident response sometimes means deliberately slowing down: carefully analyzing symptoms, verifying that a fix actually works, understanding the contributing factors well enough to prevent recurrence. MTTR punishes all of that. A team that’s genuinely improving (catching issues earlier, preventing cascades, tackling more complex problems) can see its MTTR stay flat or even go up. The metric undermines morale and leadership confidence even as the team does better work. # What fire departments get right about metrics Here’s a lesson that most software companies could learn from fire departments. Well-managed fire departments decompose their response timeline into segments and set targets only on the segments they can actually control, and expect to be fairly consistent across their incidents. Dispatch time (how long from answering the 911 call to notifying the fire crew) gets a target. Turnout time (how long from notification to crews leaving the station) gets a target. Drive time (from the crews leaving the station to arrival at the incident scene) gets a target. These are process steps that are largely consistent from one call to the next, and if they’re too slow, you can do something about it: hire more dispatchers, change how crews stage at the firehouse, build more stations. What fire departments don’t set targets for is how long it takes to put out the fire. That depends on the fire. A dumpster fire and a fully involved warehouse fire are different problems with different durations, and no fixed target could meaningfully apply to both. How long it takes to set up an incident channel, how long it takes responders to acknowledge pages, how long it takes the responders to join the channel and get started: these are your equivalent to the fire department’s dispatch chain. They’re largely consistent from one incident to the next, and you can set targets for them. If you’re not meeting the targets, there are obvious adjustments you can make: auto-create incident channels, set and enforce clearer on-call expectations, and so forth. On the other hand, the time to find the contributing factors, the time to implement a durable fix, and the total time for the incident: those depend on the details of the particular incident. A misconfigured feature flag and a cascading database failure are different problems. Track trends, investigate outliers, learn from reviews. But don’t set targets. Targets on metrics you can’t control produce gaming, demoralization, or both. # Start with questions, not metrics The most useful thing is to stop asking “what should we measure?” and start asking “what questions are we trying to answer?” Are incidents being handled well? Are we learning from them? Is the incident management program serving the business? Is our on-call workload sustainable? Each of these questions leads you to seek different evidence, some quantitative, some qualitative, and the answers are more useful than any single number on a dashboard. These aren’t easy questions to answer, but they’re better than an easy number that misleads you (like MTTR). Your dashboards should make you curious, not confident. When you see a trend, the right response isn’t “we know what’s happening.” It’s “we should dig in and find out why.” And if your incident count went up this quarter? Before you panic, ask why. You might find that your increased focus on incident management is doing exactly what it’s supposed to do. * * * _This is one of the topics I cover in depth in my upcoming book, Incident Management for DevOps and SRE. If you’d like to hear when it’s available, you can sign up atim4ds.com._ _If your company needs help with incident management right now, my consulting practice isGreatCircle.com/im._

“Oh dear. I left my laptop open, and you read a LinkedIn post from an ‘Entrepreneur, Change Agent, Futurist, Success Enabler, Innovator & Investor.’ And ...

My fourth month of unemployment. I'm fresh off an interview with a senior technical role advertised with a decent compensation range whose upper end would let me make median rent, and whose lower range would not. At the end of an excellent interview, I am informed that said compensation range,

Who was the greatest incident commander of all time? My money is on Gene Kranz. If you’ve seen the 1995 movie _Apollo 13_ (and if you haven’t, you should; it’s a gripping tale even though you know the outcome, and along the way it’s a master class in incident command), you’ve seen Ed Harris portray Kranz during one of the most harrowing incidents in the history of human spaceflight. In April 1970, an oxygen tank explosion crippled the Apollo 13 spacecraft roughly 200,000 miles from Earth, turning what was supposed to be the third Apollo landing on the Moon into a desperate fight to bring three astronauts home alive. Kranz was NASA’s lead flight director in Mission Control throughout the multi-day crisis. What makes Kranz the gold standard for incident command isn’t what he knew. It’s what he _did_ (and more importantly, what he _didn’t_ do). # His job wasn’t to solve the problem Watch the crisis scenes in the movie closely. Kranz doesn’t try to solve the technical problems himself. He has a room full of brilliant engineers for that. Each of those engineers, in turn, is backed by a “back room” of specialists focused on specific spacecraft systems. Kranz’s job isn’t to solve the problem. His job is to ensure that the problem gets solved, and that’s a subtle but critical difference. That distinction is fundamental to effective incident command, and many companies get it wrong. The incident commander role often falls to the most senior engineer by default, with no regard for the practical consequences. The technical work suffers because their best problem-solver keeps getting pulled away to handle logistics and communication, and the overall response suffers because nobody is focused full-time on running it. # What he actually did instead Kranz was technical enough to understand what his engineers were telling him, and trying to tell each other, but he wasn’t running the calculations himself. So if he wasn’t personally solving the problem, what was he doing instead? His focus was on ensuring that the right people were working the right problems at the right time, not on directly working any of the problems himself. Watch the movie scenes carefully, and you’ll see a masterclass in incident command: **He organized the response.** He set priorities, assigned responsibilities, and made sure every critical task had someone working on it. When the crisis began, he immediately reframed the mission: “From this moment on, we are improvising a new mission: how do we get our people home?” **He demanded clear information.** He asked sharp questions and required his team to give him straight answers, not hedged guesses. When the crisis began and readings were contradictory, his instinct was to cut through the speculation: “Let’s work the problem, people. Let’s not make things worse by _guessing_.” **He made decisions under uncertainty.** His engineers disagreed about whether to attempt a direct abort or use the Moon’s gravity to slingshot the crew home. Both options carried enormous risk. Kranz listened to the competing proposals, weighed the tradeoffs, and decided. Then he explained his reasoning so everyone was on the same page. **He kept them looking for possibilities, without being fixated on problems.** While his engineers were focused on the individual failures they were trying to address, Kranz stepped back and asked: “What do we have on the spacecraft that’s good?”, challenging them to look for opportunities to use capabilities in unexpected ways to further the mission. When told that a certain system wasn’t designed to do what they needed, he shot back: “I don’t care about what anything was designed to do, I care about what it CAN do.” **He facilitated, not dictated.** When engineers brought counter-proposals, he considered them and changed his mind when they were right. When discussion got heated, he reasserted calm: “Let’s hold it down, people.” He allowed debate when it was productive and cut it off when it wasn’t. **He managed the emotional temperature.** In military and emergency management circles, this quality is called “command presence.” It doesn’t mean barking orders or being stoic. It means being the steady center of gravity for the response. Responders unconsciously look to the incident commander for cues about how bad things are and whether the situation is under control. If they’re calm and focused, that calmness radiates to the team. If they’re frantic, the team gets frantic too. Kranz was the calmest person in the room precisely when the room needed calm the most. Not because the situation wasn’t dire, but because he understood that if he lost his composure, everyone else would too. # “Failure is not an option” Kranz’s famous line from the movie has become a cliché, but it’s worth revisiting. Watch the scene again. It’s not bravado. It’s a statement of intent from someone who has assessed the situation, accepted the constraints, and decided that the team is going to find a way through. He doesn’t pretend the situation isn’t terrible. He tells his team exactly how bad it is, exactly what they need to do, and then sets the expectation that they will find a way to do it. That’s what good incident command looks like. Not magical thinking, not denial, not heroics. Clear-eyed assessment, clear decisions, and relentless follow-through. # The lesson for your incident response Your incidents probably don’t involve life-or-death stakes while the whole world watches anxiously. But the model is the same. The incident commander’s job is to ensure that the problem gets solved, not to solve the problem themselves. That means organizing the response, making decisions, tracking what’s been tried and what hasn’t, communicating with stakeholders, and being the steady center of gravity so everyone else can do their best work. Many companies default to making their most senior engineer the incident commander. It seems reasonable: they have the most authority, the most experience, the most knowledge. But your incident commander doesn’t need to be your top engineer. They need to be the person who can stay calm, make decisions with incomplete information, facilitate a room full of smart people who disagree, and ensure that nothing falls through the cracks while everyone else focuses on the technical work. Gene Kranz showed what incident command looks like when it’s done right. More than half a century later, he’s still the gold standard to aspire to. * * * _I’m writing a book about incident management for software engineering teams, drawing on lessons from both the tech industry and public safety (and movies about NASA!). If you’d like to hear when it’s available, visitim4ds.com._ _If your company needs help with incident management right now, my consulting practice isGreatCircle.com/im._