iiinova — where your digital self keeps evolving

How Picnic uses incident.io to combine digital and operational reliability

And as if that’s not enough, they’re also one of the nicest and most motivated teams we have the pleasure of working with. Thanks team Picnic! 🚚🍞🛒 Full video here: www.youtube.com/watch?v=uJt...

youtube.com

000

“We are now moving to a level where we are building up structural technical resilience, meaning whenever there's an incident happening, there's no longer customers affected, and that is basically the ultimate goal that we are striving for.”

100

This comes off the back of a full migration from PagerDuty to @incident_io On-call, and a replacement of their in-house incident response tool too. And in terms of impact, I really can’t put it better than Daniel Gebler, their CTO:

100

With all this in mind, it really is a huge honour for @incident_io to be the software they use to help detect, respond to, and mitigate issues as they happen.

100

And when Picnic talk about combining the digital and operational worlds, they really mean it. There’s bespoke software running everywhere, and Grafana dashboards monitoring the status of thousands of devices, sensors, and systems.

100

and was treated to a tour of one of their automated fulfilment centres. Suffice to say, it’s an incredible operation, where robots and humans work side-by-side to get groceries ready for delivery.

100

I had the absolute pleasure of visiting the team to discuss their rollout of @incident_io

100

Behind the scenes, it's a complex hybrid business, combining a digital platform with a vast physical network of warehouses, delivery hubs, and a fleet of thousands of electric vehicles — all running in perfect coordination to keep millions of customers stocked up and smiling.

100

Picnic is an online supermarket on a mission to make grocery shopping simple, fun, and affordable. And they're clearly a company with great taste, choosing @incident_io to help them achieve their high bar for reliability.

220

At incident, we’re trying to build in the opposite direction. Instead of replacing people, we’re designing AI to show up like your best engineer. Not the brilliant jerk who fixes the thing while you're grabbing a coffee, but the one that helps you understand, decide, and act when things are on fire.

People get good at watching systems, not running them. Then something breaks, and we expect them to swoop in and save the day, with little context and limited practice.

100

But there’s a catch. Lisanne Bainbridge highlighted it back in 1983 in her paper "Ironies of Automation": the more we automate, the more we deskill.

101

My take: the only viable solution is to use AI to support the people operating these systems. It’s not unlike modern aircraft—too complex for a human alone, so we rely on automation to augment the pilot, not replace them.

As AI enables faster software development and systems become harder to fully understand, we’re left with a growing problem: how do we safely operate what we no longer fully grok?

100

After six months of watching our engineering team deep in the AI trenches, I’m more convinced than ever: AI isn't just a nice-to-have in incident management—it’s a necessity.

100

So when you're debriefing next and every fibre of your being wants to criticise someone for taking a risky action that made things worse, consider how different you'd be feeling if their heroics actually paid off.

000

Take the forklift-jet-engine saga... Had the forklift flipped over and injured 4 adults, this would be a very different post. And yet, I'm here congratulating the ingenuity.

What's interesting to me is that these off-piste actions are often highly dangerous, and when it comes to debriefing, there's a fine line between blame if it goes wrong, and celebration when it goes right.

100

But what does this have to do with incidents? Incidents are a place where things that have never been done before, happen all the time. And when it comes to debriefing, it can be fascinating to dig into the actions that are taken to understand how things really get done.

100

Then, they fired up a small tractor, balanced it's front loader on the pickup, pushed down so its full weight was on the forklift, and lo-and-behold it worked. But don't take my word for it. Check out the video of the whole thing below!

100

What did they do? First up, they got 4 adults to stand on the back of the forklift as a counterbalance. Turns out we were nowhere near heavy enough.

100

But the implications of doing so were high: The truck couldn't wait, so would have to take the engine to be unloaded elsewhere, the cost of the forklift was sunk, and the cost of hiring a crane likely significant. <Insert joke here about them being able to afford it>

100

In any kind of normal mode of operation, this is where you'd go back to the drawing board, replan and probably order a crane to unload it.

100

It showed up on a truck the size of a small country, and the owner had a forklift ready to unload it. Only the forklift was severely unpowered, and insufficiently weighted to lift the thing up. Turns out jet engines are pretty heavy at ~4.5 tons.

100

I recently stayed at an Airbnb where the owner (a self-confessed plane nerd) had ordered a disused 747 jet engine to be a show piece on the property.

100

Chris Evans @evns.io · 23/01/2025

What does unloading a 747 jet engine from a trailer have to do with incident management? Great question...

100

I sat down with @lawrencejones.dev and our AI PM earlier this week to chat more about it. Full recording here if you're interested: youtu.be/rNpwZPOUhuE?...

youtu.be

Building an AI incident responder

YouTube video by incident-io

030

Chris Evans @evns.io · 23/01/2025

We're building an AI incident responder 🔥 Not to replace humans, but to work jointly alongside them. Initially, we're focused on accelerating the investigation and diagnosis stages of an incident, helping search and correlate data points, signals and context from across your organization.

141

Chris Evans @evns.io · 15/01/2025

Fundamentally, has it done the wrong thing, or displayed the wrong thing? Two very different criticisms of the system.

1000

Chris Evans @evns.io · 15/01/2025

Feels like an awful lot of hate for Tesla's here 😅 Unclear from this what's going on. It could literally be a rendering issue, right? i.e. car knows something is there, knows it's a moving vehicle, and because it doesn't have the 3d models for a train it defaults to stretched car?

340

Chris Evans @evns.io · 06/01/2025

What's the issue you see here? Genuinely curious as I hadn't thought anything of it in the past 🤔

010

Chris Evans @evns.io · 06/01/2025

I dunno, looks like a nice warm tent 😂

Chris Evans @evns.io · 17/12/2024

When your automated "Create a Google Meet" software retries a little too aggressively 😅

1110

Sounds like we might be talking about different levels of “egregious” behaviour. Sympathise if it’s having a consistent detrimental effect on the community.

010

I should clarify, perhaps importantly, that I’m doing this as someone sponsoring a conference so supporting the ecosystem.

100

Perhaps just different worlds! As a company owner, I think it’s ok to selectively invite people who I’d like to talk to, given it’s at my own cost. I’d apply a similar lens to a group of past colleagues or friends arranging a dinner for themselves, and selectively inviting people there.

100

I'm not sure what the alternative is? If there are hundreds/thousands of people at a conference, many small, exclusive, and implicitly invite-only groups will form. And I think people regularly turn down free dinners if they're not interested in the people, domain or company, right?

100

I think it's quite reasonable? People can freely vote with their feet, and it's not possible for a conference to cater to everyone's needs. I, for example, really don't like large conference networking events. Find them a bit much and would prefer to attend a smaller dinner.