On This Page
Looking Beyond Recovery
Obviously I don’t want outages. I don’t want customer impact, I don’t want 2 AM pages, and I don’t want the business losing revenue or reputation. Nobody does. But once the immediate issue is resolved, my focus shifts. I’m no longer trying to restore service. I’m trying to understand what the outage just taught us.
Most people naturally focus on the application that failed, and they should. Something broke, it impacted customers, and the immediate priority is getting the service healthy again. Once that’s done though, I think it’s easy to miss one of the biggest opportunities an outage gives you. The application isn’t the only thing that was just tested. Your observability platform was too. Every outage tells you something about how prepared your organization was to detect, understand, and respond to the problem.
What the Outage Actually Tested
Every outage exercises far more than the application that failed. It tests the observability platform around it, the processes used to respond, and the people relying on both. Those pieces don’t exist in isolation. They’re meant to work together, and a production outage is one of the few times they are all put under pressure at the same time.
I don’t mean audit in the formal sense. You’re not going to come out of every outage with a checklist of action items, nor should you. What you do get is an honest evaluation of how your observability platform performs when people actually need it. That’s difficult to simulate, and it’s one of the reasons I think production incidents are such valuable learning opportunities.
That’s why I think every outage is an observability audit.
When I start looking back at an incident, I’m not just asking what failed. I’m trying to understand how we ended up here in the first place. That leads to a completely different set of questions. Did alerting detect the issue early enough? Did Operations understand what they were looking at? Did the engineer who got paged have enough context to start investigating immediately?
Sometimes the application is exactly where the problem starts and stops. More often than not, the outage exposes something larger. The observability platform may have detected the failure exactly as expected while gaps elsewhere in the response made the incident harder to understand or slower to resolve. To me, those gaps are all part of the observability platform, even when they aren’t software.
What Controlled Testing Cannot Recreate
One of the reasons I like Chaos Engineering is because it’s an intentional way to build confidence in a system. It validates assumptions, tests known failure scenarios, and helps teams prepare before something goes wrong. It’s an incredibly valuable practice, and one I’d encourage organizations to invest in.
That’s also why I don’t think Chaos Engineering replaces production incidents. By necessity, it operates within planned scenarios and predefined boundaries. Production outages don’t respect those boundaries. They expose the gaps nobody thought to test.
Sometimes that’s a technical gap. Sometimes it’s a process gap. Sometimes it’s simply two unrelated failures happening at the same time in a way nobody anticipated. I’ve seen situations where the technology behaved exactly as expected, but confusion around ownership or communication added far more time to the incident than the technical issue itself. Those are lessons that are difficult to manufacture in a controlled exercise because they involve people just as much as they involve systems.
Do Not Waste the Lesson
None of this means outages are a good thing. They’ve already happened, and they’ve already cost the business something. Once service has been restored, don’t waste the opportunity to understand what they taught you.
What I do appreciate is that outages expose assumptions. They expose blind spots. They expose the places where our observability platform, our processes, and our organization weren’t quite as prepared as we believed they were.
Not every outage needs another alert or dashboard. Sometimes the lesson has nothing to do with adding more telemetry at all. The value isn’t in forcing an observability change after every incident. The value is in taking the time to understand whether one is needed.
The outage may have started because an application failed, but once it’s over, you have a rare opportunity to evaluate everything that helped your organization detect, understand, and respond to it. If you’re willing to look beyond the application itself, you can strengthen the systems around it. You won’t prevent every future outage, but you can make the next one easier to understand, easier to respond to, and ultimately less painful for everyone involved.