Presentation: What Breaks Our Systems: A Taxonomy of Black Swans

I'm a Senior Staff Production Engineer at Slack, I've just started here. In terms of the work that I do here I don't know yet. This is my second week. At Google, I used to run large production services. I cared about primarily reliability and availability: is my service up, is my service performing well, are my users happy?

One of the things that I think that we are not great at as an industry is thinking about outlier events, particularly with the way that SRE has gained popularity in the last three or four years. People started talking a lot about SLOs, error budgets, and maintaining those on a week to week basis or month to month basis. I think this is great but I think that one of the things that gets lost in that view of the world is thinking about what are my systemic risks? What are the things that can go really, really wrong with my systems. What are the things that could take me down for hours, days, maybe even longer, potentially even destroy the business. I think that worrying about those kinds of scenarios should be a core part of the job of site reliability engineers, production engineers, senior engineers. That's what this talk is about.

We can't predict an unknown-unknown. The crux of this talk is that my unknown-unknown is maybe like an incident that you had six months ago. Or it might be something that follows a pattern. An example that I use pretty early on the talk is canarying. Over the last few years the practice of canary testing has become extremely common - this is where we take our code or config or any kind of change, we deploy it on a subset of our machines early on. The point of doing this is so that we can tell if it's going to fail catastrophically, or even if there’s smaller regressions we can look at them and understand impact before we deploy to our full system. What we're trying to do with canarying is not to try and find any specific bug, but trying to create this generic defense against regressions or breakages caused by change. It's not perfect but it's pretty effective. What this talk tries to look get at is what are the other emergent best practices that we're going to see in the next number of years. What are things that we can do that can defeat entire categories of black swans that might otherwise take our systems down?

This is not a canary talk so I'm not going to talk in any great depth about canarying. But what I would spend some more time on is the actual production incidents. I think I've got something like 15 incidents in this talk that are divided across six different categories. And what I'll do is spend most of the time on those. For example, the first category that I'm going to look at is hitting limits that you didn't know were there. I've got a variety of five or six different incidents to discuss under that category, everything from connections under Amazon Web Services to Postgres transaction IDs. There are some strategies to try and find these problems ahead of time. I've got more time in this slot to go into some more specifics, both on the incidents that I'm going to discuss, and more particularly on the defenses.

I think that this talk really works for everybody. I think this is something everybody should be learning early on in their careers. The truth is that we all need to keep up with how the industry is moving and that's what I've tried to do with this talk. I've tried to distill some of the knowledge about catastrophic failure that the industry as a whole has seen in the last five years. If you are okay with your system to be going away for a week, if that wouldn't damage your business, this talk is not for you. If you would be upset by that then come to this talk.

Walk away with an awareness of what are some of the risks that our systems face that we don't think about day to day and that don't show up when we think about SLOs and monthly error budgets.