Presentation: Conquering Microservices Complexity @Uber With Distributed Tracing

I mostly work on distributed tracing, in the larger scope of overall observability.

We've been doing distributed tracing at Uber for quite a while, for four years. So I think we have some fairly interesting tales to tell about how specifically we use it. And from what I see we have a slightly different take on what we use distributed tracing for. Many people focus on things like performance optimizations with the tracing tools whereas Uber’s biggest use case so far has been understanding the complexity of the microservices architecture and using tracing to do root cause analysis during outages, to figure out where the issue with a specific outage is so that people can dive in deeper. That's what I'm trying to present. We've built relatively novel visualization techniques for analysing problematic traces which I want to share.

It's primarily targeted to the developers working with distributed systems. There is also something for SREs and DevOps because if your organization has a tracing team they might do something similar, and provide these tools to a broader audience in the company. In our case it was first responders and SREs who watch the business metrics at Uber and dive into the high level outages. Those are the major beneficiaries of this tool that I'm going to describe.

The understanding that tracing data provides a goldmine of information about the behavior of your distributed system. We built tools that are based on statistical data mining, and those techniques are highly effective and very useful for any size of organization. I want people to keep in mind that data mining is a possible future for all of the tracing work that's going to happen in the next few years.

Correct. Tracing data is in a very unique position compared to many other observability signals that we get about the systems because it's the only tool that really gives us simultaneously a macro view of the system, where you can see how many services and components participate in one single request across the board, and, at the same time, it is capable of giving a very micro view. You pick one single service instance and you see what that instance is doing exactly for this particular request versus hundreds of other requests that were running concurrently in that same instance. That's the power of tracing. When you start collecting that kind of data, if you are not doing data mining on it, you are wasting its potential for improving your observability ecosystem.

I cannot speak of specific outages details, but there have been a lot of very high-level outages where several people in the past would spend 30 minutes trying to look at their various logs across different systems and metrics to figure out what exactly is going wrong. And then when this new tool became available they were able to do that within a couple of minutes, because it was very precise in pointing to the exact place where the issue was happening and where you should dig deeper. We've seen these examples. In one example, Cassandra was throwing a quorum error. If you are a first responder, you might see it as a business metric, “my trips are not getting fulfilled on the Uber marketplace”. And so, to go down from that level all the way to some storage component and say, “this is the issue,” it's very hard. Tracking this down from the top-level outage signal all the way to the storage layer, that was very powerful.