Presentation: Datadog: A Real Time Metrics Database for One Quadrillion Points/Day

Our primary challenge is scaling to meet our customer’s demand for fast queries on multi-dimensionally increasing data, but in a cost efficient manner. As our customers needs are becoming more complex, to get the insight into their systems they want to send us many more metric points with as many tags to query over. We are focused on finding ways to efficiently handle that scale, leveraging both traditional techniques like optimizing and horizontally scaling data indexes, and modern approaches like approximations (sketches) that allow us to scale sub-linearly with input load.

It's definitely not a vendor pitch. We're an engineering-driven culture and that leads us to want to share the problems we have and the ways we have solved them. We think the complexity of challenges we're tackling scaling around multiple dimensions of data growth mirror what many in the industry are seeing, and so we thought that this would be a good time to share how we're approaching that challenge.

The first challenge we face is how to build a metrics database that scales well horizontally for both point load and tag load. Our lessons there all come from embracing the distinct customer use cases that restrict us from needing to build a general time series database, and then just applying good scaleable systems design principles. One of our newer challenges though is providing accurate percentiles over streams of data. As our customers are becoming more interested in SLAs and SLOs they want to query, rather than the average or sum or traditional aggregations, instead things like the 99th percentile value of their latency request size. Naively the way you do that is storing every single point, but we're using “approximate” data structures to provide accurate and fast answers without having to scale linearly with the number of points and values that customers are sending us.

These are data structures that are in the wild. The community calls this approach “sketches”. We have open sourced our implementation as part of the Datadog agent which is Apache-licensed, so that the code is there and it's there for anyone to go and take a look at. We also have a paper that was accepted at the 2019 VLDB conference that our data science team will be talking about in August, and we'll be open-sourcing standalone versions in several languages at that time as well.

An approximation is giving you something close to the value you ask for. For example, it is impossible to compute the median of a sequence of numbers exactly in one pass without holding onto most of that data. Instead you will need to use something like a sketch, which is guaranteed to give you an approximation while holding on to much less data.

Some sketches are probabilistic, for example, they might need to choose a random hash function. Probabilistic sketches also have approximation guarantees, but they also have a failure probability where there’s a small chance that the approximation can be very bad.

We are talking to people who have similar needs to us in terms of scaling systems handling metrics and/or time series data. Our customers need answers about what's happening in their systems in as close to real time as possible. Building a scalable storage architecture and using approximations has been our method to minimize that latency and get answers to people as quickly as possible.