Presentation: Adopting Stream Processing for Instrumentation

Cribbs: With my background working on distributed databases I realized writing to disk first and then trying to process it afterwards is really expensive. It's fine for generating business intelligence afterwards, or if you want to aggregate ahead of time before you write to disk it's fine. But if you're trying to look at an individual, like a single customer who has had a bad experience, aggregates don't help you. You just want to look at the very focused events that are related to that person's bad experience. I think that stream processing is a really good fit for that sort of use case, finding the really tiny needles in a haystack.

That said people don't necessarily know how to write incremental computations over instrumentation data. How do we make it easier for people to get into this, not just to get started but to keep developing their understanding of their own data as it goes through the stream processing system?

So you have all this data flowing in: what you do with it, do you even know what's there? In some cases you do because you wrote their instrumentation for the application and it's emitting data like a request span with the latency measurement, but I think there is still a barrier to entry. Instead of thinking of it procedurally, you have to be thinking of it as a stream, and do things with windows and across different dimensions. It's an unfamiliar problem to a lot of people.

So we wanted to create a really guided interface with a lot of interactivity and contextual help, and the ability to write down what you learned when you wrote that little program, and to provide context for the next person who picks it up.

Cribbs: So it's a web user interface. You write a document with markup in the middle of it: you have code blocks which represent individual stream pipelines. At the end of the pipeline something says display graph or show a table of the last 10 events that match this criterion. As you edit it you can run it and it will show you the matching live data stream.

Cribbs: We have built up a lot of infrastructure in Elixir - it's the stuff that we're using under the hood, but we've built a pretty interesting DSL on top because we know that we can focus that on operational instrumentation data. At every stage of the stream we have a schema, so we can optimize around that.

Two of our engineers are really familiar with the Elixir ecosystem, myself one of them. But there's also some dynamism to it that makes it compelling: one of the great things that we can do is take that stream processing program in and then manipulate the AST and rewrite it and then put it into action.

Cribbs: We're using some of the tools that come with Elixir, including a web framework called Phoenix, but a lot of it is React. So much of our stack is so different from rest the company, being based on Elixir. If we want people to come in and work on our project, Elixir is enough of a barrier, so we decided to pass on things like Elm or ClojureScript - let's just have plain JavaScript.

Our department does rotations regularly, every month basically. Senior level folks are not going to have a problem with new front end languages, but if we have not-so-senior people rotating in, it's easier to do JavaScript.

Cribbs: This is an intermediate level talk. There will be some deep concepts and I would say a lot of it will be about how designed the user interface. The message is stream processing systems are awesome but you need to think about the end-user of those systems and make them accessible.