Presentation: Streaming Microservices: Contracts & Compatibility

Gwen: I’ve been concerned about the way companies manage their metadata since… 2012 probably. When I first moved from managing relational databases to managing Hadoop clusters. DBAs take metadata, especially schemas for granted. And then suddenly Hadoop was this wild west, people just dump data and no one knows how to use it. You create all those crazy dependencies between teams because whoever writes the data makes decisions that affect everyone and can break downstream apps at any time. This is even more difficult with stream processing because of the real-time and microservices nature of the applications.

I spent the last 5 years working with customers on solving this problem with different tools and environments. I feel like I have quite a lot to share.

Gwen: The relevant role is usually “enterprise architect”, because they have overall responsibility for how different applications communicate and play together. Although I hope that many responsible engineers care as well. My target audience is usually from medium to huge companies - you need to be of a certain size before questions of compatibility become important.

Gwen: I'm going to spend part of my time just telling horror stories of what happens if you don't manage your schemas (I have four years worth of horror stories to share). Then I'm going to talk about how it's a general problem. It's not about if you use Avro or if you use JSON or something thing else. It doesn't even matter if you do stream processing at all, it's a very generic problem on how components, services and teams communicate.

Then I'm going to go into some solutions, including the Confluent Schema Registry. It's open to note though; there are lots of other solutions that you can use too.

I want to end the talk with few examples of the potential in implementing this kind of centralized streams and schema of management. In addition to the immediate compatibility benefits - a centralized metadata store can be used for data discovery and for governance. I hope to share some examples of what forward-looking enterprise architects in some organizations are currently exploring.

If you use events to communicate between applications (and this includes all stream processing apps) - you absolutely need to figure out a way to detect and prevent schema compatibility issues early in the development process. You also need reasonable ways to allow schemas to change without breaking things. My talk is full of suggestions on how to do both.

Gwen: The transition from both request-response processing and batch processing to stream processing.

Every business has many applications that are either request-response or batch due to historical reasons - but the real business process they model is a continuous stream of events. Using new technologies to model the business process more accurately in the applications will help make the entire process more efficient and more timely.

I am typically wary of cutting-edge technologies and prefer to use proven systems (like Kafka!), but one of the technologies I am currently most curious about is Lift’s Envoy. I hope to learn more about it at QCon NYC.