Presentation: Survival of the Fittest - Streaming Architectures

​“Perfect is the enemy of good” ​ ​ -​ ​Voltaire

On the journey through life, we learn and adapt via trial and error - software development is no different. We realize and accept that we won’t build the perfect solution the first time around, it takes many iterations. At Gilt.com, now part of HBC Digital, we started processing and streaming event data nearly 5 years ago. Our initial solution was dramatically different from our current solution - and will likely be different from our solution 5 years from now.

The Gilt.com banner, at HBC Digital, is in the business of flash sales, which makes for some interesting use cases in the world of streaming. We release new sales of top designer labels, at up to 70% off retail, on the web and our mobile app, every day at Noon and 9pm. Around the time of these releases, we experience volume spikes between 10X and 100X on our streams.

Numerous streaming frameworks, homemade, as well as, open source, did not pass the evolutionary tests. Frameworks come and go, ​so this talk is not about the “best” framework or platform to use, rather it’s about core principles that will stand the tests of streaming evolution. Also, this talk covers major potential pitfalls that you may stumble over on your path to streaming, as well as, how to avoid these. Finally, this talk will cover what the next evolutionary step in streaming at HBC Digital. ​