Presentation: Index-Free Log Aggregation With Kafka
Abstract
We are seeing a rise in a new generation of logging solutions. Traditional approaches have been based on database indexes, but the next wave of log management platforms is a combination of techniques including streaming queries, in-memory processing, brute-force search, and filtering. We call this *Index-Free Logging*.
In this talk, we will provide an overview of the advantages of Index-Free Logging, and how we’re implementing it in Humio, a new log analytics system with a novel storage and query engine for events logs and traces. Humio was built from the ground up as a multi-tenant on-prem system so you can install it behind your own firewalls or in your own cloud infrastructure.
In a Humio setup, we then use Kafka for two things: buffering ingest and as a sequencer of events among the nodes of a Humio cluster. We love Kafka — it has made implementing a distributed system significantly easier than if we had to do the distributed queue from scratch. Using Kafka as a building block has allowed us to focus our attention on building a fast and easy search experience.
