Kafka is a vital part of data infrastructure in many organizations. When the Kafka cluster grows and more data is stored in Kafka for a longer duration, several issues related to scalability, efficiency, and operations become important to address. Kafka cluster storage is typically scaled by adding more broker nodes to the cluster. But this also adds needless memory and CPUs to the cluster making overall storage cost less efficient compared to storing the older data in external storage. Tiered storage is introduced to extend Kafka’s storage beyond the local storage available on the Kafka cluster by retaining the older data in cheaper stores, such as HDFS, S3, Azure or GCS with minimal impact on the internals of Kafka.

We will talk about

• How tiered storage addresses the above problems and also brings several other advantages.
• High level architecture of tiered storage
• Future work planned as part of tiered storage.

Speakers:

Satish Duggana: Satish is Lead - Data Infra at Uber. Apache Storm Committer/PMC. Sriharsha Chintalapani: Data Architect at Uber, Apache Kafka Committer & PMC

Sriharsha Chintalapani is the tech lead for Uber’s Data Products and Streaming Infrastructure. He is a Apache Kafka & Storm Committer and PMC.