J-Spring 2024: The Kafka Consumer: An Unexpected Journey of Data Consumption – Danica Fine

Once your data is stored on your Apache Kafka® cluster, the next step is to consume that data and do something interesting with it. Enter: Kafka Consumers. We all know how to set up a Kafka Consumer to poll data… but do you know how a consumer fetches the data from the cluster? Let’s find out! Every call to consumer.poll() is translated into a low-level request which is sent along to the brokers for fulfillment. In this session, we’ll join Kafka Consumers as they embark on their epic adventure to consume your data. First, see how these clients band together in a single fellowship and follow the guidance of their consumer group coordinator. Then, follow a request from an initial call to poll(), all the way to disk, and back to the client with your data via the broker’s final response. Along the way, we’ll explore a number of client and broker configurations that affect how these requests are handled and discuss the metrics that you can monitor to keep track of every stage of the consumer life cycle. By the end of this session, you’ll know the ins and outs of your Kafka Consumer requests, making your next debugging or performance analysis session a breeze.