J-Spring 2022: Kate Stanley – What you can learn from Connect’s use of Kafka

https://jspring.nl Kafka Connect lets you build pipelines that flow data into and out of Kafka. Its appeal lies in being able to do this declaratively without using any code and reusing a large number of plugins developed by the Kafka community. But how does Connect works under the covers? And what makes it highly scalable and at the same time very resilient to failures. In this session we will delve into how Connect works and I’ll show you the design patterns and decisions that make this possible. We will consider the way its stores and manages its internal state and explain how it’s able to automatically detect failures and dynamically balance workloads onto distributed workers via a rebalancing protocol. You will walk away with a better understanding of how both Connect and Kafka work, get some ideas for designing your next data applications and see how to use Kafka in ways you didn’t expect.