Lesson Overview

In this lesson we will learn about Kafka Consumer Groups.

About Consumer Groups

Kafka messages are produced by producers and consumed by consumers.

In many instances, it makes sense to group our consumers into logical groupings depending on how we wish to divide the work.

For instance, we might have a group of consumers which are together responsible for consuming New Order messages. We may wish that one and only one member of the group consumers the New Order messages.

This also allows us to more finely control load through the system. We could for instance have a consumer group dedicated to consuming New Orders, and a consumer group dedicated to consuming Price Updates, meaning that all of the messages are processed in a predictable way.

Consumer Groups And Partitions

Consumer groups have a tight relationship with the number of partitions from a correctness and performance perspective.

Imagine we have a topic with 10 partitions:

  • If we have 10 consumers in a group we are balanced, with each consumer servicing a different partition.
  • If we have more than 10 consumers in a group, some will sit idle.
  • If we have less than 10 consumers in a group, some consumers will process from more than one partition.

We don't necessarily need to be "balanced". This depends on the nature of the data and the requirements for failover and performance.

kafka-consumer-groups.sh

The Kafka Consumer groups script allows us to view information about the consumer groups that are currently interacting with the broker instance.

./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092

Outputs:

We can see here that there is one

Let's use the kafka-console-consumer.sh script to subscribe but using a consumer group.

./bin/kafka-console-consumer.sh --group-name pizzq_prorcessor

Let's use the kafka-console-consumer.sh script to subscribe but using a consumer group.

./bin/kafka-console-consumer.sh --group-name pizzq_prorcessor

Summary

In this lesson we took a deeper look into the consumer groups concept to see how they are used to improve reliability and throughput of your Kafka message processing.

In the next lesson we move on to look at the process for producing data and publishing it into Kafka topics.

This Lesson Requires A Free Membership

Sign Up

Already A Member? Log In

Prev LessonNext Lesson

© 2022 Timeflow Academy.