Kafka Consumer Groups

About Consumer Groups

Kafka messages are produced by producers and consumed by consumers.

In many instances, it makes sense to group our consumers into logical groupings depending on how we wish to divide the work.

For instance, we might have a group of consumers which are together responsible for consuming New Order messages. We may wish that one and only one member of the group consumers the New Order messages.

This also allows us to more finely control load through the system. We could for instance have a consumer group dedicated to consuming New Orders, and a consumer group dedicated to consuming Price Updates, meaning that all of the messages are processed in a predictable way.

Consumer Groups And Partitions

Consumer groups have a tight relationship with the number of partitions from a correctness and performance perspective.

Imagine we have a topic with 10 partitions:

• If we have 10 consumers in a group we are balanced, with each consumer servicing a different partition.
• If we have more than 10 consumers in a group, some will sit idle.
• If we have less than 10 consumers in a group, some consumers will process from more than one partition.

We don't necessarily need to be "balanced". This depends on the nature of the data and the requirements for failover and performance.

kafka-consumer-groups.sh

The Kafka Consumer groups script allows us to view information about the consumer groups that are currently interacting with the broker instance.

./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092

Outputs:

Let's use the kafka-console-consumer.sh script to subscribe but using a consumer group.

./bin/kafka-console-consumer.sh --group-name pizzq_prorcessor

Let's use the kafka-console-consumer.sh script to subscribe but using a consumer group.

./bin/kafka-console-consumer.sh --group-name pizzq_prorcessor