How To Deploy Multiple Kafka Brokers With Docker Compose?

1. Introduction

Hello, in this step-by-step guide, I will show you how to set up multiple Kafka brokers with Docker Compose 🙂

In this article, together we will:

• see the minimum Docker Compose config required to spin up 3 Kafka brokers,
• take a closer look at the configuration,
• get information about active brokers with the zookeeper-shell command,
• verify whether everything is working as expected with Console Producer and Consumer tools.

Unlike the previous article about Kafka, I would like to start this one by showing the configuration you’ll need to set up 3 Kafka brokers. After that, we will go through it as thoroughly as possible to understand it even better and in the end, we will learn how to verify and test our config.

In my articles, I focus on the practice and getting things done.

However, if you would like to get a strong understading of DevOps concepts, which are more and more in demand nowadays, then I highly recommend to check out KodeKloud courses, like this Docker and Kubernetes learning paths.

2. Multiple Kafka Brokers Docker Compose

With that being said, let’s see the docker-compose.yaml:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.1
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
  kafka1:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka1
    ports:
      - "8097:8097"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8097,INTERNAL://kafka1:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  kafka2:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka2
    ports:
      - "8098:8098"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8098,INTERNAL://kafka2:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  kafka3:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka3
    ports:
      - "8099:8099"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8099,INTERNAL://kafka3:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

Make a real progress thanks to practical examples, exercises, and quizzes.

• 64 written lessons
• 62 quizzes with a total of 269 questions
• Dedicated Facebook Support Group
• 30-DAYS Refund Guarantee

Learn More

2.1. Docker Images Versions and Containers Names

Firstly, let’s take a look at docker images versions and containers names:

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.1
    container_name: zookeeper
...
  kafka1:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka1
...
  kafka2:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka2
...
  kafka3:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka3

As we can see, in our example we will use Confluent Community Docker Image for Apache Kafka and Confluent Docker Image for Zookeeper– both in version 7.2.1.

Additionally, we explicitly set names for our containers, which are discoverable at a hostname identical to the container name. Simply put, container exposed port 9092 of kafka1 will be visible for other containers at kafka1:9092.

2.2. Containers Ports

Nextly, let’s have a look at the exposed ports:

...
  kafka1:
    ports:
      - "8097:8097"
...
  kafka2:
    ports:
      - "8098:8098"
...
  kafka3:
    ports:
      - "8099:8099"

As can be seen, we did it for each service except the zookeeper.

You might already know this, but ports is responsible for exposing container ports to the host machine. Simply put, when running docker-compose on our computer with a 123:456 setting, we will be able to access a container’s 456 port through localhost:123.

2.3. Docker Compose Depends On

The last thing, I would like to cover before we head to the environment settings of our Kafka brokers is the depends_on .

As we can see, we put it for each Kafka service specifying the zookeeper service as an argument. With this one, we simply make sure that when running a docker compose up command, the zookeeper service will be started before the brokers.

3. Multiple Kafka Brokers Environment Variables

If you’ve seen my previous article on how to set up a single-node cluster with docker-compose, then you will notice that it looks a bit different. The main reason behind this is that with multiple Kafka brokers in our node, they have to know how to communicate with each other.

3.1. Zookeeper Client Port

As the first step, let’s take a look at the ZOOKEEPER_CLIENT_PORT.

To put it simply, this one instructs Zookeeper where to listen for connections by clients- Kafka brokers in our example.

3.2. Kafka Broker ID

Following, check the KAFKA_BROKER_ID variable.

By default, when we don’t set this one, the Zookeeper will automatically assign a unique broker identifier. Nevertheless, if we would like to specify it explicitly, then we can take care of it with this environment variable. Of course, please keep in mind that this value has to be unique across our Kafka brokers.

3.3. Kafka Zookeeper Connect

As the next thing, we set the KAFKA_ZOOKEEPER_CONNECT in each service to point to the Zookeeper.

As the value, we have to set our Zookeeper container name with a port set with the ZOOKEEPER_CLIENT_PORT directive.

3.4. Kafka Listener Settings

Finally, let’s see the rest of the environment variables responsible for communication:

• KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
• KAFKA_ADVERTISED_LISTENERS
• KAFKA_INTER_BROKER_LISTENER_NAME

Unlike the previous ones, I believe it has more sense to take a look at all of them at once to get a better understanding.

With a KAFKA_ADVERTISED_LISTENERS, we set a comma-separated list of listeners with their host/IP and port. This value must be set in a multi-node environment because otherwise, clients will attempt to connect to the internal host address. Please keep in mind that the listeners’ values (EXTERNAL and INTERNAL in our case), must be unique here.

When it comes to the naming here, we can specify our own values, just as I did with EXTERNAL/INTERNAL. Nevertheless, in such a case, we have to specify the correct mapping with KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:

• EXTERNAL will be mapped to PLAINTEXT protocol,
• INTERNAL will be mapped the same way, as well

If you are wondering why have we done it, then the reason is mentioned above- listeners’ values have to be unique, so we can’t specify PLAINTEXT://localhost:8099,PLAINTEXT://kafka3:9092 .

Finally, with KAFKA_INTER_BROKER_LISTENER_NAME we explicitly specify which listener to use for inter-broker communication.

4. Multiple Kafka With Docker Compose Testing

With all of that being said, we can finally make use of our docker-compose.yaml file.

To do so, let’s build and run our services with the following command:

docker compose up

Depending on our machine, it may take a while until everything is up and running.

In the end, we should see the following:

...Recorded new controller, from now on will use broker...

4.1. Verify All Containers Running

As the next step, let’s open up a new terminal window and list up and running containers with docker ps:

# Output:
CONTAINER ID IMAGE                           COMMAND                CREATED        STATUS        PORTS                            NAMES
cf892aeb5793 confluentinc/cp-kafka:7.2.1     "/etc/confluent/dock…" 54 minutes ago Up 54 minutes 0.0.0.0:8098->8098/tcp, 9092/tcp kafka2
a10bef7b6a54 confluentinc/cp-kafka:7.2.1     "/etc/confluent/dock…" 54 minutes ago Up 54 minutes 0.0.0.0:8099->8099/tcp, 9092/tcp kafka3
46c9ca4862b6 confluentinc/cp-kafka:7.2.1     "/etc/confluent/dock…" 54 minutes ago Up 54 minutes 0.0.0.0:8097->8097/tcp, 9092/tcp kafka1
eef5ebec8519 confluentinc/cp-zookeeper:7.2.1 "/etc/confluent/dock…" 54 minutes ago Up 54 minutes 2181/tcp, 2888/tcp, 3888/tcp     zookeeper

As we can clearly see, all Kafka containers have their ports exposed to our host and Zookeeper is reachable through 2181, as well.

4.2. Verify Active Kafka Brokers Using zookeeper-shell

Following, let’s run a zookeeper-shell on the zookeeper container:

docker exec -it zookeeper /bin/zookeeper-shell localhost:2181

# Output:
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is disabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

What’s important to mention here is that we run a zookeeper-shell command from within the container. That’s the reason why the zookeeper_host value is set to localhost and accessible, even though we haven’t exposed it.

Thanks to the -it flags, we can use the shell from our terminal.

4.3. Get IDs of Currently Active Brokers

So, to get the identifiers of currently active Kafka Brokers, we can use the following:

ls /brokers/ids

# Output:
[1, 2, 3]

The above output clearly indicates that everything is working, as expected.

Note: to close the zookeeper-shell, please hit Ctrl + C (and please do it before heading to the Producer/Consumer parts)

5. Publish Messages With Console Producer

After we know all of the above, we can finally send messages using the Console Producer.

As the first step, let’s run bash from a randomly picked Kafka broker container:

docker exec -it kafka2 /bin/bash

Nextly, let’s create a new topic called randomTopic:

kafka-topics --bootstrap-server localhost:9092 --create --topic randomTopic

# Output:
Created topic randomTopic.

Please note, that given the kafka-topics script is run inside of the container, we point to the 9092 port of localhost.

Following, we can produce some messages to our new topic:

kafka-console-producer --bootstrap-server localhost:9092 --topic randomTopic 

> lorem
> ipsum

# When we finish, we have to hit ctrl + d

Of course, we could do that from any other broker container (and in order to exit bash, we need to type exit in the command prompt).

6. Read Messages With Console Consumer

As the last step, let’s connect to another Apache Kafka broker and read previously sent messages:

docker exec -it kafka3 /bin/bash

kafka-console-consumer --bootstrap-server localhost:9092 --topic randomTopic --from-beginning

# Output:
lorem
ipsum

As we can clearly see, the output contains all messages, we’ve sent using Console Producer.

7. Summary

And that would be all for this article on how to set up multiple Apache Kafka brokers using Docker Compose and produce/consume messages. As a follow-up, I would recommend checking out this Confluent article about connecting to Kafka with different configurations.

Finally, let me know your thoughts in the comment section bellow, dear friend! 😀