2.7 KiB
learning kafka
01-docker-setup
Take a look at 01-docker-setup. Ideally when you setup, you need to have a meta.properties file in the kafka-data ( /var/lib/kafka/data ) directory. This file is created when Kafka starts for the first time.
#
#Sat Jan 24 19:02:17 GMT 2026
cluster.id=5L6g3nShT-eMCtK--X86sw
directory.id=Xsizaw8IS7uzzJrI-zTLMQ
node.id=1
version=1
If this file is missing, the docker image will make one. But in case you run multiple brokers, all need to have the same cluster.id. So they can talk to each other. This is known as generate/format Kraft storage.
Neede to run: sudo chown -R 1000:1000 kafka-data/ to give the kafka user ( uid 1000 ) permission to write to the volume mounted directory. First 1000 is user id, second 1000 is group id.
02-python-conn
simple example, 1 producer, 1 consumer, no partitions or replication
03-rebalance
5 partitions
kafka-rebalance is the container name
topic is the rebal-topic
sudo docker exec kafka-rebalance /opt/kafka/bin/kafka-topics.sh --create --topic rebal-topic --partitions 5 --bootstrap-server localhost:9092
Run one producer and one consumer. The consumer will be assigned all 5 partitions. The messages will be out of order for the partitions.
Then in another terminal, run another consumer in the same group, then another similarly. The paritions will be rebalanced between the consumers automatically. If I kill one consumer, the partitions will be rebalanced again to the remaining consumers.
To see what the group assignments are, run:
sudo docker exec -it kafka-rebalance /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group rebal-group
This will show what partitions are assigned to what consumers in the group.
If you run another producer, messages will be twice as fast and all "just works".
If you create more consumers than paritions, the extra consumers will be idle.
There is no scaling down of partitions, to decrease partitions, you need to create a new topic with less partitions and migrate data over.
04-keys
If I want to have all messages for a certain user to always go in the same partition, I can use keys. Example: if messages for the user MUST be processed in order.
Change the container name and the topic
sudo chown -R 1000:1000 kafka-data
sudo docker exec kafka-keys /opt/kafka/bin/kafka-topics.sh --create --topic keys-topic --partitions 2 --bootstrap-server localhost:9092
If you forget to run the above command to create the topic, the topic will be created automatically when the producer sends the first message. But it will have only 1 partition by default.
DELETE topic:
docker exec -it kafka-keys /opt/kafka/bin/kafka-topics.sh --delete --topic keys-topic --bootstrap-server localhost:9092