4.2 KiB
learning kafka
01-docker-setup
Take a look at 01-docker-setup. Ideally when you setup, you need to have a meta.properties file in the kafka-data ( /var/lib/kafka/data ) directory. This file is created when Kafka starts for the first time.
#
#Sat Jan 24 19:02:17 GMT 2026
cluster.id=5L6g3nShT-eMCtK--X86sw
directory.id=Xsizaw8IS7uzzJrI-zTLMQ
node.id=1
version=1
If this file is missing, the docker image will make one. But in case you run multiple brokers, all need to have the same cluster.id. So they can talk to each other. This is known as generate/format Kraft storage.
Neede to run: sudo chown -R 1000:1000 kafka-data/ to give the kafka user ( uid 1000 ) permission to write to the volume mounted directory. First 1000 is user id, second 1000 is group id.
02-python-conn
simple example, 1 producer, 1 consumer, no partitions or replication
03-rebalance
5 partitions
kafka-rebalance is the container name
topic is the rebal-topic
sudo docker exec kafka-rebalance /opt/kafka/bin/kafka-topics.sh --create --topic rebal-topic --partitions 5 --bootstrap-server localhost:9092
Run one producer and one consumer. The consumer will be assigned all 5 partitions. The messages will be out of order for the partitions.
Then in another terminal, run another consumer in the same group, then another similarly. The paritions will be rebalanced between the consumers automatically. If I kill one consumer, the partitions will be rebalanced again to the remaining consumers.
To see what the group assignments are, run:
sudo docker exec -it kafka-rebalance /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group rebal-group
This will show what partitions are assigned to what consumers in the group.
If you run another producer, messages will be twice as fast and all "just works".
If you create more consumers than paritions, the extra consumers will be idle.
There is no scaling down of partitions, to decrease partitions, you need to create a new topic with less partitions and migrate data over.
04-keys
If I want to have all messages for a certain user to always go in the same partition, I can use keys. Example: if messages for the user MUST be processed in order.
Change the container name and the topic
sudo chown -R 1000:1000 kafka-data
sudo docker exec kafka-keys /opt/kafka/bin/kafka-topics.sh --create --topic keys-topic --partitions 2 --bootstrap-server localhost:9092
If you forget to run the above command to create the topic, the topic will be created automatically when the producer sends the first message. But it will have only 1 partition by default.
DELETE topic:
docker exec -it kafka-keys /opt/kafka/bin/kafka-topics.sh --delete --topic keys-topic --bootstrap-server localhost:9092
05-retention
5000 ms = 5 seconds of retention
sudo chown -R 1000:1000 kafka-data
sudo docker exec kafka-ret /opt/kafka/bin/kafka-topics.sh --create --topic ret-topic --partitions 1 --bootstrap-server localhost:9092 --config retention.ms=5000 --config segment.ms=2000
Kafka moves on from one log file to another based on size or time. These are called "segments". Each segment is a file on disk. Nothing is EVER deleted from the active segment. Only when a segment is "closed" ( rolled over to a new segment ) can it be deleted based on retention policy.
Just retention above won't work, I need to roll it over too.
06-replication
I spend a very long time on this one trying to format the kraft storage correctly. Turns out I don't
need to. Modern kafka images auto format it. If you did not get a perm error, you may have missed the
KAFKA_LOG_DIRS env var in the docker-compose.yml. I spent too long on this as did not this in my
latest compose. Adding it and giving perms as usual worked perfectly.
Too much trouble on this one but finally done.
If I run the producer and do sudo docker stop kafka-2 and sudo docker stop kafka-3, I get an error on produce as not enough replicas are available. This is because I set min.insync.replicas=2 in the docker config for default new topics. I also need to set acks='all' in the producer too as the default there is acks=1 which means just leader's ack is fine so even if replicas are down, produce will succeed.
Both settings are needed.