learning kafka

# 01-docker-setup

Take a look at 01-docker-setup. Ideally when you setup, you need to have a meta.properties file in the kafka-data ( /var/lib/kafka/data ) directory. This file is created when Kafka starts for the first time.

```
#
#Sat Jan 24 19:02:17 GMT 2026
cluster.id=5L6g3nShT-eMCtK--X86sw
directory.id=Xsizaw8IS7uzzJrI-zTLMQ
node.id=1
version=1
```

If this file is missing, the docker image will make one. But in case you run multiple brokers, all need to have the same cluster.id. So they can talk to each other. This is known as generate/format Kraft storage.

Neede to run: `sudo chown -R 1000:1000 kafka-data/` to give the kafka user ( uid 1000 ) permission to write to the volume mounted directory. First 1000 is user id, second 1000 is group id.

# 02-python-conn

simple example, 1 producer, 1 consumer, no partitions or replication

# 03-rebalance

5 partitions

`kafka-rebalance` is the container name
topic is the `rebal-topic`

```
sudo docker exec kafka-rebalance /opt/kafka/bin/kafka-topics.sh --create --topic rebal-topic --partitions 5 --bootstrap-server localhost:9092
```

Run one producer and one consumer. The consumer will be assigned all 5 partitions.
The messages will be out of order for the partitions.

Then in another terminal, run another consumer in the same group, then another similarly. The paritions will be rebalanced between the consumers automatically. If I kill one consumer, the partitions will be rebalanced again to the remaining consumers.

To see what the group assignments are, run:

```
sudo docker exec -it kafka-rebalance /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group rebal-group
```

This will show what partitions are assigned to what consumers in the group.

If you run another producer, messages will be twice as fast and all "just works".

If you create more consumers than paritions, the extra consumers will be idle.

There is no scaling down of partitions, to decrease partitions, you need to create a new topic with less partitions and migrate data over.

# 04-keys

If I want to have all messages for a certain user to always go in the same partition, I can use keys.
Example: if messages for the user MUST be processed in order.

Change the container name and the topic

```
sudo chown -R 1000:1000 kafka-data

sudo docker exec kafka-keys /opt/kafka/bin/kafka-topics.sh --create --topic keys-topic --partitions 2 --bootstrap-server localhost:9092
```

If you forget to run the above command to create the topic, the topic will be created automatically when the producer sends the first message. But it will have only 1 partition by default.

DELETE topic:

```
docker exec -it kafka-keys /opt/kafka/bin/kafka-topics.sh --delete --topic keys-topic --bootstrap-server localhost:9092
```

# 05-retention

5000 ms = 5 seconds of retention

```
sudo chown -R 1000:1000 kafka-data

sudo docker exec kafka-ret /opt/kafka/bin/kafka-topics.sh --create --topic ret-topic --partitions 1 --bootstrap-server localhost:9092 --config retention.ms=5000 --config segment.ms=2000
```

Kafka moves on from one log file to another based on size or time. These are called "segments". Each segment is a file on disk. Nothing is EVER deleted from the active segment. Only when a segment is "closed" ( rolled over to a new segment ) can it be deleted based on retention policy.

Just retention above won't work, I need to roll it over too.

# 06-replication

I spend a very long time on this one trying to format the kraft storage correctly. Turns out I don't
need to. Modern kafka images auto format it. If you did not get a perm error, you may have missed the
`KAFKA_LOG_DIRS` env var in the docker-compose.yml. I spent too long on this as did not this in my
latest compose. Adding it and giving perms as usual worked perfectly. Note that after this, the meta.properties files
have the smae cluster.id across all brokers.

Too much trouble on this one but finally done.

If I run the producer and do `sudo docker stop kafka-2` and `sudo docker stop kafka-3`, I get an error on produce as not enough replicas are available. This is because I set `min.insync.replicas=2` in the docker config for default new topics. I also need to set `acks='all'` in the producer too as the default there is `acks=1` which means just leader's ack is fine so even if replicas are down, produce will succeed.
Both settings are needed.