feat: replication example

2026-01-25 04:56:42 +00:00
parent 15c93c71fb
commit 0beb588a70
5 changed files with 178 additions and 0 deletions
@@ -85,3 +85,15 @@ sudo docker exec kafka-ret /opt/kafka/bin/kafka-topics.sh --create --topic ret-t
 Kafka moves on from one log file to another based on size or time. These are called "segments". Each segment is a file on disk. Nothing is EVER deleted from the active segment. Only when a segment is "closed" ( rolled over to a new segment ) can it be deleted based on retention policy.

 Just retention above won't work, I need to roll it over too.
+
+# 06-replication
+
+I spend a very long time on this one trying to format the kraft storage correctly. Turns out I don't
+need to. Modern kafka images auto format it. If you did not get a perm error, you may have missed the
+`KAFKA_LOG_DIRS` env var in the docker-compose.yml. I spent too long on this as did not this in my
+latest compose. Adding it and giving perms as usual worked perfectly.
+
+Too much trouble on this one but finally done.
+
+If I run the producer and do `sudo docker stop kafka-2` and `sudo docker stop kafka-3`, I get an error on produce as not enough replicas are available. This is because I set `min.insync.replicas=2` in the docker config for default new topics. I also need to set `acks='all'` in the producer too as the default there is `acks=1` which means just leader's ack is fine so even if replicas are down, produce will succeed.
+Both settings are needed.