74 lines
2.7 KiB
Markdown
74 lines
2.7 KiB
Markdown
learning kafka
|
|
|
|
# 01-docker-setup
|
|
|
|
Take a look at 01-docker-setup. Ideally when you setup, you need to have a meta.properties file in the kafka-data ( /var/lib/kafka/data ) directory. This file is created when Kafka starts for the first time.
|
|
|
|
```
|
|
#
|
|
#Sat Jan 24 19:02:17 GMT 2026
|
|
cluster.id=5L6g3nShT-eMCtK--X86sw
|
|
directory.id=Xsizaw8IS7uzzJrI-zTLMQ
|
|
node.id=1
|
|
version=1
|
|
```
|
|
|
|
If this file is missing, the docker image will make one. But in case you run multiple brokers, all need to have the same cluster.id. So they can talk to each other. This is known as generate/format Kraft storage.
|
|
|
|
Neede to run: `sudo chown -R 1000:1000 kafka-data/` to give the kafka user ( uid 1000 ) permission to write to the volume mounted directory. First 1000 is user id, second 1000 is group id.
|
|
|
|
# 02-python-conn
|
|
|
|
simple example, 1 producer, 1 consumer, no partitions or replication
|
|
|
|
# 03-rebalance
|
|
|
|
5 partitions
|
|
|
|
`kafka-rebalance` is the container name
|
|
topic is the `rebal-topic`
|
|
|
|
```
|
|
sudo docker exec kafka-rebalance /opt/kafka/bin/kafka-topics.sh --create --topic rebal-topic --partitions 5 --bootstrap-server localhost:9092
|
|
```
|
|
|
|
Run one producer and one consumer. The consumer will be assigned all 5 partitions.
|
|
The messages will be out of order for the partitions.
|
|
|
|
Then in another terminal, run another consumer in the same group, then another similarly. The paritions will be rebalanced between the consumers automatically. If I kill one consumer, the partitions will be rebalanced again to the remaining consumers.
|
|
|
|
To see what the group assignments are, run:
|
|
|
|
```
|
|
sudo docker exec -it kafka-rebalance /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group rebal-group
|
|
```
|
|
|
|
This will show what partitions are assigned to what consumers in the group.
|
|
|
|
If you run another producer, messages will be twice as fast and all "just works".
|
|
|
|
If you create more consumers than paritions, the extra consumers will be idle.
|
|
|
|
There is no scaling down of partitions, to decrease partitions, you need to create a new topic with less partitions and migrate data over.
|
|
|
|
# 04-keys
|
|
|
|
If I want to have all messages for a certain user to always go in the same partition, I can use keys.
|
|
Example: if messages for the user MUST be processed in order.
|
|
|
|
Change the container name and the topic
|
|
|
|
```
|
|
sudo chown -R 1000:1000 kafka-data
|
|
|
|
sudo docker exec kafka-keys /opt/kafka/bin/kafka-topics.sh --create --topic keys-topic --partitions 2 --bootstrap-server localhost:9092
|
|
```
|
|
|
|
If you forget to run the above command to create the topic, the topic will be created automatically when the producer sends the first message. But it will have only 1 partition by default.
|
|
|
|
DELETE topic:
|
|
|
|
```
|
|
docker exec -it kafka-keys /opt/kafka/bin/kafka-topics.sh --delete --topic keys-topic --bootstrap-server localhost:9092
|
|
```
|