173 lines
4.4 KiB
Markdown
173 lines
4.4 KiB
Markdown
# RabbitMQ Hands-On Lab: Build One End-to-End System
|
|
|
|
This lab gives you one working async system on your laptop.
|
|
|
|
## What You Will Build
|
|
|
|
A mini **order processing pipeline**:
|
|
|
|
1. `producer` sends `order.created` messages.
|
|
2. `worker` consumes orders and processes them.
|
|
3. `worker` publishes final status (`order.processed` or `order.failed`).
|
|
4. `notifier` consumes status messages and prints final result.
|
|
5. Failed messages go to a DLQ after retries.
|
|
|
|
By the end, you will have a real event flow running through RabbitMQ.
|
|
|
|
## Definition of Done
|
|
|
|
- [ ] You can run all services.
|
|
- [ ] You can submit 5 orders and see processed/failure outcomes.
|
|
- [ ] You can kill the worker and observe redelivery.
|
|
- [ ] You can force failures and see DLQ behavior.
|
|
|
|
## Lab Stack
|
|
|
|
- Language: **Python**
|
|
- Broker: RabbitMQ (Docker)
|
|
- Library: `pika`
|
|
|
|
## Step 1: Project Setup (10 min)
|
|
|
|
- [ ] Create files:
|
|
- [ ] `docker-compose.yml`
|
|
- [ ] `requirements.txt`
|
|
- [ ] `producer.py`
|
|
- [ ] `worker.py`
|
|
- [ ] `notifier.py`
|
|
|
|
- [ ] Put this in `requirements.txt`:
|
|
|
|
```txt
|
|
pika==1.3.2
|
|
```
|
|
|
|
- [ ] Put this in `docker-compose.yml`:
|
|
|
|
```yaml
|
|
version: "3.9"
|
|
services:
|
|
rabbitmq:
|
|
image: rabbitmq:3-management
|
|
ports:
|
|
- "5672:5672"
|
|
- "15672:15672"
|
|
```
|
|
|
|
- [ ] Start RabbitMQ:
|
|
|
|
```bash
|
|
docker compose up -d
|
|
```
|
|
|
|
- [ ] Open UI: `http://localhost:15672` (`guest` / `guest`).
|
|
|
|
## Step 2: RabbitMQ Topology (15 min)
|
|
|
|
Create these resources from code (preferred) or UI:
|
|
|
|
- [ ] Exchange `orders.x` (direct)
|
|
- [ ] Exchange `orders.dlx` (direct)
|
|
- [ ] Queue `orders.q`
|
|
- [ ] Queue `orders.processed.q`
|
|
- [ ] Queue `orders.failed.q`
|
|
- [ ] Queue `orders.dlq`
|
|
|
|
Bindings:
|
|
|
|
- [ ] `orders.q` <- `orders.x` with `order.created`
|
|
- [ ] `orders.processed.q` <- `orders.x` with `order.processed`
|
|
- [ ] `orders.failed.q` <- `orders.x` with `order.failed`
|
|
- [ ] `orders.dlq` <- `orders.dlx` with `order.dead`
|
|
|
|
Queue args for `orders.q`:
|
|
|
|
- [ ] `x-dead-letter-exchange=orders.dlx`
|
|
- [ ] `x-dead-letter-routing-key=order.dead`
|
|
|
|
## Step 3: Producer (20 min)
|
|
|
|
- [ ] `producer.py` should publish 5 JSON orders to routing key `order.created`.
|
|
- [ ] Each message should include:
|
|
- [ ] `order_id`
|
|
- [ ] `user_id`
|
|
- [ ] `amount`
|
|
- [ ] `should_fail` (set true for at least 1 order)
|
|
- [ ] `retry_count` (start at 0)
|
|
- [ ] Mark messages persistent.
|
|
|
|
Success check:
|
|
|
|
- [ ] Running `python producer.py` prints “Published order …” 5 times.
|
|
|
|
## Step 4: Worker (35 min)
|
|
|
|
- [ ] `worker.py` consumes from `orders.q` with manual ACK.
|
|
- [ ] For each message:
|
|
- [ ] Parse JSON
|
|
- [ ] Simulate processing (`sleep(1)`)
|
|
- [ ] If `should_fail` and `retry_count < 2`, republish same order with `retry_count + 1` to `order.created`, ACK original.
|
|
- [ ] If `should_fail` and `retry_count >= 2`, publish to `order.failed`, ACK original.
|
|
- [ ] Else publish to `order.processed`, ACK original.
|
|
|
|
Important:
|
|
|
|
- [ ] Set `prefetch_count=1`.
|
|
- [ ] Use persistent publish for all outgoing messages.
|
|
|
|
Success check:
|
|
|
|
- [ ] You see logs for retries and final outcomes.
|
|
|
|
## Step 5: Notifier (15 min)
|
|
|
|
- [ ] `notifier.py` consumes both `orders.processed.q` and `orders.failed.q`.
|
|
- [ ] Print concise final messages, e.g.:
|
|
- [ ] `ORDER 1001 PROCESSED`
|
|
- [ ] `ORDER 1003 FAILED AFTER RETRIES`
|
|
- [ ] ACK all notifier messages.
|
|
|
|
Success check:
|
|
|
|
- [ ] After running producer, notifier shows both success and failure events.
|
|
|
|
## Step 6: End-to-End Run (10 min)
|
|
|
|
Run in 3 terminals:
|
|
|
|
- [ ] Terminal A: `python worker.py`
|
|
- [ ] Terminal B: `python notifier.py`
|
|
- [ ] Terminal C: `python producer.py`
|
|
|
|
Expected behavior:
|
|
|
|
- [ ] Normal orders finish in `orders.processed.q` path.
|
|
- [ ] Failing orders retry 2 times then go `orders.failed.q`.
|
|
|
|
## Step 7: Failure Drill (10 min)
|
|
|
|
- [ ] Start worker and producer.
|
|
- [ ] Kill worker while processing a message.
|
|
- [ ] Restart worker.
|
|
- [ ] Verify unacked message is redelivered and processed.
|
|
|
|
## Step 8: DLQ Drill (10 min)
|
|
|
|
- [ ] In worker, temporarily `basic_nack(requeue=False)` for one special order.
|
|
- [ ] Verify message appears in `orders.dlq`.
|
|
- [ ] Inspect message payload from RabbitMQ UI.
|
|
|
|
## What You Learn (Concrete)
|
|
|
|
- [ ] RabbitMQ core flow (producer -> exchange -> queue -> consumer)
|
|
- [ ] Routing keys and direct exchange
|
|
- [ ] Manual ACK and redelivery
|
|
- [ ] Retry strategy and failure handling
|
|
- [ ] DLQ basics used in real systems
|
|
|
|
## Optional Next Lab
|
|
|
|
- [ ] Add HTTP API (`FastAPI`) that accepts `/orders` and publishes to RabbitMQ.
|
|
- [ ] Add publisher confirms.
|
|
- [ ] Replace classic queue with quorum queue and compare behavior.
|