169 lines
4.3 KiB
Markdown
169 lines
4.3 KiB
Markdown
# RabbitMQ Hands-On Lab: Build One End-to-End System
|
|
|
|
This lab gives you one working async system on your laptop.
|
|
|
|
## What You Will Build
|
|
|
|
A mini **order processing pipeline**:
|
|
|
|
1. `producer` sends `order.created` messages.
|
|
2. `worker` consumes orders and processes them.
|
|
3. `worker` publishes final status (`order.processed` or `order.failed`).
|
|
4. `notifier` consumes status messages and prints final result.
|
|
5. Failed messages go to a DLQ after retries.
|
|
|
|
By the end, you will have a real event flow running through RabbitMQ.
|
|
|
|
## Definition of Done
|
|
|
|
- [ ] You can run all services.
|
|
- [ ] You can submit 5 orders and see processed/failure outcomes.
|
|
- [ ] You can kill the worker and observe redelivery.
|
|
- [ ] You can force failures and see DLQ behavior.
|
|
|
|
## Lab Stack
|
|
|
|
- Language: **Python**
|
|
- Tooling: `uv`
|
|
- Broker: RabbitMQ **4+** (Docker)
|
|
- Library: `pika`
|
|
|
|
## Step 1: Project Setup (10 min)
|
|
|
|
- [x] Create files:
|
|
- [x] `docker-compose.yml`
|
|
- [x] `pyproject.toml` (created by `uv init`)
|
|
- [x] `producer.py`
|
|
- [x] `worker.py`
|
|
- [x] `notifier.py`
|
|
|
|
- [x] Initialize project and dependency with `uv`:
|
|
|
|
```bash
|
|
uv init --name rabbitmq-lab
|
|
uv add pika==1.3.2
|
|
```
|
|
|
|
- [x] Put this in `docker-compose.yml`:
|
|
|
|
```yaml
|
|
version: "3.9"
|
|
services:
|
|
rabbitmq:
|
|
image: rabbitmq:4-management
|
|
ports:
|
|
- "5672:5672"
|
|
- "15672:15672"
|
|
```
|
|
|
|
- [x] Start RabbitMQ:
|
|
|
|
```bash
|
|
docker compose up -d
|
|
```
|
|
|
|
- [x] Open UI: `http://localhost:15672` (`guest` / `guest`).
|
|
|
|
## Step 2: RabbitMQ Topology (15 min)
|
|
|
|
Create these resources from code (preferred) or UI:
|
|
|
|
- [x] Exchange `orders.x` (direct)
|
|
- [x] Exchange `orders.dlx` (direct)
|
|
- [x] Queue `orders.q`
|
|
- [x] Queue `orders.processed.q`
|
|
- [x] Queue `orders.failed.q`
|
|
- [x] Queue `orders.dlq`
|
|
|
|
Bindings:
|
|
|
|
- [x] `orders.q` <- `orders.x` with `order.created`
|
|
- [x] `orders.processed.q` <- `orders.x` with `order.processed`
|
|
- [x] `orders.failed.q` <- `orders.x` with `order.failed`
|
|
- [x] `orders.dlq` <- `orders.dlx` with `order.dead`
|
|
|
|
Queue args for `orders.q`:
|
|
|
|
- [x] `x-dead-letter-exchange=orders.dlx`
|
|
- [x] `x-dead-letter-routing-key=order.dead`
|
|
|
|
## Step 3: Producer (20 min)
|
|
|
|
- [x] `producer.py` should publish 5 JSON orders to routing key `order.created`.
|
|
- [x] Each message should include:
|
|
- [x] `order_id`
|
|
- [x] `user_id`
|
|
- [x] `amount`
|
|
- [x] `should_fail` (set true for at least 1 order)
|
|
- [x] `retry_count` (start at 0)
|
|
- [x] Mark messages persistent.
|
|
|
|
Success check:
|
|
|
|
- [x] Running `uv run producer.py` prints “Published order …” 5 times.
|
|
|
|
## Step 4: Worker (35 min)
|
|
|
|
- [x] `worker.py` consumes from `orders.q` with manual ACK.
|
|
- [x] For each message:
|
|
- [x] Parse JSON
|
|
- [x] Simulate processing (`sleep(1)`)
|
|
- [x] If `should_fail` and `retry_count < 2`, republish same order with `retry_count + 1` to `order.created`, ACK original.
|
|
- [x] If `should_fail` and `retry_count >= 2`, publish to `order.failed`, ACK original.
|
|
- [x] Else publish to `order.processed`, ACK original.
|
|
|
|
Important:
|
|
|
|
- [x] Set `prefetch_count=1`.
|
|
- [x] Use persistent publish for all outgoing messages.
|
|
|
|
Success check:
|
|
|
|
- [x] You see logs for retries and final outcomes.
|
|
|
|
## Step 5: Notifier (15 min)
|
|
|
|
- [x] `notifier.py` consumes both `orders.processed.q` and `orders.failed.q`.
|
|
- [x] Print concise final messages, e.g.:
|
|
- [x] `ORDER 1001 PROCESSED`
|
|
- [x] `ORDER 1003 FAILED AFTER RETRIES`
|
|
- [x] ACK all notifier messages.
|
|
|
|
Success check:
|
|
|
|
- [x] After running producer, notifier shows both success and failure events.
|
|
|
|
## Step 6: End-to-End Run (10 min)
|
|
|
|
Run in 3 terminals:
|
|
|
|
- [x] Terminal A: `uv run worker.py`
|
|
- [x] Terminal B: `uv run notifier.py`
|
|
- [x] Terminal C: `uv run producer.py`
|
|
|
|
Expected behavior:
|
|
|
|
- [x] Normal orders finish in `orders.processed.q` path.
|
|
- [x] Failing orders retry 2 times then go `orders.failed.q`.
|
|
|
|
## Step 7: Failure Drill (10 min)
|
|
|
|
- [x] Start worker and producer.
|
|
- [x] Kill worker while processing a message.
|
|
- [x] Restart worker.
|
|
- [x] Verify unacked message is redelivered and processed.
|
|
|
|
## Step 8: DLQ Drill (10 min)
|
|
|
|
- [x] In worker, temporarily `basic_nack(requeue=False)` for one special order.
|
|
- [x] Verify message appears in `orders.dlq`.
|
|
- [x] Inspect message payload from RabbitMQ UI.
|
|
|
|
## What You Learn (Concrete)
|
|
|
|
- [ ] RabbitMQ core flow (producer -> exchange -> queue -> consumer)
|
|
- [ ] Routing keys and direct exchange
|
|
- [ ] Manual ACK and redelivery
|
|
- [ ] Retry strategy and failure handling
|
|
- [ ] DLQ basics used in real systems
|