[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"site-settings":3,"post-rabbitmq-kafka-ru-men-jiao-cheng":20,"comments-POST-d93c5f0d-21de-4044-9055-b2bd64302821-1":63,"sidebar-data":68},{"siteName":4,"siteDescription":5,"siteUrl":6,"siteLogo":7,"authorName":8,"authorAvatar":9,"authorBio":10,"authorWechatQr":11,"mpWatermarkQr":12,"socialLinks":13,"donation":14,"auth":17,"ai":19},"小白龙博客","记录技术与生活","http:\u002F\u002Flocalhost:3000","\u002Ffavicon.png","Liam","https:\u002F\u002Fcdn.xiaolong0418.com\u002Fblog\u002F2026\u002F07\u002F54e493e2-0141-4cb3-aaad-54c66f126193.png","","https:\u002F\u002Fcdn.xiaolong0418.com\u002Fblog\u002F2026\u002F07\u002F96088464-d79f-4926-92a1-7622a09267bb.png","https:\u002F\u002Fcdn.xiaolong0418.com\u002Fblog\u002F2026\u002F07\u002F33f3eae5-3ae3-4941-8683-38d9e35ec72b.jpg",{"github":10,"twitter":10},{"enabled":15,"footerText":16},true,"感谢支持",{"githubEnabled":18,"giteeEnabled":15},false,{"enabled":15,"chatEnabled":15},{"id":21,"title":22,"slug":23,"content":24,"excerpt":25,"coverImage":26,"status":27,"isPinned":18,"pinnedAt":28,"viewCount":29,"publishedAt":30,"createdAt":31,"updatedAt":32,"category":33,"author":37,"tags":40},"d93c5f0d-21de-4044-9055-b2bd64302821","RabbitMQ 与 Kafka 入门教程：任务队列还是事件流？","rabbitmq-kafka-ru-men-jiao-cheng","# RabbitMQ 与 Kafka 入门教程：任务队列还是事件流？\n\n做后端时总会碰到这类需求：接口要尽快返回，但后面还有抓取、写日志、发通知等「慢活」。如果全堆在一次 HTTP 请求里，接口会变慢，高峰时还容易把服务打挂。\n\n消息中间件就是把「发出任务」和「真正干活」拆开：业务先把消息丢进中间件，消费者再慢慢处理。市面上最常被拿来对比的两套方案，就是 **RabbitMQ** 和 **Kafka**。\n\n本文用尽量少的概念，讲清二者分别擅长什么，以及本站（NestJS 博客）里为什么一个用 RabbitMQ、一个用 Kafka。\n\n---\n\n## 1. 先记住一个判断题\n\n| 你更像在做…… | 更合适 |\n| --- | --- |\n| 「帮我办一件事」：抓取、转码、发邮件、扣积分 | **RabbitMQ**（任务队列） |\n| 「记下一串发生过的事」：访问日志、埋点、审计流 | **Kafka**（事件流 \u002F 日志） |\n\n一句话：\n\n- **RabbitMQ**：消息像待办工单，处理完通常就没了（消费确认后删除）。\n- **Kafka**：消息像账本，写进去后还能按偏移量重放、多组消费者各自读。\n\n两者都能做异步，但心智模型和运维成本不一样。\n\n---\n\n## 2. 为什么要用中间件？\n\n没有中间件时，常见写法是：\n\n```text\n用户请求 → 接口里同步抓网页 → 写库 → 再返回\n```\n\n问题很直观：\n\n1. **慢**：抓取、第三方 API 一卡，用户就一直转圈。\n2. **脆**：下游挂了，接口也一起失败。\n3. **难扩**：流量突增时，只能把整台 API 机器一起扩，无法单独扩「干活」的进程。\n\n有了队列之后：\n\n```text\n用户请求 → 入队成功就返回 → 消费者异步处理 → 写库 \u002F 回调\n```\n\nAPI 只负责「登记任务」，真正耗时的工作交给消费者，峰值被缓冲在队列里。\n\n---\n\n## 3. RabbitMQ 入门\n\n### 3.1 核心概念（够用即可）\n\n```text\nProducer（生产者）\n    ↓\nExchange（交换机：按规则把消息分发出去）\n    ↓\nQueue（队列：真正存消息的地方）\n    ↓\nConsumer（消费者：取出并处理）\n```\n\n初学时最常见的模式是：**生产者直接往某个队列发消息**（内部往往是默认交换机 + 路由键等于队列名）。你只需要记住：\n\n- **Queue**：待办列表。\n- **Consumer**：工人。多个工人可以抢同一队列的任务（竞争消费）。\n- **ACK**：处理成功再确认；失败可不确认或拒绝，消息可重新入队。\n\n### 3.2 适合做什么？\n\n- 书签 \u002F 友链全量抓取\n- 视频去水印解析\n- 邮件、短信、Webhook\n- 需要「做完就完」的后台任务\n\n特点是：**任务有明确终点**，积压可以看队列长度，处理完消息通常不再保留。\n\n### 3.3 本站里的 RabbitMQ\n\n本站开启 `RABBITMQ_ENABLED=true` 后，用 AMQP 连接 Broker，主要队列包括：\n\n| 队列名 | 用途 |\n| --- | --- |\n| `blog.crawl.bookmark` | 书签全量抓取 |\n| `blog.crawl.friend-link` | 友链可达性 \u002F 元信息刷新 |\n| `blog.mp.video-watermark` | 小程序视频去水印任务 |\n\n管理后台仪表盘会展示这些队列的积压；健康检查也会探测 RabbitMQ 是否可达。\n\n未配置 `RABBITMQ_URL` 时，代码会**回退到进程内异步**，本地开发不装 RabbitMQ 也能跑通主流程。\n\n### 3.4 本地快速体验（Docker）\n\n```bash\ndocker run -d --name rabbitmq \\\n  -p 5672:5672 -p 15672:15672 \\\n  rabbitmq:3-management\n```\n\n浏览器打开 `http:\u002F\u002Flocalhost:15672`（默认账号密码一般是 `guest` \u002F `guest`），就能看到队列、连接和消息吞吐。生产环境请改掉默认账号，并限制管理端口的公网访问。\n\n---\n\n## 4. Kafka 入门\n\n### 4.1 核心概念\n\n```text\nProducer → Topic（主题，可分成多个 Partition）\n                ↓\n         Consumer Group（消费组）\n```\n\n关键点：\n\n- **Topic**：一类事件的名字，例如 `blog.api-request-logs`。\n- **Partition**：分区。同一分区内有序；多分片可并行消费。\n- **Offset**：消费进度，像书签一样记「读到哪一行」。\n- **Consumer Group**：同一组内的实例分摊分区；**不同组可以各自完整消费同一 Topic**（互不影响）。\n\nKafka 把消息更像**追加写的日志**：默认会保留一段时间（或按体积），而不是「消费完立刻删」。\n\n### 4.2 适合做什么？\n\n- 高吞吐写入：接口日志、埋点、审计\n- 需要回放 \u002F 多下游订阅同一数据流\n- 后续接数仓、实时统计、告警分析\n\n不太适合当成「简单任务队列」的首选：运维（ZooKeeper \u002F KRaft、磁盘、分区规划）比 RabbitMQ 重一截。\n\n### 4.3 本站里的 Kafka\n\n本站接口请求日志原先走 Redis List 批量刷库；接入 Kafka 后变成：\n\n```text\n请求结束 → produce 到 blog.api-request-logs\n         → consumer group blog-api-logs 批量 createMany 入库\n```\n\n相关配置大致是：\n\n```env\nKAFKA_ENABLED=true\nKAFKA_BROKERS=127.0.0.1:9092\nKAFKA_CLIENT_ID=blog-api\nKAFKA_API_LOG_TOPIC=blog.api-request-logs\nKAFKA_API_LOG_GROUP=blog-api-logs\n```\n\n未启用 Kafka 时，**自动回退 Redis + 定时 flush**，保证日志链路不断。\n\nPM2 多实例时，同一 Consumer Group 会分摊分区，正好匹配「多进程消费同一条日志流」的场景。\n\n---\n\n## 5. 一张表对比\n\n| 维度 | RabbitMQ | Kafka |\n| --- | --- | --- |\n| 模型 | 队列 \u002F 交换机 | 分区日志 |\n| 消息寿命 | 通常消费后删除 | 按策略保留，可重放 |\n| 吞吐 | 中等，够大多数业务任务 | 更高，偏流式写入 |\n| 多订阅者 | 要 fanout \u002F 多队列复制 | 天然：多 Consumer Group |\n| 运维复杂度 | 相对轻 | 相对重 |\n| 典型场景 | 异步任务、RPC 解耦 | 日志、埋点、事件溯源 |\n\n**不是二选一消灭对方**：本站就是并存——任务走 RabbitMQ，日志流走 Kafka。\n\n---\n\n## 6. 接入时容易踩的坑\n\n1. **把 Kafka 当小队列用**  \n   只有一两个消费者、消息量不大、也不需要重放时，RabbitMQ（或 Redis 队列）往往更简单。\n\n2. **忘记失败处理**  \n   RabbitMQ 要设计 ACK \u002F 重试 \u002F 死信；Kafka 要明确「处理失败是否提交 offset」，避免丢消息或死循环。\n\n3. **本地和线上行为不一致**  \n   本站做法是：中间件可选，未启用则回退。开发机不装 Broker 也能写业务，但联调队列行为时仍建议起 Docker。\n\n4. **只入队不消费**  \n   CLI 或临时脚本若开启「只发布」，记得别在生产主进程上误关消费者，否则积压会一直涨。\n\n5. **安全组与管理台**  \n   `5672` \u002F `9092` \u002F `15672` 不要无脑对公网全开；管理台建议仅内网或加鉴权。\n\n---\n\n## 7. 怎么在自己的项目里落地（最小步骤）\n\n### RabbitMQ 任务\n\n1. 安装 Broker，创建 vhost \u002F 用户。\n2. 业务侧：`publish(queue, payload)`。\n3. 消费者：`consume` → 处理 → `ack`。\n4. 监控队列长度；积压告警。\n\n### Kafka 日志流\n\n1. 起 Broker（单节点学习可用 KRaft 或现成发行版）。\n2. 创建 Topic（分区数按消费者规模预估）。\n3. 请求链路 `produce`；独立 consumer group 批量落库。\n4. 观察 consumer lag（积压 = 日志末端偏移 − 组已提交偏移）。\n\n本站管理后台的「消息队列」面板，就是在看 Rabbit 队列深度和 Kafka lag，方便一眼判断有没有堵。\n\n---\n\n## 8. 小结\n\n- **要异步办事** → 先想 RabbitMQ（或同类任务队列）。\n- **要高吞吐记账 \u002F 多下游读同一流** → 再上 Kafka。\n- 选型看问题形态，不看谁「更流行」。\n- 生产环境务必：鉴权、限端口、可回退、可观测（积压 + 健康检查）。\n\n如果你正在搭个人站点或中小型 NestJS 服务，完全可以像本站一样：**RabbitMQ 扛爬虫与工具任务，Kafka 扛接口日志**，两套职责清晰，出问题也好分别排查。\n\n---\n\n*文中队列名、Topic 名对应本站当前实现；若你 fork 代码，以仓库 `docs` 与环境变量为准。*\n","本文对比 RabbitMQ 与 Kafka 的核心差异：前者适合抓取、发邮件等明确终点的异步任务，后者适用于日志记录、埋点等高吞吐事件流。结合 NestJS 实战，详解两者在任务队列与数据流中的选型逻辑、配置方法及常见踩坑点，帮助开发者根据业务场景做出合适选择。","https:\u002F\u002Fcdn.xiaolong0418.com\u002Fblog\u002Fai-cover\u002F2026\u002F07\u002F64b92538-edb6-411e-8e47-e02d1ac2d468.png","PUBLISHED",null,2,"2026-07-16T13:17:45.375Z","2026-07-16T21:17:23.214Z","2026-07-16T13:40:00.010Z",{"id":34,"name":35,"slug":36},"e8d0bd45-d10c-46d3-8afb-0c072df7f8a7","技术","tech",{"id":38,"name":39,"avatar":28},"d9e86531-b988-4101-ba14-16843b265208","小白龙",[41,45,49,53,57,60],{"id":42,"name":43,"slug":44},"45c2f699-8d0b-421a-9d43-5bb0bd47e5ba","RabbitMQ","rabbitmq",{"id":46,"name":47,"slug":48},"561fc6b6-5950-4a08-8dba-ece34522dcd2","Kafka","kafka",{"id":50,"name":51,"slug":52},"ab034d3a-6e5b-4db5-a2dc-faf4ccbb63f5","Nest","nest",{"id":54,"name":55,"slug":56},"c95bbe84-bdd0-410a-86a9-e87958c55f4f","Redis","redis",{"id":58,"name":59,"slug":59},"f32faa96-f2ec-45c6-9a17-2c76062edcb0","开发工具",{"id":61,"name":62,"slug":62},"f6766d54-54fc-405e-932d-b7d550559125","服务器",{"list":64,"total":65,"page":66,"pageSize":67},[],0,1,20,{"postCount":69,"tagCount":70,"hotPosts":71,"recentComments":173,"recentDonations":213},104,50,[72,97,115,144,160],{"id":73,"title":74,"slug":75,"excerpt":76,"coverImage":77,"status":27,"isPinned":18,"pinnedAt":28,"viewCount":78,"publishedAt":79,"createdAt":79,"updatedAt":80,"category":81,"author":84,"tags":85},"7a3bd0a1-688b-4bc6-9bef-f6c53a8ff080","Pnpm安装与使用教程","pnpm安装与使用教程","本文详解Pnpm安装与核心用法。作为快速且节省磁盘空间的包管理工具，Pnpm通过硬链接实现高效存储并支持Monorepo。文中涵盖全局安装、依赖添加与移除、版本升级、脚本运行及镜像源配置等常用命令，帮助开发者掌握这一严格且高效的Node.js包管理方案。","https:\u002F\u002Fcdn.xiaolong0418.com\u002Fblog\u002Fai-cover\u002F2026\u002F07\u002Fb46c5891-f489-4991-8609-17fbb1f59c0d.png",10233,"2022-05-24T01:51:16.000Z","2026-07-14T18:40:00.013Z",{"id":82,"name":83,"slug":83},"d10456a5-e649-4741-a38f-f07f266ce5f2","开发环境",{"id":38,"name":39,"avatar":28},[86,89,93,96],{"id":87,"name":88,"slug":88},"0f1cc678-40e4-44b1-b2cf-a6fd8a1c867a","npm",{"id":90,"name":91,"slug":92},"14e9ab02-b0bb-4c85-8604-fe6f1f0f33cd","Pnpm","pnpm",{"id":94,"name":95,"slug":95},"a046060c-39ef-474a-8c85-2546aca0e2e5","代码片段",{"id":58,"name":59,"slug":59},{"id":98,"title":99,"slug":100,"excerpt":101,"coverImage":102,"status":27,"isPinned":18,"pinnedAt":28,"viewCount":103,"publishedAt":104,"createdAt":104,"updatedAt":105,"category":106,"author":108,"tags":109},"57ea9eee-45d8-4899-9bfb-519cd13690c6","Navicat Premium 16 下载与安装破解教程","navicat-premium-16-下载与安装破解教程","本文介绍在 Win11 环境下安装 Navicat Premium 16 的步骤。涵盖从官网下载程序、配置安装路径，到使用注册机关闭杀毒软件后执行补丁、生成许可证及手动激活的全过程，旨在帮助用户完成软件配置。","https:\u002F\u002Fcdn.xiaolong0418.com\u002Fblog\u002Fai-cover\u002F2026\u002F07\u002F78d5fe88-2bd5-45b7-90be-5fb394d51f8c.png",4460,"2022-09-11T11:45:15.000Z","2026-07-14T02:35:00.011Z",{"id":107,"name":59,"slug":59},"9ed9827c-9cbb-42da-80e4-d04c7fdba886",{"id":38,"name":39,"avatar":28},[110,113,114],{"id":111,"name":112,"slug":112},"d0aa41f4-68f8-48d4-a4ed-3a503ea90451","下载",{"id":58,"name":59,"slug":59},{"id":61,"name":62,"slug":62},{"id":116,"title":117,"slug":118,"excerpt":117,"coverImage":119,"status":27,"isPinned":18,"pinnedAt":28,"viewCount":120,"publishedAt":121,"createdAt":121,"updatedAt":122,"category":123,"author":126,"tags":127},"017a769c-8df4-4ef7-b485-16558fdab198","Vite快速入门教程","vite快速入门教程","https:\u002F\u002Fcdn.xiaolong0418.com\u002Fblog\u002Fai-cover\u002F2026\u002F07\u002F458813dc-889b-44f2-aae7-3ceb41b80516.png",2526,"2022-06-02T07:30:36.000Z","2026-07-16T03:40:00.008Z",{"id":124,"name":125,"slug":125},"4ea3d8af-9cc3-49bb-a9cd-34dbcdc3bd85","构建工具",{"id":38,"name":39,"avatar":28},[128,129,133,137,141],{"id":87,"name":88,"slug":88},{"id":130,"name":131,"slug":132},"20bff9cd-7848-4c16-8775-42cf12b44b30","Vue3生态","vue3生态",{"id":134,"name":135,"slug":136},"62b94c93-724f-488d-8fc5-0449971d9204","Vue3","vue3",{"id":138,"name":139,"slug":140},"66f3aeb0-84ef-45f6-a43a-944eefc9895a","Vite","vite",{"id":142,"name":143,"slug":143},"dbfc086a-73a6-4560-814d-593acb61cf98","性能优化",{"id":145,"title":146,"slug":147,"excerpt":148,"coverImage":149,"status":27,"isPinned":15,"pinnedAt":150,"viewCount":151,"publishedAt":152,"createdAt":152,"updatedAt":153,"category":154,"author":157,"tags":158},"103123bc-6e69-4464-96d3-8204b35b7a47","云服务器限时爆款《2核4G  200M 》198块钱1年","云服务器限时爆款-2核4g-200m-198块钱1年","香港云服务器限时爆款","https:\u002F\u002Fcdn.xiaolong0418.com\u002Fblog\u002F2026\u002F07\u002F5cbc189d-6476-462a-99a1-2cd05d897a03.png","2026-07-01T09:24:54.612Z",2420,"2022-05-24T01:59:14.000Z","2026-07-16T09:50:00.009Z",{"id":155,"name":156,"slug":156},"d8cbe380-54b3-4a61-a12d-5438c2918574","限时优惠",{"id":38,"name":39,"avatar":28},[159],{"id":61,"name":62,"slug":62},{"id":161,"title":162,"slug":163,"excerpt":164,"coverImage":165,"status":27,"isPinned":18,"pinnedAt":28,"viewCount":166,"publishedAt":167,"createdAt":167,"updatedAt":168,"category":169,"author":170,"tags":171},"b160f72d-9ed2-426f-aec1-9ca0a3d3224b","nvm-desktop、Nvm、Nrm","nvm-desktop-nvm-nrm","node版本管理器、下载源管理","https:\u002F\u002Fcdn.xiaolong0418.com\u002Fblog\u002F2026\u002F06\u002F6e128cd4-f101-4442-a6d4-68aa5d5c549e.png",2354,"2022-05-24T01:58:21.000Z","2026-07-12T15:44:55.368Z",{"id":82,"name":83,"slug":83},{"id":38,"name":39,"avatar":28},[172],{"id":87,"name":88,"slug":88},[174,186,196,202,208],{"id":175,"target":176,"postId":177,"parentId":28,"content":178,"createdAt":179,"user":180,"parentUser":28,"post":184},"72f0843a-9988-45da-a454-aa95736b43d9","POST","5ae0e368-2586-4030-847d-8fa2c78e839a","你好","2026-07-10T08:30:15.584Z",{"id":181,"name":182,"avatar":183},"99ca521b-0d71-42a3-8208-c0e23855f848","liam-小白龙","https:\u002F\u002Fgitee.com\u002Fassets\u002Fno_portrait.png",{"slug":185,"title":185},"前端蒙层中间挖空效果",{"id":187,"target":188,"postId":28,"parentId":189,"content":190,"createdAt":191,"user":192,"parentUser":195,"post":28},"071be543-ded7-4944-9bff-58f554f987f2","GUESTBOOK","f4f8d6e4-df47-4b51-bfbb-60f954e92125","33333","2026-06-27T01:35:00.600Z",{"id":193,"name":8,"avatar":194},"7c216ff5-de30-46f5-bfa1-dc5d5d85986d","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F46767504?v=4",{"id":193,"name":8},{"id":189,"target":188,"postId":28,"parentId":197,"content":198,"createdAt":199,"user":200,"parentUser":201,"post":28},"f628da57-d3d4-40f3-bf31-dc0ec1c661aa","4444444","2026-06-27T01:30:03.334Z",{"id":193,"name":8,"avatar":194},{"id":193,"name":8},{"id":197,"target":188,"postId":28,"parentId":203,"content":204,"createdAt":205,"user":206,"parentUser":207,"post":28},"0ebf3b5f-7cc7-44a4-b6ba-f3a0a0a13608","3333333","2026-06-27T01:29:49.323Z",{"id":193,"name":8,"avatar":194},{"id":193,"name":8},{"id":209,"target":188,"postId":28,"parentId":28,"content":210,"createdAt":211,"user":212,"parentUser":28,"post":28},"9828f20d-c871-4f1c-9cb9-796e171e0603","112222","2026-06-27T01:29:39.829Z",{"id":193,"name":8,"avatar":194},[214,217,219,221,223],{"amount":215,"message":28,"payTime":216},0.01,"2026-07-16T08:49:56.725Z",{"amount":215,"message":28,"payTime":218},"2026-07-14T08:18:40.896Z",{"amount":215,"message":28,"payTime":220},"2026-07-13T15:40:42.380Z",{"amount":215,"message":28,"payTime":222},"2026-07-13T14:26:38.767Z",{"amount":215,"message":28,"payTime":224},"2026-07-13T14:26:02.396Z"]