多场景、多 agent 的记忆系统

One Memory, Many Agents

谢锦翔（Jinxiang Xie） · 2026 年 10 月 30 日

一个人同时使用多个 LLM agent 已经很常见，比如使用 Muse 来规划日程、使用 Hermes 来追踪项目进度、使用 Codex 来执行一个具体的任务。但这些 agent 的记忆彼此隔离，同一条偏好或更正需要在每个系统里分别维护。本文介绍我为自己搭建的一套共享记忆层。所有 agent 读写同一份记忆接口，再由每个 agent 自己的 recall profile 决定召回的时机、范围和粒度。记忆的更新可以跨 session、跨 agent 自动运行。为了评估这套系统，我没有使用传统的记忆 benchmark，而是从自己的真实使用记录中构建了一个记忆评测环境。我从 agent eval 中吸取经验，结合程序化评估和基于 rubrics 的 LLM-as-a-Judge 进行评估，结合此后实际发生的对话来判断这一刻是否需要记忆，以及每条注入的记忆是否有用。在 467 个测试轮次上，相同召回预算下，本系统让 43.9% 的轮次拿到至少一条有用记忆，OpenViking 和 mem0 分别为 9.9% 和 32.3%，有害注入率为 2.0%，约为 mem0、BM25 和纯向量检索的一半。评测同时显示，在 agent 运行时，真正需要注入记忆的轮次只占 14%，而这其中约三分之二所需的信息从未被写入记忆。

本文的贡献。

1. 提出按 agent 控制召回剂量的 recall profile。同一份记忆由每个 agent 用自己的配置决定何时召回、能看哪些项目、注入什么粒度的信息。在 coding agent 上，它把普通编码提示的注入率从 18% 降到 2%，并且没有发生跨项目注入。
2. 搭建了一条没有人工审核的记忆自动更新管线，事实提炼、证据核对、与已有记忆的比对和回滚机制都自动完成。由人工标注的 38 张自动晋升卡片中有 31 张合格。
3. 我认为，记忆系统的评估不能依赖于合成数据的 benchmark，而是需要贴近真实使用记录的记忆评测环境，它衡量的是系统能否在对的时刻给出对的记忆，以及在不需要时能否保持安静。我基于个人的真实使用记录初步尝试了基于环境的记忆评估方式。

1. 背景：一个人身边的 agent 越来越多

过去一年里，我身边的 agent 数量增长得很快。自 Meta Muse 发布以来，personal agent 正逐步变成普通人的日常产品 [1]。Grok 在 8 月推出了 Grok Bot，每个 bot 都有自己的名字、职责和一台常驻的云电脑，可以全天候执行任务 [2]。开源社区也有自己部署的选择，例如 OpenClaw [3] 和我自己用作主助手的 Hermes Agent [4]。Coding agent 普及得更早，Codex、Claude Code 和 Pi [5] 已经成为我写代码时的日常工具。可以预见，以后还会出现更多面向特定领域的 agent（比如微信里的小微、飞书里的豆包工作），一个人在不同场景与多个 agent 协作可能将成为常态。

这些 agent 大多自带记忆，而且实现方式相当接近。Claude Code 用 CLAUDE.md 保存人写的长期指令，再用 auto memory 让模型自己积累经验 [6]。Codex 在本地维护一个独立于 ChatGPT 网页版的记忆库 [7]。OpenClaw 把记忆写成工作区里的纯文本 Markdown 文件 [8]。Hermes 则有两个常驻上下文的文件，MEMORY.md 记环境和经验，USER.md 记用户画像 [9]。单独看，每一种设计都合理。然而，在一个人同时使用它们的时候就会出现问题。

一个问题是记忆割裂。我在 Hermes 里定下“Python 环境一律用 uv 建独立环境”，Codex 并不知道，除非我在 Codex 里再说一遍。另一个问题是重复和矛盾。同一条偏好要在四个地方各写一次，而当相关内容有所变更，往往只有当时在用的那个 agent 更新了记忆，其余几个仍然按旧说法工作。时间一长就很难维护。

因此我希望有一套统一的记忆层，所有 agent 都从同一份记忆里读，也都把新学到的东西写回同一份记忆。它需要满足四个条件。第一，记忆只有一份真相源，改一处就对所有 agent 生效。第二，它通过统一的接口提供服务，比如通过 API 或 MCP，这样接入新 agent 时不需要改动记忆系统本身。第三，它要能跨设备共享。我的 agent 分布在本机和远程服务器上，这套服务通过 Tailscale Serve 只在我自己的私有网络（tailnet）内开放 [10]。第四，也是本文最关心的一点，每个 agent 都能单独控制召回的强度和粒度。

给 LLM agent 加外部记忆已经有不少研究。MemGPT 把有限的上下文窗口类比为操作系统的内存，让模型自己决定把哪些信息换进或换出上下文 [11]。Generative Agents 用自然语言记录 agent 的完整经历，再按时效、重要性和相关性检索，并定期把记忆综合成更高层的反思 [12]。MemoryBank 借鉴艾宾浩斯遗忘曲线，让记忆随时间衰减，并在被回忆时得到强化 [13]。Mem0 用 LLM 从对话中提炼事实，再对每条新事实选择新增、更新、删除或不变，以此维护记忆库 [14]。Zep 用带时间维度的知识图谱表示记忆，让过时的事实失效，同时保留历史 [15]。A-Mem 借鉴卡片盒笔记法，让记忆条目之间互相链接，并随着新信息不断演化 [16]。OpenViking 是火山引擎开源的上下文数据库，它把记忆、资源和技能统一组织成文件系统，并为每个节点生成三层表示：L0 是一句话摘要，L1 是概览，L2 是原文。agent 可以先读摘要，再决定是否打开全文 [17]。

这些工作关注的大多是单个 agent 怎样记得更多、更准。而我的场景多出一个维度：同一份记忆要同时服务几个角色完全不同的 agent，而它们需要的剂量相差很大。

2. 一份记忆，多种剂量

2.1 为什么召回要有剂量

直觉上，记忆给得越多越好，模型自己会挑有用的部分。已有研究并不支持这种直觉。Liu 等人发现，当相关信息位于长上下文的中间位置时，模型的表现会明显下降，即使是专门为长上下文设计的模型也是如此 [18]。Shi 等人在数学题里加入无关信息后，多种提示方法的准确率都大幅下降 [19]。Chroma 的测试也显示，随着输入变长，模型表现会以不均匀、难以预测的方式退化 [20]。Anthropic 在讨论 agent 的上下文工程时，把上下文描述成一种有限的注意力预算，并建议只放入能产生预期行为的最少量高信号内容 [21]。

检索增强生成的研究也得出了相近的结论。Mallen 等人发现，检索对冷门知识帮助很大，对模型已经掌握的热门知识帮助有限，于是提出只在必要时才检索的自适应策略，同时降低了推理成本 [22]。Self-RAG 指出，不管是否需要都检索固定数量的段落，会降低模型的通用性，因此训练模型按需决定是否检索 [23]。FLARE 则在生成过程中主动判断何时检索、检索什么 [24]。更早的人机交互研究对这个问题也有讨论。Horvitz 在提出混合主动界面的原则时强调，系统在主动介入之前，应该结合对用户意图的不确定性，权衡行动的预期收益和打扰用户的代价 [25]。

对一个共享的记忆层来说，这些结论意味着两件事。第一，召回本身有成本，无关的记忆会分散模型的注意力，也会占用本该留给任务的上下文。第二，合适的剂量取决于 agent 的角色。负责统筹全局的主助手需要更宽的视野，正在修改某个函数的 coding agent 则几乎不需要额外信息。

2.2 系统概览

图 1：系统架构。上排是读路径，每次提问都会走一遍（绿色实线，从左往右）。下排是写路径，每次会话结束后在后台静默进行（灰色虚线，从右往左）。

据此我提出了 dose-mem，图 1 展示了系统的整体结构。左侧是以 Markdown 为真相源的记忆仓库，中间是对所有 agent 提供统一接口的记忆服务，右侧是各个 agent。服务内部有两条路径：每次提问时运行的读路径，以及每次会话结束后在后台运行的写路径。

每个 agent 都带着一个身份 token 和一个 profile 接入记忆服务。身份决定它能读写哪些项目的记忆，profile 决定它每次拿到多少记忆。接入新的 agent 时，只需要签发一个 token、写一个 profile，不需要改动服务本身。服务默认只监听本机端口。

2.3 Recall profile

Recall profile 是整套设计的核心。一个 profile 由几个参数组成：触发方式决定什么时候召回，范围决定能看哪些项目，层级决定注入一句话摘要还是摘要加概览，此外还有召回的内容长度预算、最多条数和分数阈值。

第一种是给主助手用的 full profile。我的主助手是 Hermes，它负责协调我所有的研究项目，持续跟进实验和任务进度，因此需要随时了解各个项目的背景。full profile 在每一轮提问时都会尝试召回，可以跨项目检索，每轮最多注入 5 条记忆，总长度不超过 1,200 字节，每条包含一句话摘要和概览两层。

第二种是给 coding agent 用的 working profile，用于 Claude Code、Codex 和 Pi。它们是具体任务的执行者，通常由 Hermes 或我直接委派任务，在某个代码仓库里工作。working profile 只在三种情况下召回：会话的第一轮，工作切换到另一个项目时，或者我说了“按我的习惯”“上次”“之前”这类明确指向记忆的话。它只检索当前项目的记忆，再加上全局记忆中偏好、踩坑和约定三类卡片。每次最多注入 2 条一句话摘要，每条附带链接，agent 需要细节时可以自己读取原文。它的筛选分数阈值也高于 full profile。

第三种是给 personal agent 用的 personal profile，面向管理日程、邮件这类个人事务的 agent。它只召回与个人习惯相关的记忆，例如“早上 10 点喜欢来一杯咖啡”，不接触任何研究项目的内容。

2.4 基于 context 的记忆检索

记忆检索分为四步。第一步是确定这一轮属于哪个项目。系统从项目注册表里读取每个项目的名称、别名和昵称。如果 agent 显式指定了项目，就以它为准。如果没有指定，只有当提问唯一地命中某个项目时才推断范围，有歧义时保守地只检索全局记忆。对 coding agent 来说，项目由当前工作目录映射得到。

第二步是混合检索。系统把同一个查询分别交给稠密向量检索和 BM25 稀疏检索。稠密向量擅长语义上的近似匹配 [26]，BM25 擅长精确的词汇匹配 [27]，两者在中文和英文混杂、术语和数字密集的研究笔记上恰好互补。两路结果用 reciprocal rank fusion（RRF）合并，也就是按每路排名的倒数加权求和 [28]。这种方法不需要把两路分数校准到同一尺度，在检索文献里是一种稳健的融合方式。

第三步决定要不要插入记忆，这一步在设计上和“说什么”同样重要。检索结果会经过一系列过滤，分数低于 profile 阈值的条目也会被丢掉。如果没有一条合格，这一轮就什么都不注入。对 working profile，系统还要求一句话摘要本身与提问有足够的词汇支持，因为它只展示摘要，即使全文高度相关，也不应该注入一句与问题对不上的标题。

第四步是在预算内拼装。系统按排名依次放入完整的条目。

2.5 易于维护的 Markdown 真相源

所有记忆都保存在一个 Git 仓库里的 Markdown 文件中，包括每个项目的简报、决策记录、一事一卡的记忆卡片和项目注册表。日常维护主要由 AI 完成，但我随时可以打开任何一个文件阅读、比较版本或手动修改。向量索引只是从这些文件派生出来的检索状态，随时可以从 Markdown 完整重建。我选择这种设计，一是因为人能读懂并检查的记忆更容易建立信任，二是因为 OpenClaw 等系统已经证明纯文件的记忆在实践中足够好用。

每条记忆都有三层表示，这一点参考了 OpenViking。L0 是一句话结论，取自卡片的结论字段或首句。L1 是有长度上限的概览。L2 是按标题切分的原文块。与 OpenViking 不同的是，这三层完全按规则生成，不调用 LLM，因此重建索引不产生模型调用费用，每次重建的结果也一致。每张卡片还带有生命周期信息。实验结果默认 180 天后过期，决策 365 天后过期，偏好、踩坑和约定没有期限。项目在注册表中被标记为归档后，它的记忆默认不再被自动召回，但仍然可以显式读取。

3. Case study

图 2：同一句话，两种剂量。每行是同一句提示分别发给主助手（full）和 coding agent（working）时实际注入的内容，条的长短是注入字节数（上限 1,200 字节）。三个例子是线上服务的真实输出，项目名做了泛化。

图 2 用三个真实例子展示了两种 profile 的差别。每个例子都是同一句话分别发给主助手和 coding agent，再记录两边实际收到的记忆。

第一个例子是一个普通的编码步骤：“把 parse_table 改个名，顺便更新测试。”主助手的 full profile 每轮都会召回，它被塞进了 1,199 字节的“后训练经验”。这条记忆和重命名函数毫无关系，只会占用上下文。coding agent 的 working profile 没有被触发，什么都没收到。对一个正在改代码的 agent 来说，这正是合适的剂量。

第二个例子是“帮我配一下这个项目的 Python 环境”。这句话在两个 agent 中都进行了召回。两边都拿到了“每个项目用 uv 建独立环境”这条记忆。主助手还多拿到一条“大型数据放在外置盘的哪个位置”。这条记忆其实也有用，working profile 为了控制剂量没有给。

第三个例子是在 B 项目的代码仓库里，问 A 项目的论文该怎样引用软件许可证。主助手召回了 A 项目的三条记忆，其中决策记录是有用的。coding agent 被直接挡住，收到 0 字节。在 B 项目里工作的 coding agent 不应该读到 A 项目的内容，这是有意设计的隔离。如果确实需要跨项目的信息，应该从全局记忆中召回，或是由主助手来协调。

为了在更大的样本上验证这些现象，我在隔离环境里做了两组测量。第一组是 50 条独立的普通编码提示。在 full profile 下，其中 9 条（18%）被注入了记忆，在 working profile 下只有 1 条（2%）。第二组是 5 个模拟的 10 轮编码会话。在这组会话上，working profile 召回的记忆长度减少了 79%。

这种低剂量有明确的代价。在评测集上，working profile 在前 2 条结果中召回到正确来源的比例约为 30%，full profile 在前 5 条中约为 75%。也就是说，coding agent 在不少本可以受益于记忆的时刻没有拿到记忆。我接受这个取舍，原因有两个。一是 Horvitz 所说的打扰代价对 coding agent 尤其高 [25]，一条无关记忆可能把它带离当前任务。二是这个损失可以弥补，coding agent 一直可以主动调用 memory_search 工具查询记忆，自动召回只是一个低剂量的默认值。

4. 记忆的自然生长

图 3：一次没人审核的更正。这是端到端验收测试里的真实记录：先说“未验证的条目标成琥珀色”，后来改口“换成青色”。系统自动用新卡取代（supersede）旧卡，旧卡留在磁盘上，不再被召回。

记忆系统必须回答一个问题：新的记忆从哪里来，谁来保证它是对的。最稳妥的做法是让人审核每一条候选记忆。我最初也是这样设计的，结果待审核的候选在收件箱里积压到了 218 条。我不想当记忆的审核员，事实证明我也没有时间当。但错误的长期记忆比没有记忆更糟，它会在此后几十次对话里反复误导 agent，而且不容易被发现。所以写入路径的目标是用自动检查代替人工审核，同时保持准入标准不降低。

写入路径分为四步，对应图 1 下排从右往左的流程。

第一步是捕获。系统只读取已经结束的会话。从本地读取 Hermes、Codex 和 Pi 的空闲会话的历史，至少包含 3 个用户轮次。每个会话被压缩成摘要，只保留用户的话、agent 的最终回答、调用过的工具名和有限的错误分类。

第二步是提炼。LLM 把摘要拆成原子事实，每条事实都要写明所属项目和置信度，并且必须附上一段连续的原话作为证据。系统随后逐字核对这段引文是否真的出现在原始对话中，只允许空白、全半角和引号这类格式差异。没有出处的事实会被直接丢弃。这一步的做法与 Mem0 用 LLM 从对话中提炼事实的思路相近 [14]，区别在于这里强制要求可核验的原文证据。

第三步是自动审核。每条候选先经过不调用模型的确定性预筛，再用混合检索找出最相近的 12 张已有卡片，最后交给一个 LLM 评委成批判断。评委有四种决定。promote 表示作为新卡片晋升，merge 表示并入已有卡片的证据，supersede 表示取代一张旧卡片，reject 表示丢弃。supersede 的门槛最高，必须有更新、更强、可核验的来源，或者是我本人明确说过的话。

第四步是写入。写入路径只追加新卡片，从不改写旧文件。被取代的旧卡片仍然保留在仓库里，新卡片通过 supersedes 字段指向它，检索时旧卡片会被过滤掉。图 3 展示的就是这样一次更正：我先说“未验证的条目标成琥珀色”，后来改口“换成青色”，系统判定为 supersede，旧卡片留在磁盘上，从此不再被召回。

我把由自动审核上线后的 38 张卡片全部读了一遍。其中 31 张合格，例如某个项目的许可证约束、某个测试集只能用于开发诊断、某次实验结果其实是评估格式造成的假象。6 张不该进入记忆，它们分别是一条通用的数据库知识、一段带时间戳的故障经过、一份命令修复日志、一个 Python 版本快照、一次交接文档的创建记录和一条一次性的任务指令。我把这 6 种错误类型写进了评委的拒绝规则，并用归档卡片取代了这 6 张卡片。

5. 评测环境：在真实使用里测记忆

5.1 现有 benchmark 测的是什么

长期记忆领域最常用的 benchmark 是 LoCoMo [29]。它先让 LLM agent 依据人设和事件图谱生成两个人之间的长对话，每段对话平均约 300 轮，最多跨越 35 个会话，经过人工校对后，再用第三人称的问答测试模型能否答出对话中的某个事实。Mem0 的主要实验就建立在 LoCoMo 上 [14]。LongMemEval 设计了 500 道题，覆盖信息提取、跨会话推理、时间推理、知识更新和拒答五种能力，并发现长上下文模型在其上的表现下降 30% 到 60% [30]。MemoryAgentBench 把评测改为增量的多轮交互，考察准确检索、测试时学习、长程理解和选择性遗忘 [31]。

这些 benchmark 推动了很多进展，LongMemEval 也已经覆盖了知识更新和拒答。不过它们有一个共同的设定：对话是生成的，问题是事后出的，出题时已经保证答案存在于历史中。我在实际使用中关心的问题与此不同。当我对 agent 说出下一句话时，记忆系统首先要判断这一刻需不需要记忆。如果需要，它要给出对的那几条。如果不需要，它应该保持安静。这个判断每一轮都在发生，而现有 benchmark 很难测出这种能力，因为它们几乎每一道题都需要记忆。

此外，记忆系统之间的比较本身也很脆弱。Zep 最初报告在 LoCoMo 上达到 84%，Mem0 的团队指出其计算有误，认为正确数字是 58.44%，Zep 修正后报告为 75.14% [32, 33]。同一个系统、同一个 benchmark，仅仅因为评测细节不同，数字就相差二十多个百分点。这提醒我，在自己的评测中，对比系统的接入方式必须尽可能贴近它们本来的用法。

5.2 从真实使用中构建记忆环境

基于上面的考虑，我从自己真实的 agent 使用记录中构建了一个评测环境。数据来自 2026 年 9 月 21 日到 10 月 26 日之间，我与 Hermes、Codex 和 Pi 的私人会话。我从中冻结了 500 个探针，其中 363 个来自 Hermes，89 个来自 Codex，48 个来自 Pi。数据按时间切分，前两周作为开发集，用于调参和选阈值，第 3 到 5 周作为测试集。

每一个需要测试的点称为探针，每个探针是我在真实会话中说过的一句话。系统在回答这个探针时，只能看到此前的上一轮对话和此前的记忆快照。我在这句话之后实际说了什么、agent 实际做了什么，最多 6 轮，只提供给评委，绝不进入任何系统的检索请求。记忆快照来自记忆仓库的 Git 历史。我每周取一个快照，它严格早于该周所有的探针，因此系统不可能读到未来的信息。

5.3 标注与指标

为了构建合理的评价指标，我人工构建了 rubrics，每个 rubrics 的真值标签由 LLM 给出。为了减少 LLM-as-a-Judge 的偏差，评委看不到系统名称和检索分数，候选条目的顺序被随机打乱，每个判断都必须引用候选原文或后续对话中的具体片段。

对每个探针，评委需要判断两件事。第一，结合我之后实际说的话和 agent 实际做的事，这一刻是否需要记忆。第二，每条注入的记忆属于哪一类：必要（没有它就做不对），有帮助，无关，或者有害（会误导 agent）。评委还会检查当我改过口时，系统给出的是新说法还是旧说法。为了检验标签的可靠性，我采用了不同厂商的模型独立复判了随机抽取的 20% 探针，并计算模型之间的一致性。

基于这些标签，我关心几个指标。命中率是至少注入了一条有用记忆（必要或有帮助）的轮次比例，它衡量系统有多少次真正帮上了忙。有用率和有害率分别是所有注入条目中有用和有害的比例，它们衡量注入内容的质量。此外还有需要记忆的轮次中拿到有用内容的比例，以及注入的内容长度。综合分把“说”和“不说”放在一起：在需要记忆的轮次，召回到必要条目得分，在不需要的轮次，保持沉默得分，注入有害内容则扣分。所有置信区间都按会话聚类做 2,000 次 bootstrap，因为同一会话里的轮次彼此相关，把它们当作独立样本会低估不确定性 [34]。

5.4 对比系统与公平接入

对比系统有四个：OpenViking [17]、mem0 [14]、纯稠密向量检索 [26] 和纯 BM25 检索 [27]。后两个是检索领域最基本的基线，用来判断复杂系统的收益是否真的超过了简单方法。所有系统读取同一份记忆快照，使用同一个 embedding 模型（text-embedding-3-small），由同一个评委打分。每个系统只在开发集上调一个阈值，测试集只用于最终报告。

吸取 LoCoMo 争议的教训，我让每个系统都按它本来的用法接入。OpenViking 使用它为 OpenClaw 等插件做自动注入时调用的原生接口（context assembly）。mem0 使用它自己的写入方式（infer=True），由它的 LLM 从每份文档中提炼事实，并按周的顺序依次写入，每个探针只能检索到当时已经存在的记忆。

这一点在实验中被证明非常重要。我第一次做对比时，OpenViking 用的是通用检索接口，返回的多是带元数据的目录概览，mem0 则直接存入原文，不做提炼。那一轮里，它们注入内容的有用率只有 4.9% 和 5.9%。改为原生用法后，两者分别上升到 16.5% 和 19.0%，都提高到原来的三倍以上。

5.5 结果

图 4：同一份记忆、同样的召回内容预算下，各系统注入了什么。A 是注入内容的质量，B 是各系统说了多少话。测试集为 467 个已评判的真实对话轮次，分布在 83 个会话里。细线是按会话聚类的 95% 置信区间。

测试集最终包含 467 个已评判的轮次，分布在 83 个会话中。表 1 是相同注入预算下五个系统的结果。

在相同的召回内容预算下，本系统的命中率为 43.9%，比 mem0、BM25 和纯向量检索高 12 到 15 个百分点，比 OpenViking 高 34 个百分点。它的有害率为 2.0%，显著低于 mem0、纯向量检索和 BM25，大约是它们的一半，与 OpenViking 相当。在真正需要记忆的 65 个轮次中，本系统和 mem0 拿到有用内容的比例接近，分别为 56.9% 和 53.8%，都高于其余三个系统。

另一方面，单条记忆的准度在各系统之间没有显著差别，有用率都在 16% 到 21% 之间。这说明本系统的优势并不来自更会挑选单条记忆。它来自两个地方。其一是预算用得更省。本系统每条记忆只注入一句话标题，以及概览中标题之外的部分，同样的 1,200 字节平均能放下 3.2 条，其他系统只能放下 2 到 2.7 条。其二是过滤。按项目范围和版本链过滤，使本系统较少注入其他项目的内容和已被取代的旧说法，这在有害率上体现得最明显。OpenViking 的有害率也很低，但主要原因是它在 59% 的轮次里什么都没有注入，这同时也让它的命中率最低。

如果让每个系统在开发集上调出各自最优的阈值，情况就完全不同了。所有系统都变得几乎沉默，只在 1.5% 到 5.3% 的轮次注入记忆，综合分挤在 0.81 到 0.85 之间，BM25 最高，本系统为 0.84。在 65 个需要记忆的轮次里，没有一个系统召回到被标为“必要”的条目。综合分如此接近，是因为 86% 的轮次本来就不需要记忆，一个永远沉默的系统也能拿到很高的分数。所以我不认为本系统在所有方面都超过了对比系统。更准确的说法是，在愿意注入一定剂量的前提下，它能以更低的风险让更多轮次受益。

5.6 两个关于真实使用的发现

这次评测提供的额外 insights 是，真正需要记忆的时刻很少。在 467 个测试轮次中，评委认为需要记忆的只有 65 个，占 14%。我进一步检查了这 65 个轮次，发现其中 42 个（约三分之二）所需的信息在所有系统的候选中都找不到。其中最多的是“跑完了吗”这类对正在进行的任务的追问，共 21 个，这类问题需要的是任务的实时状态，而它从来不在长期记忆里。剩下的是一些从未被写入记忆的细节，例如某个 PR 的链接或某个具体的数值上限。即使把五个系统的全部候选合在一起，也有 77.7% 的轮次至少包含一条有用的条目，可见检索本身并不是瓶颈。这个结果说明，下一步最值得补的是任务状态的记录，继续改进检索的边际收益很小。这也与 LongMemEval 等 benchmark 的设定形成对照，那里每一道题都保证答案存在于历史之中 [30]。

5.7 局限

这项评测有几处明显的局限，需要在解读结果时考虑。第一，数据只来自一个用户，也就是我自己，而本系统恰好是在我的数据上开发的。其他人的使用习惯可能完全不同。

第二，评估只用了 LLM。在 90 个被复判的轮次上，条目标签的一致率为 86%（Cohen’s κ=0.41），“这一轮是否需要记忆”的一致率为 69%（κ=0.17）。按照 Landis 和 Koch 的常用划分，前者处于中等一致的下限，后者只是轻微一致 [35]。因此，与“是否需要记忆”相关的数字噪声较大，解读时应当谨慎。

第三，所有系统读取的都是同一份已经整理好的记忆，这次没有比较各系统从原始聊天中自己建立记忆的能力。

最后，原始数据涉及个人隐私，不会公开，我只公开方法和脱敏后的示例。

6. 结尾

做完这一轮工作，我对记忆系统的理解发生了变化。我原本以为难点在于记住更多的东西，现在看来，更难的是在对的时刻给对的 agent 合适的剂量，在不该说话时保持安静，并且在没有人看着的情况下，自己生长、自己改错。

还有几个问题我没有答案。第一，profile 目前是手工调的。它能否从使用中自动学习？例如 coding agent 每次主动调用 memory_search，都可以看作一次“该给的没给”的信号，可以用来调整触发条件和阈值。第二，在多用户的场景下，这种基于真实使用记录的评测要怎么做？一个人的数据说服力有限，但真实记录又天然涉及隐私。第三，需要记忆的轮次中，有三分之二缺的是任务状态。这部分信息应该由记忆系统接管，还是交给另一套专门的任务系统，我还没有想清楚。

本文的搭建说明和评测工具已开源：https://github.com/jxtse/dose-mem

One Memory, Many Agents

Jinxiang Xie · October 30, 2026

Using several LLM agents at once has become common: Muse to plan the day, Hermes to track research projects, Codex to carry out a specific task. Yet these agents keep their memories apart, so the same preference or correction has to be maintained separately in each system. This post describes a shared memory layer I built for myself. Every agent reads and writes through the same memory interface, and each agent’s own recall profile decides when to recall, what scope to search and how fine-grained the injected memory is. Memory updates run automatically across sessions and across agents. To evaluate the system I did not use a conventional memory benchmark. Instead, I built a memory evaluation environment from my own real usage. Borrowing from agent evaluation practice, I combined programmatic checks with a rubric-based LLM-as-a-Judge, which uses the conversation that actually followed to decide whether a given moment needed memory and whether each injected memory helped. On 467 test turns, under the same recall budget, this system gives at least one useful memory on 43.9% of turns, against 9.9% for OpenViking and 32.3% for mem0. Its harmful-injection rate is 2.0%, about half that of mem0, BM25 and dense retrieval. The evaluation also shows that only 14% of turns actually needed injected memory while the agent was running, and in about two thirds of those, the information needed had never been written to memory.

Contributions.

1. A recall profile that sets the recall dose per agent. Every agent reads the same memory, and its own configuration decides when to recall, which projects it may see, and how fine-grained the injected information is. For coding agents it lowers the share of routine coding prompts that receive memory from 18% to 2%, with no cross-project injection.
2. An automatic memory-update pipeline with no human review. Fact distillation, evidence checking, comparison with existing memory and rollback all run automatically. Of the 38 auto-promoted cards that I labelled by hand, 31 were good.
3. I believe memory systems cannot be evaluated with benchmarks built on synthetic data. They need an evaluation environment close to real usage, one that measures whether the system gives the right memory at the right moment and stays quiet when nothing is needed. I made a first attempt at such environment-based evaluation using my own usage records.

1. Background: more and more agents around one person

Over the past year the number of agents around me has grown quickly. Since Meta released Muse, personal agents have been turning into everyday products for ordinary people [1]. In August, Grok launched Grok Bot, where each bot has its own name, role and an always-on cloud computer and can work around the clock [2]. The open-source community has self-hosted options too, such as OpenClaw [3] and Hermes Agent [4], which I use as my main assistant. Coding agents spread even earlier: Codex, Claude Code and Pi [5] are part of how I write code every day. More domain-specific agents are clearly coming (WeChat’s Xiaowei and Doubao Work in Feishu, for example), and working with several agents across different settings is likely to become normal.

Most of these agents ship with their own memory, built in similar ways. Claude Code keeps human-written long-term instructions in CLAUDE.md and lets the model accumulate experience through auto memory [6]. Codex keeps a local memory store separate from ChatGPT on the web [7]. OpenClaw writes memory as plain Markdown files in its workspace [8]. Hermes has two files that stay in context: MEMORY.md for environment and lessons, USER.md for the user profile [9]. Each design is reasonable on its own. The trouble starts when one person uses all of them at once.

One problem is fragmented memory. If I tell Hermes that “every Python environment should be an isolated uv environment”, Codex does not know unless I say it again in Codex. Another problem is duplication and contradiction. The same preference has to be written in four places, and when something changes, usually only the agent I happened to be using updates its memory while the others keep working from the old version. Over time this becomes hard to maintain.

So I wanted a unified memory layer: every agent reads from the same memory and writes what it learns back to the same memory. It has to meet four conditions. First, memory has a single source of truth, so a change in one place takes effect for every agent. Second, it serves through a uniform interface, such as an API or MCP, so connecting a new agent needs no change to the memory system itself. Third, it can be shared across devices. My agents run on this machine and on remote servers, and the service is exposed only inside my own private network (tailnet) through Tailscale Serve [10]. Fourth, and the focus of this post, each agent can control the strength and granularity of its own recall.

There is already a good deal of research on external memory for LLM agents. MemGPT compares the limited context window to an operating system’s memory and lets the model decide what to page in and out of context [11]. Generative Agents record an agent’s full experience in natural language, retrieve it by recency, importance and relevance, and periodically synthesize memories into higher-level reflections [12]. MemoryBank borrows the Ebbinghaus forgetting curve so that memories decay over time and are reinforced when recalled [13]. Mem0 uses an LLM to distill facts from conversations and, for each new fact, chooses to add, update, delete or leave the memory store unchanged [14]. Zep represents memory as a temporal knowledge graph, invalidating outdated facts while keeping their history [15]. A-Mem borrows the Zettelkasten method, linking memory entries to each other and letting them evolve as new information arrives [16]. OpenViking, the context database open-sourced by Volcengine, organizes memory, resources and skills as one file system and gives every node three representations: L0 is a one-sentence summary, L1 an overview, L2 the original text. An agent can read the summary first and then decide whether to open the full text [17].

Most of this work is about how a single agent can remember more, and more accurately. My setting adds a dimension: the same memory has to serve several agents with entirely different roles, and the dose each one needs differs widely.

2. One memory, many doses

2.1 Why recall needs a dose

Intuitively, the more memory the better, since the model will pick out what is useful. Existing research does not support this intuition. Liu et al. found that models perform markedly worse when the relevant information sits in the middle of a long context, even models designed for long contexts [18]. Shi et al. added irrelevant information to math problems and saw accuracy fall sharply across many prompting methods [19]. Chroma’s tests also show that as input grows longer, model performance degrades in uneven and hard-to-predict ways [20]. In its discussion of context engineering for agents, Anthropic describes context as a finite attention budget and recommends including only the smallest set of high-signal content that produces the desired behaviour [21].

Research on retrieval-augmented generation points the same way. Mallen et al. found that retrieval helps a lot for long-tail knowledge but little for popular knowledge the model already holds, and proposed an adaptive strategy that retrieves only when needed, which also lowers inference cost [22]. Self-RAG notes that retrieving a fixed number of passages regardless of need hurts the model’s versatility, and trains the model to decide when to retrieve [23]. FLARE actively decides during generation when to retrieve and what to retrieve [24]. Earlier human-computer interaction research addressed the same question. In his principles of mixed-initiative user interfaces, Horvitz stressed that before acting on its own initiative a system should weigh the expected benefit of acting against the cost of interrupting the user, taking its uncertainty about the user’s intent into account [25].

For a shared memory layer these findings mean two things. First, recall has a cost: irrelevant memory distracts the model and takes up context that should go to the task. Second, the right dose depends on the agent’s role. A main assistant that coordinates everything needs a wide view, while a coding agent editing one function needs almost no extra information.

2.2 System overview

Figure 1. System architecture. The top row is the read path, run on every prompt (solid green, left to right). The bottom row is the write path, run silently in the background after each session ends (dashed grey, right to left).

Building on this, I designed dose-mem, whose overall structure is shown in Figure 1. On the left is the memory repository, with Markdown as the source of truth. In the middle is the memory service, which offers one interface to every agent. On the right are the agents. The service has two paths inside: the read path, run on every prompt, and the write path, run in the background after each session ends.

Each agent connects to the memory service with an identity token and a profile. The identity decides which projects’ memory it may read and write, and the profile decides how much memory it gets each time. Connecting a new agent only takes issuing a token and writing a profile, with no change to the service itself. By default the service listens only on a local port.

2.3 Recall profile

The recall profile is the core of the design. A profile consists of a few parameters: the trigger decides when to recall, the scope decides which projects may be searched, the level decides whether to inject only a one-line summary or the summary plus an overview, and there are also a length budget for recalled content, a maximum number of items and a score threshold.

The first is the full profile, for the main assistant. My main assistant is Hermes. It coordinates all my research projects and keeps track of experiments and tasks, so it needs the background of every project at hand. The full profile tries to recall on every prompt, may search across projects, and injects at most 5 memories per turn, no more than 1,200 bytes in total, each with a one-line summary and an overview.

The second is the working profile, for coding agents: Claude Code, Codex and Pi. They execute concrete tasks, usually delegated by Hermes or by me, inside some code repository. The working profile recalls only in three situations: on the first turn of a session, when the work moves to another project, or when I say something that points explicitly at memory, such as “the way I usually do it”, “last time” or “before”. It searches only the current project’s memory, plus three types of global cards: preferences, pitfalls and conventions. It injects at most 2 one-line summaries, each with a link the agent can follow to read the original when it needs detail. Its score threshold is also higher than the full profile’s.

The third is the personal profile, for personal agents that manage personal matters such as calendars and email. It recalls only memories about personal habits, for example “likes a cup of coffee at 10 in the morning”, and never touches research projects.

2.4 Context-aware memory retrieval

Retrieval takes four steps. The first decides which project this turn belongs to. The system reads each project’s name, aliases and nicknames from the project registry. If the agent names a project explicitly, that wins. Otherwise the scope is inferred only when the prompt matches exactly one project; when it is ambiguous, the system conservatively searches global memory only. For coding agents the project is mapped from the current working directory.

The second step is hybrid retrieval. The system sends the same query to dense vector retrieval and to BM25 sparse retrieval. Dense vectors are good at approximate semantic matches [26], and BM25 at exact lexical matches [27]; on research notes that mix Chinese and English and are dense with terms and numbers, the two complement each other well. The two result lists are merged with reciprocal rank fusion (RRF), which sums the reciprocal of each list’s rank [28]. This needs no calibration of the two scores onto one scale, and the retrieval literature regards it as a robust way to fuse rankings.

The third step decides whether to insert memory at all, which in this design matters as much as what to say. Retrieved results pass through a series of filters, and items scoring below the profile’s threshold are dropped. If nothing qualifies, nothing is injected this turn. For the working profile the system also requires the one-line summary itself to have enough lexical support from the prompt: since only the summary is shown, a title that does not match the question should not be injected even if the full text is highly relevant.

The fourth step assembles the result within the budget. The system adds whole items in rank order.

2.5 A Markdown source of truth that is easy to maintain

All memory lives in Markdown files in a Git repository: each project’s brief, its decision log, one-fact-per-card memory cards, and the project registry. Day-to-day maintenance is mostly done by AI, but I can open, diff or edit any file at any time. The vector index is only retrieval state derived from these files and can be fully rebuilt from the Markdown at any time. I chose this design for two reasons: memory that people can read and check is easier to trust, and systems such as OpenClaw have already shown that plain-file memory works well enough in practice.

Every memory has three representations, an idea taken from OpenViking. L0 is a one-sentence conclusion, taken from the card’s conclusion field or first sentence. L1 is an overview with a length cap. L2 is the original text split into chunks by heading. Unlike OpenViking, all three layers are generated by rules without calling an LLM, so rebuilding the index costs nothing in model calls and gives the same result every time. Each card also carries lifecycle information. Experimental results expire after 180 days by default, decisions after 365 days, and preferences, pitfalls and conventions never expire. Once a project is marked archived in the registry, its memory is no longer recalled automatically by default, though it can still be read explicitly.

3. Case study

Figure 2. Same prompt, two doses. Each row shows what was actually injected when the same prompt went to the main assistant (full) and to a coding agent (working); bar length is injected bytes (capped at 1,200 bytes). The three examples are real outputs of the live service, with project names generalized.

Figure 2 uses three real examples to show how the two profiles differ. In each, the same prompt was sent to the main assistant and to a coding agent, and I recorded the memory each side actually received.

The first example is a routine coding step: “Rename parse_table and update the tests.” The main assistant’s full profile recalls on every turn, and it received 1,199 bytes of “lessons from post-training”. That memory has nothing to do with renaming a function and only takes up context. The coding agent’s working profile was not triggered and received nothing. For an agent in the middle of editing code, that is exactly the right dose.

The second example is “Set up the Python environment for this project.” This prompt triggered recall in both agents. Both received the memory “use an isolated uv environment for each project”. The main assistant also received “where large data lives on the external drive”. That memory is useful too, but the working profile left it out to keep the dose small.

The third example is asking, inside project B’s repository, how project A’s paper should cite software licenses. The main assistant recalled three memories from project A, of which the decision log was useful. The coding agent was blocked outright and received 0 bytes. A coding agent working in project B should not read project A’s content; this isolation is by design. When cross-project information is really needed, it should come from global memory or be coordinated by the main assistant.

To check these effects on a larger sample, I ran two sets of measurements in an isolated environment. The first used 50 independent routine coding prompts. Under the full profile, 9 of them (18%) received memory; under the working profile only 1 (2%). The second used 5 simulated 10-turn coding sessions. On these sessions the working profile cut the length of recalled memory by 79%.

This low dose has a clear cost. On the evaluation set, the working profile recalls the correct source within its top 2 results about 30% of the time, while the full profile does so within its top 5 about 75% of the time. In other words, the coding agent misses memory at many moments when it could have benefited. I accept this trade-off for two reasons. First, the cost of interruption that Horvitz describes is especially high for coding agents [25]: one irrelevant memory can pull them away from the task at hand. Second, the loss can be made up: a coding agent can always call the memory_search tool to query memory itself, and automatic recall is only a low-dose default.

4. Memory that grows on its own

Figure 3. A correction nobody reviewed. This is a real record from the end-to-end acceptance test: I first said “mark unverified entries in amber” and later changed it to “use teal instead”. The system automatically superseded the old card with a new one; the old card stays on disk but is no longer recalled.

A memory system has to answer one question: where do new memories come from, and who makes sure they are right? The safest answer is to have a person review every candidate memory. That is how I first designed it, and the review inbox piled up to 218 candidates. I did not want to be my memory’s reviewer, and it turned out I did not have the time either. But wrong long-term memory is worse than no memory: it misleads agents again and again over dozens of later conversations, and it is hard to notice. So the goal of the write path is to replace human review with automatic checks without lowering the bar for admission.

The write path has four steps, matching the bottom row of Figure 1 from right to left.

The first step is capture. The system reads only sessions that have ended. It reads the history of idle Hermes, Codex and Pi sessions locally, each with at least 3 user turns. Each session is compressed into a digest that keeps only the user’s words, the agent’s final answers, the names of tools called, and a limited classification of errors.

The second step is distillation. An LLM splits the digest into atomic facts. Each fact must state its project and confidence and must carry a contiguous verbatim quote as evidence. The system then checks, word for word, that the quote really appears in the original conversation, allowing only formatting differences such as whitespace, full- or half-width characters and quotation marks. A fact without a source is dropped. This step is close in spirit to Mem0’s use of an LLM to distill facts from conversations [14], except that here verifiable verbatim evidence is required.

The third step is automatic review. Each candidate first passes a deterministic pre-screen that calls no model, then hybrid retrieval finds the 12 most similar existing cards, and finally an LLM judge decides in batches. The judge has four possible decisions: promote, to admit it as a new card; merge, to fold it into an existing card’s evidence; supersede, to replace an old card; and reject, to discard it. Supersede has the highest bar: it needs a newer, stronger, verifiable source, or something I said explicitly myself.

The fourth step is writing. The write path only appends new cards and never rewrites old files. A superseded card stays in the repository, the new card points to it through its supersedes field, and the old card is filtered out at retrieval time. Figure 3 shows exactly such a correction: I first said “mark unverified entries in amber” and later changed it to “use teal instead”; the system judged it a supersede, and the old card stays on disk but is never recalled again.

I read all 38 cards that automatic review promoted after launch. Thirty-one were good, for example a license constraint for one project, the fact that a certain test set may only be used for development diagnostics, and the finding that one experimental result was an artifact of the evaluation format. Six should not have entered memory: a piece of generic database knowledge, a timestamped incident timeline, a command-repair log, a Python version snapshot, a record of creating a handoff document, and a one-off task instruction. I added these six error types to the judge’s rejection rules and replaced the six cards with archived ones.

5. An evaluation environment: measuring memory in real use

5.1 What existing benchmarks measure

The most widely used benchmark for long-term memory is LoCoMo [29]. It first has LLM agents generate long conversations between two people from personas and event graphs, averaging about 300 turns and spanning up to 35 sessions; after human correction, it uses third-person question answering to test whether a model can answer a fact from the conversation. Mem0’s main experiments are built on LoCoMo [14]. LongMemEval has 500 questions covering five abilities, namely information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention, and finds that long-context models lose 30% to 60% of their performance on it [30]. MemoryAgentBench turns evaluation into incremental multi-turn interaction, testing accurate retrieval, test-time learning, long-range understanding and selective forgetting [31].

These benchmarks have driven a lot of progress, and LongMemEval already covers knowledge updates and abstention. They share one setup, though: the conversations are generated, the questions are written afterwards, and the answer is guaranteed to exist in the history when the question is written. The question I care about in practice is different. When I say my next sentence to an agent, the memory system first has to decide whether this moment needs memory. If it does, it should give the right few items. If it does not, it should stay quiet. This decision happens on every turn, and existing benchmarks can hardly measure it, because nearly every one of their questions needs memory.

Comparisons between memory systems are also fragile. Zep first reported 84% on LoCoMo; the Mem0 team pointed out a calculation error and put the correct figure at 58.44%; Zep’s corrected report was 75.14% [32, 33]. The same system on the same benchmark differed by more than twenty percentage points because of evaluation details alone. This reminded me that, in my own evaluation, the baselines must be connected in a way as close as possible to how they are meant to be used.

5.2 Building a memory environment from real use

With these considerations in mind, I built an evaluation environment from my own real agent usage. The data come from my private sessions with Hermes, Codex and Pi between September 21 and October 26, 2026. Of the 500 frozen probes, 363 come from Hermes, 89 from Codex and 48 from Pi. The data are split by time: the first two weeks form the development set, used for tuning and choosing thresholds, and weeks 3 to 5 form the test set.

Each point to be tested is called a probe, and each probe is a sentence I actually said in a real session. When answering a probe, a system can see only the previous turn and the earlier memory snapshot. What I actually said next and what the agent actually did, up to 6 turns, are shown only to the judge and never enter any system’s retrieval request. Memory snapshots come from the Git history of the memory repository. I take one snapshot per week, strictly earlier than every probe in that week, so no system can read information from the future.

5.3 Labels and metrics

To build sound metrics, I wrote the rubrics by hand, and an LLM assigns the labels under each rubric. To reduce the biases of LLM-as-a-Judge, the judge sees neither system names nor retrieval scores, candidate items are shuffled, and every judgement must cite a specific passage from the candidate text or the later conversation.

For each probe the judge decides two things. First, given what I actually said next and what the agent actually did, whether this moment needed memory. Second, which category each injected memory falls into: necessary (the task goes wrong without it), helpful, irrelevant, or harmful (it misleads the agent). The judge also checks, when I had changed my mind, whether the system gave the new version or the old one. To check label reliability, I had a model from a different vendor independently re-judge a random 20% of probes and computed the agreement between the models.

From these labels I track several metrics. Hit rate is the share of turns with at least one useful (necessary or helpful) memory injected; it measures how often the system actually helped. Useful rate and harmful rate are the shares of all injected items that are useful and harmful; they measure the quality of what is injected. There is also the share of memory-needing turns that received useful content, and the length of injected content. The composite score puts speaking and staying silent together: on turns that need memory, recalling a necessary item scores; on turns that do not, staying silent scores; injecting harmful content is penalized. All confidence intervals come from 2,000 bootstrap resamples clustered by session, because turns within one session are correlated and treating them as independent samples would understate uncertainty [34].

5.4 Baselines and fair integration

There are four baselines: OpenViking [17], mem0 [14], dense vector retrieval alone [26] and BM25 retrieval alone [27]. The last two are the most basic baselines in retrieval and show whether a complex system really beats simple methods. All systems read the same memory snapshot, use the same embedding model (text-embedding-3-small), and are scored by the same judge. Each system tunes a single threshold on the development set, and the test set is used only for the final report.

Learning from the LoCoMo dispute, I connected each system the way it is meant to be used. OpenViking uses the native interface (context assembly) that plugins such as OpenClaw’s call for automatic injection. mem0 uses its own write path (infer=True): its LLM distills facts from each document, which are written week by week in order, so each probe can only retrieve memories that already existed at that time.

This proved very important in the experiment. In my first comparison, OpenViking used the generic retrieval interface, which mostly returned directory overviews with metadata, and mem0 stored the original text directly without distillation. In that round, the useful rate of their injected content was only 4.9% and 5.9%. After switching to native usage, it rose to 16.5% and 19.0% respectively, more than three times higher.

5.5 Results

Figure 4. What each system injects from the same memory under the same recall budget. A is the quality of injected content; B is how much each system says. The test set is 467 judged real conversation turns across 83 sessions. Thin lines are session-clustered 95% confidence intervals.

The test set ended up with 467 judged turns across 83 sessions. Table 1 gives the results of the five systems under the same injection budget.

Under the same recall budget, this system’s hit rate is 43.9%, 12 to 15 percentage points higher than mem0, BM25 and dense retrieval, and 34 points higher than OpenViking. Its harmful rate is 2.0%, significantly lower than mem0, dense retrieval and BM25, about half of theirs, and comparable to OpenViking. On the 65 turns that really needed memory, this system and mem0 gave useful content at similar rates, 56.9% and 53.8%, both higher than the other three systems.

On the other hand, the accuracy of individual memories does not differ significantly between systems: useful rates all lie between 16% and 21%. So this system’s advantage does not come from picking individual memories better. It comes from two places. One is spending the budget more economically. This system injects only a one-line title per memory plus the part of the overview beyond the title, so the same 1,200 bytes hold 3.2 items on average while the other systems hold only 2 to 2.7. The other is filtering. Filtering by project scope and by version chain means this system injects less content from other projects and fewer superseded statements, which shows most clearly in the harmful rate. OpenViking’s harmful rate is also low, but mainly because it injects nothing on 59% of turns, which also gives it the lowest hit rate.

If each system is allowed to tune its own best threshold on the development set, the picture changes completely. Every system becomes almost silent, injecting memory on only 1.5% to 5.3% of turns, and composite scores bunch between 0.81 and 0.85, with BM25 highest and this system at 0.84. On the 65 memory-needing turns, no system recalled an item labelled “necessary”. The composite scores are so close because 86% of turns need no memory at all, so a system that never speaks still scores well. I therefore do not claim this system beats the baselines on every count. The more accurate statement is that, when a system is willing to inject some dose, this one lets more turns benefit at lower risk.

5.6 Findings about real use

An extra insight from this evaluation is that moments that really need memory are rare. Of the 467 test turns, the judge found only 65, or 14%, needed memory. Looking closer at those 65 turns, I found that for 42 of them (about two thirds) the information needed was not among any system’s candidates. The largest group, 21 turns, were follow-up questions about tasks in progress, such as “is it done yet?”; these need the live state of a task, which is never in long-term memory. The rest were details that had never been written to memory, such as the link to a PR or a specific numeric limit. Even pooling all five systems’ candidates, 77.7% of turns contain at least one useful item, so retrieval itself is not the bottleneck. This result says the next thing worth adding is a record of task state; further work on retrieval has small marginal returns. It also contrasts with the setup of benchmarks such as LongMemEval, where every question is guaranteed to have its answer in the history [30].

5.7 Limitations

This evaluation has several clear limitations to keep in mind when reading the results. First, the data come from a single user, me, and this system happened to be developed on my data. Other people’s usage habits may be completely different.

Second, the evaluation relies only on LLM judges. On the 90 re-judged turns, item labels agreed 86% of the time (Cohen’s κ = 0.41), and whether a turn needed memory agreed 69% of the time (κ = 0.17). By Landis and Koch’s common scale, the former sits at the low end of moderate agreement and the latter is only slight agreement [35]. Numbers related to whether memory was needed are therefore noisy and should be read with care.

Third, every system read the same already-curated memory, so this evaluation did not compare each system’s ability to build memory from raw chat on its own.

Finally, the raw data involve personal privacy and will not be released. I release only the method and anonymized examples.

6. Closing

After this round of work, my understanding of memory systems has changed. I used to think the hard part was remembering more. Now it looks harder to give the right agent the right dose at the right moment, to stay quiet when nothing should be said, and to grow and correct itself when no one is watching.

There are several questions I cannot yet answer. First, profiles are tuned by hand today. Could they be learned from use? For example, every time a coding agent calls memory_search on its own, that can be read as a signal that something it should have received was not given, and used to adjust triggers and thresholds. Second, how should this kind of evaluation on real usage records work with many users? One person’s data is limited evidence, yet real records naturally involve privacy. Third, two thirds of the memory-needing turns were missing task state. Should that information be taken over by the memory system or handed to a separate task system? I have not worked that out yet.

The build specs and the evaluation kit for this post are open source: https://github.com/jxtse/dose-mem

Cite this post

引用本文

Please cite this post as:

如需引用本文，请使用以下格式：

@article{xie2026memory,
  title = {One Memory, Many Agents: A Shared Memory Layer with Per-Agent Recall},
  author = {Jinxiang Xie},
  journal = {jxtse.github.io},
  year = {2026},
  month = {October},
  url = "https://jxtse.github.io/blog/one-memory-many-agents/#english"
}

参考文献

[1] Meta. How to get started with Muse. Meta Help Center, 2026. https://www.meta.com/help/superintelligence/1331373868832401/

[2] Introducing Grok Bot, 2026. https://x.ai/news/introducing-grok-bot

[3] OpenClaw: Your assistant, on your devices, in your chats. https://github.com/openclaw/openclaw

[4] Nous Research. Hermes Agent. https://github.com/nousresearch/hermes-agent

[5] Pi: a minimal agent harness. https://pi.dev/

[6] Anthropic. How Claude remembers your project. Claude Code Docs. https://code.claude.com/docs/en/memory

[7] OpenAI. Memories. ChatGPT Learn. https://learn.chatgpt.com/docs/customization/memories

[8] OpenClaw. Memory overview. https://docs.openclaw.ai/concepts/memory/

[9] Nous Research. Persistent Memory. Hermes Agent Docs. https://hermes-agent.nousresearch.com/docs/user-guide/features/memory

[10] Tailscale. Tailscale Serve. https://tailscale.com/docs/features/tailscale-serve

[11] Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., Gonzalez, J. E. MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560, 2023.

[12] Park, J. S., O’Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., Bernstein, M. S. Generative Agents: Interactive Simulacra of Human Behavior. UIST 2023.

[13] Zhong, W., Guo, L., Gao, Q., Ye, H., Wang, Y. MemoryBank: Enhancing Large Language Models with Long-Term Memory. AAAI 2024.

[14] Chhikara, P., Khant, D., Aryan, S., Singh, T., Yadav, D. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv:2504.19413, 2025.

[15] Rasmussen, P., Paliychuk, P., Beauvais, T., Ryan, J., Chalef, D. Zep: A Temporal Knowledge Graph Architecture for Agent Memory. arXiv:2501.13956, 2025.

[16] Xu, W., Liang, Z., Mei, K., Gao, H., Tan, J., Zhang, Y. A-Mem: Agentic Memory for LLM Agents. arXiv:2502.12110, 2025.

[17] Volcengine. OpenViking: Context Database for AI Agents. https://github.com/volcengine/OpenViking

[18] Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P. Lost in the Middle: How Language Models Use Long Contexts. TACL 12:157–173, 2024.

[19] Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., Zhou, D. Large Language Models Can Be Easily Distracted by Irrelevant Context. ICML 2023.

[20] Hong, K., Troynikov, A., Huber, J. Context Rot: How Increasing Input Tokens Impacts LLM Performance. Chroma Research, 2025. https://research.trychroma.com/context-rot

[21] Anthropic. Effective context engineering for AI agents, 2025. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

[22] Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., Hajishirzi, H. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. ACL 2023.

[23] Asai, A., Wu, Z., Wang, Y., Sil, A., Hajishirzi, H. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. ICLR 2024.

[24] Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., Neubig, G. Active Retrieval Augmented Generation. EMNLP 2023.

[25] Horvitz, E. Principles of Mixed-Initiative User Interfaces. CHI 1999.

[26] Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., Yih, W. Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020.

[27] Robertson, S., Zaragoza, H. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval 3(4):333–389, 2009.

[28] Cormack, G. V., Clarke, C. L. A., Büttcher, S. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. SIGIR 2009.

[29] Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., Fang, Y. Evaluating Very Long-Term Conversational Memory of LLM Agents. ACL 2024.

[30] Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., Yu, D. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. ICLR 2025.

[31] Hu, Y., Wang, Y., McAuley, J. Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions. arXiv:2507.05257, 2025.

[32] Zep. Is Mem0 Really SOTA in Agent Memory? 2025. https://www.getzep.com/blog/lies-damn-lies-statistics-is-mem0-really-sota-in-agent-memory/

[33] Yadav, D. Revisiting Zep’s 84% LoCoMo Claim: Corrected Evaluation & 58.44% Accuracy. GitHub issue getzep/zep-papers#5, 2025.

[34] Field, C. A., Welsh, A. H. Bootstrapping Clustered Data. Journal of the Royal Statistical Society Series B 69(3):369–390, 2007.

[35] Landis, J. R., Koch, G. G. The Measurement of Observer Agreement for Categorical Data. Biometrics 33(1):159–174, 1977.

References

[1] Meta. How to get started with Muse. Meta Help Center, 2026. https://www.meta.com/help/superintelligence/1331373868832401/

[2] Introducing Grok Bot, 2026. https://x.ai/news/introducing-grok-bot

[3] OpenClaw: Your assistant, on your devices, in your chats. https://github.com/openclaw/openclaw

[4] Nous Research. Hermes Agent. https://github.com/nousresearch/hermes-agent

[5] Pi: a minimal agent harness. https://pi.dev/

[6] Anthropic. How Claude remembers your project. Claude Code Docs. https://code.claude.com/docs/en/memory

[7] OpenAI. Memories. ChatGPT Learn. https://learn.chatgpt.com/docs/customization/memories

[8] OpenClaw. Memory overview. https://docs.openclaw.ai/concepts/memory/

[9] Nous Research. Persistent Memory. Hermes Agent Docs. https://hermes-agent.nousresearch.com/docs/user-guide/features/memory

[10] Tailscale. Tailscale Serve. https://tailscale.com/docs/features/tailscale-serve

[11] Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., Gonzalez, J. E. MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560, 2023.

[12] Park, J. S., O’Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., Bernstein, M. S. Generative Agents: Interactive Simulacra of Human Behavior. UIST 2023.

[13] Zhong, W., Guo, L., Gao, Q., Ye, H., Wang, Y. MemoryBank: Enhancing Large Language Models with Long-Term Memory. AAAI 2024.

[14] Chhikara, P., Khant, D., Aryan, S., Singh, T., Yadav, D. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv:2504.19413, 2025.

[15] Rasmussen, P., Paliychuk, P., Beauvais, T., Ryan, J., Chalef, D. Zep: A Temporal Knowledge Graph Architecture for Agent Memory. arXiv:2501.13956, 2025.

[16] Xu, W., Liang, Z., Mei, K., Gao, H., Tan, J., Zhang, Y. A-Mem: Agentic Memory for LLM Agents. arXiv:2502.12110, 2025.

[17] Volcengine. OpenViking: Context Database for AI Agents. https://github.com/volcengine/OpenViking

[18] Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P. Lost in the Middle: How Language Models Use Long Contexts. TACL 12:157–173, 2024.

[19] Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., Zhou, D. Large Language Models Can Be Easily Distracted by Irrelevant Context. ICML 2023.

[20] Hong, K., Troynikov, A., Huber, J. Context Rot: How Increasing Input Tokens Impacts LLM Performance. Chroma Research, 2025. https://research.trychroma.com/context-rot

[21] Anthropic. Effective context engineering for AI agents, 2025. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

[22] Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., Hajishirzi, H. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. ACL 2023.

[23] Asai, A., Wu, Z., Wang, Y., Sil, A., Hajishirzi, H. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. ICLR 2024.

[24] Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., Neubig, G. Active Retrieval Augmented Generation. EMNLP 2023.

[25] Horvitz, E. Principles of Mixed-Initiative User Interfaces. CHI 1999.

[26] Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., Yih, W. Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020.

[27] Robertson, S., Zaragoza, H. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval 3(4):333–389, 2009.

[28] Cormack, G. V., Clarke, C. L. A., Büttcher, S. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. SIGIR 2009.

[29] Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., Fang, Y. Evaluating Very Long-Term Conversational Memory of LLM Agents. ACL 2024.

[30] Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., Yu, D. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. ICLR 2025.

[31] Hu, Y., Wang, Y., McAuley, J. Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions. arXiv:2507.05257, 2025.

[32] Zep. Is Mem0 Really SOTA in Agent Memory? 2025. https://www.getzep.com/blog/lies-damn-lies-statistics-is-mem0-really-sota-in-agent-memory/

[33] Yadav, D. Revisiting Zep’s 84% LoCoMo Claim: Corrected Evaluation & 58.44% Accuracy. GitHub issue getzep/zep-papers#5, 2025.

[34] Field, C. A., Welsh, A. H. Bootstrapping Clustered Data. Journal of the Royal Statistical Society Series B 69(3):369–390, 2007.

[35] Landis, J. R., Koch, G. G. The Measurement of Observer Agreement for Categorical Data. Biometrics 33(1):159–174, 1977.

集体智能的天花板在哪里

Where Collective Intelligence Hits Its Ceiling

谢锦翔（Jinxiang Xie） · 2026 年 9 月 9 日

当前有很多机构在探索 collective intelligence。我最近关注的技术路线主要有两条：训练模型如何组织其他模型，以及让多个模型共同接受训练。这篇文章整理我的调研和分析，也会随着后续实验持续更新。先把判断放在前面：增加 agent 的数量和差异，并不自动产生更强的集体能力。系统甚至可能无法利用成员已经给出的正确答案。

这里需要区分超过最强单模型的平均成绩、超过同预算独立采样的候选覆盖率，以及产生成员独立探索时无法得到的解。它们是不同的主张。对于只在固定候选池中选择答案的机制，候选覆盖率确实构成上限；但这个限制不能直接推广到交互推理、任务分解、工具反馈和训练后的泛化。因此，我讨论的是当前机制在哪里遇到瓶颈，而不是证明集体智能存在一个普遍不可突破的能力上限。

范式一：Orchestration

第一条路线是 orchestration（编排），代表是 Sakana AI 的 Fugu。思路是优化一个 coordinator，负责把任务拆解、路由给合适的 subagent（worker），再把结果汇总，对外封装成单一模型接口。它建立在 Trinity 和 Conductor 两篇 ICLR 2026 工作之上。技术报告里的 Fugu-Ultra 是更完整的形态：用 LLM 生成工作流，组织各个 subagent 的任务、状态和上下文。

产品侧也在使用类似的分工：Claude Code / Codex 的 subagent 机制，以及 Cursor 的 agent swarm。学界还有 AgentFlow、Lemon 等优化编排器的工作，把训练重点放在 planner 或 orchestrator 上，而不是同时更新所有 worker。我认为这种架构相比单 agent，最直观的两个收益是：

1. 上下文隔离。 Coordinator 掌握全局进度，worker 专注单个任务。Cursor 的解释是，swarm 的可扩展性可能更多来自上下文效率：模型可以专注一个具体任务，不必同时保留所有任务的细节，从而减轻上下文过载。

2. 并行加速。 分而治之可以降低可并行任务的整体延迟。但代价同样真实：各环节相互依赖时，一个错误可能向下游传播；agent 间的通信损失，也可能让系统低于单 agent。除此之外，异构模型的能力互补和成本路由也是潜在收益，不能只用前面两点概括。

这条路线的瓶颈，更准确地说是协调与验证。Fugu 的 benchmark 成绩可以超过池中任何单个成员，但任务是否被正确拆解、必要上下文是否被传递、相互冲突的结果是否被识别，仍然需要系统解决。如果 coordinator 不加验证地采信 subagent 的结果，系统可能在不知不觉中失败；如果提供的上下文不足，worker 则可能从一开始就误解任务。

Anthropic 的游戏实验提供了一个有意思的对照：让 agent swarm 花 12 小时协作开发开放世界游戏，尝试自由组队、预设角色和指定 CEO 三种组织方式，最终游戏质量都很差。不过，不同代际模型的协作行为并不相同。Sonnet 4.6 和 Opus 4.6 难以合并彼此的工作；Opus 4.8 和 Mythos Preview 更多通过维持文件所有权、减少直接合作来避开冲突；Sonnet 5 则同时保持了较高的代码共享和 PR 合并率。

这些结果说明，摆好组织结构并不足以获得协调，但也不能据此断言协调只能来自最强模型，或仅靠模型变强就会自然出现。Trinity 本身就允许把验证交给 Verifier worker：coordinator 不必独自完成所有深度验证，难点在于它能否可靠地组织验证、处理冲突并决定何时继续。对于长时间、高依赖的任务，这仍是我对第一条路线最关心的问题。

范式二：Multi-Agent RL

第二条路线是 multi-agent RL（多智能体强化学习）。以 Co-RL 为例，它让多个不共享参数的模型同时接受 RL 训练。每个模型先从自己的多次采样中得到投票伪答案，再用这个答案监督另一个模型；在多于两个模型的设置中，监督可以沿有向环传递。它的关键发现是，异构模型家族、模型规模与输入改写带来的 cohort diversity，可以减少相关错误、缓解 self-rewarding 中的训练坍缩。

同一大方向还有 MAPoRL 和 MAGRPO，但机制并不相同。MAPoRL 包含多模型讨论、verifier 评分以及鼓励纠错和说服的奖励；MAGRPO 则优化多轮协作。因此，不能把整个 multi-agent RL 都简化成“独立采样，再固定投票”。通信协议固定，也不意味着协议中的协作行为不能被学到。

对于 Co-RL 这类依赖采样伪标签的方法，采样覆盖和监督信号质量是直接瓶颈。在某次更新中，只有实际出现并进入伪标签的答案，才能提供相应的直接监督。增加 diversity 有机会减少共同错误、扩大候选覆盖，但投票本身不会构造候选之外的新解。这是单次采样与聚合的限制，不能直接当作训练后泛化能力的定理。

Self-rewarding 的风险也需要说得更具体：当某个错误答案稳定占据众数时，用它作为伪标签可能继续强化错误。正确答案的采样概率低于一半，并不一定意味着最高票答案错误，因为其他错误可能分散到许多不同答案上。跨模型监督的价值，在于不同模型未必共享同一个错误众数；如果它们的错误高度相关，这个优势就会减弱。

一次离线投票审计

为了看清这个机制，我对 Co-RL 复现配置在训练起点的采样做了一次离线审计：8,740 道 MATH-Level345 训练题，三个模型分别是 Qwen2.5-3B、Llama-3.2-3B-Instruct 和 Qwen3-1.7B-Base，每题每模型采样 K=12。答案提取与判分沿用复现仓库的代码。

这里要区分实际训练和额外审计：实际三模型训练使用环形同伴监督；下面的跨模型投票是另一项离线分析。它先取每个模型自己的众数答案，再让每个模型投一票，选出唯一最高票答案；最高票并列时不接受标签。以下把它简称为跨模型投票。

全题正确率把没有被接受的标签计为未答对；标签覆盖率是产生被接受标签的题目比例；条件准确率只在这些题目上计算。三个指标必须一起看：投票提高了已接受标签的纯度，但放弃了部分题目，没有提高整体正确覆盖。

三个模型的模型内众数答案均错误的题目占 31.6%，跨模型投票救回率为零。这个零是结构性的：只从三个错误众数中选择，不可能得到正确答案；它不表示原始全部采样里都没有正确答案。模型间众数答案的错误相关性 φ 为 0.48–0.67，其中 Qwen2.5 与 Qwen3 这对同家族模型最高。这说明它们在当前条件下存在共同失败模式，不能直接解释为潜在知识边界的重合程度。

我还在固定的 2,000 题子样本上加入强模型的答案，观察等权投票能否保留高质量意见。小模型仍然各自提供一个众数答案；每个强模型提供一个 greedy 答案，仍然每模型一票，并列时不接受标签。

这组结果不能与上表全量训练集直接作增益比较，也不是同预算的模型能力排名。它考察的是给定答案池上的聚合行为。加入两个强模型后，已接受标签的准确率很高，但全题正确率仍低于这个条件指标。两个强模型在双方答案都能解析的题目上有 92.18% 的精确答案一致率，因此结果更接近强模型之间的一致意见获得了更大的投票影响力，而不是某个最强成员自动主导集体判断。

这次审计支持的判断是：简单等权投票会在标签纯度与覆盖率之间取舍，也可能无法保留高质量的少数意见。 它没有验证 Co-RL 训练后的能力上限，更不能证明整个 multi-agent RL 没有纠错能力。把 Co-RL 理解为跨模型伪标签监督、互相传递可用能力，是一个有帮助的视角，但它仍需与实际的环形训练机制对应起来。

群体智能的瓶颈及未来探索方向

我仍然认为，增大模型采样中有用的 diversity，再想办法利用这些差异，是群体智能值得继续探索的方向。

从采样数据分布的角度

Diversity 这一半有据可依。Co-RL 的消融支持异构 family 和输入改写减少相关错误、缓解训练坍缩。Anthropic 的实验则从反面展示了同源 agent 的低方差：30 个同模型 agent 协作开发游戏，18 个创建了同名 git 分支 mvp-game-loop；管理有限带宽任务队列时，agent 部署高频轮询，一次运行产生 240 万个请求，却只有 117 个任务被接受。个体层面的相似倾向，在集体中可能被放大成系统性失效。

困难在于，后训练常常会使输出分布变尖、降低多样性。DivPO 直接优化质量约束下的响应多样性；DPEPO 通过动作和状态转移奖励鼓励多样的并行探索；EvoDiverse 则在推理时的科学假设搜索中借鉴 parallel tempering，缓解搜索分布的坍缩。这些工作处理的对象不同，但都提示：只保留单一高分模式可能损失有价值的候选。调高 temperature 可以改变采样分布，却不能保证得到高质量、语义上互补的解。

顺着这个问题，我更关心的是：diversity 应该定义在什么对象上？Knowledge boundary 的工作提供了一个有用的概念：固定问法下答错，不代表模型在语义等价的其他问法下也答不对。因此，单次采样无法代表模型关于这个问题的全部可达能力。不过，有限的改写搜索也无法证明已经穷尽了这个范围。

对 multi-agent 来说，我想要的并非只是几个模型这次恰好答得不同，而是在保持质量的前提下，减少共同盲区、扩大互补覆盖。共享正确的基础知识是有价值的；单纯让 knowledge boundary 尽可能少重合，反而可能通过遗忘知识或制造错误实现。

这里可以借 nrehiew 那篇从分布视角看 post-training 的 blog 来组织直觉：SFT 通过外部数据提供目标，在线 RL 则通过当前策略访问到的轨迹接受反馈。前者可以示范当前模型难以采到的行为；后者的探索受当前采样分布影响。这个框架并不是严格的能力边界证明。若希望 SFT 把几个模型的分布拉开，还需要不同的高质量目标数据或显式约束；RL 也可以通过质量与多样性目标改变探索方向。

我期待未来 multi-agent system 中的各个 agent 拥有更异构、又不牺牲基本能力的分布，能提供真正互补的视角，而不只是换一个角色提示词。

从 fusion 的角度

Fusion 才是更难的一半。普通答案投票需要可判等的候选，大量开放式任务和长时间轨迹很难直接适用。When Agents Disagree 把这个问题称为 selection bottleneck：在其建模和单轮选择实验中，diverse pool 的收益取决于 selector 的质量。这个结果不应被直接推广成所有融合机制的定律，但它提出了一个好问题：如果不能识别差异中的价值，更多候选究竟是在提供机会，还是增加噪声？

一旦需要一个 learned fusion 模块读懂多个候选、判断每个候选对在哪里，再把有用部分合起来，它就开始接近第一条路线里的 coordinator。但选择一个完整候选、综合多个方案和协调长期执行，仍然是不同任务，未必应由单一模块承担。

Cursor 的 SQLite 实验提供了量化的失败记录：旧版 swarm 在前两小时产生 68,000 次 commit，累计超过 70,000 次 merge conflict，接近两小时时被暂停；冲突最严重的文件被 1,173 个 agent 改动，发生 7,771 次冲突；项目扩张成 54 个 crate，其中有三个独立的 SQL package。

更重要的是改进后的对照：同模型条件下，引入中立仲裁 agent、共享设计文档、超大文件标记等 harness 机制，显著减少了冲突。这不是模型自己通过训练学会协调的证据，却是机制设计可以改善集体表现的证据。环境和协议本来就是协作系统的一部分。

也可以通过为多个 agent 分配信息差来构造 diversity，而聚合这些信息并不容易。Anthropic 的 hidden profile 实验把事实分散给四个 agent：共享证据支持错误选项，独家信息则可以改变决策。讨论后的集体表现仍未达到单 agent 获得全部事实的 solo ceiling。这个设置说明，信息分布式存在，不等于系统能有效使用它。识别私有事实的重要性、表达它、以及让其他成员据此更新判断，都是需要解决的能力问题。

我期待更可靠的协调机制能从多样的轨迹中提取重要信息，并按照全局目标稳步推进任务。它可能来自更强的模型，也可能来自更好的验证、协议、环境反馈和训练目标。两条路线会在这里交汇。对我来说，接下来最值得追问的是：怎样制造有价值的差异，又怎样让这些差异在形成集体判断时不被抹掉？

Read in English ↓

Where Collective Intelligence Hits Its Ceiling

Jinxiang Xie · September 9, 2026

Many organizations are exploring collective intelligence. Two directions have occupied most of my recent reading: training a model to organize other models, and training multiple models together. This post collects my observations and analysis, which I expect to update as the experiments develop. My current view is straightforward: adding more agents, or making them more different, does not automatically produce stronger collective capability. A system may fail to use even the correct answers its members already provide.

We should distinguish outperforming the best individual model on average, exceeding the candidate coverage of compute-matched independent sampling, and producing solutions that members cannot reach through independent exploration. These are different claims. For a mechanism that only selects from a fixed candidate pool, candidate coverage is indeed an upper bound. That restriction does not directly extend to interactive reasoning, task decomposition, tool feedback, or generalization after training. I am interested in where current mechanisms encounter bottlenecks, not in proving a universal impossibility result for collective intelligence.

Paradigm One: Orchestration

The first direction is orchestration, exemplified by Sakana AI’s Fugu. The idea is to optimize a coordinator that decomposes tasks, routes them to suitable subagents or workers, and combines their outputs, exposing the system through a single model interface. Fugu builds on two ICLR 2026 papers, Trinity and Conductor. Fugu-Ultra, described in the technical report, is a more developed version: an LLM generates workflows and organizes subagent tasks, state, and context.

Similar divisions of labor appear in products: the subagent mechanisms in Claude Code and Codex, and Cursor’s agent swarm. Research systems such as AgentFlow and Lemon also focus optimization on a planner or orchestrator rather than jointly updating every worker. Compared with a single agent, I see two especially immediate benefits:

1. Context isolation. The coordinator tracks global progress while each worker focuses on a specific task. Cursor suggests that swarm scalability may owe more to context efficiency than to parallelism: a model can focus on one piece of work without retaining the details of every other task, reducing context overload.

2. Parallel speedup. Divide-and-conquer can reduce latency for parallelizable work. But the costs are real too. When stages depend on one another, an error can propagate downstream, and information lost in communication can leave the system worse than a single agent. These are not the only possible benefits: complementary model capabilities and cost-aware routing also matter.

The bottleneck here is more accurately described as coordination and verification. Fugu can outperform any individual member of its pool on benchmarks, but the system still has to decompose tasks correctly, transmit necessary context, and recognize conflicting results. If a coordinator accepts a subagent’s report without verification, the system may fail silently. If it supplies too little context, a worker may misunderstand the task from the outset.

Anthropic’s game-development experiment offers an interesting comparison. Agent swarms spent 12 hours building open-world games under three organizational prompts: freely forming teams, prescribed roles, and a designated CEO. The resulting games were poor in all three conditions. Yet coordination differed across model generations. Sonnet 4.6 and Opus 4.6 struggled to merge each other’s work. Opus 4.8 and Mythos Preview largely avoided conflicts by retaining ownership of files and collaborating less directly. Sonnet 5 maintained both relatively high code sharing and a high PR merge rate.

These results show that installing an organizational structure is insufficient. They do not establish that coordination can only come from the strongest models, or that stronger models will acquire it automatically. Trinity itself can delegate verification to a Verifier worker. A coordinator need not perform every difficult verification step personally; the challenge is organizing verification reliably, resolving conflicts, and deciding when to proceed. For long-running tasks with substantial dependencies, this remains my main concern about orchestration.

Paradigm Two: Multi-Agent RL

The second direction is multi-agent reinforcement learning. In Co-RL, for example, several models with separate parameters undergo RL simultaneously. Each model first obtains a voted pseudo-answer from its own samples, then uses that answer to supervise another model. With more than two models, supervision can pass along a directed ring. A central finding is that cohort diversity, introduced through heterogeneous model families, sizes, and rephrased inputs, can reduce correlated errors and mitigate the training collapse associated with self-rewarding.

MAPoRL and MAGRPO belong to the broader direction, but their mechanisms differ. MAPoRL includes discussion, verifier scoring, and incentives for correction and persuasion; MAGRPO optimizes multi-turn collaboration. It would therefore be misleading to reduce all multi-agent RL to independent sampling followed by fixed voting. A fixed communication protocol does not imply that the collaborative behavior inside it cannot be learned.

For methods that depend on sampled pseudo-labels, such as Co-RL, sampling coverage and supervision quality are immediate bottlenecks. At a given update, an answer must actually appear and enter the pseudo-label to provide the corresponding direct supervision. Diversity may reduce shared mistakes and expand coverage, but voting itself does not construct a solution outside the candidate pool. That is a restriction on a particular sampling-and-aggregation step, not a theorem about post-training generalization.

The risk in self-rewarding also deserves precision. When a wrong answer is a stable mode of the answer distribution, treating it as a pseudo-label can reinforce the mistake. A correct-answer probability below one half does not necessarily imply that the most-voted answer is wrong: incorrect answers may be distributed across many alternatives. Cross-model supervision is useful when models do not share the same wrong mode. Highly correlated errors weaken that advantage.

An Offline Voting Audit

To examine the mechanism, I audited samples from a Co-RL replication configuration at the start of training: 8,740 MATH-Level345 training questions, three models (Qwen2.5-3B, Llama-3.2-3B-Instruct, and Qwen3-1.7B-Base), and K=12 samples per model per question. Answer extraction and grading followed the replication repository’s code.

The actual training mechanism and this additional audit must be separated. The three-model training setup uses ring-based peer supervision. The cross-model vote below is a separate offline analysis: first obtain each model’s modal answer, then give each model one vote and select the unique plurality winner. Ties are rejected. I refer to this as cross-model voting below.

All-question accuracy counts rejected labels as unsuccessful answers. Label coverage is the fraction of questions receiving an accepted label. Conditional accuracy is computed only over those questions. These metrics need to be read together: voting improved the purity of accepted labels but gave up coverage, without increasing the overall fraction of questions correctly answered.

On 31.6% of questions, all three within-model modal answers were wrong. Cross-model voting rescued none of them. That zero is structural: selecting among three wrong modal answers cannot produce a correct one. It does not mean that no correct answer existed among all the original samples. Error correlations, measured by φ on modal-answer correctness, ranged from 0.48 to 0.67, with the same-family Qwen2.5–Qwen3 pair highest. This establishes shared failure patterns under the tested conditions, not the overlap of the models’ latent knowledge boundaries.

I also added strong-model answers on a fixed 2,000-question subsample to examine whether equal-weight voting could preserve high-quality opinions. Each small model still supplied one modal answer, while each strong model supplied one greedy answer. Every model received one vote, and ties were rejected.

These results should not be compared directly with the full-training-set table to calculate gains, nor are they a compute-matched model ranking. They probe aggregation over a given answer pool. With both strong models, accepted labels were highly accurate, but all-question accuracy remained below that conditional figure. The two strong models had 92.18% exact-answer agreement on questions where both answers parsed successfully. The result is therefore closer to agreement between two strong models gaining voting influence than to the strongest individual automatically dominating collective judgment.

What this audit supports is that simple equal-weight voting trades label purity against coverage and may fail to preserve high-quality minority opinions. It does not test Co-RL’s post-training capability ceiling, much less establish that multi-agent RL cannot correct errors. Thinking of Co-RL as cross-model pseudo-label supervision that transfers useful capabilities between peers is helpful, but the interpretation must remain faithful to its actual ring-based training mechanism.

Bottlenecks and Directions Worth Exploring

I still think that increasing useful diversity in model samples, then finding ways to exploit those differences, is a promising direction for collective intelligence.

The Sampling-Distribution Perspective

There is evidence for the diversity half. Co-RL’s ablations support heterogeneous families and input rephrasing as ways to reduce correlated errors and mitigate collapse. Anthropic’s experiments illustrate the opposite problem: low variance among agents built on the same model. In a game-development run, 18 of 30 agents created a branch named mvp-game-loop. In a bandwidth-limited job-queue experiment, agents deployed high-frequency polling; one run produced 2.4 million requests but only 117 accepted jobs. Similar individual tendencies can become systemic failures when amplified across a group.

One difficulty is that post-training often sharpens the output distribution and reduces diversity. DivPO directly optimizes response diversity subject to quality constraints. DPEPO encourages diverse parallel exploration through action and state-transition rewards. EvoDiverse draws on parallel tempering to counter collapse during inference-time scientific hypothesis search. These methods operate on different objects, but all suggest that retaining only a single high-scoring mode can discard valuable candidates. Raising temperature changes the sampling distribution; it does not guarantee high-quality, semantically complementary solutions.

This leads to a more basic question: what should diversity be defined over? Work on knowledge boundaries offers a useful concept. A model answering incorrectly under one wording may answer correctly under a semantically equivalent prompt. A single sample therefore cannot represent everything the model can reach on that question. Conversely, a finite paraphrase search cannot prove that this range has been exhausted.

For multi-agent systems, I want more than models happening to answer differently on a particular draw. I want to reduce shared blind spots and expand complementary coverage while preserving quality. Shared correct foundational knowledge is useful. Minimizing knowledge-boundary overlap indiscriminately could instead be achieved by forgetting knowledge or introducing errors.

I find nrehiew’s distributional view of post-training helpful for organizing this intuition. SFT supplies a target through external data, whereas online RL receives feedback on trajectories visited by the current policy. The former can demonstrate behavior the current model rarely samples; the latter’s exploration depends on its current sampling distribution. This is not a rigorous capability-boundary argument. To separate several models through SFT, we still need distinct high-quality target data or explicit constraints. RL can also reshape exploration through quality and diversity objectives.

I would like agents in future multi-agent systems to have more heterogeneous distributions without sacrificing basic competence, providing genuinely complementary perspectives rather than merely receiving different role prompts.

The Fusion Perspective

Fusion is the harder half. Ordinary answer voting requires candidates that can be tested for equivalence, which makes it difficult to apply directly to many open-ended tasks and long-horizon trajectories. When Agents Disagree calls this the selection bottleneck: in its model and single-round selection experiments, the benefit of a diverse pool depends on selector quality. This should not be generalized into a law for all fusion mechanisms, but it raises a useful question. If a system cannot recognize the value in differences, do more candidates provide opportunities or merely add noise?

Once a learned fusion module needs to read multiple candidates, identify what each gets right, and combine useful parts, it begins to resemble the coordinator in the first paradigm. Still, selecting a complete candidate, synthesizing several proposals, and coordinating long-running execution are different tasks. They need not all belong to a single module.

Cursor’s SQLite experiment provides quantitative failure records. The old swarm generated 68,000 commits during its first two hours and accumulated more than 70,000 merge conflicts before being paused near the two-hour mark. The most contested file was modified by 1,173 agents and incurred 7,771 conflicts. The project expanded to 54 crates, including three independent SQL packages.

The improved comparison matters more. With the same models, harness changes including neutral arbitration agents, shared design documents, and oversized-file flags substantially reduced conflicts. This is not evidence that models learned coordination through training. It is evidence that mechanism design can improve collective performance. Environments and protocols are part of a collaborative system.

Diversity can also be introduced by distributing different information to different agents, but aggregating it is difficult. In Anthropic’s hidden-profile experiment, facts were distributed across four agents: shared evidence favored the wrong option, while unique information could change the decision. After discussion, collective performance still fell short of the solo ceiling, where one agent received every fact. Distributed possession of information does not guarantee effective use. Recognizing the importance of private evidence, communicating it, and getting other members to update their judgments are all capabilities that need to be addressed.

I hope more reliable coordination mechanisms can extract important information from diverse trajectories and steadily advance work toward a global objective. They may come from stronger models, but also from better verification, protocols, environmental feedback, and training objectives. The two directions meet here. The question I most want to pursue is how to create valuable differences, then prevent them from being erased as a group arrives at a collective judgment.

返回中文 / Back to Chinese ↑

Cite this post

引用本文

Please cite this post as:

如需引用本文，请使用以下格式：

@article{xie2026collective,
  title = {Where Collective Intelligence Hits Its Ceiling},
  author = {Jinxiang Xie},
  journal = {jxtse.github.io},
  year = {2026},
  month = {September},
  url = "https://jxtse.github.io/blog/where-collective-intelligence-hits-its-ceiling/"
}

Cite this post

Please cite this post as:

@article{xie2026consolidation,
  title = {Consolidating an Agent That Outgrew Itself},
  author = {Jinxiang Xie},
  journal = {jxtse.github.io},
  year = {2026},
  month = {August},
  url = "https://jxtse.github.io/blog/agent-consolidation/"
}

引用本文

如需引用本文，请使用以下格式：

@article{xie2026grounded,
  title = {Staying Grounded in Fast-Paced AI Research},
  author = {Jinxiang Xie},
  journal = {jxtse.github.io},
  year = {2026},
  month = {June},
  url = "https://jxtse.github.io/blog/how-to-cope-with-fast-paced-research/"
}

引用本文

如需引用本文，请使用以下格式：

@article{xie2025claudecode,
  title = {Claude Code Skills and Subagents in Practice},
  author = {Jinxiang Xie},
  journal = {jxtse.github.io},
  year = {2025},
  month = {November},
  url = "https://jxtse.github.io/blog/claude-code-skills-subagents/"
}