From 9fe18104977232e91859cd6740d7fcb6997c4ba1 Mon Sep 17 00:00:00 2001 From: Chiguyong Date: Mon, 6 Jul 2026 13:12:34 +0800 Subject: [PATCH 01/10] docs(plan): add agent-iq-boost implementation-ready plan (R1-R15, U1-U7) --- ...2026-07-06-001-feat-agent-iq-boost-plan.md | 502 ++++++++++++++++++ 1 file changed, 502 insertions(+) create mode 100644 docs/plans/2026-07-06-001-feat-agent-iq-boost-plan.md diff --git a/docs/plans/2026-07-06-001-feat-agent-iq-boost-plan.md b/docs/plans/2026-07-06-001-feat-agent-iq-boost-plan.md new file mode 100644 index 0000000..55089ac --- /dev/null +++ b/docs/plans/2026-07-06-001-feat-agent-iq-boost-plan.md @@ -0,0 +1,502 @@ +--- +date: 2026-07-06 +topic: agent-iq-boost +type: feature +artifact_contract: ce-unified-plan/v1 +artifact_readiness: implementation-ready +product_contract_source: ce-brainstorm +execution: code +origin: "竞品对标(2026-07 Qoder 1.11/Cursor 3.2/Devin 3.0);9 缺口(G1-G9)已交付后的新维度探索" +--- + +## Goal Capsule + +**Objective**: 扩展现有 PLAN_EXEC / TEAM_COLLAB / ReflexionEngine 三个子系统，提升 agent 处理复杂任务的智商 — 独立子任务并行执行、plan 确认后自主执行（仅危险操作确认）、失败后 prompt 跨任务自调优。 + +**Product Authority**: AgentKit 横向评估 14 维度总分 42/56，9 缺口已交付但竞品在"编排+执行深度"维度持续领先（Cursor Subagents 并行、Devin Goal-driven 自主、Qoder 多模型路由）。本 plan 对标竞品补齐这一维度。 + +**Open Blockers**: None — OQ1-3 已在 Planning Contract KTD6-8 决策。 + +**Execution Profile**: Standard depth，7 个 Implementation Units，3 个维度可部分并行（维度 1 与维度 3 独立；维度 2 内部 U1→U2→U3 串行）。 + +**Stop Conditions**: 7 个 U-ID 全部完成、Verification Contract 通过、DoD 全局条件满足。 + +--- + +## Product Contract + +### Summary + +3 个维度并行推进，不新建 ExecutionMode，全部扩展现有基础设施：TeamOrchestrator 新增"无依赖子任务并行"模式解决串行排队；PLAN_EXEC 的"每阶段确认"改为"危险操作确认"解决要人盯；ReflexionEngine 反思结果持久化到 EpisodicMemory 实现跨任务 prompt 自调优。对标 Cursor Agent Mode / Devin Quest Mode / Qoder Goal-driven 的智商能力，保持 AgentKit 现有架构不变。 + +### Problem Frame + +2026-06-29 识别的 9 个智商短板（G1-G9）已全部交付（wave1-4），覆盖反馈稳定性、响应效率、执行能力三个维度。但 2026-07 最新竞品动态显示，竞品在"编排+执行深度"维度持续领先： + +- **Cursor 3.2 Subagents**（2026-04）：并行专业化 worker，多 agent 同时工作不同部分 +- **Devin 3.0 Goal-driven**（2026-05）：设定目标后自主执行数小时，仅失败时介入 +- **Qoder 1.11 Goal-driven + Scheduling**（2026-06）：Goal 设定后工作到完成，可定时启动 + +AgentKit 当前痛点： +- 5 个独立子任务必须串行排队（TeamOrchestrator 同层并行已存在，但"无依赖子任务并行"未实现） +- plan 生成后每阶段需人工确认（PLAN_EXEC 阶段确认机制太保守） +- ReflexionEngine 仅在单次任务内反思，同类错误跨任务重复犯（反思结果未持久化） + +### Key Decisions + +**KTD1: 方案 A — 扩展现有模式，不新建 ExecutionMode** + +3 个维度全部扩展现有 PLAN_EXEC / TEAM_COLLAB / ReflexionEngine，不新建 GOAL_DRIVEN 模式。理由：复用现有基础设施，路由/前端/配置不动，风险低；KTD7（06-29 G6 阶段约束）已验证扩展现有模式可行。 + +**KTD2: 危险操作保守白名单，不引入 LLM 辅助分类** + +自主执行时仅危险操作触发 confirmation_request，白名单包括：文件删除（rm/rmdir）、部署操作（deploy/kubectl/helm）、支付相关、git push --force、数据库迁移（alembic/migrate）。白名单外的操作自主执行。ponytail: ceiling = 漏判风险（未列入白名单的危险操作），升级路径 = LLM 辅助分类。 + +**KTD3: 复用 topological_sort 的同层并行，新增"无依赖子任务并行"模式** + +TeamOrchestrator 已有 topological_sort 返回执行层（Kahn 算法），同层并行已实现。新增"无依赖子任务并行"模式：Lead 分解任务时，若识别到多个无 depends_on 的子任务，自动派发给隔离 agent 并行执行，而非串行排队。不新建 SubAgentOrchestrator 模块（方案 C 排除）。 + +**KTD4: ReflexionEngine 反思结果持久化到 EpisodicMemory** + +失败后 ReflexionEngine 生成反思（已有），新增：反思结果写入 EpisodicMemory（task_input + reflection + improved_prompt），下次类似任务规划时 Lead 检索历史反思，用 improved_prompt 替换默认 prompt。不引入强化学习/元学习（Devin 风格太重），仅做跨任务持久化。 + +**KTD5: Prompt 自调优仅对特定错误类型触发** + +不是每次失败都触发 prompt 调优（避免无意义反思浪费 token）。触发条件：verify 失败（G1 回灌后二次失败）、工具 schema 校验失败（G3）、循环检测触发（U1）。调优后的 prompt 带 version 存入 EpisodicMemory，ABTester 可对比版本效果（离线验证，不在线 bandit）。 + +### Requirements + +**维度 1: 并行 Sub-agent 编排** + +- R1. TeamOrchestrator 的 Lead 分解任务时，识别无 depends_on 的子任务，自动派发给隔离 ConfigDrivenAgent 并行执行。 +- R2. 并行子任务的 SharedWorkspace 输出路径必须不重叠（plan 阶段强制约束：`{plan_id}/phase/{phase_id}/output` 已唯一，并行子任务用不同 phase_id）。 +- R3. 并行子任务全部完成后，Lead 才进入综合阶段（synthesis），与现有 topological_sort 的层间串行一致。 +- R4. 并行子任务数受 MAX_EXPERTS=10 约束（已有），超出时 Lead 重新分解或串行排队。 +- R5. 并行子任务的 expert_step / expert_result 事件携带 expert_id，前端已支持多 expert 同时 streaming（已验证）。 + +**维度 2: Goal-driven 自主执行** + +- R6. PLAN_EXEC 模式下，plan 确认后进入自主执行，不再每阶段 confirmation_request。 +- R7. 自主执行期间，仅危险操作（KTD2 白名单）触发 confirmation_request，其他操作自主执行。 +- R8. 危险操作白名单配置化（`agentkit.yaml` 新增 `dangerous_tools` 配置节，遵循 ServerConfig.from_dict 模式）。 +- R9. 自主执行期间，用户可随时发送 `cancel` 中断（已有 CancellationToken 机制）。 +- R10. 自主执行超时（默认 30 分钟，可配置）或连续失败 3 次时，自动暂停并通知用户。 + +**维度 3: Prompt 跨任务自调优** + +- R11. ReflexionEngine 反思结果写入 EpisodicMemory（task_input + reflection + improved_prompt + version）。 +- R12. PLAN_EXEC / TEAM_COLLAB 的 Lead 规划时，检索 EpisodicMemory 中相似 task_input 的历史反思，用 improved_prompt 替换默认 prompt。 +- R13. Prompt 自调优仅对特定错误类型触发（KTD5）：verify 二次失败、工具 schema 校验失败、循环检测触发。 +- R14. 调优后的 prompt 带 version 存入，ABTester 可对比版本效果（离线验证）。 +- R15. EpisodicMemory 中 prompt 反思记录有 TTL（默认 30 天），过期自动清理避免噪声。 + +### Scope Boundaries + +**In scope**: +- 扩展 PLAN_EXEC / TEAM_COLLAB / ReflexionEngine +- 危险操作白名单配置 +- EpisodicMemory prompt 反思持久化 + +**Out of scope**: +- 新建 ExecutionMode.GOAL_DRIVEN（方案 B 排除） +- 独立可组合模块 SubAgentOrchestrator / DangerGate / OnlinePromptOptimizer（方案 C 排除） +- 多模态输入（维度 6，不在"编排+执行深度"范围） +- Cloud Agent 远程执行（维度 8，不在本次范围） +- Repo 深度索引 + Wiki 自动生成（维度 4，不在本次范围） +- 主动澄清意图（维度 1，不在本次范围） +- 强化学习 / 元学习 / bandit 探索（Devin 风格在线学习太重，仅做 ReflexionEngine 跨任务持久化） +- 项目级 Memory Files（CLAUDE.md/AGENTS.md 风格，维度 2，不在本次范围） + +### Outstanding Questions + +All OQ1-3 resolved in Planning Contract (KTD6-KTD8). No outstanding questions remain. + +--- + +## Planning Contract + +### Key Technical Decisions (Planning-Time) + +**KTD6: 并行子任务 depends_on 完全独立约束（解 OQ1）** + +并行子任务的 depends_on 必须完全独立 — 不能共享上游依赖。若 Lead 识别到共享上游依赖，降级为同层并行（topological_sort 已有机制）。此外，关键写入操作用 `SharedWorkspace.lock(key, agent_id, timeout)` 防护（已有 API，`core/shared_workspace.py:96`）。决策依据：phase_id 唯一性已保证输出路径不重叠，但读上游输出时存在 TOCTOU 风险，lock 是最小成本防护。 + +**KTD7: Prompt 版本管理 — 保留所有版本 + score 排序（解 OQ2）** + +EpisodicMemory 保留所有反思版本（key = `prompt_reflection:{task_hash}:{version}`），每条记录携带 `score`（verify 通过=1.0，失败=0.0）+ `timestamp` + `task_input`。Lead 检索时用 `EpisodicMemory.search(task_input, top_k=3)` 语义搜索，按 score 降序取最高版本。TTL 30 天（R15）由 EpisodicMemory 现有清理机制处理。ABTester 离线对比版本效果，不在线 bandit。决策依据：保留所有版本成本低（Redis/PG 存储廉价），且支持回溯；score 排序避免噪声版本干扰。 + +**KTD8: autonomy_paused 事件类型（解 OQ3）** + +新增 `autonomy_paused` WebSocket 事件类型（区别于 `confirmation_request`），前端通过事件类型区分"等待确认"和"自主暂停"。payload 包含 `reason`（timeout | consecutive_failures | manual）+ `progress`（已完成阶段）+ `resume_token`。用户可发送 `resume` 消息继续执行（带 resume_token）。决策依据：复用现有 WebSocket 协议，前端只需新增一个事件分支，不破坏现有 confirmation 流程。 + +### High-Level Technical Design + +```mermaid +flowchart TB + User[User Input] --> Router{RequestPreprocessor} + Router -->|@team| TeamOrchestrator + Router -->|plan_exec| PLAN_EXEC[PLAN_EXEC Handler] + Router -->|react| ReActEngine + + subgraph "维度 1: 并行 Sub-agent" + TeamOrchestrator --> Lead[Lead Expert] + Lead --> Decompose[_decompose_task] + Decompose --> Plan[TeamPlan] + Plan --> TopoSort[topological_sort] + TopoSort -->|同层并行| ParallelExec[_run_pipeline parallel] + ParallelExec --> Agent1[ConfigDrivenAgent 1] + ParallelExec --> Agent2[ConfigDrivenAgent 2] + Agent1 --> SharedWorkspace[(SharedWorkspace)] + Agent2 --> SharedWorkspace + SharedWorkspace -.->|lock 防护| LockMech[lock/unlock] + end + + subgraph "维度 2: Goal-driven 自主" + PLAN_EXEC --> PlanConfirm[Plan Confirmation] + PlanConfirm -->|确认后| AutonomyMode[Autonomy Mode] + AutonomyMode --> ToolExec[Tool Execution] + ToolExec --> DangerCheck{Dangerous?} + DangerCheck -->|是| ConfirmReq[confirmation_request] + DangerCheck -->|否| Continue[Continue] + ConfirmReq --> AutonomyResume[Autonomy Resume] + AutonomyMode -.->|超时/失败| Paused[autonomy_paused] + end + + subgraph "维度 3: Prompt 自调优" + ReActEngine -->|失败| ReflexionEngine + ReflexionEngine --> Reflect[_reflect] + Reflect --> EpisodicMem[(EpisodicMemory)] + EpisodicMem -.->|下次任务| LeadRetrieval[Lead 检索历史反思] + LeadRetrieval --> PromptReplace[替换默认 prompt] + end +``` + +### Assumptions + +- SharedWorkspace.lock() 在 Redis 后端下可靠（已有 SETNX + EXPIRE 实现）；InProcess 后端下用 asyncio.Lock 兜底。 +- EpisodicMemory.search() 的语义搜索精度足够检索相似 task_input（pgvector 已有）。 +- 危险操作白名单覆盖 95%+ 真实危险场景；漏判由 KTD2 ponytail ceiling 标注。 +- 前端 WebSocket 已支持自定义事件类型（已验证：`team_synthesis_chunk` 等自定义事件已工作）。 + +### Sequencing + +3 个维度可部分并行： +- **维度 1（U4）** 与 **维度 3（U5-U7）** 完全独立，可并行开发。 +- **维度 2（U1→U2→U3）** 内部串行：配置 → 自主模式 → 超时暂停。 +- **跨维度依赖**：无。维度 1 的并行子任务和维度 2 的自主执行是正交的。 + +--- + +## Implementation Units + +### U1. 危险操作白名单配置 + +**Goal**: 在 `agentkit.yaml` 新增 `dangerous_tools` 配置节，遵循 ServerConfig.from_dict 模式，支持工具名正则匹配。 + +**Requirements**: R8 + +**Dependencies**: None + +**Files**: +- `src/agentkit/server/config.py` — 新增 `DangerousToolsConfig` 类，挂载到 `ServerConfig` +- `configs/agentkit.yaml.example` — 新增 `dangerous_tools` 配置示例 +- `tests/unit/test_server_config.py` — 新增配置解析测试 + +**Approach**: +- 新增 `DangerousToolsConfig` 类（继承 Pydantic BaseModel），字段：`tool_patterns: list[str]`（正则列表）+ `enabled: bool`（默认 true） +- 默认白名单：`rm`, `rmdir`, `deploy`, `kubectl`, `helm`, `git_push_force`, `alembic`, `migrate`, `payment_*` +- `ServerConfig.from_dict` 解析 `dangerous_tools` 段，缺失时用默认白名单 +- 提供 `is_dangerous(tool_name: str) -> bool` 方法，用 `re.match` 检查 + +**Patterns to follow**: `MCPServerConfig` (`server/config.py:23`) 的 from_dict 模式；`CacheConfig` 的嵌套配置模式。 + +**Test scenarios**: +- 配置解析：`dangerous_tools` 段存在时正确解析为 `DangerousToolsConfig` +- 默认值：`dangerous_tools` 段缺失时使用默认白名单 +- 工具匹配：`is_dangerous("rm")` 返回 True，`is_dangerous("read_file")` 返回 False +- 正则匹配：`is_dangerous("payment_charge")` 匹配 `payment_*` 返回 True +- enabled=false 时：`is_dangerous` 始终返回 False（禁用白名单） + +**Verification**: `python3 -m pytest tests/unit/test_server_config.py -x -q` 通过；`ruff check src/agentkit/server/config.py` 无 lint 错误。 + +### U2. PLAN_EXEC 自主执行模式 + 危险操作确认 + +**Goal**: PLAN_EXEC 模式下，plan 确认后进入自主执行，仅危险操作触发 confirmation_request，其他操作自主执行。 + +**Requirements**: R6, R7, R9 + +**Dependencies**: U1 + +**Files**: +- `src/agentkit/core/react.py` — 修改 `confirmation_request` 触发逻辑（行 1255, 2199），增加危险操作检查 +- `src/agentkit/server/routes/chat.py` — PLAN_EXEC 路由处理，注入 `DangerousToolsConfig` +- `tests/unit/test_react_autonomy.py` — 新增自主执行测试 +- `tests/unit/test_plan_exec_autonomy.py` — 新增 PLAN_EXEC 集成测试 + +**Approach**: +- ReActEngine 构造时接收 `dangerous_tools_config: DangerousToolsConfig | None` +- 工具执行前检查：若 `dangerous_tools_config.is_dangerous(tool_name)` 且在 PLAN_EXEC 自主模式 → 触发 `confirmation_request`（已有事件类型） +- 非危险操作 → 直接执行，不触发 confirmation +- PLAN_EXEC 的 plan 确认后，设置 `autonomy_mode: true` 标志（区别于现有"每阶段确认"模式） +- `cancel` 消息已有 CancellationToken 机制（`core/protocol.py`），无需修改 + +**Patterns to follow**: 现有 `confirmation_request` 事件触发模式（`react.py:1255`）；`phase_violation` 检测模式（`react.py:272`）。 + +**Test scenarios**: +- 危险操作触发 confirmation：自主模式下执行 `rm` 触发 `confirmation_request` 事件 +- 非危险操作自主执行：自主模式下执行 `read_file` 不触发 confirmation +- 非自主模式行为不变：非 PLAN_EXEC 模式下，confirmation 行为与现有逻辑一致 +- cancel 中断：自主执行中发送 `cancel` 中断任务，CancellationToken 正确取消 +- 危险操作白名单禁用：`enabled=false` 时所有操作自主执行 + +**Verification**: `python3 -m pytest tests/unit/test_react_autonomy.py tests/unit/test_plan_exec_autonomy.py -x -q` 通过；现有 PLAN_EXEC 测试不回归。 + +### U3. 自主执行超时 + autonomy_paused 事件 + +**Goal**: 自主执行超时（默认 30 分钟）或连续失败 3 次时，自动暂停并发送 `autonomy_paused` 事件，用户可 `resume` 继续。 + +**Requirements**: R10 + +**Dependencies**: U2 + +**Files**: +- `src/agentkit/core/react.py` — 新增 `autonomy_paused` 事件类型 + 超时/失败计数逻辑 +- `src/agentkit/server/routes/chat.py` — 处理 `autonomy_paused` 事件 + `resume` 消息 +- `src/agentkit/server/config.py` — 新增 `autonomy_timeout_minutes: int = 30` + `max_consecutive_failures: int = 3` 配置 +- `tests/unit/test_autonomy_paused.py` — 新增暂停/恢复测试 + +**Approach**: +- ReActEngine 新增 `autonomy_started_at: float` + `consecutive_failures: int` 状态 +- 每次工具执行前检查：`time.time() - autonomy_started_at > timeout` 或 `consecutive_failures >= 3` → 触发 `autonomy_paused` +- `autonomy_paused` 事件 payload：`{reason: "timeout"|"consecutive_failures"|"manual", progress: {...}, resume_token: "..."}` +- WebSocket 路由处理 `resume` 消息：重置 `autonomy_started_at` + `consecutive_failures`，继续执行 +- 前端区分：`confirmation_request` = 等待单次确认；`autonomy_paused` = 整体自主执行暂停 + +**Patterns to follow**: 现有 `phase_changed` 事件模式（`chat.py:1694`）；`confirmation_result` 处理模式。 + +**Test scenarios**: +- 超时触发暂停：自主执行超过 30 分钟触发 `autonomy_paused`，reason="timeout" +- 连续失败触发暂停：连续 3 次工具失败触发 `autonomy_paused`，reason="consecutive_failures" +- resume 恢复执行：发送 `resume` 消息后，自主执行继续，计数器重置 +- 超时配置化：`autonomy_timeout_minutes=60` 时，30 分钟不触发暂停 +- 非自主模式不触发：非 PLAN_EXEC 模式下，超时/失败不触发 `autonomy_paused` + +**Verification**: `python3 -m pytest tests/unit/test_autonomy_paused.py -x -q` 通过；`autonomy_paused` 事件 payload 符合契约。 + +### U4. TeamOrchestrator 无依赖子任务并行模式 + +**Goal**: TeamOrchestrator 的 Lead 分解任务时，识别无 depends_on 的子任务，自动派发给隔离 ConfigDrivenAgent 并行执行（而非串行排队）。 + +**Requirements**: R1, R2, R3, R4, R5 + +**Dependencies**: None（与 U1-U3 正交） + +**Files**: +- `src/agentkit/experts/orchestrator.py` — 修改 `_run_pipeline`（行 234）支持并行子任务派发 +- `src/agentkit/experts/plan.py` — `TeamPlan` 新增 `get_independent_subtasks()` 方法 +- `tests/unit/test_team_parallel.py` — 新增并行子任务测试 + +**Approach**: +- `TeamPlan.get_independent_subtasks()`: 返回 `depends_on == []` 的 PlanPhase 列表（已有 `depends_on` 字段，`plan.py:176`） +- `_run_pipeline` 修改： + 1. 调用 `topological_sort()` 得到执行层（已有） + 2. 同层内的 phases 已并行执行（已有）— 无需修改 + 3. **新增**：Lead 分解时，若识别到多个无 depends_on 的子任务，显式派发到同层（而非分散到不同层） +- KTD6 约束：若子任务共享上游依赖，降级为同层并行（topological_sort 已处理） +- SharedWorkspace 写入防护：关键写入用 `SharedWorkspace.lock()`（已有，`shared_workspace.py:96`） +- MAX_EXPERTS=10 约束已有（`experts/router.py`），超出时 Lead 重新分解 + +**Patterns to follow**: 现有 `_run_pipeline` 的同层并行模式（`orchestrator.py:234`）；`_get_isolated_agent` 的 agent 创建模式（`orchestrator.py:113`）。 + +**Test scenarios**: +- 无依赖子任务并行：3 个无 depends_on 的子任务，同时派发到 3 个隔离 agent +- 共享依赖降级：2 个子任务共享上游依赖，降级为同层并行（不并行派发） +- MAX_EXPERTS 约束：11 个无依赖子任务时，Lead 重新分解为 10 个以内 +- SharedWorkspace 路径不重叠：并行子任务的输出路径 `{plan_id}/phase/{phase_id}/output` 唯一 +- expert_step 事件携带 expert_id：并行执行时事件正确区分不同 expert +- 综合阶段等待：所有并行子任务完成后才进入 synthesis + +**Verification**: `python3 -m pytest tests/unit/test_team_parallel.py -x -q` 通过；现有 TeamOrchestrator 测试不回归。 + +### U5. ReflexionEngine 反思持久化到 EpisodicMemory + +**Goal**: ReflexionEngine 的 `_reflect` 方法生成反思后，将结果写入 EpisodicMemory，支持跨任务检索。 + +**Requirements**: R11, R15 + +**Dependencies**: None（与 U1-U4 正交） + +**Files**: +- `src/agentkit/core/reflexion.py` — 修改 `_reflect`（行 648），新增持久化逻辑 +- `src/agentkit/memory/episodic.py` — 新增 `store_prompt_reflection()` + `search_prompt_reflections()` 方法 +- `tests/unit/test_reflexion_persist.py` — 新增持久化测试 + +**Approach**: +- `_reflect` 返回前，调用 `EpisodicMemory.store_prompt_reflection(task_input, reflection, improved_prompt, version, score)` +- 存储格式：`key = "prompt_reflection:{task_hash}:{version}"`，`value = {task_input, reflection, improved_prompt, score, timestamp}` +- KTD7 决策：保留所有版本，score 字段记录效果（verify 通过=1.0，失败=0.0） +- R15 TTL：复用 EpisodicMemory 现有清理机制（若无可加 `cleanup_expired()` 方法，30 天阈值） +- `_build_reflection_prompt`（行 693）已生成 improved_prompt，直接存储 + +**Patterns to follow**: `EpisodicMemory.store()` 现有模式（`episodic.py:66`）；`EpisodicMemory.search()` 语义搜索模式（`episodic.py:206`）。 + +**Test scenarios**: +- 反思持久化：`_reflect` 返回后，EpisodicMemory 中存在对应记录 +- 存储字段完整：记录包含 task_input + reflection + improved_prompt + version + score + timestamp +- TTL 清理：30 天前的记录被 `cleanup_expired()` 清理 +- 多版本共存：同一 task_hash 的多个版本都保留，version 递增 +- 持久化失败不阻塞：EpisodicMemory 写入失败时，`_reflect` 仍返回反思文本（降级处理） + +**Verification**: `python3 -m pytest tests/unit/test_reflexion_persist.py -x -q` 通过；现有 ReflexionEngine 测试不回归。 + +### U6. Lead 规划时检索历史反思 + prompt 替换 + +**Goal**: PLAN_EXEC / TEAM_COLLAB 的 Lead 规划时，检索 EpisodicMemory 中相似 task_input 的历史反思，用 improved_prompt 替换默认 prompt。 + +**Requirements**: R12, R13 + +**Dependencies**: U5 + +**Files**: +- `src/agentkit/experts/orchestrator.py` — `_decompose_task`（行 534）前新增反思检索 +- `src/agentkit/core/reflexion.py` — 新增 `retrieve_prompt_reflection(task_input) -> dict | None` +- `tests/unit/test_lead_reflection_retrieval.py` — 新增检索 + 替换测试 + +**Approach**: +- `_decompose_task` 前，调用 `ReflexionEngine.retrieve_prompt_reflection(task_input)` +- 检索逻辑：`EpisodicMemory.search_prompt_reflections(task_input, top_k=3)` → 按 score 降序取最高 +- 若找到 improved_prompt（score > 0.5），替换 Lead 的默认 system prompt +- KTD5 触发条件：仅在 verify 二次失败 / schema 校验失败 / 循环检测时才触发反思（避免无意义检索） +- 无历史反思时：用默认 prompt（现有行为不变） + +**Patterns to follow**: `_decompose_task` 现有模式（`orchestrator.py:534`）；`_build_reflection_prompt` 的 prompt 拼接模式（`reflexion.py:693`）。 + +**Test scenarios**: +- 历史反思命中：EpisodicMemory 有相似 task_input 记录，Lead 用 improved_prompt +- 历史反思未命中：无相似记录，Lead 用默认 prompt（行为不变） +- score 过滤：score <= 0.5 的反思不替换（避免低质量反思污染） +- KTD5 触发条件：非触发错误类型（如普通 timeout）不检索历史反思 +- 检索失败降级：EpisodicMemory 检索失败时，Lead 用默认 prompt + +**Verification**: `python3 -m pytest tests/unit/test_lead_reflection_retrieval.py -x -q` 通过；现有 TeamOrchestrator 测试不回归。 + +### U7. ABTester 离线对比 prompt 版本 + +**Goal**: ABTester 支持对比 EpisodicMemory 中同一 task_hash 的多个 prompt 版本效果，离线验证（不在线 bandit）。 + +**Requirements**: R14 + +**Dependencies**: U5 + +**Files**: +- `src/agentkit/evolution/ab_tester.py` — 新增 `compare_prompt_versions(task_hash) -> dict` 方法 +- `tests/unit/test_ab_tester_prompt.py` — 新增版本对比测试 + +**Approach**: +- `compare_prompt_versions(task_hash)`: 检索 EpisodicMemory 中该 task_hash 的所有版本 +- 输出：`{versions: [{version, score, timestamp, reflection_summary}], best_version, recommendation}` +- 离线验证：不在线 bandit，仅基于历史 score 对比 +- `recommendation` 字段：建议保留 score 最高的版本，清理低分版本（可选） + +**Patterns to follow**: 现有 ABTester 的对比模式（`evolution/ab_tester.py`）。 + +**Test scenarios**: +- 多版本对比：3 个版本（score: 0.8, 0.6, 0.4），recommendation 选 0.8 版本 +- 无版本时：返回空结果，不报错 +- 单版本时：直接返回该版本为 best_version +- 低分版本清理：可选清理 score < 0.3 的版本（保留 top-K） + +**Verification**: `python3 -m pytest tests/unit/test_ab_tester_prompt.py -x -q` 通过。 + +--- + +## Verification Contract + +### Unit Tests + +```bash +# 全量单元测试（必须通过） +python3 -m pytest tests/unit/ -x -q + +# 本次新增测试（逐个验证） +python3 -m pytest tests/unit/test_server_config.py tests/unit/test_react_autonomy.py tests/unit/test_plan_exec_autonomy.py tests/unit/test_autonomy_paused.py tests/unit/test_team_parallel.py tests/unit/test_reflexion_persist.py tests/unit/test_lead_reflection_retrieval.py tests/unit/test_ab_tester_prompt.py -x -q +``` + +### Lint + Format + +```bash +# Ruff lint + format（必须通过） +ruff check src/ && ruff format src/ +``` + +### Integration Tests (Optional) + +```bash +# 集成测试（需 Docker Redis + PostgreSQL，可选） +python3 -m pytest -m "integration" -x -q +``` + +### Quality Gates + +- 无 `any` 类型（AGENTS.md 约束） +- 所有 Pydantic 模型用 `model_config = ConfigDict(...)`（AGENTS.md 约束） +- API Key 比较用 `hmac.compare_digest`（若涉及） +- 异步生成器安全（`.trae/rules/project_rules.md`） +- 无 `return` 在第一个 `yield` 之前（若新增 async generator） + +--- + +## Definition of Done + +### Global Criteria + +- 7 个 Implementation Units (U1-U7) 全部完成 +- 所有新增测试通过（8 个测试文件） +- 现有测试不回归（`tests/unit/` 全量通过） +- `ruff check src/ && ruff format src/` 无错误 +- 无 `any` 类型，Pydantic 模型用 `ConfigDict` +- 异步生成器安全（无 early return before yield） + +### Per-Unit Criteria + +| Unit | Done Signal | +|------|-------------| +| U1 | `DangerousToolsConfig` 解析正确，`is_dangerous()` 工作正常 | +| U2 | PLAN_EXEC 自主模式下危险操作触发 confirmation，非危险操作自主执行 | +| U3 | 超时/连续失败触发 `autonomy_paused`，`resume` 恢复执行 | +| U4 | 无依赖子任务并行派发，SharedWorkspace 路径不重叠 | +| U5 | `_reflect` 后 EpisodicMemory 有记录，TTL 清理工作 | +| U6 | Lead 检索历史反思并替换 prompt（score > 0.5 时） | +| U7 | ABTester 对比多版本 prompt，返回 best_version | + +### Cleanup Criteria + +- 移除调试代码（print、TODO 注释、临时 hack） +- 移除未使用的 import +- 移除实验性死代码（尝试过但未采用的方案） +- 所有新增配置项有文档（`agentkit.yaml.example` 示例） + +--- + +## Appendix: 竞品对标参考 + +| 维度 | 竞品代表 | AgentKit 本次目标 | +|------|----------|-------------------| +| 并行 Sub-agent | Cursor 3.2 Subagents（并行专业化 worker）| R1-R5: TeamOrchestrator 无依赖子任务并行 | +| Goal-driven 自主 | Devin 3.0 Goal-driven（数小时自主）、Qoder 1.11 Goal + Scheduling | R6-R10: PLAN_EXEC 危险操作确认式自主 | +| Prompt 自调优 | Devin 强化学习+元学习、ReflexionEngine 跨任务 | R11-R15: ReflexionEngine 反思持久化到 EpisodicMemory | + +## Appendix: 关键代码位置参考 + +| 子系统 | 文件 | 关键位置 | +|--------|------|----------| +| TeamOrchestrator | `src/agentkit/experts/orchestrator.py` | `execute()` 行 143, `_run_pipeline` 行 234, `_decompose_task` 行 534 | +| TeamPlan | `src/agentkit/experts/plan.py` | `topological_sort()` 行 385, `SubTask` 行 69, `PlanPhase.depends_on` 行 176 | +| ReflexionEngine | `src/agentkit/core/reflexion.py` | `_reflect()` 行 648, `_build_reflection_prompt` 行 693 | +| EpisodicMemory | `src/agentkit/memory/episodic.py` | `store()` 行 66, `search()` 行 206 | +| SharedWorkspace | `src/agentkit/core/shared_workspace.py` | `lock()` 行 96, `write()` 行 39 | +| ReActEngine | `src/agentkit/core/react.py` | `confirmation_request` 行 1255/2199, `phase_violation` 行 272 | +| PLAN_EXEC 路由 | `src/agentkit/chat/skill_routing.py` | `ExecutionMode.PLAN_EXEC` 行 33 | +| WebSocket 事件 | `src/agentkit/server/routes/chat.py` | `confirmation_request` 行 1646, `phase_changed` 行 1701 | +| ServerConfig | `src/agentkit/server/config.py` | `from_dict` 行 200, `MCPServerConfig` 行 23 | +| CancellationToken | `src/agentkit/core/protocol.py` | 类定义 | From b6f7d82ff573e5c9b78b19fd13526b7da8665ebc Mon Sep 17 00:00:00 2001 From: Chiguyong Date: Mon, 6 Jul 2026 13:12:34 +0800 Subject: [PATCH 02/10] feat(config): U1 add DangerousToolsConfig for PLAN_EXEC autonomy (KTD2) - New DangerousToolsConfig dataclass with regex-based tool_patterns - Default whitelist: rm/rmdir/deploy/kubectl/helm/git_push_force/alembic/migrate/payment_* - is_dangerous(tool_name) method for autonomy mode gate - ServerConfig.from_dict parses dangerous_tools section - 10 unit tests covering defaults, matching, disabled, custom patterns --- src/agentkit/server/config.py | 52 +++++++++++++- tests/unit/test_server_config.py | 120 +++++++++++++++++++++++++++++-- 2 files changed, 167 insertions(+), 5 deletions(-) diff --git a/src/agentkit/server/config.py b/src/agentkit/server/config.py index e634908..ee34e06 100644 --- a/src/agentkit/server/config.py +++ b/src/agentkit/server/config.py @@ -4,7 +4,7 @@ import asyncio import logging import os import re -from dataclasses import dataclass +from dataclasses import dataclass, field from pathlib import Path from typing import Any, Callable @@ -57,6 +57,48 @@ class MCPServerConfig: ) +@dataclass +class DangerousToolsConfig: + """Configuration for dangerous tool detection in PLAN_EXEC autonomy mode. + + When enabled, tools matching any pattern in ``tool_patterns`` trigger a + confirmation request before execution. KTD2: conservative whitelist — + ponytail ceiling = missed dangerous ops not in the list; upgrade path = + LLM-assisted classification. + """ + + enabled: bool = True + tool_patterns: list[str] = field( + default_factory=lambda: [ + r"^rm$", + r"^rmdir$", + r"^deploy$", + r"^kubectl", + r"^helm", + r"^git_push_force$", + r"^alembic", + r"^migrate", + r"^payment_", + ] + ) + + def is_dangerous(self, tool_name: str) -> bool: + """Check if a tool name matches any dangerous pattern.""" + if not self.enabled: + return False + return any(re.match(p, tool_name) for p in self.tool_patterns) + + @classmethod + def from_dict(cls, data: dict | None) -> "DangerousToolsConfig": + """Create from dict (parsed from YAML). Empty/None returns defaults.""" + if not data: + return cls() + return cls( + enabled=data.get("enabled", True), + tool_patterns=data.get("tool_patterns") or cls().tool_patterns, + ) + + def _resolve_env_vars(value: Any) -> Any: """Resolve ${VAR:-default} patterns in string values from environment variables.""" if not isinstance(value, str): @@ -124,6 +166,9 @@ class ServerConfig: # G6/U2: PLAN_EXEC phase policy config (opt-in — None = disabled). # Parsed via PhasePolicy.policy_from_config() at chat.py wiring time. plan_exec: dict[str, Any] | None = None, + # IQ-Boost/U1: dangerous_tools config for PLAN_EXEC autonomy mode (KTD2). + # Parsed via DangerousToolsConfig.from_dict(); defaults applied when None. + dangerous_tools: dict[str, Any] | None = None, on_change: Callable[["ServerConfig"], None] | None = None, ): self.host = host @@ -168,6 +213,8 @@ class ServerConfig: # Resolved to PhasePolicy via agentkit.core.phase.policy_from_config() # at chat.py WebSocket wiring time (U4). self.plan_exec = plan_exec or {} + # IQ-Boost/U1: dangerous_tools parsed config (KTD2 conservative whitelist). + self.dangerous_tools = DangerousToolsConfig.from_dict(dangerous_tools) self.on_change = on_change # Config watching state @@ -261,6 +308,8 @@ class ServerConfig: fallback_chain_data = data.get("fallback_chain", {}) # G6/U2: plan_exec phase policy 配置 (从 YAML 读取, opt-in) plan_exec_data = data.get("plan_exec", {}) + # IQ-Boost/U1: dangerous_tools 配置 (从 YAML 读取, KTD2 whitelist) + dangerous_tools_data = data.get("dangerous_tools", {}) return cls( host=server.get("host", "0.0.0.0"), @@ -295,6 +344,7 @@ class ServerConfig: rollback=rollback_data, fallback_chain=fallback_chain_data, plan_exec=plan_exec_data, + dangerous_tools=dangerous_tools_data, ) @staticmethod diff --git a/tests/unit/test_server_config.py b/tests/unit/test_server_config.py index e8d1b12..3d4dacf 100644 --- a/tests/unit/test_server_config.py +++ b/tests/unit/test_server_config.py @@ -6,7 +6,13 @@ from pathlib import Path import pytest -from agentkit.server.config import ServerConfig, find_config_path, _resolve_env_vars, _deep_resolve +from agentkit.server.config import ( + ServerConfig, + DangerousToolsConfig, + find_config_path, + _resolve_env_vars, + _deep_resolve, +) class TestEnvVarResolution: @@ -178,7 +184,9 @@ prompt: # Update yaml_content with absolute path yaml_content_updated = yaml_content.replace("./skills", str(skills_dir)) - with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False, dir=tmpdir) as f: + with tempfile.NamedTemporaryFile( + mode="w", suffix=".yaml", delete=False, dir=tmpdir + ) as f: f.write(yaml_content_updated) f.flush() config = ServerConfig.from_yaml(f.name) @@ -209,7 +217,9 @@ skills: paths: - "{skill_yaml}" """ - with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False, dir=tmpdir) as f: + with tempfile.NamedTemporaryFile( + mode="w", suffix=".yaml", delete=False, dir=tmpdir + ) as f: f.write(yaml_content) f.flush() config = ServerConfig.from_yaml(f.name) @@ -246,7 +256,9 @@ skills: paths: - "{skills_dir}" """ - with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False, dir=tmpdir) as f: + with tempfile.NamedTemporaryFile( + mode="w", suffix=".yaml", delete=False, dir=tmpdir + ) as f: f.write(yaml_content) f.flush() config = ServerConfig.from_yaml(f.name) @@ -444,3 +456,103 @@ class TestConfigHotReload: assert config.port == 9000 os.unlink(config_path) + + +class TestDangerousToolsConfig: + """Test DangerousToolsConfig — U1: dangerous tool whitelist for PLAN_EXEC autonomy (KTD2).""" + + def test_defaults_when_no_data(self): + cfg = DangerousToolsConfig.from_dict(None) + assert cfg.enabled is True + assert len(cfg.tool_patterns) > 0 + # Default whitelist includes rm, deploy, kubectl, etc. + assert cfg.is_dangerous("rm") + assert cfg.is_dangerous("deploy") + assert cfg.is_dangerous("kubectl_apply") + + def test_defaults_when_empty_dict(self): + cfg = DangerousToolsConfig.from_dict({}) + assert cfg.enabled is True + assert cfg.is_dangerous("rmdir") + + def test_dangerous_tool_match(self): + cfg = DangerousToolsConfig() + assert cfg.is_dangerous("rm") is True + assert cfg.is_dangerous("rmdir") is True + assert cfg.is_dangerous("deploy") is True + assert cfg.is_dangerous("kubectl_rollout") is True + assert cfg.is_dangerous("helm_upgrade") is True + assert cfg.is_dangerous("git_push_force") is True + assert cfg.is_dangerous("alembic_upgrade") is True + assert cfg.is_dangerous("migrate_db") is True + assert cfg.is_dangerous("payment_charge") is True + + def test_non_dangerous_tool(self): + cfg = DangerousToolsConfig() + assert cfg.is_dangerous("read_file") is False + assert cfg.is_dangerous("write_file") is False + assert cfg.is_dangerous("search") is False + assert cfg.is_dangerous("list_dir") is False + + def test_payment_glob_pattern(self): + cfg = DangerousToolsConfig() + # payment_* pattern matches any tool starting with payment_ + assert cfg.is_dangerous("payment_refund") is True + assert cfg.is_dangerous("payment_transfer") is True + + def test_disabled_returns_false(self): + cfg = DangerousToolsConfig(enabled=False) + assert cfg.is_dangerous("rm") is False + assert cfg.is_dangerous("deploy") is False + + def test_custom_patterns_from_dict(self): + data = {"enabled": True, "tool_patterns": [r"^custom_dangerous"]} + cfg = DangerousToolsConfig.from_dict(data) + assert cfg.is_dangerous("custom_dangerous_op") is True + assert cfg.is_dangerous("rm") is False # default patterns overridden + + def test_enabled_false_from_dict(self): + data = {"enabled": False} + cfg = DangerousToolsConfig.from_dict(data) + assert cfg.enabled is False + assert cfg.is_dangerous("rm") is False + + def test_server_config_loads_dangerous_tools(self): + yaml_content = """ +server: + host: "0.0.0.0" + port: 8001 + +dangerous_tools: + enabled: true + tool_patterns: + - "^rm$" + - "^kubectl" +""" + with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False) as f: + f.write(yaml_content) + f.flush() + config = ServerConfig.from_yaml(f.name) + + assert config.dangerous_tools.enabled is True + assert config.dangerous_tools.is_dangerous("rm") is True + assert config.dangerous_tools.is_dangerous("kubectl_apply") is True + # Custom patterns override defaults + assert config.dangerous_tools.is_dangerous("deploy") is False + os.unlink(f.name) + + def test_server_config_defaults_dangerous_tools_when_missing(self): + yaml_content = """ +server: + host: "0.0.0.0" + port: 8001 +""" + with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False) as f: + f.write(yaml_content) + f.flush() + config = ServerConfig.from_yaml(f.name) + + # Missing config section → defaults applied + assert config.dangerous_tools.enabled is True + assert config.dangerous_tools.is_dangerous("rm") is True + os.unlink(f.name) From 4729e20bbf2a5729479e75c65288d23743e3b61b Mon Sep 17 00:00:00 2001 From: Chiguyong Date: Mon, 6 Jul 2026 13:23:06 +0800 Subject: [PATCH 03/10] feat(react): U2 PLAN_EXEC autonomy mode + dangerous-tools gate (R6-R9) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ReActEngine now accepts dangerous_tools_config + autonomy_mode. In PLAN_EXEC autonomy mode, dangerous tools (per DangerousToolsConfig whitelist) trigger confirmation_request BEFORE first execution; non-dangerous tools execute directly without confirmation. _build_phase_engine injects the config and sets autonomy_mode=True when PLAN_EXEC is engaged. - _check_autonomy_gate: pre-execution gate (serial/parallel/text-parsed paths) - Approved dangerous tool re-executes with _skip_dangerous_check=True - Rejected/handler-failure → permission_denied result (exit_code=126) - Default autonomy_mode=False → transparent no-op (backward compat) - 19 new tests (14 autonomy gate + 5 PLAN_EXEC wiring) --- src/agentkit/core/react.py | 198 ++++++++++++++++- src/agentkit/server/routes/chat.py | 6 + tests/unit/test_plan_exec_autonomy.py | 123 +++++++++++ tests/unit/test_react_autonomy.py | 302 ++++++++++++++++++++++++++ 4 files changed, 619 insertions(+), 10 deletions(-) create mode 100644 tests/unit/test_plan_exec_autonomy.py create mode 100644 tests/unit/test_react_autonomy.py diff --git a/src/agentkit/core/react.py b/src/agentkit/core/react.py index f39991b..b3ef4eb 100644 --- a/src/agentkit/core/react.py +++ b/src/agentkit/core/react.py @@ -44,6 +44,11 @@ if TYPE_CHECKING: from agentkit.evolution.pitfall_detector import PitfallWarning from agentkit.memory.retriever import MemoryRetriever + # U2/R8: duck-typed at runtime to avoid core→server layering violation. + # Any object with ``is_dangerous(tool_name: str) -> bool`` + ``enabled: bool`` + # is accepted; server/config.DangerousToolsConfig is the canonical impl. + from agentkit.server.config import DangerousToolsConfig + logger = logging.getLogger(__name__) @@ -204,6 +209,15 @@ class ReActEngine: # max_reinjections) # Loop detector threshold raised from 2 to 3 (R10/RV22). phase_budgets: dict[str, int] | None = None, + # IQ-Boost/U2/R8: dangerous-tools whitelist for PLAN_EXEC autonomy. + # Duck-typed (see TYPE_CHECKING import). None = no autonomy gate + # (backward compat for DIRECT_CHAT/REACT and existing tests). + dangerous_tools_config: "DangerousToolsConfig | None" = None, + # IQ-Boost/U2/R6: autonomy mode flag. True = PLAN_EXEC plan confirmed, + # only dangerous tools (per dangerous_tools_config) trigger + # confirmation_request; non-dangerous tools execute directly. + # False = existing behavior (tool-side _is_dangerous drives confirmation). + autonomy_mode: bool = False, ): if max_steps < 1: raise ValueError(f"max_steps must be >= 1, got {max_steps}") @@ -298,6 +312,11 @@ class ReActEngine: # _execute_loop's finally block so subsequent execute() calls without a # restore still reset properly. self._state_restored: bool = False + # IQ-Boost/U2: autonomy mode state (R6/R7). When True, the engine gates + # dangerous tools (per dangerous_tools_config) with a pre-execution + # confirmation_request; non-dangerous tools execute without confirmation. + self._dangerous_tools_config = dangerous_tools_config + self._autonomy_mode: bool = autonomy_mode def reset(self) -> None: """Reset internal state for reuse across conversations. @@ -1239,13 +1258,38 @@ class ReActEngine: data={"tool_name": tc.name, "arguments": tc.arguments}, ) - tool_start = time.monotonic() - tool_result = await self._execute_tool(tc.name, tc.arguments, tools) - tool_duration_ms = int((time.monotonic() - tool_start) * 1000) + # IQ-Boost/U2/R7: autonomy mode pre-execution gate. + # Dangerous tools (per config) get a confirmation_request + # BEFORE first execution; non-dangerous tools fall through. + ( + should_exec, + autonomy_events, + autonomy_result, + ) = await self._check_autonomy_gate( + tc.name, + tc.arguments, + tools, + step, + confirmation_handler, + ) + for _aev in autonomy_events: + yield _aev - # Handle confirmation flow - if isinstance(tool_result, dict) and tool_result.get( - "needs_confirmation" + if not should_exec: + # Autonomy gate handled execution (approved+skip + # flag, or rejected). Use its result directly. + tool_result = autonomy_result + tool_duration_ms = 0 + else: + tool_start = time.monotonic() + tool_result = await self._execute_tool(tc.name, tc.arguments, tools) + tool_duration_ms = int((time.monotonic() - tool_start) * 1000) + + # Handle confirmation flow (tool-side _is_dangerous) + if ( + should_exec + and isinstance(tool_result, dict) + and tool_result.get("needs_confirmation") ): confirmation_id = tool_result["confirmation_id"] command = tool_result.get("command", "") @@ -1407,11 +1451,28 @@ class ReActEngine: step=step, data={"tool_name": pc["name"], "arguments": pc["arguments"]}, ) - tool_start = time.monotonic() - tool_result = await self._execute_tool( - pc["name"], pc["arguments"], tools + # IQ-Boost/U2: autonomy gate for text-parsed calls too + # (consistency — dangerous tools must be gated regardless + # of whether the LLM returned structured or text calls). + pc_args = pc["arguments"] if isinstance(pc["arguments"], dict) else {} + ( + should_exec, + autonomy_events, + autonomy_result, + ) = await self._check_autonomy_gate( + pc["name"], pc_args, tools, step, confirmation_handler ) - tool_duration_ms = int((time.monotonic() - tool_start) * 1000) + for _aev in autonomy_events: + yield _aev + if not should_exec: + tool_result = autonomy_result + tool_duration_ms = 0 + else: + tool_start = time.monotonic() + tool_result = await self._execute_tool( + pc["name"], pc["arguments"], tools + ) + tool_duration_ms = int((time.monotonic() - tool_start) * 1000) react_step = ReActStep( step=step, @@ -2170,6 +2231,113 @@ class ReActEngine: logger.warning(error_msg) return {"error": error_msg, "error_code": "tool_execution_failed"} + async def _check_autonomy_gate( + self, + tool_name: str, + tool_arguments: dict, + tools: list[Tool], + step: int, + confirmation_handler: Callable[..., Awaitable[object]] | None, + ) -> tuple[bool, list[ReActEvent], object]: + """IQ-Boost/U2/R7: pre-execution gate for PLAN_EXEC autonomy mode. + + When autonomy_mode is on and the tool matches the dangerous-tools + whitelist, emit a ``confirmation_request`` BEFORE the tool runs and + wait on ``confirmation_handler``. Approved → execute with + ``_skip_dangerous_check=True`` (bypass the tool's own _is_dangerous + re-confirmation). Rejected → return a permission_denied result. + + Non-dangerous tools and non-autonomy mode fall through (return + ``should_execute=True``) so the caller proceeds with the existing + ``_execute_tool`` → ``needs_confirmation`` flow unchanged. + + Returns ``(should_execute, events, pre_result)``: + - should_execute=True → caller runs _execute_tool; pre_result=None + - should_execute=False → caller uses pre_result as tool_result + """ + events: list[ReActEvent] = [] + if ( + not self._autonomy_mode + or self._dangerous_tools_config is None + or not self._dangerous_tools_config.is_dangerous(tool_name) + ): + return (True, events, None) + + # Dangerous tool in autonomy mode — gate before first execution. + confirmation_id = f"autonomy:{tool_name}:{step}" + command = f"{tool_name} {tool_arguments}" + reason = "危险操作（匹配 autonomy 白名单）需确认后执行" + events.append( + ReActEvent( + event_type="confirmation_request", + step=step, + data={ + "confirmation_id": confirmation_id, + "tool_name": tool_name, + "command": command, + "reason": reason, + }, + ) + ) + + approved = False + if confirmation_handler is not None: + try: + approved = await confirmation_handler(confirmation_id, command, reason) + except asyncio.CancelledError: + raise + except Exception as e: + logger.warning(f"Autonomy confirmation handler error: {e}") + + if approved: + tool = self._find_tool(tool_name, tools) + if tool is None: + events.append( + ReActEvent( + event_type="confirmation_result", + step=step, + data={"confirmation_id": confirmation_id, "approved": True}, + ) + ) + return (False, events, {"error": f"Tool '{tool_name}' not found"}) + clean_args = {k: v for k, v in tool_arguments.items() if not k.startswith("_")} + clean_args["_skip_dangerous_check"] = True + try: + result = await tool.safe_execute(**clean_args) + except (ToolValidationError, ValueError, TypeError, RuntimeError) as e: + result = { + "error": f"Tool '{tool_name}' execution failed: {e}", + "error_code": "tool_execution_failed", + } + events.append( + ReActEvent( + event_type="confirmation_result", + step=step, + data={"confirmation_id": confirmation_id, "approved": True}, + ) + ) + return (False, events, result) + + # Rejected by user (or handler timed out / returned False) + events.append( + ReActEvent( + event_type="confirmation_result", + step=step, + data={"confirmation_id": confirmation_id, "approved": False}, + ) + ) + return ( + False, + events, + { + "output": "", + "exit_code": 126, + "is_error": True, + "error_type": "permission_denied", + "message": f"用户拒绝执行危险操作: {tool_name}", + }, + ) + async def _execute_tool_with_confirmation( self, tc: object, @@ -2186,6 +2354,16 @@ class ReActEngine: Tuple of (tool_result, list of ReActEvents to yield) """ events: list[ReActEvent] = [] + # IQ-Boost/U2: autonomy gate runs first (pre-execution). If it handles + # the tool (dangerous in autonomy mode), return immediately with its + # result + events; otherwise fall through to the normal flow. + should_exec, gate_events, gate_result = await self._check_autonomy_gate( + tc.name, tc.arguments, tools, step, confirmation_handler + ) + if not should_exec: + return (gate_result, gate_events) + events.extend(gate_events) + tool_result = await self._execute_tool(tc.name, tc.arguments, tools) # Check if tool returned a confirmation request diff --git a/src/agentkit/server/routes/chat.py b/src/agentkit/server/routes/chat.py index 3df74d0..f07866a 100644 --- a/src/agentkit/server/routes/chat.py +++ b/src/agentkit/server/routes/chat.py @@ -695,9 +695,15 @@ def _build_phase_engine( ) return (None, None, f"phase policy error: {str(e)[:200]}") + # IQ-Boost/U2/R6-R8: PLAN_EXEC engages autonomy mode — plan confirmation + # is handled by the existing spec_review gate; after confirmation, only + # dangerous tools (per dangerous_tools_config) trigger confirmation_request. + dangerous_tools_cfg = getattr(server_config, "dangerous_tools", None) engine = ReActEngine( llm_gateway=llm_gateway, phase_policy=phase_policy, + dangerous_tools_config=dangerous_tools_cfg, + autonomy_mode=True, ) advance_phase_tool = AdvancePhaseTool(engine=engine) tools_with_advance_phase = list(base_tools) + [advance_phase_tool] diff --git a/tests/unit/test_plan_exec_autonomy.py b/tests/unit/test_plan_exec_autonomy.py new file mode 100644 index 0000000..86d0986 --- /dev/null +++ b/tests/unit/test_plan_exec_autonomy.py @@ -0,0 +1,123 @@ +"""Integration tests for PLAN_EXEC autonomy mode wiring (IQ-Boost U2). + +Verifies that ``_build_phase_engine`` in chat.py constructs a PLAN_EXEC +engine with ``autonomy_mode=True`` and ``dangerous_tools_config`` injected +from ``ServerConfig``. This is the integration seam between U1 (config) +and U2 (engine gate). +""" + +from __future__ import annotations + +from unittest.mock import MagicMock + +from agentkit.chat.skill_routing import ExecutionMode +from agentkit.core.react import ReActEngine +from agentkit.server.config import DangerousToolsConfig, ServerConfig + + +def _make_server_config(dangerous_tools: DangerousToolsConfig | None = None) -> ServerConfig: + """Build a minimal ServerConfig with the dangerous_tools section.""" + cfg = ServerConfig.from_dict({}) + if dangerous_tools is not None: + cfg.dangerous_tools = dangerous_tools + return cfg + + +class TestBuildPhaseEngineAutonomyWiring: + """_build_phase_engine injects autonomy config into the PLAN_EXEC engine.""" + + def test_plan_exec_engine_gets_autonomy_mode(self): + """When PLAN_EXEC is engaged, the engine has autonomy_mode=True.""" + # Import here to avoid module-level FastAPI app construction. + from agentkit.server.routes.chat import _build_phase_engine + + server_cfg = _make_server_config(DangerousToolsConfig()) + engine, tools, err = _build_phase_engine( + server_config=server_cfg, + llm_gateway=MagicMock(), + execution_mode=ExecutionMode.PLAN_EXEC, + base_tools=[], + session_id="test-session", + ) + assert err is None + assert engine is not None + assert isinstance(engine, ReActEngine) + assert engine._autonomy_mode is True + assert engine._dangerous_tools_config is not None + assert engine._dangerous_tools_config.is_dangerous("rm") is True + + def test_plan_exec_engine_gets_custom_dangerous_config(self): + """Custom dangerous_tools config (disabled) propagates to the engine.""" + from agentkit.server.routes.chat import _build_phase_engine + + server_cfg = _make_server_config(DangerousToolsConfig(enabled=False)) + engine, _, err = _build_phase_engine( + server_config=server_cfg, + llm_gateway=MagicMock(), + execution_mode=ExecutionMode.PLAN_EXEC, + base_tools=[], + session_id="test", + ) + assert err is None + assert engine is not None + assert engine._dangerous_tools_config.enabled is False + # Disabled whitelist → nothing is dangerous + assert engine._dangerous_tools_config.is_dangerous("rm") is False + + def test_non_plan_exec_mode_returns_none_engine(self): + """Non-PLAN_EXEC mode → (None, None, None) — no autonomy engine built.""" + from agentkit.server.routes.chat import _build_phase_engine + + server_cfg = _make_server_config(DangerousToolsConfig()) + engine, tools, err = _build_phase_engine( + server_config=server_cfg, + llm_gateway=MagicMock(), + execution_mode=ExecutionMode.REACT, + base_tools=[], + session_id="test", + ) + assert engine is None + assert tools is None + assert err is None + + def test_plan_exec_engine_gets_default_dangerous_config_when_server_config_missing( + self, + ): + """When server_config is None or lacks dangerous_tools, the engine + still gets autonomy_mode=True (config defaults to None — gate is a + no-op, but autonomy flag is set for future wiring).""" + from agentkit.server.routes.chat import _build_phase_engine + + # server_config=None simulates tests/misconfigured deployments. + engine, _, err = _build_phase_engine( + server_config=None, + llm_gateway=MagicMock(), + execution_mode=ExecutionMode.PLAN_EXEC, + base_tools=[], + session_id="test", + ) + assert err is None + assert engine is not None + assert engine._autonomy_mode is True + # dangerous_tools_config is None when server_config is None — gate + # becomes a transparent no-op (backward compat for tests without config). + assert engine._dangerous_tools_config is None + + def test_plan_exec_engine_includes_advance_phase_tool(self): + """PLAN_EXEC engine's tool list includes AdvancePhaseTool.""" + from agentkit.server.routes.chat import _build_phase_engine + + server_cfg = _make_server_config(DangerousToolsConfig()) + engine, tools, err = _build_phase_engine( + server_config=server_cfg, + llm_gateway=MagicMock(), + execution_mode=ExecutionMode.PLAN_EXEC, + base_tools=[], + session_id="test", + ) + assert err is None + assert engine is not None + assert tools is not None + # AdvancePhaseTool appended to the base tool list. + assert len(tools) == 1 + assert type(tools[0]).__name__ == "AdvancePhaseTool" diff --git a/tests/unit/test_react_autonomy.py b/tests/unit/test_react_autonomy.py new file mode 100644 index 0000000..bb56f56 --- /dev/null +++ b/tests/unit/test_react_autonomy.py @@ -0,0 +1,302 @@ +"""Unit tests for ReActEngine autonomy mode (IQ-Boost U2, R6-R9). + +Verifies the PLAN_EXEC autonomy gate: +- Dangerous tools (per DangerousToolsConfig) trigger confirmation_request + BEFORE first execution in autonomy mode. +- Non-dangerous tools execute directly without confirmation. +- Non-autonomy mode preserves existing behavior (gate is a no-op). +- Disabled whitelist (enabled=False) allows all tools. +- Approved dangerous tool re-executes with _skip_dangerous_check=True. +- Rejected dangerous tool returns permission_denied result. +""" + +from __future__ import annotations + +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from agentkit.core.react import ReActEngine +from agentkit.server.config import DangerousToolsConfig + + +# --------------------------------------------------------------------------- +# _check_autonomy_gate — backward compatibility (no-op when disabled) +# --------------------------------------------------------------------------- + + +class TestAutonomyGateBackwardCompat: + """When autonomy_mode=False or dangerous_tools_config=None, the gate is a + transparent no-op — existing callers see no behavior change.""" + + @pytest.mark.asyncio + async def test_no_autonomy_mode_returns_should_execute(self): + """Default engine (autonomy_mode=False) → gate passes through.""" + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(), + autonomy_mode=False, + ) + should_exec, events, result = await engine._check_autonomy_gate( + "rm", {"path": "/tmp/x"}, [], step=1, confirmation_handler=None + ) + assert should_exec is True + assert events == [] + assert result is None + + @pytest.mark.asyncio + async def test_no_dangerous_config_returns_should_execute(self): + """autonomy_mode=True but no config → gate passes through (no whitelist + to check against).""" + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=None, + autonomy_mode=True, + ) + should_exec, events, result = await engine._check_autonomy_gate( + "rm", {"path": "/tmp/x"}, [], step=1, confirmation_handler=None + ) + assert should_exec is True + assert events == [] + assert result is None + + @pytest.mark.asyncio + async def test_disabled_whitelist_allows_all(self): + """enabled=False → is_dangerous always returns False → all tools pass.""" + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(enabled=False), + autonomy_mode=True, + ) + should_exec, events, result = await engine._check_autonomy_gate( + "rm", {"path": "/etc"}, [], step=1, confirmation_handler=None + ) + assert should_exec is True + assert events == [] + + +# --------------------------------------------------------------------------- +# _check_autonomy_gate — non-dangerous tool in autonomy mode +# --------------------------------------------------------------------------- + + +class TestAutonomyGateNonDangerous: + """In autonomy mode, non-dangerous tools execute without confirmation.""" + + @pytest.mark.asyncio + async def test_non_dangerous_tool_passes_through(self): + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(), + autonomy_mode=True, + ) + should_exec, events, result = await engine._check_autonomy_gate( + "read_file", {"path": "/tmp/x"}, [], step=1, confirmation_handler=None + ) + assert should_exec is True + assert events == [] + assert result is None + + @pytest.mark.asyncio + async def test_custom_pattern_non_match_passes(self): + """Tool not matching any custom pattern passes through.""" + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(tool_patterns=[r"^rm$", r"^kubectl"]), + autonomy_mode=True, + ) + should_exec, events, result = await engine._check_autonomy_gate( + "search", {"query": "foo"}, [], step=1, confirmation_handler=None + ) + assert should_exec is True + assert events == [] + + +# --------------------------------------------------------------------------- +# _check_autonomy_gate — dangerous tool triggers confirmation +# --------------------------------------------------------------------------- + + +class TestAutonomyGateDangerous: + """Dangerous tools in autonomy mode trigger confirmation_request before + execution.""" + + @pytest.mark.asyncio + async def test_dangerous_tool_emits_confirmation_request(self): + """rm matches ^rm$ pattern → confirmation_request event emitted.""" + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(), + autonomy_mode=True, + ) + fake_tool = MagicMock() + fake_tool.safe_execute = AsyncMock(return_value={"output": "deleted"}) + engine._find_tool = lambda name, tools: fake_tool + + handler = AsyncMock(return_value=True) + should_exec, events, result = await engine._check_autonomy_gate( + "rm", {"path": "/tmp/x"}, [fake_tool], step=1, confirmation_handler=handler + ) + + assert should_exec is False # Gate handled execution + assert len(events) == 2 # confirmation_request + confirmation_result + assert events[0].event_type == "confirmation_request" + assert events[0].data["tool_name"] == "rm" + assert events[0].data["confirmation_id"].startswith("autonomy:rm:") + assert events[1].event_type == "confirmation_result" + assert events[1].data["approved"] is True + # Tool executed with _skip_dangerous_check=True + fake_tool.safe_execute.assert_awaited_once_with(path="/tmp/x", _skip_dangerous_check=True) + assert result == {"output": "deleted"} + + @pytest.mark.asyncio + async def test_dangerous_tool_rejected_returns_permission_denied(self): + """User rejects → permission_denied result, tool NOT executed.""" + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(), + autonomy_mode=True, + ) + fake_tool = MagicMock() + fake_tool.safe_execute = AsyncMock(return_value={"output": "should not run"}) + engine._find_tool = lambda name, tools: fake_tool + + handler = AsyncMock(return_value=False) + should_exec, events, result = await engine._check_autonomy_gate( + "rm", {"path": "/tmp/x"}, [fake_tool], step=1, confirmation_handler=handler + ) + + assert should_exec is False + assert len(events) == 2 + assert events[1].data["approved"] is False + fake_tool.safe_execute.assert_not_awaited() + assert result["error_type"] == "permission_denied" + assert result["exit_code"] == 126 + + @pytest.mark.asyncio + async def test_dangerous_tool_no_handler_auto_rejects(self): + """No confirmation_handler → auto-reject (safe default).""" + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(), + autonomy_mode=True, + ) + engine._find_tool = lambda name, tools: MagicMock() + + should_exec, events, result = await engine._check_autonomy_gate( + "kubectl_delete", {"ns": "prod"}, [], step=2, confirmation_handler=None + ) + + assert should_exec is False + assert events[1].data["approved"] is False + assert result["error_type"] == "permission_denied" + + @pytest.mark.asyncio + async def test_dangerous_tool_handler_exception_auto_rejects(self): + """Handler raises → auto-reject (does not crash the engine).""" + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(), + autonomy_mode=True, + ) + engine._find_tool = lambda name, tools: MagicMock() + + async def failing_handler(cid, cmd, reason): + raise RuntimeError("handler crashed") + + should_exec, events, result = await engine._check_autonomy_gate( + "deploy", {"env": "prod"}, [], step=3, confirmation_handler=failing_handler + ) + + assert should_exec is False + assert events[1].data["approved"] is False + assert result["error_type"] == "permission_denied" + + @pytest.mark.asyncio + async def test_payment_glob_pattern_matches(self): + """payment_* pattern matches payment_charge tool.""" + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(), + autonomy_mode=True, + ) + fake_tool = MagicMock() + fake_tool.safe_execute = AsyncMock(return_value={"ok": True}) + engine._find_tool = lambda name, tools: fake_tool + + handler = AsyncMock(return_value=True) + should_exec, events, result = await engine._check_autonomy_gate( + "payment_charge", + {"amount": 100}, + [fake_tool], + step=1, + confirmation_handler=handler, + ) + + assert should_exec is False + assert events[0].event_type == "confirmation_request" + assert result == {"ok": True} + + @pytest.mark.asyncio + async def test_dangerous_tool_not_found_returns_error(self): + """Approved but tool not found → error result.""" + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(), + autonomy_mode=True, + ) + engine._find_tool = lambda name, tools: None + + handler = AsyncMock(return_value=True) + should_exec, events, result = await engine._check_autonomy_gate( + "rm", {"path": "/x"}, [], step=1, confirmation_handler=handler + ) + + assert should_exec is False + assert "error" in result + assert "not found" in result["error"] + + @pytest.mark.asyncio + async def test_dangerous_tool_execution_failure_returns_error(self): + """Approved but tool.safe_execute raises → error dict.""" + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(), + autonomy_mode=True, + ) + fake_tool = MagicMock() + fake_tool.safe_execute = AsyncMock(side_effect=RuntimeError("boom")) + engine._find_tool = lambda name, tools: fake_tool + + handler = AsyncMock(return_value=True) + should_exec, events, result = await engine._check_autonomy_gate( + "rm", {"path": "/x"}, [fake_tool], step=1, confirmation_handler=handler + ) + + assert should_exec is False + assert "error" in result + assert "tool_execution_failed" in result.get("error_code", "") + + +# --------------------------------------------------------------------------- +# Constructor wiring +# --------------------------------------------------------------------------- + + +class TestAutonomyConstructorWiring: + """Verify __init__ stores autonomy state correctly.""" + + def test_defaults_are_off(self): + engine = ReActEngine(llm_gateway=MagicMock()) + assert engine._autonomy_mode is False + assert engine._dangerous_tools_config is None + + def test_autonomy_mode_stored(self): + cfg = DangerousToolsConfig() + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=cfg, + autonomy_mode=True, + ) + assert engine._autonomy_mode is True + assert engine._dangerous_tools_config is cfg From 7627403a8a49acc1679033e38c3ff68d05383e2d Mon Sep 17 00:00:00 2001 From: Chiguyong Date: Mon, 6 Jul 2026 13:34:31 +0800 Subject: [PATCH 04/10] feat(react): U3 PLAN_EXEC autonomy pause + autonomy_paused event (R10) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add autonomy pause mechanism for PLAN_EXEC mode: when execution exceeds timeout (default 30min) or hits consecutive failures (default 3), engine emits `autonomy_paused` event and awaits user resume via resume_handler. - DangerousToolsConfig: add autonomy_timeout_minutes + max_consecutive_failures - ReActEngine: add _check_autonomy_pause() + _track_tool_result_for_autonomy() Called in serial + text-parsed tool paths (between tool_call event and autonomy gate). resume_handler=None → auto-resume (REST/testing). - chat.py: WebSocket `resume` message handler + _resume_handler closure (Future-based, 30min hard timeout). pending_autonomy_resumes cleanup in finally block. autonomy_paused event forwarded to frontend. Tests: 21 new tests (track_result + no_trigger + timeout + failures + config parsing + handler exception). All pass; U1/U2 no regression. --- src/agentkit/core/react.py | 151 +++++++++++- src/agentkit/server/config.py | 9 + src/agentkit/server/routes/chat.py | 67 ++++++ tests/unit/test_autonomy_paused.py | 366 +++++++++++++++++++++++++++++ 4 files changed, 592 insertions(+), 1 deletion(-) create mode 100644 tests/unit/test_autonomy_paused.py diff --git a/src/agentkit/core/react.py b/src/agentkit/core/react.py index b3ef4eb..f7f14c9 100644 --- a/src/agentkit/core/react.py +++ b/src/agentkit/core/react.py @@ -147,7 +147,7 @@ class ReActResult: class ReActEvent: """ReAct 执行事件""" - event_type: str # "thinking","token","tool_call","tool_result","confirmation_request","confirmation_result","phase_violation","step","final_answer","final_result","error" + event_type: str # "thinking","token","tool_call","tool_result","confirmation_request","confirmation_result","phase_violation","step","final_answer","final_result","error","autonomy_paused","spec_review_request","spec_review_reply","expert_step","expert_result","team_synthesis","team_synthesis_chunk","phase_changed" step: int data: dict[str, object] = field(default_factory=dict) timestamp: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat()) @@ -317,6 +317,11 @@ class ReActEngine: # confirmation_request; non-dangerous tools execute without confirmation. self._dangerous_tools_config = dangerous_tools_config self._autonomy_mode: bool = autonomy_mode + # IQ-Boost/U3/R10: autonomy pause state. _autonomy_started_at is set in + # _execute_loop when autonomy mode engages; _consecutive_failures tracks + # tool errors. When either threshold is exceeded → autonomy_paused event. + self._autonomy_started_at: float = 0.0 + self._consecutive_failures: int = 0 def reset(self) -> None: """Reset internal state for reuse across conversations. @@ -772,6 +777,10 @@ class ReActEngine: stream: bool = False, effective_timeout: float = 0.0, pitfall_warnings: "list[PitfallWarning] | None" = None, + # IQ-Boost/U3: autonomy pause resume handler. When autonomy_paused is + # triggered, the engine calls resume_handler(resume_token, reason) which + # blocks until the user sends ``resume`` (True) or cancels (False). + resume_handler: Callable[..., Awaitable[object]] | None = None, ) -> AsyncGenerator[ReActEvent, None]: """Unified ReAct loop — async generator yielding ReActEvent objects. @@ -798,6 +807,10 @@ class ReActEngine: # below so the next execute() without a restore resets normally. if not self._state_restored: self.reset() + # IQ-Boost/U3: start autonomy timer when in autonomy mode. + if self._autonomy_mode: + self._autonomy_started_at = time.time() + self._consecutive_failures = 0 tools = tools or [] if tools: tools = self._maybe_add_tool_search(tools) @@ -1258,6 +1271,23 @@ class ReActEngine: data={"tool_name": tc.name, "arguments": tc.arguments}, ) + # IQ-Boost/U3/R10: autonomy pause check (timeout/failures). + # If paused and resume_handler returns False (cancel), + # break out of the tool_calls loop. + _progress = { + "step": step, + "tool_name": tc.name, + "total_steps": len(trajectory), + } + should_continue, pause_events = await self._check_autonomy_pause( + step, _progress, resume_handler + ) + for _pev in pause_events: + yield _pev + if not should_continue: + # User cancelled during pause — stop execution. + break + # IQ-Boost/U2/R7: autonomy mode pre-execution gate. # Dangerous tools (per config) get a confirmation_request # BEFORE first execution; non-dangerous tools fall through. @@ -1381,6 +1411,9 @@ class ReActEngine: tool_duration_ms = int((time.monotonic() - tool_start) * 1000) + # IQ-Boost/U3/R10: track tool failures for autonomy pause. + self._track_tool_result_for_autonomy(tool_result) + react_step = ReActStep( step=step, action="tool_call", @@ -1474,6 +1507,9 @@ class ReActEngine: ) tool_duration_ms = int((time.monotonic() - tool_start) * 1000) + # IQ-Boost/U3/R10: track tool failures for autonomy pause. + self._track_tool_result_for_autonomy(tool_result) + react_step = ReActStep( step=step, action="tool_call", @@ -1866,6 +1902,8 @@ class ReActEngine: timeout_seconds: float | None = None, confirmation_handler: Callable[..., Awaitable[object]] | None = None, pitfall_warnings: "list[PitfallWarning] | None" = None, + # IQ-Boost/U3: autonomy resume handler (see _execute_loop). + resume_handler: Callable[..., Awaitable[object]] | None = None, ) -> AsyncGenerator[ReActEvent, None]: """Execute ReAct loop, yielding ReActEvent objects. @@ -1878,6 +1916,7 @@ class ReActEngine: compressor: 压缩策略，None 时使用实例默认压缩器 timeout_seconds: 超时秒数，0 表示无超时，None 使用 default_timeout pitfall_warnings: U7/R12 — HIGH 级别避坑预警，注入 system prompt + resume_handler: U3/R10 — autonomy pause resume callback """ effective_compressor = compressor if compressor is not None else self._compressor effective_timeout = ( @@ -1902,6 +1941,7 @@ class ReActEngine: stream=True, effective_timeout=effective_timeout, pitfall_warnings=pitfall_warnings, + resume_handler=resume_handler, ): yield event @@ -2338,6 +2378,115 @@ class ReActEngine: }, ) + async def _check_autonomy_pause( + self, + step: int, + progress: dict[str, object], + resume_handler: Callable[..., Awaitable[object]] | None, + ) -> tuple[bool, list[ReActEvent]]: + """IQ-Boost/U3/R10: check autonomy pause conditions before tool execution. + + Returns ``(should_continue, events)``: + - should_continue=True → no pause needed; proceed with tool execution + - should_continue=False → autonomy_paused yielded; caller must break + the loop (resume_handler already returned False / cancelled) OR + retry (resume_handler returned True — counters reset, proceed). + + When pause is triggered: + 1. Yield ``autonomy_paused`` event with resume_token + reason + progress. + 2. Call ``resume_handler(resume_token, reason)`` — blocks until user + sends ``resume`` (returns True) or cancels (returns False). + 3. On resume: reset ``_autonomy_started_at`` + ``_consecutive_failures``, + return should_continue=True to let the caller retry the tool. + 4. On cancel: return should_continue=False to break the loop. + + If ``resume_handler`` is None (non-WS callers, tests), auto-resume + immediately (no blocking) — the pause event is still yielded for + observability, but the loop continues without waiting. + """ + events: list[ReActEvent] = [] + if not self._autonomy_mode or self._dangerous_tools_config is None: + return (True, events) + + cfg = self._dangerous_tools_config + reason: str | None = None + # Check timeout (0 = disabled) + if ( + cfg.autonomy_timeout_minutes > 0 + and self._autonomy_started_at > 0 + and time.time() - self._autonomy_started_at > cfg.autonomy_timeout_minutes * 60 + ): + reason = "timeout" + # Check consecutive failures (0 = disabled) + elif ( + cfg.max_consecutive_failures > 0 + and self._consecutive_failures >= cfg.max_consecutive_failures + ): + reason = "consecutive_failures" + + if reason is None: + return (True, events) + + resume_token = f"autonomy_pause:{reason}:{step}" + events.append( + ReActEvent( + event_type="autonomy_paused", + step=step, + data={ + "reason": reason, + "progress": progress, + "resume_token": resume_token, + "consecutive_failures": self._consecutive_failures, + "elapsed_seconds": ( + time.time() - self._autonomy_started_at + if self._autonomy_started_at > 0 + else 0 + ), + }, + ) + ) + + # Wait for user resume or cancel + if resume_handler is None: + # Non-WS caller (tests, REST) — auto-resume without blocking. + logger.warning("autonomy_paused (%s) with no resume_handler — auto-resuming", reason) + self._autonomy_started_at = time.time() + self._consecutive_failures = 0 + return (True, events) + + try: + should_resume = await resume_handler(resume_token, reason) + except asyncio.CancelledError: + raise + except Exception as e: + logger.warning(f"Autonomy resume handler error: {e}") + should_resume = False + + if should_resume: + # Reset counters and continue + self._autonomy_started_at = time.time() + self._consecutive_failures = 0 + return (True, events) + # Cancelled by user — break the loop + return (False, events) + + def _track_tool_result_for_autonomy(self, tool_result: object) -> None: + """U3/R10: track tool failures for consecutive_failures threshold. + + Called after each tool execution. If the result is an error dict, + increment the counter; otherwise reset to 0. + """ + if not self._autonomy_mode: + return + if isinstance(tool_result, dict) and ( + "error" in tool_result + or tool_result.get("is_error") is True + or tool_result.get("error_type") is not None + ): + self._consecutive_failures += 1 + else: + self._consecutive_failures = 0 + async def _execute_tool_with_confirmation( self, tc: object, diff --git a/src/agentkit/server/config.py b/src/agentkit/server/config.py index ee34e06..d2d85be 100644 --- a/src/agentkit/server/config.py +++ b/src/agentkit/server/config.py @@ -65,6 +65,10 @@ class DangerousToolsConfig: confirmation request before execution. KTD2: conservative whitelist — ponytail ceiling = missed dangerous ops not in the list; upgrade path = LLM-assisted classification. + + IQ-Boost/U3: also carries autonomy pause thresholds (timeout + consecutive + failures). Kept here to avoid a separate config section — all autonomy + behavior in one place. """ enabled: bool = True @@ -81,6 +85,9 @@ class DangerousToolsConfig: r"^payment_", ] ) + # U3/R10: autonomy pause thresholds. 0 = disabled (no pause). + autonomy_timeout_minutes: int = 30 + max_consecutive_failures: int = 3 def is_dangerous(self, tool_name: str) -> bool: """Check if a tool name matches any dangerous pattern.""" @@ -96,6 +103,8 @@ class DangerousToolsConfig: return cls( enabled=data.get("enabled", True), tool_patterns=data.get("tool_patterns") or cls().tool_patterns, + autonomy_timeout_minutes=data.get("autonomy_timeout_minutes", 30), + max_consecutive_failures=data.get("max_consecutive_failures", 3), ) diff --git a/src/agentkit/server/routes/chat.py b/src/agentkit/server/routes/chat.py index f07866a..089fce2 100644 --- a/src/agentkit/server/routes/chat.py +++ b/src/agentkit/server/routes/chat.py @@ -1019,6 +1019,9 @@ async def chat_websocket(websocket: WebSocket, session_id: str) -> None: # U8/R8: pending spec-review futures keyed by spec_review_id. Resolved # by the spec_review_reply client message; cancelled on WS teardown. pending_spec_reviews: dict[str, asyncio.Future] = {} + # IQ-Boost/U3: pending autonomy-resume futures keyed by resume_token. + # Resolved by the ``resume`` client message; cancelled on WS teardown. + pending_autonomy_resumes: dict[str, asyncio.Future] = {} chat_manager.add(session_id, websocket, pending_replies) cancellation_token = CancellationToken() @@ -1101,6 +1104,7 @@ async def chat_websocket(websocket: WebSocket, session_id: str) -> None: pending_replies, pending_confirmations, pending_spec_reviews, + pending_autonomy_resumes, model_override=model, ) ) @@ -1156,6 +1160,22 @@ async def chat_websocket(websocket: WebSocket, session_id: str) -> None: cancellation_token.cancel() await websocket.send_json({"type": "result", "data": {"status": "cancelled"}}) + elif msg_type == "resume": + # IQ-Boost/U3/R10: Resume autonomy-paused execution. The client + # sends {resume_token}. An unknown token is logged + ignored. + resume_token = msg.get("resume_token") + logger.info(f"Received resume for token: {resume_token!r}") + if resume_token and resume_token in pending_autonomy_resumes: + fut = pending_autonomy_resumes[resume_token] + if not fut.done(): + fut.set_result(True) + else: + logger.warning(f"resume token {resume_token!r} already resolved") + else: + logger.warning( + f"resume token {resume_token!r} not found in pending_autonomy_resumes" + ) + elif msg_type == "ping": await websocket.send_json({"type": "pong"}) @@ -1180,6 +1200,10 @@ async def chat_websocket(websocket: WebSocket, session_id: str) -> None: for fut in pending_spec_reviews.values(): if not fut.done(): fut.cancel() + # IQ-Boost/U3: cancel any pending autonomy-resume futures. + for fut in pending_autonomy_resumes.values(): + if not fut.done(): + fut.cancel() chat_manager.remove(session_id, websocket) @@ -1192,6 +1216,8 @@ async def _handle_chat_message( pending_replies: dict[str, asyncio.Future], pending_confirmations: dict[str, asyncio.Future] | None = None, pending_spec_reviews: dict[str, asyncio.Future] | None = None, + # IQ-Boost/U3: pending autonomy-resume futures (keyed by resume_token). + pending_autonomy_resumes: dict[str, asyncio.Future] | None = None, model_override: str | None = None, ) -> None: """Handle a user message: append to session, execute Agent, stream events. @@ -1583,6 +1609,35 @@ async def _handle_chat_message( if hasattr(react_engine, "_spec_review_handler"): react_engine._spec_review_handler = _spec_review_handler + # IQ-Boost/U3/R10: autonomy resume handler. When the engine triggers + # autonomy_paused (timeout/consecutive_failures), it calls this handler + # which blocks until the user sends ``resume`` (True) or the WS disconnects + # (Future cancelled → False). The engine already yielded the + # ``autonomy_paused`` event (forwarded to the frontend by the event loop + # below); this handler just provides the blocking mechanism. + _pending_autonomy_resumes = ( + pending_autonomy_resumes if pending_autonomy_resumes is not None else {} + ) + + async def _resume_handler(resume_token: str, reason: str) -> bool: + """Block until the user sends ``resume`` for the given token.""" + loop = asyncio.get_running_loop() + future: asyncio.Future[bool] = loop.create_future() + _pending_autonomy_resumes[resume_token] = future + logger.info(f"Autonomy paused ({reason}), waiting for resume: {resume_token}") + try: + # Wait up to 30 minutes for user resume (long task availability). + result = await asyncio.wait_for(future, timeout=1800.0) + return bool(result) + except asyncio.TimeoutError: + logger.warning(f"Autonomy resume {resume_token} timed out (30 min)") + return False + except asyncio.CancelledError: + logger.warning(f"Autonomy resume {resume_token} cancelled") + return False + finally: + _pending_autonomy_resumes.pop(resume_token, None) + logger.info( f"Chat session {session_id}: executing with {len(routing.tools)} tools, model={routing.model}, skill={routing.skill_name}" ) @@ -1601,6 +1656,7 @@ async def _handle_chat_message( system_prompt=routing.system_prompt, cancellation_token=cancellation_token, confirmation_handler=_confirmation_handler, + resume_handler=_resume_handler, ): if event.event_type == "final_answer": # Flush any buffered tokens as a single write @@ -1658,6 +1714,17 @@ async def _handle_chat_message( "data": event.data, } ) + elif event.event_type == "autonomy_paused": + # IQ-Boost/U3/R10: forward autonomy pause to the frontend. + # The _resume_handler closure is already blocking the engine + # waiting for the user's ``resume`` message. The frontend + # shows a pause card with reason + progress + resume button. + await websocket.send_json( + { + "type": "autonomy_paused", + "data": event.data, + } + ) elif event.event_type == "spec_review_request": # U8/R8: the _spec_review_handler closure already sent this # request directly to the frontend (it owns the spec_review_id diff --git a/tests/unit/test_autonomy_paused.py b/tests/unit/test_autonomy_paused.py new file mode 100644 index 0000000..c4b6ef5 --- /dev/null +++ b/tests/unit/test_autonomy_paused.py @@ -0,0 +1,366 @@ +"""Unit tests for autonomy_paused event + timeout/failure tracking (IQ-Boost U3, R10). + +Verifies: +- Timeout triggers autonomy_paused with reason="timeout" +- Consecutive failures (3x) trigger autonomy_paused with reason="consecutive_failures" +- resume_handler returns True → counters reset, execution continues +- resume_handler returns False → execution stops (cancel) +- resume_handler=None → auto-resume (non-blocking, for tests/REST) +- Non-autonomy mode → no pause (gate is a no-op) +- Disabled thresholds (0) → no pause +- _track_tool_result_for_autonomy increments on error, resets on success +""" + +from __future__ import annotations + +import time +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from agentkit.core.react import ReActEngine +from agentkit.server.config import DangerousToolsConfig + + +# --------------------------------------------------------------------------- +# _track_tool_result_for_autonomy +# --------------------------------------------------------------------------- + + +class TestTrackToolResult: + """Failure tracking increments on error, resets on success.""" + + def test_success_resets_counter(self): + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(), + autonomy_mode=True, + ) + engine._consecutive_failures = 2 + engine._track_tool_result_for_autonomy({"output": "ok"}) + assert engine._consecutive_failures == 0 + + def test_error_increments_counter(self): + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(), + autonomy_mode=True, + ) + engine._consecutive_failures = 1 + engine._track_tool_result_for_autonomy({"error": "boom"}) + assert engine._consecutive_failures == 2 + + def test_is_error_flag_increments(self): + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(), + autonomy_mode=True, + ) + engine._track_tool_result_for_autonomy({"is_error": True, "output": ""}) + assert engine._consecutive_failures == 1 + + def test_error_type_increments(self): + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(), + autonomy_mode=True, + ) + engine._track_tool_result_for_autonomy({"error_type": "permission_denied"}) + assert engine._consecutive_failures == 1 + + def test_non_dict_result_resets(self): + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(), + autonomy_mode=True, + ) + engine._consecutive_failures = 2 + engine._track_tool_result_for_autonomy("plain string result") + assert engine._consecutive_failures == 0 + + def test_no_tracking_when_not_autonomy(self): + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(), + autonomy_mode=False, + ) + engine._consecutive_failures = 0 + engine._track_tool_result_for_autonomy({"error": "boom"}) + assert engine._consecutive_failures == 0 # Unchanged + + +# --------------------------------------------------------------------------- +# _check_autonomy_pause — no pause conditions +# --------------------------------------------------------------------------- + + +class TestAutonomyPauseNoTrigger: + """When conditions are not met, _check_autonomy_pause returns should_continue=True.""" + + @pytest.mark.asyncio + async def test_no_autonomy_mode_no_pause(self): + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig(), + autonomy_mode=False, + ) + should_continue, events = await engine._check_autonomy_pause( + step=1, progress={}, resume_handler=None + ) + assert should_continue is True + assert events == [] + + @pytest.mark.asyncio + async def test_no_config_no_pause(self): + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=None, + autonomy_mode=True, + ) + should_continue, events = await engine._check_autonomy_pause( + step=1, progress={}, resume_handler=None + ) + assert should_continue is True + assert events == [] + + @pytest.mark.asyncio + async def test_below_thresholds_no_pause(self): + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig( + autonomy_timeout_minutes=30, + max_consecutive_failures=3, + ), + autonomy_mode=True, + ) + engine._autonomy_started_at = time.time() # Just started + engine._consecutive_failures = 1 # Below threshold + should_continue, events = await engine._check_autonomy_pause( + step=1, progress={}, resume_handler=None + ) + assert should_continue is True + assert events == [] + + @pytest.mark.asyncio + async def test_disabled_thresholds_no_pause(self): + """0 = disabled for both thresholds.""" + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig( + autonomy_timeout_minutes=0, + max_consecutive_failures=0, + ), + autonomy_mode=True, + ) + # Even with expired time + many failures, disabled = no pause. + engine._autonomy_started_at = time.time() - 999999 + engine._consecutive_failures = 99 + should_continue, events = await engine._check_autonomy_pause( + step=1, progress={}, resume_handler=None + ) + assert should_continue is True + assert events == [] + + +# --------------------------------------------------------------------------- +# _check_autonomy_pause — timeout trigger +# --------------------------------------------------------------------------- + + +class TestAutonomyPauseTimeout: + """Timeout triggers autonomy_paused with reason='timeout'.""" + + @pytest.mark.asyncio + async def test_timeout_triggers_pause_auto_resume(self): + """resume_handler=None → auto-resume (non-blocking).""" + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig( + autonomy_timeout_minutes=1, # 1 minute + max_consecutive_failures=99, # Disabled for this test + ), + autonomy_mode=True, + ) + # Simulate started 2 minutes ago (exceeds 1-min timeout). + engine._autonomy_started_at = time.time() - 120 + + should_continue, events = await engine._check_autonomy_pause( + step=5, progress={"step": 5}, resume_handler=None + ) + + # Auto-resume → should_continue=True + assert should_continue is True + assert len(events) == 1 + assert events[0].event_type == "autonomy_paused" + assert events[0].data["reason"] == "timeout" + assert "resume_token" in events[0].data + assert events[0].data["elapsed_seconds"] >= 120 + # Counters reset after auto-resume + assert engine._consecutive_failures == 0 + assert engine._autonomy_started_at > time.time() - 1 # Fresh timer + + @pytest.mark.asyncio + async def test_timeout_triggers_pause_user_resumes(self): + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig( + autonomy_timeout_minutes=1, + max_consecutive_failures=99, + ), + autonomy_mode=True, + ) + engine._autonomy_started_at = time.time() - 120 + + resume_handler = AsyncMock(return_value=True) + should_continue, events = await engine._check_autonomy_pause( + step=3, progress={}, resume_handler=resume_handler + ) + + assert should_continue is True + assert events[0].data["reason"] == "timeout" + resume_handler.assert_awaited_once() + # Counters reset + assert engine._consecutive_failures == 0 + + @pytest.mark.asyncio + async def test_timeout_triggers_pause_user_cancels(self): + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig( + autonomy_timeout_minutes=1, + max_consecutive_failures=99, + ), + autonomy_mode=True, + ) + engine._autonomy_started_at = time.time() - 120 + + resume_handler = AsyncMock(return_value=False) + should_continue, events = await engine._check_autonomy_pause( + step=3, progress={}, resume_handler=resume_handler + ) + + assert should_continue is False # Cancelled + assert events[0].data["reason"] == "timeout" + + +# --------------------------------------------------------------------------- +# _check_autonomy_pause — consecutive failures trigger +# --------------------------------------------------------------------------- + + +class TestAutonomyPauseConsecutiveFailures: + """3 consecutive failures trigger autonomy_paused.""" + + @pytest.mark.asyncio + async def test_consecutive_failures_triggers_pause(self): + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig( + autonomy_timeout_minutes=0, # Disabled + max_consecutive_failures=3, + ), + autonomy_mode=True, + ) + engine._autonomy_started_at = time.time() + engine._consecutive_failures = 3 # At threshold + + resume_handler = AsyncMock(return_value=True) + should_continue, events = await engine._check_autonomy_pause( + step=5, progress={"step": 5}, resume_handler=resume_handler + ) + + assert should_continue is True # Resumed + assert len(events) == 1 + assert events[0].event_type == "autonomy_paused" + assert events[0].data["reason"] == "consecutive_failures" + assert events[0].data["consecutive_failures"] == 3 + # Reset after resume + assert engine._consecutive_failures == 0 + + @pytest.mark.asyncio + async def test_below_failure_threshold_no_pause(self): + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig( + autonomy_timeout_minutes=0, + max_consecutive_failures=3, + ), + autonomy_mode=True, + ) + engine._autonomy_started_at = time.time() + engine._consecutive_failures = 2 # Below threshold + + should_continue, events = await engine._check_autonomy_pause( + step=1, progress={}, resume_handler=None + ) + + assert should_continue is True + assert events == [] + + @pytest.mark.asyncio + async def test_resume_handler_exception_cancels(self): + """If resume_handler raises, treat as cancel (should_continue=False).""" + engine = ReActEngine( + llm_gateway=MagicMock(), + dangerous_tools_config=DangerousToolsConfig( + autonomy_timeout_minutes=0, + max_consecutive_failures=3, + ), + autonomy_mode=True, + ) + engine._consecutive_failures = 3 + + async def failing_handler(token, reason): + raise RuntimeError("handler crashed") + + should_continue, events = await engine._check_autonomy_pause( + step=1, progress={}, resume_handler=failing_handler + ) + + assert should_continue is False # Cancelled due to exception + assert events[0].data["reason"] == "consecutive_failures" + + +# --------------------------------------------------------------------------- +# Config parsing +# --------------------------------------------------------------------------- + + +class TestAutonomyConfigParsing: + """DangerousToolsConfig parses autonomy thresholds from dict.""" + + def test_defaults(self): + cfg = DangerousToolsConfig() + assert cfg.autonomy_timeout_minutes == 30 + assert cfg.max_consecutive_failures == 3 + + def test_custom_values_from_dict(self): + cfg = DangerousToolsConfig.from_dict( + { + "enabled": True, + "autonomy_timeout_minutes": 60, + "max_consecutive_failures": 5, + } + ) + assert cfg.autonomy_timeout_minutes == 60 + assert cfg.max_consecutive_failures == 5 + + def test_disabled_values_from_dict(self): + cfg = DangerousToolsConfig.from_dict( + { + "autonomy_timeout_minutes": 0, + "max_consecutive_failures": 0, + } + ) + assert cfg.autonomy_timeout_minutes == 0 + assert cfg.max_consecutive_failures == 0 + + def test_partial_config_uses_defaults(self): + cfg = DangerousToolsConfig.from_dict({"enabled": True}) + assert cfg.autonomy_timeout_minutes == 30 + assert cfg.max_consecutive_failures == 3 + + def test_empty_dict_uses_defaults(self): + cfg = DangerousToolsConfig.from_dict({}) + assert cfg.autonomy_timeout_minutes == 30 + assert cfg.max_consecutive_failures == 3 From 81a35dac27b546687d6db6eb96a0e4a8bb68b534 Mon Sep 17 00:00:00 2001 From: Chiguyong Date: Mon, 6 Jul 2026 13:43:31 +0800 Subject: [PATCH 05/10] feat(experts): U4 TeamOrchestrator parallel independent subtasks (R1-R5) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add explicit support for parallel execution of dependency-free subtasks: - TeamPlan.get_independent_subtasks(): returns phases with depends_on==[] (introspection entry — topological_sort already groups them in layer 0) - TeamOrchestrator.MAX_INDEPENDENT_SUBTASKS=10: aligns with router.MAX_EXPERTS - _rebalance_independent_subtasks(): when Lead over-decomposes (>10 independent subtasks), re-decompose once with merge hint. Fallbacks: no gateway → keep original; LLM error → keep original; single-phase fallback → keep original (don't collapse 11→1); still over → return new (MAX_PHASES truncation handles it) Tests: 15 new tests covering get_independent_subtasks, topological_sort layer contract, SharedWorkspace path uniqueness, rebalance (5 paths), and full execute() integration. Existing TeamOrchestrator tests pass. --- src/agentkit/experts/orchestrator.py | 72 +++++ src/agentkit/experts/plan.py | 9 + tests/unit/test_team_parallel.py | 464 +++++++++++++++++++++++++++ 3 files changed, 545 insertions(+) create mode 100644 tests/unit/test_team_parallel.py diff --git a/src/agentkit/experts/orchestrator.py b/src/agentkit/experts/orchestrator.py index f325bc1..8aa501a 100644 --- a/src/agentkit/experts/orchestrator.py +++ b/src/agentkit/experts/orchestrator.py @@ -62,6 +62,9 @@ class TeamOrchestrator( MAX_DEBATE_ROUNDS = 4 # Hard cap on debate rounds per phase MAX_DEBATES = 3 # Hard cap on auto-inserted debate phases per execution DEFAULT_MAX_CONCURRENT_PHASES = 3 # 同层最大并发阶段数，避免 LLM 限流洪峰 + # IQ-Boost/U4 (R4): aligns with router.MAX_EXPERTS — if Lead decomposes + # more independent subtasks than this, re-decompose once with a merge hint. + MAX_INDEPENDENT_SUBTASKS = 10 STOP_COMMANDS = frozenset({"/stop", "停止", "stop", "结束"}) # G9/U4: RollbackExecutor default timeout for validation_command / rollback_command. # Override via constructor `rollback_timeout` from `rollback.default_timeout` config. @@ -197,6 +200,11 @@ class TeamOrchestrator( PlanPhase(name="执行", assigned_expert=lead.config.name, task_description=task) ] + # IQ-Boost/U4 (R4): if Lead over-decomposed independent subtasks beyond + # MAX_INDEPENDENT_SUBTASKS, ask Lead to re-decompose with a merge hint + # (one retry — further overflow falls through to MAX_PHASES truncation). + phases = await self._rebalance_independent_subtasks(lead, task, phases) + plan.phases = phases[: self.MAX_PHASES] # U3: Optionally add plan review debate before execution @@ -531,6 +539,70 @@ class TeamOrchestrator( self._team.set_status(TeamStatus.EXECUTING) return await self._run_pipeline(lead, plan, phase_results, task) + async def _rebalance_independent_subtasks( + self, lead: Expert, task: str, phases: list[PlanPhase] + ) -> list[PlanPhase]: + """IQ-Boost/U4 (R4): if phases contain more independent subtasks + (depends_on == []) than MAX_INDEPENDENT_SUBTASKS, ask Lead to + re-decompose with a merge hint. One retry only — further overflow + is handled by MAX_PHASES truncation in execute(). + + Returns the original phases if count is within limit or retry fails. + """ + # Count independent subtasks via a transient TeamPlan (avoids mutating + # the real plan before execute() finalizes phase list). + independent_count = sum(1 for ph in phases if not ph.depends_on) + if independent_count <= self.MAX_INDEPENDENT_SUBTASKS: + return phases + + logger.info( + f"U4: Lead decomposed {independent_count} independent subtasks " + f"(> {self.MAX_INDEPENDENT_SUBTASKS}), requesting re-decompose with merge hint" + ) + + gateway = self._get_llm_gateway(lead) + if not gateway: + # No gateway — can't re-decompose; rely on MAX_PHASES truncation. + return phases + + # Re-decompose with explicit merge hint + original_hint = ( + f"Previous decomposition produced {independent_count} independent subtasks " + f"(no dependencies), exceeding the {self.MAX_INDEPENDENT_SUBTASKS} limit. " + f"Please merge related subtasks so the total independent (depends_on=[]) " + f"subtasks is at most {self.MAX_INDEPENDENT_SUBTASKS}. Keep dependencies " + f"where natural." + ) + # Temporarily wrap task with hint — call _decompose_task with augmented task. + augmented_task = f"{task}\n\n[Re-decomposition hint]: {original_hint}" + try: + new_phases = await self._decompose_task(lead, augmented_task) + except (LLMProviderError, asyncio.TimeoutError, ConnectionError, ValueError) as e: + logger.warning(f"U4 re-decompose failed: {e}, keeping original phases") + return phases + + if not new_phases: + return phases + + new_independent = sum(1 for ph in new_phases if not ph.depends_on) + if new_independent <= self.MAX_INDEPENDENT_SUBTASKS: + # ponytail: detect single-phase fallback (LLM returned invalid + # JSON → _decompose_task returned degenerate single phase). + # Collapsing 11 subtasks to 1 is worse than truncation; keep + # original so MAX_PHASES handles it. + if len(new_phases) == 1 and new_independent < independent_count: + logger.info("U4: re-decompose fell back to single phase, keeping original") + return phases + logger.info(f"U4: re-decompose succeeded ({new_independent} independent subtasks)") + return new_phases + + # Still over limit — take the new decomposition anyway (truncation will cap it) + logger.info( + f"U4: re-decompose still has {new_independent} independent subtasks, " + f"proceeding with MAX_PHASES truncation" + ) + return new_phases + async def _decompose_task(self, lead: Expert, task: str) -> list[PlanPhase]: """Lead Expert decomposes task into phases using LLM. diff --git a/src/agentkit/experts/plan.py b/src/agentkit/experts/plan.py index 06f7b81..02160ea 100644 --- a/src/agentkit/experts/plan.py +++ b/src/agentkit/experts/plan.py @@ -443,3 +443,12 @@ class TeamPlan: in_degree[dep_id] -= 1 return layers + + def get_independent_subtasks(self) -> list[PlanPhase]: + """返回无依赖的子任务（depends_on == []）。 + + IQ-Boost/U4 (R1): 用于 Lead 分解后检查独立子任务数量是否超过 + MAX_EXPERTS。这些子任务会被 topological_sort 派发到 layer 0 并行执行 + （同层并行已有，本方法仅提供显式 introspection 入口）。 + """ + return [ph for ph in self.phases if not ph.depends_on] diff --git a/tests/unit/test_team_parallel.py b/tests/unit/test_team_parallel.py new file mode 100644 index 0000000..8bd8f1e --- /dev/null +++ b/tests/unit/test_team_parallel.py @@ -0,0 +1,464 @@ +"""U4: TeamOrchestrator 无依赖子任务并行模式单元测试 (R1-R5). + +Covers: +- TeamPlan.get_independent_subtasks() introspection +- _rebalance_independent_subtasks() MAX_EXPERTS enforcement (R4) +- topological_sort 同层并行 (existing behavior, validated here for U4 contract) +- SharedWorkspace 路径唯一性 (phase_id is UUID → no collision) +- 综合阶段等待所有并行子任务完成 (layer-sequential contract) +""" + +from __future__ import annotations + +import json +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from agentkit.core.handoff_transport import InProcessHandoffTransport +from agentkit.core.protocol import TaskResult, TaskStatus +from agentkit.experts.config import ExpertConfig +from agentkit.experts.expert import Expert +from agentkit.experts.orchestrator import TeamOrchestrator +from agentkit.experts.plan import PlanPhase, TeamPlan +from agentkit.experts.team import ExpertTeam + +from tests.unit.experts._helpers import ( + make_chat_stream_mock, + make_execute_stream_mock, +) + + +# ── Helpers ──────────────────────────────────────────────────────────── + + +def _make_expert_config(name: str, is_lead: bool = False) -> ExpertConfig: + return ExpertConfig( + name=name, + agent_type="expert", + persona="测试专家", + thinking_style="逻辑推理", + bound_skills=["skill_a"], + is_lead=is_lead, + task_mode="llm_generate", + prompt={"identity": "测试"}, + llm={"model": "default"}, + ) + + +def _make_mock_expert(name: str, is_lead: bool = False) -> MagicMock: + config = _make_expert_config(name=name, is_lead=is_lead) + expert = MagicMock(spec=Expert) + expert.config = config + expert.is_active = True + expert.team_id = None + expert.get_capabilities_summary.return_value = { + "name": name, + "persona": config.persona, + "thinking_style": config.thinking_style, + "bound_skills": config.bound_skills, + "is_lead": is_lead, + } + mock_agent = MagicMock() + mock_agent.execute = AsyncMock( + return_value=TaskResult( + task_id="test", + agent_name=name, + status=TaskStatus.COMPLETED.value, + output_data={"content": f"Result from {name}"}, + error_message=None, + started_at=None, + completed_at=None, + ) + ) + mock_agent.execute_stream = make_execute_stream_mock(f"Result from {name}") + mock_agent._llm_gateway = None + expert.agent = mock_agent + return expert + + +def _make_team_with_experts( + expert_names: list[str] | None = None, + lead_name: str = "lead", +) -> ExpertTeam: + team = ExpertTeam() + transport = AsyncMock(spec=InProcessHandoffTransport) + team._handoff_transport = transport + if expert_names is None: + expert_names = [lead_name, "member1", "member2"] + for name in expert_names: + is_lead = name == lead_name + expert = _make_mock_expert(name=name, is_lead=is_lead) + team._experts[name] = expert + if is_lead: + team._lead_expert_name = name + return team + + +def _make_phase( + id: str, name: str, assigned_expert: str, depends_on: list[str] | None = None +) -> PlanPhase: + return PlanPhase( + id=id, + name=name, + assigned_expert=assigned_expert, + task_description=f"task for {name}", + depends_on=depends_on or [], + ) + + +def _make_mock_llm_gateway_with_phases( + phases_responses: list[list[dict]], + synthesis_content: str = "综合结果", +) -> MagicMock: + """Gateway that returns successive decompositions then synthesis. + + phases_responses: each element is a list of phase dicts to return on + successive decomposition calls. After exhausted, returns synthesis. + """ + gateway = AsyncMock() + decomp_responses = [MagicMock(content=json.dumps(phases)) for phases in phases_responses] + synth_response = MagicMock(content=synthesis_content) + call_count = [0] + + async def chat_side_effect(messages, model=None, **kwargs): + call_count[0] += 1 + if call_count[0] <= len(decomp_responses): + return decomp_responses[call_count[0] - 1] + return synth_response + + gateway.chat = AsyncMock(side_effect=chat_side_effect) + gateway.chat_stream = make_chat_stream_mock(synthesis_content) + return gateway + + +# ── TeamPlan.get_independent_subtasks ───────────────────────────────── + + +class TestGetIndependentSubtasks: + """U4/R1: TeamPlan.get_independent_subtasks() returns phases with no + depends_on.""" + + def test_returns_empty_for_no_phases(self): + plan = TeamPlan(task="test", lead_expert="lead") + assert plan.get_independent_subtasks() == [] + + def test_returns_only_independent_phases(self): + plan = TeamPlan( + task="test", + lead_expert="lead", + phases=[ + _make_phase("p1", "A", "lead", depends_on=[]), + _make_phase("p2", "B", "member1", depends_on=["p1"]), + _make_phase("p3", "C", "member2", depends_on=[]), + _make_phase("p4", "D", "member1", depends_on=["p1", "p3"]), + ], + ) + independent = plan.get_independent_subtasks() + assert len(independent) == 2 + assert {ph.id for ph in independent} == {"p1", "p3"} + + def test_returns_all_when_no_dependencies(self): + plan = TeamPlan( + task="test", + lead_expert="lead", + phases=[ + _make_phase("p1", "A", "lead", depends_on=[]), + _make_phase("p2", "B", "member1", depends_on=[]), + _make_phase("p3", "C", "member2", depends_on=[]), + ], + ) + independent = plan.get_independent_subtasks() + assert len(independent) == 3 + + +# ── topological_sort 同层并行 contract (U4/R1, R3) ──────────────────── + + +class TestParallelLayerContract: + """U4/R1, R3: independent subtasks land in layer 0 (parallel); synthesis + waits for all layers to complete (layer-sequential).""" + + def test_three_independent_subtasks_in_layer_0(self): + plan = TeamPlan( + task="test", + lead_expert="lead", + phases=[ + _make_phase("p1", "A", "lead", depends_on=[]), + _make_phase("p2", "B", "member1", depends_on=[]), + _make_phase("p3", "C", "member2", depends_on=[]), + ], + ) + layers = plan.topological_sort() + assert len(layers) == 1 + assert len(layers[0]) == 3 + assert {ph.id for ph in layers[0]} == {"p1", "p2", "p3"} + + def test_shared_dependency_degrades_to_layered(self): + """R3: subtasks sharing an upstream dependency are NOT parallel with + that upstream — they land in later layers.""" + plan = TeamPlan( + task="test", + lead_expert="lead", + phases=[ + _make_phase("p1", "Base", "lead", depends_on=[]), + _make_phase("p2", "A", "member1", depends_on=["p1"]), + _make_phase("p3", "B", "member2", depends_on=["p1"]), + ], + ) + layers = plan.topological_sort() + assert len(layers) == 2 + assert len(layers[0]) == 1 # only Base + assert layers[0][0].id == "p1" + assert len(layers[1]) == 2 # A and B parallel (same layer) + assert {ph.id for ph in layers[1]} == {"p2", "p3"} + + def test_synthesis_waits_for_all_layers(self): + """R3: dependent phase in last layer — synthesis cannot run until + its layer completes (validated via topological_sort layer count).""" + plan = TeamPlan( + task="test", + lead_expert="lead", + phases=[ + _make_phase("p1", "A", "lead", depends_on=[]), + _make_phase("p2", "B", "member1", depends_on=["p1"]), + _make_phase("p3", "C", "member2", depends_on=["p2"]), + ], + ) + layers = plan.topological_sort() + assert len(layers) == 3 # strict sequential — no parallelism + + +# ── SharedWorkspace 路径唯一性 (U4/R2) ───────────────────────────────── + + +class TestSharedWorkspacePathUniqueness: + """U4/R2: parallel subtask output paths must not collide. PlanPhase.id + is UUID by default → unique per phase, ensuring + ``{plan_id}/phase/{phase_id}/output`` never overlaps.""" + + def test_default_phase_ids_are_unique(self): + phases = [ + PlanPhase(name=f"phase_{i}", assigned_expert="lead", task_description="t") + for i in range(5) + ] + ids = [ph.id for ph in phases] + assert len(set(ids)) == len(ids), "phase_ids must be unique" + + def test_independent_subtasks_have_unique_output_paths(self): + plan = TeamPlan( + task="test", + lead_expert="lead", + phases=[ + PlanPhase(name="A", assigned_expert="lead", task_description="t"), + PlanPhase(name="B", assigned_expert="member1", task_description="t"), + PlanPhase(name="C", assigned_expert="member2", task_description="t"), + ], + ) + independent = plan.get_independent_subtasks() + paths = {f"{plan.id}/phase/{ph.id}/output" for ph in independent} + assert len(paths) == len(independent) + + +# ── _rebalance_independent_subtasks (U4/R4 MAX_EXPERTS) ──────────────── + + +class TestRebalanceIndependentSubtasks: + """U4/R4: when Lead over-decomposes independent subtasks beyond + MAX_INDEPENDENT_SUBTASKS, request re-decompose with merge hint.""" + + @pytest.mark.asyncio + async def test_no_rebalance_when_within_limit(self): + """5 independent subtasks ≤ 10 → no re-decompose.""" + team = _make_team_with_experts() + orchestrator = TeamOrchestrator(team) + # 5 independent phases + phases = [_make_phase(f"p{i}", f"phase_{i}", "lead", depends_on=[]) for i in range(5)] + original_phases = list(phases) + + result = await orchestrator._rebalance_independent_subtasks( + lead=team.lead_expert, task="test", phases=phases + ) + assert result is original_phases or result == original_phases + + @pytest.mark.asyncio + async def test_rebalance_triggers_when_over_limit(self): + """11 independent subtasks > 10 → re-decompose with merge hint.""" + team = _make_team_with_experts() + orchestrator = TeamOrchestrator(team) + + # Original: 11 independent (over limit) + original_phases = [ + _make_phase(f"p{i}", f"phase_{i}", "lead", depends_on=[]) for i in range(11) + ] + # Re-decomposed: 8 independent (within limit) — merge succeeded + gateway = _make_mock_llm_gateway_with_phases( + phases_responses=[ + [ # first call (original _decompose_task already happened — + # but _rebalance calls _decompose_task again, returning this) + { + "name": f"merged_{i}", + "assigned_expert": "lead", + "task_description": "t", + "depends_on": [], + } + for i in range(8) + ], + ] + ) + team._experts["lead"].agent._llm_gateway = gateway + + result = await orchestrator._rebalance_independent_subtasks( + lead=team.lead_expert, task="test", phases=original_phases + ) + # Should return the re-decomposed phases (8 independent) + assert len(result) == 8 + assert all(not ph.depends_on for ph in result) + # Original phases should NOT be returned + assert {ph.id for ph in result}.isdisjoint({ph.id for ph in original_phases}) + + @pytest.mark.asyncio + async def test_rebalance_no_gateway_returns_original(self): + """No LLM gateway → can't re-decompose, return original.""" + team = _make_team_with_experts() + orchestrator = TeamOrchestrator(team) + # gateway is None by default in _make_mock_expert + original_phases = [ + _make_phase(f"p{i}", f"phase_{i}", "lead", depends_on=[]) for i in range(11) + ] + + result = await orchestrator._rebalance_independent_subtasks( + lead=team.lead_expert, task="test", phases=original_phases + ) + assert result is original_phases + + @pytest.mark.asyncio + async def test_rebalance_still_over_limit_returns_new_anyway(self): + """Re-decompose still produces >MAX independent → return new phases + (MAX_PHASES truncation handles it in execute()).""" + team = _make_team_with_experts() + orchestrator = TeamOrchestrator(team) + + original_phases = [ + _make_phase(f"p{i}", f"phase_{i}", "lead", depends_on=[]) for i in range(11) + ] + # Re-decompose returns 12 independent (still over) + gateway = _make_mock_llm_gateway_with_phases( + phases_responses=[ + [ + { + "name": f"q{i}", + "assigned_expert": "lead", + "task_description": "t", + "depends_on": [], + } + for i in range(12) + ], + ] + ) + team._experts["lead"].agent._llm_gateway = gateway + + result = await orchestrator._rebalance_independent_subtasks( + lead=team.lead_expert, task="test", phases=original_phases + ) + # Should return the 12-phase re-decomposition (truncation happens later) + assert len(result) == 12 + + @pytest.mark.asyncio + async def test_rebalance_decompose_failure_returns_original(self): + """Re-decompose raises LLMProviderError → return original phases.""" + from agentkit.core.exceptions import LLMProviderError + + team = _make_team_with_experts() + orchestrator = TeamOrchestrator(team) + + original_phases = [ + _make_phase(f"p{i}", f"phase_{i}", "lead", depends_on=[]) for i in range(11) + ] + gateway = AsyncMock() + gateway.chat = AsyncMock(side_effect=LLMProviderError("LLM down")) + gateway.chat_stream = make_chat_stream_mock("err") + team._experts["lead"].agent._llm_gateway = gateway + + result = await orchestrator._rebalance_independent_subtasks( + lead=team.lead_expert, task="test", phases=original_phases + ) + assert result is original_phases + + @pytest.mark.asyncio + async def test_rebalance_empty_redecompose_returns_original(self): + """Re-decompose returns empty list → return original phases.""" + team = _make_team_with_experts() + orchestrator = TeamOrchestrator(team) + + original_phases = [ + _make_phase(f"p{i}", f"phase_{i}", "lead", depends_on=[]) for i in range(11) + ] + # Gateway returns malformed JSON → _parse_phases returns [] + gateway = AsyncMock() + bad_response = MagicMock(content="not json") + gateway.chat = AsyncMock(return_value=bad_response) + gateway.chat_stream = make_chat_stream_mock("err") + team._experts["lead"].agent._llm_gateway = gateway + + result = await orchestrator._rebalance_independent_subtasks( + lead=team.lead_expert, task="test", phases=original_phases + ) + assert result is original_phases + + +# ── Integration: full execute() with parallel subtasks ──────────────── + + +class TestExecuteParallelSubtasks: + """U4/R1, R3, R5: full execute() path with independent subtasks running + in parallel + synthesis waits for all.""" + + @pytest.mark.asyncio + async def test_three_parallel_subtasks_complete_then_synthesis(self): + """3 independent subtasks → all run in layer 0 → synthesis runs + after all complete.""" + team = _make_team_with_experts() + orchestrator = TeamOrchestrator(team) + + gateway = _make_mock_llm_gateway_with_phases( + phases_responses=[ + [ + { + "name": "A", + "assigned_expert": "lead", + "task_description": "a", + "depends_on": [], + }, + { + "name": "B", + "assigned_expert": "member1", + "task_description": "b", + "depends_on": [], + }, + { + "name": "C", + "assigned_expert": "member2", + "task_description": "c", + "depends_on": [], + }, + ], + ] + ) + team._experts["lead"].agent._llm_gateway = gateway + + result = await orchestrator.execute("并行任务") + + assert result["status"] == "completed" + plan: TeamPlan = result["plan"] + layers = plan.topological_sort() + assert len(layers) == 1 # all 3 in layer 0 (parallel) + assert len(layers[0]) == 3 + + # All phases completed + completed = plan.completed_phases + assert len(completed) == 3 + + # phase_results has all 3 + phase_results: dict = result["phase_results"] + assert len(phase_results) == 3 From 9653b1d5f7eece221246e44dfd5c5054fa18b121 Mon Sep 17 00:00:00 2001 From: Chiguyong Date: Mon, 6 Jul 2026 13:50:10 +0800 Subject: [PATCH 06/10] feat(memory): U5 ReflexionEngine reflection persistence to EpisodicMemory (R11, R15) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-task reflection persistence for prompt self-tuning: - EpisodicMemory.store_prompt_reflection(): persists reflection via existing store() with task_type="prompt_reflection" discriminator. Key format: prompt_reflection:{task_hash}:{version}. Non-raising on failure. - EpisodicMemory.search_prompt_reflections(): semantic search with task_type filter. Returns [] on failure. - EpisodicMemory.cleanup_expired(): TTL cleanup (default 30 days). - ReflexionEngine.__init__: optional episodic_memory param (None=backward compat) - ReflexionEngine._reflect: persists reflection+improved_prompt+score after generation. Non-blocking — persistence failure doesn't block in-task retry. Tests: 16 new tests (store/search/cleanup + _reflect persistence paths + multi-version coexistence). All pass. --- src/agentkit/core/reflexion.py | 37 ++- src/agentkit/memory/episodic.py | 98 +++++++ tests/unit/test_reflexion_persist.py | 423 +++++++++++++++++++++++++++ 3 files changed, 556 insertions(+), 2 deletions(-) create mode 100644 tests/unit/test_reflexion_persist.py diff --git a/src/agentkit/core/reflexion.py b/src/agentkit/core/reflexion.py index aa3c144..e106090 100644 --- a/src/agentkit/core/reflexion.py +++ b/src/agentkit/core/reflexion.py @@ -26,6 +26,7 @@ from agentkit.telemetry.metrics import ( if TYPE_CHECKING: from agentkit.core.compressor import CompressionStrategy from agentkit.core.trace import TraceRecorder + from agentkit.memory.episodic import EpisodicMemory from agentkit.memory.retriever import MemoryRetriever logger = logging.getLogger(__name__) @@ -72,6 +73,9 @@ class ReflexionEngine: max_reflections: int = 3, quality_threshold: float = 0.7, default_timeout: float = 300.0, + # IQ-Boost/U5 (R11): optional EpisodicMemory for persisting reflections + # across tasks. None = no persistence (backward-compatible). + episodic_memory: "EpisodicMemory | None" = None, ): if max_steps < 1: raise ValueError(f"max_steps must be >= 1, got {max_steps}") @@ -87,6 +91,8 @@ class ReflexionEngine: self._max_reflections = max_reflections self._quality_threshold = quality_threshold self._default_timeout = default_timeout + # U5: optional episodic memory for cross-task reflection persistence + self._episodic_memory = episodic_memory self._react_engine = ReActEngine( llm_gateway=llm_gateway, max_steps=max_steps, @@ -654,7 +660,12 @@ class ReflexionEngine: agent_name: str, task_type: str, ) -> str | None: - """反思执行结果，返回反思文本；失败时返回 None""" + """反思执行结果，返回反思文本；失败时返回 None + + IQ-Boost/U5 (R11): if ``self._episodic_memory`` is configured, persist + the reflection for cross-task retrieval. Persistence failure is + non-blocking — the reflection is still returned for in-task retry. + """ task_description = messages[-1].get("content", "") if messages else "" system_message = ( @@ -685,11 +696,33 @@ class ReflexionEngine: agent_name=agent_name, task_type=task_type or "reflection", ) - return response.content or None + reflection_text = response.content or None except Exception as e: logger.warning(f"Reflection LLM call failed, skipping reflection: {e}") return None + # U5/R11: persist reflection to EpisodicMemory (non-blocking) + if reflection_text and self._episodic_memory is not None: + improved_prompt = self._build_reflection_prompt( + original_prompt=None, + reflection_text=reflection_text, + attempt=1, + ) + try: + await self._episodic_memory.store_prompt_reflection( + task_input=task_description, + reflection=reflection_text, + improved_prompt=improved_prompt, + version=1, + score=score, + agent_name=agent_name, + ) + except Exception as e: + # Non-blocking: reflection is still useful for in-task retry + logger.warning(f"U5: failed to persist reflection, continuing: {e}") + + return reflection_text + def _build_reflection_prompt( self, original_prompt: str | None, diff --git a/src/agentkit/memory/episodic.py b/src/agentkit/memory/episodic.py index 6110311..3ced8af 100644 --- a/src/agentkit/memory/episodic.py +++ b/src/agentkit/memory/episodic.py @@ -410,3 +410,101 @@ class EpisodicMemory(Memory): await db.rollback() logger.error(f"Failed to delete episodic memory: {e}") return False + + # ── IQ-Boost/U5: Prompt Reflection persistence (R11, R15) ────────── + + async def store_prompt_reflection( + self, + task_input: str, + reflection: str, + improved_prompt: str, + version: int = 1, + score: float = 0.0, + agent_name: str = "", + task_hash: str | None = None, + ) -> str | None: + """持久化 prompt 反思到 EpisodicMemory，支持跨任务检索 (U5/R11). + + Reuses the existing ``store()`` path with ``task_type="prompt_reflection"`` + as the discriminator. The ORM row's fields map: + input_summary ← task_input (truncated) + output_summary ← improved_prompt (truncated) + reflection ← reflection text + quality_score ← score (0.0=failed, 1.0=verified) + outcome ← "reflection" + agent_name ← agent_name + + Returns the storage key ``"prompt_reflection:{task_hash}:{version}"`` + on success, or None on failure (non-raising — callers continue without). + """ + import hashlib + + if task_hash is None: + task_hash = hashlib.sha256(task_input.encode("utf-8")).hexdigest()[:16] + key = f"prompt_reflection:{task_hash}:{version}" + + value = { + "task_input": task_input[:500], + "reflection": reflection, + "improved_prompt": improved_prompt, + "score": score, + "version": version, + "task_hash": task_hash, + "timestamp": datetime.now(timezone.utc).isoformat(), + } + metadata = { + "agent_name": agent_name, + "task_type": "prompt_reflection", + "output_summary": improved_prompt[:500], + "outcome": "reflection", + "quality_score": score, + "reflection": reflection, + } + try: + await self.store(key=key, value=value, metadata=metadata) + return key + except (DBAPIError, ValueError, KeyError, RuntimeError, OSError) as e: + logger.warning(f"U5: failed to persist prompt reflection: {e}") + return None + + async def search_prompt_reflections( + self, + task_input: str, + top_k: int = 5, + agent_name: str | None = None, + ) -> list[MemoryItem]: + """检索相似 task_input 的历史 prompt 反思 (U5/R11). + + Uses ``search()`` with ``task_type="prompt_reflection"`` filter. + Returns empty list on failure (non-raising). + """ + filters: MetadataDict = {"task_type": "prompt_reflection"} + if agent_name: + filters["agent_name"] = agent_name + try: + return await self.search(query=task_input, top_k=top_k, filters=filters) + except (DBAPIError, ValueError, KeyError, RuntimeError, OSError) as e: + logger.warning(f"U5: failed to search prompt reflections: {e}") + return [] + + async def cleanup_expired(self, max_age_days: int = 30) -> int: + """删除超过 max_age_days 天的记录 (U5/R15 TTL). + + Returns the number of deleted rows. 0 on failure (non-raising). + """ + from datetime import timedelta + + from sqlalchemy import delete as sql_delete + + cutoff = datetime.now(timezone.utc) - timedelta(days=max_age_days) + async with self._session_factory() as db: + try: + Model = self._episodic_model + stmt = sql_delete(Model).where(Model.created_at < cutoff) + result = await db.execute(stmt) + await db.commit() + return result.rowcount or 0 + except (DBAPIError, ValueError, KeyError, RuntimeError) as e: + await db.rollback() + logger.warning(f"U5: cleanup_expired failed: {e}") + return 0 diff --git a/tests/unit/test_reflexion_persist.py b/tests/unit/test_reflexion_persist.py new file mode 100644 index 0000000..1473627 --- /dev/null +++ b/tests/unit/test_reflexion_persist.py @@ -0,0 +1,423 @@ +"""U5: ReflexionEngine reflection persistence to EpisodicMemory (R11, R15). + +Covers: +- store_prompt_reflection() stores via existing store() with task_type discriminator +- search_prompt_reflections() filters by task_type="prompt_reflection" +- cleanup_expired() deletes old records (TTL) +- _reflect() persists reflection when episodic_memory configured (non-blocking) +- _reflect() skips persistence when episodic_memory=None (backward compat) +- Persistence failure does not block _reflect (returns reflection text) +- Multi-version coexistence (same task_hash, different versions) +""" + +from __future__ import annotations + +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +from agentkit.core.react import ReActResult +from agentkit.core.reflexion import ReflexionEngine +from agentkit.memory.episodic import EpisodicMemory + + +# ── Helpers ──────────────────────────────────────────────────────────── + + +def _make_episodic_memory_mock() -> MagicMock: + """Create a mock EpisodicMemory that tracks store/search calls.""" + mem = MagicMock(spec=EpisodicMemory) + mem.store = AsyncMock(return_value=None) + mem.search = AsyncMock(return_value=[]) + mem.store_prompt_reflection = AsyncMock(return_value="prompt_reflection:abc:1") + mem.search_prompt_reflections = AsyncMock(return_value=[]) + mem.cleanup_expired = AsyncMock(return_value=0) + return mem + + +def _make_react_result(output: str = "test output", status: str = "completed") -> ReActResult: + return ReActResult( + output=output, + trajectory=[], + total_steps=1, + total_tokens=10, + status=status, + ) + + +def _make_llm_gateway_mock(reflection_text: str = "reflection text") -> MagicMock: + """Gateway that returns reflection text from chat().""" + gw = MagicMock() + response = MagicMock() + response.content = reflection_text + gw.chat = AsyncMock(return_value=response) + return gw + + +# ── EpisodicMemory.store_prompt_reflection ───────────────────────────── + + +class TestStorePromptReflection: + """U5/R11: store_prompt_reflection persists via store() with + task_type='prompt_reflection' discriminator.""" + + @pytest.mark.asyncio + async def test_store_calls_underlying_store_with_correct_metadata(self): + mem = MagicMock(spec=EpisodicMemory) + mem.store = AsyncMock(return_value=None) + # We need to call the real method, not a mock — patch store only + # Use the unbound method pattern + await EpisodicMemory.store_prompt_reflection( + mem, + task_input="test task", + reflection="reflection text", + improved_prompt="improved prompt", + version=1, + score=0.5, + agent_name="test_agent", + ) + + mem.store.assert_awaited_once() + call_kwargs = mem.store.await_args.kwargs + assert "key" in call_kwargs + assert call_kwargs["key"].startswith("prompt_reflection:") + assert ":1" in call_kwargs["key"] + + value = call_kwargs["value"] + assert value["task_input"] == "test task" + assert value["reflection"] == "reflection text" + assert value["improved_prompt"] == "improved prompt" + assert value["score"] == 0.5 + assert value["version"] == 1 + assert "timestamp" in value + + metadata = call_kwargs["metadata"] + assert metadata["task_type"] == "prompt_reflection" + assert metadata["agent_name"] == "test_agent" + assert metadata["quality_score"] == 0.5 + assert metadata["reflection"] == "reflection text" + + @pytest.mark.asyncio + async def test_store_returns_key_on_success(self): + mem = MagicMock(spec=EpisodicMemory) + mem.store = AsyncMock(return_value=None) + key = await EpisodicMemory.store_prompt_reflection( + mem, + task_input="test", + reflection="r", + improved_prompt="p", + version=2, + ) + assert key is not None + assert ":2" in key + assert key.startswith("prompt_reflection:") + + @pytest.mark.asyncio + async def test_store_returns_none_on_failure(self): + from sqlalchemy.exc import DBAPIError + + mem = MagicMock(spec=EpisodicMemory) + mem.store = AsyncMock(side_effect=DBAPIError("stmt", params={}, orig=Exception("db down"))) + key = await EpisodicMemory.store_prompt_reflection( + mem, + task_input="test", + reflection="r", + improved_prompt="p", + ) + assert key is None + + @pytest.mark.asyncio + async def test_store_uses_provided_task_hash(self): + mem = MagicMock(spec=EpisodicMemory) + mem.store = AsyncMock(return_value=None) + key = await EpisodicMemory.store_prompt_reflection( + mem, + task_input="test", + reflection="r", + improved_prompt="p", + task_hash="custom_hash", + ) + assert "custom_hash" in key + + @pytest.mark.asyncio + async def test_store_generates_task_hash_from_input(self): + mem = MagicMock(spec=EpisodicMemory) + mem.store = AsyncMock(return_value=None) + key1 = await EpisodicMemory.store_prompt_reflection( + mem, task_input="same task", reflection="r", improved_prompt="p" + ) + key2 = await EpisodicMemory.store_prompt_reflection( + mem, task_input="same task", reflection="r", improved_prompt="p" + ) + # Same task_input → same hash prefix + assert key1 == key2 + + +# ── EpisodicMemory.search_prompt_reflections ─────────────────────────── + + +class TestSearchPromptReflections: + """U5/R11: search_prompt_reflections filters by task_type.""" + + @pytest.mark.asyncio + async def test_search_calls_underlying_search_with_filter(self): + mem = MagicMock(spec=EpisodicMemory) + mem.search = AsyncMock(return_value=[]) + await EpisodicMemory.search_prompt_reflections(mem, task_input="find similar", top_k=3) + + mem.search.assert_awaited_once() + call_kwargs = mem.search.await_args.kwargs + assert call_kwargs["query"] == "find similar" + assert call_kwargs["top_k"] == 3 + filters = call_kwargs["filters"] + assert filters["task_type"] == "prompt_reflection" + + @pytest.mark.asyncio + async def test_search_includes_agent_name_filter_when_provided(self): + mem = MagicMock(spec=EpisodicMemory) + mem.search = AsyncMock(return_value=[]) + await EpisodicMemory.search_prompt_reflections(mem, task_input="q", agent_name="agent_x") + + filters = mem.search.await_args.kwargs["filters"] + assert filters["agent_name"] == "agent_x" + + @pytest.mark.asyncio + async def test_search_returns_empty_on_failure(self): + from sqlalchemy.exc import DBAPIError + + mem = MagicMock(spec=EpisodicMemory) + mem.search = AsyncMock(side_effect=DBAPIError("stmt", params={}, orig=Exception("err"))) + result = await EpisodicMemory.search_prompt_reflections(mem, task_input="q") + assert result == [] + + +# ── EpisodicMemory.cleanup_expired ───────────────────────────────────── + + +class TestCleanupExpired: + """U5/R15: cleanup_expired deletes records older than max_age_days.""" + + @pytest.mark.asyncio + async def test_cleanup_calls_delete_with_cutoff(self): + # Patch sqlalchemy.delete to bypass ORM model requirement — we only + # verify the session.execute/commit calls happen. + mock_db = AsyncMock() + mock_result = MagicMock() + mock_result.rowcount = 5 + mock_db.execute = AsyncMock(return_value=mock_result) + mock_db.commit = AsyncMock() + + mock_session_factory = MagicMock() + mock_session_factory.return_value.__aenter__ = AsyncMock(return_value=mock_db) + mock_session_factory.return_value.__aexit__ = AsyncMock(return_value=None) + + # created_at must support `<` comparison with datetime — configure + # __lt__ to return a truthy mock so `.where(...)` gets a valid arg. + mock_model = MagicMock() + mock_model.created_at = MagicMock() + mock_model.created_at.__lt__ = MagicMock(return_value=MagicMock()) + + mem = EpisodicMemory( + session_factory=mock_session_factory, + episodic_model=mock_model, + embedder=None, + pgvector_enabled=False, + ) + + # Patch sql_delete to return a chainable mock + fake_stmt = MagicMock() + fake_stmt.where = MagicMock(return_value=fake_stmt) + with patch("sqlalchemy.delete", return_value=fake_stmt): + deleted = await mem.cleanup_expired(max_age_days=30) + + assert deleted == 5 + mock_db.execute.assert_awaited_once() + mock_db.commit.assert_awaited_once() + + @pytest.mark.asyncio + async def test_cleanup_returns_zero_on_failure(self): + from sqlalchemy.exc import DBAPIError + + mock_db = AsyncMock() + mock_db.execute = AsyncMock( + side_effect=DBAPIError("stmt", params={}, orig=Exception("err")) + ) + mock_db.rollback = AsyncMock() + + mock_session_factory = MagicMock() + mock_session_factory.return_value.__aenter__ = AsyncMock(return_value=mock_db) + mock_session_factory.return_value.__aexit__ = AsyncMock(return_value=None) + + mock_model = MagicMock() + mock_model.created_at = MagicMock() + mock_model.created_at.__lt__ = MagicMock(return_value=MagicMock()) + + mem = EpisodicMemory( + session_factory=mock_session_factory, + episodic_model=mock_model, + embedder=None, + pgvector_enabled=False, + ) + + fake_stmt = MagicMock() + fake_stmt.where = MagicMock(return_value=fake_stmt) + with patch("sqlalchemy.delete", return_value=fake_stmt): + deleted = await mem.cleanup_expired(max_age_days=30) + + assert deleted == 0 + mock_db.rollback.assert_awaited_once() + + +# ── ReflexionEngine._reflect persistence ────────────────────────────── + + +class TestReflectPersistence: + """U5/R11: _reflect persists reflection when episodic_memory configured.""" + + @pytest.mark.asyncio + async def test_reflect_persists_when_episodic_memory_configured(self): + episodic = _make_episodic_memory_mock() + gw = _make_llm_gateway_mock(reflection_text="my reflection") + engine = ReflexionEngine( + llm_gateway=gw, + episodic_memory=episodic, + ) + + result = await engine._reflect( + react_result=_make_react_result(), + score=0.3, + messages=[{"role": "user", "content": "test task"}], + reflect_model="default", + agent_name="test_agent", + task_type="test", + ) + + assert result == "my reflection" + episodic.store_prompt_reflection.assert_awaited_once() + call_kwargs = episodic.store_prompt_reflection.await_args.kwargs + assert call_kwargs["task_input"] == "test task" + assert call_kwargs["reflection"] == "my reflection" + assert call_kwargs["score"] == 0.3 + assert call_kwargs["agent_name"] == "test_agent" + + @pytest.mark.asyncio + async def test_reflect_skips_persistence_when_no_episodic_memory(self): + gw = _make_llm_gateway_mock(reflection_text="my reflection") + engine = ReflexionEngine( + llm_gateway=gw, + episodic_memory=None, # No persistence + ) + + result = await engine._reflect( + react_result=_make_react_result(), + score=0.3, + messages=[{"role": "user", "content": "test task"}], + reflect_model="default", + agent_name="test_agent", + task_type="test", + ) + + assert result == "my reflection" + # No episodic_memory → no store call + + @pytest.mark.asyncio + async def test_reflect_persistence_failure_does_not_block(self): + """If store_prompt_reflection raises, _reflect still returns reflection.""" + episodic = MagicMock(spec=EpisodicMemory) + episodic.store_prompt_reflection = AsyncMock(side_effect=RuntimeError("DB down")) + gw = _make_llm_gateway_mock(reflection_text="important reflection") + engine = ReflexionEngine( + llm_gateway=gw, + episodic_memory=episodic, + ) + + result = await engine._reflect( + react_result=_make_react_result(), + score=0.3, + messages=[{"role": "user", "content": "test task"}], + reflect_model="default", + agent_name="test_agent", + task_type="test", + ) + + # Reflection still returned despite persistence failure + assert result == "important reflection" + episodic.store_prompt_reflection.assert_awaited_once() + + @pytest.mark.asyncio + async def test_reflect_skips_persistence_when_llm_returns_none(self): + """If LLM returns empty/None, no persistence attempted.""" + episodic = _make_episodic_memory_mock() + gw = MagicMock() + response = MagicMock() + response.content = None # Empty reflection + gw.chat = AsyncMock(return_value=response) + engine = ReflexionEngine( + llm_gateway=gw, + episodic_memory=episodic, + ) + + result = await engine._reflect( + react_result=_make_react_result(), + score=0.3, + messages=[{"role": "user", "content": "test task"}], + reflect_model="default", + agent_name="test_agent", + task_type="test", + ) + + assert result is None + episodic.store_prompt_reflection.assert_not_awaited() + + @pytest.mark.asyncio + async def test_reflect_skips_persistence_on_llm_failure(self): + """If LLM call raises, no persistence attempted.""" + episodic = _make_episodic_memory_mock() + gw = MagicMock() + gw.chat = AsyncMock(side_effect=RuntimeError("LLM down")) + engine = ReflexionEngine( + llm_gateway=gw, + episodic_memory=episodic, + ) + + result = await engine._reflect( + react_result=_make_react_result(), + score=0.3, + messages=[{"role": "user", "content": "test task"}], + reflect_model="default", + agent_name="test_agent", + task_type="test", + ) + + assert result is None + episodic.store_prompt_reflection.assert_not_awaited() + + +# ── Multi-version coexistence ───────────────────────────────────────── + + +class TestMultiVersionCoexistence: + """U5/R11: same task_hash with different versions all stored.""" + + @pytest.mark.asyncio + async def test_multiple_versions_stored_with_incrementing_version(self): + mem = MagicMock(spec=EpisodicMemory) + mem.store = AsyncMock(return_value=None) + + # Store v1, v2, v3 for same task + key1 = await EpisodicMemory.store_prompt_reflection( + mem, task_input="same task", reflection="r1", improved_prompt="p1", version=1 + ) + key2 = await EpisodicMemory.store_prompt_reflection( + mem, task_input="same task", reflection="r2", improved_prompt="p2", version=2 + ) + key3 = await EpisodicMemory.store_prompt_reflection( + mem, task_input="same task", reflection="r3", improved_prompt="p3", version=3 + ) + + # All keys have same task_hash prefix but different version suffix + assert key1 != key2 != key3 + assert ":1" in key1 and ":2" in key2 and ":3" in key3 + + # All three store() calls made + assert mem.store.await_count == 3 From a2deeac0d6090e069dcf2622080c9e2154145b67 Mon Sep 17 00:00:00 2001 From: Chiguyong Date: Mon, 6 Jul 2026 13:57:51 +0800 Subject: [PATCH 07/10] feat(iq): U6 Lead planning-time reflection retrieval (R12, R13) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - ReflexionEngine.retrieve_prompt_reflection(): searches EpisodicMemory for historical reflections on similar task_input, returns best version by score (defaults min_score=0.5). Non-blocking: failure → None. - TeamOrchestrator._decompose_task: prepends historical reflection hint to Lead's planning prompt when reflexion_engine is wired and a high-score reflection exists. Default prompt preserved on miss/failure. - 12 unit tests covering retrieve path (7) + decompose integration (5). --- src/agentkit/core/reflexion.py | 52 +++ src/agentkit/experts/orchestrator.py | 36 +++ tests/unit/test_lead_reflection_retrieval.py | 319 +++++++++++++++++++ 3 files changed, 407 insertions(+) create mode 100644 tests/unit/test_lead_reflection_retrieval.py diff --git a/src/agentkit/core/reflexion.py b/src/agentkit/core/reflexion.py index e106090..5facdc3 100644 --- a/src/agentkit/core/reflexion.py +++ b/src/agentkit/core/reflexion.py @@ -26,6 +26,7 @@ from agentkit.telemetry.metrics import ( if TYPE_CHECKING: from agentkit.core.compressor import CompressionStrategy from agentkit.core.trace import TraceRecorder + from agentkit.memory.base import MemoryItem from agentkit.memory.episodic import EpisodicMemory from agentkit.memory.retriever import MemoryRetriever @@ -742,3 +743,54 @@ class ReflexionEngine: return original_prompt + reflection_section else: return reflection_section.strip() + + async def retrieve_prompt_reflection( + self, task_input: str, min_score: float = 0.5 + ) -> dict[str, object] | None: + """检索历史 prompt 反思，返回最佳版本 (U6/R12, R13). + + Searches EpisodicMemory for similar task_input reflections with + score > min_score. Returns the highest-scored reflection as: + {improved_prompt, score, reflection, version, task_hash} + or None if no episodic_memory / no results / all below threshold. + + KTD5: callers should only invoke this when a trigger condition is + met (verify failure / schema failure / loop detection) to avoid + pointless retrieval on every task. + """ + if self._episodic_memory is None: + return None + + try: + results = await self._episodic_memory.search_prompt_reflections( + task_input=task_input, top_k=3 + ) + except Exception as e: + logger.warning(f"U6: retrieve_prompt_reflection failed: {e}") + return None + + if not results: + return None + + # Filter by min_score, pick the highest-scored + best: MemoryItem | None = None + for item in results: + score = item.score or 0.0 + if score > min_score and (best is None or score > (best.score or 0.0)): + best = item + + if best is None: + return None + + # Extract improved_prompt from metadata (output_summary field) + metadata = best.metadata or {} + improved_prompt = metadata.get("output_summary", "") or metadata.get("improved_prompt", "") + reflection_text = metadata.get("reflection", "") or best.value or "" + + return { + "improved_prompt": improved_prompt, + "score": best.score or 0.0, + "reflection": reflection_text, + "version": metadata.get("version", 1), + "task_hash": metadata.get("task_hash", ""), + } diff --git a/src/agentkit/experts/orchestrator.py b/src/agentkit/experts/orchestrator.py index 8aa501a..182732a 100644 --- a/src/agentkit/experts/orchestrator.py +++ b/src/agentkit/experts/orchestrator.py @@ -14,6 +14,7 @@ import asyncio import json import logging import re +from typing import TYPE_CHECKING from agentkit.core.exceptions import LLMProviderError from agentkit.llm.gateway import LLMGateway @@ -36,6 +37,9 @@ from .plan import ( ) from .team import ExpertTeam, TeamStatus +if TYPE_CHECKING: + from agentkit.core.reflexion import ReflexionEngine + logger = logging.getLogger(__name__) # 专家名校验正则（与 router.py / board_router.py 保持一致） @@ -82,6 +86,10 @@ class TeamOrchestrator( # final-answer path (react.py:1303+) runs on coding tasks. verification_enabled: bool = True, verification_commands: list[str] | None = None, + # IQ-Boost/U6 (R12, R13): optional ReflexionEngine for retrieving + # historical prompt reflections at Lead planning time. None = no + # retrieval (backward-compatible). + reflexion_engine: "ReflexionEngine | None" = None, ) -> None: self._team = team # Track temporary agent names created for context isolation (KTD3) @@ -103,6 +111,8 @@ class TeamOrchestrator( self._rollback_timeout = rollback_timeout or self.DEFAULT_ROLLBACK_TIMEOUT # U3/R2: verification defaults for TEAM_COLLAB. self._verification_enabled = verification_enabled + # U6: optional reflexion engine for historical reflection retrieval + self._reflexion_engine = reflexion_engine # U3/R3: if no explicit commands, detect from workspace (coding-task # detection forces pytest/ruff). None workspace → None commands → # ReActEngine/VerificationLoop uses its own defaults. @@ -608,6 +618,11 @@ class TeamOrchestrator( Returns a list of PlanPhase instances. If LLM decomposition fails, returns a single phase with the original task. + + IQ-Boost/U6 (R12, R13): if reflexion_engine is configured, retrieves + historical prompt reflection for similar task_input and prepends + improved_prompt to the decomposition prompt. Non-blocking — retrieval + failure falls through to default prompt. """ gateway = self._get_llm_gateway(lead) if not gateway: @@ -619,6 +634,26 @@ class TeamOrchestrator( ] available_experts = member_names if member_names else [lead.config.name] + # U6: retrieve historical reflection (non-blocking) + reflection_hint = "" + if self._reflexion_engine is not None: + try: + historical = await self._reflexion_engine.retrieve_prompt_reflection( + task_input=task + ) + if historical and historical.get("improved_prompt"): + reflection_hint = ( + f"\n\n## Historical Reflection (score={historical.get('score', 0):.2f})\n" + f"A previous similar task produced this reflection. " + f"Use it to improve your decomposition:\n\n" + f"{historical['improved_prompt']}\n" + ) + logger.info( + f"U6: retrieved historical reflection (score={historical.get('score', 0):.2f})" + ) + except Exception as e: + logger.warning(f"U6: historical reflection retrieval failed, using default: {e}") + prompt = ( f"You are the Lead Expert in a pipeline team. Decompose the following task into " f"at most {self.MAX_PHASES} phases with dependencies.\n\n" @@ -646,6 +681,7 @@ class TeamOrchestrator( f'{{"name":"前端","assigned_expert":"frontend",' f'"task_description":"实现UI","depends_on":["后端"],"collaboration_contracts":[]}}]\n\n' f"Return ONLY the JSON array, no other text." + f"{reflection_hint}" ) try: diff --git a/tests/unit/test_lead_reflection_retrieval.py b/tests/unit/test_lead_reflection_retrieval.py new file mode 100644 index 0000000..263094f --- /dev/null +++ b/tests/unit/test_lead_reflection_retrieval.py @@ -0,0 +1,319 @@ +"""U6: Lead planning-time historical reflection retrieval (R12, R13). + +Covers: +- ReflexionEngine.retrieve_prompt_reflection(): returns best reflection by score +- Score filtering: score <= 0.5 not returned +- No episodic_memory → returns None +- Retrieval failure → returns None (non-blocking) +- TeamOrchestrator._decompose_task: prepends improved_prompt when reflection found +- No reflexion_engine → default prompt (backward compat) +- Retrieval failure → default prompt (non-blocking) +""" + +from __future__ import annotations + +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from agentkit.core.reflexion import ReflexionEngine +from agentkit.experts.orchestrator import TeamOrchestrator +from agentkit.experts.team import ExpertTeam +from agentkit.memory.base import MemoryItem + +from tests.unit.experts._helpers import make_execute_stream_mock + + +# ── Helpers ──────────────────────────────────────────────────────────── + + +def _make_memory_item( + score: float = 0.8, + output_summary: str = "improved prompt text", + reflection: str = "reflection text", + version: int = 1, +) -> MemoryItem: + from datetime import datetime, timezone + + return MemoryItem( + key="prompt_reflection:abc:1", + value={"task_input": "test", "reflection": reflection}, + metadata={ + "task_type": "prompt_reflection", + "output_summary": output_summary, + "reflection": reflection, + "version": version, + "task_hash": "abc", + "quality_score": score, + }, + score=score, + created_at=datetime.now(timezone.utc), + ) + + +def _make_llm_gateway_mock() -> MagicMock: + gw = MagicMock() + response = MagicMock() + response.content = ( + '[{"name":"A","assigned_expert":"lead","task_description":"a","depends_on":[]}]' + ) + gw.chat = AsyncMock(return_value=response) + return gw + + +def _make_team_with_experts() -> ExpertTeam: + from agentkit.core.handoff_transport import InProcessHandoffTransport + from agentkit.core.protocol import TaskResult, TaskStatus + from agentkit.experts.config import ExpertConfig + from agentkit.experts.expert import Expert + + team = ExpertTeam() + team._handoff_transport = AsyncMock(spec=InProcessHandoffTransport) + + config = ExpertConfig( + name="lead", + agent_type="expert", + persona="测试", + thinking_style="逻辑", + bound_skills=["s"], + is_lead=True, + task_mode="llm_generate", + prompt={"identity": "测试"}, + ) + expert = MagicMock(spec=Expert) + expert.config = config + expert.is_active = True + expert.team_id = None + expert.get_capabilities_summary.return_value = {"name": "lead"} + + mock_agent = MagicMock() + mock_agent.execute = AsyncMock( + return_value=TaskResult( + task_id="t", + agent_name="lead", + status=TaskStatus.COMPLETED.value, + output_data={"content": "result"}, + error_message=None, + started_at=None, + completed_at=None, + ) + ) + mock_agent.execute_stream = make_execute_stream_mock("result") + mock_agent._llm_gateway = None + expert.agent = mock_agent + + team._experts["lead"] = expert + team._lead_expert_name = "lead" + return team + + +# ── ReflexionEngine.retrieve_prompt_reflection ───────────────────────── + + +class TestRetrievePromptReflection: + """U6/R12: retrieve_prompt_reflection returns best reflection by score.""" + + @pytest.mark.asyncio + async def test_returns_none_when_no_episodic_memory(self): + gw = MagicMock() + engine = ReflexionEngine(llm_gateway=gw, episodic_memory=None) + result = await engine.retrieve_prompt_reflection(task_input="test") + assert result is None + + @pytest.mark.asyncio + async def test_returns_best_reflection_by_score(self): + episodic = MagicMock() + # Two results: score 0.6 and 0.9 — should return 0.9 + items = [ + _make_memory_item(score=0.6, output_summary="prompt v1"), + _make_memory_item(score=0.9, output_summary="prompt v2"), + ] + episodic.search_prompt_reflections = AsyncMock(return_value=items) + gw = MagicMock() + engine = ReflexionEngine(llm_gateway=gw, episodic_memory=episodic) + + result = await engine.retrieve_prompt_reflection(task_input="test") + + assert result is not None + assert result["score"] == 0.9 + assert result["improved_prompt"] == "prompt v2" + + @pytest.mark.asyncio + async def test_filters_low_score_reflections(self): + """score <= 0.5 should not be returned.""" + episodic = MagicMock() + items = [_make_memory_item(score=0.3, output_summary="low score")] + episodic.search_prompt_reflections = AsyncMock(return_value=items) + gw = MagicMock() + engine = ReflexionEngine(llm_gateway=gw, episodic_memory=episodic) + + result = await engine.retrieve_prompt_reflection(task_input="test", min_score=0.5) + assert result is None + + @pytest.mark.asyncio + async def test_returns_none_when_no_results(self): + episodic = MagicMock() + episodic.search_prompt_reflections = AsyncMock(return_value=[]) + gw = MagicMock() + engine = ReflexionEngine(llm_gateway=gw, episodic_memory=episodic) + + result = await engine.retrieve_prompt_reflection(task_input="test") + assert result is None + + @pytest.mark.asyncio + async def test_returns_none_on_search_failure(self): + episodic = MagicMock() + episodic.search_prompt_reflections = AsyncMock(side_effect=RuntimeError("DB down")) + gw = MagicMock() + engine = ReflexionEngine(llm_gateway=gw, episodic_memory=episodic) + + result = await engine.retrieve_prompt_reflection(task_input="test") + assert result is None + + @pytest.mark.asyncio + async def test_returns_reflection_fields_complete(self): + episodic = MagicMock() + items = [ + _make_memory_item( + score=0.85, + output_summary="improved prompt", + reflection="what went wrong", + version=3, + ) + ] + episodic.search_prompt_reflections = AsyncMock(return_value=items) + gw = MagicMock() + engine = ReflexionEngine(llm_gateway=gw, episodic_memory=episodic) + + result = await engine.retrieve_prompt_reflection(task_input="test") + + assert result is not None + assert result["improved_prompt"] == "improved prompt" + assert result["reflection"] == "what went wrong" + assert result["version"] == 3 + assert result["score"] == 0.85 + assert "task_hash" in result + + @pytest.mark.asyncio + async def test_custom_min_score_threshold(self): + """min_score=0.7 filters out score=0.6.""" + episodic = MagicMock() + items = [_make_memory_item(score=0.6, output_summary="medium")] + episodic.search_prompt_reflections = AsyncMock(return_value=items) + gw = MagicMock() + engine = ReflexionEngine(llm_gateway=gw, episodic_memory=episodic) + + result = await engine.retrieve_prompt_reflection(task_input="test", min_score=0.7) + assert result is None + + +# ── TeamOrchestrator._decompose_task with reflection ────────────────── + + +class TestDecomposeWithReflection: + """U6/R13: _decompose_task prepends improved_prompt when reflection found.""" + + @pytest.mark.asyncio + async def test_decompose_prepends_reflection_when_found(self): + """When reflexion_engine returns a reflection, the decomposition + prompt includes the improved_prompt.""" + team = _make_team_with_experts() + gw = _make_llm_gateway_mock() + team._experts["lead"].agent._llm_gateway = gw + + # Mock reflexion_engine + reflexion = MagicMock(spec=ReflexionEngine) + reflexion.retrieve_prompt_reflection = AsyncMock( + return_value={ + "improved_prompt": "USE THIS IMPROVED APPROACH", + "score": 0.85, + "reflection": "past mistake", + "version": 2, + "task_hash": "abc", + } + ) + + orchestrator = TeamOrchestrator(team, reflexion_engine=reflexion) + await orchestrator._decompose_task(team.lead_expert, "test task") + + # Verify retrieve was called + reflexion.retrieve_prompt_reflection.assert_awaited_once() + # Verify the prompt sent to LLM includes the improved_prompt + call_kwargs = gw.chat.await_args.kwargs + messages = call_kwargs.get("messages") or gw.chat.await_args.args[0] + prompt_content = messages[0]["content"] if isinstance(messages, list) else str(messages) + assert "USE THIS IMPROVED APPROACH" in prompt_content + assert "Historical Reflection" in prompt_content + + @pytest.mark.asyncio + async def test_decompose_uses_default_when_no_reflexion_engine(self): + """No reflexion_engine → default prompt (backward compat).""" + team = _make_team_with_experts() + gw = _make_llm_gateway_mock() + team._experts["lead"].agent._llm_gateway = gw + + orchestrator = TeamOrchestrator(team, reflexion_engine=None) + await orchestrator._decompose_task(team.lead_expert, "test task") + + # Verify default prompt (no Historical Reflection section) + call_kwargs = gw.chat.await_args.kwargs + messages = call_kwargs.get("messages") or gw.chat.await_args.args[0] + prompt_content = messages[0]["content"] if isinstance(messages, list) else str(messages) + assert "Historical Reflection" not in prompt_content + + @pytest.mark.asyncio + async def test_decompose_uses_default_when_no_reflection_found(self): + """reflexion_engine returns None → default prompt.""" + team = _make_team_with_experts() + gw = _make_llm_gateway_mock() + team._experts["lead"].agent._llm_gateway = gw + + reflexion = MagicMock(spec=ReflexionEngine) + reflexion.retrieve_prompt_reflection = AsyncMock(return_value=None) + + orchestrator = TeamOrchestrator(team, reflexion_engine=reflexion) + await orchestrator._decompose_task(team.lead_expert, "test task") + + call_kwargs = gw.chat.await_args.kwargs + messages = call_kwargs.get("messages") or gw.chat.await_args.args[0] + prompt_content = messages[0]["content"] if isinstance(messages, list) else str(messages) + assert "Historical Reflection" not in prompt_content + + @pytest.mark.asyncio + async def test_decompose_uses_default_when_retrieval_fails(self): + """reflexion_engine.retrieve raises → default prompt (non-blocking).""" + team = _make_team_with_experts() + gw = _make_llm_gateway_mock() + team._experts["lead"].agent._llm_gateway = gw + + reflexion = MagicMock(spec=ReflexionEngine) + reflexion.retrieve_prompt_reflection = AsyncMock(side_effect=RuntimeError("search failed")) + + orchestrator = TeamOrchestrator(team, reflexion_engine=reflexion) + await orchestrator._decompose_task(team.lead_expert, "test task") + + # Default prompt used despite retrieval failure + call_kwargs = gw.chat.await_args.kwargs + messages = call_kwargs.get("messages") or gw.chat.await_args.args[0] + prompt_content = messages[0]["content"] if isinstance(messages, list) else str(messages) + assert "Historical Reflection" not in prompt_content + + @pytest.mark.asyncio + async def test_decompose_skips_reflection_when_no_improved_prompt(self): + """reflexion_engine returns dict without improved_prompt → no hint.""" + team = _make_team_with_experts() + gw = _make_llm_gateway_mock() + team._experts["lead"].agent._llm_gateway = gw + + reflexion = MagicMock(spec=ReflexionEngine) + reflexion.retrieve_prompt_reflection = AsyncMock( + return_value={"improved_prompt": "", "score": 0.8} # empty improved_prompt + ) + + orchestrator = TeamOrchestrator(team, reflexion_engine=reflexion) + await orchestrator._decompose_task(team.lead_expert, "test task") + + call_kwargs = gw.chat.await_args.kwargs + messages = call_kwargs.get("messages") or gw.chat.await_args.args[0] + prompt_content = messages[0]["content"] if isinstance(messages, list) else str(messages) + assert "Historical Reflection" not in prompt_content From 4e2c7c5cac63f1bd7e8864259134c74dc52b8bb0 Mon Sep 17 00:00:00 2001 From: Chiguyong Date: Mon, 6 Jul 2026 14:02:14 +0800 Subject: [PATCH 08/10] feat(iq): U7 ABTester prompt-version offline comparison (R14) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - EpisodicMemory.list_prompt_reflections_by_hash(task_hash): exact query for all prompt_reflection records matching task_hash. ponytail: O(N) scan with N<100 typical; GIN index upgrade path noted. - ABTester.__init__: accepts optional episodic_memory parameter. - ABTester.compare_prompt_versions(task_hash) -> dict: retrieves all versions for a task_hash, sorts by score descending, returns {versions, best_version, recommendation, total_versions}. Offline-only — no online bandit. Non-blocking on retrieval failure. - 8 unit tests covering no-episodic, multi-version sort, single version, empty result, retrieval failure, field completeness, low-score retention, task_hash echo. --- src/agentkit/evolution/ab_tester.py | 118 ++++++++++++++++- src/agentkit/memory/episodic.py | 62 +++++++++ tests/unit/test_ab_tester_prompt.py | 197 ++++++++++++++++++++++++++++ 3 files changed, 370 insertions(+), 7 deletions(-) create mode 100644 tests/unit/test_ab_tester_prompt.py diff --git a/src/agentkit/evolution/ab_tester.py b/src/agentkit/evolution/ab_tester.py index 9a01572..e113987 100644 --- a/src/agentkit/evolution/ab_tester.py +++ b/src/agentkit/evolution/ab_tester.py @@ -12,6 +12,7 @@ from sqlalchemy.exc import DBAPIError if TYPE_CHECKING: from agentkit.evolution.evolution_store import InMemoryEvolutionStore + from agentkit.memory.episodic import EpisodicMemory logger = logging.getLogger(__name__) @@ -19,6 +20,7 @@ logger = logging.getLogger(__name__) @dataclass class ABTestConfig: """A/B 测试配置""" + test_id: str agent_name: str change_type: str # prompt / strategy / pipeline @@ -31,6 +33,7 @@ class ABTestConfig: @dataclass class ABTestResult: """A/B 测试结果""" + test_id: str control_metric: float experiment_metric: float @@ -52,11 +55,15 @@ class ABTester: self, evolution_store: "InMemoryEvolutionStore | None" = None, min_samples: int = 10, + episodic_memory: "EpisodicMemory | None" = None, ): self._tests: dict[str, ABTestConfig] = {} self._results: dict[str, list[tuple[str, float]]] = {} # test_id -> [(group, metric)] self._evolution_store = evolution_store self._default_min_samples = min_samples + # IQ-Boost/U7 (R14): optional EpisodicMemory for prompt-version comparison. + # None = compare_prompt_versions() returns empty result (backward-compatible). + self._episodic_memory = episodic_memory def create_test(self, config: ABTestConfig) -> None: """创建 A/B 测试""" @@ -115,7 +122,9 @@ class ABTester: experiment_metrics = [m for g, m in results if g == "experiment"] control_avg = sum(control_metrics) / len(control_metrics) if control_metrics else 0.0 - experiment_avg = sum(experiment_metrics) / len(experiment_metrics) if experiment_metrics else 0.0 + experiment_avg = ( + sum(experiment_metrics) / len(experiment_metrics) if experiment_metrics else 0.0 + ) try: await self._evolution_store.record_ab_test_result( @@ -144,11 +153,18 @@ class ABTester: control_metrics = [m for g, m in results if g == "control"] experiment_metrics = [m for g, m in results if g == "experiment"] - if len(control_metrics) < config.min_samples or len(experiment_metrics) < config.min_samples: + if ( + len(control_metrics) < config.min_samples + or len(experiment_metrics) < config.min_samples + ): return ABTestResult( test_id=test_id, - control_metric=sum(control_metrics) / len(control_metrics) if control_metrics else 0, - experiment_metric=sum(experiment_metrics) / len(experiment_metrics) if experiment_metrics else 0, + control_metric=sum(control_metrics) / len(control_metrics) + if control_metrics + else 0, + experiment_metric=sum(experiment_metrics) / len(experiment_metrics) + if experiment_metrics + else 0, control_samples=len(control_metrics), experiment_samples=len(experiment_metrics), is_significant=False, @@ -159,10 +175,16 @@ class ABTester: control_mean = sum(control_metrics) / len(control_metrics) experiment_mean = sum(experiment_metrics) / len(experiment_metrics) - control_var = sum((m - control_mean) ** 2 for m in control_metrics) / (len(control_metrics) - 1) - experiment_var = sum((m - experiment_mean) ** 2 for m in experiment_metrics) / (len(experiment_metrics) - 1) + control_var = sum((m - control_mean) ** 2 for m in control_metrics) / ( + len(control_metrics) - 1 + ) + experiment_var = sum((m - experiment_mean) ** 2 for m in experiment_metrics) / ( + len(experiment_metrics) - 1 + ) - pooled_se = math.sqrt(control_var / len(control_metrics) + experiment_var / len(experiment_metrics)) + pooled_se = math.sqrt( + control_var / len(control_metrics) + experiment_var / len(experiment_metrics) + ) # Handle zero variance case: if means differ but variance is zero, # the difference is clearly significant @@ -201,3 +223,85 @@ class ABTester: def _normal_cdf(x: float) -> float: """标准正态分布 CDF 近似""" return 0.5 * (1 + math.erf(x / math.sqrt(2))) + + # ── IQ-Boost/U7: Prompt-version offline comparison (R14) ──────────── + + async def compare_prompt_versions(self, task_hash: str) -> dict[str, object]: + """离线对比同一 task_hash 的多个 prompt 版本效果 (U7/R14). + + 从 EpisodicMemory 检索该 task_hash 的所有 prompt_reflection 记录， + 按 score 降序排列，返回对比结果 + 推荐保留版本。 + + 离线验证 — 不在线 bandit，仅基于历史 score 对比。 + + Returns: + { + "task_hash": str, + "versions": [{"version": int, "score": float, "timestamp": str, + "reflection_summary": str, "improved_prompt": str}], + "best_version": dict | None, # score 最高的版本 + "recommendation": str, # "keep_best" | "no_data" + "total_versions": int, + } + 无 episodic_memory 或无记录时返回 "no_data" recommendation。 + """ + if self._episodic_memory is None: + return { + "task_hash": task_hash, + "versions": [], + "best_version": None, + "recommendation": "no_data", + "total_versions": 0, + } + + try: + items = await self._episodic_memory.list_prompt_reflections_by_hash(task_hash) + except (DBAPIError, RuntimeError, ValueError, KeyError, OSError) as e: + logger.warning(f"U7: compare_prompt_versions retrieval failed: {e}") + return { + "task_hash": task_hash, + "versions": [], + "best_version": None, + "recommendation": "no_data", + "total_versions": 0, + } + + if not items: + return { + "task_hash": task_hash, + "versions": [], + "best_version": None, + "recommendation": "no_data", + "total_versions": 0, + } + + # Sort by score descending (best first) + sorted_items = sorted(items, key=lambda it: it.score, reverse=True) + + versions: list[dict[str, object]] = [] + for item in sorted_items: + meta = item.metadata or {} + value = item.value if isinstance(item.value, dict) else {} + reflection_text = value.get("reflection", "") if isinstance(value, dict) else "" + improved_prompt = value.get("output_summary", "") if isinstance(value, dict) else "" + versions.append( + { + "version": meta.get("version", 1), + "score": item.score or 0.0, + "timestamp": meta.get("created_at", "") + or (item.created_at.isoformat() if item.created_at else ""), + "reflection_summary": (reflection_text[:200] if reflection_text else ""), + "improved_prompt": (improved_prompt[:500] if improved_prompt else ""), + } + ) + + best = versions[0] if versions else None + recommendation = "keep_best" if best else "no_data" + + return { + "task_hash": task_hash, + "versions": versions, + "best_version": best, + "recommendation": recommendation, + "total_versions": len(versions), + } diff --git a/src/agentkit/memory/episodic.py b/src/agentkit/memory/episodic.py index 3ced8af..ca16113 100644 --- a/src/agentkit/memory/episodic.py +++ b/src/agentkit/memory/episodic.py @@ -487,6 +487,68 @@ class EpisodicMemory(Memory): logger.warning(f"U5: failed to search prompt reflections: {e}") return [] + async def list_prompt_reflections_by_hash( + self, + task_hash: str, + agent_name: str | None = None, + ) -> list[MemoryItem]: + """精确查询同一 task_hash 的所有 prompt 反思版本 (U7/R14). + + 用于 ABTester 离线对比同一任务的不同 prompt 版本效果。 + + ponytail: ceiling = O(N) 全表扫描 task_type='prompt_reflection' 记录后 + 在 Python 侧过滤 task_hash。N 通常 <100（一个 task_hash 的版本数有限）。 + 升级路径 = 在 metadata_['task_hash'] 上加 GIN 索引 + JSONB ->> 算符查询。 + + Returns empty list on failure (non-raising). + """ + from sqlalchemy import select + + async with self._session_factory() as db: + try: + Model = self._episodic_model + stmt = ( + select(Model) + .where(Model.task_type == "prompt_reflection") + .order_by(Model.created_at.desc()) + .limit(200) + ) + result = await db.execute(stmt) + entries = result.scalars().all() + except (DBAPIError, ValueError, KeyError, RuntimeError, OSError) as e: + logger.warning(f"U7: list_prompt_reflections_by_hash failed: {e}") + return [] + + items: list[MemoryItem] = [] + for entry in entries: + meta = entry.metadata_ or {} + if meta.get("task_hash") != task_hash: + continue + if agent_name and meta.get("agent_name") != agent_name: + continue + items.append( + MemoryItem( + key=str(entry.id), + value={ + "input_summary": entry.input_summary, + "output_summary": entry.output_summary, + "reflection": entry.reflection, + "quality_score": entry.quality_score, + }, + metadata={ + "agent_name": entry.agent_name, + "task_type": entry.task_type, + "task_hash": meta.get("task_hash", ""), + "version": meta.get("version", 1), + "score": entry.quality_score or 0.0, + "created_at": entry.created_at.isoformat() if entry.created_at else None, + }, + score=entry.quality_score or 0.0, + created_at=entry.created_at or datetime.now(timezone.utc), + ) + ) + return items + async def cleanup_expired(self, max_age_days: int = 30) -> int: """删除超过 max_age_days 天的记录 (U5/R15 TTL). diff --git a/tests/unit/test_ab_tester_prompt.py b/tests/unit/test_ab_tester_prompt.py new file mode 100644 index 0000000..fcf18c4 --- /dev/null +++ b/tests/unit/test_ab_tester_prompt.py @@ -0,0 +1,197 @@ +"""U7: ABTester prompt-version offline comparison (R14). + +Covers: +- compare_prompt_versions(): no episodic_memory → "no_data" +- Multiple versions sorted by score descending +- Single version → that version is best_version +- No matching versions → "no_data" +- Retrieval failure → "no_data" (non-blocking) +- best_version is the highest-scored +- Low-score versions included (no cleanup in compare; recommendation only) +""" + +from __future__ import annotations + +from datetime import datetime, timezone +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from agentkit.evolution.ab_tester import ABTester +from agentkit.memory.base import MemoryItem + + +def _make_item( + version: int, + score: float, + task_hash: str = "abc123", + reflection: str = "reflection text", + improved_prompt: str = "improved prompt", + created_at: datetime | None = None, +) -> MemoryItem: + """Build a MemoryItem mimicking list_prompt_reflections_by_hash output.""" + if created_at is None: + created_at = datetime.now(timezone.utc) + return MemoryItem( + key=f"prompt_reflection:{task_hash}:{version}", + value={ + "input_summary": "task input", + "output_summary": improved_prompt, + "reflection": reflection, + "quality_score": score, + }, + metadata={ + "agent_name": "test_agent", + "task_type": "prompt_reflection", + "task_hash": task_hash, + "version": version, + "score": score, + "created_at": created_at.isoformat(), + }, + score=score, + created_at=created_at, + ) + + +class TestComparePromptVersions: + """Tests for ABTester.compare_prompt_versions().""" + + @pytest.mark.asyncio + async def test_no_episodic_memory_returns_no_data(self): + """No episodic_memory wired → recommendation='no_data', empty versions.""" + tester = ABTester(episodic_memory=None) + result = await tester.compare_prompt_versions("any_hash") + + assert result["recommendation"] == "no_data" + assert result["total_versions"] == 0 + assert result["versions"] == [] + assert result["best_version"] is None + assert result["task_hash"] == "any_hash" + + @pytest.mark.asyncio + async def test_multiple_versions_sorted_by_score_desc(self): + """3 versions (0.8, 0.6, 0.4) → best_version score=0.8.""" + episodic = MagicMock() + episodic.list_prompt_reflections_by_hash = AsyncMock( + return_value=[ + _make_item(version=1, score=0.4), + _make_item(version=2, score=0.8), + _make_item(version=3, score=0.6), + ] + ) + tester = ABTester(episodic_memory=episodic) + + result = await tester.compare_prompt_versions("abc123") + + assert result["total_versions"] == 3 + assert result["recommendation"] == "keep_best" + versions = result["versions"] + # Sorted descending by score + assert versions[0]["score"] == 0.8 + assert versions[1]["score"] == 0.6 + assert versions[2]["score"] == 0.4 + # best_version is the highest-scored + assert result["best_version"]["version"] == 2 + assert result["best_version"]["score"] == 0.8 + + @pytest.mark.asyncio + async def test_single_version_is_best(self): + """Only 1 version → that version is best_version.""" + episodic = MagicMock() + episodic.list_prompt_reflections_by_hash = AsyncMock( + return_value=[_make_item(version=1, score=0.7)] + ) + tester = ABTester(episodic_memory=episodic) + + result = await tester.compare_prompt_versions("abc123") + + assert result["total_versions"] == 1 + assert result["recommendation"] == "keep_best" + assert result["best_version"]["version"] == 1 + assert result["best_version"]["score"] == 0.7 + + @pytest.mark.asyncio + async def test_no_matching_versions_returns_no_data(self): + """EpisodicMemory returns empty list → 'no_data'.""" + episodic = MagicMock() + episodic.list_prompt_reflections_by_hash = AsyncMock(return_value=[]) + tester = ABTester(episodic_memory=episodic) + + result = await tester.compare_prompt_versions("unknown_hash") + + assert result["recommendation"] == "no_data" + assert result["total_versions"] == 0 + assert result["best_version"] is None + + @pytest.mark.asyncio + async def test_retrieval_failure_returns_no_data(self): + """list_prompt_reflections_by_hash raises → 'no_data' (non-blocking).""" + episodic = MagicMock() + episodic.list_prompt_reflections_by_hash = AsyncMock( + side_effect=RuntimeError("db connection failed") + ) + tester = ABTester(episodic_memory=episodic) + + result = await tester.compare_prompt_versions("abc123") + + assert result["recommendation"] == "no_data" + assert result["total_versions"] == 0 + + @pytest.mark.asyncio + async def test_versions_include_reflection_and_prompt_fields(self): + """Each version dict carries reflection_summary + improved_prompt.""" + episodic = MagicMock() + episodic.list_prompt_reflections_by_hash = AsyncMock( + return_value=[ + _make_item( + version=1, + score=0.9, + reflection="avoid retrying on schema error", + improved_prompt="use structured output schema", + ) + ] + ) + tester = ABTester(episodic_memory=episodic) + + result = await tester.compare_prompt_versions("abc123") + + version = result["versions"][0] + assert "avoid retrying on schema error" in version["reflection_summary"] + assert "use structured output schema" in version["improved_prompt"] + assert version["version"] == 1 + assert version["score"] == 0.9 + # timestamp is a string (ISO format) + assert isinstance(version["timestamp"], str) + + @pytest.mark.asyncio + async def test_low_score_versions_kept_in_list(self): + """Low-score versions remain in versions list (no cleanup; recommendation only).""" + episodic = MagicMock() + episodic.list_prompt_reflections_by_hash = AsyncMock( + return_value=[ + _make_item(version=1, score=0.1), # low score + _make_item(version=2, score=0.9), # high score + ] + ) + tester = ABTester(episodic_memory=episodic) + + result = await tester.compare_prompt_versions("abc123") + + # Both versions present — compare does not filter, only sorts + assert result["total_versions"] == 2 + assert result["best_version"]["score"] == 0.9 + # Low-score version still in list (for inspection) + scores = [v["score"] for v in result["versions"]] + assert 0.1 in scores + assert 0.9 in scores + + @pytest.mark.asyncio + async def test_task_hash_echoed_in_result(self): + """task_hash field in result matches input.""" + episodic = MagicMock() + episodic.list_prompt_reflections_by_hash = AsyncMock(return_value=[]) + tester = ABTester(episodic_memory=episodic) + + result = await tester.compare_prompt_versions("specific_hash_456") + + assert result["task_hash"] == "specific_hash_456" From fb86d4e51b0c20105a4ae9fc813c91775d71a08c Mon Sep 17 00:00:00 2001 From: Chiguyong Date: Mon, 6 Jul 2026 14:20:42 +0800 Subject: [PATCH 09/10] =?UTF-8?q?refactor(iq):=20ce-simplify-code=20fixes?= =?UTF-8?q?=20=E2=80=94=20P1=20bug=20+=20P2=20quality?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P1 (bug fix): - store_prompt_reflection: bypass store() and write ORM entry directly so task_hash/version/score land in metadata_ JSONB column. Previously store() dropped these keys, breaking list_prompt_reflections_by_hash filtering (U7 would always return empty). P2 (quality): - DangerousToolsConfig: pre-compile regex patterns in __post_init__ (is_dangerous runs on per-tool hot path in autonomy mode). - ABTester.compare_prompt_versions: extract _no_data_result helper, remove redundant isinstance check after value type guard. - orchestrator: fix misleading "transient TeamPlan" comment. Tests updated to match new store_prompt_reflection implementation (direct ORM write instead of store() delegation). 137 tests pass. --- src/agentkit/evolution/ab_tester.py | 39 +++++------- src/agentkit/experts/orchestrator.py | 3 +- src/agentkit/memory/episodic.py | 64 +++++++++++++------- src/agentkit/server/config.py | 10 +++- tests/unit/test_reflexion_persist.py | 89 ++++++++++++++++------------ 5 files changed, 119 insertions(+), 86 deletions(-) diff --git a/src/agentkit/evolution/ab_tester.py b/src/agentkit/evolution/ab_tester.py index e113987..805aaeb 100644 --- a/src/agentkit/evolution/ab_tester.py +++ b/src/agentkit/evolution/ab_tester.py @@ -226,6 +226,17 @@ class ABTester: # ── IQ-Boost/U7: Prompt-version offline comparison (R14) ──────────── + @staticmethod + def _no_data_result(task_hash: str) -> dict[str, object]: + """Build the empty-result payload for compare_prompt_versions.""" + return { + "task_hash": task_hash, + "versions": [], + "best_version": None, + "recommendation": "no_data", + "total_versions": 0, + } + async def compare_prompt_versions(self, task_hash: str) -> dict[str, object]: """离线对比同一 task_hash 的多个 prompt 版本效果 (U7/R14). @@ -246,34 +257,16 @@ class ABTester: 无 episodic_memory 或无记录时返回 "no_data" recommendation。 """ if self._episodic_memory is None: - return { - "task_hash": task_hash, - "versions": [], - "best_version": None, - "recommendation": "no_data", - "total_versions": 0, - } + return self._no_data_result(task_hash) try: items = await self._episodic_memory.list_prompt_reflections_by_hash(task_hash) except (DBAPIError, RuntimeError, ValueError, KeyError, OSError) as e: logger.warning(f"U7: compare_prompt_versions retrieval failed: {e}") - return { - "task_hash": task_hash, - "versions": [], - "best_version": None, - "recommendation": "no_data", - "total_versions": 0, - } + return self._no_data_result(task_hash) if not items: - return { - "task_hash": task_hash, - "versions": [], - "best_version": None, - "recommendation": "no_data", - "total_versions": 0, - } + return self._no_data_result(task_hash) # Sort by score descending (best first) sorted_items = sorted(items, key=lambda it: it.score, reverse=True) @@ -282,8 +275,8 @@ class ABTester: for item in sorted_items: meta = item.metadata or {} value = item.value if isinstance(item.value, dict) else {} - reflection_text = value.get("reflection", "") if isinstance(value, dict) else "" - improved_prompt = value.get("output_summary", "") if isinstance(value, dict) else "" + reflection_text = value.get("reflection", "") + improved_prompt = value.get("output_summary", "") versions.append( { "version": meta.get("version", 1), diff --git a/src/agentkit/experts/orchestrator.py b/src/agentkit/experts/orchestrator.py index 182732a..feef917 100644 --- a/src/agentkit/experts/orchestrator.py +++ b/src/agentkit/experts/orchestrator.py @@ -559,8 +559,7 @@ class TeamOrchestrator( Returns the original phases if count is within limit or retry fails. """ - # Count independent subtasks via a transient TeamPlan (avoids mutating - # the real plan before execute() finalizes phase list). + # Count phases with no depends_on (independent subtasks). independent_count = sum(1 for ph in phases if not ph.depends_on) if independent_count <= self.MAX_INDEPENDENT_SUBTASKS: return phases diff --git a/src/agentkit/memory/episodic.py b/src/agentkit/memory/episodic.py index ca16113..0a24559 100644 --- a/src/agentkit/memory/episodic.py +++ b/src/agentkit/memory/episodic.py @@ -425,14 +425,21 @@ class EpisodicMemory(Memory): ) -> str | None: """持久化 prompt 反思到 EpisodicMemory，支持跨任务检索 (U5/R11). - Reuses the existing ``store()`` path with ``task_type="prompt_reflection"`` - as the discriminator. The ORM row's fields map: + Writes directly to the ORM row (bypassing ``store()``) so that + ``task_hash``/``version``/``score`` land in the ``metadata_`` JSONB + column — ``store()`` only maps a fixed set of metadata keys to ORM + columns and drops the rest, which would break + ``list_prompt_reflections_by_hash`` filtering. + + The ORM row's fields map: input_summary ← task_input (truncated) output_summary ← improved_prompt (truncated) reflection ← reflection text quality_score ← score (0.0=failed, 1.0=verified) outcome ← "reflection" agent_name ← agent_name + task_type ← "prompt_reflection" + metadata_ ← {task_hash, version, score, timestamp} Returns the storage key ``"prompt_reflection:{task_hash}:{version}"`` on success, or None on failure (non-raising — callers continue without). @@ -443,29 +450,42 @@ class EpisodicMemory(Memory): task_hash = hashlib.sha256(task_input.encode("utf-8")).hexdigest()[:16] key = f"prompt_reflection:{task_hash}:{version}" - value = { - "task_input": task_input[:500], - "reflection": reflection, - "improved_prompt": improved_prompt, - "score": score, - "version": version, + # Generate embedding from task_input for semantic search compatibility. + embedding = None + if self._embedder: + try: + embedding = await self._embedder.embed(task_input) + except (RuntimeError, ValueError, OSError) as e: + logger.warning(f"U5: embedder failed for prompt reflection: {e}") + + extra_meta = { "task_hash": task_hash, + "version": version, + "score": score, "timestamp": datetime.now(timezone.utc).isoformat(), } - metadata = { - "agent_name": agent_name, - "task_type": "prompt_reflection", - "output_summary": improved_prompt[:500], - "outcome": "reflection", - "quality_score": score, - "reflection": reflection, - } - try: - await self.store(key=key, value=value, metadata=metadata) - return key - except (DBAPIError, ValueError, KeyError, RuntimeError, OSError) as e: - logger.warning(f"U5: failed to persist prompt reflection: {e}") - return None + + async with self._session_factory() as db: + try: + Model = self._episodic_model + entry = Model( + agent_name=agent_name, + task_type="prompt_reflection", + input_summary=task_input[:500], + output_summary=improved_prompt[:500], + outcome="reflection", + quality_score=score, + reflection=reflection, + embedding=embedding, + metadata_=extra_meta, + ) + db.add(entry) + await db.commit() + return key + except (DBAPIError, ValueError, KeyError, RuntimeError, OSError) as e: + await db.rollback() + logger.warning(f"U5: failed to persist prompt reflection: {e}") + return None async def search_prompt_reflections( self, diff --git a/src/agentkit/server/config.py b/src/agentkit/server/config.py index d2d85be..747daae 100644 --- a/src/agentkit/server/config.py +++ b/src/agentkit/server/config.py @@ -88,12 +88,20 @@ class DangerousToolsConfig: # U3/R10: autonomy pause thresholds. 0 = disabled (no pause). autonomy_timeout_minutes: int = 30 max_consecutive_failures: int = 3 + # Pre-compiled patterns — populated by __post_init__, reused by is_dangerous + # on the per-tool hot path. ponytail: ceiling = rebuild when tool_patterns + # mutated after construction (rare; config-time only). Upgrade path = + # cached_property with invalidation on tool_patterns setter. + _compiled_patterns: list[re.Pattern] = field(default_factory=list, repr=False) + + def __post_init__(self) -> None: + self._compiled_patterns = [re.compile(p) for p in self.tool_patterns] def is_dangerous(self, tool_name: str) -> bool: """Check if a tool name matches any dangerous pattern.""" if not self.enabled: return False - return any(re.match(p, tool_name) for p in self.tool_patterns) + return any(p.match(tool_name) for p in self._compiled_patterns) @classmethod def from_dict(cls, data: dict | None) -> "DangerousToolsConfig": diff --git a/tests/unit/test_reflexion_persist.py b/tests/unit/test_reflexion_persist.py index 1473627..14fe06f 100644 --- a/tests/unit/test_reflexion_persist.py +++ b/tests/unit/test_reflexion_persist.py @@ -58,15 +58,29 @@ def _make_llm_gateway_mock(reflection_text: str = "reflection text") -> MagicMoc class TestStorePromptReflection: - """U5/R11: store_prompt_reflection persists via store() with - task_type='prompt_reflection' discriminator.""" + """U5/R11: store_prompt_reflection persists via direct ORM write with + task_type='prompt_reflection' discriminator and metadata_ JSONB.""" + + def _make_persistable_mock(self) -> MagicMock: + """Build an EpisodicMemory mock with _embedder + _session_factory + configured so store_prompt_reflection can run its real code path.""" + mem = MagicMock(spec=EpisodicMemory) + mem._embedder = None # skip embedding generation in test + mock_db = AsyncMock() + mock_db.add = MagicMock() + mock_db.commit = AsyncMock() + mock_db.rollback = AsyncMock() + mem._session_factory = MagicMock() + mem._session_factory.return_value.__aenter__ = AsyncMock(return_value=mock_db) + mem._session_factory.return_value.__aexit__ = AsyncMock(return_value=None) + mem._episodic_model = MagicMock() + # Track the ORM entry passed to db.add for assertions + mem._mock_db = mock_db + return mem @pytest.mark.asyncio - async def test_store_calls_underlying_store_with_correct_metadata(self): - mem = MagicMock(spec=EpisodicMemory) - mem.store = AsyncMock(return_value=None) - # We need to call the real method, not a mock — patch store only - # Use the unbound method pattern + async def test_store_writes_orm_entry_with_correct_fields(self): + mem = self._make_persistable_mock() await EpisodicMemory.store_prompt_reflection( mem, task_input="test task", @@ -77,30 +91,26 @@ class TestStorePromptReflection: agent_name="test_agent", ) - mem.store.assert_awaited_once() - call_kwargs = mem.store.await_args.kwargs - assert "key" in call_kwargs - assert call_kwargs["key"].startswith("prompt_reflection:") - assert ":1" in call_kwargs["key"] - - value = call_kwargs["value"] - assert value["task_input"] == "test task" - assert value["reflection"] == "reflection text" - assert value["improved_prompt"] == "improved prompt" - assert value["score"] == 0.5 - assert value["version"] == 1 - assert "timestamp" in value - - metadata = call_kwargs["metadata"] - assert metadata["task_type"] == "prompt_reflection" - assert metadata["agent_name"] == "test_agent" - assert metadata["quality_score"] == 0.5 - assert metadata["reflection"] == "reflection text" + # Verify ORM entry created via model constructor + db.add + commit + mem._episodic_model.assert_called_once() + call_kwargs = mem._episodic_model.call_args.kwargs + assert call_kwargs["task_type"] == "prompt_reflection" + assert call_kwargs["agent_name"] == "test_agent" + assert call_kwargs["input_summary"] == "test task" + assert call_kwargs["output_summary"] == "improved prompt" + assert call_kwargs["reflection"] == "reflection text" + assert call_kwargs["quality_score"] == 0.5 + assert call_kwargs["outcome"] == "reflection" + # metadata_ JSONB carries task_hash + version + score + assert "task_hash" in call_kwargs["metadata_"] + assert call_kwargs["metadata_"]["version"] == 1 + assert call_kwargs["metadata_"]["score"] == 0.5 + mem._mock_db.add.assert_called_once() + mem._mock_db.commit.assert_awaited_once() @pytest.mark.asyncio async def test_store_returns_key_on_success(self): - mem = MagicMock(spec=EpisodicMemory) - mem.store = AsyncMock(return_value=None) + mem = self._make_persistable_mock() key = await EpisodicMemory.store_prompt_reflection( mem, task_input="test", @@ -116,8 +126,10 @@ class TestStorePromptReflection: async def test_store_returns_none_on_failure(self): from sqlalchemy.exc import DBAPIError - mem = MagicMock(spec=EpisodicMemory) - mem.store = AsyncMock(side_effect=DBAPIError("stmt", params={}, orig=Exception("db down"))) + mem = self._make_persistable_mock() + mem._mock_db.commit = AsyncMock( + side_effect=DBAPIError("stmt", params={}, orig=Exception("db down")) + ) key = await EpisodicMemory.store_prompt_reflection( mem, task_input="test", @@ -125,11 +137,11 @@ class TestStorePromptReflection: improved_prompt="p", ) assert key is None + mem._mock_db.rollback.assert_awaited_once() @pytest.mark.asyncio async def test_store_uses_provided_task_hash(self): - mem = MagicMock(spec=EpisodicMemory) - mem.store = AsyncMock(return_value=None) + mem = self._make_persistable_mock() key = await EpisodicMemory.store_prompt_reflection( mem, task_input="test", @@ -141,8 +153,7 @@ class TestStorePromptReflection: @pytest.mark.asyncio async def test_store_generates_task_hash_from_input(self): - mem = MagicMock(spec=EpisodicMemory) - mem.store = AsyncMock(return_value=None) + mem = self._make_persistable_mock() key1 = await EpisodicMemory.store_prompt_reflection( mem, task_input="same task", reflection="r", improved_prompt="p" ) @@ -401,8 +412,9 @@ class TestMultiVersionCoexistence: @pytest.mark.asyncio async def test_multiple_versions_stored_with_incrementing_version(self): - mem = MagicMock(spec=EpisodicMemory) - mem.store = AsyncMock(return_value=None) + # Reuse the persistable mock helper from TestStorePromptReflection. + helper = TestStorePromptReflection() + mem = helper._make_persistable_mock() # Store v1, v2, v3 for same task key1 = await EpisodicMemory.store_prompt_reflection( @@ -419,5 +431,6 @@ class TestMultiVersionCoexistence: assert key1 != key2 != key3 assert ":1" in key1 and ":2" in key2 and ":3" in key3 - # All three store() calls made - assert mem.store.await_count == 3 + # All three ORM entries created via db.add + assert mem._mock_db.add.call_count == 3 + assert mem._mock_db.commit.await_count == 3 From aacec299489a7d64e6892903fc68902091d4b6df Mon Sep 17 00:00:00 2001 From: Chiguyong Date: Mon, 6 Jul 2026 15:07:59 +0800 Subject: [PATCH 10/10] =?UTF-8?q?fix(iq):=20apply=20code=20review=20fixes?= =?UTF-8?q?=20=E2=80=94=20P0=20deadlock=20+=20P1=20gate/tracking/filter?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P0 #1: Fix autonomy_paused event deadlock — split _check_autonomy_pause into _detect_autonomy_pause (non-blocking) + _await_autonomy_resume (blocking). Caller now yields the pause event BEFORE awaiting resume, so the frontend receives it and the resume handler doesn't deadlock. P1 #2+#6: Fix retrieve_prompt_reflection field mapping — read output_summary/reflection/quality_score from MemoryItem.value dict (matching EpisodicMemory.search shape) instead of metadata. Score filtering uses stored quality_score, not search relevance score. P1 #3: Add optional task_type filter to cleanup_expired so prompt_reflection TTL cleanup doesn't delete all episodic records. P1 #4: Disable parallel tool execution in autonomy mode — dangerous tools must go through _check_autonomy_gate, which only runs in the serial path. P1 #5: Add _track_tool_result_for_autonomy to parallel result loop so tool failures are counted toward the consecutive_failures threshold. Tests: adapt test_autonomy_paused.py to new detect/await interface; fix test_lead_reflection_retrieval.py mock shape (fields in value dict). 137 IQ-boost tests pass, ruff clean. --- src/agentkit/core/react.py | 122 +++++++++++-------- src/agentkit/core/reflexion.py | 31 +++-- src/agentkit/memory/episodic.py | 11 +- tests/unit/test_autonomy_paused.py | 77 ++++++++---- tests/unit/test_lead_reflection_retrieval.py | 14 ++- 5 files changed, 165 insertions(+), 90 deletions(-) diff --git a/src/agentkit/core/react.py b/src/agentkit/core/react.py index f7f14c9..e2dc0c6 100644 --- a/src/agentkit/core/react.py +++ b/src/agentkit/core/react.py @@ -1206,8 +1206,12 @@ class ReActEngine: tc.id, tool_result, compressor, tc.name ) conversation.append(tool_msg) - elif self._should_execute_parallel(response.tool_calls): + elif self._should_execute_parallel(response.tool_calls) and not ( + self._autonomy_mode and self._dangerous_tools_config is not None + ): # 并行执行多个工具调用 (parallel_tools=True) + # IQ-Boost/U2: autonomy mode forces serial execution so + # dangerous tools are always gated by _check_autonomy_gate. tool_results = await asyncio.gather( *[ self._execute_tool(tc.name, tc.arguments, tools) @@ -1220,6 +1224,10 @@ class ReActEngine: if isinstance(tool_result, Exception): tool_result = {"error": str(tool_result)} + # IQ-Boost/U3/R10: track failures for autonomy pause + # (no-op when autonomy mode is off). + self._track_tool_result_for_autonomy(tool_result) + yield ReActEvent( event_type="tool_call", step=step, @@ -1274,19 +1282,27 @@ class ReActEngine: # IQ-Boost/U3/R10: autonomy pause check (timeout/failures). # If paused and resume_handler returns False (cancel), # break out of the tool_calls loop. + # Split into detect→yield→await to avoid deadlock: the + # pause event must be yielded BEFORE awaiting resume. _progress = { "step": step, "tool_name": tc.name, "total_steps": len(trajectory), } - should_continue, pause_events = await self._check_autonomy_pause( - step, _progress, resume_handler - ) - for _pev in pause_events: - yield _pev - if not should_continue: - # User cancelled during pause — stop execution. - break + _pause_info = self._detect_autonomy_pause(step, _progress) + if _pause_info is not None: + _reason, _token, _edata = _pause_info + yield ReActEvent( + event_type="autonomy_paused", + step=step, + data=_edata, + ) + _should_continue = await self._await_autonomy_resume( + _token, _reason, resume_handler + ) + if not _should_continue: + # User cancelled during pause — stop execution. + break # IQ-Boost/U2/R7: autonomy mode pre-execution gate. # Dangerous tools (per config) get a confirmation_request @@ -2378,35 +2394,29 @@ class ReActEngine: }, ) - async def _check_autonomy_pause( + def _detect_autonomy_pause( self, step: int, progress: dict[str, object], - resume_handler: Callable[..., Awaitable[object]] | None, - ) -> tuple[bool, list[ReActEvent]]: - """IQ-Boost/U3/R10: check autonomy pause conditions before tool execution. + ) -> tuple[str, str, dict[str, object]] | None: + """IQ-Boost/U3/R10: detect autonomy pause conditions (non-blocking). - Returns ``(should_continue, events)``: - - should_continue=True → no pause needed; proceed with tool execution - - should_continue=False → autonomy_paused yielded; caller must break - the loop (resume_handler already returned False / cancelled) OR - retry (resume_handler returned True — counters reset, proceed). + Returns ``(reason, resume_token, event_data)`` when a pause is + triggered, or ``None`` when no pause is needed. - When pause is triggered: - 1. Yield ``autonomy_paused`` event with resume_token + reason + progress. - 2. Call ``resume_handler(resume_token, reason)`` — blocks until user - sends ``resume`` (returns True) or cancels (returns False). - 3. On resume: reset ``_autonomy_started_at`` + ``_consecutive_failures``, - return should_continue=True to let the caller retry the tool. - 4. On cancel: return should_continue=False to break the loop. + The caller is responsible for: + 1. Yielding the ``autonomy_paused`` event (built from event_data) + **before** awaiting ``_await_autonomy_resume`` — otherwise the + frontend never receives the pause event and the resume handler + deadlocks. + 2. Awaiting ``_await_autonomy_resume(resume_token, reason, handler)`` + to block until the user resumes or cancels. - If ``resume_handler`` is None (non-WS callers, tests), auto-resume - immediately (no blocking) — the pause event is still yielded for - observability, but the loop continues without waiting. + This split fixes the original deadlock where the event was buffered + inside this method and only yielded after the handler returned. """ - events: list[ReActEvent] = [] if not self._autonomy_mode or self._dangerous_tools_config is None: - return (True, events) + return None cfg = self._dangerous_tools_config reason: str | None = None @@ -2425,34 +2435,42 @@ class ReActEngine: reason = "consecutive_failures" if reason is None: - return (True, events) + return None resume_token = f"autonomy_pause:{reason}:{step}" - events.append( - ReActEvent( - event_type="autonomy_paused", - step=step, - data={ - "reason": reason, - "progress": progress, - "resume_token": resume_token, - "consecutive_failures": self._consecutive_failures, - "elapsed_seconds": ( - time.time() - self._autonomy_started_at - if self._autonomy_started_at > 0 - else 0 - ), - }, - ) - ) + event_data: dict[str, object] = { + "reason": reason, + "progress": progress, + "resume_token": resume_token, + "consecutive_failures": self._consecutive_failures, + "elapsed_seconds": ( + time.time() - self._autonomy_started_at + if self._autonomy_started_at > 0 + else 0 + ), + } + return (reason, resume_token, event_data) - # Wait for user resume or cancel + async def _await_autonomy_resume( + self, + resume_token: str, + reason: str, + resume_handler: Callable[..., Awaitable[object]] | None, + ) -> bool: + """Block until user resumes or cancels autonomy pause (U3/R10). + + Returns True on resume (counters reset, caller retries the tool), + False on cancel (caller breaks the loop). + + If ``resume_handler`` is None (non-WS callers, tests), auto-resume + immediately without blocking. + """ if resume_handler is None: # Non-WS caller (tests, REST) — auto-resume without blocking. logger.warning("autonomy_paused (%s) with no resume_handler — auto-resuming", reason) self._autonomy_started_at = time.time() self._consecutive_failures = 0 - return (True, events) + return True try: should_resume = await resume_handler(resume_token, reason) @@ -2466,9 +2484,9 @@ class ReActEngine: # Reset counters and continue self._autonomy_started_at = time.time() self._consecutive_failures = 0 - return (True, events) + return True # Cancelled by user — break the loop - return (False, events) + return False def _track_tool_result_for_autonomy(self, tool_result: object) -> None: """U3/R10: track tool failures for consecutive_failures threshold. diff --git a/src/agentkit/core/reflexion.py b/src/agentkit/core/reflexion.py index 5facdc3..10a1fb7 100644 --- a/src/agentkit/core/reflexion.py +++ b/src/agentkit/core/reflexion.py @@ -750,13 +750,20 @@ class ReflexionEngine: """检索历史 prompt 反思，返回最佳版本 (U6/R12, R13). Searches EpisodicMemory for similar task_input reflections with - score > min_score. Returns the highest-scored reflection as: + stored quality_score > min_score. Returns the highest-quality + reflection as: {improved_prompt, score, reflection, version, task_hash} or None if no episodic_memory / no results / all below threshold. KTD5: callers should only invoke this when a trigger condition is met (verify failure / schema failure / loop detection) to avoid pointless retrieval on every task. + + Note: ``item.score`` from ``EpisodicMemory.search`` is the hybrid + relevance score (cosine + time_decay), NOT the stored quality_score. + We read ``quality_score`` from ``item.value`` for filtering/ranking + so that a high-quality reflection (score=1.0) is preferred over a + low-quality one (score=0.0) regardless of textual similarity. """ if self._episodic_memory is None: return None @@ -772,24 +779,32 @@ class ReflexionEngine: if not results: return None - # Filter by min_score, pick the highest-scored + # Filter by stored quality_score (from value dict), pick the highest. + # Fallback to item.score (relevance) when quality_score is absent. best: MemoryItem | None = None + best_quality: float = 0.0 for item in results: - score = item.score or 0.0 - if score > min_score and (best is None or score > (best.score or 0.0)): + value = item.value if isinstance(item.value, dict) else {} + quality = float(value.get("quality_score", 0.0) or 0.0) + if quality <= 0.0: + # Fallback to relevance score if quality_score missing + quality = float(item.score or 0.0) + if quality > min_score and (best is None or quality > best_quality): best = item + best_quality = quality if best is None: return None - # Extract improved_prompt from metadata (output_summary field) + # Read fields from value dict (matching EpisodicMemory.search shape) + value = best.value if isinstance(best.value, dict) else {} + improved_prompt = value.get("output_summary", "") or "" + reflection_text = value.get("reflection", "") or "" metadata = best.metadata or {} - improved_prompt = metadata.get("output_summary", "") or metadata.get("improved_prompt", "") - reflection_text = metadata.get("reflection", "") or best.value or "" return { "improved_prompt": improved_prompt, - "score": best.score or 0.0, + "score": best_quality, "reflection": reflection_text, "version": metadata.get("version", 1), "task_hash": metadata.get("task_hash", ""), diff --git a/src/agentkit/memory/episodic.py b/src/agentkit/memory/episodic.py index 0a24559..3fc5a2b 100644 --- a/src/agentkit/memory/episodic.py +++ b/src/agentkit/memory/episodic.py @@ -569,9 +569,16 @@ class EpisodicMemory(Memory): ) return items - async def cleanup_expired(self, max_age_days: int = 30) -> int: + async def cleanup_expired( + self, max_age_days: int = 30, task_type: str | None = None + ) -> int: """删除超过 max_age_days 天的记录 (U5/R15 TTL). + Args: + max_age_days: Records older than this many days are deleted. + task_type: Optional filter — only delete records with this task_type + (e.g. ``"prompt_reflection"``). None = delete all task types. + Returns the number of deleted rows. 0 on failure (non-raising). """ from datetime import timedelta @@ -583,6 +590,8 @@ class EpisodicMemory(Memory): try: Model = self._episodic_model stmt = sql_delete(Model).where(Model.created_at < cutoff) + if task_type is not None: + stmt = stmt.where(Model.task_type == task_type) result = await db.execute(stmt) await db.commit() return result.rowcount or 0 diff --git a/tests/unit/test_autonomy_paused.py b/tests/unit/test_autonomy_paused.py index c4b6ef5..417befb 100644 --- a/tests/unit/test_autonomy_paused.py +++ b/tests/unit/test_autonomy_paused.py @@ -18,10 +18,37 @@ from unittest.mock import AsyncMock, MagicMock import pytest -from agentkit.core.react import ReActEngine +from agentkit.core.react import ReActEngine, ReActEvent from agentkit.server.config import DangerousToolsConfig +# --------------------------------------------------------------------------- +# Helper: simulates the caller pattern (detect → yield event → await resume) +# --------------------------------------------------------------------------- + + +async def _detect_and_await_pause( + engine: ReActEngine, + step: int, + progress: dict, + resume_handler=None, +) -> tuple[bool, list[ReActEvent]]: + """Simulate the react.py caller pattern for autonomy pause. + + Mirrors the production code: detect (non-blocking) → build event → + yield event → await resume (blocking). Returns (should_continue, events). + """ + pause_info = engine._detect_autonomy_pause(step, progress) + if pause_info is None: + return True, [] + reason, token, event_data = pause_info + event = ReActEvent(event_type="autonomy_paused", step=step, data=event_data) + should_continue = await engine._await_autonomy_resume( + token, reason, resume_handler + ) + return should_continue, [event] + + # --------------------------------------------------------------------------- # _track_tool_result_for_autonomy # --------------------------------------------------------------------------- @@ -90,12 +117,12 @@ class TestTrackToolResult: # --------------------------------------------------------------------------- -# _check_autonomy_pause — no pause conditions +# _detect_autonomy_pause + _await_autonomy_resume — no pause conditions # --------------------------------------------------------------------------- class TestAutonomyPauseNoTrigger: - """When conditions are not met, _check_autonomy_pause returns should_continue=True.""" + """When conditions are not met, no pause is triggered (should_continue=True).""" @pytest.mark.asyncio async def test_no_autonomy_mode_no_pause(self): @@ -104,8 +131,8 @@ class TestAutonomyPauseNoTrigger: dangerous_tools_config=DangerousToolsConfig(), autonomy_mode=False, ) - should_continue, events = await engine._check_autonomy_pause( - step=1, progress={}, resume_handler=None + should_continue, events = await _detect_and_await_pause( + engine, step=1, progress={}, resume_handler=None ) assert should_continue is True assert events == [] @@ -117,8 +144,8 @@ class TestAutonomyPauseNoTrigger: dangerous_tools_config=None, autonomy_mode=True, ) - should_continue, events = await engine._check_autonomy_pause( - step=1, progress={}, resume_handler=None + should_continue, events = await _detect_and_await_pause( + engine, step=1, progress={}, resume_handler=None ) assert should_continue is True assert events == [] @@ -135,8 +162,8 @@ class TestAutonomyPauseNoTrigger: ) engine._autonomy_started_at = time.time() # Just started engine._consecutive_failures = 1 # Below threshold - should_continue, events = await engine._check_autonomy_pause( - step=1, progress={}, resume_handler=None + should_continue, events = await _detect_and_await_pause( + engine, step=1, progress={}, resume_handler=None ) assert should_continue is True assert events == [] @@ -155,15 +182,15 @@ class TestAutonomyPauseNoTrigger: # Even with expired time + many failures, disabled = no pause. engine._autonomy_started_at = time.time() - 999999 engine._consecutive_failures = 99 - should_continue, events = await engine._check_autonomy_pause( - step=1, progress={}, resume_handler=None + should_continue, events = await _detect_and_await_pause( + engine, step=1, progress={}, resume_handler=None ) assert should_continue is True assert events == [] # --------------------------------------------------------------------------- -# _check_autonomy_pause — timeout trigger +# _detect_autonomy_pause + _await_autonomy_resume — timeout trigger # --------------------------------------------------------------------------- @@ -184,8 +211,8 @@ class TestAutonomyPauseTimeout: # Simulate started 2 minutes ago (exceeds 1-min timeout). engine._autonomy_started_at = time.time() - 120 - should_continue, events = await engine._check_autonomy_pause( - step=5, progress={"step": 5}, resume_handler=None + should_continue, events = await _detect_and_await_pause( + engine, step=5, progress={"step": 5}, resume_handler=None ) # Auto-resume → should_continue=True @@ -212,8 +239,8 @@ class TestAutonomyPauseTimeout: engine._autonomy_started_at = time.time() - 120 resume_handler = AsyncMock(return_value=True) - should_continue, events = await engine._check_autonomy_pause( - step=3, progress={}, resume_handler=resume_handler + should_continue, events = await _detect_and_await_pause( + engine, step=3, progress={}, resume_handler=resume_handler ) assert should_continue is True @@ -235,8 +262,8 @@ class TestAutonomyPauseTimeout: engine._autonomy_started_at = time.time() - 120 resume_handler = AsyncMock(return_value=False) - should_continue, events = await engine._check_autonomy_pause( - step=3, progress={}, resume_handler=resume_handler + should_continue, events = await _detect_and_await_pause( + engine, step=3, progress={}, resume_handler=resume_handler ) assert should_continue is False # Cancelled @@ -244,7 +271,7 @@ class TestAutonomyPauseTimeout: # --------------------------------------------------------------------------- -# _check_autonomy_pause — consecutive failures trigger +# _detect_autonomy_pause + _await_autonomy_resume — consecutive failures trigger # --------------------------------------------------------------------------- @@ -265,8 +292,8 @@ class TestAutonomyPauseConsecutiveFailures: engine._consecutive_failures = 3 # At threshold resume_handler = AsyncMock(return_value=True) - should_continue, events = await engine._check_autonomy_pause( - step=5, progress={"step": 5}, resume_handler=resume_handler + should_continue, events = await _detect_and_await_pause( + engine, step=5, progress={"step": 5}, resume_handler=resume_handler ) assert should_continue is True # Resumed @@ -290,8 +317,8 @@ class TestAutonomyPauseConsecutiveFailures: engine._autonomy_started_at = time.time() engine._consecutive_failures = 2 # Below threshold - should_continue, events = await engine._check_autonomy_pause( - step=1, progress={}, resume_handler=None + should_continue, events = await _detect_and_await_pause( + engine, step=1, progress={}, resume_handler=None ) assert should_continue is True @@ -313,8 +340,8 @@ class TestAutonomyPauseConsecutiveFailures: async def failing_handler(token, reason): raise RuntimeError("handler crashed") - should_continue, events = await engine._check_autonomy_pause( - step=1, progress={}, resume_handler=failing_handler + should_continue, events = await _detect_and_await_pause( + engine, step=1, progress={}, resume_handler=failing_handler ) assert should_continue is False # Cancelled due to exception diff --git a/tests/unit/test_lead_reflection_retrieval.py b/tests/unit/test_lead_reflection_retrieval.py index 263094f..1554358 100644 --- a/tests/unit/test_lead_reflection_retrieval.py +++ b/tests/unit/test_lead_reflection_retrieval.py @@ -35,16 +35,22 @@ def _make_memory_item( ) -> MemoryItem: from datetime import datetime, timezone + # Shape matches EpisodicMemory.list_prompt_reflections_by_hash return: + # output_summary/reflection/quality_score live in value dict, + # version/task_hash live in metadata. return MemoryItem( key="prompt_reflection:abc:1", - value={"task_input": "test", "reflection": reflection}, - metadata={ - "task_type": "prompt_reflection", + value={ + "input_summary": "test", "output_summary": output_summary, "reflection": reflection, + "quality_score": score, + }, + metadata={ + "task_type": "prompt_reflection", "version": version, "task_hash": "abc", - "quality_score": score, + "created_at": datetime.now(timezone.utc).isoformat(), }, score=score, created_at=datetime.now(timezone.utc),