Merge remote-tracking branch 'origin/main' into fix/private-board-llm
This commit is contained in:
Chiguyong
commit
1c33859904
|
|
@ -0,0 +1,502 @@
|
|||
---
|
||||
date: 2026-07-06
|
||||
topic: agent-iq-boost
|
||||
type: feature
|
||||
artifact_contract: ce-unified-plan/v1
|
||||
artifact_readiness: implementation-ready
|
||||
product_contract_source: ce-brainstorm
|
||||
execution: code
|
||||
origin: "竞品对标(2026-07 Qoder 1.11/Cursor 3.2/Devin 3.0);9 缺口(G1-G9)已交付后的新维度探索"
|
||||
---
|
||||
|
||||
## Goal Capsule
|
||||
|
||||
**Objective**: 扩展现有 PLAN_EXEC / TEAM_COLLAB / ReflexionEngine 三个子系统，提升 agent 处理复杂任务的智商 — 独立子任务并行执行、plan 确认后自主执行（仅危险操作确认）、失败后 prompt 跨任务自调优。
|
||||
|
||||
**Product Authority**: AgentKit 横向评估 14 维度总分 42/56，9 缺口已交付但竞品在"编排+执行深度"维度持续领先（Cursor Subagents 并行、Devin Goal-driven 自主、Qoder 多模型路由）。本 plan 对标竞品补齐这一维度。
|
||||
|
||||
**Open Blockers**: None — OQ1-3 已在 Planning Contract KTD6-8 决策。
|
||||
|
||||
**Execution Profile**: Standard depth，7 个 Implementation Units，3 个维度可部分并行（维度 1 与维度 3 独立；维度 2 内部 U1→U2→U3 串行）。
|
||||
|
||||
**Stop Conditions**: 7 个 U-ID 全部完成、Verification Contract 通过、DoD 全局条件满足。
|
||||
|
||||
---
|
||||
|
||||
## Product Contract
|
||||
|
||||
### Summary
|
||||
|
||||
3 个维度并行推进，不新建 ExecutionMode，全部扩展现有基础设施：TeamOrchestrator 新增"无依赖子任务并行"模式解决串行排队；PLAN_EXEC 的"每阶段确认"改为"危险操作确认"解决要人盯；ReflexionEngine 反思结果持久化到 EpisodicMemory 实现跨任务 prompt 自调优。对标 Cursor Agent Mode / Devin Quest Mode / Qoder Goal-driven 的智商能力，保持 AgentKit 现有架构不变。
|
||||
|
||||
### Problem Frame
|
||||
|
||||
2026-06-29 识别的 9 个智商短板（G1-G9）已全部交付（wave1-4），覆盖反馈稳定性、响应效率、执行能力三个维度。但 2026-07 最新竞品动态显示，竞品在"编排+执行深度"维度持续领先：
|
||||
|
||||
- **Cursor 3.2 Subagents**（2026-04）：并行专业化 worker，多 agent 同时工作不同部分
|
||||
- **Devin 3.0 Goal-driven**（2026-05）：设定目标后自主执行数小时，仅失败时介入
|
||||
- **Qoder 1.11 Goal-driven + Scheduling**（2026-06）：Goal 设定后工作到完成，可定时启动
|
||||
|
||||
AgentKit 当前痛点：
|
||||
- 5 个独立子任务必须串行排队（TeamOrchestrator 同层并行已存在，但"无依赖子任务并行"未实现）
|
||||
- plan 生成后每阶段需人工确认（PLAN_EXEC 阶段确认机制太保守）
|
||||
- ReflexionEngine 仅在单次任务内反思，同类错误跨任务重复犯（反思结果未持久化）
|
||||
|
||||
### Key Decisions
|
||||
|
||||
**KTD1: 方案 A — 扩展现有模式，不新建 ExecutionMode**
|
||||
|
||||
3 个维度全部扩展现有 PLAN_EXEC / TEAM_COLLAB / ReflexionEngine，不新建 GOAL_DRIVEN 模式。理由：复用现有基础设施，路由/前端/配置不动，风险低；KTD7（06-29 G6 阶段约束）已验证扩展现有模式可行。
|
||||
|
||||
**KTD2: 危险操作保守白名单，不引入 LLM 辅助分类**
|
||||
|
||||
自主执行时仅危险操作触发 confirmation_request，白名单包括：文件删除（rm/rmdir）、部署操作（deploy/kubectl/helm）、支付相关、git push --force、数据库迁移（alembic/migrate）。白名单外的操作自主执行。ponytail: ceiling = 漏判风险（未列入白名单的危险操作），升级路径 = LLM 辅助分类。
|
||||
|
||||
**KTD3: 复用 topological_sort 的同层并行，新增"无依赖子任务并行"模式**
|
||||
|
||||
TeamOrchestrator 已有 topological_sort 返回执行层（Kahn 算法），同层并行已实现。新增"无依赖子任务并行"模式：Lead 分解任务时，若识别到多个无 depends_on 的子任务，自动派发给隔离 agent 并行执行，而非串行排队。不新建 SubAgentOrchestrator 模块（方案 C 排除）。
|
||||
|
||||
**KTD4: ReflexionEngine 反思结果持久化到 EpisodicMemory**
|
||||
|
||||
失败后 ReflexionEngine 生成反思（已有），新增：反思结果写入 EpisodicMemory（task_input + reflection + improved_prompt），下次类似任务规划时 Lead 检索历史反思，用 improved_prompt 替换默认 prompt。不引入强化学习/元学习（Devin 风格太重），仅做跨任务持久化。
|
||||
|
||||
**KTD5: Prompt 自调优仅对特定错误类型触发**
|
||||
|
||||
不是每次失败都触发 prompt 调优（避免无意义反思浪费 token）。触发条件：verify 失败（G1 回灌后二次失败）、工具 schema 校验失败（G3）、循环检测触发（U1）。调优后的 prompt 带 version 存入 EpisodicMemory，ABTester 可对比版本效果（离线验证，不在线 bandit）。
|
||||
|
||||
### Requirements
|
||||
|
||||
**维度 1: 并行 Sub-agent 编排**
|
||||
|
||||
- R1. TeamOrchestrator 的 Lead 分解任务时，识别无 depends_on 的子任务，自动派发给隔离 ConfigDrivenAgent 并行执行。
|
||||
- R2. 并行子任务的 SharedWorkspace 输出路径必须不重叠（plan 阶段强制约束：`{plan_id}/phase/{phase_id}/output` 已唯一，并行子任务用不同 phase_id）。
|
||||
- R3. 并行子任务全部完成后，Lead 才进入综合阶段（synthesis），与现有 topological_sort 的层间串行一致。
|
||||
- R4. 并行子任务数受 MAX_EXPERTS=10 约束（已有），超出时 Lead 重新分解或串行排队。
|
||||
- R5. 并行子任务的 expert_step / expert_result 事件携带 expert_id，前端已支持多 expert 同时 streaming（已验证）。
|
||||
|
||||
**维度 2: Goal-driven 自主执行**
|
||||
|
||||
- R6. PLAN_EXEC 模式下，plan 确认后进入自主执行，不再每阶段 confirmation_request。
|
||||
- R7. 自主执行期间，仅危险操作（KTD2 白名单）触发 confirmation_request，其他操作自主执行。
|
||||
- R8. 危险操作白名单配置化（`agentkit.yaml` 新增 `dangerous_tools` 配置节，遵循 ServerConfig.from_dict 模式）。
|
||||
- R9. 自主执行期间，用户可随时发送 `cancel` 中断（已有 CancellationToken 机制）。
|
||||
- R10. 自主执行超时（默认 30 分钟，可配置）或连续失败 3 次时，自动暂停并通知用户。
|
||||
|
||||
**维度 3: Prompt 跨任务自调优**
|
||||
|
||||
- R11. ReflexionEngine 反思结果写入 EpisodicMemory（task_input + reflection + improved_prompt + version）。
|
||||
- R12. PLAN_EXEC / TEAM_COLLAB 的 Lead 规划时，检索 EpisodicMemory 中相似 task_input 的历史反思，用 improved_prompt 替换默认 prompt。
|
||||
- R13. Prompt 自调优仅对特定错误类型触发（KTD5）：verify 二次失败、工具 schema 校验失败、循环检测触发。
|
||||
- R14. 调优后的 prompt 带 version 存入，ABTester 可对比版本效果（离线验证）。
|
||||
- R15. EpisodicMemory 中 prompt 反思记录有 TTL（默认 30 天），过期自动清理避免噪声。
|
||||
|
||||
### Scope Boundaries
|
||||
|
||||
**In scope**:
|
||||
- 扩展 PLAN_EXEC / TEAM_COLLAB / ReflexionEngine
|
||||
- 危险操作白名单配置
|
||||
- EpisodicMemory prompt 反思持久化
|
||||
|
||||
**Out of scope**:
|
||||
- 新建 ExecutionMode.GOAL_DRIVEN（方案 B 排除）
|
||||
- 独立可组合模块 SubAgentOrchestrator / DangerGate / OnlinePromptOptimizer（方案 C 排除）
|
||||
- 多模态输入（维度 6，不在"编排+执行深度"范围）
|
||||
- Cloud Agent 远程执行（维度 8，不在本次范围）
|
||||
- Repo 深度索引 + Wiki 自动生成（维度 4，不在本次范围）
|
||||
- 主动澄清意图（维度 1，不在本次范围）
|
||||
- 强化学习 / 元学习 / bandit 探索（Devin 风格在线学习太重，仅做 ReflexionEngine 跨任务持久化）
|
||||
- 项目级 Memory Files（CLAUDE.md/AGENTS.md 风格，维度 2，不在本次范围）
|
||||
|
||||
### Outstanding Questions
|
||||
|
||||
All OQ1-3 resolved in Planning Contract (KTD6-KTD8). No outstanding questions remain.
|
||||
|
||||
---
|
||||
|
||||
## Planning Contract
|
||||
|
||||
### Key Technical Decisions (Planning-Time)
|
||||
|
||||
**KTD6: 并行子任务 depends_on 完全独立约束（解 OQ1）**
|
||||
|
||||
并行子任务的 depends_on 必须完全独立 — 不能共享上游依赖。若 Lead 识别到共享上游依赖，降级为同层并行（topological_sort 已有机制）。此外，关键写入操作用 `SharedWorkspace.lock(key, agent_id, timeout)` 防护（已有 API，`core/shared_workspace.py:96`）。决策依据：phase_id 唯一性已保证输出路径不重叠，但读上游输出时存在 TOCTOU 风险，lock 是最小成本防护。
|
||||
|
||||
**KTD7: Prompt 版本管理 — 保留所有版本 + score 排序（解 OQ2）**
|
||||
|
||||
EpisodicMemory 保留所有反思版本（key = `prompt_reflection:{task_hash}:{version}`），每条记录携带 `score`（verify 通过=1.0，失败=0.0）+ `timestamp` + `task_input`。Lead 检索时用 `EpisodicMemory.search(task_input, top_k=3)` 语义搜索，按 score 降序取最高版本。TTL 30 天（R15）由 EpisodicMemory 现有清理机制处理。ABTester 离线对比版本效果，不在线 bandit。决策依据：保留所有版本成本低（Redis/PG 存储廉价），且支持回溯；score 排序避免噪声版本干扰。
|
||||
|
||||
**KTD8: autonomy_paused 事件类型（解 OQ3）**
|
||||
|
||||
新增 `autonomy_paused` WebSocket 事件类型（区别于 `confirmation_request`），前端通过事件类型区分"等待确认"和"自主暂停"。payload 包含 `reason`（timeout | consecutive_failures | manual）+ `progress`（已完成阶段）+ `resume_token`。用户可发送 `resume` 消息继续执行（带 resume_token）。决策依据：复用现有 WebSocket 协议，前端只需新增一个事件分支，不破坏现有 confirmation 流程。
|
||||
|
||||
### High-Level Technical Design
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
User[User Input] --> Router{RequestPreprocessor}
|
||||
Router -->|@team| TeamOrchestrator
|
||||
Router -->|plan_exec| PLAN_EXEC[PLAN_EXEC Handler]
|
||||
Router -->|react| ReActEngine
|
||||
|
||||
subgraph "维度 1: 并行 Sub-agent"
|
||||
TeamOrchestrator --> Lead[Lead Expert]
|
||||
Lead --> Decompose[_decompose_task]
|
||||
Decompose --> Plan[TeamPlan]
|
||||
Plan --> TopoSort[topological_sort]
|
||||
TopoSort -->|同层并行| ParallelExec[_run_pipeline parallel]
|
||||
ParallelExec --> Agent1[ConfigDrivenAgent 1]
|
||||
ParallelExec --> Agent2[ConfigDrivenAgent 2]
|
||||
Agent1 --> SharedWorkspace[(SharedWorkspace)]
|
||||
Agent2 --> SharedWorkspace
|
||||
SharedWorkspace -.->|lock 防护| LockMech[lock/unlock]
|
||||
end
|
||||
|
||||
subgraph "维度 2: Goal-driven 自主"
|
||||
PLAN_EXEC --> PlanConfirm[Plan Confirmation]
|
||||
PlanConfirm -->|确认后| AutonomyMode[Autonomy Mode]
|
||||
AutonomyMode --> ToolExec[Tool Execution]
|
||||
ToolExec --> DangerCheck{Dangerous?}
|
||||
DangerCheck -->|是| ConfirmReq[confirmation_request]
|
||||
DangerCheck -->|否| Continue[Continue]
|
||||
ConfirmReq --> AutonomyResume[Autonomy Resume]
|
||||
AutonomyMode -.->|超时/失败| Paused[autonomy_paused]
|
||||
end
|
||||
|
||||
subgraph "维度 3: Prompt 自调优"
|
||||
ReActEngine -->|失败| ReflexionEngine
|
||||
ReflexionEngine --> Reflect[_reflect]
|
||||
Reflect --> EpisodicMem[(EpisodicMemory)]
|
||||
EpisodicMem -.->|下次任务| LeadRetrieval[Lead 检索历史反思]
|
||||
LeadRetrieval --> PromptReplace[替换默认 prompt]
|
||||
end
|
||||
```
|
||||
|
||||
### Assumptions
|
||||
|
||||
- SharedWorkspace.lock() 在 Redis 后端下可靠（已有 SETNX + EXPIRE 实现）；InProcess 后端下用 asyncio.Lock 兜底。
|
||||
- EpisodicMemory.search() 的语义搜索精度足够检索相似 task_input（pgvector 已有）。
|
||||
- 危险操作白名单覆盖 95%+ 真实危险场景；漏判由 KTD2 ponytail ceiling 标注。
|
||||
- 前端 WebSocket 已支持自定义事件类型（已验证：`team_synthesis_chunk` 等自定义事件已工作）。
|
||||
|
||||
### Sequencing
|
||||
|
||||
3 个维度可部分并行：
|
||||
- **维度 1（U4）** 与 **维度 3（U5-U7）** 完全独立，可并行开发。
|
||||
- **维度 2（U1→U2→U3）** 内部串行：配置 → 自主模式 → 超时暂停。
|
||||
- **跨维度依赖**：无。维度 1 的并行子任务和维度 2 的自主执行是正交的。
|
||||
|
||||
---
|
||||
|
||||
## Implementation Units
|
||||
|
||||
### U1. 危险操作白名单配置
|
||||
|
||||
**Goal**: 在 `agentkit.yaml` 新增 `dangerous_tools` 配置节，遵循 ServerConfig.from_dict 模式，支持工具名正则匹配。
|
||||
|
||||
**Requirements**: R8
|
||||
|
||||
**Dependencies**: None
|
||||
|
||||
**Files**:
|
||||
- `src/agentkit/server/config.py` — 新增 `DangerousToolsConfig` 类，挂载到 `ServerConfig`
|
||||
- `configs/agentkit.yaml.example` — 新增 `dangerous_tools` 配置示例
|
||||
- `tests/unit/test_server_config.py` — 新增配置解析测试
|
||||
|
||||
**Approach**:
|
||||
- 新增 `DangerousToolsConfig` 类（继承 Pydantic BaseModel），字段：`tool_patterns: list[str]`（正则列表）+ `enabled: bool`（默认 true）
|
||||
- 默认白名单：`rm`, `rmdir`, `deploy`, `kubectl`, `helm`, `git_push_force`, `alembic`, `migrate`, `payment_*`
|
||||
- `ServerConfig.from_dict` 解析 `dangerous_tools` 段，缺失时用默认白名单
|
||||
- 提供 `is_dangerous(tool_name: str) -> bool` 方法，用 `re.match` 检查
|
||||
|
||||
**Patterns to follow**: `MCPServerConfig` (`server/config.py:23`) 的 from_dict 模式；`CacheConfig` 的嵌套配置模式。
|
||||
|
||||
**Test scenarios**:
|
||||
- 配置解析：`dangerous_tools` 段存在时正确解析为 `DangerousToolsConfig`
|
||||
- 默认值：`dangerous_tools` 段缺失时使用默认白名单
|
||||
- 工具匹配：`is_dangerous("rm")` 返回 True，`is_dangerous("read_file")` 返回 False
|
||||
- 正则匹配：`is_dangerous("payment_charge")` 匹配 `payment_*` 返回 True
|
||||
- enabled=false 时：`is_dangerous` 始终返回 False（禁用白名单）
|
||||
|
||||
**Verification**: `python3 -m pytest tests/unit/test_server_config.py -x -q` 通过；`ruff check src/agentkit/server/config.py` 无 lint 错误。
|
||||
|
||||
### U2. PLAN_EXEC 自主执行模式 + 危险操作确认
|
||||
|
||||
**Goal**: PLAN_EXEC 模式下，plan 确认后进入自主执行，仅危险操作触发 confirmation_request，其他操作自主执行。
|
||||
|
||||
**Requirements**: R6, R7, R9
|
||||
|
||||
**Dependencies**: U1
|
||||
|
||||
**Files**:
|
||||
- `src/agentkit/core/react.py` — 修改 `confirmation_request` 触发逻辑（行 1255, 2199），增加危险操作检查
|
||||
- `src/agentkit/server/routes/chat.py` — PLAN_EXEC 路由处理，注入 `DangerousToolsConfig`
|
||||
- `tests/unit/test_react_autonomy.py` — 新增自主执行测试
|
||||
- `tests/unit/test_plan_exec_autonomy.py` — 新增 PLAN_EXEC 集成测试
|
||||
|
||||
**Approach**:
|
||||
- ReActEngine 构造时接收 `dangerous_tools_config: DangerousToolsConfig | None`
|
||||
- 工具执行前检查：若 `dangerous_tools_config.is_dangerous(tool_name)` 且在 PLAN_EXEC 自主模式 → 触发 `confirmation_request`（已有事件类型）
|
||||
- 非危险操作 → 直接执行，不触发 confirmation
|
||||
- PLAN_EXEC 的 plan 确认后，设置 `autonomy_mode: true` 标志（区别于现有"每阶段确认"模式）
|
||||
- `cancel` 消息已有 CancellationToken 机制（`core/protocol.py`），无需修改
|
||||
|
||||
**Patterns to follow**: 现有 `confirmation_request` 事件触发模式（`react.py:1255`）；`phase_violation` 检测模式（`react.py:272`）。
|
||||
|
||||
**Test scenarios**:
|
||||
- 危险操作触发 confirmation：自主模式下执行 `rm` 触发 `confirmation_request` 事件
|
||||
- 非危险操作自主执行：自主模式下执行 `read_file` 不触发 confirmation
|
||||
- 非自主模式行为不变：非 PLAN_EXEC 模式下，confirmation 行为与现有逻辑一致
|
||||
- cancel 中断：自主执行中发送 `cancel` 中断任务，CancellationToken 正确取消
|
||||
- 危险操作白名单禁用：`enabled=false` 时所有操作自主执行
|
||||
|
||||
**Verification**: `python3 -m pytest tests/unit/test_react_autonomy.py tests/unit/test_plan_exec_autonomy.py -x -q` 通过；现有 PLAN_EXEC 测试不回归。
|
||||
|
||||
### U3. 自主执行超时 + autonomy_paused 事件
|
||||
|
||||
**Goal**: 自主执行超时（默认 30 分钟）或连续失败 3 次时，自动暂停并发送 `autonomy_paused` 事件，用户可 `resume` 继续。
|
||||
|
||||
**Requirements**: R10
|
||||
|
||||
**Dependencies**: U2
|
||||
|
||||
**Files**:
|
||||
- `src/agentkit/core/react.py` — 新增 `autonomy_paused` 事件类型 + 超时/失败计数逻辑
|
||||
- `src/agentkit/server/routes/chat.py` — 处理 `autonomy_paused` 事件 + `resume` 消息
|
||||
- `src/agentkit/server/config.py` — 新增 `autonomy_timeout_minutes: int = 30` + `max_consecutive_failures: int = 3` 配置
|
||||
- `tests/unit/test_autonomy_paused.py` — 新增暂停/恢复测试
|
||||
|
||||
**Approach**:
|
||||
- ReActEngine 新增 `autonomy_started_at: float` + `consecutive_failures: int` 状态
|
||||
- 每次工具执行前检查：`time.time() - autonomy_started_at > timeout` 或 `consecutive_failures >= 3` → 触发 `autonomy_paused`
|
||||
- `autonomy_paused` 事件 payload：`{reason: "timeout"|"consecutive_failures"|"manual", progress: {...}, resume_token: "..."}`
|
||||
- WebSocket 路由处理 `resume` 消息：重置 `autonomy_started_at` + `consecutive_failures`，继续执行
|
||||
- 前端区分：`confirmation_request` = 等待单次确认；`autonomy_paused` = 整体自主执行暂停
|
||||
|
||||
**Patterns to follow**: 现有 `phase_changed` 事件模式（`chat.py:1694`）；`confirmation_result` 处理模式。
|
||||
|
||||
**Test scenarios**:
|
||||
- 超时触发暂停：自主执行超过 30 分钟触发 `autonomy_paused`，reason="timeout"
|
||||
- 连续失败触发暂停：连续 3 次工具失败触发 `autonomy_paused`，reason="consecutive_failures"
|
||||
- resume 恢复执行：发送 `resume` 消息后，自主执行继续，计数器重置
|
||||
- 超时配置化：`autonomy_timeout_minutes=60` 时，30 分钟不触发暂停
|
||||
- 非自主模式不触发：非 PLAN_EXEC 模式下，超时/失败不触发 `autonomy_paused`
|
||||
|
||||
**Verification**: `python3 -m pytest tests/unit/test_autonomy_paused.py -x -q` 通过；`autonomy_paused` 事件 payload 符合契约。
|
||||
|
||||
### U4. TeamOrchestrator 无依赖子任务并行模式
|
||||
|
||||
**Goal**: TeamOrchestrator 的 Lead 分解任务时，识别无 depends_on 的子任务，自动派发给隔离 ConfigDrivenAgent 并行执行（而非串行排队）。
|
||||
|
||||
**Requirements**: R1, R2, R3, R4, R5
|
||||
|
||||
**Dependencies**: None（与 U1-U3 正交）
|
||||
|
||||
**Files**:
|
||||
- `src/agentkit/experts/orchestrator.py` — 修改 `_run_pipeline`（行 234）支持并行子任务派发
|
||||
- `src/agentkit/experts/plan.py` — `TeamPlan` 新增 `get_independent_subtasks()` 方法
|
||||
- `tests/unit/test_team_parallel.py` — 新增并行子任务测试
|
||||
|
||||
**Approach**:
|
||||
- `TeamPlan.get_independent_subtasks()`: 返回 `depends_on == []` 的 PlanPhase 列表（已有 `depends_on` 字段，`plan.py:176`）
|
||||
- `_run_pipeline` 修改：
|
||||
1. 调用 `topological_sort()` 得到执行层（已有）
|
||||
2. 同层内的 phases 已并行执行（已有）— 无需修改
|
||||
3. **新增**：Lead 分解时，若识别到多个无 depends_on 的子任务，显式派发到同层（而非分散到不同层）
|
||||
- KTD6 约束：若子任务共享上游依赖，降级为同层并行（topological_sort 已处理）
|
||||
- SharedWorkspace 写入防护：关键写入用 `SharedWorkspace.lock()`（已有，`shared_workspace.py:96`）
|
||||
- MAX_EXPERTS=10 约束已有（`experts/router.py`），超出时 Lead 重新分解
|
||||
|
||||
**Patterns to follow**: 现有 `_run_pipeline` 的同层并行模式（`orchestrator.py:234`）；`_get_isolated_agent` 的 agent 创建模式（`orchestrator.py:113`）。
|
||||
|
||||
**Test scenarios**:
|
||||
- 无依赖子任务并行：3 个无 depends_on 的子任务，同时派发到 3 个隔离 agent
|
||||
- 共享依赖降级：2 个子任务共享上游依赖，降级为同层并行（不并行派发）
|
||||
- MAX_EXPERTS 约束：11 个无依赖子任务时，Lead 重新分解为 10 个以内
|
||||
- SharedWorkspace 路径不重叠：并行子任务的输出路径 `{plan_id}/phase/{phase_id}/output` 唯一
|
||||
- expert_step 事件携带 expert_id：并行执行时事件正确区分不同 expert
|
||||
- 综合阶段等待：所有并行子任务完成后才进入 synthesis
|
||||
|
||||
**Verification**: `python3 -m pytest tests/unit/test_team_parallel.py -x -q` 通过；现有 TeamOrchestrator 测试不回归。
|
||||
|
||||
### U5. ReflexionEngine 反思持久化到 EpisodicMemory
|
||||
|
||||
**Goal**: ReflexionEngine 的 `_reflect` 方法生成反思后，将结果写入 EpisodicMemory，支持跨任务检索。
|
||||
|
||||
**Requirements**: R11, R15
|
||||
|
||||
**Dependencies**: None（与 U1-U4 正交）
|
||||
|
||||
**Files**:
|
||||
- `src/agentkit/core/reflexion.py` — 修改 `_reflect`（行 648），新增持久化逻辑
|
||||
- `src/agentkit/memory/episodic.py` — 新增 `store_prompt_reflection()` + `search_prompt_reflections()` 方法
|
||||
- `tests/unit/test_reflexion_persist.py` — 新增持久化测试
|
||||
|
||||
**Approach**:
|
||||
- `_reflect` 返回前，调用 `EpisodicMemory.store_prompt_reflection(task_input, reflection, improved_prompt, version, score)`
|
||||
- 存储格式：`key = "prompt_reflection:{task_hash}:{version}"`，`value = {task_input, reflection, improved_prompt, score, timestamp}`
|
||||
- KTD7 决策：保留所有版本，score 字段记录效果（verify 通过=1.0，失败=0.0）
|
||||
- R15 TTL：复用 EpisodicMemory 现有清理机制（若无可加 `cleanup_expired()` 方法，30 天阈值）
|
||||
- `_build_reflection_prompt`（行 693）已生成 improved_prompt，直接存储
|
||||
|
||||
**Patterns to follow**: `EpisodicMemory.store()` 现有模式（`episodic.py:66`）；`EpisodicMemory.search()` 语义搜索模式（`episodic.py:206`）。
|
||||
|
||||
**Test scenarios**:
|
||||
- 反思持久化：`_reflect` 返回后，EpisodicMemory 中存在对应记录
|
||||
- 存储字段完整：记录包含 task_input + reflection + improved_prompt + version + score + timestamp
|
||||
- TTL 清理：30 天前的记录被 `cleanup_expired()` 清理
|
||||
- 多版本共存：同一 task_hash 的多个版本都保留，version 递增
|
||||
- 持久化失败不阻塞：EpisodicMemory 写入失败时，`_reflect` 仍返回反思文本（降级处理）
|
||||
|
||||
**Verification**: `python3 -m pytest tests/unit/test_reflexion_persist.py -x -q` 通过；现有 ReflexionEngine 测试不回归。
|
||||
|
||||
### U6. Lead 规划时检索历史反思 + prompt 替换
|
||||
|
||||
**Goal**: PLAN_EXEC / TEAM_COLLAB 的 Lead 规划时，检索 EpisodicMemory 中相似 task_input 的历史反思，用 improved_prompt 替换默认 prompt。
|
||||
|
||||
**Requirements**: R12, R13
|
||||
|
||||
**Dependencies**: U5
|
||||
|
||||
**Files**:
|
||||
- `src/agentkit/experts/orchestrator.py` — `_decompose_task`（行 534）前新增反思检索
|
||||
- `src/agentkit/core/reflexion.py` — 新增 `retrieve_prompt_reflection(task_input) -> dict | None`
|
||||
- `tests/unit/test_lead_reflection_retrieval.py` — 新增检索 + 替换测试
|
||||
|
||||
**Approach**:
|
||||
- `_decompose_task` 前，调用 `ReflexionEngine.retrieve_prompt_reflection(task_input)`
|
||||
- 检索逻辑：`EpisodicMemory.search_prompt_reflections(task_input, top_k=3)` → 按 score 降序取最高
|
||||
- 若找到 improved_prompt（score > 0.5），替换 Lead 的默认 system prompt
|
||||
- KTD5 触发条件：仅在 verify 二次失败 / schema 校验失败 / 循环检测时才触发反思（避免无意义检索）
|
||||
- 无历史反思时：用默认 prompt（现有行为不变）
|
||||
|
||||
**Patterns to follow**: `_decompose_task` 现有模式（`orchestrator.py:534`）；`_build_reflection_prompt` 的 prompt 拼接模式（`reflexion.py:693`）。
|
||||
|
||||
**Test scenarios**:
|
||||
- 历史反思命中：EpisodicMemory 有相似 task_input 记录，Lead 用 improved_prompt
|
||||
- 历史反思未命中：无相似记录，Lead 用默认 prompt（行为不变）
|
||||
- score 过滤：score <= 0.5 的反思不替换（避免低质量反思污染）
|
||||
- KTD5 触发条件：非触发错误类型（如普通 timeout）不检索历史反思
|
||||
- 检索失败降级：EpisodicMemory 检索失败时，Lead 用默认 prompt
|
||||
|
||||
**Verification**: `python3 -m pytest tests/unit/test_lead_reflection_retrieval.py -x -q` 通过；现有 TeamOrchestrator 测试不回归。
|
||||
|
||||
### U7. ABTester 离线对比 prompt 版本
|
||||
|
||||
**Goal**: ABTester 支持对比 EpisodicMemory 中同一 task_hash 的多个 prompt 版本效果，离线验证（不在线 bandit）。
|
||||
|
||||
**Requirements**: R14
|
||||
|
||||
**Dependencies**: U5
|
||||
|
||||
**Files**:
|
||||
- `src/agentkit/evolution/ab_tester.py` — 新增 `compare_prompt_versions(task_hash) -> dict` 方法
|
||||
- `tests/unit/test_ab_tester_prompt.py` — 新增版本对比测试
|
||||
|
||||
**Approach**:
|
||||
- `compare_prompt_versions(task_hash)`: 检索 EpisodicMemory 中该 task_hash 的所有版本
|
||||
- 输出：`{versions: [{version, score, timestamp, reflection_summary}], best_version, recommendation}`
|
||||
- 离线验证：不在线 bandit，仅基于历史 score 对比
|
||||
- `recommendation` 字段：建议保留 score 最高的版本，清理低分版本（可选）
|
||||
|
||||
**Patterns to follow**: 现有 ABTester 的对比模式（`evolution/ab_tester.py`）。
|
||||
|
||||
**Test scenarios**:
|
||||
- 多版本对比：3 个版本（score: 0.8, 0.6, 0.4），recommendation 选 0.8 版本
|
||||
- 无版本时：返回空结果，不报错
|
||||
- 单版本时：直接返回该版本为 best_version
|
||||
- 低分版本清理：可选清理 score < 0.3 的版本（保留 top-K）
|
||||
|
||||
**Verification**: `python3 -m pytest tests/unit/test_ab_tester_prompt.py -x -q` 通过。
|
||||
|
||||
---
|
||||
|
||||
## Verification Contract
|
||||
|
||||
### Unit Tests
|
||||
|
||||
```bash
|
||||
# 全量单元测试（必须通过）
|
||||
python3 -m pytest tests/unit/ -x -q
|
||||
|
||||
# 本次新增测试（逐个验证）
|
||||
python3 -m pytest tests/unit/test_server_config.py tests/unit/test_react_autonomy.py tests/unit/test_plan_exec_autonomy.py tests/unit/test_autonomy_paused.py tests/unit/test_team_parallel.py tests/unit/test_reflexion_persist.py tests/unit/test_lead_reflection_retrieval.py tests/unit/test_ab_tester_prompt.py -x -q
|
||||
```
|
||||
|
||||
### Lint + Format
|
||||
|
||||
```bash
|
||||
# Ruff lint + format（必须通过）
|
||||
ruff check src/ && ruff format src/
|
||||
```
|
||||
|
||||
### Integration Tests (Optional)
|
||||
|
||||
```bash
|
||||
# 集成测试（需 Docker Redis + PostgreSQL，可选）
|
||||
python3 -m pytest -m "integration" -x -q
|
||||
```
|
||||
|
||||
### Quality Gates
|
||||
|
||||
- 无 `any` 类型（AGENTS.md 约束）
|
||||
- 所有 Pydantic 模型用 `model_config = ConfigDict(...)`（AGENTS.md 约束）
|
||||
- API Key 比较用 `hmac.compare_digest`（若涉及）
|
||||
- 异步生成器安全（`.trae/rules/project_rules.md`）
|
||||
- 无 `return` 在第一个 `yield` 之前（若新增 async generator）
|
||||
|
||||
---
|
||||
|
||||
## Definition of Done
|
||||
|
||||
### Global Criteria
|
||||
|
||||
- 7 个 Implementation Units (U1-U7) 全部完成
|
||||
- 所有新增测试通过（8 个测试文件）
|
||||
- 现有测试不回归（`tests/unit/` 全量通过）
|
||||
- `ruff check src/ && ruff format src/` 无错误
|
||||
- 无 `any` 类型，Pydantic 模型用 `ConfigDict`
|
||||
- 异步生成器安全（无 early return before yield）
|
||||
|
||||
### Per-Unit Criteria
|
||||
|
||||
| Unit | Done Signal |
|
||||
|------|-------------|
|
||||
| U1 | `DangerousToolsConfig` 解析正确，`is_dangerous()` 工作正常 |
|
||||
| U2 | PLAN_EXEC 自主模式下危险操作触发 confirmation，非危险操作自主执行 |
|
||||
| U3 | 超时/连续失败触发 `autonomy_paused`，`resume` 恢复执行 |
|
||||
| U4 | 无依赖子任务并行派发，SharedWorkspace 路径不重叠 |
|
||||
| U5 | `_reflect` 后 EpisodicMemory 有记录，TTL 清理工作 |
|
||||
| U6 | Lead 检索历史反思并替换 prompt（score > 0.5 时） |
|
||||
| U7 | ABTester 对比多版本 prompt，返回 best_version |
|
||||
|
||||
### Cleanup Criteria
|
||||
|
||||
- 移除调试代码（print、TODO 注释、临时 hack）
|
||||
- 移除未使用的 import
|
||||
- 移除实验性死代码（尝试过但未采用的方案）
|
||||
- 所有新增配置项有文档（`agentkit.yaml.example` 示例）
|
||||
|
||||
---
|
||||
|
||||
## Appendix: 竞品对标参考
|
||||
|
||||
| 维度 | 竞品代表 | AgentKit 本次目标 |
|
||||
|------|----------|-------------------|
|
||||
| 并行 Sub-agent | Cursor 3.2 Subagents（并行专业化 worker）| R1-R5: TeamOrchestrator 无依赖子任务并行 |
|
||||
| Goal-driven 自主 | Devin 3.0 Goal-driven（数小时自主）、Qoder 1.11 Goal + Scheduling | R6-R10: PLAN_EXEC 危险操作确认式自主 |
|
||||
| Prompt 自调优 | Devin 强化学习+元学习、ReflexionEngine 跨任务 | R11-R15: ReflexionEngine 反思持久化到 EpisodicMemory |
|
||||
|
||||
## Appendix: 关键代码位置参考
|
||||
|
||||
| 子系统 | 文件 | 关键位置 |
|
||||
|--------|------|----------|
|
||||
| TeamOrchestrator | `src/agentkit/experts/orchestrator.py` | `execute()` 行 143, `_run_pipeline` 行 234, `_decompose_task` 行 534 |
|
||||
| TeamPlan | `src/agentkit/experts/plan.py` | `topological_sort()` 行 385, `SubTask` 行 69, `PlanPhase.depends_on` 行 176 |
|
||||
| ReflexionEngine | `src/agentkit/core/reflexion.py` | `_reflect()` 行 648, `_build_reflection_prompt` 行 693 |
|
||||
| EpisodicMemory | `src/agentkit/memory/episodic.py` | `store()` 行 66, `search()` 行 206 |
|
||||
| SharedWorkspace | `src/agentkit/core/shared_workspace.py` | `lock()` 行 96, `write()` 行 39 |
|
||||
| ReActEngine | `src/agentkit/core/react.py` | `confirmation_request` 行 1255/2199, `phase_violation` 行 272 |
|
||||
| PLAN_EXEC 路由 | `src/agentkit/chat/skill_routing.py` | `ExecutionMode.PLAN_EXEC` 行 33 |
|
||||
| WebSocket 事件 | `src/agentkit/server/routes/chat.py` | `confirmation_request` 行 1646, `phase_changed` 行 1701 |
|
||||
| ServerConfig | `src/agentkit/server/config.py` | `from_dict` 行 200, `MCPServerConfig` 行 23 |
|
||||
| CancellationToken | `src/agentkit/core/protocol.py` | 类定义 |
|
||||
|
|
@ -44,6 +44,11 @@ if TYPE_CHECKING:
|
|||
from agentkit.evolution.pitfall_detector import PitfallWarning
|
||||
from agentkit.memory.retriever import MemoryRetriever
|
||||
|
||||
# U2/R8: duck-typed at runtime to avoid core→server layering violation.
|
||||
# Any object with ``is_dangerous(tool_name: str) -> bool`` + ``enabled: bool``
|
||||
# is accepted; server/config.DangerousToolsConfig is the canonical impl.
|
||||
from agentkit.server.config import DangerousToolsConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
|
|
@ -142,7 +147,7 @@ class ReActResult:
|
|||
class ReActEvent:
|
||||
"""ReAct 执行事件"""
|
||||
|
||||
event_type: str # "thinking","token","tool_call","tool_result","confirmation_request","confirmation_result","phase_violation","step","final_answer","final_result","error"
|
||||
event_type: str # "thinking","token","tool_call","tool_result","confirmation_request","confirmation_result","phase_violation","step","final_answer","final_result","error","autonomy_paused","spec_review_request","spec_review_reply","expert_step","expert_result","team_synthesis","team_synthesis_chunk","phase_changed"
|
||||
step: int
|
||||
data: dict[str, object] = field(default_factory=dict)
|
||||
timestamp: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat())
|
||||
|
|
@ -204,6 +209,15 @@ class ReActEngine:
|
|||
# max_reinjections)
|
||||
# Loop detector threshold raised from 2 to 3 (R10/RV22).
|
||||
phase_budgets: dict[str, int] | None = None,
|
||||
# IQ-Boost/U2/R8: dangerous-tools whitelist for PLAN_EXEC autonomy.
|
||||
# Duck-typed (see TYPE_CHECKING import). None = no autonomy gate
|
||||
# (backward compat for DIRECT_CHAT/REACT and existing tests).
|
||||
dangerous_tools_config: "DangerousToolsConfig | None" = None,
|
||||
# IQ-Boost/U2/R6: autonomy mode flag. True = PLAN_EXEC plan confirmed,
|
||||
# only dangerous tools (per dangerous_tools_config) trigger
|
||||
# confirmation_request; non-dangerous tools execute directly.
|
||||
# False = existing behavior (tool-side _is_dangerous drives confirmation).
|
||||
autonomy_mode: bool = False,
|
||||
):
|
||||
if max_steps < 1:
|
||||
raise ValueError(f"max_steps must be >= 1, got {max_steps}")
|
||||
|
|
@ -298,6 +312,16 @@ class ReActEngine:
|
|||
# _execute_loop's finally block so subsequent execute() calls without a
|
||||
# restore still reset properly.
|
||||
self._state_restored: bool = False
|
||||
# IQ-Boost/U2: autonomy mode state (R6/R7). When True, the engine gates
|
||||
# dangerous tools (per dangerous_tools_config) with a pre-execution
|
||||
# confirmation_request; non-dangerous tools execute without confirmation.
|
||||
self._dangerous_tools_config = dangerous_tools_config
|
||||
self._autonomy_mode: bool = autonomy_mode
|
||||
# IQ-Boost/U3/R10: autonomy pause state. _autonomy_started_at is set in
|
||||
# _execute_loop when autonomy mode engages; _consecutive_failures tracks
|
||||
# tool errors. When either threshold is exceeded → autonomy_paused event.
|
||||
self._autonomy_started_at: float = 0.0
|
||||
self._consecutive_failures: int = 0
|
||||
|
||||
def reset(self) -> None:
|
||||
"""Reset internal state for reuse across conversations.
|
||||
|
|
@ -753,6 +777,10 @@ class ReActEngine:
|
|||
stream: bool = False,
|
||||
effective_timeout: float = 0.0,
|
||||
pitfall_warnings: "list[PitfallWarning] | None" = None,
|
||||
# IQ-Boost/U3: autonomy pause resume handler. When autonomy_paused is
|
||||
# triggered, the engine calls resume_handler(resume_token, reason) which
|
||||
# blocks until the user sends ``resume`` (True) or cancels (False).
|
||||
resume_handler: Callable[..., Awaitable[object]] | None = None,
|
||||
) -> AsyncGenerator[ReActEvent, None]:
|
||||
"""Unified ReAct loop — async generator yielding ReActEvent objects.
|
||||
|
||||
|
|
@ -779,6 +807,10 @@ class ReActEngine:
|
|||
# below so the next execute() without a restore resets normally.
|
||||
if not self._state_restored:
|
||||
self.reset()
|
||||
# IQ-Boost/U3: start autonomy timer when in autonomy mode.
|
||||
if self._autonomy_mode:
|
||||
self._autonomy_started_at = time.time()
|
||||
self._consecutive_failures = 0
|
||||
tools = tools or []
|
||||
if tools:
|
||||
tools = self._maybe_add_tool_search(tools)
|
||||
|
|
@ -1174,8 +1206,12 @@ class ReActEngine:
|
|||
tc.id, tool_result, compressor, tc.name
|
||||
)
|
||||
conversation.append(tool_msg)
|
||||
elif self._should_execute_parallel(response.tool_calls):
|
||||
elif self._should_execute_parallel(response.tool_calls) and not (
|
||||
self._autonomy_mode and self._dangerous_tools_config is not None
|
||||
):
|
||||
# 并行执行多个工具调用 (parallel_tools=True)
|
||||
# IQ-Boost/U2: autonomy mode forces serial execution so
|
||||
# dangerous tools are always gated by _check_autonomy_gate.
|
||||
tool_results = await asyncio.gather(
|
||||
*[
|
||||
self._execute_tool(tc.name, tc.arguments, tools)
|
||||
|
|
@ -1188,6 +1224,10 @@ class ReActEngine:
|
|||
if isinstance(tool_result, Exception):
|
||||
tool_result = {"error": str(tool_result)}
|
||||
|
||||
# IQ-Boost/U3/R10: track failures for autonomy pause
|
||||
# (no-op when autonomy mode is off).
|
||||
self._track_tool_result_for_autonomy(tool_result)
|
||||
|
||||
yield ReActEvent(
|
||||
event_type="tool_call",
|
||||
step=step,
|
||||
|
|
@ -1239,13 +1279,63 @@ class ReActEngine:
|
|||
data={"tool_name": tc.name, "arguments": tc.arguments},
|
||||
)
|
||||
|
||||
tool_start = time.monotonic()
|
||||
tool_result = await self._execute_tool(tc.name, tc.arguments, tools)
|
||||
tool_duration_ms = int((time.monotonic() - tool_start) * 1000)
|
||||
# IQ-Boost/U3/R10: autonomy pause check (timeout/failures).
|
||||
# If paused and resume_handler returns False (cancel),
|
||||
# break out of the tool_calls loop.
|
||||
# Split into detect→yield→await to avoid deadlock: the
|
||||
# pause event must be yielded BEFORE awaiting resume.
|
||||
_progress = {
|
||||
"step": step,
|
||||
"tool_name": tc.name,
|
||||
"total_steps": len(trajectory),
|
||||
}
|
||||
_pause_info = self._detect_autonomy_pause(step, _progress)
|
||||
if _pause_info is not None:
|
||||
_reason, _token, _edata = _pause_info
|
||||
yield ReActEvent(
|
||||
event_type="autonomy_paused",
|
||||
step=step,
|
||||
data=_edata,
|
||||
)
|
||||
_should_continue = await self._await_autonomy_resume(
|
||||
_token, _reason, resume_handler
|
||||
)
|
||||
if not _should_continue:
|
||||
# User cancelled during pause — stop execution.
|
||||
break
|
||||
|
||||
# Handle confirmation flow
|
||||
if isinstance(tool_result, dict) and tool_result.get(
|
||||
"needs_confirmation"
|
||||
# IQ-Boost/U2/R7: autonomy mode pre-execution gate.
|
||||
# Dangerous tools (per config) get a confirmation_request
|
||||
# BEFORE first execution; non-dangerous tools fall through.
|
||||
(
|
||||
should_exec,
|
||||
autonomy_events,
|
||||
autonomy_result,
|
||||
) = await self._check_autonomy_gate(
|
||||
tc.name,
|
||||
tc.arguments,
|
||||
tools,
|
||||
step,
|
||||
confirmation_handler,
|
||||
)
|
||||
for _aev in autonomy_events:
|
||||
yield _aev
|
||||
|
||||
if not should_exec:
|
||||
# Autonomy gate handled execution (approved+skip
|
||||
# flag, or rejected). Use its result directly.
|
||||
tool_result = autonomy_result
|
||||
tool_duration_ms = 0
|
||||
else:
|
||||
tool_start = time.monotonic()
|
||||
tool_result = await self._execute_tool(tc.name, tc.arguments, tools)
|
||||
tool_duration_ms = int((time.monotonic() - tool_start) * 1000)
|
||||
|
||||
# Handle confirmation flow (tool-side _is_dangerous)
|
||||
if (
|
||||
should_exec
|
||||
and isinstance(tool_result, dict)
|
||||
and tool_result.get("needs_confirmation")
|
||||
):
|
||||
confirmation_id = tool_result["confirmation_id"]
|
||||
command = tool_result.get("command", "")
|
||||
|
|
@ -1337,6 +1427,9 @@ class ReActEngine:
|
|||
|
||||
tool_duration_ms = int((time.monotonic() - tool_start) * 1000)
|
||||
|
||||
# IQ-Boost/U3/R10: track tool failures for autonomy pause.
|
||||
self._track_tool_result_for_autonomy(tool_result)
|
||||
|
||||
react_step = ReActStep(
|
||||
step=step,
|
||||
action="tool_call",
|
||||
|
|
@ -1407,11 +1500,31 @@ class ReActEngine:
|
|||
step=step,
|
||||
data={"tool_name": pc["name"], "arguments": pc["arguments"]},
|
||||
)
|
||||
tool_start = time.monotonic()
|
||||
tool_result = await self._execute_tool(
|
||||
pc["name"], pc["arguments"], tools
|
||||
# IQ-Boost/U2: autonomy gate for text-parsed calls too
|
||||
# (consistency — dangerous tools must be gated regardless
|
||||
# of whether the LLM returned structured or text calls).
|
||||
pc_args = pc["arguments"] if isinstance(pc["arguments"], dict) else {}
|
||||
(
|
||||
should_exec,
|
||||
autonomy_events,
|
||||
autonomy_result,
|
||||
) = await self._check_autonomy_gate(
|
||||
pc["name"], pc_args, tools, step, confirmation_handler
|
||||
)
|
||||
tool_duration_ms = int((time.monotonic() - tool_start) * 1000)
|
||||
for _aev in autonomy_events:
|
||||
yield _aev
|
||||
if not should_exec:
|
||||
tool_result = autonomy_result
|
||||
tool_duration_ms = 0
|
||||
else:
|
||||
tool_start = time.monotonic()
|
||||
tool_result = await self._execute_tool(
|
||||
pc["name"], pc["arguments"], tools
|
||||
)
|
||||
tool_duration_ms = int((time.monotonic() - tool_start) * 1000)
|
||||
|
||||
# IQ-Boost/U3/R10: track tool failures for autonomy pause.
|
||||
self._track_tool_result_for_autonomy(tool_result)
|
||||
|
||||
react_step = ReActStep(
|
||||
step=step,
|
||||
|
|
@ -1805,6 +1918,8 @@ class ReActEngine:
|
|||
timeout_seconds: float | None = None,
|
||||
confirmation_handler: Callable[..., Awaitable[object]] | None = None,
|
||||
pitfall_warnings: "list[PitfallWarning] | None" = None,
|
||||
# IQ-Boost/U3: autonomy resume handler (see _execute_loop).
|
||||
resume_handler: Callable[..., Awaitable[object]] | None = None,
|
||||
) -> AsyncGenerator[ReActEvent, None]:
|
||||
"""Execute ReAct loop, yielding ReActEvent objects.
|
||||
|
||||
|
|
@ -1817,6 +1932,7 @@ class ReActEngine:
|
|||
compressor: 压缩策略，None 时使用实例默认压缩器
|
||||
timeout_seconds: 超时秒数，0 表示无超时，None 使用 default_timeout
|
||||
pitfall_warnings: U7/R12 — HIGH 级别避坑预警，注入 system prompt
|
||||
resume_handler: U3/R10 — autonomy pause resume callback
|
||||
"""
|
||||
effective_compressor = compressor if compressor is not None else self._compressor
|
||||
effective_timeout = (
|
||||
|
|
@ -1841,6 +1957,7 @@ class ReActEngine:
|
|||
stream=True,
|
||||
effective_timeout=effective_timeout,
|
||||
pitfall_warnings=pitfall_warnings,
|
||||
resume_handler=resume_handler,
|
||||
):
|
||||
yield event
|
||||
|
||||
|
|
@ -2170,6 +2287,224 @@ class ReActEngine:
|
|||
logger.warning(error_msg)
|
||||
return {"error": error_msg, "error_code": "tool_execution_failed"}
|
||||
|
||||
async def _check_autonomy_gate(
|
||||
self,
|
||||
tool_name: str,
|
||||
tool_arguments: dict,
|
||||
tools: list[Tool],
|
||||
step: int,
|
||||
confirmation_handler: Callable[..., Awaitable[object]] | None,
|
||||
) -> tuple[bool, list[ReActEvent], object]:
|
||||
"""IQ-Boost/U2/R7: pre-execution gate for PLAN_EXEC autonomy mode.
|
||||
|
||||
When autonomy_mode is on and the tool matches the dangerous-tools
|
||||
whitelist, emit a ``confirmation_request`` BEFORE the tool runs and
|
||||
wait on ``confirmation_handler``. Approved → execute with
|
||||
``_skip_dangerous_check=True`` (bypass the tool's own _is_dangerous
|
||||
re-confirmation). Rejected → return a permission_denied result.
|
||||
|
||||
Non-dangerous tools and non-autonomy mode fall through (return
|
||||
``should_execute=True``) so the caller proceeds with the existing
|
||||
``_execute_tool`` → ``needs_confirmation`` flow unchanged.
|
||||
|
||||
Returns ``(should_execute, events, pre_result)``:
|
||||
- should_execute=True → caller runs _execute_tool; pre_result=None
|
||||
- should_execute=False → caller uses pre_result as tool_result
|
||||
"""
|
||||
events: list[ReActEvent] = []
|
||||
if (
|
||||
not self._autonomy_mode
|
||||
or self._dangerous_tools_config is None
|
||||
or not self._dangerous_tools_config.is_dangerous(tool_name)
|
||||
):
|
||||
return (True, events, None)
|
||||
|
||||
# Dangerous tool in autonomy mode — gate before first execution.
|
||||
confirmation_id = f"autonomy:{tool_name}:{step}"
|
||||
command = f"{tool_name} {tool_arguments}"
|
||||
reason = "危险操作（匹配 autonomy 白名单）需确认后执行"
|
||||
events.append(
|
||||
ReActEvent(
|
||||
event_type="confirmation_request",
|
||||
step=step,
|
||||
data={
|
||||
"confirmation_id": confirmation_id,
|
||||
"tool_name": tool_name,
|
||||
"command": command,
|
||||
"reason": reason,
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
approved = False
|
||||
if confirmation_handler is not None:
|
||||
try:
|
||||
approved = await confirmation_handler(confirmation_id, command, reason)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.warning(f"Autonomy confirmation handler error: {e}")
|
||||
|
||||
if approved:
|
||||
tool = self._find_tool(tool_name, tools)
|
||||
if tool is None:
|
||||
events.append(
|
||||
ReActEvent(
|
||||
event_type="confirmation_result",
|
||||
step=step,
|
||||
data={"confirmation_id": confirmation_id, "approved": True},
|
||||
)
|
||||
)
|
||||
return (False, events, {"error": f"Tool '{tool_name}' not found"})
|
||||
clean_args = {k: v for k, v in tool_arguments.items() if not k.startswith("_")}
|
||||
clean_args["_skip_dangerous_check"] = True
|
||||
try:
|
||||
result = await tool.safe_execute(**clean_args)
|
||||
except (ToolValidationError, ValueError, TypeError, RuntimeError) as e:
|
||||
result = {
|
||||
"error": f"Tool '{tool_name}' execution failed: {e}",
|
||||
"error_code": "tool_execution_failed",
|
||||
}
|
||||
events.append(
|
||||
ReActEvent(
|
||||
event_type="confirmation_result",
|
||||
step=step,
|
||||
data={"confirmation_id": confirmation_id, "approved": True},
|
||||
)
|
||||
)
|
||||
return (False, events, result)
|
||||
|
||||
# Rejected by user (or handler timed out / returned False)
|
||||
events.append(
|
||||
ReActEvent(
|
||||
event_type="confirmation_result",
|
||||
step=step,
|
||||
data={"confirmation_id": confirmation_id, "approved": False},
|
||||
)
|
||||
)
|
||||
return (
|
||||
False,
|
||||
events,
|
||||
{
|
||||
"output": "",
|
||||
"exit_code": 126,
|
||||
"is_error": True,
|
||||
"error_type": "permission_denied",
|
||||
"message": f"用户拒绝执行危险操作: {tool_name}",
|
||||
},
|
||||
)
|
||||
|
||||
def _detect_autonomy_pause(
|
||||
self,
|
||||
step: int,
|
||||
progress: dict[str, object],
|
||||
) -> tuple[str, str, dict[str, object]] | None:
|
||||
"""IQ-Boost/U3/R10: detect autonomy pause conditions (non-blocking).
|
||||
|
||||
Returns ``(reason, resume_token, event_data)`` when a pause is
|
||||
triggered, or ``None`` when no pause is needed.
|
||||
|
||||
The caller is responsible for:
|
||||
1. Yielding the ``autonomy_paused`` event (built from event_data)
|
||||
**before** awaiting ``_await_autonomy_resume`` — otherwise the
|
||||
frontend never receives the pause event and the resume handler
|
||||
deadlocks.
|
||||
2. Awaiting ``_await_autonomy_resume(resume_token, reason, handler)``
|
||||
to block until the user resumes or cancels.
|
||||
|
||||
This split fixes the original deadlock where the event was buffered
|
||||
inside this method and only yielded after the handler returned.
|
||||
"""
|
||||
if not self._autonomy_mode or self._dangerous_tools_config is None:
|
||||
return None
|
||||
|
||||
cfg = self._dangerous_tools_config
|
||||
reason: str | None = None
|
||||
# Check timeout (0 = disabled)
|
||||
if (
|
||||
cfg.autonomy_timeout_minutes > 0
|
||||
and self._autonomy_started_at > 0
|
||||
and time.time() - self._autonomy_started_at > cfg.autonomy_timeout_minutes * 60
|
||||
):
|
||||
reason = "timeout"
|
||||
# Check consecutive failures (0 = disabled)
|
||||
elif (
|
||||
cfg.max_consecutive_failures > 0
|
||||
and self._consecutive_failures >= cfg.max_consecutive_failures
|
||||
):
|
||||
reason = "consecutive_failures"
|
||||
|
||||
if reason is None:
|
||||
return None
|
||||
|
||||
resume_token = f"autonomy_pause:{reason}:{step}"
|
||||
event_data: dict[str, object] = {
|
||||
"reason": reason,
|
||||
"progress": progress,
|
||||
"resume_token": resume_token,
|
||||
"consecutive_failures": self._consecutive_failures,
|
||||
"elapsed_seconds": (
|
||||
time.time() - self._autonomy_started_at
|
||||
if self._autonomy_started_at > 0
|
||||
else 0
|
||||
),
|
||||
}
|
||||
return (reason, resume_token, event_data)
|
||||
|
||||
async def _await_autonomy_resume(
|
||||
self,
|
||||
resume_token: str,
|
||||
reason: str,
|
||||
resume_handler: Callable[..., Awaitable[object]] | None,
|
||||
) -> bool:
|
||||
"""Block until user resumes or cancels autonomy pause (U3/R10).
|
||||
|
||||
Returns True on resume (counters reset, caller retries the tool),
|
||||
False on cancel (caller breaks the loop).
|
||||
|
||||
If ``resume_handler`` is None (non-WS callers, tests), auto-resume
|
||||
immediately without blocking.
|
||||
"""
|
||||
if resume_handler is None:
|
||||
# Non-WS caller (tests, REST) — auto-resume without blocking.
|
||||
logger.warning("autonomy_paused (%s) with no resume_handler — auto-resuming", reason)
|
||||
self._autonomy_started_at = time.time()
|
||||
self._consecutive_failures = 0
|
||||
return True
|
||||
|
||||
try:
|
||||
should_resume = await resume_handler(resume_token, reason)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.warning(f"Autonomy resume handler error: {e}")
|
||||
should_resume = False
|
||||
|
||||
if should_resume:
|
||||
# Reset counters and continue
|
||||
self._autonomy_started_at = time.time()
|
||||
self._consecutive_failures = 0
|
||||
return True
|
||||
# Cancelled by user — break the loop
|
||||
return False
|
||||
|
||||
def _track_tool_result_for_autonomy(self, tool_result: object) -> None:
|
||||
"""U3/R10: track tool failures for consecutive_failures threshold.
|
||||
|
||||
Called after each tool execution. If the result is an error dict,
|
||||
increment the counter; otherwise reset to 0.
|
||||
"""
|
||||
if not self._autonomy_mode:
|
||||
return
|
||||
if isinstance(tool_result, dict) and (
|
||||
"error" in tool_result
|
||||
or tool_result.get("is_error") is True
|
||||
or tool_result.get("error_type") is not None
|
||||
):
|
||||
self._consecutive_failures += 1
|
||||
else:
|
||||
self._consecutive_failures = 0
|
||||
|
||||
async def _execute_tool_with_confirmation(
|
||||
self,
|
||||
tc: object,
|
||||
|
|
@ -2186,6 +2521,16 @@ class ReActEngine:
|
|||
Tuple of (tool_result, list of ReActEvents to yield)
|
||||
"""
|
||||
events: list[ReActEvent] = []
|
||||
# IQ-Boost/U2: autonomy gate runs first (pre-execution). If it handles
|
||||
# the tool (dangerous in autonomy mode), return immediately with its
|
||||
# result + events; otherwise fall through to the normal flow.
|
||||
should_exec, gate_events, gate_result = await self._check_autonomy_gate(
|
||||
tc.name, tc.arguments, tools, step, confirmation_handler
|
||||
)
|
||||
if not should_exec:
|
||||
return (gate_result, gate_events)
|
||||
events.extend(gate_events)
|
||||
|
||||
tool_result = await self._execute_tool(tc.name, tc.arguments, tools)
|
||||
|
||||
# Check if tool returned a confirmation request
|
||||
|
|
|
|||
|
|
@ -26,6 +26,8 @@ from agentkit.telemetry.metrics import (
|
|||
if TYPE_CHECKING:
|
||||
from agentkit.core.compressor import CompressionStrategy
|
||||
from agentkit.core.trace import TraceRecorder
|
||||
from agentkit.memory.base import MemoryItem
|
||||
from agentkit.memory.episodic import EpisodicMemory
|
||||
from agentkit.memory.retriever import MemoryRetriever
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
|
@ -72,6 +74,9 @@ class ReflexionEngine:
|
|||
max_reflections: int = 3,
|
||||
quality_threshold: float = 0.7,
|
||||
default_timeout: float = 300.0,
|
||||
# IQ-Boost/U5 (R11): optional EpisodicMemory for persisting reflections
|
||||
# across tasks. None = no persistence (backward-compatible).
|
||||
episodic_memory: "EpisodicMemory | None" = None,
|
||||
):
|
||||
if max_steps < 1:
|
||||
raise ValueError(f"max_steps must be >= 1, got {max_steps}")
|
||||
|
|
@ -87,6 +92,8 @@ class ReflexionEngine:
|
|||
self._max_reflections = max_reflections
|
||||
self._quality_threshold = quality_threshold
|
||||
self._default_timeout = default_timeout
|
||||
# U5: optional episodic memory for cross-task reflection persistence
|
||||
self._episodic_memory = episodic_memory
|
||||
self._react_engine = ReActEngine(
|
||||
llm_gateway=llm_gateway,
|
||||
max_steps=max_steps,
|
||||
|
|
@ -654,7 +661,12 @@ class ReflexionEngine:
|
|||
agent_name: str,
|
||||
task_type: str,
|
||||
) -> str | None:
|
||||
"""反思执行结果，返回反思文本；失败时返回 None"""
|
||||
"""反思执行结果，返回反思文本；失败时返回 None
|
||||
|
||||
IQ-Boost/U5 (R11): if ``self._episodic_memory`` is configured, persist
|
||||
the reflection for cross-task retrieval. Persistence failure is
|
||||
non-blocking — the reflection is still returned for in-task retry.
|
||||
"""
|
||||
task_description = messages[-1].get("content", "") if messages else ""
|
||||
|
||||
system_message = (
|
||||
|
|
@ -685,11 +697,33 @@ class ReflexionEngine:
|
|||
agent_name=agent_name,
|
||||
task_type=task_type or "reflection",
|
||||
)
|
||||
return response.content or None
|
||||
reflection_text = response.content or None
|
||||
except Exception as e:
|
||||
logger.warning(f"Reflection LLM call failed, skipping reflection: {e}")
|
||||
return None
|
||||
|
||||
# U5/R11: persist reflection to EpisodicMemory (non-blocking)
|
||||
if reflection_text and self._episodic_memory is not None:
|
||||
improved_prompt = self._build_reflection_prompt(
|
||||
original_prompt=None,
|
||||
reflection_text=reflection_text,
|
||||
attempt=1,
|
||||
)
|
||||
try:
|
||||
await self._episodic_memory.store_prompt_reflection(
|
||||
task_input=task_description,
|
||||
reflection=reflection_text,
|
||||
improved_prompt=improved_prompt,
|
||||
version=1,
|
||||
score=score,
|
||||
agent_name=agent_name,
|
||||
)
|
||||
except Exception as e:
|
||||
# Non-blocking: reflection is still useful for in-task retry
|
||||
logger.warning(f"U5: failed to persist reflection, continuing: {e}")
|
||||
|
||||
return reflection_text
|
||||
|
||||
def _build_reflection_prompt(
|
||||
self,
|
||||
original_prompt: str | None,
|
||||
|
|
@ -709,3 +743,69 @@ class ReflexionEngine:
|
|||
return original_prompt + reflection_section
|
||||
else:
|
||||
return reflection_section.strip()
|
||||
|
||||
async def retrieve_prompt_reflection(
|
||||
self, task_input: str, min_score: float = 0.5
|
||||
) -> dict[str, object] | None:
|
||||
"""检索历史 prompt 反思，返回最佳版本 (U6/R12, R13).
|
||||
|
||||
Searches EpisodicMemory for similar task_input reflections with
|
||||
stored quality_score > min_score. Returns the highest-quality
|
||||
reflection as:
|
||||
{improved_prompt, score, reflection, version, task_hash}
|
||||
or None if no episodic_memory / no results / all below threshold.
|
||||
|
||||
KTD5: callers should only invoke this when a trigger condition is
|
||||
met (verify failure / schema failure / loop detection) to avoid
|
||||
pointless retrieval on every task.
|
||||
|
||||
Note: ``item.score`` from ``EpisodicMemory.search`` is the hybrid
|
||||
relevance score (cosine + time_decay), NOT the stored quality_score.
|
||||
We read ``quality_score`` from ``item.value`` for filtering/ranking
|
||||
so that a high-quality reflection (score=1.0) is preferred over a
|
||||
low-quality one (score=0.0) regardless of textual similarity.
|
||||
"""
|
||||
if self._episodic_memory is None:
|
||||
return None
|
||||
|
||||
try:
|
||||
results = await self._episodic_memory.search_prompt_reflections(
|
||||
task_input=task_input, top_k=3
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"U6: retrieve_prompt_reflection failed: {e}")
|
||||
return None
|
||||
|
||||
if not results:
|
||||
return None
|
||||
|
||||
# Filter by stored quality_score (from value dict), pick the highest.
|
||||
# Fallback to item.score (relevance) when quality_score is absent.
|
||||
best: MemoryItem | None = None
|
||||
best_quality: float = 0.0
|
||||
for item in results:
|
||||
value = item.value if isinstance(item.value, dict) else {}
|
||||
quality = float(value.get("quality_score", 0.0) or 0.0)
|
||||
if quality <= 0.0:
|
||||
# Fallback to relevance score if quality_score missing
|
||||
quality = float(item.score or 0.0)
|
||||
if quality > min_score and (best is None or quality > best_quality):
|
||||
best = item
|
||||
best_quality = quality
|
||||
|
||||
if best is None:
|
||||
return None
|
||||
|
||||
# Read fields from value dict (matching EpisodicMemory.search shape)
|
||||
value = best.value if isinstance(best.value, dict) else {}
|
||||
improved_prompt = value.get("output_summary", "") or ""
|
||||
reflection_text = value.get("reflection", "") or ""
|
||||
metadata = best.metadata or {}
|
||||
|
||||
return {
|
||||
"improved_prompt": improved_prompt,
|
||||
"score": best_quality,
|
||||
"reflection": reflection_text,
|
||||
"version": metadata.get("version", 1),
|
||||
"task_hash": metadata.get("task_hash", ""),
|
||||
}
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@ from sqlalchemy.exc import DBAPIError
|
|||
|
||||
if TYPE_CHECKING:
|
||||
from agentkit.evolution.evolution_store import InMemoryEvolutionStore
|
||||
from agentkit.memory.episodic import EpisodicMemory
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
|
@ -19,6 +20,7 @@ logger = logging.getLogger(__name__)
|
|||
@dataclass
|
||||
class ABTestConfig:
|
||||
"""A/B 测试配置"""
|
||||
|
||||
test_id: str
|
||||
agent_name: str
|
||||
change_type: str # prompt / strategy / pipeline
|
||||
|
|
@ -31,6 +33,7 @@ class ABTestConfig:
|
|||
@dataclass
|
||||
class ABTestResult:
|
||||
"""A/B 测试结果"""
|
||||
|
||||
test_id: str
|
||||
control_metric: float
|
||||
experiment_metric: float
|
||||
|
|
@ -52,11 +55,15 @@ class ABTester:
|
|||
self,
|
||||
evolution_store: "InMemoryEvolutionStore | None" = None,
|
||||
min_samples: int = 10,
|
||||
episodic_memory: "EpisodicMemory | None" = None,
|
||||
):
|
||||
self._tests: dict[str, ABTestConfig] = {}
|
||||
self._results: dict[str, list[tuple[str, float]]] = {} # test_id -> [(group, metric)]
|
||||
self._evolution_store = evolution_store
|
||||
self._default_min_samples = min_samples
|
||||
# IQ-Boost/U7 (R14): optional EpisodicMemory for prompt-version comparison.
|
||||
# None = compare_prompt_versions() returns empty result (backward-compatible).
|
||||
self._episodic_memory = episodic_memory
|
||||
|
||||
def create_test(self, config: ABTestConfig) -> None:
|
||||
"""创建 A/B 测试"""
|
||||
|
|
@ -115,7 +122,9 @@ class ABTester:
|
|||
experiment_metrics = [m for g, m in results if g == "experiment"]
|
||||
|
||||
control_avg = sum(control_metrics) / len(control_metrics) if control_metrics else 0.0
|
||||
experiment_avg = sum(experiment_metrics) / len(experiment_metrics) if experiment_metrics else 0.0
|
||||
experiment_avg = (
|
||||
sum(experiment_metrics) / len(experiment_metrics) if experiment_metrics else 0.0
|
||||
)
|
||||
|
||||
try:
|
||||
await self._evolution_store.record_ab_test_result(
|
||||
|
|
@ -144,11 +153,18 @@ class ABTester:
|
|||
control_metrics = [m for g, m in results if g == "control"]
|
||||
experiment_metrics = [m for g, m in results if g == "experiment"]
|
||||
|
||||
if len(control_metrics) < config.min_samples or len(experiment_metrics) < config.min_samples:
|
||||
if (
|
||||
len(control_metrics) < config.min_samples
|
||||
or len(experiment_metrics) < config.min_samples
|
||||
):
|
||||
return ABTestResult(
|
||||
test_id=test_id,
|
||||
control_metric=sum(control_metrics) / len(control_metrics) if control_metrics else 0,
|
||||
experiment_metric=sum(experiment_metrics) / len(experiment_metrics) if experiment_metrics else 0,
|
||||
control_metric=sum(control_metrics) / len(control_metrics)
|
||||
if control_metrics
|
||||
else 0,
|
||||
experiment_metric=sum(experiment_metrics) / len(experiment_metrics)
|
||||
if experiment_metrics
|
||||
else 0,
|
||||
control_samples=len(control_metrics),
|
||||
experiment_samples=len(experiment_metrics),
|
||||
is_significant=False,
|
||||
|
|
@ -159,10 +175,16 @@ class ABTester:
|
|||
control_mean = sum(control_metrics) / len(control_metrics)
|
||||
experiment_mean = sum(experiment_metrics) / len(experiment_metrics)
|
||||
|
||||
control_var = sum((m - control_mean) ** 2 for m in control_metrics) / (len(control_metrics) - 1)
|
||||
experiment_var = sum((m - experiment_mean) ** 2 for m in experiment_metrics) / (len(experiment_metrics) - 1)
|
||||
control_var = sum((m - control_mean) ** 2 for m in control_metrics) / (
|
||||
len(control_metrics) - 1
|
||||
)
|
||||
experiment_var = sum((m - experiment_mean) ** 2 for m in experiment_metrics) / (
|
||||
len(experiment_metrics) - 1
|
||||
)
|
||||
|
||||
pooled_se = math.sqrt(control_var / len(control_metrics) + experiment_var / len(experiment_metrics))
|
||||
pooled_se = math.sqrt(
|
||||
control_var / len(control_metrics) + experiment_var / len(experiment_metrics)
|
||||
)
|
||||
|
||||
# Handle zero variance case: if means differ but variance is zero,
|
||||
# the difference is clearly significant
|
||||
|
|
@ -201,3 +223,78 @@ class ABTester:
|
|||
def _normal_cdf(x: float) -> float:
|
||||
"""标准正态分布 CDF 近似"""
|
||||
return 0.5 * (1 + math.erf(x / math.sqrt(2)))
|
||||
|
||||
# ── IQ-Boost/U7: Prompt-version offline comparison (R14) ────────────
|
||||
|
||||
@staticmethod
|
||||
def _no_data_result(task_hash: str) -> dict[str, object]:
|
||||
"""Build the empty-result payload for compare_prompt_versions."""
|
||||
return {
|
||||
"task_hash": task_hash,
|
||||
"versions": [],
|
||||
"best_version": None,
|
||||
"recommendation": "no_data",
|
||||
"total_versions": 0,
|
||||
}
|
||||
|
||||
async def compare_prompt_versions(self, task_hash: str) -> dict[str, object]:
|
||||
"""离线对比同一 task_hash 的多个 prompt 版本效果 (U7/R14).
|
||||
|
||||
从 EpisodicMemory 检索该 task_hash 的所有 prompt_reflection 记录，
|
||||
按 score 降序排列，返回对比结果 + 推荐保留版本。
|
||||
|
||||
离线验证 — 不在线 bandit，仅基于历史 score 对比。
|
||||
|
||||
Returns:
|
||||
{
|
||||
"task_hash": str,
|
||||
"versions": [{"version": int, "score": float, "timestamp": str,
|
||||
"reflection_summary": str, "improved_prompt": str}],
|
||||
"best_version": dict | None, # score 最高的版本
|
||||
"recommendation": str, # "keep_best" | "no_data"
|
||||
"total_versions": int,
|
||||
}
|
||||
无 episodic_memory 或无记录时返回 "no_data" recommendation。
|
||||
"""
|
||||
if self._episodic_memory is None:
|
||||
return self._no_data_result(task_hash)
|
||||
|
||||
try:
|
||||
items = await self._episodic_memory.list_prompt_reflections_by_hash(task_hash)
|
||||
except (DBAPIError, RuntimeError, ValueError, KeyError, OSError) as e:
|
||||
logger.warning(f"U7: compare_prompt_versions retrieval failed: {e}")
|
||||
return self._no_data_result(task_hash)
|
||||
|
||||
if not items:
|
||||
return self._no_data_result(task_hash)
|
||||
|
||||
# Sort by score descending (best first)
|
||||
sorted_items = sorted(items, key=lambda it: it.score, reverse=True)
|
||||
|
||||
versions: list[dict[str, object]] = []
|
||||
for item in sorted_items:
|
||||
meta = item.metadata or {}
|
||||
value = item.value if isinstance(item.value, dict) else {}
|
||||
reflection_text = value.get("reflection", "")
|
||||
improved_prompt = value.get("output_summary", "")
|
||||
versions.append(
|
||||
{
|
||||
"version": meta.get("version", 1),
|
||||
"score": item.score or 0.0,
|
||||
"timestamp": meta.get("created_at", "")
|
||||
or (item.created_at.isoformat() if item.created_at else ""),
|
||||
"reflection_summary": (reflection_text[:200] if reflection_text else ""),
|
||||
"improved_prompt": (improved_prompt[:500] if improved_prompt else ""),
|
||||
}
|
||||
)
|
||||
|
||||
best = versions[0] if versions else None
|
||||
recommendation = "keep_best" if best else "no_data"
|
||||
|
||||
return {
|
||||
"task_hash": task_hash,
|
||||
"versions": versions,
|
||||
"best_version": best,
|
||||
"recommendation": recommendation,
|
||||
"total_versions": len(versions),
|
||||
}
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ import asyncio
|
|||
import json
|
||||
import logging
|
||||
import re
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from agentkit.core.exceptions import LLMProviderError
|
||||
from agentkit.llm.gateway import LLMGateway
|
||||
|
|
@ -36,6 +37,9 @@ from .plan import (
|
|||
)
|
||||
from .team import ExpertTeam, TeamStatus
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from agentkit.core.reflexion import ReflexionEngine
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# 专家名校验正则（与 router.py / board_router.py 保持一致）
|
||||
|
|
@ -62,6 +66,9 @@ class TeamOrchestrator(
|
|||
MAX_DEBATE_ROUNDS = 4 # Hard cap on debate rounds per phase
|
||||
MAX_DEBATES = 3 # Hard cap on auto-inserted debate phases per execution
|
||||
DEFAULT_MAX_CONCURRENT_PHASES = 3 # 同层最大并发阶段数，避免 LLM 限流洪峰
|
||||
# IQ-Boost/U4 (R4): aligns with router.MAX_EXPERTS — if Lead decomposes
|
||||
# more independent subtasks than this, re-decompose once with a merge hint.
|
||||
MAX_INDEPENDENT_SUBTASKS = 10
|
||||
STOP_COMMANDS = frozenset({"/stop", "停止", "stop", "结束"})
|
||||
# G9/U4: RollbackExecutor default timeout for validation_command / rollback_command.
|
||||
# Override via constructor `rollback_timeout` from `rollback.default_timeout` config.
|
||||
|
|
@ -79,6 +86,10 @@ class TeamOrchestrator(
|
|||
# final-answer path (react.py:1303+) runs on coding tasks.
|
||||
verification_enabled: bool = True,
|
||||
verification_commands: list[str] | None = None,
|
||||
# IQ-Boost/U6 (R12, R13): optional ReflexionEngine for retrieving
|
||||
# historical prompt reflections at Lead planning time. None = no
|
||||
# retrieval (backward-compatible).
|
||||
reflexion_engine: "ReflexionEngine | None" = None,
|
||||
) -> None:
|
||||
self._team = team
|
||||
# Track temporary agent names created for context isolation (KTD3)
|
||||
|
|
@ -100,6 +111,8 @@ class TeamOrchestrator(
|
|||
self._rollback_timeout = rollback_timeout or self.DEFAULT_ROLLBACK_TIMEOUT
|
||||
# U3/R2: verification defaults for TEAM_COLLAB.
|
||||
self._verification_enabled = verification_enabled
|
||||
# U6: optional reflexion engine for historical reflection retrieval
|
||||
self._reflexion_engine = reflexion_engine
|
||||
# U3/R3: if no explicit commands, detect from workspace (coding-task
|
||||
# detection forces pytest/ruff). None workspace → None commands →
|
||||
# ReActEngine/VerificationLoop uses its own defaults.
|
||||
|
|
@ -197,6 +210,11 @@ class TeamOrchestrator(
|
|||
PlanPhase(name="执行", assigned_expert=lead.config.name, task_description=task)
|
||||
]
|
||||
|
||||
# IQ-Boost/U4 (R4): if Lead over-decomposed independent subtasks beyond
|
||||
# MAX_INDEPENDENT_SUBTASKS, ask Lead to re-decompose with a merge hint
|
||||
# (one retry — further overflow falls through to MAX_PHASES truncation).
|
||||
phases = await self._rebalance_independent_subtasks(lead, task, phases)
|
||||
|
||||
plan.phases = phases[: self.MAX_PHASES]
|
||||
|
||||
# U3: Optionally add plan review debate before execution
|
||||
|
|
@ -531,11 +549,79 @@ class TeamOrchestrator(
|
|||
self._team.set_status(TeamStatus.EXECUTING)
|
||||
return await self._run_pipeline(lead, plan, phase_results, task)
|
||||
|
||||
async def _rebalance_independent_subtasks(
|
||||
self, lead: Expert, task: str, phases: list[PlanPhase]
|
||||
) -> list[PlanPhase]:
|
||||
"""IQ-Boost/U4 (R4): if phases contain more independent subtasks
|
||||
(depends_on == []) than MAX_INDEPENDENT_SUBTASKS, ask Lead to
|
||||
re-decompose with a merge hint. One retry only — further overflow
|
||||
is handled by MAX_PHASES truncation in execute().
|
||||
|
||||
Returns the original phases if count is within limit or retry fails.
|
||||
"""
|
||||
# Count phases with no depends_on (independent subtasks).
|
||||
independent_count = sum(1 for ph in phases if not ph.depends_on)
|
||||
if independent_count <= self.MAX_INDEPENDENT_SUBTASKS:
|
||||
return phases
|
||||
|
||||
logger.info(
|
||||
f"U4: Lead decomposed {independent_count} independent subtasks "
|
||||
f"(> {self.MAX_INDEPENDENT_SUBTASKS}), requesting re-decompose with merge hint"
|
||||
)
|
||||
|
||||
gateway = self._get_llm_gateway(lead)
|
||||
if not gateway:
|
||||
# No gateway — can't re-decompose; rely on MAX_PHASES truncation.
|
||||
return phases
|
||||
|
||||
# Re-decompose with explicit merge hint
|
||||
original_hint = (
|
||||
f"Previous decomposition produced {independent_count} independent subtasks "
|
||||
f"(no dependencies), exceeding the {self.MAX_INDEPENDENT_SUBTASKS} limit. "
|
||||
f"Please merge related subtasks so the total independent (depends_on=[]) "
|
||||
f"subtasks is at most {self.MAX_INDEPENDENT_SUBTASKS}. Keep dependencies "
|
||||
f"where natural."
|
||||
)
|
||||
# Temporarily wrap task with hint — call _decompose_task with augmented task.
|
||||
augmented_task = f"{task}\n\n[Re-decomposition hint]: {original_hint}"
|
||||
try:
|
||||
new_phases = await self._decompose_task(lead, augmented_task)
|
||||
except (LLMProviderError, asyncio.TimeoutError, ConnectionError, ValueError) as e:
|
||||
logger.warning(f"U4 re-decompose failed: {e}, keeping original phases")
|
||||
return phases
|
||||
|
||||
if not new_phases:
|
||||
return phases
|
||||
|
||||
new_independent = sum(1 for ph in new_phases if not ph.depends_on)
|
||||
if new_independent <= self.MAX_INDEPENDENT_SUBTASKS:
|
||||
# ponytail: detect single-phase fallback (LLM returned invalid
|
||||
# JSON → _decompose_task returned degenerate single phase).
|
||||
# Collapsing 11 subtasks to 1 is worse than truncation; keep
|
||||
# original so MAX_PHASES handles it.
|
||||
if len(new_phases) == 1 and new_independent < independent_count:
|
||||
logger.info("U4: re-decompose fell back to single phase, keeping original")
|
||||
return phases
|
||||
logger.info(f"U4: re-decompose succeeded ({new_independent} independent subtasks)")
|
||||
return new_phases
|
||||
|
||||
# Still over limit — take the new decomposition anyway (truncation will cap it)
|
||||
logger.info(
|
||||
f"U4: re-decompose still has {new_independent} independent subtasks, "
|
||||
f"proceeding with MAX_PHASES truncation"
|
||||
)
|
||||
return new_phases
|
||||
|
||||
async def _decompose_task(self, lead: Expert, task: str) -> list[PlanPhase]:
|
||||
"""Lead Expert decomposes task into phases using LLM.
|
||||
|
||||
Returns a list of PlanPhase instances. If LLM decomposition fails,
|
||||
returns a single phase with the original task.
|
||||
|
||||
IQ-Boost/U6 (R12, R13): if reflexion_engine is configured, retrieves
|
||||
historical prompt reflection for similar task_input and prepends
|
||||
improved_prompt to the decomposition prompt. Non-blocking — retrieval
|
||||
failure falls through to default prompt.
|
||||
"""
|
||||
gateway = self._get_llm_gateway(lead)
|
||||
if not gateway:
|
||||
|
|
@ -547,6 +633,26 @@ class TeamOrchestrator(
|
|||
]
|
||||
available_experts = member_names if member_names else [lead.config.name]
|
||||
|
||||
# U6: retrieve historical reflection (non-blocking)
|
||||
reflection_hint = ""
|
||||
if self._reflexion_engine is not None:
|
||||
try:
|
||||
historical = await self._reflexion_engine.retrieve_prompt_reflection(
|
||||
task_input=task
|
||||
)
|
||||
if historical and historical.get("improved_prompt"):
|
||||
reflection_hint = (
|
||||
f"\n\n## Historical Reflection (score={historical.get('score', 0):.2f})\n"
|
||||
f"A previous similar task produced this reflection. "
|
||||
f"Use it to improve your decomposition:\n\n"
|
||||
f"{historical['improved_prompt']}\n"
|
||||
)
|
||||
logger.info(
|
||||
f"U6: retrieved historical reflection (score={historical.get('score', 0):.2f})"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"U6: historical reflection retrieval failed, using default: {e}")
|
||||
|
||||
prompt = (
|
||||
f"You are the Lead Expert in a pipeline team. Decompose the following task into "
|
||||
f"at most {self.MAX_PHASES} phases with dependencies.\n\n"
|
||||
|
|
@ -574,6 +680,7 @@ class TeamOrchestrator(
|
|||
f'{{"name":"前端","assigned_expert":"frontend",'
|
||||
f'"task_description":"实现UI","depends_on":["后端"],"collaboration_contracts":[]}}]\n\n'
|
||||
f"Return ONLY the JSON array, no other text."
|
||||
f"{reflection_hint}"
|
||||
)
|
||||
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -443,3 +443,12 @@ class TeamPlan:
|
|||
in_degree[dep_id] -= 1
|
||||
|
||||
return layers
|
||||
|
||||
def get_independent_subtasks(self) -> list[PlanPhase]:
|
||||
"""返回无依赖的子任务（depends_on == []）。
|
||||
|
||||
IQ-Boost/U4 (R1): 用于 Lead 分解后检查独立子任务数量是否超过
|
||||
MAX_EXPERTS。这些子任务会被 topological_sort 派发到 layer 0 并行执行
|
||||
（同层并行已有，本方法仅提供显式 introspection 入口）。
|
||||
"""
|
||||
return [ph for ph in self.phases if not ph.depends_on]
|
||||
|
|
|
|||
|
|
@ -410,3 +410,192 @@ class EpisodicMemory(Memory):
|
|||
await db.rollback()
|
||||
logger.error(f"Failed to delete episodic memory: {e}")
|
||||
return False
|
||||
|
||||
# ── IQ-Boost/U5: Prompt Reflection persistence (R11, R15) ──────────
|
||||
|
||||
async def store_prompt_reflection(
|
||||
self,
|
||||
task_input: str,
|
||||
reflection: str,
|
||||
improved_prompt: str,
|
||||
version: int = 1,
|
||||
score: float = 0.0,
|
||||
agent_name: str = "",
|
||||
task_hash: str | None = None,
|
||||
) -> str | None:
|
||||
"""持久化 prompt 反思到 EpisodicMemory，支持跨任务检索 (U5/R11).
|
||||
|
||||
Writes directly to the ORM row (bypassing ``store()``) so that
|
||||
``task_hash``/``version``/``score`` land in the ``metadata_`` JSONB
|
||||
column — ``store()`` only maps a fixed set of metadata keys to ORM
|
||||
columns and drops the rest, which would break
|
||||
``list_prompt_reflections_by_hash`` filtering.
|
||||
|
||||
The ORM row's fields map:
|
||||
input_summary ← task_input (truncated)
|
||||
output_summary ← improved_prompt (truncated)
|
||||
reflection ← reflection text
|
||||
quality_score ← score (0.0=failed, 1.0=verified)
|
||||
outcome ← "reflection"
|
||||
agent_name ← agent_name
|
||||
task_type ← "prompt_reflection"
|
||||
metadata_ ← {task_hash, version, score, timestamp}
|
||||
|
||||
Returns the storage key ``"prompt_reflection:{task_hash}:{version}"``
|
||||
on success, or None on failure (non-raising — callers continue without).
|
||||
"""
|
||||
import hashlib
|
||||
|
||||
if task_hash is None:
|
||||
task_hash = hashlib.sha256(task_input.encode("utf-8")).hexdigest()[:16]
|
||||
key = f"prompt_reflection:{task_hash}:{version}"
|
||||
|
||||
# Generate embedding from task_input for semantic search compatibility.
|
||||
embedding = None
|
||||
if self._embedder:
|
||||
try:
|
||||
embedding = await self._embedder.embed(task_input)
|
||||
except (RuntimeError, ValueError, OSError) as e:
|
||||
logger.warning(f"U5: embedder failed for prompt reflection: {e}")
|
||||
|
||||
extra_meta = {
|
||||
"task_hash": task_hash,
|
||||
"version": version,
|
||||
"score": score,
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
}
|
||||
|
||||
async with self._session_factory() as db:
|
||||
try:
|
||||
Model = self._episodic_model
|
||||
entry = Model(
|
||||
agent_name=agent_name,
|
||||
task_type="prompt_reflection",
|
||||
input_summary=task_input[:500],
|
||||
output_summary=improved_prompt[:500],
|
||||
outcome="reflection",
|
||||
quality_score=score,
|
||||
reflection=reflection,
|
||||
embedding=embedding,
|
||||
metadata_=extra_meta,
|
||||
)
|
||||
db.add(entry)
|
||||
await db.commit()
|
||||
return key
|
||||
except (DBAPIError, ValueError, KeyError, RuntimeError, OSError) as e:
|
||||
await db.rollback()
|
||||
logger.warning(f"U5: failed to persist prompt reflection: {e}")
|
||||
return None
|
||||
|
||||
async def search_prompt_reflections(
|
||||
self,
|
||||
task_input: str,
|
||||
top_k: int = 5,
|
||||
agent_name: str | None = None,
|
||||
) -> list[MemoryItem]:
|
||||
"""检索相似 task_input 的历史 prompt 反思 (U5/R11).
|
||||
|
||||
Uses ``search()`` with ``task_type="prompt_reflection"`` filter.
|
||||
Returns empty list on failure (non-raising).
|
||||
"""
|
||||
filters: MetadataDict = {"task_type": "prompt_reflection"}
|
||||
if agent_name:
|
||||
filters["agent_name"] = agent_name
|
||||
try:
|
||||
return await self.search(query=task_input, top_k=top_k, filters=filters)
|
||||
except (DBAPIError, ValueError, KeyError, RuntimeError, OSError) as e:
|
||||
logger.warning(f"U5: failed to search prompt reflections: {e}")
|
||||
return []
|
||||
|
||||
async def list_prompt_reflections_by_hash(
|
||||
self,
|
||||
task_hash: str,
|
||||
agent_name: str | None = None,
|
||||
) -> list[MemoryItem]:
|
||||
"""精确查询同一 task_hash 的所有 prompt 反思版本 (U7/R14).
|
||||
|
||||
用于 ABTester 离线对比同一任务的不同 prompt 版本效果。
|
||||
|
||||
ponytail: ceiling = O(N) 全表扫描 task_type='prompt_reflection' 记录后
|
||||
在 Python 侧过滤 task_hash。N 通常 <100（一个 task_hash 的版本数有限）。
|
||||
升级路径 = 在 metadata_['task_hash'] 上加 GIN 索引 + JSONB ->> 算符查询。
|
||||
|
||||
Returns empty list on failure (non-raising).
|
||||
"""
|
||||
from sqlalchemy import select
|
||||
|
||||
async with self._session_factory() as db:
|
||||
try:
|
||||
Model = self._episodic_model
|
||||
stmt = (
|
||||
select(Model)
|
||||
.where(Model.task_type == "prompt_reflection")
|
||||
.order_by(Model.created_at.desc())
|
||||
.limit(200)
|
||||
)
|
||||
result = await db.execute(stmt)
|
||||
entries = result.scalars().all()
|
||||
except (DBAPIError, ValueError, KeyError, RuntimeError, OSError) as e:
|
||||
logger.warning(f"U7: list_prompt_reflections_by_hash failed: {e}")
|
||||
return []
|
||||
|
||||
items: list[MemoryItem] = []
|
||||
for entry in entries:
|
||||
meta = entry.metadata_ or {}
|
||||
if meta.get("task_hash") != task_hash:
|
||||
continue
|
||||
if agent_name and meta.get("agent_name") != agent_name:
|
||||
continue
|
||||
items.append(
|
||||
MemoryItem(
|
||||
key=str(entry.id),
|
||||
value={
|
||||
"input_summary": entry.input_summary,
|
||||
"output_summary": entry.output_summary,
|
||||
"reflection": entry.reflection,
|
||||
"quality_score": entry.quality_score,
|
||||
},
|
||||
metadata={
|
||||
"agent_name": entry.agent_name,
|
||||
"task_type": entry.task_type,
|
||||
"task_hash": meta.get("task_hash", ""),
|
||||
"version": meta.get("version", 1),
|
||||
"score": entry.quality_score or 0.0,
|
||||
"created_at": entry.created_at.isoformat() if entry.created_at else None,
|
||||
},
|
||||
score=entry.quality_score or 0.0,
|
||||
created_at=entry.created_at or datetime.now(timezone.utc),
|
||||
)
|
||||
)
|
||||
return items
|
||||
|
||||
async def cleanup_expired(
|
||||
self, max_age_days: int = 30, task_type: str | None = None
|
||||
) -> int:
|
||||
"""删除超过 max_age_days 天的记录 (U5/R15 TTL).
|
||||
|
||||
Args:
|
||||
max_age_days: Records older than this many days are deleted.
|
||||
task_type: Optional filter — only delete records with this task_type
|
||||
(e.g. ``"prompt_reflection"``). None = delete all task types.
|
||||
|
||||
Returns the number of deleted rows. 0 on failure (non-raising).
|
||||
"""
|
||||
from datetime import timedelta
|
||||
|
||||
from sqlalchemy import delete as sql_delete
|
||||
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(days=max_age_days)
|
||||
async with self._session_factory() as db:
|
||||
try:
|
||||
Model = self._episodic_model
|
||||
stmt = sql_delete(Model).where(Model.created_at < cutoff)
|
||||
if task_type is not None:
|
||||
stmt = stmt.where(Model.task_type == task_type)
|
||||
result = await db.execute(stmt)
|
||||
await db.commit()
|
||||
return result.rowcount or 0
|
||||
except (DBAPIError, ValueError, KeyError, RuntimeError) as e:
|
||||
await db.rollback()
|
||||
logger.warning(f"U5: cleanup_expired failed: {e}")
|
||||
return 0
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ import asyncio
|
|||
import logging
|
||||
import os
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable
|
||||
|
||||
|
|
@ -57,6 +57,65 @@ class MCPServerConfig:
|
|||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class DangerousToolsConfig:
|
||||
"""Configuration for dangerous tool detection in PLAN_EXEC autonomy mode.
|
||||
|
||||
When enabled, tools matching any pattern in ``tool_patterns`` trigger a
|
||||
confirmation request before execution. KTD2: conservative whitelist —
|
||||
ponytail ceiling = missed dangerous ops not in the list; upgrade path =
|
||||
LLM-assisted classification.
|
||||
|
||||
IQ-Boost/U3: also carries autonomy pause thresholds (timeout + consecutive
|
||||
failures). Kept here to avoid a separate config section — all autonomy
|
||||
behavior in one place.
|
||||
"""
|
||||
|
||||
enabled: bool = True
|
||||
tool_patterns: list[str] = field(
|
||||
default_factory=lambda: [
|
||||
r"^rm$",
|
||||
r"^rmdir$",
|
||||
r"^deploy$",
|
||||
r"^kubectl",
|
||||
r"^helm",
|
||||
r"^git_push_force$",
|
||||
r"^alembic",
|
||||
r"^migrate",
|
||||
r"^payment_",
|
||||
]
|
||||
)
|
||||
# U3/R10: autonomy pause thresholds. 0 = disabled (no pause).
|
||||
autonomy_timeout_minutes: int = 30
|
||||
max_consecutive_failures: int = 3
|
||||
# Pre-compiled patterns — populated by __post_init__, reused by is_dangerous
|
||||
# on the per-tool hot path. ponytail: ceiling = rebuild when tool_patterns
|
||||
# mutated after construction (rare; config-time only). Upgrade path =
|
||||
# cached_property with invalidation on tool_patterns setter.
|
||||
_compiled_patterns: list[re.Pattern] = field(default_factory=list, repr=False)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
self._compiled_patterns = [re.compile(p) for p in self.tool_patterns]
|
||||
|
||||
def is_dangerous(self, tool_name: str) -> bool:
|
||||
"""Check if a tool name matches any dangerous pattern."""
|
||||
if not self.enabled:
|
||||
return False
|
||||
return any(p.match(tool_name) for p in self._compiled_patterns)
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict | None) -> "DangerousToolsConfig":
|
||||
"""Create from dict (parsed from YAML). Empty/None returns defaults."""
|
||||
if not data:
|
||||
return cls()
|
||||
return cls(
|
||||
enabled=data.get("enabled", True),
|
||||
tool_patterns=data.get("tool_patterns") or cls().tool_patterns,
|
||||
autonomy_timeout_minutes=data.get("autonomy_timeout_minutes", 30),
|
||||
max_consecutive_failures=data.get("max_consecutive_failures", 3),
|
||||
)
|
||||
|
||||
|
||||
def _resolve_env_vars(value: Any) -> Any:
|
||||
"""Resolve ${VAR:-default} patterns in string values from environment variables."""
|
||||
if not isinstance(value, str):
|
||||
|
|
@ -124,6 +183,9 @@ class ServerConfig:
|
|||
# G6/U2: PLAN_EXEC phase policy config (opt-in — None = disabled).
|
||||
# Parsed via PhasePolicy.policy_from_config() at chat.py wiring time.
|
||||
plan_exec: dict[str, Any] | None = None,
|
||||
# IQ-Boost/U1: dangerous_tools config for PLAN_EXEC autonomy mode (KTD2).
|
||||
# Parsed via DangerousToolsConfig.from_dict(); defaults applied when None.
|
||||
dangerous_tools: dict[str, Any] | None = None,
|
||||
on_change: Callable[["ServerConfig"], None] | None = None,
|
||||
):
|
||||
self.host = host
|
||||
|
|
@ -168,6 +230,8 @@ class ServerConfig:
|
|||
# Resolved to PhasePolicy via agentkit.core.phase.policy_from_config()
|
||||
# at chat.py WebSocket wiring time (U4).
|
||||
self.plan_exec = plan_exec or {}
|
||||
# IQ-Boost/U1: dangerous_tools parsed config (KTD2 conservative whitelist).
|
||||
self.dangerous_tools = DangerousToolsConfig.from_dict(dangerous_tools)
|
||||
self.on_change = on_change
|
||||
|
||||
# Config watching state
|
||||
|
|
@ -261,6 +325,8 @@ class ServerConfig:
|
|||
fallback_chain_data = data.get("fallback_chain", {})
|
||||
# G6/U2: plan_exec phase policy 配置 (从 YAML 读取, opt-in)
|
||||
plan_exec_data = data.get("plan_exec", {})
|
||||
# IQ-Boost/U1: dangerous_tools 配置 (从 YAML 读取, KTD2 whitelist)
|
||||
dangerous_tools_data = data.get("dangerous_tools", {})
|
||||
|
||||
return cls(
|
||||
host=server.get("host", "0.0.0.0"),
|
||||
|
|
@ -295,6 +361,7 @@ class ServerConfig:
|
|||
rollback=rollback_data,
|
||||
fallback_chain=fallback_chain_data,
|
||||
plan_exec=plan_exec_data,
|
||||
dangerous_tools=dangerous_tools_data,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
|
|
|
|||
|
|
@ -695,9 +695,15 @@ def _build_phase_engine(
|
|||
)
|
||||
return (None, None, f"phase policy error: {str(e)[:200]}")
|
||||
|
||||
# IQ-Boost/U2/R6-R8: PLAN_EXEC engages autonomy mode — plan confirmation
|
||||
# is handled by the existing spec_review gate; after confirmation, only
|
||||
# dangerous tools (per dangerous_tools_config) trigger confirmation_request.
|
||||
dangerous_tools_cfg = getattr(server_config, "dangerous_tools", None)
|
||||
engine = ReActEngine(
|
||||
llm_gateway=llm_gateway,
|
||||
phase_policy=phase_policy,
|
||||
dangerous_tools_config=dangerous_tools_cfg,
|
||||
autonomy_mode=True,
|
||||
)
|
||||
advance_phase_tool = AdvancePhaseTool(engine=engine)
|
||||
tools_with_advance_phase = list(base_tools) + [advance_phase_tool]
|
||||
|
|
@ -1013,6 +1019,9 @@ async def chat_websocket(websocket: WebSocket, session_id: str) -> None:
|
|||
# U8/R8: pending spec-review futures keyed by spec_review_id. Resolved
|
||||
# by the spec_review_reply client message; cancelled on WS teardown.
|
||||
pending_spec_reviews: dict[str, asyncio.Future] = {}
|
||||
# IQ-Boost/U3: pending autonomy-resume futures keyed by resume_token.
|
||||
# Resolved by the ``resume`` client message; cancelled on WS teardown.
|
||||
pending_autonomy_resumes: dict[str, asyncio.Future] = {}
|
||||
chat_manager.add(session_id, websocket, pending_replies)
|
||||
|
||||
cancellation_token = CancellationToken()
|
||||
|
|
@ -1095,6 +1104,7 @@ async def chat_websocket(websocket: WebSocket, session_id: str) -> None:
|
|||
pending_replies,
|
||||
pending_confirmations,
|
||||
pending_spec_reviews,
|
||||
pending_autonomy_resumes,
|
||||
model_override=model,
|
||||
)
|
||||
)
|
||||
|
|
@ -1150,6 +1160,22 @@ async def chat_websocket(websocket: WebSocket, session_id: str) -> None:
|
|||
cancellation_token.cancel()
|
||||
await websocket.send_json({"type": "result", "data": {"status": "cancelled"}})
|
||||
|
||||
elif msg_type == "resume":
|
||||
# IQ-Boost/U3/R10: Resume autonomy-paused execution. The client
|
||||
# sends {resume_token}. An unknown token is logged + ignored.
|
||||
resume_token = msg.get("resume_token")
|
||||
logger.info(f"Received resume for token: {resume_token!r}")
|
||||
if resume_token and resume_token in pending_autonomy_resumes:
|
||||
fut = pending_autonomy_resumes[resume_token]
|
||||
if not fut.done():
|
||||
fut.set_result(True)
|
||||
else:
|
||||
logger.warning(f"resume token {resume_token!r} already resolved")
|
||||
else:
|
||||
logger.warning(
|
||||
f"resume token {resume_token!r} not found in pending_autonomy_resumes"
|
||||
)
|
||||
|
||||
elif msg_type == "ping":
|
||||
await websocket.send_json({"type": "pong"})
|
||||
|
||||
|
|
@ -1174,6 +1200,10 @@ async def chat_websocket(websocket: WebSocket, session_id: str) -> None:
|
|||
for fut in pending_spec_reviews.values():
|
||||
if not fut.done():
|
||||
fut.cancel()
|
||||
# IQ-Boost/U3: cancel any pending autonomy-resume futures.
|
||||
for fut in pending_autonomy_resumes.values():
|
||||
if not fut.done():
|
||||
fut.cancel()
|
||||
chat_manager.remove(session_id, websocket)
|
||||
|
||||
|
||||
|
|
@ -1186,6 +1216,8 @@ async def _handle_chat_message(
|
|||
pending_replies: dict[str, asyncio.Future],
|
||||
pending_confirmations: dict[str, asyncio.Future] | None = None,
|
||||
pending_spec_reviews: dict[str, asyncio.Future] | None = None,
|
||||
# IQ-Boost/U3: pending autonomy-resume futures (keyed by resume_token).
|
||||
pending_autonomy_resumes: dict[str, asyncio.Future] | None = None,
|
||||
model_override: str | None = None,
|
||||
) -> None:
|
||||
"""Handle a user message: append to session, execute Agent, stream events.
|
||||
|
|
@ -1577,6 +1609,35 @@ async def _handle_chat_message(
|
|||
if hasattr(react_engine, "_spec_review_handler"):
|
||||
react_engine._spec_review_handler = _spec_review_handler
|
||||
|
||||
# IQ-Boost/U3/R10: autonomy resume handler. When the engine triggers
|
||||
# autonomy_paused (timeout/consecutive_failures), it calls this handler
|
||||
# which blocks until the user sends ``resume`` (True) or the WS disconnects
|
||||
# (Future cancelled → False). The engine already yielded the
|
||||
# ``autonomy_paused`` event (forwarded to the frontend by the event loop
|
||||
# below); this handler just provides the blocking mechanism.
|
||||
_pending_autonomy_resumes = (
|
||||
pending_autonomy_resumes if pending_autonomy_resumes is not None else {}
|
||||
)
|
||||
|
||||
async def _resume_handler(resume_token: str, reason: str) -> bool:
|
||||
"""Block until the user sends ``resume`` for the given token."""
|
||||
loop = asyncio.get_running_loop()
|
||||
future: asyncio.Future[bool] = loop.create_future()
|
||||
_pending_autonomy_resumes[resume_token] = future
|
||||
logger.info(f"Autonomy paused ({reason}), waiting for resume: {resume_token}")
|
||||
try:
|
||||
# Wait up to 30 minutes for user resume (long task availability).
|
||||
result = await asyncio.wait_for(future, timeout=1800.0)
|
||||
return bool(result)
|
||||
except asyncio.TimeoutError:
|
||||
logger.warning(f"Autonomy resume {resume_token} timed out (30 min)")
|
||||
return False
|
||||
except asyncio.CancelledError:
|
||||
logger.warning(f"Autonomy resume {resume_token} cancelled")
|
||||
return False
|
||||
finally:
|
||||
_pending_autonomy_resumes.pop(resume_token, None)
|
||||
|
||||
logger.info(
|
||||
f"Chat session {session_id}: executing with {len(routing.tools)} tools, model={routing.model}, skill={routing.skill_name}"
|
||||
)
|
||||
|
|
@ -1595,6 +1656,7 @@ async def _handle_chat_message(
|
|||
system_prompt=routing.system_prompt,
|
||||
cancellation_token=cancellation_token,
|
||||
confirmation_handler=_confirmation_handler,
|
||||
resume_handler=_resume_handler,
|
||||
):
|
||||
if event.event_type == "final_answer":
|
||||
# Flush any buffered tokens as a single write
|
||||
|
|
@ -1652,6 +1714,17 @@ async def _handle_chat_message(
|
|||
"data": event.data,
|
||||
}
|
||||
)
|
||||
elif event.event_type == "autonomy_paused":
|
||||
# IQ-Boost/U3/R10: forward autonomy pause to the frontend.
|
||||
# The _resume_handler closure is already blocking the engine
|
||||
# waiting for the user's ``resume`` message. The frontend
|
||||
# shows a pause card with reason + progress + resume button.
|
||||
await websocket.send_json(
|
||||
{
|
||||
"type": "autonomy_paused",
|
||||
"data": event.data,
|
||||
}
|
||||
)
|
||||
elif event.event_type == "spec_review_request":
|
||||
# U8/R8: the _spec_review_handler closure already sent this
|
||||
# request directly to the frontend (it owns the spec_review_id
|
||||
|
|
|
|||
|
|
@ -0,0 +1,197 @@
|
|||
"""U7: ABTester prompt-version offline comparison (R14).
|
||||
|
||||
Covers:
|
||||
- compare_prompt_versions(): no episodic_memory → "no_data"
|
||||
- Multiple versions sorted by score descending
|
||||
- Single version → that version is best_version
|
||||
- No matching versions → "no_data"
|
||||
- Retrieval failure → "no_data" (non-blocking)
|
||||
- best_version is the highest-scored
|
||||
- Low-score versions included (no cleanup in compare; recommendation only)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from agentkit.evolution.ab_tester import ABTester
|
||||
from agentkit.memory.base import MemoryItem
|
||||
|
||||
|
||||
def _make_item(
|
||||
version: int,
|
||||
score: float,
|
||||
task_hash: str = "abc123",
|
||||
reflection: str = "reflection text",
|
||||
improved_prompt: str = "improved prompt",
|
||||
created_at: datetime | None = None,
|
||||
) -> MemoryItem:
|
||||
"""Build a MemoryItem mimicking list_prompt_reflections_by_hash output."""
|
||||
if created_at is None:
|
||||
created_at = datetime.now(timezone.utc)
|
||||
return MemoryItem(
|
||||
key=f"prompt_reflection:{task_hash}:{version}",
|
||||
value={
|
||||
"input_summary": "task input",
|
||||
"output_summary": improved_prompt,
|
||||
"reflection": reflection,
|
||||
"quality_score": score,
|
||||
},
|
||||
metadata={
|
||||
"agent_name": "test_agent",
|
||||
"task_type": "prompt_reflection",
|
||||
"task_hash": task_hash,
|
||||
"version": version,
|
||||
"score": score,
|
||||
"created_at": created_at.isoformat(),
|
||||
},
|
||||
score=score,
|
||||
created_at=created_at,
|
||||
)
|
||||
|
||||
|
||||
class TestComparePromptVersions:
|
||||
"""Tests for ABTester.compare_prompt_versions()."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_episodic_memory_returns_no_data(self):
|
||||
"""No episodic_memory wired → recommendation='no_data', empty versions."""
|
||||
tester = ABTester(episodic_memory=None)
|
||||
result = await tester.compare_prompt_versions("any_hash")
|
||||
|
||||
assert result["recommendation"] == "no_data"
|
||||
assert result["total_versions"] == 0
|
||||
assert result["versions"] == []
|
||||
assert result["best_version"] is None
|
||||
assert result["task_hash"] == "any_hash"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_multiple_versions_sorted_by_score_desc(self):
|
||||
"""3 versions (0.8, 0.6, 0.4) → best_version score=0.8."""
|
||||
episodic = MagicMock()
|
||||
episodic.list_prompt_reflections_by_hash = AsyncMock(
|
||||
return_value=[
|
||||
_make_item(version=1, score=0.4),
|
||||
_make_item(version=2, score=0.8),
|
||||
_make_item(version=3, score=0.6),
|
||||
]
|
||||
)
|
||||
tester = ABTester(episodic_memory=episodic)
|
||||
|
||||
result = await tester.compare_prompt_versions("abc123")
|
||||
|
||||
assert result["total_versions"] == 3
|
||||
assert result["recommendation"] == "keep_best"
|
||||
versions = result["versions"]
|
||||
# Sorted descending by score
|
||||
assert versions[0]["score"] == 0.8
|
||||
assert versions[1]["score"] == 0.6
|
||||
assert versions[2]["score"] == 0.4
|
||||
# best_version is the highest-scored
|
||||
assert result["best_version"]["version"] == 2
|
||||
assert result["best_version"]["score"] == 0.8
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_single_version_is_best(self):
|
||||
"""Only 1 version → that version is best_version."""
|
||||
episodic = MagicMock()
|
||||
episodic.list_prompt_reflections_by_hash = AsyncMock(
|
||||
return_value=[_make_item(version=1, score=0.7)]
|
||||
)
|
||||
tester = ABTester(episodic_memory=episodic)
|
||||
|
||||
result = await tester.compare_prompt_versions("abc123")
|
||||
|
||||
assert result["total_versions"] == 1
|
||||
assert result["recommendation"] == "keep_best"
|
||||
assert result["best_version"]["version"] == 1
|
||||
assert result["best_version"]["score"] == 0.7
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_matching_versions_returns_no_data(self):
|
||||
"""EpisodicMemory returns empty list → 'no_data'."""
|
||||
episodic = MagicMock()
|
||||
episodic.list_prompt_reflections_by_hash = AsyncMock(return_value=[])
|
||||
tester = ABTester(episodic_memory=episodic)
|
||||
|
||||
result = await tester.compare_prompt_versions("unknown_hash")
|
||||
|
||||
assert result["recommendation"] == "no_data"
|
||||
assert result["total_versions"] == 0
|
||||
assert result["best_version"] is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retrieval_failure_returns_no_data(self):
|
||||
"""list_prompt_reflections_by_hash raises → 'no_data' (non-blocking)."""
|
||||
episodic = MagicMock()
|
||||
episodic.list_prompt_reflections_by_hash = AsyncMock(
|
||||
side_effect=RuntimeError("db connection failed")
|
||||
)
|
||||
tester = ABTester(episodic_memory=episodic)
|
||||
|
||||
result = await tester.compare_prompt_versions("abc123")
|
||||
|
||||
assert result["recommendation"] == "no_data"
|
||||
assert result["total_versions"] == 0
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_versions_include_reflection_and_prompt_fields(self):
|
||||
"""Each version dict carries reflection_summary + improved_prompt."""
|
||||
episodic = MagicMock()
|
||||
episodic.list_prompt_reflections_by_hash = AsyncMock(
|
||||
return_value=[
|
||||
_make_item(
|
||||
version=1,
|
||||
score=0.9,
|
||||
reflection="avoid retrying on schema error",
|
||||
improved_prompt="use structured output schema",
|
||||
)
|
||||
]
|
||||
)
|
||||
tester = ABTester(episodic_memory=episodic)
|
||||
|
||||
result = await tester.compare_prompt_versions("abc123")
|
||||
|
||||
version = result["versions"][0]
|
||||
assert "avoid retrying on schema error" in version["reflection_summary"]
|
||||
assert "use structured output schema" in version["improved_prompt"]
|
||||
assert version["version"] == 1
|
||||
assert version["score"] == 0.9
|
||||
# timestamp is a string (ISO format)
|
||||
assert isinstance(version["timestamp"], str)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_low_score_versions_kept_in_list(self):
|
||||
"""Low-score versions remain in versions list (no cleanup; recommendation only)."""
|
||||
episodic = MagicMock()
|
||||
episodic.list_prompt_reflections_by_hash = AsyncMock(
|
||||
return_value=[
|
||||
_make_item(version=1, score=0.1), # low score
|
||||
_make_item(version=2, score=0.9), # high score
|
||||
]
|
||||
)
|
||||
tester = ABTester(episodic_memory=episodic)
|
||||
|
||||
result = await tester.compare_prompt_versions("abc123")
|
||||
|
||||
# Both versions present — compare does not filter, only sorts
|
||||
assert result["total_versions"] == 2
|
||||
assert result["best_version"]["score"] == 0.9
|
||||
# Low-score version still in list (for inspection)
|
||||
scores = [v["score"] for v in result["versions"]]
|
||||
assert 0.1 in scores
|
||||
assert 0.9 in scores
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_task_hash_echoed_in_result(self):
|
||||
"""task_hash field in result matches input."""
|
||||
episodic = MagicMock()
|
||||
episodic.list_prompt_reflections_by_hash = AsyncMock(return_value=[])
|
||||
tester = ABTester(episodic_memory=episodic)
|
||||
|
||||
result = await tester.compare_prompt_versions("specific_hash_456")
|
||||
|
||||
assert result["task_hash"] == "specific_hash_456"
|
||||
|
|
@ -0,0 +1,393 @@
|
|||
"""Unit tests for autonomy_paused event + timeout/failure tracking (IQ-Boost U3, R10).
|
||||
|
||||
Verifies:
|
||||
- Timeout triggers autonomy_paused with reason="timeout"
|
||||
- Consecutive failures (3x) trigger autonomy_paused with reason="consecutive_failures"
|
||||
- resume_handler returns True → counters reset, execution continues
|
||||
- resume_handler returns False → execution stops (cancel)
|
||||
- resume_handler=None → auto-resume (non-blocking, for tests/REST)
|
||||
- Non-autonomy mode → no pause (gate is a no-op)
|
||||
- Disabled thresholds (0) → no pause
|
||||
- _track_tool_result_for_autonomy increments on error, resets on success
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from agentkit.core.react import ReActEngine, ReActEvent
|
||||
from agentkit.server.config import DangerousToolsConfig
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helper: simulates the caller pattern (detect → yield event → await resume)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _detect_and_await_pause(
|
||||
engine: ReActEngine,
|
||||
step: int,
|
||||
progress: dict,
|
||||
resume_handler=None,
|
||||
) -> tuple[bool, list[ReActEvent]]:
|
||||
"""Simulate the react.py caller pattern for autonomy pause.
|
||||
|
||||
Mirrors the production code: detect (non-blocking) → build event →
|
||||
yield event → await resume (blocking). Returns (should_continue, events).
|
||||
"""
|
||||
pause_info = engine._detect_autonomy_pause(step, progress)
|
||||
if pause_info is None:
|
||||
return True, []
|
||||
reason, token, event_data = pause_info
|
||||
event = ReActEvent(event_type="autonomy_paused", step=step, data=event_data)
|
||||
should_continue = await engine._await_autonomy_resume(
|
||||
token, reason, resume_handler
|
||||
)
|
||||
return should_continue, [event]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _track_tool_result_for_autonomy
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestTrackToolResult:
|
||||
"""Failure tracking increments on error, resets on success."""
|
||||
|
||||
def test_success_resets_counter(self):
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
engine._consecutive_failures = 2
|
||||
engine._track_tool_result_for_autonomy({"output": "ok"})
|
||||
assert engine._consecutive_failures == 0
|
||||
|
||||
def test_error_increments_counter(self):
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
engine._consecutive_failures = 1
|
||||
engine._track_tool_result_for_autonomy({"error": "boom"})
|
||||
assert engine._consecutive_failures == 2
|
||||
|
||||
def test_is_error_flag_increments(self):
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
engine._track_tool_result_for_autonomy({"is_error": True, "output": ""})
|
||||
assert engine._consecutive_failures == 1
|
||||
|
||||
def test_error_type_increments(self):
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
engine._track_tool_result_for_autonomy({"error_type": "permission_denied"})
|
||||
assert engine._consecutive_failures == 1
|
||||
|
||||
def test_non_dict_result_resets(self):
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
engine._consecutive_failures = 2
|
||||
engine._track_tool_result_for_autonomy("plain string result")
|
||||
assert engine._consecutive_failures == 0
|
||||
|
||||
def test_no_tracking_when_not_autonomy(self):
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(),
|
||||
autonomy_mode=False,
|
||||
)
|
||||
engine._consecutive_failures = 0
|
||||
engine._track_tool_result_for_autonomy({"error": "boom"})
|
||||
assert engine._consecutive_failures == 0 # Unchanged
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _detect_autonomy_pause + _await_autonomy_resume — no pause conditions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAutonomyPauseNoTrigger:
|
||||
"""When conditions are not met, no pause is triggered (should_continue=True)."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_autonomy_mode_no_pause(self):
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(),
|
||||
autonomy_mode=False,
|
||||
)
|
||||
should_continue, events = await _detect_and_await_pause(
|
||||
engine, step=1, progress={}, resume_handler=None
|
||||
)
|
||||
assert should_continue is True
|
||||
assert events == []
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_config_no_pause(self):
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=None,
|
||||
autonomy_mode=True,
|
||||
)
|
||||
should_continue, events = await _detect_and_await_pause(
|
||||
engine, step=1, progress={}, resume_handler=None
|
||||
)
|
||||
assert should_continue is True
|
||||
assert events == []
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_below_thresholds_no_pause(self):
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(
|
||||
autonomy_timeout_minutes=30,
|
||||
max_consecutive_failures=3,
|
||||
),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
engine._autonomy_started_at = time.time() # Just started
|
||||
engine._consecutive_failures = 1 # Below threshold
|
||||
should_continue, events = await _detect_and_await_pause(
|
||||
engine, step=1, progress={}, resume_handler=None
|
||||
)
|
||||
assert should_continue is True
|
||||
assert events == []
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_disabled_thresholds_no_pause(self):
|
||||
"""0 = disabled for both thresholds."""
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(
|
||||
autonomy_timeout_minutes=0,
|
||||
max_consecutive_failures=0,
|
||||
),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
# Even with expired time + many failures, disabled = no pause.
|
||||
engine._autonomy_started_at = time.time() - 999999
|
||||
engine._consecutive_failures = 99
|
||||
should_continue, events = await _detect_and_await_pause(
|
||||
engine, step=1, progress={}, resume_handler=None
|
||||
)
|
||||
assert should_continue is True
|
||||
assert events == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _detect_autonomy_pause + _await_autonomy_resume — timeout trigger
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAutonomyPauseTimeout:
|
||||
"""Timeout triggers autonomy_paused with reason='timeout'."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_timeout_triggers_pause_auto_resume(self):
|
||||
"""resume_handler=None → auto-resume (non-blocking)."""
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(
|
||||
autonomy_timeout_minutes=1, # 1 minute
|
||||
max_consecutive_failures=99, # Disabled for this test
|
||||
),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
# Simulate started 2 minutes ago (exceeds 1-min timeout).
|
||||
engine._autonomy_started_at = time.time() - 120
|
||||
|
||||
should_continue, events = await _detect_and_await_pause(
|
||||
engine, step=5, progress={"step": 5}, resume_handler=None
|
||||
)
|
||||
|
||||
# Auto-resume → should_continue=True
|
||||
assert should_continue is True
|
||||
assert len(events) == 1
|
||||
assert events[0].event_type == "autonomy_paused"
|
||||
assert events[0].data["reason"] == "timeout"
|
||||
assert "resume_token" in events[0].data
|
||||
assert events[0].data["elapsed_seconds"] >= 120
|
||||
# Counters reset after auto-resume
|
||||
assert engine._consecutive_failures == 0
|
||||
assert engine._autonomy_started_at > time.time() - 1 # Fresh timer
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_timeout_triggers_pause_user_resumes(self):
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(
|
||||
autonomy_timeout_minutes=1,
|
||||
max_consecutive_failures=99,
|
||||
),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
engine._autonomy_started_at = time.time() - 120
|
||||
|
||||
resume_handler = AsyncMock(return_value=True)
|
||||
should_continue, events = await _detect_and_await_pause(
|
||||
engine, step=3, progress={}, resume_handler=resume_handler
|
||||
)
|
||||
|
||||
assert should_continue is True
|
||||
assert events[0].data["reason"] == "timeout"
|
||||
resume_handler.assert_awaited_once()
|
||||
# Counters reset
|
||||
assert engine._consecutive_failures == 0
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_timeout_triggers_pause_user_cancels(self):
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(
|
||||
autonomy_timeout_minutes=1,
|
||||
max_consecutive_failures=99,
|
||||
),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
engine._autonomy_started_at = time.time() - 120
|
||||
|
||||
resume_handler = AsyncMock(return_value=False)
|
||||
should_continue, events = await _detect_and_await_pause(
|
||||
engine, step=3, progress={}, resume_handler=resume_handler
|
||||
)
|
||||
|
||||
assert should_continue is False # Cancelled
|
||||
assert events[0].data["reason"] == "timeout"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _detect_autonomy_pause + _await_autonomy_resume — consecutive failures trigger
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAutonomyPauseConsecutiveFailures:
|
||||
"""3 consecutive failures trigger autonomy_paused."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_consecutive_failures_triggers_pause(self):
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(
|
||||
autonomy_timeout_minutes=0, # Disabled
|
||||
max_consecutive_failures=3,
|
||||
),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
engine._autonomy_started_at = time.time()
|
||||
engine._consecutive_failures = 3 # At threshold
|
||||
|
||||
resume_handler = AsyncMock(return_value=True)
|
||||
should_continue, events = await _detect_and_await_pause(
|
||||
engine, step=5, progress={"step": 5}, resume_handler=resume_handler
|
||||
)
|
||||
|
||||
assert should_continue is True # Resumed
|
||||
assert len(events) == 1
|
||||
assert events[0].event_type == "autonomy_paused"
|
||||
assert events[0].data["reason"] == "consecutive_failures"
|
||||
assert events[0].data["consecutive_failures"] == 3
|
||||
# Reset after resume
|
||||
assert engine._consecutive_failures == 0
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_below_failure_threshold_no_pause(self):
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(
|
||||
autonomy_timeout_minutes=0,
|
||||
max_consecutive_failures=3,
|
||||
),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
engine._autonomy_started_at = time.time()
|
||||
engine._consecutive_failures = 2 # Below threshold
|
||||
|
||||
should_continue, events = await _detect_and_await_pause(
|
||||
engine, step=1, progress={}, resume_handler=None
|
||||
)
|
||||
|
||||
assert should_continue is True
|
||||
assert events == []
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_resume_handler_exception_cancels(self):
|
||||
"""If resume_handler raises, treat as cancel (should_continue=False)."""
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(
|
||||
autonomy_timeout_minutes=0,
|
||||
max_consecutive_failures=3,
|
||||
),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
engine._consecutive_failures = 3
|
||||
|
||||
async def failing_handler(token, reason):
|
||||
raise RuntimeError("handler crashed")
|
||||
|
||||
should_continue, events = await _detect_and_await_pause(
|
||||
engine, step=1, progress={}, resume_handler=failing_handler
|
||||
)
|
||||
|
||||
assert should_continue is False # Cancelled due to exception
|
||||
assert events[0].data["reason"] == "consecutive_failures"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Config parsing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAutonomyConfigParsing:
|
||||
"""DangerousToolsConfig parses autonomy thresholds from dict."""
|
||||
|
||||
def test_defaults(self):
|
||||
cfg = DangerousToolsConfig()
|
||||
assert cfg.autonomy_timeout_minutes == 30
|
||||
assert cfg.max_consecutive_failures == 3
|
||||
|
||||
def test_custom_values_from_dict(self):
|
||||
cfg = DangerousToolsConfig.from_dict(
|
||||
{
|
||||
"enabled": True,
|
||||
"autonomy_timeout_minutes": 60,
|
||||
"max_consecutive_failures": 5,
|
||||
}
|
||||
)
|
||||
assert cfg.autonomy_timeout_minutes == 60
|
||||
assert cfg.max_consecutive_failures == 5
|
||||
|
||||
def test_disabled_values_from_dict(self):
|
||||
cfg = DangerousToolsConfig.from_dict(
|
||||
{
|
||||
"autonomy_timeout_minutes": 0,
|
||||
"max_consecutive_failures": 0,
|
||||
}
|
||||
)
|
||||
assert cfg.autonomy_timeout_minutes == 0
|
||||
assert cfg.max_consecutive_failures == 0
|
||||
|
||||
def test_partial_config_uses_defaults(self):
|
||||
cfg = DangerousToolsConfig.from_dict({"enabled": True})
|
||||
assert cfg.autonomy_timeout_minutes == 30
|
||||
assert cfg.max_consecutive_failures == 3
|
||||
|
||||
def test_empty_dict_uses_defaults(self):
|
||||
cfg = DangerousToolsConfig.from_dict({})
|
||||
assert cfg.autonomy_timeout_minutes == 30
|
||||
assert cfg.max_consecutive_failures == 3
|
||||
|
|
@ -0,0 +1,325 @@
|
|||
"""U6: Lead planning-time historical reflection retrieval (R12, R13).
|
||||
|
||||
Covers:
|
||||
- ReflexionEngine.retrieve_prompt_reflection(): returns best reflection by score
|
||||
- Score filtering: score <= 0.5 not returned
|
||||
- No episodic_memory → returns None
|
||||
- Retrieval failure → returns None (non-blocking)
|
||||
- TeamOrchestrator._decompose_task: prepends improved_prompt when reflection found
|
||||
- No reflexion_engine → default prompt (backward compat)
|
||||
- Retrieval failure → default prompt (non-blocking)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from agentkit.core.reflexion import ReflexionEngine
|
||||
from agentkit.experts.orchestrator import TeamOrchestrator
|
||||
from agentkit.experts.team import ExpertTeam
|
||||
from agentkit.memory.base import MemoryItem
|
||||
|
||||
from tests.unit.experts._helpers import make_execute_stream_mock
|
||||
|
||||
|
||||
# ── Helpers ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _make_memory_item(
|
||||
score: float = 0.8,
|
||||
output_summary: str = "improved prompt text",
|
||||
reflection: str = "reflection text",
|
||||
version: int = 1,
|
||||
) -> MemoryItem:
|
||||
from datetime import datetime, timezone
|
||||
|
||||
# Shape matches EpisodicMemory.list_prompt_reflections_by_hash return:
|
||||
# output_summary/reflection/quality_score live in value dict,
|
||||
# version/task_hash live in metadata.
|
||||
return MemoryItem(
|
||||
key="prompt_reflection:abc:1",
|
||||
value={
|
||||
"input_summary": "test",
|
||||
"output_summary": output_summary,
|
||||
"reflection": reflection,
|
||||
"quality_score": score,
|
||||
},
|
||||
metadata={
|
||||
"task_type": "prompt_reflection",
|
||||
"version": version,
|
||||
"task_hash": "abc",
|
||||
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||
},
|
||||
score=score,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
)
|
||||
|
||||
|
||||
def _make_llm_gateway_mock() -> MagicMock:
|
||||
gw = MagicMock()
|
||||
response = MagicMock()
|
||||
response.content = (
|
||||
'[{"name":"A","assigned_expert":"lead","task_description":"a","depends_on":[]}]'
|
||||
)
|
||||
gw.chat = AsyncMock(return_value=response)
|
||||
return gw
|
||||
|
||||
|
||||
def _make_team_with_experts() -> ExpertTeam:
|
||||
from agentkit.core.handoff_transport import InProcessHandoffTransport
|
||||
from agentkit.core.protocol import TaskResult, TaskStatus
|
||||
from agentkit.experts.config import ExpertConfig
|
||||
from agentkit.experts.expert import Expert
|
||||
|
||||
team = ExpertTeam()
|
||||
team._handoff_transport = AsyncMock(spec=InProcessHandoffTransport)
|
||||
|
||||
config = ExpertConfig(
|
||||
name="lead",
|
||||
agent_type="expert",
|
||||
persona="测试",
|
||||
thinking_style="逻辑",
|
||||
bound_skills=["s"],
|
||||
is_lead=True,
|
||||
task_mode="llm_generate",
|
||||
prompt={"identity": "测试"},
|
||||
)
|
||||
expert = MagicMock(spec=Expert)
|
||||
expert.config = config
|
||||
expert.is_active = True
|
||||
expert.team_id = None
|
||||
expert.get_capabilities_summary.return_value = {"name": "lead"}
|
||||
|
||||
mock_agent = MagicMock()
|
||||
mock_agent.execute = AsyncMock(
|
||||
return_value=TaskResult(
|
||||
task_id="t",
|
||||
agent_name="lead",
|
||||
status=TaskStatus.COMPLETED.value,
|
||||
output_data={"content": "result"},
|
||||
error_message=None,
|
||||
started_at=None,
|
||||
completed_at=None,
|
||||
)
|
||||
)
|
||||
mock_agent.execute_stream = make_execute_stream_mock("result")
|
||||
mock_agent._llm_gateway = None
|
||||
expert.agent = mock_agent
|
||||
|
||||
team._experts["lead"] = expert
|
||||
team._lead_expert_name = "lead"
|
||||
return team
|
||||
|
||||
|
||||
# ── ReflexionEngine.retrieve_prompt_reflection ─────────────────────────
|
||||
|
||||
|
||||
class TestRetrievePromptReflection:
|
||||
"""U6/R12: retrieve_prompt_reflection returns best reflection by score."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_returns_none_when_no_episodic_memory(self):
|
||||
gw = MagicMock()
|
||||
engine = ReflexionEngine(llm_gateway=gw, episodic_memory=None)
|
||||
result = await engine.retrieve_prompt_reflection(task_input="test")
|
||||
assert result is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_returns_best_reflection_by_score(self):
|
||||
episodic = MagicMock()
|
||||
# Two results: score 0.6 and 0.9 — should return 0.9
|
||||
items = [
|
||||
_make_memory_item(score=0.6, output_summary="prompt v1"),
|
||||
_make_memory_item(score=0.9, output_summary="prompt v2"),
|
||||
]
|
||||
episodic.search_prompt_reflections = AsyncMock(return_value=items)
|
||||
gw = MagicMock()
|
||||
engine = ReflexionEngine(llm_gateway=gw, episodic_memory=episodic)
|
||||
|
||||
result = await engine.retrieve_prompt_reflection(task_input="test")
|
||||
|
||||
assert result is not None
|
||||
assert result["score"] == 0.9
|
||||
assert result["improved_prompt"] == "prompt v2"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_filters_low_score_reflections(self):
|
||||
"""score <= 0.5 should not be returned."""
|
||||
episodic = MagicMock()
|
||||
items = [_make_memory_item(score=0.3, output_summary="low score")]
|
||||
episodic.search_prompt_reflections = AsyncMock(return_value=items)
|
||||
gw = MagicMock()
|
||||
engine = ReflexionEngine(llm_gateway=gw, episodic_memory=episodic)
|
||||
|
||||
result = await engine.retrieve_prompt_reflection(task_input="test", min_score=0.5)
|
||||
assert result is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_returns_none_when_no_results(self):
|
||||
episodic = MagicMock()
|
||||
episodic.search_prompt_reflections = AsyncMock(return_value=[])
|
||||
gw = MagicMock()
|
||||
engine = ReflexionEngine(llm_gateway=gw, episodic_memory=episodic)
|
||||
|
||||
result = await engine.retrieve_prompt_reflection(task_input="test")
|
||||
assert result is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_returns_none_on_search_failure(self):
|
||||
episodic = MagicMock()
|
||||
episodic.search_prompt_reflections = AsyncMock(side_effect=RuntimeError("DB down"))
|
||||
gw = MagicMock()
|
||||
engine = ReflexionEngine(llm_gateway=gw, episodic_memory=episodic)
|
||||
|
||||
result = await engine.retrieve_prompt_reflection(task_input="test")
|
||||
assert result is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_returns_reflection_fields_complete(self):
|
||||
episodic = MagicMock()
|
||||
items = [
|
||||
_make_memory_item(
|
||||
score=0.85,
|
||||
output_summary="improved prompt",
|
||||
reflection="what went wrong",
|
||||
version=3,
|
||||
)
|
||||
]
|
||||
episodic.search_prompt_reflections = AsyncMock(return_value=items)
|
||||
gw = MagicMock()
|
||||
engine = ReflexionEngine(llm_gateway=gw, episodic_memory=episodic)
|
||||
|
||||
result = await engine.retrieve_prompt_reflection(task_input="test")
|
||||
|
||||
assert result is not None
|
||||
assert result["improved_prompt"] == "improved prompt"
|
||||
assert result["reflection"] == "what went wrong"
|
||||
assert result["version"] == 3
|
||||
assert result["score"] == 0.85
|
||||
assert "task_hash" in result
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_custom_min_score_threshold(self):
|
||||
"""min_score=0.7 filters out score=0.6."""
|
||||
episodic = MagicMock()
|
||||
items = [_make_memory_item(score=0.6, output_summary="medium")]
|
||||
episodic.search_prompt_reflections = AsyncMock(return_value=items)
|
||||
gw = MagicMock()
|
||||
engine = ReflexionEngine(llm_gateway=gw, episodic_memory=episodic)
|
||||
|
||||
result = await engine.retrieve_prompt_reflection(task_input="test", min_score=0.7)
|
||||
assert result is None
|
||||
|
||||
|
||||
# ── TeamOrchestrator._decompose_task with reflection ──────────────────
|
||||
|
||||
|
||||
class TestDecomposeWithReflection:
|
||||
"""U6/R13: _decompose_task prepends improved_prompt when reflection found."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_decompose_prepends_reflection_when_found(self):
|
||||
"""When reflexion_engine returns a reflection, the decomposition
|
||||
prompt includes the improved_prompt."""
|
||||
team = _make_team_with_experts()
|
||||
gw = _make_llm_gateway_mock()
|
||||
team._experts["lead"].agent._llm_gateway = gw
|
||||
|
||||
# Mock reflexion_engine
|
||||
reflexion = MagicMock(spec=ReflexionEngine)
|
||||
reflexion.retrieve_prompt_reflection = AsyncMock(
|
||||
return_value={
|
||||
"improved_prompt": "USE THIS IMPROVED APPROACH",
|
||||
"score": 0.85,
|
||||
"reflection": "past mistake",
|
||||
"version": 2,
|
||||
"task_hash": "abc",
|
||||
}
|
||||
)
|
||||
|
||||
orchestrator = TeamOrchestrator(team, reflexion_engine=reflexion)
|
||||
await orchestrator._decompose_task(team.lead_expert, "test task")
|
||||
|
||||
# Verify retrieve was called
|
||||
reflexion.retrieve_prompt_reflection.assert_awaited_once()
|
||||
# Verify the prompt sent to LLM includes the improved_prompt
|
||||
call_kwargs = gw.chat.await_args.kwargs
|
||||
messages = call_kwargs.get("messages") or gw.chat.await_args.args[0]
|
||||
prompt_content = messages[0]["content"] if isinstance(messages, list) else str(messages)
|
||||
assert "USE THIS IMPROVED APPROACH" in prompt_content
|
||||
assert "Historical Reflection" in prompt_content
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_decompose_uses_default_when_no_reflexion_engine(self):
|
||||
"""No reflexion_engine → default prompt (backward compat)."""
|
||||
team = _make_team_with_experts()
|
||||
gw = _make_llm_gateway_mock()
|
||||
team._experts["lead"].agent._llm_gateway = gw
|
||||
|
||||
orchestrator = TeamOrchestrator(team, reflexion_engine=None)
|
||||
await orchestrator._decompose_task(team.lead_expert, "test task")
|
||||
|
||||
# Verify default prompt (no Historical Reflection section)
|
||||
call_kwargs = gw.chat.await_args.kwargs
|
||||
messages = call_kwargs.get("messages") or gw.chat.await_args.args[0]
|
||||
prompt_content = messages[0]["content"] if isinstance(messages, list) else str(messages)
|
||||
assert "Historical Reflection" not in prompt_content
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_decompose_uses_default_when_no_reflection_found(self):
|
||||
"""reflexion_engine returns None → default prompt."""
|
||||
team = _make_team_with_experts()
|
||||
gw = _make_llm_gateway_mock()
|
||||
team._experts["lead"].agent._llm_gateway = gw
|
||||
|
||||
reflexion = MagicMock(spec=ReflexionEngine)
|
||||
reflexion.retrieve_prompt_reflection = AsyncMock(return_value=None)
|
||||
|
||||
orchestrator = TeamOrchestrator(team, reflexion_engine=reflexion)
|
||||
await orchestrator._decompose_task(team.lead_expert, "test task")
|
||||
|
||||
call_kwargs = gw.chat.await_args.kwargs
|
||||
messages = call_kwargs.get("messages") or gw.chat.await_args.args[0]
|
||||
prompt_content = messages[0]["content"] if isinstance(messages, list) else str(messages)
|
||||
assert "Historical Reflection" not in prompt_content
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_decompose_uses_default_when_retrieval_fails(self):
|
||||
"""reflexion_engine.retrieve raises → default prompt (non-blocking)."""
|
||||
team = _make_team_with_experts()
|
||||
gw = _make_llm_gateway_mock()
|
||||
team._experts["lead"].agent._llm_gateway = gw
|
||||
|
||||
reflexion = MagicMock(spec=ReflexionEngine)
|
||||
reflexion.retrieve_prompt_reflection = AsyncMock(side_effect=RuntimeError("search failed"))
|
||||
|
||||
orchestrator = TeamOrchestrator(team, reflexion_engine=reflexion)
|
||||
await orchestrator._decompose_task(team.lead_expert, "test task")
|
||||
|
||||
# Default prompt used despite retrieval failure
|
||||
call_kwargs = gw.chat.await_args.kwargs
|
||||
messages = call_kwargs.get("messages") or gw.chat.await_args.args[0]
|
||||
prompt_content = messages[0]["content"] if isinstance(messages, list) else str(messages)
|
||||
assert "Historical Reflection" not in prompt_content
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_decompose_skips_reflection_when_no_improved_prompt(self):
|
||||
"""reflexion_engine returns dict without improved_prompt → no hint."""
|
||||
team = _make_team_with_experts()
|
||||
gw = _make_llm_gateway_mock()
|
||||
team._experts["lead"].agent._llm_gateway = gw
|
||||
|
||||
reflexion = MagicMock(spec=ReflexionEngine)
|
||||
reflexion.retrieve_prompt_reflection = AsyncMock(
|
||||
return_value={"improved_prompt": "", "score": 0.8} # empty improved_prompt
|
||||
)
|
||||
|
||||
orchestrator = TeamOrchestrator(team, reflexion_engine=reflexion)
|
||||
await orchestrator._decompose_task(team.lead_expert, "test task")
|
||||
|
||||
call_kwargs = gw.chat.await_args.kwargs
|
||||
messages = call_kwargs.get("messages") or gw.chat.await_args.args[0]
|
||||
prompt_content = messages[0]["content"] if isinstance(messages, list) else str(messages)
|
||||
assert "Historical Reflection" not in prompt_content
|
||||
|
|
@ -0,0 +1,123 @@
|
|||
"""Integration tests for PLAN_EXEC autonomy mode wiring (IQ-Boost U2).
|
||||
|
||||
Verifies that ``_build_phase_engine`` in chat.py constructs a PLAN_EXEC
|
||||
engine with ``autonomy_mode=True`` and ``dangerous_tools_config`` injected
|
||||
from ``ServerConfig``. This is the integration seam between U1 (config)
|
||||
and U2 (engine gate).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from agentkit.chat.skill_routing import ExecutionMode
|
||||
from agentkit.core.react import ReActEngine
|
||||
from agentkit.server.config import DangerousToolsConfig, ServerConfig
|
||||
|
||||
|
||||
def _make_server_config(dangerous_tools: DangerousToolsConfig | None = None) -> ServerConfig:
|
||||
"""Build a minimal ServerConfig with the dangerous_tools section."""
|
||||
cfg = ServerConfig.from_dict({})
|
||||
if dangerous_tools is not None:
|
||||
cfg.dangerous_tools = dangerous_tools
|
||||
return cfg
|
||||
|
||||
|
||||
class TestBuildPhaseEngineAutonomyWiring:
|
||||
"""_build_phase_engine injects autonomy config into the PLAN_EXEC engine."""
|
||||
|
||||
def test_plan_exec_engine_gets_autonomy_mode(self):
|
||||
"""When PLAN_EXEC is engaged, the engine has autonomy_mode=True."""
|
||||
# Import here to avoid module-level FastAPI app construction.
|
||||
from agentkit.server.routes.chat import _build_phase_engine
|
||||
|
||||
server_cfg = _make_server_config(DangerousToolsConfig())
|
||||
engine, tools, err = _build_phase_engine(
|
||||
server_config=server_cfg,
|
||||
llm_gateway=MagicMock(),
|
||||
execution_mode=ExecutionMode.PLAN_EXEC,
|
||||
base_tools=[],
|
||||
session_id="test-session",
|
||||
)
|
||||
assert err is None
|
||||
assert engine is not None
|
||||
assert isinstance(engine, ReActEngine)
|
||||
assert engine._autonomy_mode is True
|
||||
assert engine._dangerous_tools_config is not None
|
||||
assert engine._dangerous_tools_config.is_dangerous("rm") is True
|
||||
|
||||
def test_plan_exec_engine_gets_custom_dangerous_config(self):
|
||||
"""Custom dangerous_tools config (disabled) propagates to the engine."""
|
||||
from agentkit.server.routes.chat import _build_phase_engine
|
||||
|
||||
server_cfg = _make_server_config(DangerousToolsConfig(enabled=False))
|
||||
engine, _, err = _build_phase_engine(
|
||||
server_config=server_cfg,
|
||||
llm_gateway=MagicMock(),
|
||||
execution_mode=ExecutionMode.PLAN_EXEC,
|
||||
base_tools=[],
|
||||
session_id="test",
|
||||
)
|
||||
assert err is None
|
||||
assert engine is not None
|
||||
assert engine._dangerous_tools_config.enabled is False
|
||||
# Disabled whitelist → nothing is dangerous
|
||||
assert engine._dangerous_tools_config.is_dangerous("rm") is False
|
||||
|
||||
def test_non_plan_exec_mode_returns_none_engine(self):
|
||||
"""Non-PLAN_EXEC mode → (None, None, None) — no autonomy engine built."""
|
||||
from agentkit.server.routes.chat import _build_phase_engine
|
||||
|
||||
server_cfg = _make_server_config(DangerousToolsConfig())
|
||||
engine, tools, err = _build_phase_engine(
|
||||
server_config=server_cfg,
|
||||
llm_gateway=MagicMock(),
|
||||
execution_mode=ExecutionMode.REACT,
|
||||
base_tools=[],
|
||||
session_id="test",
|
||||
)
|
||||
assert engine is None
|
||||
assert tools is None
|
||||
assert err is None
|
||||
|
||||
def test_plan_exec_engine_gets_default_dangerous_config_when_server_config_missing(
|
||||
self,
|
||||
):
|
||||
"""When server_config is None or lacks dangerous_tools, the engine
|
||||
still gets autonomy_mode=True (config defaults to None — gate is a
|
||||
no-op, but autonomy flag is set for future wiring)."""
|
||||
from agentkit.server.routes.chat import _build_phase_engine
|
||||
|
||||
# server_config=None simulates tests/misconfigured deployments.
|
||||
engine, _, err = _build_phase_engine(
|
||||
server_config=None,
|
||||
llm_gateway=MagicMock(),
|
||||
execution_mode=ExecutionMode.PLAN_EXEC,
|
||||
base_tools=[],
|
||||
session_id="test",
|
||||
)
|
||||
assert err is None
|
||||
assert engine is not None
|
||||
assert engine._autonomy_mode is True
|
||||
# dangerous_tools_config is None when server_config is None — gate
|
||||
# becomes a transparent no-op (backward compat for tests without config).
|
||||
assert engine._dangerous_tools_config is None
|
||||
|
||||
def test_plan_exec_engine_includes_advance_phase_tool(self):
|
||||
"""PLAN_EXEC engine's tool list includes AdvancePhaseTool."""
|
||||
from agentkit.server.routes.chat import _build_phase_engine
|
||||
|
||||
server_cfg = _make_server_config(DangerousToolsConfig())
|
||||
engine, tools, err = _build_phase_engine(
|
||||
server_config=server_cfg,
|
||||
llm_gateway=MagicMock(),
|
||||
execution_mode=ExecutionMode.PLAN_EXEC,
|
||||
base_tools=[],
|
||||
session_id="test",
|
||||
)
|
||||
assert err is None
|
||||
assert engine is not None
|
||||
assert tools is not None
|
||||
# AdvancePhaseTool appended to the base tool list.
|
||||
assert len(tools) == 1
|
||||
assert type(tools[0]).__name__ == "AdvancePhaseTool"
|
||||
|
|
@ -0,0 +1,302 @@
|
|||
"""Unit tests for ReActEngine autonomy mode (IQ-Boost U2, R6-R9).
|
||||
|
||||
Verifies the PLAN_EXEC autonomy gate:
|
||||
- Dangerous tools (per DangerousToolsConfig) trigger confirmation_request
|
||||
BEFORE first execution in autonomy mode.
|
||||
- Non-dangerous tools execute directly without confirmation.
|
||||
- Non-autonomy mode preserves existing behavior (gate is a no-op).
|
||||
- Disabled whitelist (enabled=False) allows all tools.
|
||||
- Approved dangerous tool re-executes with _skip_dangerous_check=True.
|
||||
- Rejected dangerous tool returns permission_denied result.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from agentkit.core.react import ReActEngine
|
||||
from agentkit.server.config import DangerousToolsConfig
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _check_autonomy_gate — backward compatibility (no-op when disabled)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAutonomyGateBackwardCompat:
|
||||
"""When autonomy_mode=False or dangerous_tools_config=None, the gate is a
|
||||
transparent no-op — existing callers see no behavior change."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_autonomy_mode_returns_should_execute(self):
|
||||
"""Default engine (autonomy_mode=False) → gate passes through."""
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(),
|
||||
autonomy_mode=False,
|
||||
)
|
||||
should_exec, events, result = await engine._check_autonomy_gate(
|
||||
"rm", {"path": "/tmp/x"}, [], step=1, confirmation_handler=None
|
||||
)
|
||||
assert should_exec is True
|
||||
assert events == []
|
||||
assert result is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_dangerous_config_returns_should_execute(self):
|
||||
"""autonomy_mode=True but no config → gate passes through (no whitelist
|
||||
to check against)."""
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=None,
|
||||
autonomy_mode=True,
|
||||
)
|
||||
should_exec, events, result = await engine._check_autonomy_gate(
|
||||
"rm", {"path": "/tmp/x"}, [], step=1, confirmation_handler=None
|
||||
)
|
||||
assert should_exec is True
|
||||
assert events == []
|
||||
assert result is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_disabled_whitelist_allows_all(self):
|
||||
"""enabled=False → is_dangerous always returns False → all tools pass."""
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(enabled=False),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
should_exec, events, result = await engine._check_autonomy_gate(
|
||||
"rm", {"path": "/etc"}, [], step=1, confirmation_handler=None
|
||||
)
|
||||
assert should_exec is True
|
||||
assert events == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _check_autonomy_gate — non-dangerous tool in autonomy mode
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAutonomyGateNonDangerous:
|
||||
"""In autonomy mode, non-dangerous tools execute without confirmation."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_non_dangerous_tool_passes_through(self):
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
should_exec, events, result = await engine._check_autonomy_gate(
|
||||
"read_file", {"path": "/tmp/x"}, [], step=1, confirmation_handler=None
|
||||
)
|
||||
assert should_exec is True
|
||||
assert events == []
|
||||
assert result is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_custom_pattern_non_match_passes(self):
|
||||
"""Tool not matching any custom pattern passes through."""
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(tool_patterns=[r"^rm$", r"^kubectl"]),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
should_exec, events, result = await engine._check_autonomy_gate(
|
||||
"search", {"query": "foo"}, [], step=1, confirmation_handler=None
|
||||
)
|
||||
assert should_exec is True
|
||||
assert events == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _check_autonomy_gate — dangerous tool triggers confirmation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAutonomyGateDangerous:
|
||||
"""Dangerous tools in autonomy mode trigger confirmation_request before
|
||||
execution."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dangerous_tool_emits_confirmation_request(self):
|
||||
"""rm matches ^rm$ pattern → confirmation_request event emitted."""
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
fake_tool = MagicMock()
|
||||
fake_tool.safe_execute = AsyncMock(return_value={"output": "deleted"})
|
||||
engine._find_tool = lambda name, tools: fake_tool
|
||||
|
||||
handler = AsyncMock(return_value=True)
|
||||
should_exec, events, result = await engine._check_autonomy_gate(
|
||||
"rm", {"path": "/tmp/x"}, [fake_tool], step=1, confirmation_handler=handler
|
||||
)
|
||||
|
||||
assert should_exec is False # Gate handled execution
|
||||
assert len(events) == 2 # confirmation_request + confirmation_result
|
||||
assert events[0].event_type == "confirmation_request"
|
||||
assert events[0].data["tool_name"] == "rm"
|
||||
assert events[0].data["confirmation_id"].startswith("autonomy:rm:")
|
||||
assert events[1].event_type == "confirmation_result"
|
||||
assert events[1].data["approved"] is True
|
||||
# Tool executed with _skip_dangerous_check=True
|
||||
fake_tool.safe_execute.assert_awaited_once_with(path="/tmp/x", _skip_dangerous_check=True)
|
||||
assert result == {"output": "deleted"}
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dangerous_tool_rejected_returns_permission_denied(self):
|
||||
"""User rejects → permission_denied result, tool NOT executed."""
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
fake_tool = MagicMock()
|
||||
fake_tool.safe_execute = AsyncMock(return_value={"output": "should not run"})
|
||||
engine._find_tool = lambda name, tools: fake_tool
|
||||
|
||||
handler = AsyncMock(return_value=False)
|
||||
should_exec, events, result = await engine._check_autonomy_gate(
|
||||
"rm", {"path": "/tmp/x"}, [fake_tool], step=1, confirmation_handler=handler
|
||||
)
|
||||
|
||||
assert should_exec is False
|
||||
assert len(events) == 2
|
||||
assert events[1].data["approved"] is False
|
||||
fake_tool.safe_execute.assert_not_awaited()
|
||||
assert result["error_type"] == "permission_denied"
|
||||
assert result["exit_code"] == 126
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dangerous_tool_no_handler_auto_rejects(self):
|
||||
"""No confirmation_handler → auto-reject (safe default)."""
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
engine._find_tool = lambda name, tools: MagicMock()
|
||||
|
||||
should_exec, events, result = await engine._check_autonomy_gate(
|
||||
"kubectl_delete", {"ns": "prod"}, [], step=2, confirmation_handler=None
|
||||
)
|
||||
|
||||
assert should_exec is False
|
||||
assert events[1].data["approved"] is False
|
||||
assert result["error_type"] == "permission_denied"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dangerous_tool_handler_exception_auto_rejects(self):
|
||||
"""Handler raises → auto-reject (does not crash the engine)."""
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
engine._find_tool = lambda name, tools: MagicMock()
|
||||
|
||||
async def failing_handler(cid, cmd, reason):
|
||||
raise RuntimeError("handler crashed")
|
||||
|
||||
should_exec, events, result = await engine._check_autonomy_gate(
|
||||
"deploy", {"env": "prod"}, [], step=3, confirmation_handler=failing_handler
|
||||
)
|
||||
|
||||
assert should_exec is False
|
||||
assert events[1].data["approved"] is False
|
||||
assert result["error_type"] == "permission_denied"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_payment_glob_pattern_matches(self):
|
||||
"""payment_* pattern matches payment_charge tool."""
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
fake_tool = MagicMock()
|
||||
fake_tool.safe_execute = AsyncMock(return_value={"ok": True})
|
||||
engine._find_tool = lambda name, tools: fake_tool
|
||||
|
||||
handler = AsyncMock(return_value=True)
|
||||
should_exec, events, result = await engine._check_autonomy_gate(
|
||||
"payment_charge",
|
||||
{"amount": 100},
|
||||
[fake_tool],
|
||||
step=1,
|
||||
confirmation_handler=handler,
|
||||
)
|
||||
|
||||
assert should_exec is False
|
||||
assert events[0].event_type == "confirmation_request"
|
||||
assert result == {"ok": True}
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dangerous_tool_not_found_returns_error(self):
|
||||
"""Approved but tool not found → error result."""
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
engine._find_tool = lambda name, tools: None
|
||||
|
||||
handler = AsyncMock(return_value=True)
|
||||
should_exec, events, result = await engine._check_autonomy_gate(
|
||||
"rm", {"path": "/x"}, [], step=1, confirmation_handler=handler
|
||||
)
|
||||
|
||||
assert should_exec is False
|
||||
assert "error" in result
|
||||
assert "not found" in result["error"]
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dangerous_tool_execution_failure_returns_error(self):
|
||||
"""Approved but tool.safe_execute raises → error dict."""
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=DangerousToolsConfig(),
|
||||
autonomy_mode=True,
|
||||
)
|
||||
fake_tool = MagicMock()
|
||||
fake_tool.safe_execute = AsyncMock(side_effect=RuntimeError("boom"))
|
||||
engine._find_tool = lambda name, tools: fake_tool
|
||||
|
||||
handler = AsyncMock(return_value=True)
|
||||
should_exec, events, result = await engine._check_autonomy_gate(
|
||||
"rm", {"path": "/x"}, [fake_tool], step=1, confirmation_handler=handler
|
||||
)
|
||||
|
||||
assert should_exec is False
|
||||
assert "error" in result
|
||||
assert "tool_execution_failed" in result.get("error_code", "")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constructor wiring
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAutonomyConstructorWiring:
|
||||
"""Verify __init__ stores autonomy state correctly."""
|
||||
|
||||
def test_defaults_are_off(self):
|
||||
engine = ReActEngine(llm_gateway=MagicMock())
|
||||
assert engine._autonomy_mode is False
|
||||
assert engine._dangerous_tools_config is None
|
||||
|
||||
def test_autonomy_mode_stored(self):
|
||||
cfg = DangerousToolsConfig()
|
||||
engine = ReActEngine(
|
||||
llm_gateway=MagicMock(),
|
||||
dangerous_tools_config=cfg,
|
||||
autonomy_mode=True,
|
||||
)
|
||||
assert engine._autonomy_mode is True
|
||||
assert engine._dangerous_tools_config is cfg
|
||||
|
|
@ -0,0 +1,436 @@
|
|||
"""U5: ReflexionEngine reflection persistence to EpisodicMemory (R11, R15).
|
||||
|
||||
Covers:
|
||||
- store_prompt_reflection() stores via existing store() with task_type discriminator
|
||||
- search_prompt_reflections() filters by task_type="prompt_reflection"
|
||||
- cleanup_expired() deletes old records (TTL)
|
||||
- _reflect() persists reflection when episodic_memory configured (non-blocking)
|
||||
- _reflect() skips persistence when episodic_memory=None (backward compat)
|
||||
- Persistence failure does not block _reflect (returns reflection text)
|
||||
- Multi-version coexistence (same task_hash, different versions)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from agentkit.core.react import ReActResult
|
||||
from agentkit.core.reflexion import ReflexionEngine
|
||||
from agentkit.memory.episodic import EpisodicMemory
|
||||
|
||||
|
||||
# ── Helpers ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _make_episodic_memory_mock() -> MagicMock:
|
||||
"""Create a mock EpisodicMemory that tracks store/search calls."""
|
||||
mem = MagicMock(spec=EpisodicMemory)
|
||||
mem.store = AsyncMock(return_value=None)
|
||||
mem.search = AsyncMock(return_value=[])
|
||||
mem.store_prompt_reflection = AsyncMock(return_value="prompt_reflection:abc:1")
|
||||
mem.search_prompt_reflections = AsyncMock(return_value=[])
|
||||
mem.cleanup_expired = AsyncMock(return_value=0)
|
||||
return mem
|
||||
|
||||
|
||||
def _make_react_result(output: str = "test output", status: str = "completed") -> ReActResult:
|
||||
return ReActResult(
|
||||
output=output,
|
||||
trajectory=[],
|
||||
total_steps=1,
|
||||
total_tokens=10,
|
||||
status=status,
|
||||
)
|
||||
|
||||
|
||||
def _make_llm_gateway_mock(reflection_text: str = "reflection text") -> MagicMock:
|
||||
"""Gateway that returns reflection text from chat()."""
|
||||
gw = MagicMock()
|
||||
response = MagicMock()
|
||||
response.content = reflection_text
|
||||
gw.chat = AsyncMock(return_value=response)
|
||||
return gw
|
||||
|
||||
|
||||
# ── EpisodicMemory.store_prompt_reflection ─────────────────────────────
|
||||
|
||||
|
||||
class TestStorePromptReflection:
|
||||
"""U5/R11: store_prompt_reflection persists via direct ORM write with
|
||||
task_type='prompt_reflection' discriminator and metadata_ JSONB."""
|
||||
|
||||
def _make_persistable_mock(self) -> MagicMock:
|
||||
"""Build an EpisodicMemory mock with _embedder + _session_factory
|
||||
configured so store_prompt_reflection can run its real code path."""
|
||||
mem = MagicMock(spec=EpisodicMemory)
|
||||
mem._embedder = None # skip embedding generation in test
|
||||
mock_db = AsyncMock()
|
||||
mock_db.add = MagicMock()
|
||||
mock_db.commit = AsyncMock()
|
||||
mock_db.rollback = AsyncMock()
|
||||
mem._session_factory = MagicMock()
|
||||
mem._session_factory.return_value.__aenter__ = AsyncMock(return_value=mock_db)
|
||||
mem._session_factory.return_value.__aexit__ = AsyncMock(return_value=None)
|
||||
mem._episodic_model = MagicMock()
|
||||
# Track the ORM entry passed to db.add for assertions
|
||||
mem._mock_db = mock_db
|
||||
return mem
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_store_writes_orm_entry_with_correct_fields(self):
|
||||
mem = self._make_persistable_mock()
|
||||
await EpisodicMemory.store_prompt_reflection(
|
||||
mem,
|
||||
task_input="test task",
|
||||
reflection="reflection text",
|
||||
improved_prompt="improved prompt",
|
||||
version=1,
|
||||
score=0.5,
|
||||
agent_name="test_agent",
|
||||
)
|
||||
|
||||
# Verify ORM entry created via model constructor + db.add + commit
|
||||
mem._episodic_model.assert_called_once()
|
||||
call_kwargs = mem._episodic_model.call_args.kwargs
|
||||
assert call_kwargs["task_type"] == "prompt_reflection"
|
||||
assert call_kwargs["agent_name"] == "test_agent"
|
||||
assert call_kwargs["input_summary"] == "test task"
|
||||
assert call_kwargs["output_summary"] == "improved prompt"
|
||||
assert call_kwargs["reflection"] == "reflection text"
|
||||
assert call_kwargs["quality_score"] == 0.5
|
||||
assert call_kwargs["outcome"] == "reflection"
|
||||
# metadata_ JSONB carries task_hash + version + score
|
||||
assert "task_hash" in call_kwargs["metadata_"]
|
||||
assert call_kwargs["metadata_"]["version"] == 1
|
||||
assert call_kwargs["metadata_"]["score"] == 0.5
|
||||
mem._mock_db.add.assert_called_once()
|
||||
mem._mock_db.commit.assert_awaited_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_store_returns_key_on_success(self):
|
||||
mem = self._make_persistable_mock()
|
||||
key = await EpisodicMemory.store_prompt_reflection(
|
||||
mem,
|
||||
task_input="test",
|
||||
reflection="r",
|
||||
improved_prompt="p",
|
||||
version=2,
|
||||
)
|
||||
assert key is not None
|
||||
assert ":2" in key
|
||||
assert key.startswith("prompt_reflection:")
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_store_returns_none_on_failure(self):
|
||||
from sqlalchemy.exc import DBAPIError
|
||||
|
||||
mem = self._make_persistable_mock()
|
||||
mem._mock_db.commit = AsyncMock(
|
||||
side_effect=DBAPIError("stmt", params={}, orig=Exception("db down"))
|
||||
)
|
||||
key = await EpisodicMemory.store_prompt_reflection(
|
||||
mem,
|
||||
task_input="test",
|
||||
reflection="r",
|
||||
improved_prompt="p",
|
||||
)
|
||||
assert key is None
|
||||
mem._mock_db.rollback.assert_awaited_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_store_uses_provided_task_hash(self):
|
||||
mem = self._make_persistable_mock()
|
||||
key = await EpisodicMemory.store_prompt_reflection(
|
||||
mem,
|
||||
task_input="test",
|
||||
reflection="r",
|
||||
improved_prompt="p",
|
||||
task_hash="custom_hash",
|
||||
)
|
||||
assert "custom_hash" in key
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_store_generates_task_hash_from_input(self):
|
||||
mem = self._make_persistable_mock()
|
||||
key1 = await EpisodicMemory.store_prompt_reflection(
|
||||
mem, task_input="same task", reflection="r", improved_prompt="p"
|
||||
)
|
||||
key2 = await EpisodicMemory.store_prompt_reflection(
|
||||
mem, task_input="same task", reflection="r", improved_prompt="p"
|
||||
)
|
||||
# Same task_input → same hash prefix
|
||||
assert key1 == key2
|
||||
|
||||
|
||||
# ── EpisodicMemory.search_prompt_reflections ───────────────────────────
|
||||
|
||||
|
||||
class TestSearchPromptReflections:
|
||||
"""U5/R11: search_prompt_reflections filters by task_type."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_calls_underlying_search_with_filter(self):
|
||||
mem = MagicMock(spec=EpisodicMemory)
|
||||
mem.search = AsyncMock(return_value=[])
|
||||
await EpisodicMemory.search_prompt_reflections(mem, task_input="find similar", top_k=3)
|
||||
|
||||
mem.search.assert_awaited_once()
|
||||
call_kwargs = mem.search.await_args.kwargs
|
||||
assert call_kwargs["query"] == "find similar"
|
||||
assert call_kwargs["top_k"] == 3
|
||||
filters = call_kwargs["filters"]
|
||||
assert filters["task_type"] == "prompt_reflection"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_includes_agent_name_filter_when_provided(self):
|
||||
mem = MagicMock(spec=EpisodicMemory)
|
||||
mem.search = AsyncMock(return_value=[])
|
||||
await EpisodicMemory.search_prompt_reflections(mem, task_input="q", agent_name="agent_x")
|
||||
|
||||
filters = mem.search.await_args.kwargs["filters"]
|
||||
assert filters["agent_name"] == "agent_x"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_returns_empty_on_failure(self):
|
||||
from sqlalchemy.exc import DBAPIError
|
||||
|
||||
mem = MagicMock(spec=EpisodicMemory)
|
||||
mem.search = AsyncMock(side_effect=DBAPIError("stmt", params={}, orig=Exception("err")))
|
||||
result = await EpisodicMemory.search_prompt_reflections(mem, task_input="q")
|
||||
assert result == []
|
||||
|
||||
|
||||
# ── EpisodicMemory.cleanup_expired ─────────────────────────────────────
|
||||
|
||||
|
||||
class TestCleanupExpired:
|
||||
"""U5/R15: cleanup_expired deletes records older than max_age_days."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cleanup_calls_delete_with_cutoff(self):
|
||||
# Patch sqlalchemy.delete to bypass ORM model requirement — we only
|
||||
# verify the session.execute/commit calls happen.
|
||||
mock_db = AsyncMock()
|
||||
mock_result = MagicMock()
|
||||
mock_result.rowcount = 5
|
||||
mock_db.execute = AsyncMock(return_value=mock_result)
|
||||
mock_db.commit = AsyncMock()
|
||||
|
||||
mock_session_factory = MagicMock()
|
||||
mock_session_factory.return_value.__aenter__ = AsyncMock(return_value=mock_db)
|
||||
mock_session_factory.return_value.__aexit__ = AsyncMock(return_value=None)
|
||||
|
||||
# created_at must support `<` comparison with datetime — configure
|
||||
# __lt__ to return a truthy mock so `.where(...)` gets a valid arg.
|
||||
mock_model = MagicMock()
|
||||
mock_model.created_at = MagicMock()
|
||||
mock_model.created_at.__lt__ = MagicMock(return_value=MagicMock())
|
||||
|
||||
mem = EpisodicMemory(
|
||||
session_factory=mock_session_factory,
|
||||
episodic_model=mock_model,
|
||||
embedder=None,
|
||||
pgvector_enabled=False,
|
||||
)
|
||||
|
||||
# Patch sql_delete to return a chainable mock
|
||||
fake_stmt = MagicMock()
|
||||
fake_stmt.where = MagicMock(return_value=fake_stmt)
|
||||
with patch("sqlalchemy.delete", return_value=fake_stmt):
|
||||
deleted = await mem.cleanup_expired(max_age_days=30)
|
||||
|
||||
assert deleted == 5
|
||||
mock_db.execute.assert_awaited_once()
|
||||
mock_db.commit.assert_awaited_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cleanup_returns_zero_on_failure(self):
|
||||
from sqlalchemy.exc import DBAPIError
|
||||
|
||||
mock_db = AsyncMock()
|
||||
mock_db.execute = AsyncMock(
|
||||
side_effect=DBAPIError("stmt", params={}, orig=Exception("err"))
|
||||
)
|
||||
mock_db.rollback = AsyncMock()
|
||||
|
||||
mock_session_factory = MagicMock()
|
||||
mock_session_factory.return_value.__aenter__ = AsyncMock(return_value=mock_db)
|
||||
mock_session_factory.return_value.__aexit__ = AsyncMock(return_value=None)
|
||||
|
||||
mock_model = MagicMock()
|
||||
mock_model.created_at = MagicMock()
|
||||
mock_model.created_at.__lt__ = MagicMock(return_value=MagicMock())
|
||||
|
||||
mem = EpisodicMemory(
|
||||
session_factory=mock_session_factory,
|
||||
episodic_model=mock_model,
|
||||
embedder=None,
|
||||
pgvector_enabled=False,
|
||||
)
|
||||
|
||||
fake_stmt = MagicMock()
|
||||
fake_stmt.where = MagicMock(return_value=fake_stmt)
|
||||
with patch("sqlalchemy.delete", return_value=fake_stmt):
|
||||
deleted = await mem.cleanup_expired(max_age_days=30)
|
||||
|
||||
assert deleted == 0
|
||||
mock_db.rollback.assert_awaited_once()
|
||||
|
||||
|
||||
# ── ReflexionEngine._reflect persistence ──────────────────────────────
|
||||
|
||||
|
||||
class TestReflectPersistence:
|
||||
"""U5/R11: _reflect persists reflection when episodic_memory configured."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reflect_persists_when_episodic_memory_configured(self):
|
||||
episodic = _make_episodic_memory_mock()
|
||||
gw = _make_llm_gateway_mock(reflection_text="my reflection")
|
||||
engine = ReflexionEngine(
|
||||
llm_gateway=gw,
|
||||
episodic_memory=episodic,
|
||||
)
|
||||
|
||||
result = await engine._reflect(
|
||||
react_result=_make_react_result(),
|
||||
score=0.3,
|
||||
messages=[{"role": "user", "content": "test task"}],
|
||||
reflect_model="default",
|
||||
agent_name="test_agent",
|
||||
task_type="test",
|
||||
)
|
||||
|
||||
assert result == "my reflection"
|
||||
episodic.store_prompt_reflection.assert_awaited_once()
|
||||
call_kwargs = episodic.store_prompt_reflection.await_args.kwargs
|
||||
assert call_kwargs["task_input"] == "test task"
|
||||
assert call_kwargs["reflection"] == "my reflection"
|
||||
assert call_kwargs["score"] == 0.3
|
||||
assert call_kwargs["agent_name"] == "test_agent"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reflect_skips_persistence_when_no_episodic_memory(self):
|
||||
gw = _make_llm_gateway_mock(reflection_text="my reflection")
|
||||
engine = ReflexionEngine(
|
||||
llm_gateway=gw,
|
||||
episodic_memory=None, # No persistence
|
||||
)
|
||||
|
||||
result = await engine._reflect(
|
||||
react_result=_make_react_result(),
|
||||
score=0.3,
|
||||
messages=[{"role": "user", "content": "test task"}],
|
||||
reflect_model="default",
|
||||
agent_name="test_agent",
|
||||
task_type="test",
|
||||
)
|
||||
|
||||
assert result == "my reflection"
|
||||
# No episodic_memory → no store call
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reflect_persistence_failure_does_not_block(self):
|
||||
"""If store_prompt_reflection raises, _reflect still returns reflection."""
|
||||
episodic = MagicMock(spec=EpisodicMemory)
|
||||
episodic.store_prompt_reflection = AsyncMock(side_effect=RuntimeError("DB down"))
|
||||
gw = _make_llm_gateway_mock(reflection_text="important reflection")
|
||||
engine = ReflexionEngine(
|
||||
llm_gateway=gw,
|
||||
episodic_memory=episodic,
|
||||
)
|
||||
|
||||
result = await engine._reflect(
|
||||
react_result=_make_react_result(),
|
||||
score=0.3,
|
||||
messages=[{"role": "user", "content": "test task"}],
|
||||
reflect_model="default",
|
||||
agent_name="test_agent",
|
||||
task_type="test",
|
||||
)
|
||||
|
||||
# Reflection still returned despite persistence failure
|
||||
assert result == "important reflection"
|
||||
episodic.store_prompt_reflection.assert_awaited_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reflect_skips_persistence_when_llm_returns_none(self):
|
||||
"""If LLM returns empty/None, no persistence attempted."""
|
||||
episodic = _make_episodic_memory_mock()
|
||||
gw = MagicMock()
|
||||
response = MagicMock()
|
||||
response.content = None # Empty reflection
|
||||
gw.chat = AsyncMock(return_value=response)
|
||||
engine = ReflexionEngine(
|
||||
llm_gateway=gw,
|
||||
episodic_memory=episodic,
|
||||
)
|
||||
|
||||
result = await engine._reflect(
|
||||
react_result=_make_react_result(),
|
||||
score=0.3,
|
||||
messages=[{"role": "user", "content": "test task"}],
|
||||
reflect_model="default",
|
||||
agent_name="test_agent",
|
||||
task_type="test",
|
||||
)
|
||||
|
||||
assert result is None
|
||||
episodic.store_prompt_reflection.assert_not_awaited()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reflect_skips_persistence_on_llm_failure(self):
|
||||
"""If LLM call raises, no persistence attempted."""
|
||||
episodic = _make_episodic_memory_mock()
|
||||
gw = MagicMock()
|
||||
gw.chat = AsyncMock(side_effect=RuntimeError("LLM down"))
|
||||
engine = ReflexionEngine(
|
||||
llm_gateway=gw,
|
||||
episodic_memory=episodic,
|
||||
)
|
||||
|
||||
result = await engine._reflect(
|
||||
react_result=_make_react_result(),
|
||||
score=0.3,
|
||||
messages=[{"role": "user", "content": "test task"}],
|
||||
reflect_model="default",
|
||||
agent_name="test_agent",
|
||||
task_type="test",
|
||||
)
|
||||
|
||||
assert result is None
|
||||
episodic.store_prompt_reflection.assert_not_awaited()
|
||||
|
||||
|
||||
# ── Multi-version coexistence ─────────────────────────────────────────
|
||||
|
||||
|
||||
class TestMultiVersionCoexistence:
|
||||
"""U5/R11: same task_hash with different versions all stored."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_multiple_versions_stored_with_incrementing_version(self):
|
||||
# Reuse the persistable mock helper from TestStorePromptReflection.
|
||||
helper = TestStorePromptReflection()
|
||||
mem = helper._make_persistable_mock()
|
||||
|
||||
# Store v1, v2, v3 for same task
|
||||
key1 = await EpisodicMemory.store_prompt_reflection(
|
||||
mem, task_input="same task", reflection="r1", improved_prompt="p1", version=1
|
||||
)
|
||||
key2 = await EpisodicMemory.store_prompt_reflection(
|
||||
mem, task_input="same task", reflection="r2", improved_prompt="p2", version=2
|
||||
)
|
||||
key3 = await EpisodicMemory.store_prompt_reflection(
|
||||
mem, task_input="same task", reflection="r3", improved_prompt="p3", version=3
|
||||
)
|
||||
|
||||
# All keys have same task_hash prefix but different version suffix
|
||||
assert key1 != key2 != key3
|
||||
assert ":1" in key1 and ":2" in key2 and ":3" in key3
|
||||
|
||||
# All three ORM entries created via db.add
|
||||
assert mem._mock_db.add.call_count == 3
|
||||
assert mem._mock_db.commit.await_count == 3
|
||||
|
|
@ -6,7 +6,13 @@ from pathlib import Path
|
|||
|
||||
import pytest
|
||||
|
||||
from agentkit.server.config import ServerConfig, find_config_path, _resolve_env_vars, _deep_resolve
|
||||
from agentkit.server.config import (
|
||||
ServerConfig,
|
||||
DangerousToolsConfig,
|
||||
find_config_path,
|
||||
_resolve_env_vars,
|
||||
_deep_resolve,
|
||||
)
|
||||
|
||||
|
||||
class TestEnvVarResolution:
|
||||
|
|
@ -178,7 +184,9 @@ prompt:
|
|||
# Update yaml_content with absolute path
|
||||
yaml_content_updated = yaml_content.replace("./skills", str(skills_dir))
|
||||
|
||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False, dir=tmpdir) as f:
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".yaml", delete=False, dir=tmpdir
|
||||
) as f:
|
||||
f.write(yaml_content_updated)
|
||||
f.flush()
|
||||
config = ServerConfig.from_yaml(f.name)
|
||||
|
|
@ -209,7 +217,9 @@ skills:
|
|||
paths:
|
||||
- "{skill_yaml}"
|
||||
"""
|
||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False, dir=tmpdir) as f:
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".yaml", delete=False, dir=tmpdir
|
||||
) as f:
|
||||
f.write(yaml_content)
|
||||
f.flush()
|
||||
config = ServerConfig.from_yaml(f.name)
|
||||
|
|
@ -246,7 +256,9 @@ skills:
|
|||
paths:
|
||||
- "{skills_dir}"
|
||||
"""
|
||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False, dir=tmpdir) as f:
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".yaml", delete=False, dir=tmpdir
|
||||
) as f:
|
||||
f.write(yaml_content)
|
||||
f.flush()
|
||||
config = ServerConfig.from_yaml(f.name)
|
||||
|
|
@ -444,3 +456,103 @@ class TestConfigHotReload:
|
|||
assert config.port == 9000
|
||||
|
||||
os.unlink(config_path)
|
||||
|
||||
|
||||
class TestDangerousToolsConfig:
|
||||
"""Test DangerousToolsConfig — U1: dangerous tool whitelist for PLAN_EXEC autonomy (KTD2)."""
|
||||
|
||||
def test_defaults_when_no_data(self):
|
||||
cfg = DangerousToolsConfig.from_dict(None)
|
||||
assert cfg.enabled is True
|
||||
assert len(cfg.tool_patterns) > 0
|
||||
# Default whitelist includes rm, deploy, kubectl, etc.
|
||||
assert cfg.is_dangerous("rm")
|
||||
assert cfg.is_dangerous("deploy")
|
||||
assert cfg.is_dangerous("kubectl_apply")
|
||||
|
||||
def test_defaults_when_empty_dict(self):
|
||||
cfg = DangerousToolsConfig.from_dict({})
|
||||
assert cfg.enabled is True
|
||||
assert cfg.is_dangerous("rmdir")
|
||||
|
||||
def test_dangerous_tool_match(self):
|
||||
cfg = DangerousToolsConfig()
|
||||
assert cfg.is_dangerous("rm") is True
|
||||
assert cfg.is_dangerous("rmdir") is True
|
||||
assert cfg.is_dangerous("deploy") is True
|
||||
assert cfg.is_dangerous("kubectl_rollout") is True
|
||||
assert cfg.is_dangerous("helm_upgrade") is True
|
||||
assert cfg.is_dangerous("git_push_force") is True
|
||||
assert cfg.is_dangerous("alembic_upgrade") is True
|
||||
assert cfg.is_dangerous("migrate_db") is True
|
||||
assert cfg.is_dangerous("payment_charge") is True
|
||||
|
||||
def test_non_dangerous_tool(self):
|
||||
cfg = DangerousToolsConfig()
|
||||
assert cfg.is_dangerous("read_file") is False
|
||||
assert cfg.is_dangerous("write_file") is False
|
||||
assert cfg.is_dangerous("search") is False
|
||||
assert cfg.is_dangerous("list_dir") is False
|
||||
|
||||
def test_payment_glob_pattern(self):
|
||||
cfg = DangerousToolsConfig()
|
||||
# payment_* pattern matches any tool starting with payment_
|
||||
assert cfg.is_dangerous("payment_refund") is True
|
||||
assert cfg.is_dangerous("payment_transfer") is True
|
||||
|
||||
def test_disabled_returns_false(self):
|
||||
cfg = DangerousToolsConfig(enabled=False)
|
||||
assert cfg.is_dangerous("rm") is False
|
||||
assert cfg.is_dangerous("deploy") is False
|
||||
|
||||
def test_custom_patterns_from_dict(self):
|
||||
data = {"enabled": True, "tool_patterns": [r"^custom_dangerous"]}
|
||||
cfg = DangerousToolsConfig.from_dict(data)
|
||||
assert cfg.is_dangerous("custom_dangerous_op") is True
|
||||
assert cfg.is_dangerous("rm") is False # default patterns overridden
|
||||
|
||||
def test_enabled_false_from_dict(self):
|
||||
data = {"enabled": False}
|
||||
cfg = DangerousToolsConfig.from_dict(data)
|
||||
assert cfg.enabled is False
|
||||
assert cfg.is_dangerous("rm") is False
|
||||
|
||||
def test_server_config_loads_dangerous_tools(self):
|
||||
yaml_content = """
|
||||
server:
|
||||
host: "0.0.0.0"
|
||||
port: 8001
|
||||
|
||||
dangerous_tools:
|
||||
enabled: true
|
||||
tool_patterns:
|
||||
- "^rm$"
|
||||
- "^kubectl"
|
||||
"""
|
||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False) as f:
|
||||
f.write(yaml_content)
|
||||
f.flush()
|
||||
config = ServerConfig.from_yaml(f.name)
|
||||
|
||||
assert config.dangerous_tools.enabled is True
|
||||
assert config.dangerous_tools.is_dangerous("rm") is True
|
||||
assert config.dangerous_tools.is_dangerous("kubectl_apply") is True
|
||||
# Custom patterns override defaults
|
||||
assert config.dangerous_tools.is_dangerous("deploy") is False
|
||||
os.unlink(f.name)
|
||||
|
||||
def test_server_config_defaults_dangerous_tools_when_missing(self):
|
||||
yaml_content = """
|
||||
server:
|
||||
host: "0.0.0.0"
|
||||
port: 8001
|
||||
"""
|
||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False) as f:
|
||||
f.write(yaml_content)
|
||||
f.flush()
|
||||
config = ServerConfig.from_yaml(f.name)
|
||||
|
||||
# Missing config section → defaults applied
|
||||
assert config.dangerous_tools.enabled is True
|
||||
assert config.dangerous_tools.is_dangerous("rm") is True
|
||||
os.unlink(f.name)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,464 @@
|
|||
"""U4: TeamOrchestrator 无依赖子任务并行模式单元测试 (R1-R5).
|
||||
|
||||
Covers:
|
||||
- TeamPlan.get_independent_subtasks() introspection
|
||||
- _rebalance_independent_subtasks() MAX_EXPERTS enforcement (R4)
|
||||
- topological_sort 同层并行 (existing behavior, validated here for U4 contract)
|
||||
- SharedWorkspace 路径唯一性 (phase_id is UUID → no collision)
|
||||
- 综合阶段等待所有并行子任务完成 (layer-sequential contract)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from agentkit.core.handoff_transport import InProcessHandoffTransport
|
||||
from agentkit.core.protocol import TaskResult, TaskStatus
|
||||
from agentkit.experts.config import ExpertConfig
|
||||
from agentkit.experts.expert import Expert
|
||||
from agentkit.experts.orchestrator import TeamOrchestrator
|
||||
from agentkit.experts.plan import PlanPhase, TeamPlan
|
||||
from agentkit.experts.team import ExpertTeam
|
||||
|
||||
from tests.unit.experts._helpers import (
|
||||
make_chat_stream_mock,
|
||||
make_execute_stream_mock,
|
||||
)
|
||||
|
||||
|
||||
# ── Helpers ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _make_expert_config(name: str, is_lead: bool = False) -> ExpertConfig:
|
||||
return ExpertConfig(
|
||||
name=name,
|
||||
agent_type="expert",
|
||||
persona="测试专家",
|
||||
thinking_style="逻辑推理",
|
||||
bound_skills=["skill_a"],
|
||||
is_lead=is_lead,
|
||||
task_mode="llm_generate",
|
||||
prompt={"identity": "测试"},
|
||||
llm={"model": "default"},
|
||||
)
|
||||
|
||||
|
||||
def _make_mock_expert(name: str, is_lead: bool = False) -> MagicMock:
|
||||
config = _make_expert_config(name=name, is_lead=is_lead)
|
||||
expert = MagicMock(spec=Expert)
|
||||
expert.config = config
|
||||
expert.is_active = True
|
||||
expert.team_id = None
|
||||
expert.get_capabilities_summary.return_value = {
|
||||
"name": name,
|
||||
"persona": config.persona,
|
||||
"thinking_style": config.thinking_style,
|
||||
"bound_skills": config.bound_skills,
|
||||
"is_lead": is_lead,
|
||||
}
|
||||
mock_agent = MagicMock()
|
||||
mock_agent.execute = AsyncMock(
|
||||
return_value=TaskResult(
|
||||
task_id="test",
|
||||
agent_name=name,
|
||||
status=TaskStatus.COMPLETED.value,
|
||||
output_data={"content": f"Result from {name}"},
|
||||
error_message=None,
|
||||
started_at=None,
|
||||
completed_at=None,
|
||||
)
|
||||
)
|
||||
mock_agent.execute_stream = make_execute_stream_mock(f"Result from {name}")
|
||||
mock_agent._llm_gateway = None
|
||||
expert.agent = mock_agent
|
||||
return expert
|
||||
|
||||
|
||||
def _make_team_with_experts(
|
||||
expert_names: list[str] | None = None,
|
||||
lead_name: str = "lead",
|
||||
) -> ExpertTeam:
|
||||
team = ExpertTeam()
|
||||
transport = AsyncMock(spec=InProcessHandoffTransport)
|
||||
team._handoff_transport = transport
|
||||
if expert_names is None:
|
||||
expert_names = [lead_name, "member1", "member2"]
|
||||
for name in expert_names:
|
||||
is_lead = name == lead_name
|
||||
expert = _make_mock_expert(name=name, is_lead=is_lead)
|
||||
team._experts[name] = expert
|
||||
if is_lead:
|
||||
team._lead_expert_name = name
|
||||
return team
|
||||
|
||||
|
||||
def _make_phase(
|
||||
id: str, name: str, assigned_expert: str, depends_on: list[str] | None = None
|
||||
) -> PlanPhase:
|
||||
return PlanPhase(
|
||||
id=id,
|
||||
name=name,
|
||||
assigned_expert=assigned_expert,
|
||||
task_description=f"task for {name}",
|
||||
depends_on=depends_on or [],
|
||||
)
|
||||
|
||||
|
||||
def _make_mock_llm_gateway_with_phases(
|
||||
phases_responses: list[list[dict]],
|
||||
synthesis_content: str = "综合结果",
|
||||
) -> MagicMock:
|
||||
"""Gateway that returns successive decompositions then synthesis.
|
||||
|
||||
phases_responses: each element is a list of phase dicts to return on
|
||||
successive decomposition calls. After exhausted, returns synthesis.
|
||||
"""
|
||||
gateway = AsyncMock()
|
||||
decomp_responses = [MagicMock(content=json.dumps(phases)) for phases in phases_responses]
|
||||
synth_response = MagicMock(content=synthesis_content)
|
||||
call_count = [0]
|
||||
|
||||
async def chat_side_effect(messages, model=None, **kwargs):
|
||||
call_count[0] += 1
|
||||
if call_count[0] <= len(decomp_responses):
|
||||
return decomp_responses[call_count[0] - 1]
|
||||
return synth_response
|
||||
|
||||
gateway.chat = AsyncMock(side_effect=chat_side_effect)
|
||||
gateway.chat_stream = make_chat_stream_mock(synthesis_content)
|
||||
return gateway
|
||||
|
||||
|
||||
# ── TeamPlan.get_independent_subtasks ─────────────────────────────────
|
||||
|
||||
|
||||
class TestGetIndependentSubtasks:
|
||||
"""U4/R1: TeamPlan.get_independent_subtasks() returns phases with no
|
||||
depends_on."""
|
||||
|
||||
def test_returns_empty_for_no_phases(self):
|
||||
plan = TeamPlan(task="test", lead_expert="lead")
|
||||
assert plan.get_independent_subtasks() == []
|
||||
|
||||
def test_returns_only_independent_phases(self):
|
||||
plan = TeamPlan(
|
||||
task="test",
|
||||
lead_expert="lead",
|
||||
phases=[
|
||||
_make_phase("p1", "A", "lead", depends_on=[]),
|
||||
_make_phase("p2", "B", "member1", depends_on=["p1"]),
|
||||
_make_phase("p3", "C", "member2", depends_on=[]),
|
||||
_make_phase("p4", "D", "member1", depends_on=["p1", "p3"]),
|
||||
],
|
||||
)
|
||||
independent = plan.get_independent_subtasks()
|
||||
assert len(independent) == 2
|
||||
assert {ph.id for ph in independent} == {"p1", "p3"}
|
||||
|
||||
def test_returns_all_when_no_dependencies(self):
|
||||
plan = TeamPlan(
|
||||
task="test",
|
||||
lead_expert="lead",
|
||||
phases=[
|
||||
_make_phase("p1", "A", "lead", depends_on=[]),
|
||||
_make_phase("p2", "B", "member1", depends_on=[]),
|
||||
_make_phase("p3", "C", "member2", depends_on=[]),
|
||||
],
|
||||
)
|
||||
independent = plan.get_independent_subtasks()
|
||||
assert len(independent) == 3
|
||||
|
||||
|
||||
# ── topological_sort 同层并行 contract (U4/R1, R3) ────────────────────
|
||||
|
||||
|
||||
class TestParallelLayerContract:
|
||||
"""U4/R1, R3: independent subtasks land in layer 0 (parallel); synthesis
|
||||
waits for all layers to complete (layer-sequential)."""
|
||||
|
||||
def test_three_independent_subtasks_in_layer_0(self):
|
||||
plan = TeamPlan(
|
||||
task="test",
|
||||
lead_expert="lead",
|
||||
phases=[
|
||||
_make_phase("p1", "A", "lead", depends_on=[]),
|
||||
_make_phase("p2", "B", "member1", depends_on=[]),
|
||||
_make_phase("p3", "C", "member2", depends_on=[]),
|
||||
],
|
||||
)
|
||||
layers = plan.topological_sort()
|
||||
assert len(layers) == 1
|
||||
assert len(layers[0]) == 3
|
||||
assert {ph.id for ph in layers[0]} == {"p1", "p2", "p3"}
|
||||
|
||||
def test_shared_dependency_degrades_to_layered(self):
|
||||
"""R3: subtasks sharing an upstream dependency are NOT parallel with
|
||||
that upstream — they land in later layers."""
|
||||
plan = TeamPlan(
|
||||
task="test",
|
||||
lead_expert="lead",
|
||||
phases=[
|
||||
_make_phase("p1", "Base", "lead", depends_on=[]),
|
||||
_make_phase("p2", "A", "member1", depends_on=["p1"]),
|
||||
_make_phase("p3", "B", "member2", depends_on=["p1"]),
|
||||
],
|
||||
)
|
||||
layers = plan.topological_sort()
|
||||
assert len(layers) == 2
|
||||
assert len(layers[0]) == 1 # only Base
|
||||
assert layers[0][0].id == "p1"
|
||||
assert len(layers[1]) == 2 # A and B parallel (same layer)
|
||||
assert {ph.id for ph in layers[1]} == {"p2", "p3"}
|
||||
|
||||
def test_synthesis_waits_for_all_layers(self):
|
||||
"""R3: dependent phase in last layer — synthesis cannot run until
|
||||
its layer completes (validated via topological_sort layer count)."""
|
||||
plan = TeamPlan(
|
||||
task="test",
|
||||
lead_expert="lead",
|
||||
phases=[
|
||||
_make_phase("p1", "A", "lead", depends_on=[]),
|
||||
_make_phase("p2", "B", "member1", depends_on=["p1"]),
|
||||
_make_phase("p3", "C", "member2", depends_on=["p2"]),
|
||||
],
|
||||
)
|
||||
layers = plan.topological_sort()
|
||||
assert len(layers) == 3 # strict sequential — no parallelism
|
||||
|
||||
|
||||
# ── SharedWorkspace 路径唯一性 (U4/R2) ─────────────────────────────────
|
||||
|
||||
|
||||
class TestSharedWorkspacePathUniqueness:
|
||||
"""U4/R2: parallel subtask output paths must not collide. PlanPhase.id
|
||||
is UUID by default → unique per phase, ensuring
|
||||
``{plan_id}/phase/{phase_id}/output`` never overlaps."""
|
||||
|
||||
def test_default_phase_ids_are_unique(self):
|
||||
phases = [
|
||||
PlanPhase(name=f"phase_{i}", assigned_expert="lead", task_description="t")
|
||||
for i in range(5)
|
||||
]
|
||||
ids = [ph.id for ph in phases]
|
||||
assert len(set(ids)) == len(ids), "phase_ids must be unique"
|
||||
|
||||
def test_independent_subtasks_have_unique_output_paths(self):
|
||||
plan = TeamPlan(
|
||||
task="test",
|
||||
lead_expert="lead",
|
||||
phases=[
|
||||
PlanPhase(name="A", assigned_expert="lead", task_description="t"),
|
||||
PlanPhase(name="B", assigned_expert="member1", task_description="t"),
|
||||
PlanPhase(name="C", assigned_expert="member2", task_description="t"),
|
||||
],
|
||||
)
|
||||
independent = plan.get_independent_subtasks()
|
||||
paths = {f"{plan.id}/phase/{ph.id}/output" for ph in independent}
|
||||
assert len(paths) == len(independent)
|
||||
|
||||
|
||||
# ── _rebalance_independent_subtasks (U4/R4 MAX_EXPERTS) ────────────────
|
||||
|
||||
|
||||
class TestRebalanceIndependentSubtasks:
|
||||
"""U4/R4: when Lead over-decomposes independent subtasks beyond
|
||||
MAX_INDEPENDENT_SUBTASKS, request re-decompose with merge hint."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_rebalance_when_within_limit(self):
|
||||
"""5 independent subtasks ≤ 10 → no re-decompose."""
|
||||
team = _make_team_with_experts()
|
||||
orchestrator = TeamOrchestrator(team)
|
||||
# 5 independent phases
|
||||
phases = [_make_phase(f"p{i}", f"phase_{i}", "lead", depends_on=[]) for i in range(5)]
|
||||
original_phases = list(phases)
|
||||
|
||||
result = await orchestrator._rebalance_independent_subtasks(
|
||||
lead=team.lead_expert, task="test", phases=phases
|
||||
)
|
||||
assert result is original_phases or result == original_phases
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rebalance_triggers_when_over_limit(self):
|
||||
"""11 independent subtasks > 10 → re-decompose with merge hint."""
|
||||
team = _make_team_with_experts()
|
||||
orchestrator = TeamOrchestrator(team)
|
||||
|
||||
# Original: 11 independent (over limit)
|
||||
original_phases = [
|
||||
_make_phase(f"p{i}", f"phase_{i}", "lead", depends_on=[]) for i in range(11)
|
||||
]
|
||||
# Re-decomposed: 8 independent (within limit) — merge succeeded
|
||||
gateway = _make_mock_llm_gateway_with_phases(
|
||||
phases_responses=[
|
||||
[ # first call (original _decompose_task already happened —
|
||||
# but _rebalance calls _decompose_task again, returning this)
|
||||
{
|
||||
"name": f"merged_{i}",
|
||||
"assigned_expert": "lead",
|
||||
"task_description": "t",
|
||||
"depends_on": [],
|
||||
}
|
||||
for i in range(8)
|
||||
],
|
||||
]
|
||||
)
|
||||
team._experts["lead"].agent._llm_gateway = gateway
|
||||
|
||||
result = await orchestrator._rebalance_independent_subtasks(
|
||||
lead=team.lead_expert, task="test", phases=original_phases
|
||||
)
|
||||
# Should return the re-decomposed phases (8 independent)
|
||||
assert len(result) == 8
|
||||
assert all(not ph.depends_on for ph in result)
|
||||
# Original phases should NOT be returned
|
||||
assert {ph.id for ph in result}.isdisjoint({ph.id for ph in original_phases})
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rebalance_no_gateway_returns_original(self):
|
||||
"""No LLM gateway → can't re-decompose, return original."""
|
||||
team = _make_team_with_experts()
|
||||
orchestrator = TeamOrchestrator(team)
|
||||
# gateway is None by default in _make_mock_expert
|
||||
original_phases = [
|
||||
_make_phase(f"p{i}", f"phase_{i}", "lead", depends_on=[]) for i in range(11)
|
||||
]
|
||||
|
||||
result = await orchestrator._rebalance_independent_subtasks(
|
||||
lead=team.lead_expert, task="test", phases=original_phases
|
||||
)
|
||||
assert result is original_phases
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rebalance_still_over_limit_returns_new_anyway(self):
|
||||
"""Re-decompose still produces >MAX independent → return new phases
|
||||
(MAX_PHASES truncation handles it in execute())."""
|
||||
team = _make_team_with_experts()
|
||||
orchestrator = TeamOrchestrator(team)
|
||||
|
||||
original_phases = [
|
||||
_make_phase(f"p{i}", f"phase_{i}", "lead", depends_on=[]) for i in range(11)
|
||||
]
|
||||
# Re-decompose returns 12 independent (still over)
|
||||
gateway = _make_mock_llm_gateway_with_phases(
|
||||
phases_responses=[
|
||||
[
|
||||
{
|
||||
"name": f"q{i}",
|
||||
"assigned_expert": "lead",
|
||||
"task_description": "t",
|
||||
"depends_on": [],
|
||||
}
|
||||
for i in range(12)
|
||||
],
|
||||
]
|
||||
)
|
||||
team._experts["lead"].agent._llm_gateway = gateway
|
||||
|
||||
result = await orchestrator._rebalance_independent_subtasks(
|
||||
lead=team.lead_expert, task="test", phases=original_phases
|
||||
)
|
||||
# Should return the 12-phase re-decomposition (truncation happens later)
|
||||
assert len(result) == 12
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rebalance_decompose_failure_returns_original(self):
|
||||
"""Re-decompose raises LLMProviderError → return original phases."""
|
||||
from agentkit.core.exceptions import LLMProviderError
|
||||
|
||||
team = _make_team_with_experts()
|
||||
orchestrator = TeamOrchestrator(team)
|
||||
|
||||
original_phases = [
|
||||
_make_phase(f"p{i}", f"phase_{i}", "lead", depends_on=[]) for i in range(11)
|
||||
]
|
||||
gateway = AsyncMock()
|
||||
gateway.chat = AsyncMock(side_effect=LLMProviderError("LLM down"))
|
||||
gateway.chat_stream = make_chat_stream_mock("err")
|
||||
team._experts["lead"].agent._llm_gateway = gateway
|
||||
|
||||
result = await orchestrator._rebalance_independent_subtasks(
|
||||
lead=team.lead_expert, task="test", phases=original_phases
|
||||
)
|
||||
assert result is original_phases
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rebalance_empty_redecompose_returns_original(self):
|
||||
"""Re-decompose returns empty list → return original phases."""
|
||||
team = _make_team_with_experts()
|
||||
orchestrator = TeamOrchestrator(team)
|
||||
|
||||
original_phases = [
|
||||
_make_phase(f"p{i}", f"phase_{i}", "lead", depends_on=[]) for i in range(11)
|
||||
]
|
||||
# Gateway returns malformed JSON → _parse_phases returns []
|
||||
gateway = AsyncMock()
|
||||
bad_response = MagicMock(content="not json")
|
||||
gateway.chat = AsyncMock(return_value=bad_response)
|
||||
gateway.chat_stream = make_chat_stream_mock("err")
|
||||
team._experts["lead"].agent._llm_gateway = gateway
|
||||
|
||||
result = await orchestrator._rebalance_independent_subtasks(
|
||||
lead=team.lead_expert, task="test", phases=original_phases
|
||||
)
|
||||
assert result is original_phases
|
||||
|
||||
|
||||
# ── Integration: full execute() with parallel subtasks ────────────────
|
||||
|
||||
|
||||
class TestExecuteParallelSubtasks:
|
||||
"""U4/R1, R3, R5: full execute() path with independent subtasks running
|
||||
in parallel + synthesis waits for all."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_three_parallel_subtasks_complete_then_synthesis(self):
|
||||
"""3 independent subtasks → all run in layer 0 → synthesis runs
|
||||
after all complete."""
|
||||
team = _make_team_with_experts()
|
||||
orchestrator = TeamOrchestrator(team)
|
||||
|
||||
gateway = _make_mock_llm_gateway_with_phases(
|
||||
phases_responses=[
|
||||
[
|
||||
{
|
||||
"name": "A",
|
||||
"assigned_expert": "lead",
|
||||
"task_description": "a",
|
||||
"depends_on": [],
|
||||
},
|
||||
{
|
||||
"name": "B",
|
||||
"assigned_expert": "member1",
|
||||
"task_description": "b",
|
||||
"depends_on": [],
|
||||
},
|
||||
{
|
||||
"name": "C",
|
||||
"assigned_expert": "member2",
|
||||
"task_description": "c",
|
||||
"depends_on": [],
|
||||
},
|
||||
],
|
||||
]
|
||||
)
|
||||
team._experts["lead"].agent._llm_gateway = gateway
|
||||
|
||||
result = await orchestrator.execute("并行任务")
|
||||
|
||||
assert result["status"] == "completed"
|
||||
plan: TeamPlan = result["plan"]
|
||||
layers = plan.topological_sort()
|
||||
assert len(layers) == 1 # all 3 in layer 0 (parallel)
|
||||
assert len(layers[0]) == 3
|
||||
|
||||
# All phases completed
|
||||
completed = plan.completed_phases
|
||||
assert len(completed) == 3
|
||||
|
||||
# phase_results has all 3
|
||||
phase_results: dict = result["phase_results"]
|
||||
assert len(phase_results) == 3
|
||||
Loading…
Reference in New Issue