AgentKit 能力基准测试报告
测试概要
- 时间: 2026-06-17T04:52:53.863927+00:00
- 版本: 0.1.0
- 模式: all
- 运行次数: 1
- 总体准确率: 95.2% ± 0.0%
与行业 Benchmark 对比
| Benchmark |
测试对象 |
AgentKit 对应 |
| SWE-bench |
LLM 代码修复 |
— (测 LLM 非框架) |
| ToolBench |
工具调用 |
tool_search 维度 |
| AgentBench |
Agent 系统 |
全部维度 |
维度结果
1. 预处理准确度 (Preprocessing Accuracy) [Mock]
| 指标 |
值 |
| Accuracy |
100.0% ± 0.0% |
| 95% CI |
[79.6%, 100.0%] |
| Precision |
100.0% |
| Recall |
100.0% |
| F1 |
100.0% |
| Latency p50 |
0.01ms |
| Latency p95 |
0.06ms |
| Latency p99 |
0.11ms |
| Consistency |
100.0% |
| Total / Pass / Fail |
15 / 15 / 0 |
按类别分布
| 类别 |
用例数 |
通过 |
准确率 |
| greeting |
4 |
4 |
100.0% |
| tool_query |
5 |
5 |
100.0% |
| skill_prefix |
3 |
3 |
100.0% |
| complex |
3 |
3 |
100.0% |
按难度分布
| 难度 |
用例数 |
通过 |
准确率 |
| easy |
5 |
5 |
100.0% |
| medium |
7 |
7 |
100.0% |
| hard |
3 |
3 |
100.0% |
2. 过拟合检测 (Overfitting Detection) [Mock]
| 指标 |
值 |
| Accuracy |
100.0% ± 0.0% |
| 95% CI |
[56.5%, 100.0%] |
| Precision |
100.0% |
| Recall |
100.0% |
| F1 |
100.0% |
| Latency p50 |
0.03ms |
| Latency p95 |
0.06ms |
| Latency p99 |
0.06ms |
| Consistency |
100.0% |
| Total / Pass / Fail |
5 / 5 / 0 |
按类别分布
| 类别 |
用例数 |
通过 |
准确率 |
| ip_check |
1 |
1 |
100.0% |
| search |
1 |
1 |
100.0% |
| greeting |
1 |
1 |
100.0% |
| tool_use |
1 |
1 |
100.0% |
| complex |
1 |
1 |
100.0% |
按难度分布
| 难度 |
用例数 |
通过 |
准确率 |
| medium |
3 |
3 |
100.0% |
| easy |
1 |
1 |
100.0% |
| hard |
1 |
1 |
100.0% |
3. 效率测试 (Efficiency) [Mock]
| 指标 |
值 |
| Accuracy |
100.0% ± 0.0% |
| 95% CI |
[56.5%, 100.0%] |
| Precision |
0.0% |
| Recall |
0.0% |
| F1 |
0.0% |
| Latency p50 |
0.33ms |
| Latency p95 |
0.62ms |
| Latency p99 |
0.66ms |
| Consistency |
100.0% |
| Total / Pass / Fail |
5 / 5 / 0 |
按类别分布
| 类别 |
用例数 |
通过 |
准确率 |
| preprocess_latency |
3 |
3 |
100.0% |
| tool_search_latency |
2 |
2 |
100.0% |
按难度分布
| 难度 |
用例数 |
通过 |
准确率 |
| easy |
2 |
2 |
100.0% |
| medium |
3 |
3 |
100.0% |
4. 工具搜索 (Tool Search) [Mock]
| 指标 |
值 |
| Accuracy |
100.0% ± 0.0% |
| 95% CI |
[72.2%, 100.0%] |
| Precision |
83.3% |
| Recall |
83.3% |
| F1 |
83.3% |
| Latency p50 |
0.02ms |
| Latency p95 |
0.03ms |
| Latency p99 |
0.03ms |
| Consistency |
100.0% |
| Total / Pass / Fail |
10 / 10 / 0 |
按类别分布
| 类别 |
用例数 |
通过 |
准确率 |
| exact_match |
5 |
5 |
100.0% |
| fuzzy_match |
2 |
2 |
100.0% |
| no_match |
2 |
2 |
100.0% |
| top_k |
1 |
1 |
100.0% |
按难度分布
| 难度 |
用例数 |
通过 |
准确率 |
| easy |
7 |
7 |
100.0% |
| medium |
3 |
3 |
100.0% |
5. 事件模型 (Event Model) [Mock]
| 指标 |
值 |
| Accuracy |
100.0% ± 0.0% |
| 95% CI |
[61.0%, 100.0%] |
| Precision |
0.0% |
| Recall |
0.0% |
| F1 |
0.0% |
| Latency p50 |
0.06ms |
| Latency p95 |
16.00ms |
| Latency p99 |
20.24ms |
| Consistency |
100.0% |
| Total / Pass / Fail |
6 / 6 / 0 |
按类别分布
| 类别 |
用例数 |
通过 |
准确率 |
| sq_lifecycle |
3 |
3 |
100.0% |
| eq_lifecycle |
3 |
3 |
100.0% |
按难度分布
| 难度 |
用例数 |
通过 |
准确率 |
| easy |
6 |
6 |
100.0% |
6. 规格管理 (Spec Management) [Mock]
| 指标 |
值 |
| Accuracy |
100.0% ± 0.0% |
| 95% CI |
[64.6%, 100.0%] |
| Precision |
0.0% |
| Recall |
0.0% |
| F1 |
0.0% |
| Latency p50 |
1.38ms |
| Latency p95 |
3.46ms |
| Latency p99 |
4.01ms |
| Consistency |
100.0% |
| Total / Pass / Fail |
7 / 7 / 0 |
按类别分布
| 类别 |
用例数 |
通过 |
准确率 |
| crud |
5 |
5 |
100.0% |
| edge |
2 |
2 |
100.0% |
按难度分布
| 难度 |
用例数 |
通过 |
准确率 |
| easy |
6 |
6 |
100.0% |
| medium |
1 |
1 |
100.0% |
7. 验证循环 (Verification Loop) [Mock]
| 指标 |
值 |
| Accuracy |
100.0% ± 0.0% |
| 95% CI |
[56.5%, 100.0%] |
| Precision |
0.0% |
| Recall |
0.0% |
| F1 |
0.0% |
| Latency p50 |
22.00ms |
| Latency p95 |
411.57ms |
| Latency p99 |
487.06ms |
| Consistency |
100.0% |
| Total / Pass / Fail |
5 / 5 / 0 |
按类别分布
| 类别 |
用例数 |
通过 |
准确率 |
| basic |
2 |
2 |
100.0% |
| retry |
1 |
1 |
100.0% |
| timeout |
1 |
1 |
100.0% |
| multi |
1 |
1 |
100.0% |
按难度分布
| 难度 |
用例数 |
通过 |
准确率 |
| easy |
2 |
2 |
100.0% |
| medium |
3 |
3 |
100.0% |
8. LLM 推理能力 (LLM Reasoning) [LLM]
| 指标 |
值 |
| Accuracy |
60.0% ± 0.0% |
| 95% CI |
[23.1%, 88.2%] |
| Precision |
0.0% |
| Recall |
0.0% |
| F1 |
0.0% |
| Latency p50 |
25149.49ms |
| Latency p95 |
30001.17ms |
| Latency p99 |
30001.23ms |
| Consistency |
100.0% |
| Total / Pass / Fail |
5 / 3 / 2 |
按类别分布
| 类别 |
用例数 |
通过 |
准确率 |
| intent_understanding |
1 |
1 |
100.0% |
| tool_selection |
1 |
1 |
100.0% |
| multi_step |
1 |
0 |
0.0% |
| code_generation |
1 |
1 |
100.0% |
| error_recovery |
1 |
0 |
0.0% |
按难度分布
| 难度 |
用例数 |
通过 |
准确率 |
| easy |
1 |
1 |
100.0% |
| medium |
2 |
2 |
100.0% |
| hard |
2 |
0 |
0.0% |
失败用例分析
| 用例 ID |
类别 |
难度 |
期望 |
实际 |
根因 |
| llm-003 |
multi_step |
hard |
react |
timeout |
timeout |
| llm-005 |
error_recovery |
hard |
react |
timeout |
timeout |
9. GUI 集成测试 (GUI Integration) [GUI]
| 指标 |
值 |
| Accuracy |
80.0% ± 0.0% |
| 95% CI |
[37.5%, 96.4%] |
| Precision |
80.0% |
| Recall |
80.0% |
| F1 |
80.0% |
| Latency p50 |
0.00ms |
| Latency p95 |
0.00ms |
| Latency p99 |
0.00ms |
| Consistency |
100.0% |
| Total / Pass / Fail |
5 / 4 / 1 |
按类别分布
| 类别 |
用例数 |
通过 |
准确率 |
| service_startup |
1 |
1 |
100.0% |
| api_availability |
2 |
2 |
100.0% |
| websocket |
1 |
0 |
0.0% |
| frontend |
1 |
1 |
100.0% |
按难度分布
| 难度 |
用例数 |
通过 |
准确率 |
| easy |
2 |
2 |
100.0% |
| medium |
2 |
2 |
100.0% |
| hard |
1 |
0 |
0.0% |
失败用例分析
| 用例 ID |
类别 |
难度 |
期望 |
实际 |
根因 |
| gui-004 |
websocket |
hard |
connected |
failed |
gui_failure |
基线对比
| 维度 |
基线准确率 |
当前准确率 |
变化 |
| preprocessing |
100.0% |
100.0% |
— |
| overfitting |
100.0% |
100.0% |
— |
| efficiency |
100.0% |
100.0% |
— |
| tool_search |
100.0% |
100.0% |
— |
| event_model |
100.0% |
100.0% |
— |
| spec_management |
100.0% |
100.0% |
— |
| verification |
100.0% |
100.0% |
— |
| llm_reasoning |
0.0% |
60.0% |
↑ |
| gui_integration |
0.0% |
80.0% |
↑ |
问题总结与改进建议
- verification: P95 延迟 411.57ms 较高，建议优化性能
- llm_reasoning: 准确率 60.0% 低于 90%，建议检查失败用例并优化
- llm_reasoning: P95 延迟 30001.17ms 较高，建议优化性能
- gui_integration: 准确率 80.0% 低于 90%，建议检查失败用例并优化