feat(bitable): add bitable companion service with full P0-P2 fixes
Bitable is a multi-dimensional table companion service that runs alongside the main AgentKit server. It provides structured data storage with formula fields, views, and ingestion pipelines. Major components: - Domain models (Pydantic v2): Table, Field, Record, View, RecalcTask - SQLAlchemy 2 async ORM with independent bitable PostgreSQL schema - Formula engine: AST parser, DAG, Kahn topological sort, safe eval - RecalcWorker: atomic task claiming (FOR UPDATE SKIP LOCKED), topo-order processing, stale-threshold reaper for crash recovery - REST API (/api/v1/bitable): tables, fields, records, views, files - BitableTool: agent-facing tool with batch chunking (500/batch) - CLI: agentkit bitable subcommands (create, list, import-excel, etc.) - Frontend: Vue 3 + vxe-table grid with field management, views, filters - Ingestion: Excel (openpyxl), database reflection, API collector Security fixes (ce-code-review P0 + ce-debug P1): - SQL injection prevention (field_id validation, parameterized queries) - IDOR protection (_check_table_ownership on all table-level endpoints) - SSRF prevention (URL scheme + private IP validation in parse_excel_url) - OOM prevention (streaming file upload, batch delete, batch insert) - Atomic recalc task claiming (FOR UPDATE SKIP LOCKED) - Formula engine cache invalidation on field changes - Composite cursor pagination for non-id sort orders - Batch upsert (eliminates N+1 queries) - Sync I/O offloaded to thread pool in async contexts - Internal token auth (X-Internal-Token, hmac.compare_digest) - PK unique index enforcement Test coverage: 88 unit tests (95 skipped without Docker)
This commit is contained in:
chiguyong
parent
567cbc9c9b
commit
bbbf9cd40a
|
|
@ -47,3 +47,8 @@ src/agentkit/server/static/
|
|||
|
||||
# Runtime data (auth DB, conversation DB, etc.)
|
||||
data/
|
||||
|
||||
# Agent skill tooling (local-only, not project code)
|
||||
.agents/
|
||||
.trae/
|
||||
.aider*
|
||||
|
|
|
|||
305
AGENTS.md
305
AGENTS.md
|
|
@ -1,203 +1,206 @@
|
|||
# Fischer AgentKit — Project Context
|
||||
# Fischer AgentKit — 项目上下文
|
||||
|
||||
## Rules
|
||||
## 规则
|
||||
|
||||
- Python >= 3.11, type hints required, `pydantic>=2.0` for all data models
|
||||
- Ruff for lint + format: `ruff check src/ && ruff format src/` (target py311, line-length 100)
|
||||
- Tests: `pytest` (asyncio_mode=auto), markers: `integration`, `redis`, `postgres`
|
||||
- Never use `any` type — use proper Pydantic models or `Unknown`
|
||||
- API key comparison must use `hmac.compare_digest` (constant-time)
|
||||
- Expert names validated with `_EXPERT_NAME_RE = re.compile(r"^[a-zA-Z0-9_-]{1,64}$")`
|
||||
- HandoffTransport queues bounded (`maxsize=1024`), close uses sentinel pattern
|
||||
- Frontend: Vue 3 + TypeScript + Ant Design Vue, Pinia stores, no `require()` calls
|
||||
- **Async generator safety**: Never use early `return` before the first `yield` in `async def` — use `return; yield` pattern instead (see `.trae/rules/project_rules.md`)
|
||||
- Python >= 3.11，必须使用类型注解，所有数据模型使用 `pydantic>=2.0`
|
||||
- 使用 Ruff 进行 lint 和格式化：`ruff check src/ && ruff format src/`（目标 py311，行宽 100）
|
||||
- 测试：`pytest`（asyncio\_mode=auto），标记：`integration`、`redis`、`postgres`
|
||||
- 禁止使用 `any` 类型 — 使用合适的 Pydantic 模型或 `Unknown`
|
||||
- API Key 比较必须使用 `hmac.compare_digest`（恒定时间比较）
|
||||
- 专家名称使用 `_EXPERT_NAME_RE = re.compile(r"^[a-zA-Z0-9_-]{1,64}$")` 校验
|
||||
- HandoffTransport 队列有界（`maxsize=1024`），关闭使用 sentinel 模式
|
||||
- 前端：Vue 3 + TypeScript + Ant Design Vue，Pinia stores，禁止 `require()` 调用
|
||||
- **异步生成器安全**：在 `async def` 中禁止在第一个 `yield` 之前使用 `return` — 改用 `return; yield` 模式（见 `.trae/rules/project_rules.md`）
|
||||
- 所有回复必须是中文
|
||||
|
||||
## Tech Stack
|
||||
## 技术栈
|
||||
|
||||
- **Backend**: Python 3.11+, FastAPI, Uvicorn, Pydantic v2, SQLAlchemy 2 (async)
|
||||
- **Frontend**: Vue 3, TypeScript, Vite 5, Ant Design Vue 4, Pinia, Vue Router 4
|
||||
- **Desktop**: Tauri 2.x (Rust shell + Python sidecar)
|
||||
- **Infra**: Redis (bus/cache/state), PostgreSQL + pgvector (episodic memory)
|
||||
- **CLI**: Typer + Rich
|
||||
- **Exact versions**: see `pyproject.toml` (Python), `package.json` (Node)
|
||||
- **后端**：Python 3.11+、FastAPI、Uvicorn、Pydantic v2、SQLAlchemy 2（async）
|
||||
- **前端**：Vue 3、TypeScript、Vite 5、Ant Design Vue 4、Pinia、Vue Router 4
|
||||
- **桌面端**：Tauri 2.x（Rust 外壳 + Python sidecar）
|
||||
- **基础设施**：Redis（总线/缓存/状态）、PostgreSQL + pgvector（情景记忆）
|
||||
- **CLI**：Typer + Rich
|
||||
- **精确版本**：见 `pyproject.toml`（Python）、`package.json`（Node）
|
||||
|
||||
## Commands
|
||||
## 命令
|
||||
|
||||
```bash
|
||||
# Backend
|
||||
pip install -e ".[dev]" # Install with dev deps
|
||||
agentkit gui --port 8002 # Web GUI (frontend + API)
|
||||
agentkit serve --port 8001 # API-only server
|
||||
agentkit chat # CLI interactive chat
|
||||
agentkit init # Generate agentkit.yaml
|
||||
agentkit version / doctor / usage # Utility commands
|
||||
agentkit task submit/status/list/cancel # Task management
|
||||
agentkit skill list/load/info # Skill management
|
||||
agentkit pair --name X # Generate API key for external system
|
||||
pytest # Run all tests
|
||||
pytest -m "not integration" # Unit tests only
|
||||
ruff check src/ && ruff format src/ # Lint + format
|
||||
# 后端
|
||||
pip install -e ".[dev]" # 安装开发依赖
|
||||
agentkit gui --port 8002 # Web GUI（前端 + API）
|
||||
agentkit serve --port 8001 # 仅 API 服务
|
||||
agentkit chat # CLI 交互式聊天
|
||||
agentkit init # 生成 agentkit.yaml
|
||||
agentkit version / doctor / usage # 工具命令
|
||||
agentkit task submit/status/list/cancel # 任务管理
|
||||
agentkit skill list/load/info # 技能管理
|
||||
agentkit pair --name X # 为外部系统生成 API Key
|
||||
pytest # 运行所有测试
|
||||
pytest -m "not integration" # 仅单元测试
|
||||
ruff check src/ && ruff format src/ # Lint + 格式化
|
||||
|
||||
# Frontend
|
||||
# 前端
|
||||
cd src/agentkit/server/frontend
|
||||
npm install # Install deps
|
||||
npm run dev # Vite dev server (proxy /api -> :8000)
|
||||
npm run build:frontend # Production build -> ../static
|
||||
npm run typecheck # TypeScript check
|
||||
npm install # 安装依赖
|
||||
npm run dev # Vite 开发服务器（代理 /api -> :8000）
|
||||
npm run build:frontend # 生产构建 -> ../static
|
||||
npm run typecheck # TypeScript 检查
|
||||
|
||||
# Desktop
|
||||
# 桌面端
|
||||
cd src/agentkit/server/frontend
|
||||
npm run tauri dev # Tauri dev mode
|
||||
npm run tauri build # Tauri production build
|
||||
npm run tauri dev # Tauri 开发模式
|
||||
npm run tauri build # Tauri 生产构建
|
||||
|
||||
# Docker
|
||||
docker-compose up -d # AgentKit + Redis + PostgreSQL
|
||||
```
|
||||
|
||||
## Architecture
|
||||
## 架构
|
||||
|
||||
### Request Flow
|
||||
### 请求流程
|
||||
|
||||
```
|
||||
User Input
|
||||
├─ @board prefix -> BoardRouter (experts/board_router.py) -> BoardOrchestrator (multi-round discussion)
|
||||
├─ @team prefix -> ExpertTeamRouter (experts/router.py) -> TeamOrchestrator (pipeline collaboration)
|
||||
└─ otherwise -> RequestPreprocessor (chat/request_preprocessor.py)
|
||||
Layer 0: @skill:xxx prefix -> explicit skill selection (SKILL_REACT or skill's configured mode)
|
||||
Layer 1: Trivial-input regex (~0ms, 0 tokens) -> DIRECT_CHAT
|
||||
(greetings, identity, factual Q&A, math, translation; guarded by _TOOL_CONTEXT_RE)
|
||||
Default: -> REACT (LLM decides tool usage autonomously in the agent loop)
|
||||
用户输入
|
||||
├─ @board 前缀 -> BoardRouter (experts/board_router.py) -> BoardOrchestrator（多轮讨论）
|
||||
├─ @team 前缀 -> ExpertTeamRouter (experts/router.py) -> TeamOrchestrator（流水线协作）
|
||||
└─ 其他 -> RequestPreprocessor (chat/request_preprocessor.py)
|
||||
Layer 0: @skill:xxx 前缀 -> 显式技能选择（SKILL_REACT 或技能配置的模式）
|
||||
Layer 1: 琐碎输入正则（~0ms，0 tokens）-> DIRECT_CHAT
|
||||
（问候、身份、事实问答、数学、翻译；由 _TOOL_CONTEXT_RE 守护）
|
||||
默认: -> REACT（LLM 在 agent 循环中自主决定工具使用）
|
||||
-> ExecutionMode: DIRECT_CHAT / REACT / SKILL_REACT / REWOO / REFLEXION / PLAN_EXEC / TEAM_COLLAB
|
||||
(chat handler currently supports DIRECT_CHAT, REACT, SKILL_REACT; others raise "not yet supported")
|
||||
（chat handler 当前支持 DIRECT_CHAT、REACT、SKILL_REACT；其余抛出 "not yet supported"）
|
||||
```
|
||||
|
||||
**Note**: The old 3-layer `CostAwareRouter` (with `RegexRules` / `HeuristicClassifier` / `SemanticRouter` / `Vickrey Auction`) has been replaced by `RequestPreprocessor`. The `IntentRouter` (`router/intent.py`) exists but is not wired into the chat flow. `AuctionHouse` with Vickrey auction lives in `marketplace/auction.py` (marketplace subsystem, not routing).
|
||||
**注意**：旧的 3 层 `CostAwareRouter`（含 `RegexRules` / `HeuristicClassifier` / `SemanticRouter` / `Vickrey Auction`）已被 `RequestPreprocessor` 替换。`IntentRouter`（`router/intent.py`）存在但未接入 chat 流程。`AuctionHouse`（Vickrey 拍卖）位于 `marketplace/auction.py`（属于 marketplace 子系统，非路由）。
|
||||
|
||||
### Agent Hierarchy
|
||||
### Agent 层级
|
||||
|
||||
```
|
||||
BaseAgent (core/base.py) — abstract, execute() is final
|
||||
+-- ConfigDrivenAgent (core/config_driven.py) — YAML-driven, 3 task modes
|
||||
BaseAgent (core/base.py) — 抽象基类，execute() 是 final 方法
|
||||
+-- ConfigDrivenAgent (core/config_driven.py) — YAML 驱动，3 种任务模式
|
||||
+-- ReActEngine (core/react.py) — Think->Act->Observe
|
||||
+-- ReflexionAgent (core/reflexion.py) — reflection-driven
|
||||
+-- ReWOOAgent (core/rewoo.py) — plan-without-observation
|
||||
+-- StandaloneAgent (core/standalone.py) — standalone runner
|
||||
+-- ReflexionAgent (core/reflexion.py) — 反思驱动
|
||||
+-- ReWOOAgent (core/rewoo.py) — 无观察规划
|
||||
+-- StandaloneAgent (core/standalone.py) — 独立运行器
|
||||
```
|
||||
|
||||
### Expert Team Mode (Pipeline)
|
||||
### 专家团队模式（流水线）
|
||||
|
||||
```
|
||||
ExpertConfig (extends AgentConfig) -> Expert (wraps ConfigDrivenAgent via AgentPool)
|
||||
ExpertTeam: manages experts, shared workspace, team status (FORMING→PLANNING→EXECUTING→SYNTHESIZING→COMPLETED)
|
||||
TeamOrchestrator: pipeline execution — Lead decomposes task into PlanPhase with depends_on, topological sort, parallel layers
|
||||
PlanPhase: id, name, assigned_expert, task_description, depends_on, status (PENDING/RUNNING/COMPLETED/FAILED)
|
||||
TeamPlan: phases with dependencies, topological_sort() returns execution layers (Kahn's algorithm)
|
||||
ExpertTeamRouter: @team prefix routing, @team:dev_team template expansion, name validation, MAX_EXPERTS=10
|
||||
HandoffTransport: InProcess (asyncio.Queue) + Redis Pub/Sub — used for event broadcasting only
|
||||
ExpertConfig（继承 AgentConfig）-> Expert（通过 AgentPool 包装 ConfigDrivenAgent）
|
||||
ExpertTeam：管理专家、共享工作区、团队状态（FORMING→PLANNING→EXECUTING→SYNTHESIZING→COMPLETED）
|
||||
TeamOrchestrator：流水线执行 — Lead 将任务分解为带 depends_on 的 PlanPhase，拓扑排序，并行分层
|
||||
PlanPhase：id、name、assigned_expert、task_description、depends_on、status（PENDING/RUNNING/COMPLETED/FAILED）
|
||||
TeamPlan：带依赖的阶段，topological_sort() 返回执行层（Kahn 算法）
|
||||
ExpertTeamRouter：@team 前缀路由、@team:dev_team 模板展开、名称校验、MAX_EXPERTS=10
|
||||
HandoffTransport：InProcess（asyncio.Queue）+ Redis Pub/Sub — 仅用于事件广播
|
||||
```
|
||||
|
||||
**Pipeline Flow**:
|
||||
1. `@team` prefix triggers team mode (or `@team:dev_team` for template, `@team:expert1,expert2` for explicit)
|
||||
2. `ExpertTeam.create_team()` sets status to PLANNING
|
||||
3. Lead Expert decomposes task into phases via LLM (fallback to single phase on failure)
|
||||
4. `topological_sort()` arranges phases into layers (same-layer parallel, inter-layer serial)
|
||||
5. Each phase creates an isolated `ConfigDrivenAgent` via `AgentPool.create_agent` (context isolation, KTD3)
|
||||
6. Phase outputs passed via `SharedWorkspace` (`{plan_id}/phase/{phase_id}/output`)
|
||||
7. Lead synthesizes results (BEST strategy)
|
||||
8. On all-phases-fail: fallback to single agent mode
|
||||
**流水线流程**：
|
||||
|
||||
**Event Sequence**: `team_formed` → `plan_update` → `phase_started` → `expert_step` → `expert_result` → `phase_completed` → `team_synthesis` → `team_dissolved`
|
||||
1. `@team` 前缀触发团队模式（或 `@team:dev_team` 用模板，`@team:expert1,expert2` 显式指定）
|
||||
2. `ExpertTeam.create_team()` 将状态置为 PLANNING
|
||||
3. Lead Expert 通过 LLM 将任务分解为阶段（失败时回退为单阶段）
|
||||
4. `topological_sort()` 将阶段排成层（同层并行，层间串行）
|
||||
5. 每个阶段通过 `AgentPool.create_agent` 创建隔离的 `ConfigDrivenAgent`（上下文隔离，KTD3）
|
||||
6. 阶段输出通过 `SharedWorkspace` 传递（`{plan_id}/phase/{phase_id}/output`）
|
||||
7. Lead 综合结果（BEST 策略）
|
||||
8. 所有阶段都失败时：回退到单 agent 模式
|
||||
|
||||
**Team Templates**: `configs/experts/dev_team.yaml` stores member list in `bound_skills` field (tech_lead, frontend_engineer, backend_engineer, qa_engineer, code_reviewer)
|
||||
**事件序列**：`team_formed` → `plan_update` → `phase_started` → `expert_step` → `expert_result` → `phase_completed` → `team_synthesis` → `team_dissolved`
|
||||
|
||||
Lifecycle: FORMING -> PLANNING -> EXECUTING -> SYNTHESIZING -> COMPLETED -> DISSOLVED
|
||||
On failure: fallback to single-agent mode (lead or first active expert).
|
||||
**团队模板**：`configs/experts/dev_team.yaml` 在 `bound_skills` 字段存储成员列表（tech\_lead、frontend\_engineer、backend\_engineer、qa\_engineer、code\_reviewer）
|
||||
|
||||
### Module Map
|
||||
生命周期：FORMING -> PLANNING -> EXECUTING -> SYNTHESIZING -> COMPLETED -> DISSOLVED
|
||||
失败时：回退到单 agent 模式（lead 或第一个活跃专家）。
|
||||
|
||||
| Layer | Modules | Purpose |
|
||||
|-------|---------|---------|
|
||||
| API | `server/`, `cli/` | FastAPI routes + Typer CLI |
|
||||
| Auth | `server/auth/` | JWT + RBAC + terminal security (6-layer whitelist) |
|
||||
| Service | `core/`, `chat/`, `skills/`, `experts/` | Agent engine, routing, skills, expert teams |
|
||||
| Data | `memory/`, `session/`, `bus/` | Persistence, sessions, messaging |
|
||||
| Utility | `llm/`, `tools/`, `evolution/`, `quality/`, `mcp/` | LLM gateway, tools, self-evolution, quality, MCP |
|
||||
| Client | `client/` | ConfigSync, RemoteLLMProvider integration |
|
||||
### 模块映射
|
||||
|
||||
### Key Subsystems
|
||||
| 层级 | 模块 | 用途 |
|
||||
| --- | ---------------------------------------------- | ------------------------------- |
|
||||
| API | `server/`、`cli/` | FastAPI 路由 + Typer CLI |
|
||||
| 认证 | `server/auth/` | JWT + RBAC + 终端安全（6 层白名单） |
|
||||
| 服务 | `core/`、`chat/`、`skills/`、`experts/` | Agent 引擎、路由、技能、专家团队 |
|
||||
| 数据 | `memory/`、`session/`、`bus/` | 持久化、会话、消息 |
|
||||
| 工具 | `llm/`、`tools/`、`evolution/`、`quality/`、`mcp/` | LLM 网关、工具、自进化、质量、MCP |
|
||||
| 客户端 | `client/` | ConfigSync、RemoteLLMProvider 集成 |
|
||||
|
||||
- **LLM Gateway** (`llm/`): 6 providers (OpenAI/Anthropic/Gemini/Doubao/Wenxin/Yuanbao), fallback, semantic cache, usage tracking, RemoteLLMProvider (client→server proxy with 401 refresh retry)
|
||||
- **Memory** (`memory/`): 4-layer (SOUL/USER/MEMORY/DAILY), WorkingMemory (Redis), EpisodicMemory (PG+pgvector), SemanticMemory (HTTP RAG)
|
||||
- **Evolution** (`evolution/`): Reflector, PromptOptimizer (genetic), PitfallDetector, ABTester
|
||||
- **Tools** (`tools/`): 21 built-in + MCP extension, composition (SequentialChain/ParallelFanOut/DynamicSelector)
|
||||
- **Pipeline** (`orchestrator/`): PipelineEngine, SagaOrchestrator, DynamicPipeline, HandoffManager
|
||||
- **Bus** (`bus/`): MemoryBus (in-process), RedisBus (distributed)
|
||||
- **Auth** (`server/auth/`): JWT (access 15min + refresh 7d, HS256), API Key (constant-time compare), 3-level RBAC (member/operator/admin + permission bits), 6-layer terminal security (blocklist→shell-ops→builtin→global→user→session→danger), bcrypt password hashing (rounds=12)
|
||||
### 关键子系统
|
||||
|
||||
### Server Routes (22 modules)
|
||||
- **LLM 网关**（`llm/`）：6 个 provider（OpenAI/Anthropic/Gemini/Doubao/Wenxin/Yuanbao）、fallback、语义缓存、用量追踪、RemoteLLMProvider（client→server 代理，带 401 刷新重试）
|
||||
- **记忆**（`memory/`）：4 层（SOUL/USER/MEMORY/DAILY）、WorkingMemory（Redis）、EpisodicMemory（PG+pgvector）、SemanticMemory（HTTP RAG）
|
||||
- **进化**（`evolution/`）：Reflector、PromptOptimizer（遗传算法）、PitfallDetector、ABTester
|
||||
- **工具**（`tools/`）：21 个内置 + MCP 扩展，组合（SequentialChain/ParallelFanOut/DynamicSelector）
|
||||
- **流水线**（`orchestrator/`）：PipelineEngine、SagaOrchestrator、DynamicPipeline、HandoffManager
|
||||
- **总线**（`bus/`）：MemoryBus（进程内）、RedisBus（分布式）
|
||||
- **认证**（`server/auth/`）：JWT（access 15min + refresh 7d，HS256）、API Key（恒定时间比较）、3 级 RBAC（member/operator/admin + 权限位）、6 层终端安全（blocklist→shell-ops→builtin→global→user→session→danger）、bcrypt 密码哈希（rounds=12）
|
||||
|
||||
| Prefix | Module | Purpose |
|
||||
|--------|--------|---------|
|
||||
| `/api/v1/agents` | agents.py | Agent CRUD |
|
||||
| `/api/v1/tasks` | tasks.py | Task submit/query/cancel |
|
||||
| `/api/v1/skills` | skills.py | Skill register/list |
|
||||
| `/api/v1/chat` | chat.py | Chat REST + WebSocket |
|
||||
| `/api/v1/ws` | ws.py | WebSocket channel |
|
||||
| `/api/v1/llm` | llm.py | LLM usage |
|
||||
| `/api/v1/llm/chat` | llm_gateway.py | LLM gateway proxy (JWT auth, SSE streaming) |
|
||||
| `/api/v1/health` | health.py | Health check |
|
||||
| `/api/v1/metrics` | metrics.py | Metrics |
|
||||
| `/api/v1/evolution` | evolution.py + evolution_dashboard.py | Self-evolution API |
|
||||
| `/api/v1/memory` | memory.py | Memory management |
|
||||
| `/api/v1/portal` | portal.py | Portal |
|
||||
| `/api/v1/kb` | kb_management.py | Knowledge base |
|
||||
| `/api/v1/skill-mgmt` | skill_management.py | Skill management |
|
||||
| `/api/v1/workflows` | workflows.py | Workflows |
|
||||
| `/api/v1/terminal` | terminal.py | Local terminal (client sidecar PTY) |
|
||||
| `/api/v1/terminal/server` | terminal_server.py | Server terminal (server PTY + admin approval) |
|
||||
| `/api/v1/terminal` | terminal_whitelist.py | Whitelist/blocklist/audit-log management |
|
||||
| `/api/v1/settings` | settings.py | Settings |
|
||||
| `/api/v1/auth` | auth.py | Login/refresh/logout/me |
|
||||
| `/api/v1/system` | system.py | System resources (SYSTEM_CONFIG permission) |
|
||||
| `/api/v1/config` | config_sync.py | Config version + sync (polling) |
|
||||
### 服务端路由（22 个模块）
|
||||
|
||||
### WebSocket Chat Protocol
|
||||
| 前缀 | 模块 | 用途 |
|
||||
| ------------------------- | -------------------------------------- | ------------------------- |
|
||||
| `/api/v1/agents` | agents.py | Agent CRUD |
|
||||
| `/api/v1/tasks` | tasks.py | 任务提交/查询/取消 |
|
||||
| `/api/v1/skills` | skills.py | 技能注册/列表 |
|
||||
| `/api/v1/chat` | chat.py | Chat REST + WebSocket |
|
||||
| `/api/v1/ws` | ws.py | WebSocket 通道 |
|
||||
| `/api/v1/llm` | llm.py | LLM 用量 |
|
||||
| `/api/v1/llm/chat` | llm\_gateway.py | LLM 网关代理（JWT 认证，SSE 流式） |
|
||||
| `/api/v1/health` | health.py | 健康检查 |
|
||||
| `/api/v1/metrics` | metrics.py | 指标 |
|
||||
| `/api/v1/evolution` | evolution.py + evolution\_dashboard.py | 自进化 API |
|
||||
| `/api/v1/memory` | memory.py | 记忆管理 |
|
||||
| `/api/v1/portal` | portal.py | Portal |
|
||||
| `/api/v1/kb` | kb\_management.py | 知识库 |
|
||||
| `/api/v1/skill-mgmt` | skill\_management.py | 技能管理 |
|
||||
| `/api/v1/workflows` | workflows.py | 工作流 |
|
||||
| `/api/v1/terminal` | terminal.py | 本地终端（client sidecar PTY） |
|
||||
| `/api/v1/terminal/server` | terminal\_server.py | 服务端终端（server PTY + 管理员审批） |
|
||||
| `/api/v1/terminal` | terminal\_whitelist.py | 白名单/黑名单/审计日志管理 |
|
||||
| `/api/v1/settings` | settings.py | 设置 |
|
||||
| `/api/v1/auth` | auth.py | 登录/刷新/登出/me |
|
||||
| `/api/v1/system` | system.py | 系统资源（需 SYSTEM\_CONFIG 权限） |
|
||||
| `/api/v1/config` | config\_sync.py | 配置版本 + 同步（轮询） |
|
||||
|
||||
Client -> Server: `message`, `reply`, `confirmation_reply`, `cancel`, `ping`
|
||||
Server -> Client: `connected`, `token`, `thinking`, `step`, `final_answer`, `skill_match`, `confirmation_request`, `confirmation_result`, `ask_human`, `error`, `pong`
|
||||
Expert Team events: `team_formed`, `expert_step`, `expert_result`, `plan_update`, `phase_started`, `phase_completed`, `phase_failed`, `team_synthesis`, `team_dissolved`
|
||||
### WebSocket Chat 协议
|
||||
|
||||
### Frontend Pages
|
||||
Client -> Server：`message`、`reply`、`confirmation_reply`、`cancel`、`ping`
|
||||
Server -> Client：`connected`、`token`、`thinking`、`step`、`final_answer`、`skill_match`、`confirmation_request`、`confirmation_result`、`ask_human`、`error`、`pong`
|
||||
专家团队事件：`team_formed`、`expert_step`、`expert_result`、`plan_update`、`phase_started`、`phase_completed`、`phase_failed`、`team_synthesis`、`team_dissolved`
|
||||
|
||||
- `/agent/chat` — Chat with Expert Team view
|
||||
- `/agent/code` — Code/workflow
|
||||
- `/agent/monitor` — Evolution dashboard
|
||||
- `/computer-use` — Desktop control
|
||||
- `/login` — Login page (JWT auth)
|
||||
- Terminal panel — Local + server terminal with whitelist manager
|
||||
### 前端页面
|
||||
|
||||
### Configuration Priority
|
||||
- `/agent/chat` — 专家团队聊天视图
|
||||
- `/agent/code` — 代码/工作流
|
||||
- `/agent/monitor` — 进化看板
|
||||
- `/computer-use` — 桌面控制
|
||||
- `/login` — 登录页（JWT 认证）
|
||||
- 终端面板 — 本地 + 服务端终端，含白名单管理器
|
||||
|
||||
CLI args > `agentkit.yaml` > env vars (`${VAR:-default}`) > `.env` > hardcoded defaults
|
||||
### 配置优先级
|
||||
|
||||
Config search: `--config` path > `./agentkit.yaml` > `~/.agentkit/agentkit.yaml`
|
||||
CLI 参数 > `agentkit.yaml` > 环境变量（`${VAR:-default}`）> `.env` > 硬编码默认值
|
||||
|
||||
## Conventions
|
||||
配置查找：`--config` 路径 > `./agentkit.yaml` > `~/.agentkit/agentkit.yaml`
|
||||
|
||||
- Skill configs: `configs/skills/*.yaml` (16 presets, unified as `SkillConfig`)
|
||||
- Skill categories: `agent_template` (execution engines: react/direct/rewoo/reflexion/plan_exec/goal_driven) vs `business_skill` (domain skills). Classified via `_ENGINE_TEMPLATE_NAMES` in `server/routes/skill_management.py`. Frontend groups by `category` field — `SkillsView` two-column layout, `SkillCard`/`SkillsTab` show type tags (引擎/技能) and category-based icons
|
||||
- LLM configs: `agentkit.yaml` llm section (unified with server config)
|
||||
- Pipeline configs: `configs/pipelines/*.yaml`
|
||||
- Expert templates: `configs/experts/*.yaml` (5 programming experts + dev_team team template), registered via `ExpertTemplateRegistry`
|
||||
- Team templates: `bound_skills` field stores member list (e.g., `dev_team.yaml` lists tech_lead, frontend_engineer, backend_engineer, qa_engineer, code_reviewer)
|
||||
- All Pydantic models use `model_config = ConfigDict(...)` not `class Config`
|
||||
- Test files: `tests/unit/` and `tests/integration/`
|
||||
- Frontend stores: Pinia, one per domain (chat, team, settings)
|
||||
- Frontend components: `src/agentkit/server/frontend/src/components/`
|
||||
## 约定
|
||||
|
||||
## Boundaries
|
||||
- 技能配置：`configs/skills/*.yaml`（16 个预设，统一为 `SkillConfig`）
|
||||
- 技能分类：`agent_template`（执行引擎：react/direct/rewoo/reflexion/plan\_exec/goal\_driven）vs `business_skill`（领域技能）。通过 `server/routes/skill_management.py` 中的 `_ENGINE_TEMPLATE_NAMES` 分类。前端按 `category` 字段分组 — `SkillsView` 双栏布局，`SkillCard`/`SkillsTab` 显示类型标签（引擎/技能）和基于分类的图标
|
||||
- LLM 配置：`agentkit.yaml` llm 段（与服务端配置统一）
|
||||
- 流水线配置：`configs/pipelines/*.yaml`
|
||||
- 专家模板：`configs/experts/*.yaml`（5 个编程专家 + dev\_team 团队模板），通过 `ExpertTemplateRegistry` 注册
|
||||
- 团队模板：`bound_skills` 字段存储成员列表（如 `dev_team.yaml` 列出 tech\_lead、frontend\_engineer、backend\_engineer、qa\_engineer、code\_reviewer）
|
||||
- 所有 Pydantic 模型使用 `model_config = ConfigDict(...)` 而非 `class Config`
|
||||
- 测试文件：`tests/unit/` 和 `tests/integration/`
|
||||
- 前端 stores：Pinia，每个领域一个（chat、team、settings）
|
||||
- 前端组件：`src/agentkit/server/frontend/src/components/`
|
||||
|
||||
## 边界
|
||||
|
||||
- 未经明确请求不得修改 `pyproject.toml` 版本
|
||||
- 禁止直接推送到 main — 使用 feature 分支
|
||||
- 集成测试需要 Docker（Redis + PostgreSQL）
|
||||
- 桌面端构建需要 Rust 工具链 + PyInstaller
|
||||
|
||||
- Never modify `pyproject.toml` version without explicit request
|
||||
- Never push to main directly — use feature branches
|
||||
- Integration tests require Docker (Redis + PostgreSQL)
|
||||
- Desktop builds require Rust toolchain + PyInstaller
|
||||
|
|
|
|||
12
README.md
12
README.md
|
|
@ -1,16 +1,20 @@
|
|||
# Fischer AgentKit
|
||||
|
||||
统一 AI Agent 开发框架 -- 将 LLM、Tool、Prompt 组装为可执行的 Skill，通过 ReAct 推理引擎自主完成任务，支持记忆持久化、自进化、Pipeline 编排和桌面客户端。
|
||||
企业级统一 AI Agent 门户平台 -- 面向企业用户与开发者，将 LLM、Tool、Prompt 组装为可执行的 Skill，通过 ReAct 推理引擎自主完成任务，支持记忆持久化、自进化、Pipeline 编排、专家团队协作和桌面客户端。
|
||||
|
||||
## 项目简介
|
||||
|
||||
AgentKit 解决的核心问题：**从写 150 行 Agent 代码降为 10-20 行 YAML 配置**。
|
||||
AgentKit 是企业级统一 AI Agent 门户平台，目标用户覆盖**企业用户**与**开发者**：
|
||||
|
||||
传统方式下，每新增一个 Agent 需要编写子类、处理 LLM 调用、管理工具绑定、校验输出质量。AgentKit 将这些能力标准化为可组合模块，开发者只需编写 YAML 配置即可定义一个完整的 Skill（Prompt + Tool + 质量门禁），框架自动完成 ReAct 推理循环、模型路由降级、产出质量检查和标准化输出。
|
||||
- **企业用户**：通过 Web GUI / 桌面客户端开箱即用，零代码配置 Skill、专家团队、知识库，直接获得多专家协作、文档生成、桌面操控等能力
|
||||
- **开发者**：通过 Python 库 / CLI / HTTP API 深度集成，将 150 行 Agent 代码降为 10-20 行 YAML 配置，框架自动完成 ReAct 推理循环、模型路由降级、产出质量检查和标准化输出
|
||||
|
||||
AgentKit 将 LLM、Tool、Prompt 标准化为可组合模块，开发者只需编写 YAML 配置即可定义一个完整的 Skill（Prompt + Tool + 质量门禁）；企业用户通过门户界面即可编排专家团队、监控自进化、管理知识库与终端安全。
|
||||
|
||||
核心定位：
|
||||
|
||||
- **配置驱动** -- YAML 定义 Skill，无需写 Agent 子类
|
||||
- **门户平台** -- 统一入口聚合 Skill、专家团队、知识库、终端、自进化等能力，企业用户开箱即用
|
||||
- **配置驱动** -- YAML 定义 Skill，开发者无需写 Agent 子类
|
||||
- **生产就绪** -- 内置质量门禁、模型降级、用量统计、级联检测、状态持久化
|
||||
- **四种使用** -- Python 库引用、CLI 聊天、Web GUI、桌面客户端
|
||||
- **专家团队** -- Expert Team Mode，多专家协作执行复杂任务，前端以多角色对话流呈现
|
||||
|
|
|
|||
|
|
@ -0,0 +1,214 @@
|
|||
# 多维表格（Bitable）伴生服务需求文档
|
||||
|
||||
- **日期**：2026-06-24
|
||||
- **状态**：已对齐，待规划
|
||||
- **范围分级**：Deep — feature
|
||||
- **后续**：交由 `/ce-plan` 做实现规划
|
||||
|
||||
---
|
||||
|
||||
## 1. 问题与机会
|
||||
|
||||
AgentKit 当前缺少一个**统一的持久化结构化数据落地载体**。当出现以下需求时，没有合适的地方承接数据：
|
||||
|
||||
- 多系统数据汇总（需把多个来源的结构化数据合并到一处）
|
||||
- 本地 Excel 上传后持久化（当前 Excel 仅能单向导出或解析为文本进 RAG，无法作为可编辑的结构化表留存）
|
||||
- 外部数据采集（爬虫/API 抓取的结果需要按字段落地为可查询、可视图、可分析的表）
|
||||
|
||||
现状：Excel 导出是单向的（`src/agentkit/documents/renderers/excel_renderer.py`），Excel 解析只转文本进知识库（`src/agentkit/memory/document_loader.py`），`MultiSourceRetriever` 只做读侧多源检索，`SharedWorkspace` 是带 TTL 的临时 KV。**没有任何模块能把异构数据源的结构化数据持久化为可编辑、可视图、可计算的多维表格。**
|
||||
|
||||
机会：引入多维表格伴生服务，作为 AgentKit 异构数据的统一落地载体。Agent 作为数据的主要作者（采集写入），用户在落地后的表上精修、配视图、做分析。这既补齐了结构化数据持久化的缺口，又让 Agent 获得"数据编排者"的战略能力。
|
||||
|
||||
## 2. 主要使用者与核心价值
|
||||
|
||||
| 维度 | 决策 |
|
||||
|------|------|
|
||||
| **形态** | 混合模式——Agent 采集 + 用户精修 |
|
||||
| **Agent 角色** | 数据作者：执行三类采集（Excel/数据库/爬虫API），按字段写入多维表格 |
|
||||
| **用户角色** | 数据精修者与分析者：在落地后的表上编辑用户列、配置视图、做分析 |
|
||||
| **核心价值** | 异构数据源的统一持久化落地载体，其上承载视图、分析、公式、引用 |
|
||||
|
||||
**三类采集场景**（均为 Agent 驱动）：
|
||||
1. 上传 Excel 或提供在线 Excel 地址 → 读取内容 → 按字段写入多维表格
|
||||
2. 指定数据库 → 根据数据表 → 生成多张多维表格数据表
|
||||
3. 根据指令执行数据采集（爬虫或具体 API）→ 获取到的数据按字段写入多维表格
|
||||
|
||||
## 3. 服务架构
|
||||
|
||||
**多维表格是 AgentKit 的伴生服务**：
|
||||
|
||||
- **逻辑独立**：自有 API/CLI、自有领域模型（表/字段/记录/视图/公式）、自有存储边界
|
||||
- **当前共部署**：物理上与 AgentKit 同进程或同部署单元，UI 级集成于 AgentKit 前端
|
||||
- **调用边界**：AgentKit ↔ 多维表格走 API/CLI，**不做进程内紧耦合**
|
||||
- **未来演进**：可零成本抽离为独立服务，只是部署变更，不改代码
|
||||
- **存储边界**：当前共享 AgentKit 的 PostgreSQL，使用**独立 schema** 隔离；未来抽离时迁移
|
||||
|
||||
> 设计含义：所有跨服务交互按"远程调用"心智设计，即使当前是本地调用。字段所有权模型、upsert 语义、公式引擎都**内建在多维表格服务自身**，而非套在外部工具上。
|
||||
|
||||
## 4. 关键产品决策
|
||||
|
||||
### 4.1 写入语义：按主键 upsert + 字段所有权模型
|
||||
|
||||
Agent 重复采集时，按表的主键字段 upsert：
|
||||
|
||||
- **匹配到的记录**：更新"数据列"（Agent 管理的列），保留"用户列"
|
||||
- **未匹配的记录**：新增
|
||||
- **用户列**永不被 Agent 覆盖
|
||||
|
||||
**字段所有权**（每列标记为"数据列"或"用户列"）：
|
||||
|
||||
- **自动推断**：公式列、引用列、手动标注列 → 用户列；Agent 采集写入的列 → 数据列
|
||||
- **Agent 声明**：Agent 采集时可显式声明列的所有权，覆盖自动推断
|
||||
- 公式列天然是用户列（派生的，永不被覆盖）
|
||||
|
||||
### 4.2 公式与引用：身份核心，深度分阶段
|
||||
|
||||
公式列和引用列是"多维表格"身份的核心，**必须从 v1 存在**，但支持深度分阶段扩展：
|
||||
|
||||
- **v1**：基础公式（算术、字符串、简单聚合如 SUM/AVG/COUNT）+ 基础引用（lookup 到另一张表的字段）
|
||||
- **v2+**：高级公式（日期、条件、跨表 rollup）+ 函数库扩展
|
||||
|
||||
**公式重算策略**：异步重算 + "计算中"状态标记。Agent 写入数据列后，依赖该列的公式列进入"计算中"状态，由后台异步重算管道更新。避免同步重算阻塞写入，代价是短暂的不一致窗口（用户可见"计算中"标记）。
|
||||
|
||||
### 4.3 规模与存储：可演进
|
||||
|
||||
- **起步规模**：单表 < 10 万行，总表 < 1000 张（部门级）
|
||||
- **架构目标**：支持未来向大规模（10万+行）演进
|
||||
- **存储选型**：规范化存储（字段定义、记录、单元格分离）+ 索引 + 分页，**不**用 JSONB 整表塞单行
|
||||
- **大规模演进路径**（v3）：列式存储、分区、物化视图、异步重算管道
|
||||
|
||||
## 5. 能力范围与分阶段
|
||||
|
||||
用户提出的 6 项能力，按复杂度与依赖关系分三阶段：
|
||||
|
||||
| 能力 | 复杂度 | v1 | v2 | v3 |
|
||||
|------|--------|----|----|----|
|
||||
| ① 模块搭建（服务骨架+领域模型+API） | 基础 | ✅ | — | — |
|
||||
| ② 数据采集落地（Excel/DB/爬虫API 三类） | 中 | ✅ | — | — |
|
||||
| ③ 多视图展示 | 中 | 网格视图 | 看板/甘特/画廊 | 表单 |
|
||||
| ④ 分析计算 | 中高 | — | 分组/透视 | 高级聚合 |
|
||||
| ⑤ 公式列+引用列 | 高 | 基础公式+lookup | 高级公式+rollup | 函数库扩展 |
|
||||
| ⑥ 图片+附件 | 低中 | ✅ | — | — |
|
||||
|
||||
### v1：核心闭环验证
|
||||
|
||||
验证"Agent 采集 → 持久化落地 → 用户查看/精修"的核心闭环：
|
||||
|
||||
- 服务骨架：领域模型（表/字段/记录/视图）、API/CLI、独立 schema 存储
|
||||
- 字段所有权模型 + 按主键 upsert 语义
|
||||
- 三类采集落地（Excel 上传/URL、数据库导入、爬虫/API 采集）
|
||||
- 网格视图（表格视图，支持排序/筛选/分页/单元格编辑）
|
||||
- 基础公式列（算术、字符串、SUM/AVG/COUNT 等简单聚合）
|
||||
- 基础引用列（lookup 到另一张表的字段）
|
||||
- 图片/附件字段类型（复用现有文件上传能力）
|
||||
- 异步公式重算 + "计算中"状态
|
||||
|
||||
### v2：多视图与分析
|
||||
|
||||
- 看板视图（按分组字段分列展示）
|
||||
- 甘特视图（按日期字段排时间线）
|
||||
- 画廊视图（以图片/附件为主视觉的卡片展示）
|
||||
- 高级公式（日期函数、条件函数、跨表 rollup）
|
||||
- 分析能力（分组聚合、透视表）
|
||||
|
||||
### v3：规模化与协作
|
||||
|
||||
- 表单视图（以表单形式收集数据写入表）
|
||||
- 公式函数库扩展
|
||||
- 大规模优化（列式存储、分区、物化视图、异步重算管道升级）
|
||||
- 多人实时协作
|
||||
|
||||
## 6. 方案探索与推荐
|
||||
|
||||
### 方案 1：自建多维表格引擎，分阶段交付
|
||||
|
||||
在 AgentKit 内构建原生 bitable 子系统，规范化存储，字段所有权模型原生内建，公式引擎自建分阶段。Agent 通过新增的 bitable API/CLI 写入。
|
||||
|
||||
- **优点**：完全可控；与现有栈（PG/Redis/Vue/FastAPI）匹配；字段所有权模型原生；upsert 语义无摩擦
|
||||
- **缺点**：构建工作量最大；公式引擎是硬骨头；需长期维护
|
||||
- **风险**：公式引擎范围蔓延；大规模重算性能
|
||||
|
||||
### 方案 2：集成开源多维表格（APITable/NocoDB）作为子服务
|
||||
|
||||
部署开源 bitable 作为伴生服务，AgentKit 通过其 API 让 Agent 写入，用户编辑在 bitable 原生 UI 完成，上层叠加 upsert-保留用户列逻辑。
|
||||
|
||||
- **优点**：视图/公式/附件白送，成熟，最快获完整功能
|
||||
- **缺点**：AGPL 协议有传染性风险（若商业化）；upsert-保留用户列需硬套（外部 bitable 无字段所有权概念）；集成走 API 较松
|
||||
- **风险**：协议冲突；外部模型与需求偏离
|
||||
|
||||
### 方案 3（挑战者）：Agent 结构化数据底座优先，UI 作为叠加层
|
||||
|
||||
反转优先级：多维表格首先是 Agent 的持久化结构化工作记忆/输出底座，用户侧多视图 UI 是读写底座的叠加层。
|
||||
|
||||
- **优点**：最大化 Agent 协同；战略差异化；底座可驱动表格 UI 之外的能力
|
||||
- **缺点**：用户主动建表流程次要；底座抽象需更多架构思考
|
||||
|
||||
### 推荐：方案 1 自建 + 方案 3 底座心智
|
||||
|
||||
**理由**：
|
||||
|
||||
1. 三类采集场景全是 Agent 驱动——本质是"Agent 作为数据作者"，方案 3 心智天然契合；但用户也要精修，需方案 1 的完整引擎
|
||||
2. upsert-保留用户列的字段所有权模型是定制的——外部 bitable 没有此概念，硬套很痛；AGPL 对可能商业化的产品是真实风险
|
||||
3. 现有基础设施齐全（PG + pgvector + Redis + SQLAlchemy 2 + Vue3 + Ant Design Vue），自建边际成本可控
|
||||
4. 伴生服务架构约束天然要求 API/CLI 边界——方案 1 自建反而最契合，因为所有权模型内建在服务自身
|
||||
5. 分阶段控制风险——v1 先验证核心闭环
|
||||
|
||||
**与方案 3 的融合**：架构上以"Agent 的持久化结构化数据底座"心智设计领域模型，使多维表格不仅是"一个表格功能"，而是 Agent 的数据编排能力的载体。这让底座未来可驱动表格 UI 之外的能力（仪表盘、报表、Agent 记忆）。
|
||||
|
||||
## 7. 范围边界
|
||||
|
||||
### 本次范围内（v1）
|
||||
|
||||
- 多维表格伴生服务骨架（领域模型、API/CLI、独立 schema 存储）
|
||||
- 字段所有权模型 + 按主键 upsert 语义
|
||||
- 三类采集落地（Excel/DB/爬虫API）
|
||||
- 网格视图
|
||||
- 基础公式列 + 基础引用列（lookup）
|
||||
- 图片/附件字段
|
||||
- 异步公式重算
|
||||
|
||||
### 延后（v2/v3）
|
||||
|
||||
- 看板/甘特/画廊/表单视图
|
||||
- 高级公式（日期/条件/跨表 rollup）+ 函数库扩展
|
||||
- 分析能力（分组/透视）
|
||||
- 大规模优化（列式/分区/物化视图）
|
||||
- 多人实时协作
|
||||
|
||||
### 本产品身份之外
|
||||
|
||||
- 不做通用电子表格（非单元格自由编辑，是字段化记录模型）
|
||||
- 不做 ETL/数据管道平台（采集是 Agent 驱动的按需执行，非定时调度管道）
|
||||
- 不做 BI 仪表盘产品（分析能力服务于表格内聚合，非独立 BI）
|
||||
- 不替代知识库 RAG（多维表格是结构化数据载体，非非结构化文档检索）
|
||||
|
||||
## 8. 假设与依赖
|
||||
|
||||
- **假设**：Agent 已具备执行采集任务的能力（爬虫/API 调用），多维表格只承接"写入"环节，不负责采集执行本身
|
||||
- **假设**：共享 PostgreSQL 的性能足以支撑 v1/v2 规模；v3 大规模时再评估独立数据库或列式存储
|
||||
- **依赖**：现有文件上传能力（`src/agentkit/server/routes/chat.py` 的上传端点、`data/uploads/` 存储）可复用于附件字段
|
||||
- **依赖**：Agent 工具系统（`src/agentkit/tools/base.py` 的 `execute() -> dict` 契约）可扩展新增 bitable 写入工具
|
||||
- **假设**：公式异步重算的"计算中"窗口（秒级）对用户可接受
|
||||
|
||||
## 9. 成功标准
|
||||
|
||||
**v1 验证成功的标志**：
|
||||
|
||||
1. Agent 能把一份 Excel 上传的数据按字段写入多维表格，并在网格视图中查看
|
||||
2. Agent 能指定一个数据库表，生成对应的多维表格
|
||||
3. Agent 能执行一次 API 采集，把返回数据按字段写入多维表格
|
||||
4. 用户能在网格视图中编辑单元格、新增公式列（如 `=SUM(数据列)`）、看到异步重算结果
|
||||
5. Agent 对同一表重复采集时，按主键 upsert 更新数据列，用户的公式列和手动编辑保留不变
|
||||
6. 多维表格服务通过 API/CLI 被 AgentKit 调用，无进程内紧耦合
|
||||
|
||||
## 10. 下一步
|
||||
|
||||
本需求文档交由 `/ce-plan` 做实现规划，重点规划：
|
||||
|
||||
- v1 的领域模型设计（表/字段/记录/视图/公式 的实体关系）
|
||||
- 独立 schema 的存储设计（规范化表结构、索引、分页）
|
||||
- API/CLI 接口设计（CRUD + 采集写入 + upsert + 公式重算触发）
|
||||
- 字段所有权模型的实现机制（自动推断 + Agent 声明）
|
||||
- 异步重算管道设计
|
||||
- Agent bitable 写入工具设计
|
||||
- 前端网格视图组件选型与集成
|
||||
|
|
@ -0,0 +1,182 @@
|
|||
---
|
||||
date: 2026-06-24
|
||||
topic: portal-platform-evolution
|
||||
---
|
||||
|
||||
# AgentKit 门户平台整体演进路线
|
||||
|
||||
## Summary
|
||||
|
||||
按优先级串行推进 AgentKit 门户平台演进：先建独立 RAG 平台对标 MaxKB 功能对等，再扩展多端接入与 MCP Server，最后生态替换降本（MCP/Celery/LiteLLM）。不设硬性时间，按完成度推进。
|
||||
|
||||
## Problem Frame
|
||||
|
||||
AgentKit 定位为企业级统一 AI Agent 门户平台，面向企业用户与开发者。对标 MaxKB（开源企业级智能体平台，GitHub 19k+ stars）后发现，当前能力堆栈在多个方向存在差距：
|
||||
|
||||
- **RAG 工业级管道**：AgentKit 现有 `memory/` 模块是开发者级组件库（基础分块 + pgvector 语义检索 + time_decay 重排），MaxKB 是工业级产品功能（双索引检索 + 智能分段 + 问题生成 + 术语表 + 命中处理模式）。RAG 是门户平台服务企业知识库场景的底线能力。
|
||||
- **平台触达**：AgentKit 仅有飞书/Confluence/通用 HTTP 三种 RAG 适配器，MaxKB 原生支持企微/钉钉/飞书/Slack 多端接入。门户平台需要触达企业现有协作工具。
|
||||
- **MCP Server**：AgentKit 已有 MCP Client（`mcp/client.py`）和 MCP Server（`mcp/server.py`）基础实现，但尚未将 Skill/专家团队发布为 MCP 工具。门户平台应完善 MCP Server 的 Skill/专家团队发布能力。
|
||||
- **自研 vs 生态**：AgentKit 大量自研（Agent 引擎/LLM 网关/工作流画布/MCP 客户端/记忆系统/消息总线），commodity 层维护成本高。
|
||||
|
||||
本次演进为**预防性演进 + 必备功能补齐**，非救火式驱动。目标是补齐门户平台应有的能力，使 AgentKit 在企业级 AI Agent 平台赛道具备完整竞争力。
|
||||
|
||||
## Key Decisions
|
||||
|
||||
**串行演进策略（方案 A）。** 按优先级串行推进，每个方向充分交付后再进入下一个。理由：用户要求"对标 MaxKB 功能对等"，MVP 驱动难以一次达标；不设硬性时间契合串行节奏；预防性演进无紧急压力，可保障交付质量。
|
||||
|
||||
**RAG 平台并行独立。** 新建独立 RAG 平台模块，现有 `memory/` 保留给 Agent 记忆（WorkingMemory/EpisodicMemory/SemanticMemory）。理由：职责分离，避免 RAG 与 Agent 记忆耦合；RAG 平台作为门户平台基础设施服务于企业用户知识库场景，Agent 记忆服务于 Agent 运行时。
|
||||
|
||||
**开放引入生态依赖。** commodity 层优先用生态，降低维护成本。约束：需注意开源协议合规，且对已完成或进行中的特性保持向后兼容。差异化层（Agent 引擎/专家团队/自进化/终端安全）保持自研。
|
||||
|
||||
**保留现有工作流。** FlowCanvas 不替换为 LogicFlow，现有工作流画布保持自研。理由：避免破坏现有节点类型（SkillNode/ApprovalNode/ConditionNode/ParallelNode）和工作流。
|
||||
|
||||
**对标 MaxKB 功能对等。** RAG 工业级管道的成功标准是功能对等：双索引检索/智能分段/问题生成/术语表/命中处理模式/rerank 全部具备。
|
||||
|
||||
## Requirements
|
||||
|
||||
### RAG 工业级管道（优先级 1）
|
||||
|
||||
R1. 新建独立 RAG 平台模块，与现有 `memory/` 模块职责分离，现有 `memory/` 保留给 Agent 记忆使用。现有 `memory/local_rag.py` 的 `LocalRAGService`（pgvector + 分块 + 嵌入 + 语义检索）需明确迁移策略：吸收/扩展至新平台、提取至新模块、或新建并废弃 LocalRAGService。
|
||||
|
||||
R2. 支持双索引检索：pgvector 语义检索 + PostgreSQL 全文检索（`search_vector`），提供 `embedding`（语义）/ `keywords`（全文）/ `blend`（混合）三种检索模式。检索模式由企业用户按知识库配置默认值，Agent 运行时可按查询特征覆盖。
|
||||
|
||||
R3. 在 RAG 平台模块中实现智能分段与高级分段能力（可参考现有 `memory/chunking.py` 的分块基础），提供分段预览能力，企业用户可在向量化前查看分段结果。
|
||||
|
||||
R4. 支持问题自动生成：为文档段落自动生成相关问题/问法，提升检索召回率。
|
||||
|
||||
R5. 支持术语表（Termbase）：通过全文检索分词增强，提升中文场景检索准确率。
|
||||
|
||||
R6. 支持命中处理模式：模型优化模式（LLM 基于检索结果生成回答）与直接回答模式（直接返回匹配段落），按知识库配置默认模式，企业用户在知识库设置中选择，Agent 可按查询场景覆盖。
|
||||
|
||||
R7. 支持 rerank 重排：检索结果经 rerank 模型重排后返回，提升相关性排序。
|
||||
|
||||
R8. 扩展现有 `KnowledgeBaseView`/`DocumentUpload`/`SearchTest` 组件，提供可视化文档管理：文档上传/分段预览/检索测试，企业用户可通过前端界面管理知识库。知识库必须实施 per-KB 访问控制（owner/authorized-users），Agent 检索必须限定于调用用户授权的知识库。文档上传必须验证文件类型（白名单）、强制大小限制、并在索引前净化解析内容（markdown sanitize、PDF 解析安全）。
|
||||
|
||||
### 平台触达扩展（优先级 2）
|
||||
|
||||
R9. 支持多端消息接入：企微/钉钉/飞书/Slack 消息适配器，企业用户可通过现有协作工具使用 AgentKit。各平台适配器必须验证平台提供的请求签名/token（飞书 encrypt_key、钉钉 token、企微信 EncodingAESKey）后处理消息，拒绝未认证请求。所有第三方平台凭证必须存储于 secrets store（非明文配置），定义轮换策略与访问审计。
|
||||
|
||||
R10. 完善现有 MCP Server（`mcp/server.py`）：支持将 Skill/专家团队发布为 MCP 工具，供外部 AI 系统调用。MCP 工具调用必须要求认证与授权（复用现有 JWT+RBAC 或 API Key 机制），发布 Skill/专家团队为 MCP 工具需管理员级授权。
|
||||
|
||||
### 生态替换降本（优先级 3）
|
||||
|
||||
**目标**：将 commodity 层（MCP 客户端/异步任务/LLM Provider 适配）迁移至生态方案，降低自研维护成本，使团队聚焦差异化能力。成功标准：替换后现有功能行为不变，维护代码量减少。
|
||||
|
||||
R11. MCP 客户端替换为 `langchain-mcp-adapters`：跟进行业协议演进，降低自研 3 传输层（Stdio/HTTP/SSE）的维护成本。
|
||||
|
||||
R12. 引入 Celery 异步任务：与现有 asyncio 原生共存，承接文档向量化/批量任务（利用现有 Redis 作为 broker，不引入新基础设施），提供任务持久化/重试/调度能力。提供异步任务可视化：进度展示、失败通知与重试、任务历史。
|
||||
|
||||
R13. LLM Provider 底层替换为 LiteLLM：上层网关逻辑（fallback/缓存/用量追踪）保留自研，底层 provider 适配走 LiteLLM 统一接口。
|
||||
|
||||
## Actors
|
||||
|
||||
A1. **企业用户** — 通过前端界面管理知识库（上传文档/配置分段/测试检索）、配置多端接入、发布 MCP 工具。
|
||||
|
||||
A2. **开发者** — 通过 API/MCP Server 集成 AgentKit 能力到外部系统。
|
||||
|
||||
A3. **Agent** — 运行时调用 RAG 平台检索知识库内容，支撑问答与决策。
|
||||
|
||||
## Key Flows
|
||||
|
||||
F1. RAG 文档处理流程
|
||||
- **Trigger:** 企业用户上传文档到知识库。
|
||||
- **Actors:** A1, A3
|
||||
- **Steps:** 文档解析 → 分段（智能/高级）→ 分段预览 → 向量化 → 全文索引建立 → 问题自动生成 → 可用。
|
||||
- **Outcome:** 文档进入知识库，可被 Agent 检索。
|
||||
|
||||
F2. RAG 检索流程
|
||||
- **Trigger:** Agent 需要检索知识库回答用户问题。
|
||||
- **Actors:** A3
|
||||
- **Steps:** 查询接收 → 检索模式选择（embedding/keywords/blend）→ 双索引检索 → rerank 重排 → 命中处理（模型优化/直接回答）→ 返回结果。
|
||||
- **Outcome:** Agent 获得相关知识库内容。
|
||||
|
||||
F3. 多端消息接入流程
|
||||
- **Trigger:** 企业用户通过企微/钉钉/飞书/Slack 发送消息。
|
||||
- **Actors:** A1, A3
|
||||
- **Steps:** 消息适配器接收 → 转换为 AgentKit 标准格式 → Agent 处理 → 响应转换为目标平台格式 → 返回。
|
||||
- **Outcome:** 企业用户通过协作工具获得 Agent 响应。
|
||||
|
||||
F4. MCP Server 发布流程
|
||||
- **Trigger:** 企业用户或开发者将 Skill/专家团队发布为 MCP 工具。
|
||||
- **Actors:** A1, A2
|
||||
- **Steps:** 选择 Skill/专家团队 → 配置 MCP endpoint → 发布 → 外部 AI 系统可通过 MCP 协议调用。
|
||||
- **Outcome:** AgentKit 能力通过 MCP 协议对外输出。
|
||||
|
||||
## Scope Boundaries
|
||||
|
||||
### Deferred for later
|
||||
|
||||
- 本地模型支持（Ollama）——后续迭代，服务企业私有化部署场景。
|
||||
- 现有 `memory/` 模块重构——保留给 Agent 记忆用，不在本次演进范围。
|
||||
|
||||
### Outside this product's identity
|
||||
|
||||
- FlowCanvas→LogicFlow 替换——保留现有工作流，不替换。
|
||||
- Agent 引擎（ReActEngine/ReWOO/Reflexion/PlanExec）——保持自研，是核心差异化能力。
|
||||
- 专家团队编排（流水线 + 私董会）——保持自研，生态无对应方案。
|
||||
- 自进化系统（16 组件）——保持自研，独有能力。
|
||||
- 终端安全（6 层白名单）——保持自研，安全必须自主可控。
|
||||
|
||||
## Dependencies / Assumptions
|
||||
|
||||
- **开源协议合规**：生态替换涉及的依赖协议需宽松可商用。Celery (BSD-3)、LiteLLM (MIT)、langchain-mcp-adapters 需确认协议。（注：LogicFlow 已因保留 FlowCanvas 决策排除，非协议原因——Apache-2.0 本身可商用）
|
||||
- **现有特性向后兼容**：生态替换（MCP 客户端/Celery/LiteLLM）不能破坏现有功能，需提供迁移路径。
|
||||
- **pgvector 基础设施**：RAG 平台与现有 EpisodicMemory 共用 pgvector 基础设施，但数据模型独立。
|
||||
- **前端组件复用**：现有 KnowledgeBaseView/DocumentUpload/SearchTest 组件可能需要重构以支撑 RAG 平台可视化文档管理。
|
||||
|
||||
## Outstanding Questions
|
||||
|
||||
### Resolve Before Planning
|
||||
|
||||
- **[P0 安全] MCP Server 端点缺少认证/授权决策**：R10 暴露 Skills/Expert Teams 为 MCP 工具，F4 描述发布流程但未提及认证、授权、限流或访问控制。未认证的 MCP 端点允许任何可达客户端调用 Skills、读取 Expert Team 输出，若工具具备文件系统或 shell 访问权限则构成远程代码执行面。需在规划前明确：认证方案（API Key / JWT 复用 / OAuth / mTLS）、授权模型（按 skill / team / tenant）、限流策略。（ce-doc-review 延期，security-lens，置信度 100）
|
||||
- **[P0 安全] 多端消息适配器缺少输入验证**：R9 多平台消息接入（飞书/钉钉/企业微信），F3 未提及签名校验、来源认证或速率限制。外部平台消息为不可信输入边界，缺少验证允许伪造消息、注入恶意内容、触发未授权 Skill 执行。需在规划前明确：各平台签名机制（飞书 encrypt_key、钉钉 token、企微信 EncodingAESKey）、消息格式校验、重放攻击防护。（ce-doc-review 延期，security-lens，置信度 100）
|
||||
- **[P0 安全] 文档上传缺少内容净化**：R1/R7 涉及用户上传文档（PDF/Word/Markdown/TXT），需求未提及内容净化、恶意文件检测或大小限制。上传文档可能包含恶意脚本（XSS via markdown）、超大文件导致 OOM、嵌入恶意宏。需在规划前明确：文件类型白名单、大小限制、内容扫描、markdown sanitize、PDF 解析安全。（ce-doc-review 延期，security-lens，置信度 100）
|
||||
|
||||
### Deferred to Planning
|
||||
|
||||
- 飞书消息接入适配器与现有飞书 RAG 适配器（`memory/adapters/feishu.py`）的复用程度——后续确认。
|
||||
- RAG 平台数据模型（Knowledge/Document/Paragraph/Problem/Embedding）的具体设计——ce-plan 决策。
|
||||
- Celery 与现有 asyncio 原生的共存策略——ce-plan 决策。
|
||||
- LiteLLM 与现有自研 Provider 的迁移路径——ce-plan 决策。
|
||||
- **[P1 战略] 对标 MaxKB ≠ 竞争力**：功能对等是底线而非差异化。需明确 AgentKit 相对 MaxKB 的差异化定位（专家团队/自进化/终端安全等独有能力如何与 RAG 平台协同）。（ce-doc-review 延期，product-lens，置信度 75）
|
||||
- **[P1 前提] 演进前提缺少用户痛点证据**：文档以"对标 MaxKB 发现差距"为前提，但未提供用户痛点证据（用户反馈/流失原因/竞品丢失原因）。需确认：是否有用户因缺少 RAG/多端/MCP Server 而流失或抱怨？（ce-doc-review 延期，product-lens，置信度 75，ROOT）
|
||||
- **[P1 前提] 未评估"不做"基线**：未评估不执行此演进计划的后果。需确认：若不演进，AgentKit 的实际损失是什么？（ce-doc-review 延期，adversarial，置信度 75，DEPENDENT）
|
||||
- **[P1 战略] 同质化追赶 vs 差异化构建**：RAG/多端/MCP Server 是 commodity 能力，MaxKB 已有。需确认：投入大量资源追赶 commodity 是否优于强化差异化能力？（ce-doc-review 延期，product-lens，置信度 75）
|
||||
- **[P1 战略] 重建 MaxKB 缺乏正当性论证**：RAG 平台并行独立意味着重建 MaxKB 已有的工业级管道。需确认：为何不直接集成 MaxKB 或 fork？（ce-doc-review 延期，adversarial，置信度 75，ROOT）
|
||||
- **[P1 基础设施] 并行 RAG 平台导致基础设施翻倍**：独立 RAG 平台与现有 `memory/` 共用 pgvector 但数据模型独立，可能导致维护两套 RAG 基础设施。需确认：长期是否合并？（ce-doc-review 延期，product-lens，置信度 75，DEPENDENT）
|
||||
- **[P1 设计] RAG 平台信息架构未定义**：RAG 平台的前端信息架构（知识库列表/文档管理/检索测试/配置页）未定义。归 ce-plan 设计。（ce-doc-review 延期，design-lens，置信度 75）
|
||||
- **[P1 设计] 多端配置流程缺失**：R9 多端消息接入缺少配置流程（如何添加平台/配置凭证/测试连通性）。归 ce-plan 设计。（ce-doc-review 延期，design-lens，置信度 75）
|
||||
- **[P1 安全] 知识库访问控制未指定**：R8 可视化文档管理未指定访问控制（谁可查看/编辑/删除知识库）。需明确 RBAC 模型。（ce-doc-review 延期，security-lens，置信度 75）
|
||||
- **[P1 安全] 适配器凭证管理未定义**：R9 多端适配器需要管理各平台 API 凭证（app_id/app_secret/token），需求未提及凭证存储与轮换。归 ce-plan 设计。（ce-doc-review 延期，security-lens，置信度 75）
|
||||
- **[P1 安全] MCP 发布授权未指定**：R10 MCP Server 发布流程未指定谁有权发布 MCP 工具。需明确发布权限模型。（ce-doc-review 延期，security-lens，置信度 75）
|
||||
- **[P1 技术] PG 全文检索对中文不适用**：R2 依赖 PostgreSQL 全文检索，但 PG 原生全文检索对中文支持差（缺中文分词）。需确认：是否使用 pg_jieba/zhparser 扩展或外部搜索引擎？（ce-doc-review 延期，feasibility，置信度 75）
|
||||
- **[P1 优先级] R12 优先级阻断 P1 交付**：R12（Celery）在 P3，但 R1-R8（RAG 管道）的文档向量化需要异步任务能力。需确认：P1 是否依赖 R12？（ce-doc-review 延期，scope-guardian，置信度 75）
|
||||
- **[P2 证据] Celery 替换缺少必要性证据**：R12 引入 Celery 但未提供 asyncio 原生不足的证据（具体场景/性能瓶颈/故障案例）。需补充必要性论证。（ce-doc-review 延期，feasibility，置信度 75）
|
||||
- **[P2 量化] LiteLLM 节省未量化**：R13 引入 LiteLLM 但未量化节省（维护代码量/开发效率/兼容 provider 数）。需补充量化数据。（ce-doc-review 延期，feasibility，置信度 75）
|
||||
- **[P2 设计] MCP 配置流程未指定**：R10 MCP Server 发布流程缺少配置细节（endpoint 路径/工具命名/参数定义）。归 ce-plan 设计。（ce-doc-review 延期，design-lens，置信度 75）
|
||||
- **[P2 设计] 分块预览未定义**：R3 分段预览的交互模式未定义（预览界面/编辑能力/重新分段）。归 ce-plan 设计。（ce-doc-review 延期，design-lens，置信度 75）
|
||||
- **[P2 技术] 消息总线替换未处理**：现有 `bus/`（MemoryBus/RedisBus）在生态替换中未提及。需确认：是否保留自研？（ce-doc-review 延期，feasibility，置信度 75）
|
||||
- **[P2 技术] Rerank 模型未处理**：R7 rerank 未指定模型（本地/API/开源）。归 ce-plan 决策。（ce-doc-review 延期，feasibility，置信度 75）
|
||||
- **[P1 战略] MaxKB 对等框架解决错误问题**：MaxKB 是 RAG 知识库产品，AgentKit 是 Agent 平台。对标不同产品类别的功能对等可能构建不服务于实际用户的能力。需重构为用户结果导向的成功标准。（ce-doc-review 第 2 轮延期，product-lens+adversarial，置信度 100）
|
||||
- **[P1 战略] 串行策略饿死差异化投入**：串行执行意味着差异化能力（Agent 引擎/专家团队/自进化）在 P1/P2 完成前零投入。需考虑预留差异化并行轨道。（ce-doc-review 第 2 轮延期，product-lens，置信度 75）
|
||||
- **[P1 战略] Build-vs-buy 未评估**：R1-R8 从零构建工业级 RAG 管道，未评估集成 MaxKB 或采用 RAG 框架（LlamaIndex/Haystack）。需补充 build-vs-buy 评估。（ce-doc-review 第 2 轮延期，product-lens+adversarial，置信度 100）
|
||||
- **[P1 前提] 核心替换向后兼容性是假设非验证**：R11/R12/R13 替换三个核心组件但假设"现有功能行为不变"。现有 LLM 网关有 6 provider + fallback + 语义缓存 + RemoteLLMProvider 代理。需将向后兼容从假设转为验证前提。（ce-doc-review 第 2 轮延期，adversarial，置信度 75）
|
||||
- **[P1 设计] 文档处理失败状态缺失**：F1 未定义解析失败/不支持格式/向量化错误的用户可见状态。归 ce-plan 设计。（ce-doc-review 第 2 轮延期，design-lens，置信度 75）
|
||||
- **[P1 设计] 分段预览交互模式未定义**：R3 "查看分段结果"是只读还是可编辑（合并/拆分/重新分段）未定义。归 ce-plan 设计。（ce-doc-review 第 2 轮延期，design-lens，置信度 75）
|
||||
- **[P1 设计] 多端配置与认证流程缺失**：F3 缺少多端 onboarding 流程（webhook 配置/OAuth/app 凭证/连通性测试）。归 ce-plan 设计。（ce-doc-review 第 2 轮延期，design-lens，置信度 75）
|
||||
- **[P2 战略] 维护成本痛点延期至 P3**：Problem Frame 声明"commodity 层维护成本高"但解决方案在最低优先级 P3。需考虑将高杠杆替换（如 LiteLLM）提前并行。（ce-doc-review 第 2 轮延期，product-lens，置信度 75）
|
||||
- **[P2 战略] R11-R13 是技术债非产品需求**：R11-R13 成功标准是"现有功能行为不变"（零用户可见影响），无 Actor 受益。需考虑移至独立工程债轨道。（ce-doc-review 第 2 轮延期，product-lens，置信度 75）
|
||||
- **[P2 战略] 门户触达(P2)反转门户价值主张**：门户平台的核心价值是触达，但多端接入在 P2。需考虑在 P1 并行交付至少一个高价值渠道。（ce-doc-review 第 2 轮延期，product-lens，置信度 75）
|
||||
- **[P2 技术] Celery 缺乏必要性论证**：R12 引入 Celery 但未论证 asyncio 不足。文档向量化是 CPU-bound（可用 ProcessPoolExecutor），批量任务是 I/O-bound（asyncio 强项）。需补充具体场景。（ce-doc-review 第 2 轮延期，scope-guardian+adversarial，置信度 75）
|
||||
- **[P2 技术] LiteLLM 替换覆盖缺口**：R13 未评估 LiteLLM 对 6 个现有 provider（尤其 Doubao/Wenxin/Yuanbao）的覆盖，以及语义缓存/用量追踪/RemoteLLMProvider 代理等网关特性。需补充 feature-gap 分析。（ce-doc-review 第 2 轮延期，scope-guardian+adversarial，置信度 75）
|
||||
- **[P2 战略] 串行策略阻断 MCP Server**：R10 完善现有 MCP 基础设施，无依赖 RAG 或多端。串行策略将其阻断在两个无关工作流之后。需考虑解耦 R10。（ce-doc-review 第 2 轮延期，adversarial，置信度 75）
|
||||
- **[P2 设计] MCP 发布配置细节未定义**：F4 "配置 MCP endpoint"未定义配置字段（工具名称/描述/输入 schema/鉴权方式/发布前测试）。归 ce-plan 设计。（ce-doc-review 第 2 轮延期，design-lens，置信度 75）
|
||||
- **[P2 设计] 新 RAG 平台门户 IA 未定义**：R1 新建独立 RAG 平台模块，但未定义其在门户导航中的位置（顶级 section 还是扩展现有知识库管理区）。归 ce-plan 设计。（ce-doc-review 第 2 轮延期，design-lens，置信度 75）
|
||||
|
||||
## Sources / Research
|
||||
|
||||
- MaxKB 系统架构：https://maxkb.cn/docs/v1/system_arch/
|
||||
- MaxKB 技术解析：https://juejin.cn/post/7650428235188420651
|
||||
- MaxKB GitHub：https://github.com/1Panel-dev/MaxKB
|
||||
- AgentKit 代码库：`memory/`（RAG 基础组件）、`server/routes/`（22 路由模块）、`src/agentkit/server/frontend/src/components/`（前端组件）
|
||||
- AgentKit 项目规则：`AGENTS.md`、`CLAUDE.md`
|
||||
|
|
@ -0,0 +1,868 @@
|
|||
---
|
||||
title: "feat: 多维表格（Bitable）伴生服务 v1"
|
||||
status: active
|
||||
date: 2026-06-24
|
||||
deepened: 2026-06-24
|
||||
type: feat
|
||||
origin: docs/brainstorms/2026-06-24-bitable-module-requirements.md
|
||||
---
|
||||
|
||||
# 多维表格（Bitable）伴生服务 v1 实现规划
|
||||
|
||||
## Summary
|
||||
|
||||
为 AgentKit 引入多维表格伴生服务，作为异构数据源（Excel/数据库/爬虫API）的统一持久化落地载体。Agent 是数据的主要作者（采集写入），用户在落地后的表上精修、配视图、做分析。v1 验证"采集→落地→网格视图→基础公式→附件"核心闭环。
|
||||
|
||||
本服务逻辑独立（自有 API/CLI/领域模型/存储），当前共部署、UI 级集成，未来可零成本抽离。
|
||||
|
||||
## Problem Frame
|
||||
|
||||
AgentKit 缺少统一的持久化结构化数据落地载体。Excel 导出是单向的（`src/agentkit/documents/renderers/excel_renderer.py`），Excel 解析只转文本进 RAG（`src/agentkit/memory/document_loader.py`），`SharedWorkspace` 是带 TTL 的临时 KV。没有模块能把异构数据源的结构化数据持久化为可编辑、可视图、可计算的多维表格。
|
||||
|
||||
详见 origin: `docs/brainstorms/2026-06-24-bitable-module-requirements.md`。
|
||||
|
||||
## Requirements
|
||||
|
||||
源自需求文档 v1 范围：
|
||||
|
||||
| ID | 需求 | 来源 |
|
||||
|----|------|------|
|
||||
| R1 | 服务骨架：领域模型（表/字段/记录/视图）、API/CLI、独立 schema 存储 | 需求文档 §5 v1 |
|
||||
| R2 | 字段所有权模型 + 按主键 upsert 语义（数据列归 Agent，用户列保留） | 需求文档 §4.1 |
|
||||
| R3 | 三类采集落地（Excel 上传/URL、数据库导入、爬虫/API 采集） | 需求文档 §2 |
|
||||
| R4 | 网格视图（排序/筛选/分页/单元格编辑） | 需求文档 §5 v1 |
|
||||
| R5 | 基础公式列（算术/字符串/SUM/AVG/COUNT）+ 基础引用列（lookup） | 需求文档 §4.2 |
|
||||
| R6 | 图片/附件字段类型（复用现有文件上传能力） | 需求文档 §5 v1 |
|
||||
| R7 | 异步公式重算 + "计算中"状态标记 | 需求文档 §4.2 |
|
||||
| R8 | 伴生服务架构：API/CLI 调用边界，不做进程内紧耦合 | 需求文档 §3 |
|
||||
|
||||
**成功标准**（源自需求文档 §9）：Agent 能把 Excel/DB/API 数据写入多维表格；用户能编辑单元格、新增公式列、看到异步重算结果；重复采集时按主键 upsert 保留用户列；服务通过 API/CLI 被调用。
|
||||
|
||||
---
|
||||
|
||||
## Key Technical Decisions
|
||||
|
||||
### KTD1: 存储选用 PostgreSQL（非 SQLite），跟随 evolution/memory 模式
|
||||
|
||||
现有伴生子系统（calendar/documents/auth）用 SQLite + 独立 `.db` 文件。bitable **偏离此模式**，改用 PostgreSQL + 独立 schema，跟随 `src/agentkit/evolution/pg_store.py` 和 `src/agentkit/memory/models.py` 的 PostgreSQL 模式。
|
||||
|
||||
**理由**：需求文档要求可演进到单表 10万+行 + 并发写入（Agent 采集 + 用户编辑同时）。SQLite 的并发写锁和单文件规模是硬瓶颈。PostgreSQL 的 JSONB 查询能力、行级并发、索引支持是 bitable 的刚需。
|
||||
|
||||
**代价**：bitable 要求部署环境配置 PostgreSQL（不像 calendar/documents 开箱即用 SQLite）。这是可接受的——需求文档已明确"共享 PG + 独立 schema"。
|
||||
|
||||
**模式参考**：`src/agentkit/evolution/pg_store.py`（PGBase 独立 + 延迟初始化 + 锁防并发）、`src/agentkit/memory/models.py`（SQLAlchemy 2 declarative + JSONB + pgvector）。
|
||||
|
||||
### KTD2: 存储模型——字段定义表 + 记录表（JSONB 存值）
|
||||
|
||||
不用 EAV（一行一单元格，100k×20=200万行太慢），不用动态列（加列要 DDL）。采用：
|
||||
|
||||
- **字段定义表** `bitable_fields`：每行一个字段定义（名称、类型、配置、所有权）
|
||||
- **记录表** `bitable_records`：每行一条记录，`values` 列为 JSONB（`{field_id: value}`）
|
||||
|
||||
这是 Airtable/飞书多维表格的标准模式。JSONB 支持 GIN 索引和 `->>` 查询，兼顾灵活性与查询性能。加列/删列只改字段定义表，不动记录表结构。
|
||||
|
||||
### KTD3: 公式引擎——自研 Python 轻量引擎
|
||||
|
||||
不引入 HyperFormula（商业付费）、pycel（GPL 传染风险）、formulas（EUPL 边界模糊）。自研，因为 v1 函数集小（10-50 个）。
|
||||
|
||||
**架构**：`ast`/`pyparsing` 解析公式为 AST → 构建 DAG（字段依赖关系）→ Kahn 算法拓扑排序 → DFS 检测循环引用 → 增量重算（仅重算受影响下游）。
|
||||
|
||||
**重算策略**：数据列写入 → 标记依赖该列的公式列为"计算中" → 异步队列按拓扑序重算 → 结果写回记录 JSONB → 状态置"完成"。
|
||||
|
||||
`ponytail:` 自研引擎的 O(V+E) 拓扑重算在万级公式单元格下足够；若未来公式量到十万级或需 Excel 100% 兼容，升级路径为迁移到 Univer 引擎（Apache-2.0，免费商用）。
|
||||
|
||||
### KTD4: 网格视图组件——vxe-table（MIT）
|
||||
|
||||
不选 Handsontable（商业付费）、ag-grid Enterprise（付费功能）、a-table 裸用（10k+ 行无虚拟滚动）。选 vxe-table：Vue 3 原生 + TS、MIT、横向+纵向虚拟滚动、可编辑 CRUD、自定义渲染器（插槽实现附件/图片/公式列）。
|
||||
|
||||
公式列由后端计算后回填值，前端只渲染（不前端算公式）。
|
||||
|
||||
### KTD5: 服务边界——REST API 即使共部署也走 HTTP
|
||||
|
||||
需求文档要求"API/CLI 调用边界，不做进程内紧耦合"。即使 bitable 与 AgentKit 共进程部署，Agent 调用 bitable 也走 localhost REST API（`/api/v1/bitable/*`），而非直接 import service 类。
|
||||
|
||||
**理由**：满足伴生服务契约，未来抽离为零成本。代价是本地 HTTP 往返开销（可忽略）。
|
||||
|
||||
**例外**：CLI 命令（`agentkit bitable ...`）可直接调用 service 层（CLI 是运维工具，不是运行时调用路径）。
|
||||
|
||||
### KTD6: 字段所有权——field 元数据 `owner` 字段 + 自动推断
|
||||
|
||||
`bitable_fields` 表增加 `owner` 列（`agent` | `user`）。自动推断规则：公式列/引用列/手动标注列 → `user`；Agent 采集写入的列 → `agent`。Agent 采集时可显式声明覆盖推断。
|
||||
|
||||
upsert 时只更新 `owner=agent` 的字段值，`owner=user` 的字段值原样保留。
|
||||
|
||||
### KTD7: 公式引擎安全约束——受限 AST walker + 白名单节点
|
||||
|
||||
`ast.parse` 后**禁止直接 `eval()`**。必须实现受限 AST walker，仅允许白名单节点类型：`Expression`、`BinOp`、`UnaryOp`、`BoolOp`、`Compare`、`Call`（仅已注册函数）、`Name`（仅字段引用）、`Constant`、`IfExp`。
|
||||
|
||||
**禁用节点**：`Attribute`（防 `__import__`）、`Subscript`、`Lambda`、`Import`/`ImportFrom`、`Assign`/`AugAssign`、`For`/`While`、`FunctionDef`/`ClassDef`、`Subscript`、`Await`、`Yield`。遇到禁用节点立即抛出 `FormulaSecurityError`。
|
||||
|
||||
**理由**：公式字符串来自用户输入和 Agent 输出，是信任边界。`ast.eval` 的 `eval` 模式仍允许 `__builtins__` 访问。受限 walker 是唯一安全方案。
|
||||
|
||||
**模式参考**：Python `ast` 模块的 `NodeVisitor` + 白名单校验，类似 bandit 的 AST 检查模式。
|
||||
|
||||
### KTD8: Upsert 用 `jsonb_set` 逐字段合并，禁止整行替换
|
||||
|
||||
upsert 更新 agent 列时，**禁止** `UPDATE ... SET values = :new_values`（整行替换会覆盖 user 列）。必须用 `jsonb_set` 逐字段合并：
|
||||
|
||||
```sql
|
||||
-- ponytail: 逐字段 jsonb_set，O(字段数) per record，万级批量 upsert 可接受
|
||||
UPDATE bitable_records
|
||||
SET values = jsonb_set(values, :field_path, :field_value, true)
|
||||
WHERE id = :record_id
|
||||
```
|
||||
|
||||
对每条记录的每个 agent 列执行一次 `jsonb_set`，或在单条 SQL 中嵌套多个 `jsonb_set`。user 列（`owner=user`）的值绝不出现在 UPDATE 语句中。
|
||||
|
||||
**理由**：整行替换是 upsert 语义破坏的最常见实现错误。`jsonb_set` 逐字段合并是唯一能保证"只更新 agent 列、保留 user 列"的正确实现。
|
||||
|
||||
### KTD9: 记录分页用 cursor-based，非 offset-based
|
||||
|
||||
`GET /tables/{id}/records` 分页用 cursor（`?cursor=...&limit=50`），非 `?offset=0&limit=50`。
|
||||
|
||||
**理由**：offset 分页在 100k 行时深翻页慢（`OFFSET 50000` 仍扫描前 5 万行）。cursor 分页用 `WHERE id > :cursor ORDER BY id LIMIT :limit`，恒定性能。代价是不支持随机跳页（v1 不需要——网格视图是连续滚动）。
|
||||
|
||||
`ponytail:` cursor 分页不支持跳页；未来若需"跳到第 N 页"，升级路径为 keyset + 估算偏移或预计算页索引。
|
||||
|
||||
### KTD10: vxe-table 与 Ant Design Vue CSS 隔离
|
||||
|
||||
vxe-table 引入全局 CSS（`.vxe-*` 前缀），可能与 Ant Design Vue 的 `.ant-*` 样式冲突。隔离策略：
|
||||
|
||||
1. vxe-table 样式通过 `@import` 局部引入到 `BitableGrid.vue` 的 `<style scoped>` 不可行（vxe-table 用全局类），改为在 `main.ts` 中 `import 'vxe-table/lib/style.css'` 且**只在 bitable 路由组件挂载时确保已加载**
|
||||
2. bitable 网格容器用 `.bitable-grid-scope` 包裹，vxe-table 的样式覆盖限定在该 scope 下（`.bitable-grid-scope .vxe-table { ... }`）
|
||||
3. 字体/颜色变量对齐 Ant Design Vue 的 token（`var(--ant-primary-color)` 等），避免视觉割裂
|
||||
|
||||
**理由**：vxe-table 和 Ant Design Vue 都是全局样式注入型组件库，不隔离会导致样式互相污染。
|
||||
|
||||
### KTD11: 内部 Agent→bitable HTTP 服务间认证
|
||||
|
||||
Agent 通过 BitableTool 调用 bitable REST API 时，不走用户 JWT 认证（Agent 无用户会话）。改用**内部服务令牌**：
|
||||
|
||||
- `agentkit.yaml` 配置 `bitable.internal_token`（启动时生成或手动配置）
|
||||
- BitableTool 请求头携带 `X-Internal-Token: <token>`
|
||||
- bitable 路由的 `require_authenticated` 依赖增加内部令牌分支：`Authorization: Bearer <jwt>` **或** `X-Internal-Token: <token>` 二选一
|
||||
- 内部令牌仅授权 bitable 端点，不授权其他 API
|
||||
|
||||
**理由**：伴生服务架构要求 REST API 边界（KTD5），但 Agent 无用户会话。内部令牌是服务间认证的标准模式，比禁用认证安全，比共享 JWT 简单。
|
||||
|
||||
`ponytail:` 内部令牌是静态共享密钥，适合单实例部署；未来多实例/独立部署时升级为 mTLS 或 OAuth2 client credentials。
|
||||
|
||||
---
|
||||
|
||||
## High-Level Technical Design
|
||||
|
||||
### 组件架构
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph AgentKit
|
||||
Agent[Agent Loop] -->|HTTP API| BitableRoutes[Bitable Routes]
|
||||
CLI[agentkit bitable CLI] -->|direct call| BitableService
|
||||
BitableRoutes --> BitableService[BitableService]
|
||||
BitableService --> BitableRepo[Repository]
|
||||
BitableService --> FormulaEngine[Formula Engine]
|
||||
BitableService --> RecalcQueue[Recalc Queue]
|
||||
RecalcWorker[Recalc Worker] -->|consume| RecalcQueue
|
||||
RecalcWorker --> FormulaEngine
|
||||
RecalcWorker --> BitableRepo
|
||||
BitableRepo --> PG[(PostgreSQL\nbitable schema)]
|
||||
end
|
||||
subgraph Ingestion
|
||||
Agent -->|Excel/DB/Crawler| BitableTool[BitableTool]
|
||||
BitableTool -->|HTTP| BitableRoutes
|
||||
end
|
||||
subgraph Frontend
|
||||
GridView[Grid View\nvxe-table] -->|HTTP| BitableRoutes
|
||||
GridStore[Pinia Store] --> GridView
|
||||
end
|
||||
```
|
||||
|
||||
### 数据模型 ERD
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
bitable_tables ||--o{ bitable_fields : has
|
||||
bitable_tables ||--o{ bitable_records : has
|
||||
bitable_tables ||--o{ bitable_views : has
|
||||
bitable_fields ||--o{ bitable_recalc_queue : triggers
|
||||
|
||||
bitable_tables {
|
||||
string id PK
|
||||
string name
|
||||
string description
|
||||
string primary_key_field_id FK
|
||||
string owner_user_id
|
||||
timestamp created_at
|
||||
timestamp updated_at
|
||||
}
|
||||
bitable_fields {
|
||||
string id PK
|
||||
string table_id FK
|
||||
string name
|
||||
string field_type "text/number/date/select/attachment/image/formula/lookup"
|
||||
jsonb config "options, formula_expr, lookup_target"
|
||||
string owner "agent|user"
|
||||
timestamp created_at
|
||||
}
|
||||
bitable_records {
|
||||
string id PK
|
||||
string table_id FK
|
||||
jsonb values "{field_id: value}"
|
||||
timestamp created_at
|
||||
timestamp updated_at
|
||||
}
|
||||
bitable_views {
|
||||
string id PK
|
||||
string table_id FK
|
||||
string name
|
||||
string view_type "grid|kanban|gantt|gallery|form"
|
||||
jsonb config "filters, sorts, groupings, hidden_fields"
|
||||
timestamp created_at
|
||||
}
|
||||
bitable_recalc_queue {
|
||||
string id PK
|
||||
string table_id FK
|
||||
string record_id FK
|
||||
string field_id FK
|
||||
string status "pending|calculating|done|error"
|
||||
string error_message
|
||||
timestamp queued_at
|
||||
timestamp completed_at
|
||||
}
|
||||
```
|
||||
|
||||
### 公式异步重算流程
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Agent
|
||||
participant API
|
||||
participant Service
|
||||
participant Queue as Recalc Queue
|
||||
participant Worker as Recalc Worker
|
||||
participant Engine as Formula Engine
|
||||
participant DB
|
||||
|
||||
Agent->>API: POST /records (upsert, data columns)
|
||||
API->>Service: upsert_records(table_id, records, pk)
|
||||
Service->>DB: upsert (update agent-owned columns only)
|
||||
Service->>Service: detect affected formula fields (DAG lookup)
|
||||
Service->>DB: mark formula cells "calculating"
|
||||
Service->>Queue: enqueue recalc tasks (record_id, field_id)
|
||||
Service-->>API: 202 Accepted (records saved, formulas calculating)
|
||||
API-->>Agent: 202 + recalc pending count
|
||||
|
||||
loop async
|
||||
Worker->>Queue: dequeue task
|
||||
Worker->>DB: load source values + formula expr
|
||||
Worker->>Engine: evaluate(formula_expr, source_values)
|
||||
Engine-->>Worker: result | error
|
||||
Worker->>DB: write result to record values JSONB
|
||||
Worker->>DB: mark cell "done" | "error"
|
||||
end
|
||||
|
||||
Note over Agent,API: Frontend polls or gets WS update for "done" status
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Scope Boundaries
|
||||
|
||||
### 本次范围内（v1）
|
||||
|
||||
- bitable 伴生服务骨架（领域模型、PostgreSQL schema、REST API、CLI）
|
||||
- 字段所有权模型 + 按主键 upsert 语义
|
||||
- 三类采集落地（Excel/DB/爬虫API）通过 BitableTool
|
||||
- 网格视图（vxe-table，排序/筛选/分页/编辑）
|
||||
- 基础公式列（算术/字符串/SUM/AVG/COUNT）+ 基础引用列（lookup）
|
||||
- 图片/附件字段类型
|
||||
- 异步公式重算 + "计算中"状态
|
||||
|
||||
### 延后（v2/v3，见需求文档 §5）
|
||||
|
||||
- 看板/甘特/画廊/表单视图
|
||||
- 高级公式（日期/条件/跨表 rollup）+ 函数库扩展
|
||||
- 分析能力（分组/透视）
|
||||
- 大规模优化（列式/分区/物化视图）
|
||||
- 多人实时协作
|
||||
|
||||
### 本产品身份之外
|
||||
|
||||
- 不做通用电子表格（字段化记录模型，非单元格自由编辑）
|
||||
- 不做 ETL/数据管道平台（采集是 Agent 按需执行，非定时调度）
|
||||
- 不做 BI 仪表盘产品
|
||||
- 不替代知识库 RAG
|
||||
|
||||
### Deferred to Follow-Up Work
|
||||
|
||||
- bitable 数据导出为 Excel/CSV（现有 `excel_renderer.py` 可后续适配）
|
||||
- bitable 记录的语义检索（pgvector 索引，类似 episodic memory）
|
||||
- 多维表格与 Agent 记忆系统的联动（bitable 作为 episodic memory 的结构化补充）
|
||||
- WebSocket 实时推送公式重算完成事件（v1 用轮询）
|
||||
|
||||
---
|
||||
|
||||
## Implementation Units
|
||||
|
||||
### U1. 领域模型 + PostgreSQL Schema + 服务骨架
|
||||
|
||||
**Goal:** 搭建 bitable 子系统的领域模型、数据库 schema 和服务骨架，为后续所有单元提供基础。
|
||||
|
||||
**Requirements:** R1, R8
|
||||
|
||||
**Dependencies:** 无（基础单元）
|
||||
|
||||
**Files:**
|
||||
- `src/agentkit/bitable/__init__.py`（新建）
|
||||
- `src/agentkit/bitable/models.py`（新建，Pydantic v2 数据模型）
|
||||
- `src/agentkit/bitable/db.py`（新建，PostgreSQL schema + init 函数 + 迁移机制）
|
||||
- `src/agentkit/bitable/repository.py`（新建，数据访问层）
|
||||
- `src/agentkit/bitable/service.py`（新建，业务逻辑层骨架）
|
||||
- `src/agentkit/server/app.py`（修改，lifespan 中初始化 bitable）
|
||||
- `tests/unit/bitable/test_models.py`（新建）
|
||||
- `tests/unit/bitable/test_db.py`（新建）
|
||||
|
||||
**Approach:**
|
||||
- Pydantic 模型：`Table`、`Field`（含 `FieldType` 枚举：text/number/date/select/multiselect/attachment/image/formula/lookup）、`Record`、`View`、`RecalcTask`。所有模型用 `model_config = ConfigDict(...)`。
|
||||
- PostgreSQL schema：5 张表（`bitable_tables`、`bitable_fields`、`bitable_records`、`bitable_views`、`bitable_recalc_queue`）+ 1 张元数据表（`bitable_meta`），置于独立 schema `bitable`。`bitable_records.values` 用 JSONB + GIN 索引。
|
||||
- **主键唯一约束**：`bitable_tables` 的 `primary_key_field_id` 指定的字段在 `bitable_records.values` 中对应值必须唯一。通过在 `bitable_records` 上建函数索引 `CREATE UNIQUE INDEX ... ON bitable_records (table_id, (values->>'{pk_field_id}')) WHERE values ? '{pk_field_id}'` 实现。upsert 按此索引匹配。
|
||||
- **Recalc queue 索引**：`bitable_recalc_queue` 在 `(status, queued_at)` 上建索引（worker 按 status=pending + queued_at 排序消费）；在 `(record_id, field_id)` 上建唯一索引防重复入队。
|
||||
- **Schema 迁移机制**：采用 `src/agentkit/server/auth/models.py` 的 `_SCHEMA_VERSION` 模式。`bitable_meta` 表存 `schema_version`，`init_bitable_db()` 读取当前版本，按版本号顺序执行迁移脚本（`migrations/v1__init.sql`、`migrations/v2__add_index.sql` 等）。首次创建版本=1，后续每次 init 检查版本并执行未应用的迁移。
|
||||
- init 函数 `init_bitable_db()`：参考 `src/agentkit/evolution/pg_store.py` 的延迟初始化 + 锁防并发模式。`CREATE SCHEMA IF NOT EXISTS bitable` + 按版本执行迁移。
|
||||
- service 骨架：`BitableService` 类，注入 repository，暴露后续单元将实现的方法签名。
|
||||
- app.py lifespan：`try/except` 包裹初始化，失败时 `logger.exception` 不崩溃（参考 calendar 子系统初始化模式，`src/agentkit/server/app.py` 第 406-428 行）。
|
||||
|
||||
**Patterns to follow:**
|
||||
- PostgreSQL 模式：`src/agentkit/evolution/pg_store.py`（PGBase 独立、延迟初始化、锁防并发）
|
||||
- SQLAlchemy 2 declarative：`src/agentkit/server/auth/models.py`（`DeclarativeBase` + `Mapped` + `mapped_column` + `_SCHEMA_VERSION` 迁移机制）
|
||||
- JSONB 元数据：`src/agentkit/memory/models.py`（`metadata_` 字段用 JSONB）
|
||||
- 表名安全校验：`src/agentkit/evolution/experience_store.py`（`_SAFE_TABLE_NAME_PATTERN` 防 SQL 注入）
|
||||
|
||||
**Test scenarios:**
|
||||
- Happy path: `init_bitable_db()` 创建 schema 和 6 张表（含 `bitable_meta`），幂等重复调用不报错
|
||||
- Happy path: Pydantic 模型序列化/反序列化 round-trip 正确（Table/Field/Record/View）
|
||||
- Edge case: `Field` 的 `config` JSONB 在不同 field_type 下结构正确（formula 类型有 `formula_expr`，lookup 类型有 `lookup_target`，select 类型有 `options`）
|
||||
- Edge case: `Record.values` JSONB 为空 `{}` 时合法（新记录无值）
|
||||
- Covers 迁移: `bitable_meta.schema_version` 初始为 1；模拟 v2 迁移脚本存在时，第二次 init 执行 v2 迁移并更新版本号
|
||||
- Covers 主键约束: 设置主键字段后，插入两条相同主键值的记录触发唯一约束冲突
|
||||
- Covers 队列去重: `bitable_recalc_queue` 的 `(record_id, field_id)` 唯一索引阻止重复入队
|
||||
- Error path: PostgreSQL 不可用时 `init_bitable_db()` 抛出明确异常，app.py lifespan 捕获后 bitable 降级
|
||||
- Integration: app.py lifespan 初始化后 `app.state.bitable_service` 存在
|
||||
|
||||
**Verification:** `init_bitable_db()` 成功创建 schema + 6 张表 + 迁移版本记录；主键唯一约束生效；Pydantic 模型可正确序列化；app 启动后 bitable service 可用（或降级记日志）。
|
||||
|
||||
---
|
||||
|
||||
### U2. CRUD API + 字段所有权 + Upsert 语义
|
||||
|
||||
**Goal:** 实现 bitable 的 REST API（表/字段/记录/视图 CRUD），含字段所有权模型和按主键 upsert 语义。
|
||||
|
||||
**Requirements:** R1, R2, R8
|
||||
|
||||
**Dependencies:** U1
|
||||
|
||||
**Files:**
|
||||
- `src/agentkit/server/routes/bitable.py`（新建，FastAPI 路由）
|
||||
- `src/agentkit/bitable/service.py`（修改，实现 CRUD + upsert 逻辑）
|
||||
- `src/agentkit/bitable/repository.py`（修改，实现数据访问）
|
||||
- `src/agentkit/server/app.py`（修改，注册路由 `app.include_router(bitable_routes.router, prefix="/api/v1")`）
|
||||
- `tests/unit/bitable/test_service.py`（新建）
|
||||
- `tests/unit/bitable/test_routes.py`（新建）
|
||||
|
||||
**Approach:**
|
||||
- 路由：`router = APIRouter(prefix="/bitable", tags=["bitable"])`，最终前缀 `/api/v1/bitable`。参考 `src/agentkit/server/routes/calendar.py`。
|
||||
- 端点：
|
||||
- 表：`POST /tables`、`GET /tables`、`GET /tables/{id}`、`PATCH /tables/{id}`、`DELETE /tables/{id}`
|
||||
- 字段：`POST /tables/{id}/fields`、`GET /tables/{id}/fields`、`PATCH /fields/{id}`、`DELETE /fields/{id}`
|
||||
- 记录：`POST /tables/{id}/records`（批量插入）、`GET /tables/{id}/records`（cursor 分页+筛选+排序，见 KTD9）、`PATCH /records/{id}`、`DELETE /tables/{id}/records`（批量删除）
|
||||
- 视图：`POST /tables/{id}/views`、`GET /tables/{id}/views`、`PATCH /views/{id}`
|
||||
- 字段所有权：创建字段时 `owner` 默认 `user`；Agent 通过 BitableTool 写入时声明 `owner=agent`。upsert 时只更新 `owner=agent` 的字段值。
|
||||
- **Upsert 实现（KTD8）**：upsert 更新阶段**禁止** `UPDATE ... SET values = :new_values`。必须用 `jsonb_set` 逐字段合并——对每条记录的每个 agent 列执行 `jsonb_set(values, '{field_id}', :value, true)`。user 列值绝不出现在 UPDATE 语句中。批量 upsert 用单事务包裹，失败回滚。
|
||||
- **字段删除依赖检查**：删除字段前检查：(1) 是否有公式字段引用该字段（DAG 反向查找）；(2) 是否是表的主键字段；(3) 是否有视图的 filter/sort 配置引用该字段。有依赖时返回 409 Conflict + 依赖列表，不直接删除。强制删除时（`?force=true`）级联清理：公式字段标记为 error、视图配置移除该字段引用、记录 JSONB 中移除该字段 key（`values - '{field_id}'`）。
|
||||
- **视图过滤翻译**：视图的 `config.filters`（如 `[{field_id, op, value}]`）在查询记录时翻译为 JSONB 查询条件。`op` 支持 `eq`/`ne`/`contains`/`gt`/`lt`/`is_empty`。翻译为 `WHERE values->>'{field_id}' {op_sql} :value`。排序翻译为 `ORDER BY values->>'{field_id}'`。注意：JSONB `->>` 返回 text，number/date 比较需 cast（`CAST(values->>'{field_id}' AS NUMERIC)`）。
|
||||
- 认证：`require_authenticated` 依赖（参考 `src/agentkit/server/auth/dependencies.py`）+ 内部令牌分支（KTD11）。
|
||||
- 服务访问：路由通过 `request.app.state.bitable_service` 获取 service（参考 calendar.py 模式）。
|
||||
|
||||
**Patterns to follow:**
|
||||
- 路由模块：`src/agentkit/server/routes/calendar.py`（`APIRouter` + `app.state` 服务访问 + 503 降级）
|
||||
- 认证依赖：`src/agentkit/server/auth/dependencies.py`（`require_authenticated`、`require_permission`）
|
||||
- 表名/字段名校验：`_SAFE_TABLE_NAME_PATTERN` 模式防注入
|
||||
- JSONB 查询：`src/agentkit/memory/models.py` 的 JSONB 操作模式
|
||||
|
||||
**Test scenarios:**
|
||||
- Happy path: 创建表 → 添加字段 → 插入记录 → 查询记录（cursor 分页）完整流程
|
||||
- Happy path: upsert 模式——首次插入 3 条记录，第二次 upsert 同主键不同数据列值，记录数不变、数据列更新
|
||||
- Covers R2 + KTD8: upsert 时用户列（`owner=user`）值不被覆盖——先手动设置 user 列值，再 upsert，user 列值不变。验证 SQL 层面用 `jsonb_set` 而非整行替换
|
||||
- Covers 字段删除: 删除被公式引用的字段返回 409 + 依赖列表；`?force=true` 后级联清理公式字段状态
|
||||
- Covers 字段删除: 删除主键字段返回 409
|
||||
- Covers 视图过滤: 视图配置 `filter: [{field_id, op: "gt", value: 100}]` 查询时正确过滤 number 字段（CAST 为 NUMERIC）
|
||||
- Edge case: 主键字段未设置时 upsert 报 400
|
||||
- Edge case: 批量插入空数组返回成功且 count=0
|
||||
- Edge case: cursor 分页——第一页返回 next_cursor，第二页用该 cursor 获取后续记录，无更多数据时 next_cursor 为 null
|
||||
- Covers 并发: 两个并发 upsert 同主键不同 agent 列——一个成功一个等待，最终两列都更新（行级锁）
|
||||
- Error path: 表不存在时所有操作返回 404
|
||||
- Error path: 字段类型不匹配（往 number 字段写非数字）的校验行为
|
||||
- Integration: 字段所有权自动推断——Agent 声明 `owner=agent` 的字段，upsert 后该字段更新；`owner=user` 的字段不更新
|
||||
|
||||
**Verification:** API 端点可 CRUD 表/字段/记录/视图；upsert 用 `jsonb_set` 正确保留用户列；字段删除有依赖检查；视图过滤正确翻译为 JSONB 查询；cursor 分页正确。
|
||||
|
||||
---
|
||||
|
||||
### U3. 公式引擎 + 异步重算管道 + 基础引用列
|
||||
|
||||
**Goal:** 实现自研 Python 公式引擎（解析、依赖图、重算）和异步重算管道，支持基础公式（算术/字符串/SUM/AVG/COUNT）和基础引用列（lookup）。
|
||||
|
||||
**Requirements:** R5, R7
|
||||
|
||||
**Dependencies:** U1, U2
|
||||
|
||||
**Files:**
|
||||
- `src/agentkit/bitable/formula/__init__.py`（新建）
|
||||
- `src/agentkit/bitable/formula/parser.py`（新建，公式解析为 AST）
|
||||
- `src/agentkit/bitable/formula/engine.py`（新建，DAG + 拓扑排序 + 求值）
|
||||
- `src/agentkit/bitable/formula/functions.py`（新建，内置函数库）
|
||||
- `src/agentkit/bitable/recalc_worker.py`（新建，异步重算 worker）
|
||||
- `src/agentkit/bitable/service.py`（修改，写入时触发重算入队）
|
||||
- `src/agentkit/server/app.py`（修改，lifespan 启动 recalc worker）
|
||||
- `tests/unit/bitable/test_formula_parser.py`（新建）
|
||||
- `tests/unit/bitable/test_formula_engine.py`（新建）
|
||||
- `tests/unit/bitable/test_recalc.py`（新建）
|
||||
|
||||
**Approach:**
|
||||
- **解析器**（`parser.py`）：用 `ast` 模块解析公式字符串（如 `=SUM({field_abc}) + {field_xyz} * 2`）为 AST。字段引用用 `{field_id}` 语法。支持算术运算符、字符串拼接、函数调用。
|
||||
- **AST 安全约束（KTD7）**：`ast.parse` 后实现受限 `NodeVisitor`，仅允许白名单节点（`Expression`/`BinOp`/`UnaryOp`/`BoolOp`/`Compare`/`Call`仅已注册函数/`Name`仅字段引用/`Constant`/`IfExp`）。禁用 `Attribute`/`Subscript`/`Lambda`/`Import`/`Assign`/`For`/`While`/`FunctionDef`/`ClassDef`/`Await`/`Yield`。遇到禁用节点抛 `FormulaSecurityError`。**禁止** `eval()`/`exec()`。
|
||||
- **引擎**（`engine.py`）：
|
||||
- 构建 DAG：遍历 AST 提取字段依赖，建立字段间依赖图
|
||||
- 拓扑排序：Kahn 算法，确定重算顺序
|
||||
- 循环检测：DFS 检测循环引用，抛出 `CircularReferenceError`
|
||||
- 求值：按拓扑序遍历 AST，解析字段引用（从 record values 读取），调用函数库
|
||||
- **函数库**（`functions.py`）：v1 实现 `SUM`、`AVG`、`COUNT`、`MIN`、`MAX`、`CONCAT`、`ABS`、`ROUND`、`IF`、`LEN`。每个函数注册到 `FUNCTION_REGISTRY`。
|
||||
- **聚合函数语义边界**：`SUM({field_id})` 中 `{field_id}` 引用整列（聚合上下文），返回标量；`{field_id} + 1` 中 `{field_id}` 引用当前记录的值（行上下文），返回标量。**区分规则**：聚合函数的参数为列引用时聚合整列，否则按行求值。`SUM({f1} + {f2})` = 对每行 `f1+f2` 求和（聚合）；`{f1} + SUM({f2})` = 当前行 f1 + f2 列总和（混合）。解析器在 AST 层面标记聚合上下文，引擎按标记决定取列值还是取行值。
|
||||
- **引用列（lookup）**：lookup 字段的 `config.lookup_target = {table_id, field_id, filter_field_id, filter_value}`。求值时从目标表查询匹配记录的指定字段值。复用引擎的字段引用解析机制。lookup 是只读引用，不参与 DAG 环检测（不会形成环）。
|
||||
- **异步重算管道**（`recalc_worker.py`）：
|
||||
- 数据列写入后，service 检测受影响的公式字段（DAG 反向查找下游）
|
||||
- 标记 `bitable_recalc_queue` 为 `pending`，记录 JSONB 中公式字段值置为 `{__status: "calculating"}`。入队时利用 `(record_id, field_id)` 唯一索引去重（已有 pending 任务则跳过，ON CONFLICT DO NOTHING）
|
||||
- worker 从队列消费任务，按拓扑序重算，结果写回 JSONB，状态置 `done`/`error`
|
||||
- **事务边界**：每个 recalc task 的"读取源值→求值→写回结果→标记完成"在单事务中完成，避免读到半更新状态。写回用 `jsonb_set` 只更新公式字段值，不影响其他字段。
|
||||
- **Worker 崩溃恢复**：worker 启动时扫描 `status='calculating'` 的任务（上次崩溃残留），重置为 `pending` 重新入队。app.py lifespan 中 worker 作为 asyncio task 启动，关闭时发 sentinel 优雅停止。
|
||||
- **Reaper 机制**：定时任务（每 5 分钟）扫描 `status='pending'` 且 `queued_at` 超过 10 分钟的任务，重置为 `pending` 并重新入队（防 worker 卡死）。
|
||||
- worker 在 app.py lifespan 中作为 asyncio task 启动，关闭时优雅停止
|
||||
- **异步生成器安全**：若 worker 用 `async def` + `yield` 消费队列，遵守 `return; yield` 模式（见 `.trae/rules/project_rules.md`）。
|
||||
|
||||
**Execution note:** 公式引擎核心逻辑（解析 + AST 安全 + DAG + 求值）建议测试先行——先写解析器、安全约束和循环检测的测试，再实现。
|
||||
|
||||
**Patterns to follow:**
|
||||
- 异步任务生命周期：`src/agentkit/server/app.py` lifespan 中 calendar/evolution 子系统的启动/关闭模式
|
||||
- 异步生成器安全：`.trae/rules/project_rules.md`（`return; yield` 模式）
|
||||
- 队列消费模式：`src/agentkit/experts/team.py` 的 `HandoffTransport`（bounded queue + sentinel 关闭）
|
||||
- AST 安全检查：Python `ast.NodeVisitor` 白名单模式
|
||||
|
||||
**Test scenarios:**
|
||||
- Happy path: `=1+2*3` 解析为 AST 并求值得 7
|
||||
- Happy path: `=SUM({f1})` 其中 f1 列值为 [1,2,3] 求值得 6（聚合上下文）
|
||||
- Happy path: `=CONCAT({f1}, "-", {f2})` 求值得 "a-b"（行上下文）
|
||||
- Happy path: `={f1} + SUM({f2})` 混合上下文——当前行 f1 + f2 列总和
|
||||
- Happy path: lookup 字段从目标表查询匹配记录的值
|
||||
- Covers KTD7 安全: 公式 `=__import__('os')` 抛出 `FormulaSecurityError`（Attribute 节点被禁）
|
||||
- Covers KTD7 安全: 公式 `=(lambda: 1)()` 抛出 `FormulaSecurityError`（Lambda 节点被禁）
|
||||
- Covers KTD7 安全: 公式 `=eval('1+1')` 抛出 `FormulaSecurityError`（Call 节点函数名不在注册表）
|
||||
- Covers 聚合语义: `SUM({f1})` 聚合整列 vs `{f1} + 1` 引用当前行——两者语义不同，测试验证
|
||||
- Edge case: 公式引用不存在的 field_id 报错
|
||||
- Edge case: 空值参与运算的语义（SUM 忽略空值，算术遇空值报错）
|
||||
- Covers R7: 数据列写入后，公式字段进入 "calculating" 状态，worker 重算后变为 "done" 且值正确
|
||||
- Covers 崩溃恢复: 模拟 worker 在 `calculating` 状态崩溃 → 重启后任务重置为 `pending` 并重算成功
|
||||
- Covers 去重: 同一 (record_id, field_id) 并发入队两次 → 队列中只有一个任务（唯一索引）
|
||||
- Covers 事务: recalc task 执行期间另一请求更新源字段 → task 在自己的事务中读到一致快照
|
||||
- Error path: 循环引用 `f1 = f2 + 1`、`f2 = f1 + 1` 抛出 `CircularReferenceError`
|
||||
- Error path: 公式语法错误（括号不匹配）抛出 `FormulaParseError`
|
||||
- Error path: 函数不存在（`=UNKNOWN()`)抛出 `UnknownFunctionError`
|
||||
- Integration: 多条记录批量 upsert 后，所有受影响公式字段被入队重算，最终值正确
|
||||
|
||||
**Verification:** 公式可解析、求值；AST 安全约束阻止注入；聚合函数语义正确；循环引用被检测；异步重算管道正确更新公式值且支持崩溃恢复；lookup 引用列正确跨表取值。
|
||||
|
||||
---
|
||||
|
||||
### U4. Agent BitableTool + 三类采集落地
|
||||
|
||||
**Goal:** 实现 `BitableTool`（Agent 工具），支持三类数据采集写入：Excel 上传/URL、数据库导入、爬虫/API 采集。
|
||||
|
||||
**Requirements:** R3, R8
|
||||
|
||||
**Dependencies:** U2
|
||||
|
||||
**Files:**
|
||||
- `src/agentkit/tools/bitable_tool.py`（新建，`BitableTool(Tool)`）
|
||||
- `src/agentkit/bitable/ingestion/__init__.py`（新建）
|
||||
- `src/agentkit/bitable/ingestion/excel.py`（新建，Excel 解析+写入）
|
||||
- `src/agentkit/bitable/ingestion/database.py`（新建，数据库导入）
|
||||
- `src/agentkit/bitable/ingestion/api_collector.py`（新建，API/爬虫采集）
|
||||
- `src/agentkit/server/app.py`（修改，注册 BitableTool 到 tool_registry）
|
||||
- `tests/unit/bitable/test_bitable_tool.py`（新建）
|
||||
- `tests/unit/bitable/test_ingestion_excel.py`（新建）
|
||||
|
||||
**Approach:**
|
||||
- **BitableTool**：继承 `Tool`（`src/agentkit/tools/base.py`），`input_schema` 用 `action` 枚举区分操作。通过 HTTP 调用 bitable REST API（KTD5：即使共部署也走 HTTP）。
|
||||
- actions: `create_table`、`import_excel`、`import_database`、`collect_api`、`upsert_records`、`query_records`
|
||||
- 工具描述用英文（供 LLM function calling）
|
||||
- **内部认证（KTD11）**：HTTP 请求头携带 `X-Internal-Token: <token>`，token 从 `agentkit.yaml` 的 `bitable.internal_token` 读取。BitableTool 初始化时注入 token。
|
||||
- **批量分块**：`upsert_records` 和 `import_excel`/`import_database` 写入时，单次 HTTP 请求最多 500 条记录（`BATCH_SIZE=500`）。超过则分块发送，每块独立 HTTP 请求。失败时返回已成功块数 + 失败块详情，支持断点续传（调用方传入 `resume_from` 跳过已成功块）。
|
||||
- **Excel 导入**（`excel.py`）：复用 `src/agentkit/memory/document_loader.py` 的 `_parse_xlsx` 逻辑（openpyxl），但改为返回结构化数据（`{sheet_name: [{col: val}]}`）而非 Markdown 文本。支持文件上传和 URL 两种输入。按 sheet 创建表，首行为字段名，自动推断字段类型。
|
||||
- **数据库导入**（`database.py`）：接收连接串 + 表名列表，用 SQLAlchemy 反射读取表结构，按表生成 bitable 表+字段，批量导入数据。字段类型映射（DB int→number、varchar→text、timestamp→date）。
|
||||
- **API 采集**（`api_collector.py`）：Agent 已有爬虫/API 调用能力（工具系统），此模块只负责"把 Agent 采集到的结构化数据按字段写入 bitable"。接收 `{records: [...], field_mapping: {...}}`，调用 upsert API 写入。
|
||||
- **注册**：app.py lifespan 中 `app.state.tool_registry.register(BitableTool(...))`，参考 calendar_tool 注册模式（`src/agentkit/server/app.py` 第 422 行）。
|
||||
|
||||
**Patterns to follow:**
|
||||
- 工具基类：`src/agentkit/tools/base.py`（`Tool` 抽象类 + `safe_execute`）
|
||||
- 工具实现：`src/agentkit/tools/calendar_tool.py`（`action` 枚举 + service 注入 + input_schema）
|
||||
- Excel 解析：`src/agentkit/memory/document_loader.py` 的 `_parse_xlsx`（openpyxl + `MAX_ROWS_PER_SHEET`）
|
||||
- 工具注册：`src/agentkit/server/app.py` lifespan（`tool_registry.register`）
|
||||
|
||||
**Test scenarios:**
|
||||
- Happy path: `import_excel` 上传 .xlsx 文件 → 创建表 + 字段 + 记录，数据正确
|
||||
- Happy path: `import_excel` 提供 URL → 下载 + 解析 + 写入
|
||||
- Happy path: `import_database` 指定表名 → 生成 bitable 表，字段类型映射正确
|
||||
- Happy path: `collect_api` 接收 records + field_mapping → upsert 写入
|
||||
- Covers KTD11 认证: BitableTool 请求头携带 `X-Internal-Token`，无 token 时 bitable API 返回 401
|
||||
- Covers 批量分块: 1200 条记录的 upsert → 分 3 次 HTTP 请求（500+500+200），全部成功
|
||||
- Covers 批量分块: 1200 条记录，第 2 块失败 → 返回已成功 500 条 + 失败详情，`resume_from=500` 续传
|
||||
- Covers R3: 三类采集场景各自端到端写入成功
|
||||
- Edge case: Excel 空表（只有表头无数据行）→ 创建表+字段，0 条记录
|
||||
- Edge case: Excel 合并单元格——仅左上角有值（已知局限，参考 `document_loader.py`）
|
||||
- Edge case: 数据库表无主键——bitable 表自动生成 id 字段作为主键
|
||||
- Error path: Excel 文件格式损坏 → 明确错误信息
|
||||
- Error path: 数据库连接失败 → 错误返回，不创建 bitable 表
|
||||
- Error path: bitable REST API 不可用（503）→ 工具返回明确错误
|
||||
- Integration: Agent 通过 BitableTool 创建表并写入数据后，前端网格视图可查看到数据
|
||||
|
||||
**Verification:** Agent 能通过 BitableTool 执行三类采集写入；内部认证正确；批量分块写入可靠；数据正确落地到 bitable 表。
|
||||
|
||||
---
|
||||
|
||||
### U5a. 前端网格视图 + Store + API 客户端 + 公式轮询
|
||||
|
||||
**Goal:** 实现前端网格视图（vxe-table），支持排序/筛选/cursor 分页/单元格编辑，配套 Pinia store、API 客户端和公式重算状态轮询。
|
||||
|
||||
**Requirements:** R4, R7（前端侧）
|
||||
|
||||
**Dependencies:** U2
|
||||
|
||||
**Files:**
|
||||
- `src/agentkit/server/frontend/src/api/bitable.ts`（新建，`BitableApiClient extends BaseApiClient` + 类型定义）
|
||||
- `src/agentkit/server/frontend/src/stores/bitable.ts`（新建，`defineStore('bitable', ...)`）
|
||||
- `src/agentkit/server/frontend/src/views/BitableView.vue`（新建，主视图，全屏布局）
|
||||
- `src/agentkit/server/frontend/src/components/bitable/BitableGrid.vue`（新建，vxe-table 网格组件）
|
||||
- `src/agentkit/server/frontend/src/components/bitable/TableViewList.vue`（新建，表列表侧栏）
|
||||
- `src/agentkit/server/frontend/src/router/index.ts`（修改，添加 `/agent/bitable` 路由，全屏布局）
|
||||
- `src/agentkit/server/frontend/src/main.ts`（修改，`import 'vxe-table/lib/style.css'`）
|
||||
- `src/agentkit/server/frontend/package.json`（修改，添加 vxe-table 依赖）
|
||||
|
||||
**Approach:**
|
||||
- **API 客户端**（`bitable.ts`）：`BitableApiClient extends BaseApiClient`（`src/agentkit/server/frontend/src/api/base.ts`），`API_BASE = '/api/v1/bitable'`。类型定义同文件（`IBitableTable`、`IBitableField`、`IBitableRecord`、`IBitableView`），参考 `src/agentkit/server/frontend/src/api/calendar.ts` 模式。
|
||||
- **Pinia store**（`bitable.ts`）：Composition API 风格，state（tables、currentTable、fields、records、views、loading、recalcPendingCount），actions（loadTables、selectTable、loadRecords、updateCell、addField、pollRecalcStatus）。参考 `src/agentkit/server/frontend/src/stores/calendar.ts`。
|
||||
- **网格组件**（`BitableGrid.vue`）：vxe-table v4，配置：
|
||||
- 虚拟滚动：`virtualXConfig` + `virtualYConfig`（支持 10k+ 行）
|
||||
- 可编辑单元格：`edit-config`，按字段类型渲染编辑器（text/number/date/select）
|
||||
- 自定义列渲染器：附件/图片列用插槽渲染（U6 实现），公式列显示值或"计算中"标记
|
||||
- 排序/筛选：vxe-table 内置 + 服务端 cursor 分页
|
||||
- **CSS 隔离（KTD10）**：容器用 `.bitable-grid-scope` 包裹，vxe-table 样式覆盖限定在该 scope 下
|
||||
- **公式轮询策略**：store 中 `pollRecalcStatus` action——当记录中存在 `__status: "calculating"` 的公式字段时，每 2s 轮询 `GET /tables/{id}/records?fields=calculating_only`。所有公式字段变为 `done`/`error` 后停止轮询。切换表或组件卸载时清理定时器。
|
||||
- **主视图**（`BitableView.vue`）：左侧表列表 + 右侧网格。**全屏布局**（非 AgentLayout 的 55% 象限），bitable 需要完整宽度展示网格。
|
||||
- **路由**：`/agent` children 中添加 `{ path: 'bitable', name: 'agent-bitable', meta: { title: '多维表格', panel: 'full' }, component: () => import('@/views/BitableView.vue') }`。`meta.panel = 'full'` 表示全屏（需在 AgentLayout 中支持 full panel 类型，或直接用独立 layout）。
|
||||
- **依赖**：`npm install vxe-table`（MIT）。
|
||||
|
||||
**Patterns to follow:**
|
||||
- API 客户端：`src/agentkit/server/frontend/src/api/calendar.ts`（`BaseApiClient` 继承 + 类型同文件 + `isXxx` 类型守卫）
|
||||
- Pinia store：`src/agentkit/server/frontend/src/stores/calendar.ts`（Composition API + 错误中文化 + notification）
|
||||
- 路由注册：`src/agentkit/server/frontend/src/router/index.ts`（`/agent` children + `meta.panel`）
|
||||
- 组件结构：`src/agentkit/server/frontend/src/components/calendar/`（主视图 + 子组件分层）
|
||||
|
||||
**Test scenarios:**
|
||||
- Happy path: 打开 `/agent/bitable` → 显示表列表 → 选择表 → 网格加载数据（cursor 分页）
|
||||
- Happy path: 双击单元格 → 编辑 → 保存 → 数据持久化到后端
|
||||
- Happy path: 点击列头排序 → 服务端排序 → 数据刷新
|
||||
- Happy path: 滚动到底部 → cursor 分页加载下一页
|
||||
- Covers R7 轮询: 公式列显示"计算中"标记 → 2s 后轮询 → 重算完成更新为值 → 轮询停止
|
||||
- Covers KTD10 隔离: vxe-table 样式不污染 Ant Design Vue 组件（视觉检查）
|
||||
- Edge case: 10k 行数据虚拟滚动流畅（无卡顿）
|
||||
- Edge case: 空表（0 条记录）显示空状态
|
||||
- Edge case: 切换表时轮询定时器被清理（无内存泄漏）
|
||||
- Error path: 后端 503（bitable 未初始化）→ 显示降级提示
|
||||
- Integration: Agent 通过 BitableTool 写入数据后，前端刷新可见
|
||||
|
||||
**Verification:** 前端可查看/编辑 bitable 数据；10k 行虚拟滚动流畅；公式列正确显示计算状态并轮询更新；CSS 隔离无污染。
|
||||
|
||||
---
|
||||
|
||||
### U5b. 表/字段管理 UI
|
||||
|
||||
**Goal:** 实现表创建、字段管理（新增/编辑/删除/类型配置）的前端 UI，使用户能自主建表和配置字段（不依赖 Agent 采集）。
|
||||
|
||||
**Requirements:** R1, R2
|
||||
|
||||
**Dependencies:** U5a
|
||||
|
||||
**Files:**
|
||||
- `src/agentkit/server/frontend/src/components/bitable/TableCreateModal.vue`（新建，创建表对话框）
|
||||
- `src/agentkit/server/frontend/src/components/bitable/FieldManagePanel.vue`（新建，字段管理面板）
|
||||
- `src/agentkit/server/frontend/src/components/bitable/FieldConfigForm.vue`（新建，字段配置表单——按类型动态渲染）
|
||||
- `src/agentkit/server/frontend/src/stores/bitable.ts`（修改，添加 createTable、addField、updateField、deleteField actions）
|
||||
- `src/agentkit/server/frontend/src/api/bitable.ts`（修改，补充表/字段 CRUD 方法）
|
||||
|
||||
**Approach:**
|
||||
- **创建表对话框**（`TableCreateModal.vue`）：`a-modal` + `a-form`，输入表名、描述、主键字段名。提交后调用 `POST /tables`。
|
||||
- **字段管理面板**（`FieldManagePanel.vue`）：侧滑面板（`a-drawer`），列出当前表所有字段（名称、类型、owner 标签），支持新增/编辑/删除。删除时调用 API，若返回 409（有依赖）则显示依赖列表确认框。
|
||||
- **字段配置表单**（`FieldConfigForm.vue`）：按 `field_type` 动态渲染配置项：
|
||||
- text/number：无额外配置
|
||||
- date：日期格式
|
||||
- select/multiselect：选项列表（可增删）
|
||||
- formula：公式表达式输入框 + 实时语法校验（调用后端 `POST /bitable/fields/validate-formula`）
|
||||
- lookup：目标表+字段+过滤条件选择器
|
||||
- attachment/image：无额外配置
|
||||
- **owner 标签**：字段列表中 `owner=agent` 显示蓝色"Agent"标签，`owner=user` 显示绿色"用户"标签。
|
||||
|
||||
**Patterns to follow:**
|
||||
- 对话框/抽屉：Ant Design Vue 的 `a-modal`/`a-drawer` 模式
|
||||
- 动态表单：`a-form` + `v-if` 按 type 渲染
|
||||
- 表单校验：`src/agentkit/server/frontend/src/components/calendar/` 中的表单校验模式
|
||||
|
||||
**Test scenarios:**
|
||||
- Happy path: 点击"新建表" → 填写表名+主键字段 → 提交 → 表列表刷新
|
||||
- Happy path: 打开字段管理 → 新增 formula 字段 → 输入公式 → 语法校验通过 → 保存
|
||||
- Happy path: 编辑 select 字段 → 增删选项 → 保存 → 网格中该列下拉选项更新
|
||||
- Covers 字段删除: 删除被公式引用的字段 → 显示 409 依赖列表 → 确认强制删除 → 公式字段标记 error
|
||||
- Edge case: 公式语法错误 → 实时校验显示错误提示，保存按钮禁用
|
||||
- Edge case: lookup 字段配置——选择目标表后加载该表字段列表
|
||||
- Edge case: owner 标签正确显示（agent 蓝色 / user 绿色）
|
||||
- Error path: 网络错误 → notification 提示
|
||||
|
||||
**Verification:** 用户可通过 UI 创建表、管理字段（含公式语法校验）、删除字段（含依赖检查）；字段 owner 标签正确。
|
||||
|
||||
---
|
||||
|
||||
### U5c. 视图配置 UI
|
||||
|
||||
**Goal:** 实现视图管理（创建/切换/配置筛选排序）的前端 UI，使用户能保存和切换不同的数据查看视角。
|
||||
|
||||
**Requirements:** R4
|
||||
|
||||
**Dependencies:** U5a
|
||||
|
||||
**Files:**
|
||||
- `src/agentkit/server/frontend/src/components/bitable/ViewSwitcher.vue`（新建，视图切换器）
|
||||
- `src/agentkit/server/frontend/src/components/bitable/ViewConfigPanel.vue`（新建，视图配置面板——筛选/排序/隐藏字段）
|
||||
- `src/agentkit/server/frontend/src/components/bitable/FilterBuilder.vue`（新建，筛选条件构建器）
|
||||
- `src/agentkit/server/frontend/src/stores/bitable.ts`（修改，添加 createView、updateView、switchView actions）
|
||||
- `src/agentkit/server/frontend/src/api/bitable.ts`（修改，补充视图 CRUD 方法）
|
||||
|
||||
**Approach:**
|
||||
- **视图切换器**（`ViewSwitcher.vue`）：网格顶部 tab 栏，列出当前表的所有视图，点击切换。"+"按钮创建新视图（输入名称+类型，v1 仅 grid）。
|
||||
- **视图配置面板**（`ViewConfigPanel.vue`）：`a-drawer`，三个 tab：筛选、排序、隐藏字段。
|
||||
- 筛选：`FilterBuilder` 组件，支持增删筛选条件（field + op + value），op 选项按字段类型动态变化
|
||||
- 排序：字段选择 + 升序/降序
|
||||
- 隐藏字段：字段列表 checkbox，勾选隐藏
|
||||
- **FilterBuilder**（`FilterBuilder.vue`）：条件列表，每行 `a-select`（字段）+ `a-select`（操作符）+ `a-input`/`a-date-picker`（值）。op 选项：text→eq/ne/contains/is_empty；number→eq/ne/gt/lt/gte/lte；date→eq/gt/lt/between。
|
||||
- 配置变更实时保存到视图 `config`（`PATCH /views/{id}`），网格立即重新查询。
|
||||
|
||||
**Patterns to follow:**
|
||||
- Tab 栏：Ant Design Vue `a-tabs`
|
||||
- 条件构建器：动态表单行 + `a-select` 联动
|
||||
|
||||
**Test scenarios:**
|
||||
- Happy path: 创建视图"高价订单" → 配置筛选 `amount > 1000` → 网格只显示匹配记录
|
||||
- Happy path: 切换视图 → 网格数据按视图配置刷新
|
||||
- Happy path: 配置排序 `created_at desc` → 网格按创建时间倒序
|
||||
- Happy path: 隐藏字段 → 网格中该列消失
|
||||
- Edge case: 筛选 number 字段 → op 选项为 eq/ne/gt/lt（非 contains）
|
||||
- Edge case: 筛选 date 字段 → 值输入为 `a-date-picker`
|
||||
- Edge case: 多个筛选条件 AND 组合
|
||||
- Edge case: 视图配置变更实时保存 + 网格刷新
|
||||
- Error path: 筛选值类型不匹配 → 校验提示
|
||||
|
||||
**Verification:** 用户可创建/切换视图；筛选/排序/隐藏字段配置生效并实时保存；不同字段类型的筛选操作符正确。
|
||||
|
||||
---
|
||||
|
||||
### U6. 图片/附件字段类型
|
||||
|
||||
**Goal:** 实现图片和附件字段类型，复用现有文件上传能力，支持上传、存储、预览。
|
||||
|
||||
**Requirements:** R6
|
||||
|
||||
**Dependencies:** U2, U5a
|
||||
|
||||
**Files:**
|
||||
- `src/agentkit/server/routes/bitable.py`（修改，添加附件上传端点）
|
||||
- `src/agentkit/bitable/service.py`（修改，附件存储逻辑 + 记录删除时附件清理）
|
||||
- `src/agentkit/server/frontend/src/components/bitable/AttachmentCell.vue`（新建，附件单元格渲染）
|
||||
- `src/agentkit/server/frontend/src/components/bitable/ImageCell.vue`（新建，图片单元格渲染，懒加载）
|
||||
- `tests/unit/bitable/test_attachment.py`（新建）
|
||||
|
||||
**Approach:**
|
||||
- **后端**：复用现有上传端点模式（`src/agentkit/server/routes/chat.py` 第 1240-1276 行的 `POST /api/v1/chat/upload`），新建 `POST /api/v1/bitable/tables/{id}/upload`，存储到 `data/uploads/bitable/`。附件值在记录 JSONB 中存为 `[{filename, stored_name, url, size, mime_type}]` 数组。
|
||||
- **记录删除时附件清理**：`DELETE /tables/{id}/records` 和 `DELETE /records/{id}` 时，service 层先读取待删记录的 attachment/image 字段值，删除对应物理文件，再删除记录。用 `try/except` 包裹文件删除（文件丢失不阻断记录删除，只记日志）。
|
||||
- **前端**：`AttachmentCell.vue` 渲染附件列表（文件名+下载链接），`ImageCell.vue` 渲染图片缩略图（点击预览）。作为 vxe-table 的自定义渲染器插槽。
|
||||
- **图片懒加载**：`ImageCell.vue` 用 `IntersectionObserver` 或 vxe-table 的虚拟滚动可见性回调——仅视口内的图片加载 `src`，视口外的用占位图。缩略图用 `?thumbnail=true` 参数请求后端生成缩略图（或前端 CSS 缩放）。
|
||||
- **字段类型**：`attachment`（通用文件）和 `image`（仅图片，前端限制 accept）。
|
||||
|
||||
**Patterns to follow:**
|
||||
- 文件上传：`src/agentkit/server/routes/chat.py`（`MAX_UPLOAD_SIZE` + `data/uploads/` 存储 + 下载端点）
|
||||
- 前端上传组件：`src/agentkit/server/frontend/src/components/kb/DocumentUpload.vue`（`a-upload-dragger`）
|
||||
- 文件展示：`src/agentkit/server/frontend/src/components/chat/messages/FileAttachment.vue`（文件类型识别）
|
||||
|
||||
**Test scenarios:**
|
||||
- Happy path: 上传图片 → 记录 image 字段存为 `[{filename, url, ...}]` → 前端显示缩略图
|
||||
- Happy path: 上传 PDF 附件 → attachment 字段存值 → 前端显示文件名+下载链接
|
||||
- Covers 附件清理: 删除含附件的记录 → 物理文件被删除 → 记录被删除
|
||||
- Covers 附件清理: 删除记录时文件已丢失 → 记录仍被删除（不阻断），日志记录
|
||||
- Covers 懒加载: 10k 行含图片记录 → 仅视口内图片加载，滚动流畅
|
||||
- Edge case: 上传超过大小限制的文件 → 413 错误
|
||||
- Edge case: image 字段上传非图片文件 → 校验拒绝
|
||||
- Edge case: 一条记录的 attachment 字段上传多个文件 → 数组存储多个
|
||||
- Error path: 上传时磁盘满 → 明确错误
|
||||
- Integration: Agent 通过 BitableTool 写入含附件的记录 → 前端正确渲染
|
||||
|
||||
**Verification:** 图片/附件可上传、存储、预览、下载；记录删除时附件文件被清理；图片懒加载保证大表滚动流畅；字段值正确存为 JSONB 数组。
|
||||
|
||||
---
|
||||
|
||||
### U7. CLI 子命令
|
||||
|
||||
**Goal:** 实现 `agentkit bitable` CLI 子命令组，供运维和脚本化操作。
|
||||
|
||||
**Requirements:** R1, R8
|
||||
|
||||
**Dependencies:** U2
|
||||
|
||||
**Files:**
|
||||
- `src/agentkit/cli/bitable.py`（新建，`bitable_app = typer.Typer(...)`）
|
||||
- `src/agentkit/cli/main.py`（修改，`app.add_typer(bitable_app, name="bitable")`）
|
||||
- `tests/unit/bitable/test_cli.py`（新建）
|
||||
|
||||
**Approach:**
|
||||
- 子命令：`list-tables`、`create-table`、`import-excel`、`export-excel`（复用 `excel_renderer.py`）、`query`。
|
||||
- CLI 直接调用 `BitableService`（KTD5 例外：CLI 是运维工具，非运行时调用路径）。
|
||||
- 参考 `src/agentkit/cli/task.py` 的 `task_app` 模式。
|
||||
|
||||
**Patterns to follow:**
|
||||
- CLI 子命令组：`src/agentkit/cli/task.py`（`typer.Typer` + `@app.command`）
|
||||
- 注册：`src/agentkit/cli/main.py`（`app.add_typer`）
|
||||
- 输出格式：Typer + Rich（项目 CLI 已用 Rich）
|
||||
|
||||
**Test scenarios:**
|
||||
- Happy path: `agentkit bitable list-tables` 列出所有表
|
||||
- Happy path: `agentkit bitable create-table --name "测试"` 创建表
|
||||
- Happy path: `agentkit bitable import-excel --file data.xlsx --table "导入表"` 导入 Excel
|
||||
- Edge case: 表不存在时 `query` 报错
|
||||
- Error path: bitable 未初始化时 CLI 报明确错误
|
||||
|
||||
**Verification:** CLI 子命令可执行表管理和数据导入操作。
|
||||
|
||||
---
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
### PG Schema Fixture
|
||||
|
||||
所有涉及 PostgreSQL 的测试用 `@pytest.mark.postgres` 标记。`conftest.py` 提供 `bitable_db` fixture：
|
||||
|
||||
- 每个测试函数前 `DROP SCHEMA bitable CASCADE` + 重新 `init_bitable_db()`，保证隔离
|
||||
- 测试后自动清理（teardown）
|
||||
- PG 不可用时 `pytest.skip("PostgreSQL not available")`，不报错
|
||||
|
||||
**文件**：`tests/unit/bitable/conftest.py`（新建，`bitable_db` fixture + `bitable_service` fixture）
|
||||
|
||||
### 测试标记
|
||||
|
||||
| 标记 | 用途 | 命令 |
|
||||
|------|------|------|
|
||||
| `@pytest.mark.postgres` | 需要 PG 的测试 | `pytest -m postgres` |
|
||||
| `@pytest.mark.integration` | 端到端集成测试 | `pytest -m integration` |
|
||||
| 无标记 | 纯单元测试（公式解析、Pydantic 模型等） | `pytest -m "not postgres and not integration"` |
|
||||
|
||||
### HTTP Mock 策略
|
||||
|
||||
BitableTool 测试（U4）不真实调用 HTTP——用 `httpx.MockTransport` 或 `unittest.mock.patch` mock bitable REST API 响应：
|
||||
|
||||
- Happy path：mock 返回 202 + 正常数据
|
||||
- Error path：mock 返回 503 / 401 / 500
|
||||
- 批量分块：mock 验证请求被正确分块
|
||||
|
||||
**不 mock 的场景**：U2 路由测试用 FastAPI `TestClient` 真实调用（但 service 层可 mock repository）。
|
||||
|
||||
### 工厂 Fixture
|
||||
|
||||
`conftest.py` 提供工厂函数快速创建测试数据：
|
||||
|
||||
- `make_table(name="test_table", ...)` → 创建表 + 返回 table_id
|
||||
- `make_field(table_id, name="f1", field_type="text", owner="agent", ...)` → 创建字段 + 返回 field_id
|
||||
- `make_record(table_id, values={"f1": "value1", ...})` → 创建记录 + 返回 record_id
|
||||
- `make_formula_field(table_id, name="calc", formula_expr="=SUM({f1})", ...)` → 创建公式字段
|
||||
|
||||
### 并发测试
|
||||
|
||||
U2/U3 的并发场景（并发 upsert、并发入队）用 `asyncio.gather` 并发执行多个操作，验证：
|
||||
|
||||
- 并发 upsert 同主键不同列 → 行级锁，最终两列都更新
|
||||
- 并发入队同 (record_id, field_id) → 唯一索引去重，队列中只有一个任务
|
||||
|
||||
### 异步生成器安全测试
|
||||
|
||||
若 recalc worker 用 `async def` + `yield`，测试验证 `return; yield` 模式（项目规则要求）：
|
||||
|
||||
- worker 在队列为空时 `return` → 不触发 `'async for' requires __aiter__` 错误
|
||||
- worker 正常消费 → yield 每个任务
|
||||
|
||||
---
|
||||
|
||||
## Risks & Dependencies
|
||||
|
||||
### 风险
|
||||
|
||||
| 风险 | 影响 | 缓解 |
|
||||
|------|------|------|
|
||||
| 公式引擎循环引用/跨表引用边界 case 多 | 重算错误或死循环 | DFS 循环检测 + 超时保护 + v1 限制跨表引用仅 lookup（只读引用，不形成环） |
|
||||
| 公式 AST 安全约束遗漏 | 代码注入（`__import__`/`eval`） | KTD7 白名单 AST walker + 安全测试覆盖（`=__import__`/`=eval`/`=lambda` 用例） |
|
||||
| vxe-table 虚拟滚动 + 可编辑 + 自定义列三者集成复杂 | 前端开发周期拉长 | U5a 前先做最小原型验证（单表 + 1万行 + 编辑） |
|
||||
| vxe-table 与 Ant Design Vue CSS 冲突 | 样式互相污染 | KTD10 CSS 隔离策略（`.bitable-grid-scope` 包裹 + token 对齐） |
|
||||
| PostgreSQL 部署依赖 | 无 PG 环境无法用 bitable | lifespan 降级处理（参考 calendar 模式），CLI doctor 检查 PG 可用性 |
|
||||
| Agent 通过 HTTP 调用 bitable 的延迟 | 采集写入慢 | 本地 HTTP 延迟可忽略；批量写入用批量端点 + 分块（U4 BATCH_SIZE=500） |
|
||||
| JSONB 记录存储在 100k 行时的查询性能 | 分页/筛选慢 | GIN 索引 + cursor 分页（KTD9）+ v3 评估列式存储 |
|
||||
| Recalc worker 崩溃后任务卡在 calculating | 公式永远不更新 | U3 崩溃恢复（启动时重置 calculating→pending）+ reaper 定时清理 |
|
||||
| 内部令牌泄露 | 未授权访问 bitable API | 令牌存 `agentkit.yaml`（不进 git），未来升级 mTLS（KTD11 ponytail 注明） |
|
||||
| Schema 迁移失败 | bitable 不可用 | U1 采用 `_SCHEMA_VERSION` 模式，迁移在事务中执行，失败回滚 |
|
||||
|
||||
### 依赖
|
||||
|
||||
- PostgreSQL 已配置且可连接（bitable 不像 calendar/documents 有 SQLite 回退）
|
||||
- 现有文件上传基础设施（`data/uploads/` + 上传端点模式）可复用
|
||||
- Agent 工具系统（`src/agentkit/tools/base.py`）可扩展
|
||||
- vxe-table npm 包可安装（MIT，无许可证风险）
|
||||
|
||||
---
|
||||
|
||||
## System-Wide Impact
|
||||
|
||||
- **部署**：bitable 要求 PostgreSQL（与 calendar/documents 的 SQLite 零依赖不同）。部署文档需注明 PG 是 bitable 的硬依赖。`agentkit.yaml` 需新增 `bitable.internal_token` 配置。
|
||||
- **Agent 能力**：Agent 获得"结构化数据落地"能力，可通过 BitableTool 把采集结果持久化为可编辑表格。这改变了 Agent 输出的形态（从纯文本/文件 → 持久化结构化数据）。
|
||||
- **前端**：新增 `/agent/bitable` 路由（全屏布局，非象限），引入 vxe-table 依赖（与现有 Ant Design Vue 并存，KTD10 CSS 隔离）。前端工作量从 1 个单元扩展到 3 个单元（U5a/U5b/U5c）。
|
||||
- **CLI**：新增 `agentkit bitable` 子命令组。
|
||||
- **数据库**：新增 `bitable` schema（6 张表含 `bitable_meta`），不影响现有 schema。采用 `_SCHEMA_VERSION` 迁移机制。
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
- ~~**公式列的"计算中"状态前端感知方式**~~：**已解决**——U5a 采用 2s 间隔轮询，公式字段全 done/error 后停止。WebSocket 推送延后到 v2。
|
||||
- **数据库导入的字段类型映射完整性**：v1 覆盖常见类型（int/varchar/text/timestamp/bool/decimal），复杂类型（json/array/enum）如何处理？实现时按需扩展。
|
||||
- **多用户并发编辑同一记录**：v1 不做多人实时协作（v3），但 Agent 写入和用户编辑可能同时发生。v1 用乐观锁（`updated_at` 版本号）还是 last-write-wins？倾向 last-write-wins（upsert 语义已保证用户列不被覆盖，冲突面小）。实现时确认。
|
||||
- **bitable 全屏路由的 layout 实现**：`meta.panel = 'full'` 需要在 AgentLayout 中新增 full panel 类型，还是为 bitable 用独立 layout（脱离 AgentLayout）？实现时确认——倾向独立 layout（bitable 是伴生服务，UI 独立性更好）。
|
||||
- **公式语法校验端点**：U5b 提到 `POST /bitable/fields/validate-formula` 端点用于前端实时校验。这个端点在 U2 还是 U3 实现？倾向 U3（公式引擎在 U3），U2 只做 CRUD。
|
||||
|
||||
---
|
||||
|
||||
## Sources & Research
|
||||
|
||||
- **本地模式研究**：`src/agentkit/server/app.py`（路由注册 + lifespan）、`src/agentkit/server/auth/models.py`（SQLAlchemy 2 模式）、`src/agentkit/evolution/pg_store.py`（PostgreSQL 模式）、`src/agentkit/cli/task.py`（CLI 模式）、`src/agentkit/server/frontend/src/api/calendar.ts`（API 客户端模式）、`src/agentkit/server/frontend/src/stores/calendar.ts`（Pinia store 模式）、`src/agentkit/tools/calendar_tool.py`（工具实现模式）
|
||||
- **公式引擎选型**：Hyperformula（GPLv3/商业双授权，商业需付费）、formulas（EUPL 1.1）、pycel（GPL-3.0，不推荐）、Univer（Apache-2.0，升级路径）。结论：自研避免许可证风险。
|
||||
- **网格组件选型**：vxe-table（MIT，Vue 3 原生，首选）、ag-grid Community（MIT，但分组/透视需 Enterprise 付费）、Handsontable（商业付费）、Univer（Apache-2.0，完整套件，备选）。结论：vxe-table。
|
||||
- **公式重算算法**：业界标准为"标记脏 cell → 拓扑序增量重算"，Luckysheet/Univer/HyperFormula 均采用此模式。
|
||||
|
|
@ -0,0 +1,521 @@
|
|||
---
|
||||
title: "feat: 长程任务可靠性优化 — 中间件管道、循环检测、并发限制、检查点、状态卸载"
|
||||
status: active
|
||||
date: 2026-06-24
|
||||
type: feat
|
||||
origin: "DeerFlow 2.0 SuperAgent Harness 架构对比分析（2026-06-24 对话）"
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
基于 DeerFlow 2.0（字节跳动开源 SuperAgent Harness，74K+ Stars）的架构对比分析，为 AgentKit 补齐长程任务（持续数分钟到数小时）可靠性的 5 个关键缺口：
|
||||
|
||||
1. **循环检测（LoopDetection）** — ReAct 循环内滑动窗口 hash 检测重复工具调用，消除 ~30% 的 token 浪费。
|
||||
2. **子代理并发限制（SubagentLimit）** — Expert Team 同层并行阶段加 Semaphore，避免 LLM 限流洪峰。
|
||||
3. **主动压缩触发（Headroom 压缩）** — 基于 token 用量预测主动触发压缩，避免单次请求超限。
|
||||
4. **SharedWorkspace Redis 化 + 状态卸载** — 阶段输出主动卸载到 Redis/磁盘，上下文只保留摘要，长程任务 token 降 50%+。
|
||||
5. **中间件管道架构** — 统一中间件协议（洋葱模型），将散落的横切关注点（压缩/计量/安全/循环检测）集中化，并行接入验证后移除旧路径。
|
||||
6. **Pipeline 检查点与断点续跑** — 阶段级 checkpoint，崩溃后可从最后完成阶段恢复。
|
||||
7. **技能加载审计** — 确认 disclosure_level 运行时是否真正按需加载，补齐渐进式加载缺口（若存在）。
|
||||
|
||||
明确**不**做：全盘迁移 LangGraph（自研架构保持灵活）、重写现有编排逻辑、Docker 沙箱默认引入（仅文档化边界）、IM 原生集成（标记为可选插件）、ACP 集成（MCP 已覆盖）。
|
||||
|
||||
## Problem Frame
|
||||
|
||||
DeerFlow 2.0 的核心能力是让 Agent 自主执行持续数小时的复杂工作流而不"迷失"。其工程基础是 14 层有序中间件管道（洋葱模型）+ 文件系统优先的状态管理 + 沙箱隔离 + 循环检测 + 子代理并发限制。
|
||||
|
||||
对照 AgentKit 现状：
|
||||
|
||||
| DeerFlow 能力 | AgentKit 现状 | 差距 |
|
||||
|---|---|---|
|
||||
| 14 层有序中间件管道 | 横切关注点硬编码在 `ReActEngine._execute_loop` 和 `execute_stream` 内，无统一管道 | **缺失** |
|
||||
| LoopDetectionMiddleware（滑动窗口 hash） | ReAct 循环无循环检测，只有 `max_steps` 兜底 | **缺失** |
|
||||
| SubagentLimitMiddleware（max 3/轮） | `asyncio.gather` 同层全部并行，无 Semaphore | **缺失** |
|
||||
| SummarizationMiddleware（主动触发） | 压缩在 ReAct 内被动调用，阈值 8000 tokens 硬编码 | **部分**（被动 vs 主动） |
|
||||
| 文件系统优先状态管理 | SharedWorkspace 默认内存 dict，不持久化 | **缺失** |
|
||||
| Checkpoint（节点级，断点续跑） | PipelineStateManager 只写不读，无 resume 逻辑 | **缺失** |
|
||||
| 技能渐进式加载（先元数据，触发时加载完整） | disclosure_level 字段存在但运行时不使用，启动时一次性全量加载 | **部分**（字段有，逻辑无） |
|
||||
|
||||
关键洞察：AgentKit 的自研编排（拓扑排序 + Board 辩论 + 4 层记忆 + 自演化）比 DeerFlow 更丰富，但**长程任务护栏**（循环检测、并发限制、统一中间件、检查点）是明显短板。补齐这些护栏是让 AgentKit 也能支撑"数小时自主执行"的关键。
|
||||
|
||||
## Requirements
|
||||
|
||||
- **R1**：ReAct 循环（非流式 `_execute_loop` + 流式 `execute_stream`）内加入循环检测，滑动窗口检测最近 N 步的 `(tool_name, arguments_hash)` 重复，触发时注入"你正在重复调用 X，请改变策略"系统消息而非直接中断。
|
||||
- **R2**：`TeamOrchestrator` 同层并行阶段（`asyncio.gather`）加 `asyncio.Semaphore`，默认 max 3 并发，可配置；辩论阶段并行同样加限制。
|
||||
- **R3**：`ContextCompressor` 包装为基于 headroom 预测的主动压缩——当 token 用量 / 模型上限 > 80% 时主动触发压缩，而非固定阈值 8000。
|
||||
- **R4**：`SharedWorkspace` 在 ExpertTeam 创建时传入 Redis client（复用 `app.state.working_redis_client`），阶段输出主动写入 Redis；上下文中只保留摘要 + 引用路径。
|
||||
- **R5**：新增 `core/middleware.py`，定义 `Middleware` 协议（`before`/`after`）+ `MiddlewareChain`（洋葱模型，有序执行）；将现有 ContextCompressor、终端安全、循环检测包装为中间件；并行接入（新旧共存），验证后移除旧路径。
|
||||
- **R6**：新增 `orchestrator/checkpoint.py`，实现 `PipelineCheckpoint`（阶段级，Redis 存储，键 `agentkit:pipeline:checkpoint:{plan_id}:{phase_id}`，TTL 7 天）；`TeamOrchestrator` 在阶段完成后 save checkpoint；新增 `resume(plan_id)` 方法从 checkpoint 恢复。
|
||||
- **R7**：审计 `SkillRegistry` 运行时加载路径，确认 `disclosure_level` 是否真正按需加载；若非，在 `build_skill_system_prompt` 中实现 Level 0 概要注入 + ReAct 循环内 `skill_search` 工具按需加载完整内容。
|
||||
- **R8**：所有优化项配置化，新增 `agentkit.yaml` 的 `pipeline` 节（`checkpoint.enabled`、`max_concurrent_phases`、`loop_detection.window_size` 等），遵循 `ServerConfig.from_dict` 模式。
|
||||
- **R9**：每个优化项附最小自检测试（ponytail 规则），参考 `test_pipeline_state.py` 的 `TestPipelineStateRedis` 类模式（AsyncMock + 直接注入 mock）。
|
||||
|
||||
## Key Technical Decisions
|
||||
|
||||
### KTD1：中间件管道并行接入，不直接替换现有请求流
|
||||
|
||||
**决策**：中间件管道与现有 `ReActEngine._execute_loop` / `execute_stream` 内的横切逻辑并行运行，通过 feature flag 控制（`pipeline.middleware.enabled`），验证稳定后再移除旧路径。
|
||||
|
||||
**理由**：直接替换风险高——`_execute_loop`（行 281-831）和 `execute_stream`（行 833-1504）各有 500+ 行，横切逻辑（压缩、trace、memory、telemetry）深度交织。并行接入允许灰度验证，单点故障不影响现有功能。
|
||||
|
||||
**代价**：短期内代码重复（新旧路径共存），工作量翻倍。可接受——长程任务可靠性是核心能力，不值得为省工作量冒回归风险。
|
||||
|
||||
### KTD2：循环检测不直接中断，注入纠正消息
|
||||
|
||||
**决策**：循环检测触发时，不抛异常或 break 循环，而是向 conversation 注入系统消息"你正在重复调用 {tool_name}，请改变策略"，给 LLM 一次自我纠正机会。连续 2 次检测仍重复则强制中断。
|
||||
|
||||
**理由**：DeerFlow 的 LoopDetectionMiddleware 也是先警告后中断。直接中断会丢失已有上下文，且 LLM 可能只是"卡顿"而非真正死循环（如等待外部资源）。纠正消息是最低成本的恢复机制。
|
||||
|
||||
**代价**：多消耗 1-2 轮 LLM 调用。可接受——比无限循环烧 token 好。
|
||||
|
||||
### KTD3：检查点粒度为阶段级（Phase-level），不做节点级
|
||||
|
||||
**决策**：`PipelineCheckpoint` 只在 `PlanPhase` 完成后保存（`_execute_execution_phase` 行 633 后），保存 `plan_id`、`phase_id`、`phase.result`、`plan.topological_sort()` 当前进度。不做 ReAct 循环内的节点级 checkpoint。
|
||||
|
||||
**理由**：节点级 checkpoint（类似 LangGraph）需要状态模式强约束，与 AgentKit 的 `dict[str, Any]` 上下文模型冲突，实现复杂度高。阶段级已满足"崩溃后从最后完成阶段恢复"的核心需求，覆盖 90% 的长程任务恢复场景。
|
||||
|
||||
**代价**：崩溃在阶段中间时，该阶段需重跑。可接受——单阶段通常 1-5 分钟，重跑成本低于全量重跑。
|
||||
|
||||
### KTD4：SharedWorkspace Redis 化，但不引入磁盘文件系统
|
||||
|
||||
**决策**：`ExpertTeam` 创建时传入 Redis client（复用 `app.state.working_redis_client`），阶段输出写入 Redis。不引入 DeerFlow 式的磁盘文件系统（`/mnt/skills/`、`/workspace/`）。
|
||||
|
||||
**理由**：AgentKit 已有 Redis 基础设施（`PipelineStateRedis` 的 `_safe_redis_call` + Memory fallback 模式成熟），复用成本最低。磁盘文件系统需要容器化部署配合，违反 ponytail（"不引入新依赖"）。
|
||||
|
||||
**代价**：Redis 存储有 TTL 限制（7 天），不适合永久归档。可接受——阶段输出是中间产物，永久归档应由 EpisodicMemory（PG+pgvector）负责。
|
||||
|
||||
### KTD5：技能渐进式加载——先审计，按需实现
|
||||
|
||||
**决策**：U5 首先是审计单元——确认 `disclosure_level` 在运行时是否被使用。若已按需加载，该单元无代码产出（仅审计报告）。若未按需加载，在 `build_skill_system_prompt` 实现 Level 0 概要注入 + ReAct 循环内新增 `skill_search` 工具。
|
||||
|
||||
**理由**：研究显示 `disclosure_level` 字段存在但运行时默认为 1（全量加载），疑似未真正按需加载。但需审计确认后再决定是否实现，避免过度工程。
|
||||
|
||||
**代价**：U5 可能无代码产出。可接受——审计结论本身有价值（确认差距是否存在）。
|
||||
|
||||
---
|
||||
|
||||
## Implementation Units
|
||||
|
||||
### U1. 循环检测（LoopDetection）
|
||||
|
||||
**Goal**：在 ReAct 循环内加入滑动窗口 hash 检测，识别重复工具调用模式，触发纠正消息。
|
||||
|
||||
**Requirements**：R1
|
||||
|
||||
**Dependencies**：无（独立优化项）
|
||||
|
||||
**Files**：
|
||||
- `src/agentkit/core/react.py`（修改：`_execute_loop` 行 388-392 + `execute_stream` 行 948-952 插入检测逻辑）
|
||||
- `tests/unit/test_react_engine.py`（修改：新增 `TestLoopDetection` 类）
|
||||
|
||||
**Approach**：
|
||||
- 在 `ReActEngine.__init__` 增加 `_loop_window: deque = deque(maxlen=5)` 和 `_loop_threshold: int = 2`
|
||||
- 检测逻辑：每步工具调用后，计算 `(tool_name, json.dumps(arguments, sort_keys=True))` 的 hash，存入 `_loop_window`
|
||||
- 若窗口内同一 hash 出现次数 ≥ `_loop_threshold`，向 conversation 注入系统消息
|
||||
- 连续 2 次检测仍重复，抛 `LoopDetectedError`（新增异常，继承 `TaskCancelledError`）
|
||||
- 插入点：`react.py` 行 388（`step += 1` 后）、行 948（流式对应位置）
|
||||
- 配置化：`loop_detection.window_size`、`loop_detection.threshold`（agentkit.yaml `pipeline` 节）
|
||||
|
||||
**Patterns to follow**：`cancellation_token.check()`（行 391）的协作式检查模式——检测但不阻塞，给 LLM 纠正机会。
|
||||
|
||||
**Test scenarios**：
|
||||
- **Happy path**：正常工具调用不触发检测，5 步内不同工具调用，无系统消息注入
|
||||
- **Edge case**：连续 3 次相同工具 + 相同参数 → 第 2 次注入纠正消息，第 3 次抛 `LoopDetectedError`
|
||||
- **Edge case**：相同工具不同参数 → 不触发检测（参数 hash 不同）
|
||||
- **Error path**：`LoopDetectedError` 被 `_execute_loop` 的 try/except 捕获，返回 `ReActResult` 带 `error` 字段
|
||||
- **Integration**：流式路径 `execute_stream` 同样触发检测，通过 SSE 事件 `step` 广播纠正消息
|
||||
|
||||
**Verification**：运行 `python3 -m pytest tests/unit/test_react_engine.py::TestLoopDetection -x -q` 全部通过；模拟重复调用场景验证检测生效。
|
||||
|
||||
---
|
||||
|
||||
### U2. 子代理并发限制（SubagentLimit）
|
||||
|
||||
**Goal**：Expert Team 同层并行阶段加 Semaphore，避免 LLM 限流洪峰。
|
||||
|
||||
**Requirements**：R2
|
||||
|
||||
**Dependencies**：无（独立优化项）
|
||||
|
||||
**Files**：
|
||||
- `src/agentkit/experts/orchestrator.py`（修改：`__init__` 行 75-84 + `execute` 行 205-208 + `_execute_debate_phase` 行 949-955）
|
||||
- `tests/unit/experts/test_team_orchestrator.py`（修改：新增 `TestConcurrencyLimit` 类）
|
||||
|
||||
**Approach**：
|
||||
- `TeamOrchestrator.__init__` 增加 `max_concurrent_phases: int = 3`，创建 `self._phase_semaphore = asyncio.Semaphore(max_concurrent_phases)`
|
||||
- 行 205-208 的 `asyncio.gather` 改为包裹 `_bounded_execute(phase)`：
|
||||
```
|
||||
async def _bounded_execute(phase):
|
||||
async with self._phase_semaphore:
|
||||
return await self._execute_phase(phase, plan)
|
||||
```
|
||||
- 辩论阶段行 949-955 同样包裹 `_bounded_debate_arg(expert)`
|
||||
- 配置化：`pipeline.max_concurrent_phases`（agentkit.yaml），默认 3
|
||||
- `MAX_PHASES = 10`（行 67）保持不变，作为阶段总数上限
|
||||
|
||||
**Patterns to follow**：`chat.py` 行 761 `_MAX_CONCURRENT_TASKS = 4` 的 per-session 并发限制模式。
|
||||
|
||||
**Test scenarios**：
|
||||
- **Happy path**：3 个阶段同层，全部并行执行（semaphore 不阻塞）
|
||||
- **Edge case**：5 个阶段同层，验证最多 3 个同时执行（用 `asyncio.Event` 同步验证并发数）
|
||||
- **Edge case**：`max_concurrent_phases=1` 时退化为串行执行
|
||||
- **Error path**：某阶段失败不影响 semaphore 释放（`async with` 保证释放）
|
||||
- **Integration**：辩论阶段 4 个专家并行，验证 semaphore 限制同样生效
|
||||
|
||||
**Verification**：运行 `python3 -m pytest tests/unit/experts/test_team_orchestrator.py::TestConcurrencyLimit -x -q` 全部通过。
|
||||
|
||||
---
|
||||
|
||||
### U3. 主动压缩触发（Headroom 压缩）
|
||||
|
||||
**Goal**：基于 token 用量预测主动触发压缩，避免单次请求超限。
|
||||
|
||||
**Requirements**：R3
|
||||
|
||||
**Dependencies**：无（独立优化项）
|
||||
|
||||
**Files**：
|
||||
- `src/agentkit/core/compressor.py`（修改：新增 `HeadroomCompressionMiddleware` 类）
|
||||
- `src/agentkit/core/react.py`（修改：`_should_compress` 行 1652-1663 改为基于 headroom 预测）
|
||||
- `tests/unit/test_react_engine.py`（修改：新增 `TestHeadroomCompression` 类）
|
||||
|
||||
**Approach**：
|
||||
- `ContextCompressor` 新增 `model_context_limit: int` 参数（默认 128000，从 LLMConfig 读取）
|
||||
- `_should_compress` 改为：`estimate_tokens(conversation) / model_context_limit > 0.8`
|
||||
- 保留固定阈值 8000 作为下限（小模型场景）
|
||||
- 压缩触发后，将压缩前的 conversation checkpoint 到 SharedWorkspace（与 U4 协同）
|
||||
- 配置化：`pipeline.compression.headroom_threshold`（默认 0.8）、`pipeline.compression.min_tokens`（默认 8000）
|
||||
|
||||
**Patterns to follow**：现有 `_should_compress`（行 1652-1663）的阈值判断模式，只是阈值来源从硬编码改为 headroom 比例。
|
||||
|
||||
**Test scenarios**：
|
||||
- **Happy path**：conversation 100K tokens，model_limit 128K → 触发压缩（100K/128K = 0.78 > 0.8? 否，不触发；调整测试数据为 110K → 0.86 > 0.8 触发）
|
||||
- **Edge case**：conversation 5K tokens，model_limit 128K → 不触发（低于 min_tokens 8000）
|
||||
- **Edge case**：model_limit 8K（小模型），conversation 7K → 触发（7K/8K = 0.875 > 0.8）
|
||||
- **Error path**：压缩器不可用（`is_available()=False`）→ 跳过压缩，记录 warning
|
||||
|
||||
**Verification**：运行 `python3 -m pytest tests/unit/test_react_engine.py::TestHeadroomCompression -x -q` 全部通过。
|
||||
|
||||
---
|
||||
|
||||
### U4. SharedWorkspace Redis 化 + 状态卸载
|
||||
|
||||
**Goal**：ExpertTeam 创建时传入 Redis client，阶段输出主动写入 Redis，上下文只保留摘要。
|
||||
|
||||
**Requirements**：R4
|
||||
|
||||
**Dependencies**：无（独立优化项，但与 U6 中间件管道协同）
|
||||
|
||||
**Files**：
|
||||
- `src/agentkit/experts/team.py`（修改：`__init__` 行 67 接收 `redis_client` 参数）
|
||||
- `src/agentkit/server/routes/chat.py`（修改：`_execute_team_collab` 行 407-410 传入 `app.state.working_redis_client`）
|
||||
- `src/agentkit/experts/orchestrator.py`（修改：`_execute_execution_phase` 行 633 后增加上下文摘要替换）
|
||||
- `tests/unit/test_orchestrator.py`（修改：新增 `TestSharedWorkspaceRedis` 类）
|
||||
|
||||
**Approach**：
|
||||
- `ExpertTeam.__init__` 增加 `redis_client: Any = None` 参数，传给 `SharedWorkspace(redis_client=redis_client)`
|
||||
- `chat.py` 行 407 `ExpertTeam(...)` 传入 `redis_client=app_state.working_redis_client`
|
||||
- `orchestrator.py` 行 633 `workspace.write` 后，将 `phase.result` 的完整内容写入 Redis，上下文中只保留摘要 + 引用路径：
|
||||
```
|
||||
summary = result.get("content", "")[:500] + "..."
|
||||
ref_key = f"{plan.id}/phase/{phase.id}/output"
|
||||
# 后续阶段读取依赖输出时，从 Redis 读取完整内容
|
||||
```
|
||||
- `orchestrator.py` 行 496-501 读取依赖输出的逻辑改为：先从内存 `plan.phases` 读，若不存在从 Redis `workspace.read` 读
|
||||
|
||||
**Patterns to follow**：`PipelineStateRedis._safe_redis_call`（行 183-223）的 Redis + Memory fallback 模式——Redis 失败时降级到内存 dict，不阻断执行。
|
||||
|
||||
**Test scenarios**：
|
||||
- **Happy path**：阶段完成后，Redis 中存在 `{plan_id}/phase/{phase_id}/output` 键，值为完整 result
|
||||
- **Edge case**：Redis 不可用 → 降级到内存 dict，执行不中断（`_safe_redis_call` 模式）
|
||||
- **Edge case**：后续阶段读取依赖输出，内存无（模拟崩溃恢复）→ 从 Redis 读取成功
|
||||
- **Integration**：5 阶段流水线，验证上下文中只保留摘要，完整输出在 Redis
|
||||
|
||||
**Verification**：运行 `python3 -m pytest tests/unit/test_orchestrator.py::TestSharedWorkspaceRedis -x -q` 全部通过；手动验证 Redis 中存在阶段输出键。
|
||||
|
||||
---
|
||||
|
||||
### U5. 技能加载审计与渐进式加载
|
||||
|
||||
**Goal**：审计 `disclosure_level` 运行时使用情况；若未按需加载，实现 Level 0 概要注入 + `skill_search` 工具。
|
||||
|
||||
**Requirements**：R7
|
||||
|
||||
**Dependencies**：无（审计单元，可能无代码产出）
|
||||
|
||||
**Files**：
|
||||
- `src/agentkit/skills/registry.py`（审计：`get` 方法 + 运行时调用路径）
|
||||
- `src/agentkit/chat/skill_routing.py`（审计 + 可能修改：`build_skill_system_prompt` 行 101-126）
|
||||
- `src/agentkit/core/react.py`（可能修改：新增 `skill_search` 工具，参考 `_maybe_add_tool_search` 行 1600-1623）
|
||||
- `tests/unit/test_skill_routing.py`（可能新增：`TestProgressiveSkillLoading` 类）
|
||||
|
||||
**Approach**：
|
||||
- **审计阶段**：
|
||||
1. 确认 `SkillLoader.load_from_skill_md`（`loader.py` 行 89）的 `disclosure_level` 默认值
|
||||
2. 确认 `build_skill_system_prompt`（`skill_routing.py` 行 101-126）是否根据 `disclosure_level` 动态选择 sections
|
||||
3. 确认 ReAct 循环内是否有 `skill_search` 工具（类似 `tool_search`）
|
||||
4. 输出审计报告：`disclosure_level` 是否真正按需加载
|
||||
- **实现阶段（若审计确认未按需加载）**：
|
||||
1. `build_skill_system_prompt` 根据 `disclosure_level=0` 只注入 name + description
|
||||
2. ReAct 循环内新增 `skill_search(query: str)` 工具，从 SkillRegistry 检索匹配 skill 的完整内容
|
||||
3. LLM 调用 `skill_search` 后，完整 skill 内容注入到 conversation
|
||||
- **若审计确认已按需加载**：该单元仅输出审计报告，无代码改动
|
||||
|
||||
**Patterns to follow**：`_maybe_add_tool_search`（`react.py` 行 1600-1623）的 `tool_search` 工具实现模式——动态注入工具 schema，LLM 按需调用。
|
||||
|
||||
**Test scenarios**（若实现阶段触发）：
|
||||
- **Happy path**：`disclosure_level=0` 时，system_prompt 只含 skill name + description
|
||||
- **Happy path**：LLM 调用 `skill_search("research")` → 返回 research skill 的完整 instructions
|
||||
- **Edge case**：`skill_search` 查询无匹配 → 返回"无匹配 skill"提示
|
||||
- **Edge case**：`disclosure_level=1`（默认）→ 行为与现状一致（全量加载），向后兼容
|
||||
|
||||
**Verification**：审计报告完成；若实现，运行 `python3 -m pytest tests/unit/test_skill_routing.py::TestProgressiveSkillLoading -x -q` 全部通过。
|
||||
|
||||
---
|
||||
|
||||
### U6. 中间件管道架构
|
||||
|
||||
**Goal**：定义统一中间件协议（洋葱模型），将横切关注点集中化，并行接入验证后移除旧路径。
|
||||
|
||||
**Requirements**：R5
|
||||
|
||||
**Dependencies**：U1（循环检测迁移为中间件）、U3（压缩迁移为中间件）
|
||||
|
||||
**Files**：
|
||||
- `src/agentkit/core/middleware.py`（新建：`Middleware` 协议 + `MiddlewareChain` + `RequestContext`）
|
||||
- `src/agentkit/core/react.py`（修改：`execute` / `execute_stream` 入口接入中间件链，feature flag 控制）
|
||||
- `src/agentkit/server/routes/chat.py`（修改：ReActEngine 创建时注入中间件链）
|
||||
- `tests/unit/test_middleware.py`（新建：`TestMiddlewareChain` 类）
|
||||
|
||||
**Approach**：
|
||||
- **新建 `core/middleware.py`**：
|
||||
```
|
||||
class RequestContext: # 请求上下文，贯穿中间件链
|
||||
conversation, tools, system_prompt, trajectory, step, token_usage, ...
|
||||
|
||||
class Middleware(Protocol):
|
||||
async def before(self, ctx: RequestContext) -> RequestContext: ...
|
||||
async def after(self, ctx: RequestContext, result: Any) -> Any: ...
|
||||
|
||||
class MiddlewareChain:
|
||||
def __init__(self, middlewares: list[Middleware]): ...
|
||||
async def execute(self, ctx: RequestContext, handler: Callable) -> Any:
|
||||
# 洋葱模型：before 由外到内，after 由内到外
|
||||
```
|
||||
- **初始中间件集**（精简版，6 层，非 DeerFlow 的 14 层）：
|
||||
1. `ThreadDataMiddleware` — 初始化 RequestContext
|
||||
2. `SummarizationMiddleware` — 包装现有 ContextCompressor（U3 的 headroom 压缩）
|
||||
3. `TokenUsageMiddleware` — token 计量
|
||||
4. `LoopDetectionMiddleware` — 包装 U1 的循环检测
|
||||
5. `SubagentLimitMiddleware` — 包装 U2 的并发限制（作用于 Expert Team）
|
||||
6. `TerminalSecurityMiddleware` — 包装现有 6 层终端安全
|
||||
- **并行接入**：
|
||||
- `ReActEngine.__init__` 增加 `middleware_chain: MiddlewareChain | None = None`
|
||||
- `execute` / `execute_stream` 入口：若 `middleware_chain` 存在且 `pipeline.middleware.enabled=True`，走中间件路径；否则走现有路径
|
||||
- `chat.py` 行 1066-1068 ReActEngine 创建时注入中间件链
|
||||
- **迁移顺序**（验证后逐步移除旧路径）：
|
||||
1. 先接入 SummarizationMiddleware + TokenUsageMiddleware（低风险）
|
||||
2. 再接入 LoopDetectionMiddleware + SubagentLimitMiddleware（中风险）
|
||||
3. 最后接入 TerminalSecurityMiddleware（高风险，需充分回归）
|
||||
- 配置化：`pipeline.middleware.enabled`（默认 False，灰度开启）
|
||||
|
||||
**Patterns to follow**：DeerFlow 的洋葱模型——`before` 由外到内、`after` 由内到外，顺序依赖通过 `@Next`/`@Prev` 装饰器声明（首版可用固定顺序列表简化）。
|
||||
|
||||
**Test scenarios**：
|
||||
- **Happy path**：3 个中间件按序执行，`before` 顺序 A→B→C，`after` 顺序 C→B→A
|
||||
- **Edge case**：某中间件 `before` 抛异常 → 后续中间件不执行，`after` 链不触发
|
||||
- **Edge case**：`middleware_chain=None` → 走现有路径，行为不变（向后兼容）
|
||||
- **Integration**：`SummarizationMiddleware` 触发压缩后，`LoopDetectionMiddleware` 仍能检测（中间件间状态通过 RequestContext 传递）
|
||||
- **Integration**：feature flag 开关切换，新旧路径行为一致
|
||||
|
||||
**Verification**：运行 `python3 -m pytest tests/unit/test_middleware.py -x -q` 全部通过；灰度开启中间件路径，运行现有 ReAct 测试套件验证无回归。
|
||||
|
||||
---
|
||||
|
||||
### U7. Pipeline 检查点与断点续跑
|
||||
|
||||
**Goal**：阶段级 checkpoint，崩溃后可从最后完成阶段恢复。
|
||||
|
||||
**Requirements**：R6
|
||||
|
||||
**Dependencies**：U4（SharedWorkspace Redis 化提供存储基础）
|
||||
|
||||
**Files**：
|
||||
- `src/agentkit/orchestrator/checkpoint.py`（新建：`PipelineCheckpoint` 类）
|
||||
- `src/agentkit/experts/orchestrator.py`（修改：`__init__` 注入 checkpoint + `_execute_execution_phase` 行 633 后 save + 新增 `resume` 方法）
|
||||
- `src/agentkit/server/routes/tasks.py`（修改：新增 `POST /api/v1/tasks/{id}/resume` 端点）
|
||||
- `tests/unit/test_pipeline_checkpoint.py`（新建：`TestPipelineCheckpoint` 类）
|
||||
|
||||
**Approach**：
|
||||
- **新建 `orchestrator/checkpoint.py`**：
|
||||
```
|
||||
class PipelineCheckpoint:
|
||||
def __init__(self, redis_client, prefix="agentkit:pipeline:checkpoint"): ...
|
||||
async def save(self, plan_id: str, phase: PlanPhase, plan_status: str) -> None:
|
||||
# 键：{prefix}:{plan_id}:{phase_id}
|
||||
# 值：JSON {phase_id, phase_name, phase_result, phase_status, plan_status, saved_at}
|
||||
# TTL: 7 天（与 PipelineStateRedis._TTL_SECONDS 一致）
|
||||
async def load(self, plan_id: str) -> CheckpointData | None:
|
||||
# 返回最后一个 COMPLETED 阶段的 checkpoint
|
||||
async def list_checkpoints(self, plan_id: str) -> list[CheckpointData]: ...
|
||||
async def clear(self, plan_id: str) -> None: ...
|
||||
```
|
||||
- **复用 `PipelineStateRedis._safe_redis_call` 模式**：Redis 失败降级到内存 dict，不阻断执行
|
||||
- **TeamOrchestrator 接入**：
|
||||
- `__init__` 增加 `checkpoint: PipelineCheckpoint | None = None`
|
||||
- `_execute_execution_phase` 行 633（phase COMPLETED 后）调用 `checkpoint.save(plan.id, phase, plan.status)`
|
||||
- 新增 `async def resume(self, plan_id: str) -> OrchestrationResult`：
|
||||
1. `checkpoint.load(plan_id)` 获取最后完成阶段
|
||||
2. 重建 `TeamPlan`，标记已完成阶段为 COMPLETED
|
||||
3. 从下一未完成阶段继续执行
|
||||
- **API 端点**：`POST /api/v1/tasks/{id}/resume` → 调用 `TeamOrchestrator.resume(plan_id)`
|
||||
- 配置化：`pipeline.checkpoint.enabled`（默认 False）、`pipeline.checkpoint.ttl_seconds`（默认 604800）
|
||||
|
||||
**Patterns to follow**：`PipelineStateRedis`（`pipeline_state.py` 行 178-314）的 Redis + Memory fallback + 键命名 + TTL + JSON 序列化模式。`test_pipeline_state.py` 的 `TestPipelineStateRedis` 类的测试模式（AsyncMock + 直接注入 mock）。
|
||||
|
||||
**Test scenarios**：
|
||||
- **Happy path**：`save(plan_id, phase, "executing")` 后 `load(plan_id)` 返回该 phase 的 checkpoint 数据
|
||||
- **Happy path**：3 个阶段完成，`load` 返回第 3 个阶段的 checkpoint（最后一个 COMPLETED）
|
||||
- **Edge case**：`load` 不存在的 plan_id → 返回 None
|
||||
- **Edge case**：Redis 不可用 → 降级到内存 dict，save/load 仍工作
|
||||
- **Error path**：`save` 时 Redis 异常 → 记录 warning，不阻断阶段执行（`_safe_redis_call` 模式）
|
||||
- **Integration**：`resume(plan_id)` 从 checkpoint 恢复，跳过已完成阶段，执行未完成阶段
|
||||
- **Integration**：checkpoint TTL 过期后 `load` 返回 None
|
||||
|
||||
**Verification**：运行 `python3 -m pytest tests/unit/test_pipeline_checkpoint.py -x -q` 全部通过；手动模拟崩溃恢复场景验证 resume 生效。
|
||||
|
||||
---
|
||||
|
||||
## Scope Boundaries
|
||||
|
||||
### In Scope
|
||||
|
||||
- ReAct 循环内循环检测（U1）
|
||||
- Expert Team 并发限制（U2）
|
||||
- 主动压缩触发（U3）
|
||||
- SharedWorkspace Redis 化 + 状态卸载（U4）
|
||||
- 技能加载审计与渐进式加载（U5）
|
||||
- 中间件管道架构（U6，并行接入）
|
||||
- Pipeline 检查点与断点续跑（U7，阶段级）
|
||||
- 所有优化项的配置化（agentkit.yaml `pipeline` 节）
|
||||
- 每个优化项的最小自检测试
|
||||
|
||||
### Out of Scope
|
||||
|
||||
- 全盘迁移到 LangGraph（明确不建议，自研架构保持灵活）
|
||||
- 重写现有编排逻辑（拓扑排序、Board 辩论、4 层记忆等保持不变）
|
||||
- Docker 沙箱默认引入（仅文档化命令级安全的边界，Docker 沙箱作为可选插件未来考虑）
|
||||
- IM 原生集成（Telegram/Slack/Feishu，标记为可选插件，非本期）
|
||||
- ACP 集成（MCP 已覆盖更通用的协议）
|
||||
- 节点级 checkpoint（KTD3 决策，阶段级已满足核心需求）
|
||||
- DeerFlow 式磁盘文件系统（KTD4 决策，复用 Redis）
|
||||
|
||||
### Deferred to Follow-Up Work
|
||||
|
||||
- 中间件管道的 `@Next`/`@Prev` 装饰器声明顺序依赖（首版用固定顺序列表）
|
||||
- 全局 LLM 并发限制（`LLMGateway` 内 semaphore，本期只做 Expert Team 层）
|
||||
- ReAct conversation 的 checkpoint（本期只做 Pipeline 阶段级）
|
||||
- 中间件路径移除旧路径（需灰度验证 1-2 周后单独执行）
|
||||
- `skill_search` 工具的语义检索（本期若实现，用关键词匹配，语义检索延后）
|
||||
|
||||
---
|
||||
|
||||
## High-Level Technical Design
|
||||
|
||||
### 中间件管道洋葱模型
|
||||
|
||||
```
|
||||
请求进入 ReActEngine.execute()
|
||||
↓
|
||||
MiddlewareChain.execute(ctx, handler)
|
||||
↓
|
||||
┌─ ThreadDataMiddleware.before ─────────────────────────┐
|
||||
│ ┌─ SummarizationMiddleware.before ────────────────┐ │
|
||||
│ │ ┌─ TokenUsageMiddleware.before ─────────────┐ │ │
|
||||
│ │ │ ┌─ LoopDetectionMiddleware.before ────┐ │ │ │
|
||||
│ │ │ │ ┌─ TerminalSecurityMiddleware.before ┐│ │ │ │
|
||||
│ │ │ │ │ handler(ctx) ← ReAct 循环 ││ │ │ │
|
||||
│ │ │ │ └─ TerminalSecurityMiddleware.after ─┘│ │ │ │
|
||||
│ │ │ └─ LoopDetectionMiddleware.after ────────┘ │ │ │
|
||||
│ │ └─ TokenUsageMiddleware.after ─────────────────┘ │ │
|
||||
│ └─ SummarizationMiddleware.after ────────────────────┘ │
|
||||
└─ ThreadDataMiddleware.after ─────────────────────────────┘
|
||||
↓
|
||||
返回结果
|
||||
```
|
||||
|
||||
### 检查点恢复流程
|
||||
|
||||
```
|
||||
TeamOrchestrator.execute(plan)
|
||||
↓
|
||||
每阶段完成后: checkpoint.save(plan_id, phase, plan.status)
|
||||
↓
|
||||
[崩溃发生]
|
||||
↓
|
||||
用户调用 POST /api/v1/tasks/{id}/resume
|
||||
↓
|
||||
TeamOrchestrator.resume(plan_id)
|
||||
├─ checkpoint.load(plan_id) → 获取最后 COMPLETED 阶段
|
||||
├─ 重建 TeamPlan，标记已完成阶段
|
||||
├─ topological_sort() → 找到下一未完成层
|
||||
└─ 从下一层继续执行（与 execute 相同的并行+串行逻辑）
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Risks & Dependencies
|
||||
|
||||
### Risks
|
||||
|
||||
| 风险 | 概率 | 影响 | 缓解 |
|
||||
|---|---|---|---|
|
||||
| 中间件管道重构引入回归 | 中 | 高（ReAct 循环是核心路径） | 并行接入 + feature flag 灰度（KTD1） |
|
||||
| 循环检测误判（合法重复调用被中断） | 低 | 中（用户体验） | 先警告后中断（KTD2），阈值可配置 |
|
||||
| SharedWorkspace Redis 化后性能下降 | 低 | 中（Redis 延迟 vs 内存 dict） | `_safe_redis_call` fallback 到内存，异步写入 |
|
||||
| 检查点序列化大对象导致 Redis 膨胀 | 中 | 低（TTL 7 天自动清理） | phase.result 只存摘要 + 引用，完整内容已在 SharedWorkspace |
|
||||
| U5 审计确认无需改动，单元"无产出" | 中 | 低（审计结论有价值） | KTD5 已明确接受此可能性 |
|
||||
|
||||
### Dependencies
|
||||
|
||||
- **Redis 基础设施**：U4、U7 依赖 Redis 可用（已有 `app.state.working_redis_client`，U4 接入）
|
||||
- **现有测试套件**：U6 中间件管道接入后需运行全量 ReAct/Team 测试验证无回归
|
||||
- **agentkit.yaml 配置**：U1-U7 均需新增 `pipeline` 配置节（U8 统一处理）
|
||||
|
||||
---
|
||||
|
||||
## System-Wide Impact
|
||||
|
||||
### 影响方
|
||||
|
||||
- **终端用户**：长程任务（@team 模式）更稳定，不再因循环/限流崩溃；崩溃后可恢复
|
||||
- **开发者**：新增中间件扩展点，横切逻辑可插拔；配置项增加 `pipeline` 节
|
||||
- **运维**：Redis 使用量增加（checkpoint + SharedWorkspace），需监控内存
|
||||
- **API 消费者**：新增 `POST /api/v1/tasks/{id}/resume` 端点
|
||||
|
||||
### 兼容性
|
||||
|
||||
- 所有优化项默认关闭（feature flag），不影响现有行为
|
||||
- `agentkit.yaml` 新增 `pipeline` 节，旧配置无此节时取默认值
|
||||
- `SharedWorkspace` Redis 化后，内存模式仍可用（fallback）
|
||||
- 中间件管道 `middleware_chain=None` 时走现有路径
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Examples
|
||||
|
||||
- **AE1**：用户发起 @team 任务（5 阶段），阶段 3 因 LLM 限流失败 → 系统自动重试（U2 并发限制避免洪峰），任务完成
|
||||
- **AE2**：用户发起 ReAct 任务，LLM 连续 3 次调用相同工具相同参数 → 第 2 次注入纠正消息，LLM 改变策略，任务完成（U1 循环检测）
|
||||
- **AE3**：用户发起 @team 任务（10 阶段），阶段 7 时服务崩溃 → 用户调用 `/resume`，从阶段 7 继续，任务完成（U7 检查点）
|
||||
- **AE4**：用户发起长对话（50 轮），token 用量接近模型上限 → 系统主动压缩历史，对话继续不中断（U3 headroom 压缩）
|
||||
- **AE5**：开发者新增"审计日志"横切逻辑 → 实现 `Middleware` 协议，注入 `MiddlewareChain`，无需修改 ReActEngine（U6 中间件管道）
|
||||
|
||||
---
|
||||
|
||||
## Sources & Research
|
||||
|
||||
- **DeerFlow 2.0 架构分析**（2026-06-24 对话）：14 层中间件管道、Lead Agent + Sub-Agent 编排、沙箱隔离、长期记忆、技能渐进式加载
|
||||
- **DeerFlow 2.0 源码拆解**（CSDN, 2026-05-10）：14 层 Middleware 严格有序、Sub-Agent 并发编排、结构化记忆
|
||||
- **DeerFlow 2.0 Deep Dive**（guancyxx.cn, 2026-05-07）：Lead Agent + Subagent 模式、SubagentLimitMiddleware、LoopDetectionMiddleware
|
||||
- **AgentKit 现有架构**：`react.py` ReAct 循环、`orchestrator.py` Team 编排、`pipeline_state.py` 状态持久化、`shared_workspace.py` 共享工作区
|
||||
- **外部研究 load-bearing**：是——DeerFlow 的中间件管道设计直接影响了 KTD1（并行接入策略）、KTD2（循环检测不中断）、U6（中间件协议设计）
|
||||
|
|
@ -0,0 +1,508 @@
|
|||
---
|
||||
title: "AgentKit 门户平台整体演进路线"
|
||||
type: feat
|
||||
date: 2026-06-24
|
||||
origin: docs/brainstorms/2026-06-24-portal-platform-evolution-requirements.md
|
||||
---
|
||||
|
||||
# AgentKit 门户平台整体演进路线
|
||||
|
||||
## Summary
|
||||
|
||||
按优先级串行推进 AgentKit 门户平台演进：P1 用 LlamaIndex 构建工业级 RAG 管道 + TaskIQ 异步任务基础；P2 扩展多端消息接入与 MCP Server 认证发布；P3 用 LiteLLM/langchain-mcp-adapters 替换 commodity 层降本。差异化能力（Agent 引擎/专家团队/自进化/终端安全）保持自研，不在本计划范围内。
|
||||
|
||||
## Problem Frame
|
||||
|
||||
AgentKit 定位为企业级统一 AI Agent 门户平台。对标 MaxKB 后发现四方面差距：RAG 管道是开发者级组件非工业级产品（上传端点从未调用向量化，KB 存储仅内存）；平台触达仅有 RAG 数据源适配器无消息适配器；MCP Server 零认证且未支持 Skill/专家团队发布；commodity 层大量自研维护成本高。
|
||||
|
||||
本次演进为预防性演进 + 必备功能补齐。目标是补齐门户平台应有的能力，使 AgentKit 在企业级 AI Agent 平台赛道具备完整竞争力。
|
||||
|
||||
## Requirements
|
||||
|
||||
### RAG 工业级管道（P1）
|
||||
|
||||
R1. 企业用户可上传文档到知识库，文档经 LlamaIndex 管道处理（解析→分段→预览→向量化→索引）后可被检索。现有 `memory/local_rag.py` 的 `LocalRAGService` 对 KB 场景废弃，Agent 记忆（WorkingMemory/EpisodicMemory/SemanticMemory）保留不动。
|
||||
|
||||
R2. 知识库支持双索引检索：pgvector 语义检索 + PostgreSQL 全文检索（jieba 中文分词），提供 embedding/keywords/blend 三种模式。检索模式由企业用户按知识库配置默认值，Agent 运行时可按查询特征覆盖。
|
||||
|
||||
R3. 文档分段支持智能分段与高级分段（LlamaIndex IngestionPipeline），企业用户可在向量化前预览分段结果（只读预览，编辑能力延后）。
|
||||
|
||||
R4. 系统为文档段落自动生成相关问题（LLM-based，参考现有 `memory/contextual_retrieval.py` 的 ContextualChunker 模式），提升检索召回率。
|
||||
|
||||
R5. 系统支持术语表（Termbase），通过 jieba 自定义词典增强中文分词，提升领域术语检索准确率。
|
||||
|
||||
R6. 知识库支持命中处理模式：模型优化模式（LLM 基于检索结果生成回答）与直接回答模式（直接返回匹配段落），按 KB 配置默认模式，Agent 可按查询场景覆盖。
|
||||
|
||||
R7. 检索结果经 rerank 模型重排后返回（API-based reranker，可配置 Cohere Rerank 或 BGE-Reranker）。
|
||||
|
||||
R8. 知识库实施 per-KB 访问控制（owner/authorized-users），文档上传验证文件类型（白名单）、强制大小限制、索引前净化解析内容（markdown sanitize、PDF 解析安全）。Agent 检索限定于调用用户授权的知识库。
|
||||
|
||||
R9. 知识库元数据持久化到 PostgreSQL（KB 源、文档记录、ACL），重启不丢失。现有内存 `KnowledgeSourceStore` 替换为 PG 后端。
|
||||
|
||||
R10. 文档向量化通过 TaskIQ 异步执行（复用现有 Redis 作为 broker），提供进度展示、失败通知与重试、任务历史。文档状态模型：pending → parsing → segmenting → vectorizing → indexed | failed（含 error_message）。
|
||||
|
||||
### 平台触达扩展（P2）
|
||||
|
||||
R11. 系统支持企微/钉钉/飞书/Slack 消息接入。各适配器验证平台签名/token（飞书 encrypt_key、钉钉 token、企微 EncodingAESKey）后处理消息，拒绝未认证请求。
|
||||
|
||||
R12. 平台凭证存储于加密 DB 列（master key 来自环境变量），定义轮换策略与访问审计。现有明文 `ProviderConfig.api_key` 同步迁移。
|
||||
|
||||
R13. MCP Server 合并至主 FastAPI app（`/api/v1/mcp/` 路由），所有端点要求认证与授权（复用 `require_permission` + API Key）。现有独立 `mcp/server.py` 重构为路由工厂。
|
||||
|
||||
R14. 企业用户/开发者可将 Skill/专家团队发布为 MCP 工具（配置：工具名称、描述、输入 schema、鉴权方式、速率限制），发布需管理员级授权。外部 AI 系统通过 MCP 协议认证调用。
|
||||
|
||||
### 生态替换降本（P3）
|
||||
|
||||
R15. LLM Provider 底层替换为 LiteLLM：6 个直接 API provider（OpenAI/Anthropic/Gemini/Doubao/Wenxin/Yuanbao）走 LiteLLM 统一接口。上层网关逻辑（fallback/用量追踪/部门级配额）保留自研。`RemoteLLMProvider`（客户端→服务端代理）保留不动。
|
||||
|
||||
R16. MCP 客户端替换为 `langchain-mcp-adapters`：跟进行业协议演进，降低自研 3 传输层（Stdio/HTTP/SSE）维护成本。
|
||||
|
||||
R17. 自研语义缓存替换为 LiteLLM 内置 Redis Semantic Cache。阈值调优（默认 0.87，约 13% 误命中风险）。现有 `llm/cache.py` 废弃。
|
||||
|
||||
## Key Technical Decisions
|
||||
|
||||
**KTD1: LlamaIndex 作为 RAG 管道框架。** 外部研究确认 LlamaIndex 2026 原生覆盖所有所需能力（双索引/智能分段/rerank/问题生成），pgvector 一等支持。相比从零构建避免重复造轮子；相比集成 MaxKB 独立服务避免引入 Django + 独立 PG 的运维复杂度。风险：LlamaIndex 频繁 breaking changes → 通过版本锁定 + 集成测试缓解。
|
||||
|
||||
**KTD2: TaskIQ 替代 Celery 作为异步任务队列。** 外部研究确认 Celery 对 asyncio 原生栈过度设计（引入 broker + worker + beat = 3 个新运维组件）。TaskIQ 提供一等 FastAPI 集成、Redis broker 支持、asyncio 原生。R12 提前至 P1 解决文档向量化阻塞事件循环的优先级依赖冲突。ARQ 已废弃不采用。
|
||||
|
||||
**KTD3: KB 元数据持久化到 PostgreSQL。** 遵循 `memory/episodic.py` 的 EpisodicMemory 模式（SQLAlchemy async session + PG）。KB 源、文档记录、ACL 存关系表，embedding 存 pgvector。替换现有内存 `KnowledgeSourceStore`。
|
||||
|
||||
**KTD4: MCP Server 合并至主 FastAPI app。** 现有 `mcp/server.py` 独立 app 零认证是 RCE 面（终端工具存在）。合并为 `/api/v1/mcp/` 子路由，复用 `require_permission` + `APIKeyHeader` 认证。独立 `MCPServer` 类重构为路由工厂。
|
||||
|
||||
**KTD5: Per-KB ACL 通过新 `kb_acl` 表实现。** 遵循现有 `filter_kb_sources_by_department` 模式，新增 `kb_acl` 表（kb_id, user_id, role: owner/viewer）。Agent 检索时通过 `filter_kb_by_user_acl()` 过滤，与部门级过滤并行。
|
||||
|
||||
**KTD6: 应用层 jieba 分词实现中文全文检索。** PostgreSQL 内置 `tsvector` 不支持中文分词。在 Python 层用 jieba 分词后写入 `tsvector`，避免安装 PG 扩展（pg_jieba/zhparser）的运维复杂度，也避免引入 Elasticsearch 外部搜索引擎。术语表通过 jieba 自定义词典实现。
|
||||
|
||||
**KTD7: LiteLLM 替换直接 provider，RemoteLLMProvider 保留。** LiteLLM 原生支持 Volcengine/Doubao；Wenxin/Yuanbao 通过 OpenAI 兼容端点。`RemoteLLMProvider` 是客户端→服务端代理（架构上不同于直接 API provider），LiteLLM 无法替代。语义缓存/fallback/用量追踪保留自研上层逻辑，底层 provider 适配走 LiteLLM。
|
||||
|
||||
**KTD8: 加密 DB 列存储平台凭证。** 单部署企业场景下，加密 DB 列 + 环境变量 master key 足够。避免引入 HashiCorp Vault 外部依赖。凭证轮换通过 API 触发 re-encrypt。访问审计记录到现有审计日志。
|
||||
|
||||
**KTD9: 新建 `src/agentkit/rag_platform/` 顶层模块。** 与 `memory/` 职责分离：`rag_platform/` 服务企业知识库场景，`memory/` 服务 Agent 运行时记忆。`LocalRAGService` 对 KB 场景废弃，Agent 记忆（EpisodicMemory）保留使用。
|
||||
|
||||
**KTD10: 成功标准从"MaxKB 功能对等"重构为"用户结果导向"。** MaxKB 是 RAG 知识库产品，AgentKit 是 Agent 平台。对标不同产品类别的功能对等可能构建不服务于实际用户的能力。成功标准改为：企业用户可上传文档、配置检索、测试召回、通过多端使用 Agent 检索知识库。
|
||||
|
||||
## High-Level Technical Design
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph P1["P1: RAG 工业级管道 + 异步任务"]
|
||||
U1[U1 RAG 平台骨架] --> U2[U2 KB 持久化 + ACL]
|
||||
U2 --> U3[U3 文档处理管道]
|
||||
U3 --> U4[U4 双索引检索]
|
||||
U4 --> U5[U5 Rerank/问题生成/术语表]
|
||||
U5 --> U6[U6 命中处理 + KB 设置]
|
||||
U2 --> U7[U7 上传安全 + 净化]
|
||||
U3 --> U8[U8 TaskIQ 异步任务]
|
||||
U6 --> U9[U9 前端 KB 管理]
|
||||
U8 --> U9
|
||||
end
|
||||
|
||||
subgraph P2["P2: 平台触达扩展"]
|
||||
U10[U10 适配器骨架 + secrets] --> U11[U11 飞书 IM]
|
||||
U11 --> U12[U12 钉钉/企微/Slack]
|
||||
U13[U13 MCP 认证 + 合并]
|
||||
U13 --> U14[U14 Skill/团队 MCP 发布]
|
||||
end
|
||||
|
||||
subgraph P3["P3: 生态替换降本"]
|
||||
U15[U15 LiteLLM Provider]
|
||||
U16[U16 langchain-mcp-adapters]
|
||||
U17[U17 LiteLLM 语义缓存]
|
||||
end
|
||||
|
||||
P1 --> P2 --> P3
|
||||
```
|
||||
|
||||
### 文档处理管道状态机
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> pending: 上传
|
||||
pending --> parsing: TaskIQ 接收
|
||||
parsing --> segmenting: 解析成功
|
||||
parsing --> failed: 解析失败
|
||||
segmenting --> vectorizing: 预览确认/自动
|
||||
segmenting --> failed: 分段失败
|
||||
vectorizing --> indexed: 向量化+索引成功
|
||||
vectorizing --> failed: 向量化失败
|
||||
failed --> pending: 用户重试
|
||||
indexed --> [*]
|
||||
```
|
||||
|
||||
### 检索流程
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Q[Agent 查询] --> ACL{Per-KB ACL 过滤}
|
||||
ACL -->|授权| Mode{检索模式}
|
||||
Mode -->|embedding| Sem[pgvector 语义]
|
||||
Mode -->|keywords| FT[PG 全文 jieba]
|
||||
Mode -->|blend| Both[双索引合并]
|
||||
Sem --> Rerank[rerank 模型]
|
||||
FT --> Rerank
|
||||
Both --> Rerank
|
||||
Rerank --> Hit{命中处理}
|
||||
Hit -->|model_opt| LLM[LLM 生成回答]
|
||||
Hit -->|direct| Return[返回匹配段落]
|
||||
LLM --> Result[检索结果]
|
||||
Return --> Result
|
||||
```
|
||||
|
||||
## Implementation Units
|
||||
|
||||
### P1: RAG 工业级管道 + 异步任务基础
|
||||
|
||||
---
|
||||
|
||||
### U1. RAG 平台模块骨架 + LlamaIndex 集成
|
||||
|
||||
- **Goal:** 创建 `src/agentkit/rag_platform/` 模块，集成 LlamaIndex 作为管道框架，连接现有 pgvector。
|
||||
- **Files:**
|
||||
- `src/agentkit/rag_platform/__init__.py` — 模块入口
|
||||
- `src/agentkit/rag_platform/models.py` — Pydantic 数据模型（KB、Document、Chunk、QueryResult）
|
||||
- `src/agentkit/rag_platform/pipeline.py` — LlamaIndex IngestionPipeline 封装
|
||||
- `src/agentkit/rag_platform/indexing.py` — pgvector 索引管理（LlamaIndex PGVectorStore）
|
||||
- **Patterns:** 遵循现有模块结构（cf. `memory/__init__.py`, `mcp/__init__.py`）；LlamaIndex `PGVectorStore` 连接现有 PostgreSQL。
|
||||
- **Test scenarios:**
|
||||
- LlamaIndex PGVectorStore 连接现有 pgvector 扩展
|
||||
- 基础 ingest（文档 → chunk → embedding → pgvector INSERT）端到端工作
|
||||
- 基础 query（query → embedding → pgvector cosine 检索）返回结果
|
||||
- **Verification:** `pytest tests/unit/rag_platform/test_pipeline.py`
|
||||
|
||||
---
|
||||
|
||||
### U2. KB 持久化存储 + Per-KB 访问控制
|
||||
|
||||
- **Goal:** 替换内存 `KnowledgeSourceStore` 为 PostgreSQL 持久化，实现 per-KB ACL。
|
||||
- **Files:**
|
||||
- `src/agentkit/rag_platform/store.py` — KB/Document 持久化（SQLAlchemy async）
|
||||
- `src/agentkit/rag_platform/acl.py` — per-KB ACL 逻辑
|
||||
- `src/agentkit/server/auth/models.py` — 新增 `kb_acl` 表模型
|
||||
- `src/agentkit/server/routes/kb_management.py` — 替换 `KnowledgeSourceStore` 调用
|
||||
- **Patterns:** 遵循 `memory/episodic.py` 的 EpisodicMemory PG 模式（async session）；遵循 `filter_kb_sources_by_department` 模式实现 `filter_kb_by_user_acl()`。
|
||||
- **Test scenarios:**
|
||||
- KB 元数据写入 PG，重启后仍存在
|
||||
- owner 用户可查询自己的 KB
|
||||
- 非 authorized 用户查询 KB 被拒绝
|
||||
- Agent 检索时 ACL 过滤生效（仅返回授权 KB 的结果）
|
||||
- **Verification:** `pytest tests/unit/rag_platform/test_store.py tests/unit/rag_platform/test_acl.py`
|
||||
|
||||
---
|
||||
|
||||
### U3. 文档处理管道（LlamaIndex 分段 + 预览 + 向量化）
|
||||
|
||||
- **Goal:** 连接上传→解析→分段→预览→向量化→索引管道，使用 LlamaIndex IngestionPipeline。
|
||||
- **Files:**
|
||||
- `src/agentkit/rag_platform/document_processor.py` — 文档处理管道
|
||||
- `src/agentkit/rag_platform/preview.py` — 分段预览 API
|
||||
- `src/agentkit/server/routes/kb_management.py` — 重写 `upload_document()` 端点
|
||||
- **Patterns:** LlamaIndex `IngestionPipeline`（SentenceSplitter + MetadataExtractor）；现有 `memory/document_loader.py` 的 `DocumentLoader` 用于解析；文档状态模型（pending→parsing→segmenting→vectorizing→indexed|failed）。
|
||||
- **Test scenarios:**
|
||||
- 上传 PDF → 解析 → 分段 → 返回预览（只读）
|
||||
- 确认预览 → 向量化 → 索引 → 可检索
|
||||
- 解析失败 → 状态 failed + error_message
|
||||
- 向量化失败 → 状态 failed + error_message
|
||||
- 重复上传同一文档 → 拒绝或更新（非创建重复）
|
||||
- **Verification:** `pytest tests/unit/rag_platform/test_document_processor.py`
|
||||
|
||||
---
|
||||
|
||||
### U4. 双索引检索（pgvector 语义 + PG 全文检索 with jieba）
|
||||
|
||||
- **Goal:** 实现双索引检索，支持 embedding/keywords/blend 三种模式。
|
||||
- **Files:**
|
||||
- `src/agentkit/rag_platform/retrieval.py` — 检索逻辑（三模式）
|
||||
- `src/agentkit/rag_platform/fulltext.py` — jieba 分词 + tsvector 写入/查询
|
||||
- **Patterns:** LlamaIndex hybrid retriever（VectorStoreRetriever + NLSQLRetriever 或自定义）；jieba 在 Python 层分词后写入 `search_vector` 列。
|
||||
- **Test scenarios:**
|
||||
- embedding 模式：语义检索返回相关结果
|
||||
- keywords 模式：中文全文检索返回包含关键词的结果
|
||||
- blend 模式：合并语义+全文结果，去重排序
|
||||
- 查询无结果时返回空列表（非报错）
|
||||
- **Verification:** `pytest tests/unit/rag_platform/test_retrieval.py tests/unit/rag_platform/test_fulltext.py`
|
||||
|
||||
---
|
||||
|
||||
### U5. Rerank + 问题生成 + 术语表
|
||||
|
||||
- **Goal:** 添加 rerank 模型重排、问题自动生成、术语表支持。
|
||||
- **Files:**
|
||||
- `src/agentkit/rag_platform/rerank.py` — rerank 模型集成（Cohere/BGE-Reranker 可配置）
|
||||
- `src/agentkit/rag_platform/question_gen.py` — LLM-based 问题生成
|
||||
- `src/agentkit/rag_platform/termbase.py` — 术语表管理 + jieba 自定义词典
|
||||
- **Patterns:** LlamaIndex rerankers（`CohereRerank` 或 `SentenceTransformerRerank`）；参考 `memory/contextual_retrieval.py` 的 ContextualChunker 模式生成问题；jieba `load_userdict()` 加载术语表。
|
||||
- **Test scenarios:**
|
||||
- rerank 后结果相关性顺序改善
|
||||
- 问题生成产生与段落内容相关的问题
|
||||
- 术语表中的领域术语被正确分词
|
||||
- 术语表增强后检索召回率提升
|
||||
- **Verification:** `pytest tests/unit/rag_platform/test_rerank.py tests/unit/rag_platform/test_question_gen.py tests/unit/rag_platform/test_termbase.py`
|
||||
|
||||
---
|
||||
|
||||
### U6. 命中处理模式 + KB 设置
|
||||
|
||||
- **Goal:** 实现命中处理模式（模型优化/直接回答）+ KB 级别设置。
|
||||
- **Files:**
|
||||
- `src/agentkit/rag_platform/hit_processing.py` — 命中处理逻辑
|
||||
- `src/agentkit/rag_platform/settings.py` — KB 设置模型（检索模式默认/命中处理默认/授权用户）
|
||||
- `src/agentkit/server/routes/kb_management.py` — KB 设置端点
|
||||
- **Patterns:** 模型优化模式调用现有 LLM Gateway；直接回答模式返回匹配段落；KB 设置存 PG。
|
||||
- **Test scenarios:**
|
||||
- model_opt 模式：LLM 基于检索结果生成回答
|
||||
- direct 模式：直接返回匹配段落
|
||||
- KB 设置默认模式生效
|
||||
- Agent 运行时覆盖默认模式
|
||||
- **Verification:** `pytest tests/unit/rag_platform/test_hit_processing.py tests/unit/rag_platform/test_settings.py`
|
||||
|
||||
---
|
||||
|
||||
### U7. 文件上传安全 + 内容净化
|
||||
|
||||
- **Goal:** 后端文件类型白名单 + 内容净化。
|
||||
- **Files:**
|
||||
- `src/agentkit/rag_platform/sanitize.py` — 内容净化（markdown sanitize + PDF 安全）
|
||||
- `src/agentkit/server/routes/kb_management.py` — 上传端点增加白名单验证
|
||||
- **Patterns:** `DocumentLoader._detect_format()` 映射作为白名单源；`bleach` 或 `markdown` 库净化 HTML/markdown；PDF 解析限制页面数/大小。
|
||||
- **Test scenarios:**
|
||||
- 白名单外文件类型被拒绝（.exe, .sh 等）
|
||||
- 超大小限制文件被拒绝
|
||||
- markdown 中 `<script>` 标签被净化
|
||||
- PDF 解析不触发已知 CVE（限制解析器行为）
|
||||
- **Verification:** `pytest tests/unit/rag_platform/test_sanitize.py`
|
||||
|
||||
---
|
||||
|
||||
### U8. TaskIQ 异步任务集成
|
||||
|
||||
- **Goal:** 集成 TaskIQ 实现异步文档向量化和批量任务。
|
||||
- **Files:**
|
||||
- `src/agentkit/rag_platform/tasks.py` — TaskIQ 任务定义（向量化、批量索引）
|
||||
- `src/agentkit/server/app.py` — TaskIQ startup/shutdown
|
||||
- `src/agentkit/server/task_store.py` — 扩展状态跟踪（复用现有 TaskStore 模式）
|
||||
- **Patterns:** TaskIQ FastAPI 集成（`TaskiqMiddleware`）；现有 Redis 作为 broker；`TaskStore` 状态模型（PENDING/RUNNING/COMPLETED/FAILED）。
|
||||
- **Test scenarios:**
|
||||
- 大文档（50MB PDF）向量化在后台执行，不阻塞事件循环
|
||||
- 任务状态可查询（pending→running→completed）
|
||||
- 任务失败后自动重试（可配置重试次数）
|
||||
- 任务历史可查询
|
||||
- **Verification:** `pytest tests/unit/rag_platform/test_tasks.py`
|
||||
|
||||
---
|
||||
|
||||
### U9. 前端 KB 管理扩展
|
||||
|
||||
- **Goal:** 扩展 KnowledgeBaseView/DocumentUpload/SearchTest 组件，增加分段预览、状态展示、KB 设置。
|
||||
- **Files:**
|
||||
- `src/agentkit/server/frontend/src/components/knowledge/KnowledgeBaseView.vue` — 扩展
|
||||
- `src/agentkit/server/frontend/src/components/knowledge/DocumentUpload.vue` — 增加状态展示
|
||||
- `src/agentkit/server/frontend/src/components/knowledge/SearchTest.vue` — 扩展检索测试
|
||||
- `src/agentkit/server/frontend/src/components/knowledge/SegmentPreview.vue` — 新建分段预览组件
|
||||
- `src/agentkit/server/frontend/src/components/knowledge/KBSettings.vue` — 新建 KB 设置组件
|
||||
- **Patterns:** 现有 Ant Design Vue 组件模式；WebSocket 推送文档处理状态（复用现有 ws 协议）。
|
||||
- **Test scenarios:**
|
||||
- 上传后显示处理进度（pending→parsing→...→indexed）
|
||||
- 分段预览显示分段结果（只读）
|
||||
- 检索测试支持三模式切换
|
||||
- KB 设置可配置检索模式默认/命中处理默认/授权用户
|
||||
- **Verification:** `npm run typecheck` + 手动验证
|
||||
|
||||
### P2: 平台触达扩展
|
||||
|
||||
---
|
||||
|
||||
### U10. 多端消息适配器骨架 + secrets store
|
||||
|
||||
- **Goal:** 创建 MessageAdapter 协议 + secrets store 基础设施。
|
||||
- **Files:**
|
||||
- `src/agentkit/channels/__init__.py` — 模块入口
|
||||
- `src/agentkit/channels/base.py` — MessageAdapter 协议（receive_message/send_message/verify_signature）
|
||||
- `src/agentkit/channels/secrets.py` — 加密 DB 列 secrets store
|
||||
- `src/agentkit/server/routes/channels.py` — 渠道管理端点
|
||||
- **Patterns:** 新 `MessageAdapter` 协议（cf. `memory/adapters/base.py` 的 KBAdapter）；加密 DB 列（AES-256，master key 来自环境变量）。
|
||||
- **Test scenarios:**
|
||||
- secrets 写入后加密存储（非明文）
|
||||
- secrets 读取时解密
|
||||
- MessageAdapter 协议被所有适配器实现
|
||||
- 渠道管理端点 CRUD 工作
|
||||
- **Verification:** `pytest tests/unit/channels/test_secrets.py tests/unit/channels/test_base.py`
|
||||
|
||||
---
|
||||
|
||||
### U11. 飞书 IM 适配器（端到端）
|
||||
|
||||
- **Goal:** 实现飞书 IM 消息适配器端到端。
|
||||
- **Files:**
|
||||
- `src/agentkit/channels/feishu.py` — 飞书 IM 适配器
|
||||
- `src/agentkit/server/routes/channels.py` — 飞书 webhook 端点
|
||||
- **Patterns:** 飞书 webhook 签名验证（encrypt_key + AES 解密）；消息格式转换（飞书事件 → AgentKit 标准消息）；现有 chat handler 集成（`RequestPreprocessor` → `ExecutionMode`）。
|
||||
- **Test scenarios:**
|
||||
- 飞书消息 → webhook → 签名验证 → Agent 处理 → 响应返回飞书
|
||||
- 无效签名请求被拒绝
|
||||
- 文本消息正确转换
|
||||
- Agent 响应正确格式化返回飞书
|
||||
- **Verification:** `pytest tests/unit/channels/test_feishu.py`
|
||||
|
||||
---
|
||||
|
||||
### U12. 钉钉/企微/Slack 适配器
|
||||
|
||||
- **Goal:** 按飞书模式实现其余平台适配器。
|
||||
- **Files:**
|
||||
- `src/agentkit/channels/dingtalk.py` — 钉钉适配器
|
||||
- `src/agentkit/channels/wecom.py` — 企微适配器
|
||||
- `src/agentkit/channels/slack.py` — Slack 适配器
|
||||
- **Patterns:** 遵循 U11 飞书模式；平台特定签名验证（钉钉 token、企微 EncodingAESKey、Slack signing secret）。
|
||||
- **Test scenarios:**
|
||||
- 每个平台端到端消息流
|
||||
- 每个平台签名验证拒绝无效请求
|
||||
- **Verification:** `pytest tests/unit/channels/test_dingtalk.py tests/unit/channels/test_wecom.py tests/unit/channels/test_slack.py`
|
||||
|
||||
---
|
||||
|
||||
### U13. MCP Server 认证 + 合并至主 app
|
||||
|
||||
- **Goal:** 将 MCP Server 合并至主 FastAPI app，添加认证。
|
||||
- **Files:**
|
||||
- `src/agentkit/mcp/server.py` — 重构为路由工厂（`create_mcp_router()`）
|
||||
- `src/agentkit/server/app.py` — 挂载 MCP 路由到 `/api/v1/mcp/`
|
||||
- **Patterns:** `require_permission` 依赖注入；`APIKeyHeader` 认证；现有 `ToolRegistry.list_tools()` 暴露为 MCP 工具。
|
||||
- **Test scenarios:**
|
||||
- 无认证调用 `/api/v1/mcp/tools/list` 被拒绝（401）
|
||||
- 有效 API Key 调用返回工具列表
|
||||
- 有效 JWT + 权限调用返回工具列表
|
||||
- 现有工具仍可通过 MCP 协议调用
|
||||
- **Verification:** `pytest tests/unit/mcp/test_server_auth.py`
|
||||
|
||||
---
|
||||
|
||||
### U14. Skill/专家团队 MCP 发布
|
||||
|
||||
- **Goal:** 支持 Skill/专家团队发布为 MCP 工具。
|
||||
- **Files:**
|
||||
- `src/agentkit/mcp/publisher.py` — Skill/Team → MCP Tool 适配器
|
||||
- `src/agentkit/server/routes/mcp_publish.py` — 发布管理端点
|
||||
- `src/agentkit/mcp/server.py` — 扩展工具列表包含已发布 Skill/Team
|
||||
- **Patterns:** Tool 适配器包装（Skill/Team → `Tool` 接口）；管理员级授权（`Permission.ADMIN`）；配置字段（工具名称/描述/输入 schema/鉴权方式/速率限制）。
|
||||
- **Test scenarios:**
|
||||
- 管理员发布 Skill 为 MCP 工具
|
||||
- 非管理员发布被拒绝
|
||||
- 外部系统通过 MCP 认证调用已发布 Skill
|
||||
- 专家团队发布为 MCP 工具
|
||||
- 已发布工具在 `/api/v1/mcp/tools/list` 中可见
|
||||
- **Verification:** `pytest tests/unit/mcp/test_publisher.py`
|
||||
|
||||
### P3: 生态替换降本
|
||||
|
||||
---
|
||||
|
||||
### U15. LiteLLM Provider 替换
|
||||
|
||||
- **Goal:** 用 LiteLLM 替换 6 个直接 API provider 适配器。
|
||||
- **Files:**
|
||||
- `src/agentkit/llm/providers.py` — 重写为 LiteLLM 统一接口
|
||||
- `src/agentkit/llm/gateway.py` — 适配上层网关逻辑
|
||||
- `src/agentkit/llm/config.py` — provider 配置更新
|
||||
- **Patterns:** LiteLLM `completion()` / `acompletion()` 统一接口；保留自研 fallback 链/用量追踪/部门级配额；`RemoteLLMProvider` 保留不动。
|
||||
- **Test scenarios:**
|
||||
- 6 个 provider 通过 LiteLLM 调用成功（OpenAI/Anthropic/Gemini/Doubao/Wenxin/Yuanbao）
|
||||
- fallback 链在 provider 失败时切换
|
||||
- 用量追踪记录正确
|
||||
- 部门级配额生效
|
||||
- `RemoteLLMProvider` 仍正常工作（不受影响）
|
||||
- 流式响应正常工作
|
||||
- **Verification:** `pytest tests/unit/llm/test_providers.py tests/unit/llm/test_gateway.py`
|
||||
|
||||
---
|
||||
|
||||
### U16. langchain-mcp-adapters 替换 MCP 客户端
|
||||
|
||||
- **Goal:** 用 langchain-mcp-adapters 替换自研 MCP 客户端传输层。
|
||||
- **Files:**
|
||||
- `src/agentkit/mcp/client.py` — 重写为 langchain-mcp-adapters 封装
|
||||
- `src/agentkit/mcp/transport.py` — 废弃（Stdio/HTTP/SSE 传输由 langchain-mcp-adapters 提供）
|
||||
- **Patterns:** `langchain-mcp-adapters` 的 `ClientSession` + 传输层；现有 `MCPTool` 包装保留（将远程 MCP 工具暴露为本地 `Tool`）。
|
||||
- **Test scenarios:**
|
||||
- 现有 MCP 工具调用通过新客户端工作
|
||||
- Stdio 传输连接成功
|
||||
- HTTP 传输连接成功
|
||||
- SSE 传输连接成功
|
||||
- `MCPTool` 包装仍正常工作
|
||||
- **Verification:** `pytest tests/unit/mcp/test_client.py`
|
||||
|
||||
---
|
||||
|
||||
### U17. LiteLLM 语义缓存集成
|
||||
|
||||
- **Goal:** 用 LiteLLM 内置 Redis Semantic Cache 替换自研语义缓存。
|
||||
- **Files:**
|
||||
- `src/agentkit/llm/cache.py` — 重写为 LiteLLM 缓存配置（或废弃，由 gateway 直接配置）
|
||||
- `src/agentkit/llm/gateway.py` — 集成 LiteLLM caching 配置
|
||||
- **Patterns:** LiteLLM `RedisSemanticCache`；阈值调优（默认 0.87）；缓存 key 包含 system prompt + temperature。
|
||||
- **Test scenarios:**
|
||||
- 语义相似查询命中缓存
|
||||
- 不同 system prompt 不命中缓存
|
||||
- 缓存命中率可统计
|
||||
- 阈值调优生效
|
||||
- **Verification:** `pytest tests/unit/llm/test_cache.py`
|
||||
|
||||
## Scope Boundaries
|
||||
|
||||
### In scope
|
||||
|
||||
- P1: RAG 工业级管道（LlamaIndex 集成 + 双索引 + rerank + 问题生成 + 术语表 + 命中处理 + per-KB ACL + 上传安全 + TaskIQ 异步 + 前端扩展）
|
||||
- P2: 多端消息接入（飞书/钉钉/企微/Slack）+ secrets store + MCP Server 认证 + Skill/团队 MCP 发布
|
||||
- P3: LiteLLM provider 替换 + langchain-mcp-adapters 客户端替换 + LiteLLM 语义缓存替换
|
||||
|
||||
### Deferred for later
|
||||
|
||||
- 分段预览编辑能力（合并/拆分/重新分段）— P1 只做只读预览
|
||||
- 外部 secrets manager（HashiCorp Vault / 云 KMS）— P2 用加密 DB 列，单部署足够
|
||||
- 分块预览的高级交互模式 — 归 ce-work 设计
|
||||
- RAG 平台门户 IA 精细化（顶级 section vs 扩展现有）— P1 前端扩展先复用现有 KnowledgeBaseView
|
||||
|
||||
### Outside this product's identity
|
||||
|
||||
- Agent 引擎/专家团队/自进化/终端安全 — 差异化能力，保持自研，不在本计划范围
|
||||
- FlowCanvas 工作流画布 — 不替换为 LogicFlow，保持自研
|
||||
- 消息总线（`bus/`）— 紧耦合 Agent 事件系统，非 commodity，保持自研
|
||||
- MaxKB 深度集成（共享 DB/embedding model）— MaxKB 作为独立服务集成仅作备选方案，不在本计划实施
|
||||
|
||||
## System-Wide Impact
|
||||
|
||||
- **数据生命周期:** 新增 KB/Document/Chunk/kb_acl 表到 PostgreSQL；pgvector 索引扩展（现有 Agent 记忆表不动）；`search_vector` 列新增到 KB chunk 表。
|
||||
- **认证边界:** MCP Server 从零认证变为要求 JWT/API Key；KB 检索新增 per-KB ACL 层；平台凭证从明文变为加密存储。
|
||||
- **性能姿态:** 文档向量化从同步（阻塞事件循环）变为 TaskIQ 异步；检索从单索引变为双索引 + rerank（增加延迟但提升相关性）。
|
||||
- **共享基础设施:** Redis 新增 TaskIQ broker 角色（与现有 bus/cache 共存）；PostgreSQL 新增 RAG 平台 schema。
|
||||
- **Agent/工具对等:** Agent 运行时通过 RAG 平台检索 KB 内容；外部 AI 系统通过 MCP 调用已发布 Skill/团队。
|
||||
- **向后兼容:** `LocalRAGService` 对 KB 场景废弃但保留（Agent 记忆仍用）；`RemoteLLMProvider` 保留不动；现有 `ToolRegistry` 工具仍可通过 MCP 访问。
|
||||
|
||||
## Risks & Dependencies
|
||||
|
||||
- **LlamaIndex breaking changes:** LlamaIndex 频繁发布 breaking changes。缓解：版本锁定（`pyproject.toml` pin major version）+ 集成测试覆盖核心管道。
|
||||
- **jieba 中文分词质量:** jieba 默认词典可能不覆盖领域术语。缓解：术语表通过 `jieba.load_userdict()` 扩展；检索测试覆盖中文场景。
|
||||
- **TaskIQ 成熟度:** TaskIQ 社区较小（~2k★）。缓解：API 简单，必要时可替换为 SAQ 或回退到 ProcessPoolExecutor。
|
||||
- **LiteLLM 中文 provider 覆盖:** Wenxin/Yuanbao 通过 OpenAI 兼容端点，可能缺少 provider 特定功能。缓解：feature-gap 分析在 U15 实施时执行；保留自定义 handler 作为 fallback。
|
||||
- **MCP Server 合并破坏现有集成:** 合并至主 app 可能影响现有 MCP 客户端调用。缓解：保持 `/api/v1/mcp/` 路径与现有 MCP 协议兼容；迁移测试。
|
||||
- **向后兼容性验证:** R15-R17 替换核心组件，"现有功能行为不变"是假设非验证。缓解：每个替换单元必须有 feature-parity 测试（现有行为 → 新实现行为对比）。
|
||||
- **GPL v3 合规边界:** MaxKB 作为备选集成方案的 GPL v3 许可证。缓解：本计划不实施 MaxKB 集成（仅作备选）；若未来集成，通过 REST API 独立服务调用（不修改/分发 MaxKB 代码）。
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **门户触达(P2)反转门户价值主张:** 门户平台核心价值是触达，但多端接入在 P2。是否在 P1 并行交付至少一个高价值渠道（如飞书 IM）？默认假设：不并行，P1 聚焦 RAG 管道，P2 再做多端。
|
||||
2. **R11-R13 是技术债非产品需求:** R15-R17（原 R11-R13）成功标准是"现有功能行为不变"（零用户可见影响），无 Actor 受益。是否移至独立工程债轨道？默认假设：保留在 P3，作为 commodity 层降本。
|
||||
3. **rerank 模型选择:** U5 rerank 模型未指定（Cohere Rerank vs BGE-Reranker vs 其他）。默认假设：API-based（Cohere Rerank 或 BGE-Reranker via Xinference），可配置。
|
||||
4. **多端 onboarding 流程细节:** U10-U12 的管理员配置流程（webhook URL 生成、app 凭证配置、连通性测试）需在实施时细化。默认假设：admin 导航到渠道配置 → 选择平台 → 输入凭证 → 系统生成 webhook URL → 管理员在平台配置 → 连通性测试。
|
||||
|
||||
## Sources / Research
|
||||
|
||||
- **LlamaIndex 2026:** 14 index types, sparse+dense hybrid retrieval, auto-rerank, pgvector first-class support, LlamaParse for complex PDFs. Pitfall: frequent breaking changes. — https://blog.csdn.net/yanxilou/article/details/162178538
|
||||
- **LlamaIndex vs Haystack 2026:** Architecture and decision matrix. — https://myengineeringpath.dev/tools/llamaindex-vs-haystack/
|
||||
- **LiteLLM v1.89.x:** 157 providers, 2784 models. Volcengine/Doubao native; Wenxin/Yuanbao via OpenAI-compatible endpoint. Redis Semantic Cache, fallback chains, virtual keys, spend tracking built-in. P95 proxy overhead 8ms @ 1k RPS. — https://docs.litellm.ai/docs/providers
|
||||
- **LiteLLM semantic caching:** Threshold tuning critical (0.87 ≈ 13% false-positive collision risk). Cache key excludes system prompt/temperature by default. — https://theneuralbase.com/litellm/learn/intermediate/semantic-caching-similar-prompts/
|
||||
- **TaskIQ:** Modern async task queue with first-class FastAPI integration. ARQ deprecated — TaskIQ is spiritual successor. — https://markaicode.com/alternatives/rq-alternatives/
|
||||
- **Celery vs asyncio:** Celery overkill for asyncio-native stack. Adds broker + worker + beat = 3 new operational components. — https://theneuralbase.com/celery-for-ml/learn/advanced/vs-dramatiq/
|
||||
- **MaxKB v2.7.0:** 21.4k★, GPLv3, Django + Vue, ships Celery internally. OpenAI-compatible chat API. Cannot share PG/pgvector — brings its own. GPL v3 not a blocker for SaaS-style REST integration (confirmed by FSF FAQ, Chinese judicial precedent 不乱买案). — https://maxkb.cn/docs/v2/user_manual/chat_to_API/
|
||||
- **GPL v3 commercial use:** Internal-use loophole, distribution trigger. SaaS/network-service via REST does not trigger copyleft. — https://legalclarity.org/can-you-use-gplv3-in-a-commercial-application/
|
||||
- **AgentKit 代码库:** `memory/local_rag.py`（LocalRAGService — pgvector + 分块 + 嵌入 + 语义检索）、`memory/chunking.py`（TextChunker/StructuralChunker）、`memory/contextual_retrieval.py`（ContextualChunker — LLM 生成上下文前缀）、`mcp/server.py`（零认证 MCP Server）、`mcp/client.py`（3 传输层 MCP 客户端）、`llm/gateway.py`（6 provider + fallback + 语义缓存 + 用量追踪）、`llm/cache.py`（自研语义缓存）、`server/routes/kb_management.py`（KB 管理端点 — 上传未调用向量化）、`server/auth/`（JWT + RBAC + API Key + 部门级过滤）、`server/task_store.py`（任务状态存储 — 仅状态非执行）、`bus/`（MemoryBus/RedisBus）
|
||||
- **需求文档:** `docs/brainstorms/2026-06-24-portal-platform-evolution-requirements.md`（经两轮 ce-doc-review，36 项延期至 planning）
|
||||
162
skills-lock.json
162
skills-lock.json
|
|
@ -1,12 +1,174 @@
|
|||
{
|
||||
"version": 1,
|
||||
"skills": {
|
||||
"ce-brainstorm": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-brainstorm/SKILL.md",
|
||||
"computedHash": "ee435337c207aac2eb75ff3b1d2f689d3e0f9a0b834e70a18b292c4419f2fa82"
|
||||
},
|
||||
"ce-code-review": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-code-review/SKILL.md",
|
||||
"computedHash": "ed4ea6717d72837b6364b7ba8ad0bbd26c1ef1d62f592fd0832ce80d0b74fb1e"
|
||||
},
|
||||
"ce-commit": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-commit/SKILL.md",
|
||||
"computedHash": "8217b388a1c6e206423313faabbb85b779dfd8b9cc28893ac7022e73b12d686e"
|
||||
},
|
||||
"ce-commit-push-pr": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-commit-push-pr/SKILL.md",
|
||||
"computedHash": "fc0477ae0527b917676f291ae4b72f74909848746845d7e1e291cdc250636c18"
|
||||
},
|
||||
"ce-compound": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-compound/SKILL.md",
|
||||
"computedHash": "8d5c2024f3ff700d16a8144efcba296b2f540e218e700a8e2cafb76fc6a57be3"
|
||||
},
|
||||
"ce-compound-refresh": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-compound-refresh/SKILL.md",
|
||||
"computedHash": "210ab636264392d81ed9eebb1ad109225163a954d25374a58d9e7c4824df6af1"
|
||||
},
|
||||
"ce-debug": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-debug/SKILL.md",
|
||||
"computedHash": "f69714bb7069d733de0601c801b8b1f9f696b9e2813420f002164d07f2aec4e4"
|
||||
},
|
||||
"ce-doc-review": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-doc-review/SKILL.md",
|
||||
"computedHash": "59b054eb7e8a25ba44d1a0364b872a469e30272b1b4fb6b9b0d7062330769870"
|
||||
},
|
||||
"ce-dogfood-beta": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-dogfood-beta/SKILL.md",
|
||||
"computedHash": "ef164695bb3a5779dfa66dd5bcb8032255eb8912b875360d38716631f1c73870"
|
||||
},
|
||||
"ce-ideate": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-ideate/SKILL.md",
|
||||
"computedHash": "3d91386c84afcb25e5cddd8d82b96374a416f34a8c9d739e2d85674869d496b8"
|
||||
},
|
||||
"ce-optimize": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-optimize/SKILL.md",
|
||||
"computedHash": "7d1021e840bfa6c37147f5bea3bc9add8dd7f24cb6350d67fb5bd2df7b25e4cc"
|
||||
},
|
||||
"ce-plan": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-plan/SKILL.md",
|
||||
"computedHash": "4a35c906bf49f09134dbc0422dcfe9cdef7f9036c067a7bf266fe51ce40c354e"
|
||||
},
|
||||
"ce-polish": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-polish/SKILL.md",
|
||||
"computedHash": "520e3d5347c3523c3c8810a92ab6a3f4dca552514c2e7c18ac81ea03def5f180"
|
||||
},
|
||||
"ce-product-pulse": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-product-pulse/SKILL.md",
|
||||
"computedHash": "bf15852134296d1a8bf331d41a9b8c0e5411372767e5b01ba11a1b04e26b753b"
|
||||
},
|
||||
"ce-promote": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-promote/SKILL.md",
|
||||
"computedHash": "280f6003ad0cdde596f5c77edfcc63823f9d5066e800d4ee9e05828609aa352f"
|
||||
},
|
||||
"ce-proof": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-proof/SKILL.md",
|
||||
"computedHash": "fc9c51d14ac15fce8db15e5c20469aa1a83632c68a58a4e2ba8ac5e17d2f0d8b"
|
||||
},
|
||||
"ce-resolve-pr-feedback": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-resolve-pr-feedback/SKILL.md",
|
||||
"computedHash": "3c100452baa3e70d9be374ebacb6dc347627c59ad9d692ba31dcdb581aa2380c"
|
||||
},
|
||||
"ce-riffrec-feedback-analysis": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-riffrec-feedback-analysis/SKILL.md",
|
||||
"computedHash": "83f98244f23ef3c13e34497da979281e16957c66cb459fba53a86ee12eebc138"
|
||||
},
|
||||
"ce-setup": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-setup/SKILL.md",
|
||||
"computedHash": "e5e4c6452730ac27e42ee0a13a724898693cced7c50edccb0becba3d99a0e4b6"
|
||||
},
|
||||
"ce-simplify-code": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-simplify-code/SKILL.md",
|
||||
"computedHash": "82e0e4af40d1022ba6867a81bb5e6bfedaf8f238dfb62a9543dc57dfc6412383"
|
||||
},
|
||||
"ce-strategy": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-strategy/SKILL.md",
|
||||
"computedHash": "511506bd9bb029b37d8e0ef7dd0e52a75d8d535d4f181219b561080d8cd8c0f7"
|
||||
},
|
||||
"ce-test-browser": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-test-browser/SKILL.md",
|
||||
"computedHash": "7d95df13ca6a901720e9f41b38b3e0fdf81a32b199e3e522dbe5f08cc903cbda"
|
||||
},
|
||||
"ce-test-xcode": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-test-xcode/SKILL.md",
|
||||
"computedHash": "d11af303f77f3fd2493fcee09afded96a1035587971a0214b6d8814a72329b64"
|
||||
},
|
||||
"ce-work": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-work/SKILL.md",
|
||||
"computedHash": "a77908268a8638f0971fc869128cee50ece7cc152952e10157b84f3ac0730053"
|
||||
},
|
||||
"ce-work-beta": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-work-beta/SKILL.md",
|
||||
"computedHash": "9bf08db9c23dcf4a50ab61cf3d116aae829fa8a66f4ac2937f08c18e265fba1d"
|
||||
},
|
||||
"ce-worktree": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/ce-worktree/SKILL.md",
|
||||
"computedHash": "ed83a34f43ab07252dcb8812b8b0c15ebc57c3760694e997cd8c021c90052875"
|
||||
},
|
||||
"find-skills": {
|
||||
"source": "vercel-labs/skills",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/find-skills/SKILL.md",
|
||||
"computedHash": "9e1c8b3103f92fa8092568a44fe64858de7c5c9dc65ce4bea8f168080e889cfd"
|
||||
},
|
||||
"lfg": {
|
||||
"source": "EveryInc/compound-engineering-plugin",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/lfg/SKILL.md",
|
||||
"computedHash": "802083dbce8f8cbf3dc1d7cafabf20bdc6211d022da8eeb9a1e4dc13ec4b92c0"
|
||||
},
|
||||
"open-code-review": {
|
||||
"source": "alibaba/open-code-review",
|
||||
"sourceType": "github",
|
||||
|
|
|
|||
|
|
@ -0,0 +1,30 @@
|
|||
"""Bitable companion service — multi-dimensional table for AgentKit.
|
||||
|
||||
Provides structured data persistence with field ownership, upsert semantics,
|
||||
formula columns, and grid views. Logically independent (own API/CLI/models/
|
||||
storage), currently co-deployed, UI-level integrated.
|
||||
"""
|
||||
|
||||
from agentkit.bitable.models import (
|
||||
Field,
|
||||
FieldOwner,
|
||||
FieldType,
|
||||
Record,
|
||||
RecalcStatus,
|
||||
RecalcTask,
|
||||
Table,
|
||||
View,
|
||||
ViewType,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"Field",
|
||||
"FieldOwner",
|
||||
"FieldType",
|
||||
"Record",
|
||||
"RecalcStatus",
|
||||
"RecalcTask",
|
||||
"Table",
|
||||
"View",
|
||||
"ViewType",
|
||||
]
|
||||
|
|
@ -0,0 +1,321 @@
|
|||
"""PostgreSQL schema and initialization for the bitable subsystem.
|
||||
|
||||
Uses an independent ``bitable`` schema within the shared PostgreSQL instance.
|
||||
Follows the lazy-init + lock pattern from ``evolution/pg_store.py`` and the
|
||||
``_SCHEMA_VERSION`` migration pattern from ``server/auth/models.py``.
|
||||
|
||||
Schema versioning
|
||||
-----------------
|
||||
:data:`_SCHEMA_VERSION` tracks the current bitable DB schema. The
|
||||
``bitable_meta`` table stores the version so subsequent restarts skip
|
||||
already-applied migrations.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import uuid as _uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import (
|
||||
Column,
|
||||
DateTime,
|
||||
Index,
|
||||
String,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
text,
|
||||
)
|
||||
from sqlalchemy.dialects.postgresql import JSONB
|
||||
from sqlalchemy.orm import DeclarativeBase
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Current schema version — bump when adding migrations.
|
||||
_SCHEMA_VERSION = 1
|
||||
|
||||
_META_SCHEMA_VERSION_KEY = "schema_version"
|
||||
|
||||
|
||||
def _utcnow() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _uuid_str() -> str:
|
||||
return str(_uuid.uuid4())
|
||||
|
||||
|
||||
class BitableBase(DeclarativeBase):
|
||||
"""Declarative base for bitable ORM models (independent schema)."""
|
||||
|
||||
|
||||
class TableModel(BitableBase):
|
||||
"""ORM model for ``bitable.bitable_tables``."""
|
||||
|
||||
__tablename__ = "bitable_tables"
|
||||
__table_args__ = {"schema": "bitable"}
|
||||
|
||||
id = Column(String, primary_key=True, default=_uuid_str)
|
||||
name = Column(String, nullable=False)
|
||||
description = Column(Text, default="")
|
||||
primary_key_field_id = Column(String, nullable=True)
|
||||
owner_user_id = Column(String, nullable=True)
|
||||
created_at = Column(DateTime(timezone=True), default=_utcnow)
|
||||
updated_at = Column(DateTime(timezone=True), default=_utcnow, onupdate=_utcnow)
|
||||
|
||||
|
||||
class FieldModel(BitableBase):
|
||||
"""ORM model for ``bitable.bitable_fields``."""
|
||||
|
||||
__tablename__ = "bitable_fields"
|
||||
__table_args__ = (
|
||||
Index("ix_bitable_fields_table_id", "table_id"),
|
||||
{"schema": "bitable"},
|
||||
)
|
||||
|
||||
id = Column(String, primary_key=True, default=_uuid_str)
|
||||
table_id = Column(String, nullable=False)
|
||||
name = Column(String, nullable=False)
|
||||
field_type = Column(String, nullable=False)
|
||||
config = Column(JSONB, default=dict)
|
||||
owner = Column(String, default="user")
|
||||
created_at = Column(DateTime(timezone=True), default=_utcnow)
|
||||
|
||||
|
||||
class RecordModel(BitableBase):
|
||||
"""ORM model for ``bitable.bitable_records``.
|
||||
|
||||
``values`` is JSONB mapping ``{field_id: value}``. A GIN index supports
|
||||
efficient JSONB key existence checks. A unique expression index on
|
||||
``(table_id, values->>pk_field_id)`` enforces primary-key uniqueness
|
||||
(created dynamically in ``_apply_v1_schema`` because the pk field id is
|
||||
per-table, not a fixed column).
|
||||
"""
|
||||
|
||||
__tablename__ = "bitable_records"
|
||||
__table_args__ = (
|
||||
Index("ix_bitable_records_table_id", "table_id"),
|
||||
Index("ix_bitable_records_values_gin", "values", postgresql_using="gin"),
|
||||
{"schema": "bitable"},
|
||||
)
|
||||
|
||||
id = Column(String, primary_key=True, default=_uuid_str)
|
||||
table_id = Column(String, nullable=False)
|
||||
values = Column(JSONB, default=dict)
|
||||
created_at = Column(DateTime(timezone=True), default=_utcnow)
|
||||
updated_at = Column(DateTime(timezone=True), default=_utcnow, onupdate=_utcnow)
|
||||
|
||||
|
||||
class ViewModel(BitableBase):
|
||||
"""ORM model for ``bitable.bitable_views``."""
|
||||
|
||||
__tablename__ = "bitable_views"
|
||||
__table_args__ = (
|
||||
Index("ix_bitable_views_table_id", "table_id"),
|
||||
{"schema": "bitable"},
|
||||
)
|
||||
|
||||
id = Column(String, primary_key=True, default=_uuid_str)
|
||||
table_id = Column(String, nullable=False)
|
||||
name = Column(String, nullable=False)
|
||||
view_type = Column(String, default="grid")
|
||||
config = Column(JSONB, default=dict)
|
||||
created_at = Column(DateTime(timezone=True), default=_utcnow)
|
||||
|
||||
|
||||
class RecalcQueueModel(BitableBase):
|
||||
"""ORM model for ``bitable.bitable_recalc_queue``.
|
||||
|
||||
The ``(record_id, field_id)`` unique index prevents duplicate enqueues.
|
||||
The ``(status, queued_at)`` index supports efficient worker consumption.
|
||||
"""
|
||||
|
||||
__tablename__ = "bitable_recalc_queue"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("record_id", "field_id", name="uq_recalc_record_field"),
|
||||
Index("ix_recalc_status_queued", "status", "queued_at"),
|
||||
{"schema": "bitable"},
|
||||
)
|
||||
|
||||
id = Column(String, primary_key=True, default=_uuid_str)
|
||||
table_id = Column(String, nullable=False)
|
||||
record_id = Column(String, nullable=False)
|
||||
field_id = Column(String, nullable=False)
|
||||
status = Column(String, default="pending")
|
||||
error_message = Column(Text, nullable=True)
|
||||
queued_at = Column(DateTime(timezone=True), default=_utcnow)
|
||||
completed_at = Column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class MetaModel(BitableBase):
|
||||
"""ORM model for ``bitable.bitable_meta`` — schema version tracking."""
|
||||
|
||||
__tablename__ = "bitable_meta"
|
||||
__table_args__ = {"schema": "bitable"}
|
||||
|
||||
key = Column(String, primary_key=True)
|
||||
value = Column(String, nullable=False)
|
||||
updated_at = Column(DateTime(timezone=True), default=_utcnow, onupdate=_utcnow)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Schema is created via BitableBase.metadata.create_all (see BitableDB.init).
|
||||
# The ORM models above are the single source of truth for table/index DDL.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _resolve_database_url(database_url: str | None = None) -> str | None:
|
||||
"""Resolve PostgreSQL connection URL.
|
||||
|
||||
Priority: explicit arg > env ``DATABASE_URL`` > ``AGENTKIT_DATABASE_URL``.
|
||||
"""
|
||||
if database_url:
|
||||
return database_url
|
||||
return os.environ.get("DATABASE_URL") or os.environ.get("AGENTKIT_DATABASE_URL")
|
||||
|
||||
|
||||
class BitableDB:
|
||||
"""PostgreSQL connection manager for bitable (lazy init + lock).
|
||||
|
||||
Usage::
|
||||
|
||||
db = BitableDB(database_url="postgresql+asyncpg://...")
|
||||
await db.init()
|
||||
# ... use db.engine / db.session_factory ...
|
||||
await db.close()
|
||||
"""
|
||||
|
||||
def __init__(self, database_url: str | None = None) -> None:
|
||||
self._database_url = database_url or _resolve_database_url()
|
||||
self._engine: Any = None
|
||||
self._session_factory: Any = None
|
||||
self._initialized = False
|
||||
self._init_lock = asyncio.Lock()
|
||||
|
||||
@property
|
||||
def database_url(self) -> str | None:
|
||||
return self._database_url
|
||||
|
||||
@property
|
||||
def engine(self) -> Any:
|
||||
return self._engine
|
||||
|
||||
@property
|
||||
def session_factory(self) -> Any:
|
||||
return self._session_factory
|
||||
|
||||
async def _ensure_initialized(self) -> None:
|
||||
"""Lazy-init async engine and session factory (with lock)."""
|
||||
if self._initialized:
|
||||
return
|
||||
|
||||
async with self._init_lock:
|
||||
if self._initialized:
|
||||
return
|
||||
|
||||
if not self._database_url:
|
||||
raise RuntimeError(
|
||||
"No database URL configured for bitable. "
|
||||
"Set DATABASE_URL or AGENTKIT_DATABASE_URL env var."
|
||||
)
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
self._engine = create_async_engine(self._database_url, echo=False)
|
||||
self._session_factory = sessionmaker(
|
||||
self._engine, class_=AsyncSession, expire_on_commit=False
|
||||
)
|
||||
self._initialized = True
|
||||
|
||||
async def init(self) -> None:
|
||||
"""Initialize engine + create schema and tables (idempotent).
|
||||
|
||||
Runs schema migrations based on ``_SCHEMA_VERSION`` stored in
|
||||
``bitable_meta``. Safe to call on every startup.
|
||||
"""
|
||||
await self._ensure_initialized()
|
||||
|
||||
async with self._engine.begin() as conn:
|
||||
# 1. Create the bitable schema (idempotent) — must precede
|
||||
# metadata.create_all since tables live in this schema.
|
||||
await conn.execute(text("CREATE SCHEMA IF NOT EXISTS bitable"))
|
||||
|
||||
# 2. Create all tables/indexes/constraints from ORM metadata.
|
||||
# Uses run_sync because asyncpg doesn't support multi-statement
|
||||
# text() execution; metadata.create_all emits one DDL statement
|
||||
# per table and handles schema-qualified names + GIN indexes.
|
||||
await conn.run_sync(BitableBase.metadata.create_all)
|
||||
|
||||
# 3. Check current schema version
|
||||
result = await conn.execute(
|
||||
text("SELECT value FROM bitable.bitable_meta WHERE key = :key"),
|
||||
{"key": _META_SCHEMA_VERSION_KEY},
|
||||
)
|
||||
row = result.fetchone()
|
||||
current_version = int(row[0]) if row else 0
|
||||
|
||||
# 4. Apply migrations if needed (future versions add elif blocks here)
|
||||
if current_version < _SCHEMA_VERSION:
|
||||
# V1: initial schema (already created above)
|
||||
# Future: if current_version < 2: await _apply_v2_migration(conn)
|
||||
await conn.execute(
|
||||
text(
|
||||
"INSERT INTO bitable.bitable_meta (key, value, updated_at) "
|
||||
"VALUES (:key, :value, NOW()) "
|
||||
"ON CONFLICT (key) DO UPDATE SET value = :value, updated_at = NOW()"
|
||||
),
|
||||
{"key": _META_SCHEMA_VERSION_KEY, "value": str(_SCHEMA_VERSION)},
|
||||
)
|
||||
logger.info("bitable schema migrated: v%d → v%d", current_version, _SCHEMA_VERSION)
|
||||
|
||||
async def close(self) -> None:
|
||||
"""Close engine, release all connections."""
|
||||
if self._engine is not None:
|
||||
await self._engine.dispose()
|
||||
self._engine = None
|
||||
self._session_factory = None
|
||||
self._initialized = False
|
||||
|
||||
async def __aenter__(self) -> "BitableDB":
|
||||
await self.init()
|
||||
return self
|
||||
|
||||
async def __aexit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None:
|
||||
await self.close()
|
||||
|
||||
|
||||
# Module-level singleton (initialized by app lifespan)
|
||||
_db: BitableDB | None = None
|
||||
|
||||
|
||||
async def init_bitable_db(database_url: str | None = None) -> BitableDB:
|
||||
"""Initialize the bitable database (module-level singleton).
|
||||
|
||||
Called from ``app.py`` lifespan. On failure, raises — the caller should
|
||||
catch and degrade gracefully (bitable API returns 503).
|
||||
"""
|
||||
global _db
|
||||
if _db is not None and _db.is_initialized:
|
||||
return _db
|
||||
|
||||
_db = BitableDB(database_url=database_url)
|
||||
await _db.init()
|
||||
logger.info("bitable DB initialized (schema version %d)", _SCHEMA_VERSION)
|
||||
return _db
|
||||
|
||||
|
||||
def get_bitable_db() -> BitableDB | None:
|
||||
"""Return the module-level bitable DB singleton (or None if not init'd)."""
|
||||
return _db
|
||||
|
||||
|
||||
async def close_bitable_db() -> None:
|
||||
"""Close the module-level bitable DB singleton."""
|
||||
global _db
|
||||
if _db is not None:
|
||||
await _db.close()
|
||||
_db = None
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
"""Formula engine for the bitable subsystem.
|
||||
|
||||
Self-contained Python formula engine (KTD3) — no external dependencies.
|
||||
|
||||
Submodules:
|
||||
- :mod:`agentkit.bitable.formula.parser` — parse formula strings to AST
|
||||
- :mod:`agentkit.bitable.formula.engine` — DAG, topological sort, evaluation
|
||||
- :mod:`agentkit.bitable.formula.functions` — built-in function registry
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from agentkit.bitable.formula.engine import (
|
||||
CircularReferenceError,
|
||||
FormulaEngine,
|
||||
)
|
||||
from agentkit.bitable.formula.functions import FUNCTION_REGISTRY
|
||||
from agentkit.bitable.formula.parser import (
|
||||
FormulaParseError,
|
||||
FormulaSecurityError,
|
||||
UnknownFunctionError,
|
||||
parse_formula,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"CircularReferenceError",
|
||||
"FormulaEngine",
|
||||
"FormulaParseError",
|
||||
"FormulaSecurityError",
|
||||
"FUNCTION_REGISTRY",
|
||||
"UnknownFunctionError",
|
||||
"parse_formula",
|
||||
]
|
||||
|
|
@ -0,0 +1,293 @@
|
|||
"""Formula engine — DAG, topological sort, and evaluation.
|
||||
|
||||
Builds a dependency graph of formula fields, detects circular references,
|
||||
and evaluates formulas in topological order.
|
||||
|
||||
Aggregate context (KTD3):
|
||||
- ``SUM({f1})`` — f1 is an aggregate reference (entire column)
|
||||
- ``{f1} + 1`` — f1 is a row reference (current record's value)
|
||||
- ``{f1} + SUM({f2})`` — mixed: row f1 + column f2 sum
|
||||
|
||||
The engine distinguishes these by checking if a field reference appears
|
||||
as a direct argument to an aggregate function.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
from collections import deque
|
||||
from typing import Any
|
||||
|
||||
from agentkit.bitable.formula.functions import AGGREGATE_FUNCTIONS, FUNCTION_REGISTRY
|
||||
from agentkit.bitable.formula.parser import (
|
||||
FormulaParseError,
|
||||
evaluate_ast,
|
||||
parse_formula,
|
||||
)
|
||||
|
||||
|
||||
class CircularReferenceError(Exception):
|
||||
"""Raised when formula fields form a circular dependency."""
|
||||
|
||||
|
||||
class FormulaEngine:
|
||||
"""Formula engine: parse, build DAG, detect cycles, evaluate.
|
||||
|
||||
Usage::
|
||||
|
||||
engine = FormulaEngine()
|
||||
# Register formula fields
|
||||
engine.add_formula(field_id="calc", formula="=SUM({src})", field_refs={"src"})
|
||||
# Evaluate for a specific record
|
||||
result = engine.evaluate(field_id="calc", field_values={"src": [1, 2, 3]})
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
# field_id → (ast_tree, field_mapping, aggregate_refs, row_refs)
|
||||
self._formulas: dict[str, _FormulaEntry] = {}
|
||||
# DAG: field_id → set of field_ids it depends on
|
||||
self._dag: dict[str, set[str]] = {}
|
||||
|
||||
def add_formula(self, field_id: str, formula: str) -> None:
|
||||
"""Register a formula for a field.
|
||||
|
||||
Raises:
|
||||
FormulaParseError: Syntax error.
|
||||
FormulaSecurityError: Disallowed AST node.
|
||||
UnknownFunctionError: Unregistered function.
|
||||
CircularReferenceError: Adding this formula creates a cycle.
|
||||
"""
|
||||
tree, field_mapping = parse_formula(formula, set(FUNCTION_REGISTRY.keys()))
|
||||
|
||||
# Classify field refs into aggregate vs row context
|
||||
aggregate_refs, row_refs = _classify_refs(tree, field_mapping)
|
||||
|
||||
entry = _FormulaEntry(
|
||||
tree=tree,
|
||||
field_mapping=field_mapping,
|
||||
aggregate_refs=aggregate_refs,
|
||||
row_refs=row_refs,
|
||||
formula=formula,
|
||||
)
|
||||
self._formulas[field_id] = entry
|
||||
# Update DAG: this field depends on all referenced fields
|
||||
self._dag[field_id] = aggregate_refs | row_refs
|
||||
|
||||
# Check for cycles
|
||||
cycle = _detect_cycle(self._dag)
|
||||
if cycle:
|
||||
# Rollback
|
||||
del self._formulas[field_id]
|
||||
del self._dag[field_id]
|
||||
raise CircularReferenceError(f"Circular reference detected: {' → '.join(cycle)}")
|
||||
|
||||
def remove_formula(self, field_id: str) -> None:
|
||||
"""Remove a formula from the engine."""
|
||||
self._formulas.pop(field_id, None)
|
||||
self._dag.pop(field_id, None)
|
||||
# Remove edges pointing to this field
|
||||
for deps in self._dag.values():
|
||||
deps.discard(field_id)
|
||||
|
||||
def get_dependencies(self, field_id: str) -> set[str]:
|
||||
"""Get the set of field IDs that ``field_id`` depends on."""
|
||||
return self._dag.get(field_id, set()).copy()
|
||||
|
||||
def get_dependents(self, field_id: str) -> set[str]:
|
||||
"""Get the set of formula field IDs that depend on ``field_id``."""
|
||||
return {fid for fid, deps in self._dag.items() if field_id in deps}
|
||||
|
||||
def topological_order(self) -> list[str]:
|
||||
"""Return all formula field IDs in topological order (Kahn's algorithm)."""
|
||||
return _topological_sort(self._dag)
|
||||
|
||||
def evaluate(
|
||||
self,
|
||||
field_id: str,
|
||||
row_values: dict[str, Any],
|
||||
column_values: dict[str, list[Any]] | None = None,
|
||||
) -> Any:
|
||||
"""Evaluate a formula field for a specific record.
|
||||
|
||||
Args:
|
||||
field_id: The formula field to evaluate.
|
||||
row_values: Field ID → value for the current record (row context).
|
||||
column_values: Field ID → list of all values in that column
|
||||
(aggregate context). Required for aggregate references.
|
||||
|
||||
Returns:
|
||||
The computed value.
|
||||
|
||||
Raises:
|
||||
KeyError: Field ID not registered.
|
||||
FormulaParseError: Field reference not found in values.
|
||||
"""
|
||||
if field_id not in self._formulas:
|
||||
raise KeyError(f"Formula not registered: {field_id}")
|
||||
|
||||
entry = self._formulas[field_id]
|
||||
column_values = column_values or {}
|
||||
|
||||
# Build the field_values dict for the evaluator
|
||||
# Aggregate refs get column values (lists), row refs get row values (scalars)
|
||||
eval_values: dict[str, Any] = {}
|
||||
|
||||
# Map real field IDs to safe names
|
||||
for safe_name, real_id in entry.field_mapping.items():
|
||||
if real_id in entry.aggregate_refs:
|
||||
eval_values[safe_name] = column_values.get(real_id, [])
|
||||
else:
|
||||
eval_values[safe_name] = row_values.get(real_id)
|
||||
|
||||
return evaluate_ast(entry.tree, eval_values, FUNCTION_REGISTRY)
|
||||
|
||||
def evaluate_all_for_record(
|
||||
self,
|
||||
row_values: dict[str, Any],
|
||||
column_values: dict[str, list[Any]] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Evaluate all registered formulas for a record.
|
||||
|
||||
Returns a dict of field_id → computed value.
|
||||
Formulas are evaluated in topological order so that formula-to-formula
|
||||
dependencies are resolved correctly.
|
||||
"""
|
||||
results: dict[str, Any] = {}
|
||||
column_values = column_values or {}
|
||||
|
||||
for field_id in self.topological_order():
|
||||
# Include already-computed formula results in row_values
|
||||
merged_row = {**row_values, **results}
|
||||
try:
|
||||
results[field_id] = self.evaluate(field_id, merged_row, column_values)
|
||||
except (FormulaParseError, ZeroDivisionError, TypeError) as e:
|
||||
results[field_id] = {"__error": str(e)}
|
||||
|
||||
return results
|
||||
|
||||
|
||||
# ── Internal data structures ──────────────────────────────
|
||||
|
||||
|
||||
class _FormulaEntry:
|
||||
"""Parsed formula metadata."""
|
||||
|
||||
__slots__ = ("tree", "field_mapping", "aggregate_refs", "row_refs", "formula")
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
tree: ast.Expression,
|
||||
field_mapping: dict[str, str],
|
||||
aggregate_refs: set[str],
|
||||
row_refs: set[str],
|
||||
formula: str,
|
||||
) -> None:
|
||||
self.tree = tree
|
||||
self.field_mapping = field_mapping
|
||||
self.aggregate_refs = aggregate_refs
|
||||
self.row_refs = row_refs
|
||||
self.formula = formula
|
||||
|
||||
|
||||
# ── DAG utilities ─────────────────────────────────────────
|
||||
|
||||
|
||||
def _detect_cycle(dag: dict[str, set[str]]) -> list[str] | None:
|
||||
"""Detect a cycle in the DAG using DFS. Returns the cycle path or None."""
|
||||
WHITE, GRAY, BLACK = 0, 1, 2
|
||||
color: dict[str, int] = {node: WHITE for node in dag}
|
||||
parent: dict[str, str | None] = {node: None for node in dag}
|
||||
|
||||
def _dfs(node: str) -> list[str] | None:
|
||||
color[node] = GRAY
|
||||
for neighbor in dag.get(node, set()):
|
||||
if neighbor not in color:
|
||||
color[neighbor] = WHITE
|
||||
parent[neighbor] = None
|
||||
if color[neighbor] == GRAY:
|
||||
# Found cycle — reconstruct path
|
||||
cycle = [neighbor]
|
||||
current = node
|
||||
while current is not None and current != neighbor:
|
||||
cycle.append(current)
|
||||
current = parent.get(current)
|
||||
cycle.append(neighbor)
|
||||
cycle.reverse()
|
||||
return cycle
|
||||
if color[neighbor] == WHITE:
|
||||
parent[neighbor] = node
|
||||
result = _dfs(neighbor)
|
||||
if result is not None:
|
||||
return result
|
||||
color[node] = BLACK
|
||||
return None
|
||||
|
||||
for node in dag:
|
||||
if color.get(node, WHITE) == WHITE:
|
||||
result = _dfs(node)
|
||||
if result is not None:
|
||||
return result
|
||||
return None
|
||||
|
||||
|
||||
def _topological_sort(dag: dict[str, set[str]]) -> list[str]:
|
||||
"""Kahn's algorithm for topological sort."""
|
||||
# Build in-degree map
|
||||
in_degree: dict[str, int] = {node: 0 for node in dag}
|
||||
for node, deps in dag.items():
|
||||
for dep in deps:
|
||||
if dep in in_degree:
|
||||
in_degree[node] += 1
|
||||
|
||||
# Start with nodes that have no dependencies
|
||||
queue = deque(node for node, degree in in_degree.items() if degree == 0)
|
||||
result: list[str] = []
|
||||
|
||||
while queue:
|
||||
node = queue.popleft()
|
||||
result.append(node)
|
||||
# Find all nodes that depend on this node
|
||||
for other_node, deps in dag.items():
|
||||
if node in deps:
|
||||
in_degree[other_node] -= 1
|
||||
if in_degree[other_node] == 0 and other_node not in result:
|
||||
queue.append(other_node)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def _classify_refs(
|
||||
tree: ast.Expression, field_mapping: dict[str, str]
|
||||
) -> tuple[set[str], set[str]]:
|
||||
"""Classify field references into aggregate (column) and row context.
|
||||
|
||||
A field reference is aggregate if it appears as a direct argument to
|
||||
an aggregate function (SUM/AVG/COUNT/MIN/MAX). Otherwise it's row context.
|
||||
"""
|
||||
aggregate_refs: set[str] = set()
|
||||
row_refs: set[str] = set()
|
||||
|
||||
# Get all safe names → real field IDs
|
||||
safe_to_real = field_mapping
|
||||
|
||||
class _Classifier(ast.NodeVisitor):
|
||||
def visit_Call(self, node: ast.Call) -> None:
|
||||
if isinstance(node.func, ast.Name) and node.func.id in AGGREGATE_FUNCTIONS:
|
||||
# Direct arguments to aggregate functions are column refs
|
||||
for arg in node.args:
|
||||
if isinstance(arg, ast.Name) and arg.id in safe_to_real:
|
||||
aggregate_refs.add(safe_to_real[arg.id])
|
||||
else:
|
||||
# Non-field args (e.g., literals) — visit normally
|
||||
self.visit(arg)
|
||||
else:
|
||||
self.generic_visit(node)
|
||||
|
||||
def visit_Name(self, node: ast.Name) -> None:
|
||||
if node.id in safe_to_real:
|
||||
real_id = safe_to_real[node.id]
|
||||
if real_id not in aggregate_refs:
|
||||
row_refs.add(real_id)
|
||||
|
||||
_Classifier().visit(tree)
|
||||
return aggregate_refs, row_refs
|
||||
|
|
@ -0,0 +1,101 @@
|
|||
"""Built-in formula functions for the bitable formula engine.
|
||||
|
||||
v1 implements: SUM, AVG, COUNT, MIN, MAX, CONCAT, ABS, ROUND, IF, LEN.
|
||||
|
||||
Aggregate functions (SUM/AVG/COUNT/MIN/MAX) accept a list of values
|
||||
(the entire column) and return a scalar. Non-aggregate functions
|
||||
(ABS/ROUND/IF/LEN/CONCAT) operate on scalars.
|
||||
|
||||
The engine determines whether to pass a column (list) or scalar (row value)
|
||||
based on the calling context — see :mod:`agentkit.bitable.formula.engine`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Callable
|
||||
|
||||
# ── Aggregate functions (operate on lists) ────────────────
|
||||
|
||||
|
||||
def _sum(values: list[Any]) -> float | int:
|
||||
"""Sum of numeric values, ignoring None/empty."""
|
||||
total = 0
|
||||
for v in values:
|
||||
if v is None or v == "":
|
||||
continue
|
||||
total += v
|
||||
return total
|
||||
|
||||
|
||||
def _avg(values: list[Any]) -> float:
|
||||
"""Average of numeric values, ignoring None/empty."""
|
||||
nums = [v for v in values if v is not None and v != ""]
|
||||
if not nums:
|
||||
return 0.0
|
||||
return sum(nums) / len(nums)
|
||||
|
||||
|
||||
def _count(values: list[Any]) -> int:
|
||||
"""Count of non-empty values."""
|
||||
return sum(1 for v in values if v is not None and v != "")
|
||||
|
||||
|
||||
def _min(values: list[Any]) -> Any:
|
||||
"""Minimum of numeric values, ignoring None/empty."""
|
||||
nums = [v for v in values if v is not None and v != ""]
|
||||
if not nums:
|
||||
return 0
|
||||
return min(nums)
|
||||
|
||||
|
||||
def _max(values: list[Any]) -> Any:
|
||||
"""Maximum of numeric values, ignoring None/empty."""
|
||||
nums = [v for v in values if v is not None and v != ""]
|
||||
if not nums:
|
||||
return 0
|
||||
return max(nums)
|
||||
|
||||
|
||||
# ── Scalar functions ──────────────────────────────────────
|
||||
|
||||
|
||||
def _abs(value: Any) -> Any:
|
||||
return abs(value)
|
||||
|
||||
|
||||
def _round(value: Any, digits: int = 0) -> float:
|
||||
return round(value, digits)
|
||||
|
||||
|
||||
def _if(condition: Any, true_val: Any, false_val: Any = None) -> Any:
|
||||
return true_val if condition else false_val
|
||||
|
||||
|
||||
def _len(value: Any) -> int:
|
||||
if value is None:
|
||||
return 0
|
||||
return len(str(value))
|
||||
|
||||
|
||||
def _concat(*args: Any) -> str:
|
||||
"""Concatenate all arguments as strings."""
|
||||
return "".join(str(a) for a in args if a is not None)
|
||||
|
||||
|
||||
# ── Registry ──────────────────────────────────────────────
|
||||
|
||||
# Functions that aggregate a column (receive a list of all column values)
|
||||
AGGREGATE_FUNCTIONS: frozenset[str] = frozenset({"SUM", "AVG", "COUNT", "MIN", "MAX"})
|
||||
|
||||
FUNCTION_REGISTRY: dict[str, Callable[..., Any]] = {
|
||||
"SUM": _sum,
|
||||
"AVG": _avg,
|
||||
"COUNT": _count,
|
||||
"MIN": _min,
|
||||
"MAX": _max,
|
||||
"ABS": _abs,
|
||||
"ROUND": _round,
|
||||
"IF": _if,
|
||||
"LEN": _len,
|
||||
"CONCAT": _concat,
|
||||
}
|
||||
|
|
@ -0,0 +1,311 @@
|
|||
"""Formula parser — converts formula strings to safe Python AST.
|
||||
|
||||
KTD7 security: uses ``ast.parse`` then a restricted ``NodeVisitor`` that
|
||||
only allows whitelist nodes. **Never** uses ``eval()`` / ``exec()``.
|
||||
|
||||
Formula syntax:
|
||||
- Starts with ``=`` (stripped before parsing).
|
||||
- Field references: ``{field_id}`` — converted to ``Name(field_id)`` nodes.
|
||||
- Arithmetic: ``+``, ``-``, ``*``, ``/``, ``%``, ``**``.
|
||||
- Comparison: ``==``, ``!=``, ``<``, ``>``, ``<=``, ``>=``.
|
||||
- Boolean: ``and``, ``or``, ``not``.
|
||||
- String concat: ``+`` on strings, or ``CONCAT(...)``.
|
||||
- Function calls: ``SUM(...)``, ``AVG(...)``, etc. (registered functions only).
|
||||
- Conditional: ``IF(cond, a, b)`` or Python ``a if cond else b``.
|
||||
- Literals: numbers, strings (single/double quotes).
|
||||
|
||||
Examples::
|
||||
=1+2*3 → 7
|
||||
=SUM({f1}) → aggregate sum of column f1
|
||||
={f1} + {f2} → row-level sum of fields f1 and f2
|
||||
=CONCAT({f1}, "-") → string concatenation
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from agentkit.bitable.formula.functions import FUNCTION_REGISTRY
|
||||
|
||||
# ── Exceptions ────────────────────────────────────────────
|
||||
|
||||
|
||||
class FormulaParseError(Exception):
|
||||
"""Raised when a formula string cannot be parsed."""
|
||||
|
||||
|
||||
class FormulaSecurityError(Exception):
|
||||
"""Raised when a formula AST contains a disallowed node (KTD7)."""
|
||||
|
||||
|
||||
class UnknownFunctionError(Exception):
|
||||
"""Raised when a formula calls a function not in the registry."""
|
||||
|
||||
|
||||
# ── Field reference substitution ──────────────────────────
|
||||
|
||||
# Match {field_id} — field IDs are UUIDs or alphanumeric.
|
||||
_FIELD_REF_RE = re.compile(r"\{([a-zA-Z0-9_-]+)\}")
|
||||
|
||||
|
||||
def _substitute_field_refs(formula: str) -> tuple[str, dict[str, str]]:
|
||||
"""Replace ``{field_id}`` with ``_f_<safe_name>`` (a Python Name node).
|
||||
|
||||
Field IDs are UUIDs that may start with a digit, which is invalid in Python
|
||||
identifiers. We prefix with ``_f_`` and replace hyphens with underscores.
|
||||
A reverse mapping is returned so the engine can map back to real field IDs.
|
||||
"""
|
||||
mapping: dict[str, str] = {}
|
||||
|
||||
def _replace(match: re.Match[str]) -> str:
|
||||
field_id = match.group(1)
|
||||
# Convert UUID-style field_id to a valid Python identifier
|
||||
safe_name = "_f_" + field_id.replace("-", "_")
|
||||
mapping[safe_name] = field_id
|
||||
return safe_name
|
||||
|
||||
result = _FIELD_REF_RE.sub(_replace, formula)
|
||||
return result, mapping
|
||||
|
||||
|
||||
# ── AST whitelist (KTD7) ──────────────────────────────────
|
||||
|
||||
_ALLOWED_NODES: frozenset[type[ast.AST]] = frozenset(
|
||||
{
|
||||
ast.Expression,
|
||||
ast.BinOp,
|
||||
ast.UnaryOp,
|
||||
ast.BoolOp,
|
||||
ast.Compare,
|
||||
ast.Call,
|
||||
ast.Name,
|
||||
ast.Constant,
|
||||
ast.IfExp,
|
||||
ast.Load,
|
||||
ast.Add,
|
||||
ast.Sub,
|
||||
ast.Mult,
|
||||
ast.Div,
|
||||
ast.Mod,
|
||||
ast.Pow,
|
||||
ast.USub,
|
||||
ast.UAdd,
|
||||
ast.Not,
|
||||
ast.And,
|
||||
ast.Or,
|
||||
ast.Eq,
|
||||
ast.NotEq,
|
||||
ast.Lt,
|
||||
ast.Gt,
|
||||
ast.LtE,
|
||||
ast.GtE,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class _SecurityVisitor(ast.NodeVisitor):
|
||||
"""Visit AST nodes, reject any not in the whitelist (KTD7)."""
|
||||
|
||||
def __init__(self, allowed_functions: set[str]) -> None:
|
||||
self._allowed_functions = allowed_functions
|
||||
|
||||
def generic_visit(self, node: ast.AST) -> None: # noqa: D401
|
||||
node_type = type(node)
|
||||
if node_type not in _ALLOWED_NODES:
|
||||
raise FormulaSecurityError(
|
||||
f"Disallowed AST node: {node_type.__name__}. Formula contains unsafe constructs."
|
||||
)
|
||||
super().generic_visit(node)
|
||||
|
||||
def visit_Call(self, node: ast.Call) -> None:
|
||||
# Check that the function being called is a registered Name
|
||||
if not isinstance(node.func, ast.Name):
|
||||
raise FormulaSecurityError(
|
||||
"Only direct function calls by name are allowed. "
|
||||
"Method calls and attribute access are forbidden."
|
||||
)
|
||||
if node.func.id not in self._allowed_functions:
|
||||
raise UnknownFunctionError(
|
||||
f"Unknown function: '{node.func.id}'. Allowed: {sorted(self._allowed_functions)}"
|
||||
)
|
||||
self.generic_visit(node)
|
||||
|
||||
|
||||
# ── Public API ────────────────────────────────────────────
|
||||
|
||||
|
||||
def parse_formula(
|
||||
formula: str, allowed_functions: set[str] | None = None
|
||||
) -> tuple[ast.Expression, dict[str, str]]:
|
||||
"""Parse a formula string into a safe AST.
|
||||
|
||||
Args:
|
||||
formula: Formula string, optionally starting with ``=``.
|
||||
allowed_functions: Set of registered function names. If None,
|
||||
all functions are allowed (used for syntax-only validation).
|
||||
|
||||
Returns:
|
||||
Tuple of (AST expression, field_ref_mapping) where
|
||||
field_ref_mapping maps safe Python identifiers to original field IDs.
|
||||
|
||||
Raises:
|
||||
FormulaParseError: Syntax error in formula.
|
||||
FormulaSecurityError: Formula contains disallowed AST nodes.
|
||||
UnknownFunctionError: Formula calls an unregistered function.
|
||||
"""
|
||||
expr = formula.strip()
|
||||
if expr.startswith("="):
|
||||
expr = expr[1:]
|
||||
|
||||
if not expr:
|
||||
raise FormulaParseError("Empty formula")
|
||||
|
||||
# Substitute field references {field_id} → safe_name
|
||||
substituted, field_mapping = _substitute_field_refs(expr)
|
||||
|
||||
try:
|
||||
tree = ast.parse(substituted, mode="eval")
|
||||
except SyntaxError as e:
|
||||
raise FormulaParseError(f"Syntax error in formula: {e}") from e
|
||||
|
||||
# Security check
|
||||
# When allowed_functions is None, use all registered functions (syntax-only validation)
|
||||
if allowed_functions is None:
|
||||
allowed = set(FUNCTION_REGISTRY.keys())
|
||||
else:
|
||||
allowed = allowed_functions
|
||||
visitor = _SecurityVisitor(allowed)
|
||||
visitor.visit(tree)
|
||||
|
||||
return tree, field_mapping # type: ignore[return-value]
|
||||
|
||||
|
||||
def evaluate_ast(
|
||||
tree: ast.Expression,
|
||||
field_values: dict[str, Any],
|
||||
functions: dict[str, Any],
|
||||
) -> Any:
|
||||
"""Evaluate a parsed formula AST against field values and functions.
|
||||
|
||||
This is NOT ``eval()`` — it's a manual AST walker that only processes
|
||||
whitelist nodes. Field references (Name nodes) are resolved from
|
||||
``field_values``; function calls from ``functions``.
|
||||
|
||||
Args:
|
||||
tree: Parsed AST from :func:`parse_formula`.
|
||||
field_values: Mapping of field safe-name → value (scalar or list for aggregates).
|
||||
functions: Mapping of function name → callable.
|
||||
|
||||
Returns:
|
||||
The computed value.
|
||||
"""
|
||||
return _eval_node(tree.body, field_values, functions)
|
||||
|
||||
|
||||
def _eval_node(node: ast.AST, fields: dict[str, Any], functions: dict[str, Any]) -> Any:
|
||||
"""Recursively evaluate an AST node."""
|
||||
if isinstance(node, ast.Constant):
|
||||
return node.value
|
||||
|
||||
if isinstance(node, ast.Name):
|
||||
if node.id not in fields:
|
||||
raise FormulaParseError(f"Unknown field reference: {node.id}")
|
||||
return fields[node.id]
|
||||
|
||||
if isinstance(node, ast.BinOp):
|
||||
left = _eval_node(node.left, fields, functions)
|
||||
right = _eval_node(node.right, fields, functions)
|
||||
return _apply_binop(node.op, left, right)
|
||||
|
||||
if isinstance(node, ast.UnaryOp):
|
||||
operand = _eval_node(node.operand, fields, functions)
|
||||
if isinstance(node.op, ast.USub):
|
||||
return -operand
|
||||
if isinstance(node.op, ast.UAdd):
|
||||
return +operand
|
||||
if isinstance(node.op, ast.Not):
|
||||
return not operand
|
||||
raise FormulaSecurityError(f"Disallowed unary op: {type(node.op).__name__}")
|
||||
|
||||
if isinstance(node, ast.BoolOp):
|
||||
values = [_eval_node(v, fields, functions) for v in node.values]
|
||||
if isinstance(node.op, ast.And):
|
||||
result = True
|
||||
for v in values:
|
||||
result = result and v
|
||||
if not result:
|
||||
return result
|
||||
return result
|
||||
if isinstance(node.op, ast.Or):
|
||||
result = False
|
||||
for v in values:
|
||||
result = result or v
|
||||
if not result:
|
||||
return result
|
||||
return result
|
||||
raise FormulaSecurityError(f"Disallowed bool op: {type(node.op).__name__}")
|
||||
|
||||
if isinstance(node, ast.Compare):
|
||||
left = _eval_node(node.left, fields, functions)
|
||||
for op, comparator in zip(node.ops, node.comparators):
|
||||
right = _eval_node(comparator, fields, functions)
|
||||
if not _apply_compare(op, left, right):
|
||||
return False
|
||||
left = right
|
||||
return True
|
||||
|
||||
if isinstance(node, ast.IfExp):
|
||||
test = _eval_node(node.test, fields, functions)
|
||||
if test:
|
||||
return _eval_node(node.body, fields, functions)
|
||||
return _eval_node(node.orelse, fields, functions)
|
||||
|
||||
if isinstance(node, ast.Call):
|
||||
if not isinstance(node.func, ast.Name):
|
||||
raise FormulaSecurityError("Only named function calls allowed")
|
||||
func_name = node.func.id
|
||||
if func_name not in functions:
|
||||
raise UnknownFunctionError(f"Unknown function: {func_name}")
|
||||
args = [_eval_node(a, fields, functions) for a in node.args]
|
||||
return functions[func_name](*args)
|
||||
|
||||
raise FormulaSecurityError(f"Disallowed node during evaluation: {type(node).__name__}")
|
||||
|
||||
|
||||
def _apply_binop(op: ast.AST, left: Any, right: Any) -> Any:
|
||||
"""Apply a binary operator."""
|
||||
if isinstance(op, ast.Add):
|
||||
# String concat or numeric addition
|
||||
if isinstance(left, str) or isinstance(right, str):
|
||||
return f"{left}{right}"
|
||||
return left + right
|
||||
if isinstance(op, ast.Sub):
|
||||
return left - right
|
||||
if isinstance(op, ast.Mult):
|
||||
return left * right
|
||||
if isinstance(op, ast.Div):
|
||||
return left / right
|
||||
if isinstance(op, ast.Mod):
|
||||
return left % right
|
||||
if isinstance(op, ast.Pow):
|
||||
return left**right
|
||||
raise FormulaSecurityError(f"Disallowed binary op: {type(op).__name__}")
|
||||
|
||||
|
||||
def _apply_compare(op: ast.AST, left: Any, right: Any) -> bool:
|
||||
"""Apply a comparison operator."""
|
||||
if isinstance(op, ast.Eq):
|
||||
return left == right
|
||||
if isinstance(op, ast.NotEq):
|
||||
return left != right
|
||||
if isinstance(op, ast.Lt):
|
||||
return left < right
|
||||
if isinstance(op, ast.Gt):
|
||||
return left > right
|
||||
if isinstance(op, ast.LtE):
|
||||
return left <= right
|
||||
if isinstance(op, ast.GtE):
|
||||
return left >= right
|
||||
raise FormulaSecurityError(f"Disallowed compare op: {type(op).__name__}")
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
"""Data ingestion modules for the bitable subsystem.
|
||||
|
||||
Pure data transformation — no HTTP calls. The BitableTool orchestrates
|
||||
HTTP writes to the bitable REST API; these modules only parse/transform
|
||||
external data into bitable-ready field definitions and record lists.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from agentkit.bitable.ingestion.api_collector import transform_records
|
||||
from agentkit.bitable.ingestion.database import (
|
||||
DB_TYPE_MAP,
|
||||
import_table,
|
||||
infer_field_type,
|
||||
)
|
||||
from agentkit.bitable.ingestion.excel import (
|
||||
ParsedSheet,
|
||||
parse_excel,
|
||||
parse_excel_bytes,
|
||||
parse_excel_url,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"DB_TYPE_MAP",
|
||||
"ParsedSheet",
|
||||
"import_table",
|
||||
"infer_field_type",
|
||||
"parse_excel",
|
||||
"parse_excel_bytes",
|
||||
"parse_excel_url",
|
||||
"transform_records",
|
||||
]
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
"""API collector — transform Agent-collected structured data for bitable upsert.
|
||||
|
||||
The Agent already has crawl/API tools (web_crawl, web_search, etc.). This
|
||||
module only handles the "shape" transformation: map arbitrary JSON records
|
||||
to bitable field IDs via a ``field_mapping`` dict, then return records
|
||||
ready for the upsert API.
|
||||
|
||||
Usage::
|
||||
|
||||
transformed = transform_records(
|
||||
records=[{"name": "Alice", "age": 30}],
|
||||
field_mapping={"name": "fld_abc", "age": "fld_def"},
|
||||
)
|
||||
# → [{"fld_abc": "Alice", "fld_def": 30}]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
def transform_records(
|
||||
records: list[dict[str, Any]],
|
||||
field_mapping: dict[str, str],
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Map source record keys to bitable field IDs via field_mapping.
|
||||
|
||||
Keys not in ``field_mapping`` are dropped. Values are passed through
|
||||
as-is (the bitable upsert API handles type coercion).
|
||||
|
||||
Args:
|
||||
records: List of source records (arbitrary keys).
|
||||
field_mapping: ``{source_key: bitable_field_id}``.
|
||||
|
||||
Returns:
|
||||
List of records with bitable field IDs as keys.
|
||||
"""
|
||||
if not records:
|
||||
return []
|
||||
if not field_mapping:
|
||||
return []
|
||||
|
||||
transformed: list[dict[str, Any]] = []
|
||||
for rec in records:
|
||||
out: dict[str, Any] = {}
|
||||
for src_key, field_id in field_mapping.items():
|
||||
if src_key in rec:
|
||||
out[field_id] = rec[src_key]
|
||||
if out:
|
||||
transformed.append(out)
|
||||
return transformed
|
||||
|
|
@ -0,0 +1,171 @@
|
|||
"""Database ingestion — reflect external DB tables into bitable-ready data.
|
||||
|
||||
Uses SQLAlchemy reflection to read table structure and rows. The caller
|
||||
(BitableTool) then creates a bitable table + fields and upserts the rows
|
||||
via the bitable REST API.
|
||||
|
||||
Type mapping (KTD: DB → bitable):
|
||||
INTEGER / BIGINT / SMALLINT / NUMERIC / FLOAT / DECIMAL → number
|
||||
VARCHAR / TEXT / CHAR / UUID → text
|
||||
TIMESTAMP / DATETIME / DATE → date
|
||||
BOOLEAN → text (v1: no bool type)
|
||||
JSON / JSONB → text
|
||||
fallback → text
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import (
|
||||
BigInteger,
|
||||
Boolean,
|
||||
Date,
|
||||
DateTime,
|
||||
Float,
|
||||
Integer,
|
||||
Numeric,
|
||||
SmallInteger,
|
||||
String,
|
||||
Text,
|
||||
create_engine,
|
||||
inspect,
|
||||
select,
|
||||
)
|
||||
from sqlalchemy.engine import Engine
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ponytail: Static mapping covers all common SQL types. Unknown types fall
|
||||
# back to text — safe but lossy. Upgrade path: add entries as needed.
|
||||
DB_TYPE_MAP: dict[type, str] = {
|
||||
Integer: "number",
|
||||
BigInteger: "number",
|
||||
SmallInteger: "number",
|
||||
Numeric: "number",
|
||||
Float: "number",
|
||||
String: "text",
|
||||
Text: "text",
|
||||
DateTime: "date",
|
||||
Date: "date",
|
||||
Boolean: "text",
|
||||
}
|
||||
|
||||
# Batch size for reading rows from the source DB
|
||||
READ_BATCH = 1000
|
||||
|
||||
|
||||
def infer_field_type(sqla_type: Any) -> str:
|
||||
"""Map a SQLAlchemy column type instance or class to a bitable field type.
|
||||
|
||||
Handles both type instances (``Integer()``) and type classes (``Integer``).
|
||||
Falls back to ``"text"`` for unknown types.
|
||||
"""
|
||||
for sqla_cls, bitable_type in DB_TYPE_MAP.items():
|
||||
if isinstance(sqla_type, sqla_cls):
|
||||
return bitable_type
|
||||
# If sqla_type is a class (not instance), check subclass relationship
|
||||
if isinstance(sqla_type, type):
|
||||
for sqla_cls, bitable_type in DB_TYPE_MAP.items():
|
||||
if issubclass(sqla_type, sqla_cls):
|
||||
return bitable_type
|
||||
return "text"
|
||||
|
||||
|
||||
def import_table(
|
||||
connection_string: str,
|
||||
table_name: str,
|
||||
*,
|
||||
max_rows: int = 50_000,
|
||||
) -> dict[str, Any]:
|
||||
"""Reflect a single table from an external DB.
|
||||
|
||||
Returns ``{"table_name": str, "fields": [...], "records": [...],
|
||||
"primary_key": str | None, "row_count": int}``.
|
||||
|
||||
Raises ``ConnectionError`` if the DB is unreachable.
|
||||
"""
|
||||
try:
|
||||
engine = create_engine(connection_string)
|
||||
except Exception as e:
|
||||
raise ConnectionError(f"Failed to create engine for connection string: {e}") from e
|
||||
|
||||
try:
|
||||
return _reflect_and_read(engine, table_name, max_rows)
|
||||
finally:
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def _reflect_and_read(engine: Engine, table_name: str, max_rows: int) -> dict[str, Any]:
|
||||
"""Reflect one table and read its rows."""
|
||||
insp = inspect(engine)
|
||||
|
||||
# Validate table exists
|
||||
if table_name not in insp.get_table_names():
|
||||
raise ValueError(f"Table {table_name!r} not found in database")
|
||||
|
||||
from sqlalchemy import Table, MetaData
|
||||
|
||||
metadata = MetaData()
|
||||
table = Table(table_name, metadata, autoload_with=engine)
|
||||
|
||||
# Build field definitions
|
||||
fields: list[dict[str, Any]] = []
|
||||
pk_columns = list(table.primary_key.columns)
|
||||
pk_name = pk_columns[0].name if pk_columns else None
|
||||
|
||||
for col in table.columns:
|
||||
field_type = infer_field_type(col.type)
|
||||
fields.append(
|
||||
{
|
||||
"name": col.name,
|
||||
"field_type": field_type,
|
||||
"is_primary_key": col.name == pk_name,
|
||||
}
|
||||
)
|
||||
|
||||
# If no PK, auto-generate one
|
||||
if pk_name is None:
|
||||
fields.insert(0, {"name": "id", "field_type": "text", "is_primary_key": True})
|
||||
pk_name = "id"
|
||||
|
||||
# Read rows
|
||||
records: list[dict[str, Any]] = []
|
||||
with engine.connect() as conn:
|
||||
result = conn.execute(select(table))
|
||||
for i, row in enumerate(result):
|
||||
if i >= max_rows:
|
||||
logger.warning("Table %r truncated at %d rows during import", table_name, max_rows)
|
||||
break
|
||||
rec: dict[str, Any] = {}
|
||||
for col in table.columns:
|
||||
val = getattr(row, col.name, None)
|
||||
if val is not None:
|
||||
val = _serialize(val)
|
||||
rec[col.name] = val
|
||||
records.append(rec)
|
||||
|
||||
return {
|
||||
"table_name": table_name,
|
||||
"fields": fields,
|
||||
"records": records,
|
||||
"primary_key": pk_name,
|
||||
"row_count": len(records),
|
||||
}
|
||||
|
||||
|
||||
def _serialize(val: Any) -> Any:
|
||||
"""Serialize a DB value to JSON-safe form."""
|
||||
from datetime import date, datetime
|
||||
from decimal import Decimal
|
||||
|
||||
if isinstance(val, datetime):
|
||||
return val.isoformat()
|
||||
if isinstance(val, date):
|
||||
return val.isoformat()
|
||||
if isinstance(val, Decimal):
|
||||
return float(val)
|
||||
if isinstance(val, bytes):
|
||||
return val.decode("utf-8", errors="replace")
|
||||
return val
|
||||
|
|
@ -0,0 +1,249 @@
|
|||
"""Excel ingestion — parse .xlsx into structured sheets for bitable import.
|
||||
|
||||
Reuses openpyxl (already a dependency via ``agentkit.memory.document_loader``).
|
||||
Returns structured data ``{sheet_name: [{col: val}, ...]}`` rather than
|
||||
Markdown text. First row is treated as field names; types are auto-inferred
|
||||
from column values.
|
||||
|
||||
Known limitations (same as ``document_loader._parse_xlsx``):
|
||||
- ``data_only=True`` returns ``None`` for formulas never opened in Excel.
|
||||
- Merged cells: only the top-left cell has a value; others are ``None``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import ipaddress
|
||||
import logging
|
||||
import socket
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
MAX_ROWS_PER_SHEET = 10_000
|
||||
MAX_CELL_CHARS = 10_000
|
||||
|
||||
|
||||
@dataclass
|
||||
class ParsedSheet:
|
||||
"""One parsed Excel sheet ready for bitable import."""
|
||||
|
||||
name: str
|
||||
columns: list[str] = field(default_factory=list)
|
||||
field_types: list[str] = field(default_factory=list) # "text" | "number" | "date"
|
||||
records: list[dict[str, Any]] = field(default_factory=list)
|
||||
|
||||
|
||||
def parse_excel(file_path: str | Path) -> list[ParsedSheet]:
|
||||
"""Parse an .xlsx file from disk into structured sheets."""
|
||||
path = Path(file_path)
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(f"Excel file not found: {path}")
|
||||
content = path.read_bytes()
|
||||
return parse_excel_bytes(content)
|
||||
|
||||
|
||||
def parse_excel_url(url: str, *, timeout: float = 30.0) -> list[ParsedSheet]:
|
||||
"""Download an .xlsx from a URL and parse it.
|
||||
|
||||
Validates the URL to prevent SSRF: only http/https schemes are allowed,
|
||||
and the resolved host must not be a private, loopback, link-local, or
|
||||
reserved IP address. Hostnames are resolved before the request so the
|
||||
check covers DNS rebinding to internal IPs.
|
||||
"""
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme not in ("http", "https"):
|
||||
raise ValueError(f"Disallowed URL scheme: {parsed.scheme!r} (only http/https)")
|
||||
if not parsed.hostname:
|
||||
raise ValueError("URL has no hostname")
|
||||
|
||||
_assert_safe_host(parsed.hostname)
|
||||
|
||||
resp = httpx.get(url, timeout=timeout, follow_redirects=False)
|
||||
# Follow redirects manually, re-validating each Location (KTD: SSRF guard).
|
||||
seen_redirects = 0
|
||||
while resp.is_redirect and seen_redirects < 5:
|
||||
seen_redirects += 1
|
||||
next_url = httpx.URL(url).join(resp.headers["location"])
|
||||
if next_url.scheme not in ("http", "https") or not next_url.host:
|
||||
raise ValueError(f"Unsafe redirect target: {next_url}")
|
||||
_assert_safe_host(next_url.host)
|
||||
resp = httpx.get(next_url, timeout=timeout, follow_redirects=False)
|
||||
url = str(next_url)
|
||||
|
||||
resp.raise_for_status()
|
||||
return parse_excel_bytes(resp.content)
|
||||
|
||||
|
||||
def _assert_safe_host(host: str) -> None:
|
||||
"""Raise ``ValueError`` if ``host`` resolves to a private/loopback/reserved IP.
|
||||
|
||||
Accepts IPv4/IPv6 literals and DNS names. DNS names are resolved and every
|
||||
returned address is checked — any private/loopback/link-local/reserved
|
||||
address blocks the request.
|
||||
"""
|
||||
# ponytail: blocks RFC1918, loopback, link-local, and reserved ranges.
|
||||
# Ceiling: does not defend against DNS rebinding after the check (TOCTOU);
|
||||
# upgrade path is to pin resolved IP in the httpx transport.
|
||||
try:
|
||||
addr = ipaddress.ip_address(host)
|
||||
except ValueError:
|
||||
# Hostname — resolve and check all A/AAAA records.
|
||||
try:
|
||||
infos = socket.getaddrinfo(host, None)
|
||||
except socket.gaierror as e:
|
||||
raise ValueError(f"Cannot resolve host {host!r}: {e}") from e
|
||||
for info in infos:
|
||||
sockaddr = info[4]
|
||||
ip_str = sockaddr[0]
|
||||
try:
|
||||
addr = ipaddress.ip_address(ip_str)
|
||||
except ValueError:
|
||||
continue
|
||||
if _is_unsafe_ip(addr):
|
||||
raise ValueError(f"Host {host!r} resolves to private/reserved IP {addr}")
|
||||
return
|
||||
|
||||
if _is_unsafe_ip(addr):
|
||||
raise ValueError(f"Host {host!r} is a private/loopback/reserved IP: {addr}")
|
||||
|
||||
|
||||
def _is_unsafe_ip(addr: ipaddress._BaseAddress) -> bool:
|
||||
"""True if the address is private, loopback, link-local, reserved, or multicast."""
|
||||
return (
|
||||
addr.is_private
|
||||
or addr.is_loopback
|
||||
or addr.is_link_local
|
||||
or addr.is_reserved
|
||||
or addr.is_multicast
|
||||
or addr.is_unspecified
|
||||
)
|
||||
|
||||
|
||||
def parse_excel_bytes(content: bytes) -> list[ParsedSheet]:
|
||||
"""Parse Excel content from bytes. Raises ``ValueError`` on corrupt files."""
|
||||
try:
|
||||
from openpyxl import load_workbook
|
||||
except ImportError as e:
|
||||
raise ImportError("openpyxl is required for Excel ingestion") from e
|
||||
|
||||
try:
|
||||
wb = load_workbook(io.BytesIO(content), data_only=True, read_only=True)
|
||||
except Exception as e:
|
||||
raise ValueError(f"Failed to parse Excel file: {e}") from e
|
||||
|
||||
sheets: list[ParsedSheet] = []
|
||||
try:
|
||||
for ws in wb.worksheets:
|
||||
sheet = _parse_worksheet(ws)
|
||||
if sheet is not None:
|
||||
sheets.append(sheet)
|
||||
finally:
|
||||
wb.close()
|
||||
return sheets
|
||||
|
||||
|
||||
def _parse_worksheet(ws) -> ParsedSheet | None:
|
||||
"""Parse a single worksheet. Returns ``None`` for completely empty sheets."""
|
||||
rows_iter = ws.iter_rows(values_only=True)
|
||||
|
||||
# First row = headers
|
||||
try:
|
||||
header_row = next(rows_iter)
|
||||
except StopIteration:
|
||||
return None # empty sheet
|
||||
|
||||
headers = [str(v).strip() if v is not None else f"col_{i}" for i, v in enumerate(header_row)]
|
||||
# Deduplicate headers
|
||||
seen: dict[str, int] = {}
|
||||
clean_headers: list[str] = []
|
||||
for h in headers:
|
||||
if h in seen:
|
||||
seen[h] += 1
|
||||
clean_headers.append(f"{h}_{seen[h]}")
|
||||
else:
|
||||
seen[h] = 0
|
||||
clean_headers.append(h)
|
||||
|
||||
# Collect data rows
|
||||
data_rows: list[tuple] = []
|
||||
for row in rows_iter:
|
||||
if len(data_rows) >= MAX_ROWS_PER_SHEET:
|
||||
logger.warning("Sheet %r truncated at %d rows", ws.title, MAX_ROWS_PER_SHEET)
|
||||
break
|
||||
data_rows.append(row)
|
||||
|
||||
# Infer field types and build records
|
||||
col_count = len(clean_headers)
|
||||
field_types = _infer_column_types(data_rows, col_count)
|
||||
|
||||
records: list[dict[str, Any]] = []
|
||||
for row in data_rows:
|
||||
rec: dict[str, Any] = {}
|
||||
for i, col_name in enumerate(clean_headers):
|
||||
val = row[i] if i < len(row) else None
|
||||
if val is not None:
|
||||
val = _coerce_value(val, field_types[i])
|
||||
rec[col_name] = val
|
||||
records.append(rec)
|
||||
|
||||
return ParsedSheet(
|
||||
name=ws.title,
|
||||
columns=clean_headers,
|
||||
field_types=field_types,
|
||||
records=records,
|
||||
)
|
||||
|
||||
|
||||
def _infer_column_types(rows: list[tuple], col_count: int) -> list[str]:
|
||||
"""Infer bitable field type per column: 'number', 'date', or 'text'."""
|
||||
from datetime import date, datetime
|
||||
|
||||
types: list[str] = []
|
||||
for col_idx in range(col_count):
|
||||
is_number = True
|
||||
is_date = True
|
||||
has_value = False
|
||||
for row in rows:
|
||||
if col_idx >= len(row):
|
||||
continue
|
||||
val = row[col_idx]
|
||||
if val is None or val == "":
|
||||
continue
|
||||
has_value = True
|
||||
if isinstance(val, bool):
|
||||
is_number = False
|
||||
is_date = False
|
||||
break
|
||||
if not isinstance(val, (int, float)):
|
||||
is_number = False
|
||||
if not isinstance(val, (datetime, date)):
|
||||
is_date = False
|
||||
if not is_number and not is_date:
|
||||
break
|
||||
if not has_value:
|
||||
types.append("text")
|
||||
elif is_number:
|
||||
types.append("number")
|
||||
elif is_date:
|
||||
types.append("date")
|
||||
else:
|
||||
types.append("text")
|
||||
return types
|
||||
|
||||
|
||||
def _coerce_value(val: Any, field_type: str) -> Any:
|
||||
"""Coerce a cell value to the inferred field type. Truncate long strings."""
|
||||
if field_type == "date":
|
||||
from datetime import datetime
|
||||
|
||||
if isinstance(val, datetime):
|
||||
return val.isoformat()
|
||||
if isinstance(val, str) and len(val) > MAX_CELL_CHARS:
|
||||
return val[:MAX_CELL_CHARS]
|
||||
return val
|
||||
|
|
@ -0,0 +1,132 @@
|
|||
"""Pydantic v2 data models for the bitable subsystem.
|
||||
|
||||
All models use ``model_config = ConfigDict(...)`` per project convention.
|
||||
Field configs are stored as JSONB in PostgreSQL; the Pydantic models here
|
||||
define the application-layer shape.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from enum import Enum
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field as PydanticField
|
||||
|
||||
|
||||
def _utcnow() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
class FieldType(str, Enum):
|
||||
"""Supported field types in bitable tables."""
|
||||
|
||||
text = "text"
|
||||
number = "number"
|
||||
date = "date"
|
||||
select = "select"
|
||||
multiselect = "multiselect"
|
||||
attachment = "attachment"
|
||||
image = "image"
|
||||
formula = "formula"
|
||||
lookup = "lookup"
|
||||
|
||||
|
||||
class FieldOwner(str, Enum):
|
||||
"""Who owns a field's data — determines upsert merge behavior."""
|
||||
|
||||
agent = "agent"
|
||||
user = "user"
|
||||
|
||||
|
||||
class ViewType(str, Enum):
|
||||
"""Supported view types (v1: grid only)."""
|
||||
|
||||
grid = "grid"
|
||||
kanban = "kanban"
|
||||
gantt = "gantt"
|
||||
gallery = "gallery"
|
||||
form = "form"
|
||||
|
||||
|
||||
class RecalcStatus(str, Enum):
|
||||
"""Status of an asynchronous formula recalculation task."""
|
||||
|
||||
pending = "pending"
|
||||
calculating = "calculating"
|
||||
done = "done"
|
||||
error = "error"
|
||||
|
||||
|
||||
class Table(BaseModel):
|
||||
"""A bitable table — collection of fields and records."""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: str
|
||||
name: str
|
||||
description: str = ""
|
||||
primary_key_field_id: str | None = None
|
||||
owner_user_id: str | None = None
|
||||
created_at: datetime = PydanticField(default_factory=_utcnow)
|
||||
updated_at: datetime = PydanticField(default_factory=_utcnow)
|
||||
|
||||
|
||||
class Field(BaseModel):
|
||||
"""A column definition in a bitable table.
|
||||
|
||||
``config`` varies by ``field_type``:
|
||||
- select/multiselect: ``{"options": [{"label": "...", "value": "..."}]}``
|
||||
- formula: ``{"formula_expr": "=SUM({field_abc})"}``
|
||||
- lookup: ``{"lookup_target": {"table_id": "...", "field_id": "...", "filter_field_id": "...", "filter_value": "..."}}``
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: str
|
||||
table_id: str
|
||||
name: str
|
||||
field_type: FieldType
|
||||
config: dict[str, Any] = PydanticField(default_factory=dict)
|
||||
owner: FieldOwner = FieldOwner.user
|
||||
created_at: datetime = PydanticField(default_factory=_utcnow)
|
||||
|
||||
|
||||
class Record(BaseModel):
|
||||
"""A row in a bitable table. ``values`` maps field_id → value."""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: str
|
||||
table_id: str
|
||||
values: dict[str, Any] = PydanticField(default_factory=dict)
|
||||
created_at: datetime = PydanticField(default_factory=_utcnow)
|
||||
updated_at: datetime = PydanticField(default_factory=_utcnow)
|
||||
|
||||
|
||||
class View(BaseModel):
|
||||
"""A saved view of a table (filters, sorts, hidden fields)."""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: str
|
||||
table_id: str
|
||||
name: str
|
||||
view_type: ViewType = ViewType.grid
|
||||
config: dict[str, Any] = PydanticField(default_factory=dict)
|
||||
created_at: datetime = PydanticField(default_factory=_utcnow)
|
||||
|
||||
|
||||
class RecalcTask(BaseModel):
|
||||
"""An asynchronous formula recalculation task."""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: str
|
||||
table_id: str
|
||||
record_id: str
|
||||
field_id: str
|
||||
status: RecalcStatus = RecalcStatus.pending
|
||||
error_message: str | None = None
|
||||
queued_at: datetime = PydanticField(default_factory=_utcnow)
|
||||
completed_at: datetime | None = None
|
||||
|
|
@ -0,0 +1,266 @@
|
|||
"""Async recalc worker for formula fields.
|
||||
|
||||
Consumes recalc tasks from the queue, evaluates formulas, and writes results
|
||||
back to records. Supports crash recovery (resets stale ``calculating`` tasks
|
||||
on startup) and graceful shutdown.
|
||||
|
||||
Lifecycle (managed by app.py lifespan):
|
||||
worker = RecalcWorker(db, service)
|
||||
await worker.start() # starts background task + crash recovery
|
||||
...
|
||||
await worker.stop() # waits for shutdown
|
||||
|
||||
The worker runs as an asyncio task, polling the queue every ``poll_interval``
|
||||
seconds. Each task is processed in its own transaction.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from agentkit.bitable.db import BitableDB
|
||||
from agentkit.bitable.formula.engine import FormulaEngine
|
||||
from agentkit.bitable.models import FieldType, RecalcStatus
|
||||
from agentkit.bitable.repository import BitableRepository
|
||||
from agentkit.bitable.service import BitableService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_DEFAULT_POLL_INTERVAL = 0.5 # seconds between queue polls
|
||||
_DEFAULT_REAPER_INTERVAL = 300 # 5 minutes
|
||||
_DEFAULT_STALE_THRESHOLD = 600 # 10 minutes
|
||||
|
||||
|
||||
class RecalcWorker:
|
||||
"""Background worker that processes formula recalc tasks.
|
||||
|
||||
Usage::
|
||||
|
||||
worker = RecalcWorker(db, service)
|
||||
await worker.start()
|
||||
# ... worker runs in background ...
|
||||
await worker.stop()
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
db: BitableDB,
|
||||
service: BitableService,
|
||||
poll_interval: float = _DEFAULT_POLL_INTERVAL,
|
||||
reaper_interval: float = _DEFAULT_REAPER_INTERVAL,
|
||||
stale_threshold: float = _DEFAULT_STALE_THRESHOLD,
|
||||
) -> None:
|
||||
self._db = db
|
||||
self._service = service
|
||||
self._repo = BitableRepository(db)
|
||||
self._poll_interval = poll_interval
|
||||
self._reaper_interval = reaper_interval
|
||||
self._stale_threshold = stale_threshold
|
||||
self._task: asyncio.Task[None] | None = None
|
||||
self._reaper_task: asyncio.Task[None] | None = None
|
||||
self._stop_event = asyncio.Event()
|
||||
# Per-table formula engines (cached, rebuilt when fields change)
|
||||
self._engines: dict[str, FormulaEngine] = {}
|
||||
|
||||
async def start(self) -> None:
|
||||
"""Start the worker. Performs crash recovery first."""
|
||||
# Crash recovery: reset stale 'calculating' tasks to 'pending'.
|
||||
# On startup, all calculating tasks are stale (worker was down).
|
||||
# Use threshold=0 to reset all calculating tasks immediately.
|
||||
reset_count = await self._repo.reset_stale_recalc_tasks(stale_threshold=0.0)
|
||||
if reset_count > 0:
|
||||
logger.info("RecalcWorker: reset %d stale tasks to pending", reset_count)
|
||||
|
||||
self._stop_event.clear()
|
||||
self._task = asyncio.create_task(self._run(), name="recalc-worker")
|
||||
self._reaper_task = asyncio.create_task(self._run_reaper(), name="recalc-reaper")
|
||||
logger.info("RecalcWorker started")
|
||||
|
||||
async def stop(self) -> None:
|
||||
"""Gracefully stop the worker."""
|
||||
self._stop_event.set()
|
||||
if self._task is not None:
|
||||
self._task.cancel()
|
||||
try:
|
||||
await self._task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
self._task = None
|
||||
if self._reaper_task is not None:
|
||||
self._reaper_task.cancel()
|
||||
try:
|
||||
await self._reaper_task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
self._reaper_task = None
|
||||
logger.info("RecalcWorker stopped")
|
||||
|
||||
async def _run(self) -> None:
|
||||
"""Main worker loop — poll queue, process tasks."""
|
||||
while not self._stop_event.is_set():
|
||||
try:
|
||||
# Atomic claim (P1 #6): tasks are marked 'calculating' in the
|
||||
# same transaction, so concurrent workers never grab the same task.
|
||||
tasks = await self._repo.claim_recalc_tasks(limit=10)
|
||||
if not tasks:
|
||||
await asyncio.sleep(self._poll_interval)
|
||||
continue
|
||||
|
||||
# P1 #7: sort tasks by topological order so formula-to-formula
|
||||
# dependencies resolve correctly within a batch. Tasks for the
|
||||
# same record are ordered so that if B depends on A, A is
|
||||
# processed first (A's result is written back before B reads it).
|
||||
tasks = await self._sort_by_topological_order(tasks)
|
||||
|
||||
for task in tasks:
|
||||
if self._stop_event.is_set():
|
||||
break
|
||||
await self.process_task(task)
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
except Exception:
|
||||
logger.exception("RecalcWorker error in main loop")
|
||||
await asyncio.sleep(self._poll_interval)
|
||||
|
||||
async def _sort_by_topological_order(self, tasks: list[Any]) -> list[Any]:
|
||||
"""Sort claimed tasks so dependencies are processed first (P1 #7).
|
||||
|
||||
Groups tasks by table_id, builds (or reuses) the engine to get the
|
||||
topological order, and assigns each task a sort key of
|
||||
``(table_id, record_id, topo_index)``. Tasks for fields not in the
|
||||
DAG get topo_index = infinity (processed last).
|
||||
"""
|
||||
if len(tasks) <= 1:
|
||||
return tasks
|
||||
|
||||
# Build topo index per table: {table_id: {field_id: position}}
|
||||
topo_index: dict[str, dict[str, int]] = {}
|
||||
table_ids = {t.table_id for t in tasks}
|
||||
for tid in table_ids:
|
||||
engine = await self._get_or_build_engine(tid)
|
||||
if engine is None:
|
||||
topo_index[tid] = {}
|
||||
continue
|
||||
order = engine.topological_order()
|
||||
topo_index[tid] = {fid: i for i, fid in enumerate(order)}
|
||||
|
||||
def _key(t: Any) -> tuple[str, str, int]:
|
||||
idx = topo_index.get(t.table_id, {}).get(t.field_id, 1 << 30)
|
||||
return (t.table_id, t.record_id, idx)
|
||||
|
||||
return sorted(tasks, key=_key)
|
||||
|
||||
async def _run_reaper(self) -> None:
|
||||
"""Reaper loop — reset stale calculating tasks periodically.
|
||||
|
||||
Only resets tasks older than ``stale_threshold`` (P1 #10), so active
|
||||
tasks being processed by a live worker are not interrupted.
|
||||
"""
|
||||
while not self._stop_event.is_set():
|
||||
try:
|
||||
await asyncio.sleep(self._reaper_interval)
|
||||
count = await self._repo.reset_stale_recalc_tasks(
|
||||
stale_threshold=self._stale_threshold
|
||||
)
|
||||
if count > 0:
|
||||
logger.info(
|
||||
"RecalcWorker reaper: reset %d stale tasks (threshold=%ss)",
|
||||
count,
|
||||
self._stale_threshold,
|
||||
)
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
except Exception:
|
||||
logger.exception("RecalcWorker reaper error")
|
||||
|
||||
async def process_task(self, task: Any) -> None:
|
||||
"""Process a single recalc task: evaluate formula → write result.
|
||||
|
||||
The task is expected to already be in ``calculating`` status when
|
||||
called from the worker loop (atomic claim sets it). When called
|
||||
synchronously via ``service.process_recalc_task``, this method
|
||||
marks it calculating first (idempotent — re-marking is harmless).
|
||||
"""
|
||||
# Idempotent: mark calculating (no-op if already calculating via claim).
|
||||
await self._repo.update_recalc_status(task.id, RecalcStatus.calculating)
|
||||
|
||||
try:
|
||||
field = await self._repo.get_field(task.field_id)
|
||||
if field is None or field.field_type != FieldType.formula:
|
||||
await self._repo.update_recalc_status(
|
||||
task.id, RecalcStatus.error, "Field not found or not a formula"
|
||||
)
|
||||
return
|
||||
|
||||
formula_expr = field.config.get("formula_expr", "")
|
||||
if not formula_expr:
|
||||
await self._repo.update_recalc_status(
|
||||
task.id, RecalcStatus.error, "No formula_expr in field config"
|
||||
)
|
||||
return
|
||||
|
||||
engine = await self._get_or_build_engine(task.table_id)
|
||||
if engine is None:
|
||||
await self._repo.update_recalc_status(
|
||||
task.id, RecalcStatus.error, "No formula fields in table"
|
||||
)
|
||||
return
|
||||
|
||||
record = await self._repo.get_record(task.record_id)
|
||||
if record is None:
|
||||
await self._repo.update_recalc_status(
|
||||
task.id, RecalcStatus.error, "Record not found"
|
||||
)
|
||||
return
|
||||
|
||||
deps = engine.get_dependencies(task.field_id)
|
||||
column_values: dict[str, list[Any]] = {}
|
||||
for dep_field_id in deps:
|
||||
column_values[dep_field_id] = await self._repo.get_column_values(
|
||||
task.table_id, dep_field_id
|
||||
)
|
||||
|
||||
result = engine.evaluate(
|
||||
task.field_id,
|
||||
row_values=record.values,
|
||||
column_values=column_values,
|
||||
)
|
||||
|
||||
await self._repo.set_formula_value(task.record_id, task.field_id, result)
|
||||
await self._repo.update_recalc_status(task.id, RecalcStatus.done)
|
||||
|
||||
except Exception as e:
|
||||
logger.exception("RecalcWorker: error processing task %s", task.id)
|
||||
await self._repo.update_recalc_status(task.id, RecalcStatus.error, str(e)[:500])
|
||||
|
||||
async def _get_or_build_engine(self, table_id: str) -> FormulaEngine | None:
|
||||
"""Get or build a FormulaEngine for a table.
|
||||
|
||||
Returns None if the table has no formula fields.
|
||||
"""
|
||||
if table_id in self._engines:
|
||||
return self._engines[table_id]
|
||||
|
||||
fields = await self._repo.list_fields(table_id)
|
||||
formula_fields = [f for f in fields if f.field_type == FieldType.formula]
|
||||
|
||||
if not formula_fields:
|
||||
return None
|
||||
|
||||
engine = FormulaEngine()
|
||||
for f in formula_fields:
|
||||
formula_expr = f.config.get("formula_expr", "")
|
||||
if formula_expr:
|
||||
try:
|
||||
engine.add_formula(f.id, formula_expr)
|
||||
except Exception:
|
||||
logger.exception("RecalcWorker: failed to register formula for field %s", f.id)
|
||||
|
||||
self._engines[table_id] = engine
|
||||
return engine
|
||||
|
||||
def invalidate_engine(self, table_id: str) -> None:
|
||||
"""Invalidate the cached formula engine for a table (call when fields change)."""
|
||||
self._engines.pop(table_id, None)
|
||||
|
|
@ -0,0 +1,803 @@
|
|||
"""Data access layer for the bitable subsystem.
|
||||
|
||||
All PostgreSQL operations go through this repository. The service layer
|
||||
(:mod:`agentkit.bitable.service`) calls these methods — routes never
|
||||
access the repository directly.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import delete, func, insert, select, text, update
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
|
||||
from agentkit.bitable.db import (
|
||||
BitableDB,
|
||||
FieldModel,
|
||||
RecordModel,
|
||||
RecalcQueueModel,
|
||||
TableModel,
|
||||
ViewModel,
|
||||
_uuid_str,
|
||||
)
|
||||
from agentkit.bitable.models import (
|
||||
Field,
|
||||
FieldOwner,
|
||||
FieldType,
|
||||
Record,
|
||||
RecalcStatus,
|
||||
RecalcTask,
|
||||
Table,
|
||||
View,
|
||||
ViewType,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class BitableRepository:
|
||||
"""Async repository for bitable CRUD operations.
|
||||
|
||||
Usage::
|
||||
|
||||
repo = BitableRepository(db)
|
||||
table = await repo.create_table(name="My Table")
|
||||
"""
|
||||
|
||||
def __init__(self, db: BitableDB) -> None:
|
||||
self._db = db
|
||||
|
||||
@property
|
||||
def _session_factory(self):
|
||||
return self._db.session_factory
|
||||
|
||||
# ── Tables ──────────────────────────────────────────────
|
||||
|
||||
async def create_table(
|
||||
self,
|
||||
name: str,
|
||||
description: str = "",
|
||||
primary_key_field_id: str | None = None,
|
||||
owner_user_id: str | None = None,
|
||||
) -> Table:
|
||||
"""Create a new bitable table."""
|
||||
async with self._session_factory() as session:
|
||||
stmt = (
|
||||
insert(TableModel)
|
||||
.values(
|
||||
id=_uuid_str(),
|
||||
name=name,
|
||||
description=description,
|
||||
primary_key_field_id=primary_key_field_id,
|
||||
owner_user_id=owner_user_id,
|
||||
)
|
||||
.returning(TableModel)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
entity = result.scalar_one()
|
||||
await session.commit()
|
||||
return Table.model_validate(entity)
|
||||
|
||||
async def get_table(self, table_id: str) -> Table | None:
|
||||
"""Get a table by ID."""
|
||||
async with self._session_factory() as session:
|
||||
stmt = select(TableModel).where(TableModel.id == table_id)
|
||||
result = await session.execute(stmt)
|
||||
entity = result.scalars().first()
|
||||
return Table.model_validate(entity) if entity else None
|
||||
|
||||
async def list_tables(self, owner_user_id: str | None = None) -> list[Table]:
|
||||
"""List all tables (optionally filtered by owner)."""
|
||||
async with self._session_factory() as session:
|
||||
stmt = select(TableModel)
|
||||
if owner_user_id:
|
||||
stmt = stmt.where(TableModel.owner_user_id == owner_user_id)
|
||||
stmt = stmt.order_by(TableModel.created_at.desc())
|
||||
result = await session.execute(stmt)
|
||||
return [Table.model_validate(e) for e in result.scalars().all()]
|
||||
|
||||
async def update_table(self, table_id: str, **kwargs: Any) -> Table | None:
|
||||
"""Update a table's attributes."""
|
||||
async with self._session_factory() as session:
|
||||
stmt = (
|
||||
update(TableModel)
|
||||
.where(TableModel.id == table_id)
|
||||
.values(**kwargs)
|
||||
.returning(TableModel)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
entity = result.scalars().first()
|
||||
await session.commit()
|
||||
return Table.model_validate(entity) if entity else None
|
||||
|
||||
async def create_pk_unique_index(self, table_id: str, pk_field_id: str) -> None:
|
||||
"""Create a unique expression index on the PK field for this table.
|
||||
|
||||
Enforces that ``values->>pk_field_id`` is unique per table. Required
|
||||
for correct upsert semantics (KTD8) — without it, duplicate PK values
|
||||
can be silently inserted and ``find_record_by_pk`` does full scans.
|
||||
"""
|
||||
# ponytail: pk_field_id is a system UUID from TableModel, validated here
|
||||
# before interpolation into a SQL identifier.
|
||||
if not re.match(r"^[a-f0-9-]{36}$", pk_field_id):
|
||||
raise ValueError(f"Invalid pk_field_id format: {pk_field_id}")
|
||||
index_name = f"ix_bitable_records_pk_{table_id.replace('-', '_')}"
|
||||
sql = text(
|
||||
f"CREATE UNIQUE INDEX IF NOT EXISTS {index_name} "
|
||||
f"ON bitable.bitable_records (table_id, (values->>'{pk_field_id}')) "
|
||||
f"WHERE values ? :pk_path"
|
||||
)
|
||||
async with self._session_factory() as session:
|
||||
await session.execute(sql, {"pk_path": pk_field_id})
|
||||
await session.commit()
|
||||
|
||||
async def delete_table(self, table_id: str) -> bool:
|
||||
"""Delete a table and all its fields, records, views, and recalc tasks."""
|
||||
async with self._session_factory() as session:
|
||||
await session.execute(delete(FieldModel).where(FieldModel.table_id == table_id))
|
||||
await session.execute(delete(RecordModel).where(RecordModel.table_id == table_id))
|
||||
await session.execute(delete(ViewModel).where(ViewModel.table_id == table_id))
|
||||
await session.execute(
|
||||
delete(RecalcQueueModel).where(RecalcQueueModel.table_id == table_id)
|
||||
)
|
||||
result = await session.execute(delete(TableModel).where(TableModel.id == table_id))
|
||||
await session.commit()
|
||||
return result.rowcount > 0
|
||||
|
||||
# ── Fields ──────────────────────────────────────────────
|
||||
|
||||
async def create_field(
|
||||
self,
|
||||
table_id: str,
|
||||
name: str,
|
||||
field_type: FieldType,
|
||||
config: dict[str, Any] | None = None,
|
||||
owner: FieldOwner = FieldOwner.user,
|
||||
) -> Field:
|
||||
"""Create a new field in a table."""
|
||||
async with self._session_factory() as session:
|
||||
stmt = (
|
||||
insert(FieldModel)
|
||||
.values(
|
||||
id=_uuid_str(),
|
||||
table_id=table_id,
|
||||
name=name,
|
||||
field_type=field_type.value,
|
||||
config=config or {},
|
||||
owner=owner.value,
|
||||
)
|
||||
.returning(FieldModel)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
entity = result.scalar_one()
|
||||
await session.commit()
|
||||
return Field.model_validate(entity)
|
||||
|
||||
async def get_field(self, field_id: str) -> Field | None:
|
||||
"""Get a field by ID."""
|
||||
async with self._session_factory() as session:
|
||||
stmt = select(FieldModel).where(FieldModel.id == field_id)
|
||||
result = await session.execute(stmt)
|
||||
entity = result.scalars().first()
|
||||
return Field.model_validate(entity) if entity else None
|
||||
|
||||
async def list_fields(self, table_id: str) -> list[Field]:
|
||||
"""List all fields in a table."""
|
||||
async with self._session_factory() as session:
|
||||
stmt = (
|
||||
select(FieldModel)
|
||||
.where(FieldModel.table_id == table_id)
|
||||
.order_by(FieldModel.created_at)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
return [Field.model_validate(e) for e in result.scalars().all()]
|
||||
|
||||
async def update_field(self, field_id: str, **kwargs: Any) -> Field | None:
|
||||
"""Update a field's attributes."""
|
||||
async with self._session_factory() as session:
|
||||
stmt = (
|
||||
update(FieldModel)
|
||||
.where(FieldModel.id == field_id)
|
||||
.values(**kwargs)
|
||||
.returning(FieldModel)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
entity = result.scalars().first()
|
||||
await session.commit()
|
||||
return Field.model_validate(entity) if entity else None
|
||||
|
||||
async def delete_field(self, field_id: str) -> bool:
|
||||
"""Delete a field."""
|
||||
async with self._session_factory() as session:
|
||||
result = await session.execute(delete(FieldModel).where(FieldModel.id == field_id))
|
||||
await session.commit()
|
||||
return result.rowcount > 0
|
||||
|
||||
# ── Records ─────────────────────────────────────────────
|
||||
|
||||
async def create_record(self, table_id: str, values: dict[str, Any] | None = None) -> Record:
|
||||
"""Create a new record."""
|
||||
async with self._session_factory() as session:
|
||||
stmt = (
|
||||
insert(RecordModel)
|
||||
.values(
|
||||
id=_uuid_str(),
|
||||
table_id=table_id,
|
||||
values=values or {},
|
||||
)
|
||||
.returning(RecordModel)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
entity = result.scalar_one()
|
||||
await session.commit()
|
||||
return Record.model_validate(entity)
|
||||
|
||||
async def create_records_batch(
|
||||
self, table_id: str, records_values: list[dict[str, Any]]
|
||||
) -> list[Record]:
|
||||
"""Batch-insert multiple records (P2 #19: eliminates per-record INSERT).
|
||||
|
||||
Returns the created records in insertion order.
|
||||
"""
|
||||
if not records_values:
|
||||
return []
|
||||
async with self._session_factory() as session:
|
||||
rows = [
|
||||
{"id": _uuid_str(), "table_id": table_id, "values": vals} for vals in records_values
|
||||
]
|
||||
stmt = insert(RecordModel).values(rows).returning(RecordModel)
|
||||
result = await session.execute(stmt)
|
||||
entities = result.scalars().all()
|
||||
await session.commit()
|
||||
return [Record.model_validate(e) for e in entities]
|
||||
|
||||
async def get_record(self, record_id: str) -> Record | None:
|
||||
"""Get a record by ID."""
|
||||
async with self._session_factory() as session:
|
||||
stmt = select(RecordModel).where(RecordModel.id == record_id)
|
||||
result = await session.execute(stmt)
|
||||
entity = result.scalars().first()
|
||||
return Record.model_validate(entity) if entity else None
|
||||
|
||||
async def list_records(
|
||||
self,
|
||||
table_id: str,
|
||||
cursor: str | None = None,
|
||||
limit: int = 50,
|
||||
) -> tuple[list[Record], str | None]:
|
||||
"""List records in a table with cursor-based pagination.
|
||||
|
||||
Returns ``(records, next_cursor)``. ``next_cursor`` is ``None`` when
|
||||
there are no more records.
|
||||
"""
|
||||
async with self._session_factory() as session:
|
||||
stmt = (
|
||||
select(RecordModel)
|
||||
.where(RecordModel.table_id == table_id)
|
||||
.order_by(RecordModel.id)
|
||||
.limit(limit + 1) # fetch one extra to check if there's a next page
|
||||
)
|
||||
if cursor:
|
||||
stmt = stmt.where(RecordModel.id > cursor)
|
||||
|
||||
result = await session.execute(stmt)
|
||||
entities = result.scalars().all()
|
||||
|
||||
if len(entities) > limit:
|
||||
next_cursor = entities[limit - 1].id
|
||||
entities = entities[:limit]
|
||||
else:
|
||||
next_cursor = None
|
||||
|
||||
return [Record.model_validate(e) for e in entities], next_cursor
|
||||
|
||||
async def update_record_values(self, record_id: str, values: dict[str, Any]) -> Record | None:
|
||||
"""Update a record's values (full replace)."""
|
||||
async with self._session_factory() as session:
|
||||
stmt = (
|
||||
update(RecordModel)
|
||||
.where(RecordModel.id == record_id)
|
||||
.values(values=values)
|
||||
.returning(RecordModel)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
entity = result.scalars().first()
|
||||
await session.commit()
|
||||
return Record.model_validate(entity) if entity else None
|
||||
|
||||
async def delete_record(self, record_id: str) -> bool:
|
||||
"""Delete a record."""
|
||||
async with self._session_factory() as session:
|
||||
result = await session.execute(delete(RecordModel).where(RecordModel.id == record_id))
|
||||
await session.commit()
|
||||
return result.rowcount > 0
|
||||
|
||||
async def delete_records_by_table(self, table_id: str) -> int:
|
||||
"""Delete all records in a table. Returns count deleted."""
|
||||
async with self._session_factory() as session:
|
||||
result = await session.execute(
|
||||
delete(RecordModel).where(RecordModel.table_id == table_id)
|
||||
)
|
||||
await session.commit()
|
||||
return result.rowcount
|
||||
|
||||
# ── Views ───────────────────────────────────────────────
|
||||
|
||||
async def create_view(
|
||||
self,
|
||||
table_id: str,
|
||||
name: str,
|
||||
view_type: ViewType = ViewType.grid,
|
||||
config: dict[str, Any] | None = None,
|
||||
) -> View:
|
||||
"""Create a new view."""
|
||||
async with self._session_factory() as session:
|
||||
stmt = (
|
||||
insert(ViewModel)
|
||||
.values(
|
||||
id=_uuid_str(),
|
||||
table_id=table_id,
|
||||
name=name,
|
||||
view_type=view_type.value,
|
||||
config=config or {},
|
||||
)
|
||||
.returning(ViewModel)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
entity = result.scalar_one()
|
||||
await session.commit()
|
||||
return View.model_validate(entity)
|
||||
|
||||
async def list_views(self, table_id: str) -> list[View]:
|
||||
"""List all views in a table."""
|
||||
async with self._session_factory() as session:
|
||||
stmt = (
|
||||
select(ViewModel)
|
||||
.where(ViewModel.table_id == table_id)
|
||||
.order_by(ViewModel.created_at)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
return [View.model_validate(e) for e in result.scalars().all()]
|
||||
|
||||
async def update_view(self, view_id: str, **kwargs: Any) -> View | None:
|
||||
"""Update a view's attributes."""
|
||||
async with self._session_factory() as session:
|
||||
stmt = (
|
||||
update(ViewModel)
|
||||
.where(ViewModel.id == view_id)
|
||||
.values(**kwargs)
|
||||
.returning(ViewModel)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
entity = result.scalars().first()
|
||||
await session.commit()
|
||||
return View.model_validate(entity) if entity else None
|
||||
|
||||
# ── Recalc Queue ────────────────────────────────────────
|
||||
|
||||
async def enqueue_recalc(
|
||||
self, table_id: str, record_id: str, field_id: str
|
||||
) -> RecalcTask | None:
|
||||
"""Enqueue a recalc task. Returns None if duplicate (ON CONFLICT DO NOTHING)."""
|
||||
async with self._session_factory() as session:
|
||||
stmt = (
|
||||
pg_insert(RecalcQueueModel)
|
||||
.values(
|
||||
id=_uuid_str(),
|
||||
table_id=table_id,
|
||||
record_id=record_id,
|
||||
field_id=field_id,
|
||||
status=RecalcStatus.pending.value,
|
||||
)
|
||||
.on_conflict_do_nothing(constraint="uq_recalc_record_field")
|
||||
.returning(RecalcQueueModel)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
entity = result.scalars().first()
|
||||
await session.commit()
|
||||
return RecalcTask.model_validate(entity) if entity else None
|
||||
|
||||
async def get_pending_recalc_tasks(self, limit: int = 50) -> list[RecalcTask]:
|
||||
"""Get pending recalc tasks ordered by queue time.
|
||||
|
||||
Deprecated for concurrent workers — use :meth:`claim_recalc_tasks`
|
||||
for atomic claim. Kept for introspection/tests.
|
||||
"""
|
||||
async with self._session_factory() as session:
|
||||
stmt = (
|
||||
select(RecalcQueueModel)
|
||||
.where(RecalcQueueModel.status == RecalcStatus.pending.value)
|
||||
.order_by(RecalcQueueModel.queued_at)
|
||||
.limit(limit)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
return [RecalcTask.model_validate(e) for e in result.scalars().all()]
|
||||
|
||||
async def claim_recalc_tasks(self, limit: int = 10) -> list[RecalcTask]:
|
||||
"""Atomically claim up to ``limit`` pending tasks (P1 #6).
|
||||
|
||||
Uses ``FOR UPDATE SKIP LOCKED`` so multiple workers never grab the
|
||||
same task. Each claimed task is set to ``calculating`` in the same
|
||||
transaction and returned. Callers process the returned tasks and
|
||||
then call :meth:`update_recalc_status` with ``done``/``error``.
|
||||
"""
|
||||
# ponytail: PostgreSQL-specific FOR UPDATE SKIP LOCKED. Ceiling: this
|
||||
# binds the repository to PG; upgrade path is a dialect check.
|
||||
async with self._session_factory() as session:
|
||||
# Subselect pending tasks with row-level locks, skipping locked rows.
|
||||
subq = (
|
||||
select(RecalcQueueModel.id)
|
||||
.where(RecalcQueueModel.status == RecalcStatus.pending.value)
|
||||
.order_by(RecalcQueueModel.queued_at)
|
||||
.limit(limit)
|
||||
.with_for_update(skip_locked=True)
|
||||
).subquery()
|
||||
|
||||
stmt = (
|
||||
update(RecalcQueueModel)
|
||||
.where(RecalcQueueModel.id.in_(select(subq.c.id)))
|
||||
.values(status=RecalcStatus.calculating.value)
|
||||
.returning(RecalcQueueModel)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
await session.commit()
|
||||
return [RecalcTask.model_validate(e) for e in result.scalars().all()]
|
||||
|
||||
async def update_recalc_status(
|
||||
self,
|
||||
task_id: str,
|
||||
status: RecalcStatus,
|
||||
error_message: str | None = None,
|
||||
) -> None:
|
||||
"""Update a recalc task's status."""
|
||||
async with self._session_factory() as session:
|
||||
kwargs: dict[str, Any] = {"status": status.value}
|
||||
if error_message is not None:
|
||||
kwargs["error_message"] = error_message
|
||||
if status in (RecalcStatus.done, RecalcStatus.error):
|
||||
kwargs["completed_at"] = func.now()
|
||||
stmt = update(RecalcQueueModel).where(RecalcQueueModel.id == task_id).values(**kwargs)
|
||||
await session.execute(stmt)
|
||||
await session.commit()
|
||||
|
||||
async def reset_stale_recalc_tasks(self, stale_threshold: float = 600.0) -> int:
|
||||
"""Reset 'calculating' tasks older than ``stale_threshold`` seconds back to 'pending'.
|
||||
|
||||
Only tasks whose ``queued_at`` is older than ``now - stale_threshold``
|
||||
are reset — this avoids resetting tasks that a live worker is currently
|
||||
processing (crash recovery, not active-task interruption). Returns count.
|
||||
"""
|
||||
# ponytail: stale_threshold is the max age a calculating task should
|
||||
# reach. 600s default = 10min; a healthy worker finishes in milliseconds.
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(seconds=stale_threshold)
|
||||
async with self._session_factory() as session:
|
||||
stmt = (
|
||||
update(RecalcQueueModel)
|
||||
.where(RecalcQueueModel.status == RecalcStatus.calculating.value)
|
||||
.where(RecalcQueueModel.queued_at < cutoff)
|
||||
.values(status=RecalcStatus.pending.value)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
await session.commit()
|
||||
return result.rowcount
|
||||
|
||||
# ── Upsert (KTD8: jsonb_set per agent field) ───────────
|
||||
|
||||
async def find_record_by_pk(
|
||||
self, table_id: str, pk_field_id: str, pk_value: str
|
||||
) -> RecordModel | None:
|
||||
"""Find a record by primary key value (JSONB key lookup)."""
|
||||
async with self._session_factory() as session:
|
||||
# ponytail: field_id is a system-generated UUID, safe to interpolate.
|
||||
# pk_value is parameterized.
|
||||
sql = text(
|
||||
"SELECT * FROM bitable.bitable_records "
|
||||
"WHERE table_id = :table_id AND values->>:pk_path = :pk_value LIMIT 1"
|
||||
)
|
||||
result = await session.execute(
|
||||
sql, {"table_id": table_id, "pk_path": pk_field_id, "pk_value": pk_value}
|
||||
)
|
||||
row = result.fetchone()
|
||||
return RecordModel(**row._mapping) if row else None
|
||||
|
||||
async def find_records_by_pk_batch(
|
||||
self, table_id: str, pk_field_id: str, pk_values: list[str]
|
||||
) -> dict[str, RecordModel]:
|
||||
"""Batch-find records by PK values (P1 #14: eliminates N+1 queries).
|
||||
|
||||
Returns a dict mapping pk_value (str) → RecordModel.
|
||||
"""
|
||||
if not pk_values:
|
||||
return {}
|
||||
async with self._session_factory() as session:
|
||||
# ponytail: field_id is a system UUID, safe to interpolate.
|
||||
# pk_values are parameterized via ANY(:values).
|
||||
sql = text(
|
||||
"SELECT * FROM bitable.bitable_records "
|
||||
"WHERE table_id = :table_id AND values->>:pk_path = ANY(:pk_values)"
|
||||
)
|
||||
result = await session.execute(
|
||||
sql,
|
||||
{
|
||||
"table_id": table_id,
|
||||
"pk_path": pk_field_id,
|
||||
"pk_values": pk_values,
|
||||
},
|
||||
)
|
||||
rows = result.fetchall()
|
||||
result_map: dict[str, RecordModel] = {}
|
||||
for r in rows:
|
||||
values = r._mapping["values"]
|
||||
if isinstance(values, str):
|
||||
import json as _json
|
||||
|
||||
values = _json.loads(values)
|
||||
pk_val = values.get(pk_field_id) if isinstance(values, dict) else None
|
||||
if pk_val is not None:
|
||||
result_map[str(pk_val)] = RecordModel(**r._mapping)
|
||||
return result_map
|
||||
|
||||
async def upsert_record_agent_fields(
|
||||
self, record_id: str, agent_field_values: dict[str, Any]
|
||||
) -> None:
|
||||
"""Update agent-owned fields using jsonb_set (KTD8).
|
||||
|
||||
Chains jsonb_set calls so user-owned fields are never touched.
|
||||
"""
|
||||
if not agent_field_values:
|
||||
return
|
||||
import json
|
||||
|
||||
async with self._session_factory() as session:
|
||||
# Build nested jsonb_set: jsonb_set(jsonb_set(values, '{f1}', CAST(:v0 AS jsonb), true), ...)
|
||||
# ponytail: field_ids are system UUIDs, safe to interpolate into path literals.
|
||||
# Use CAST(:param AS jsonb) instead of :param::jsonb — asyncpg dialect
|
||||
# misparses the `::` as part of the param name.
|
||||
inner = "values"
|
||||
params: dict[str, Any] = {"record_id": record_id}
|
||||
for i, (field_id, value) in enumerate(agent_field_values.items()):
|
||||
param_key = f"v{i}"
|
||||
inner = f"jsonb_set({inner}, '{{{field_id}}}', CAST(:{param_key} AS jsonb), true)"
|
||||
params[param_key] = json.dumps(value)
|
||||
sql = text(f"UPDATE bitable.bitable_records SET values = {inner} WHERE id = :record_id")
|
||||
await session.execute(sql, params)
|
||||
await session.commit()
|
||||
|
||||
# ── View-filtered record listing (KTD9) ────────────────
|
||||
|
||||
async def list_records_filtered(
|
||||
self,
|
||||
table_id: str,
|
||||
filters: list[dict[str, Any]] | None = None,
|
||||
sorts: list[dict[str, Any]] | None = None,
|
||||
cursor: str | None = None,
|
||||
limit: int = 50,
|
||||
) -> tuple[list[Record], str | None]:
|
||||
"""List records with view filters/sorts + cursor pagination.
|
||||
|
||||
filters: [{"field_id": "...", "op": "eq|ne|gt|lt|contains|is_empty", "value": ...}]
|
||||
sorts: [{"field_id": "...", "direction": "asc|desc"}]
|
||||
|
||||
Cursor is a base64-encoded JSON of ``{"id": ..., "sv": [sort_val, ...]}``.
|
||||
When no sorts are given, the cursor degrades to id-only pagination.
|
||||
With sorts, a row-value comparison ``(sort_cols..., id) > (cursor_vals...)``
|
||||
ensures stable pagination across non-id orderings (P1 #11).
|
||||
"""
|
||||
import base64
|
||||
import json as _json
|
||||
|
||||
async with self._session_factory() as session:
|
||||
# Build raw SQL with JSONB filter/sort translation.
|
||||
# ponytail: field_ids in filters/sorts are system UUIDs (validated by service layer).
|
||||
where_clauses = ["table_id = :table_id"]
|
||||
params: dict[str, Any] = {"table_id": table_id}
|
||||
|
||||
if filters:
|
||||
for i, f in enumerate(filters):
|
||||
fid = f["field_id"]
|
||||
op = f["op"]
|
||||
val = f.get("value")
|
||||
param_name = f"fv{i}"
|
||||
if op == "is_empty":
|
||||
where_clauses.append(f"(values->>'{fid}') IS NULL OR values->>'{fid}' = ''")
|
||||
elif op == "eq":
|
||||
where_clauses.append(f"values->>'{fid}' = :{param_name}")
|
||||
params[param_name] = str(val)
|
||||
elif op == "ne":
|
||||
where_clauses.append(f"values->>'{fid}' != :{param_name}")
|
||||
params[param_name] = str(val)
|
||||
elif op == "contains":
|
||||
where_clauses.append(f"values->>'{fid}' LIKE :{param_name}")
|
||||
params[param_name] = f"%{val}%"
|
||||
elif op in ("gt", "lt", "gte", "lte"):
|
||||
op_map = {"gt": ">", "lt": "<", "gte": ">=", "lte": "<="}
|
||||
where_clauses.append(
|
||||
f"CAST(values->>'{fid}' AS NUMERIC) {op_map[op]} :{param_name}"
|
||||
)
|
||||
params[param_name] = val
|
||||
|
||||
# Build sort expressions and cursor (P1 #11: composite cursor).
|
||||
sort_exprs: list[str] = [] # SQL expressions for ORDER BY
|
||||
sort_dirs: list[str] = [] # "ASC" or "DESC" per sort column
|
||||
if sorts:
|
||||
for s in sorts:
|
||||
fid = s["field_id"]
|
||||
direction = "ASC" if s.get("direction", "asc") == "asc" else "DESC"
|
||||
sort_exprs.append(f"values->>'{fid}'")
|
||||
sort_dirs.append(direction)
|
||||
# Always append id as the final tiebreaker for stable ordering.
|
||||
sort_exprs.append("id")
|
||||
sort_dirs.append(
|
||||
"ASC" if not sorts or sorts[0].get("direction", "asc") == "asc" else "DESC"
|
||||
)
|
||||
|
||||
order_by = ", ".join(f"{expr} {dir_}" for expr, dir_ in zip(sort_exprs, sort_dirs))
|
||||
|
||||
# Cursor: composite row-value comparison.
|
||||
if cursor:
|
||||
try:
|
||||
decoded = _json.loads(base64.b64decode(cursor).decode("utf-8"))
|
||||
cursor_id = decoded["id"]
|
||||
cursor_sort_vals = decoded.get("sv", [])
|
||||
except (ValueError, KeyError) as e:
|
||||
raise ValueError(f"Invalid cursor: {e}") from e
|
||||
|
||||
if not cursor_sort_vals:
|
||||
# No sorts — simple id comparison.
|
||||
where_clauses.append("id > :cursor_id")
|
||||
params["cursor_id"] = cursor_id
|
||||
else:
|
||||
# Row-value comparison: (col1, col2, ..., id) > (v1, v2, ..., cursor_id)
|
||||
# For DESC columns, invert the comparison direction per-column
|
||||
# by negating the value or using < instead of >.
|
||||
# PostgreSQL row values: (a,b) > (x,y) means a>x OR (a=x AND b>y).
|
||||
# For DESC, we want a<x OR (a=x AND b<x), so we use < and the
|
||||
# ORDER BY already has DESC.
|
||||
# Build the LHS and RHS of the row comparison.
|
||||
lhs_parts: list[str] = []
|
||||
rhs_params: list[str] = []
|
||||
for i, expr in enumerate(sort_exprs):
|
||||
lhs_parts.append(expr)
|
||||
param_name = f"csv{i}"
|
||||
rhs_params.append(f":{param_name}")
|
||||
params[param_name] = cursor_sort_vals[i]
|
||||
# Last element is id.
|
||||
lhs_parts.append("id")
|
||||
params["cursor_id"] = cursor_id
|
||||
rhs_params.append(":cursor_id")
|
||||
|
||||
# Determine comparison direction: if first sort is DESC, use <.
|
||||
# Mixed-direction sorts are rare; for correctness with mixed
|
||||
# directions, we'd need per-column operators. For now, use the
|
||||
# first sort's direction (covers the common single-sort case).
|
||||
# ponytail: ceiling — mixed ASC/DESC sorts use first direction only.
|
||||
comp_op = "<" if sort_dirs[0] == "DESC" else ">"
|
||||
lhs = f"({', '.join(lhs_parts)})"
|
||||
rhs = f"({', '.join(rhs_params)})"
|
||||
where_clauses.append(f"{lhs} {comp_op} {rhs}")
|
||||
|
||||
where_sql = " AND ".join(where_clauses)
|
||||
sql = text(
|
||||
f"SELECT * FROM bitable.bitable_records WHERE {where_sql} "
|
||||
f"ORDER BY {order_by} LIMIT :limit"
|
||||
)
|
||||
params["limit"] = limit + 1
|
||||
|
||||
result = await session.execute(sql, params)
|
||||
rows = result.fetchall()
|
||||
|
||||
if len(rows) > limit:
|
||||
last_row = rows[limit - 1]
|
||||
last_mapping = last_row._mapping
|
||||
# Build composite cursor from sort values + id.
|
||||
# Sort values are extracted as text to match `values->>'fid'` expressions.
|
||||
sv: list[Any] = []
|
||||
last_values = last_mapping.get("values")
|
||||
if isinstance(last_values, str):
|
||||
# asyncpg may return JSONB as str in raw text() queries.
|
||||
try:
|
||||
last_values = _json.loads(last_values)
|
||||
except (ValueError, TypeError):
|
||||
last_values = {}
|
||||
if sorts and isinstance(last_values, dict):
|
||||
for s in sorts:
|
||||
fid = s["field_id"]
|
||||
val = last_values.get(fid)
|
||||
sv.append(str(val) if val is not None else None)
|
||||
cursor_data = {"id": last_mapping["id"], "sv": sv}
|
||||
next_cursor = base64.b64encode(_json.dumps(cursor_data).encode("utf-8")).decode(
|
||||
"utf-8"
|
||||
)
|
||||
rows = rows[:limit]
|
||||
else:
|
||||
next_cursor = None
|
||||
|
||||
return [Record.model_validate(RecordModel(**r._mapping)) for r in rows], next_cursor
|
||||
|
||||
# ── Field deletion dependency queries ──────────────────
|
||||
|
||||
async def find_formula_fields_referencing(
|
||||
self, table_id: str, target_field_id: str
|
||||
) -> list[Field]:
|
||||
"""Find formula fields that reference target_field_id in their formula_expr."""
|
||||
async with self._session_factory() as session:
|
||||
# JSONB config->>'formula_expr' LIKE '%{target_field_id}%'
|
||||
sql = text(
|
||||
"SELECT * FROM bitable.bitable_fields "
|
||||
"WHERE table_id = :table_id AND field_type = 'formula' "
|
||||
"AND config->>'formula_expr' LIKE :pattern"
|
||||
)
|
||||
result = await session.execute(
|
||||
sql,
|
||||
{"table_id": table_id, "pattern": f"%{target_field_id}%"},
|
||||
)
|
||||
return [Field.model_validate(FieldModel(**r._mapping)) for r in result.fetchall()]
|
||||
|
||||
async def find_views_referencing_field(self, table_id: str, field_id: str) -> list[View]:
|
||||
"""Find views whose config (filters/sorts/hidden_fields) references field_id."""
|
||||
async with self._session_factory() as session:
|
||||
# Check if field_id appears anywhere in config JSONB
|
||||
sql = text(
|
||||
"SELECT * FROM bitable.bitable_views "
|
||||
"WHERE table_id = :table_id AND config::text LIKE :pattern"
|
||||
)
|
||||
result = await session.execute(
|
||||
sql,
|
||||
{"table_id": table_id, "pattern": f"%{field_id}%"},
|
||||
)
|
||||
return [View.model_validate(ViewModel(**r._mapping)) for r in result.fetchall()]
|
||||
|
||||
async def remove_field_from_records(self, table_id: str, field_id: str) -> int:
|
||||
"""Remove a field key from all records' JSONB values (force delete cleanup)."""
|
||||
async with self._session_factory() as session:
|
||||
sql = text(
|
||||
"UPDATE bitable.bitable_records SET values = values - :field_path "
|
||||
"WHERE table_id = :table_id"
|
||||
)
|
||||
result = await session.execute(sql, {"field_path": field_id, "table_id": table_id})
|
||||
await session.commit()
|
||||
return result.rowcount
|
||||
|
||||
# ── Recalc support (U3) ────────────────────────────────
|
||||
|
||||
async def get_column_values(self, table_id: str, field_id: str) -> list[Any]:
|
||||
"""Get all values for a field across all records in a table (for aggregates).
|
||||
|
||||
Returns a list of values (preserving order by record id). Missing values
|
||||
are included as None so aggregate functions can skip them.
|
||||
"""
|
||||
async with self._session_factory() as session:
|
||||
sql = text(
|
||||
"SELECT values->:field_id AS val FROM bitable.bitable_records "
|
||||
"WHERE table_id = :table_id ORDER BY id"
|
||||
)
|
||||
result = await session.execute(sql, {"field_id": field_id, "table_id": table_id})
|
||||
return [row[0] for row in result.fetchall()]
|
||||
|
||||
async def set_formula_value(self, record_id: str, field_id: str, value: Any) -> None:
|
||||
"""Set a single formula field value in a record's JSONB (jsonb_set)."""
|
||||
import json
|
||||
|
||||
async with self._session_factory() as session:
|
||||
# ponytail: field_id is a system UUID, safe to interpolate into path literal.
|
||||
# Use CAST AS jsonb instead of ::jsonb (asyncpg dialect misparses ::).
|
||||
sql = text(
|
||||
f"UPDATE bitable.bitable_records "
|
||||
f"SET values = jsonb_set(values, '{{{field_id}}}', CAST(:val AS jsonb), true) "
|
||||
f"WHERE id = :record_id"
|
||||
)
|
||||
await session.execute(sql, {"val": json.dumps(value), "record_id": record_id})
|
||||
await session.commit()
|
||||
|
||||
async def get_all_records(self, table_id: str) -> list[Record]:
|
||||
"""Get all records in a table (for column value extraction in recalc)."""
|
||||
async with self._session_factory() as session:
|
||||
stmt = (
|
||||
select(RecordModel).where(RecordModel.table_id == table_id).order_by(RecordModel.id)
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
return [Record.model_validate(e) for e in result.scalars().all()]
|
||||
|
|
@ -0,0 +1,477 @@
|
|||
"""Business logic layer for the bitable subsystem.
|
||||
|
||||
The service wraps :class:`BitableRepository` and adds business rules:
|
||||
field ownership, upsert semantics, formula recalc triggering, etc.
|
||||
|
||||
Routes and CLI call this layer — never the repository directly.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from agentkit.bitable.db import BitableDB
|
||||
from agentkit.bitable.models import (
|
||||
Field,
|
||||
FieldOwner,
|
||||
FieldType,
|
||||
Record,
|
||||
RecalcTask,
|
||||
Table,
|
||||
View,
|
||||
ViewType,
|
||||
)
|
||||
from agentkit.bitable.repository import BitableRepository
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class FieldDependencyError(Exception):
|
||||
"""Raised when deleting a field that has dependencies (formula refs, PK, views)."""
|
||||
|
||||
def __init__(self, message: str, dependencies: dict[str, Any]) -> None:
|
||||
super().__init__(message)
|
||||
self.dependencies = dependencies
|
||||
|
||||
|
||||
class BitableService:
|
||||
"""Bitable business logic service.
|
||||
|
||||
Usage::
|
||||
|
||||
service = BitableService(db)
|
||||
table = await service.create_table(name="Orders")
|
||||
"""
|
||||
|
||||
def __init__(self, db: BitableDB) -> None:
|
||||
self._db = db
|
||||
self._repo = BitableRepository(db)
|
||||
self._recalc_worker: Any = None # RecalcWorker, set via set_recalc_worker
|
||||
|
||||
@property
|
||||
def repo(self) -> BitableRepository:
|
||||
return self._repo
|
||||
|
||||
def set_recalc_worker(self, worker: Any) -> None:
|
||||
"""Register the long-lived RecalcWorker so field changes can invalidate its engine cache.
|
||||
|
||||
Called after both service and worker are constructed (breaks the
|
||||
service<->worker construction cycle). When unset, field mutations
|
||||
simply skip cache invalidation — the next worker start rebuilds engines.
|
||||
"""
|
||||
self._recalc_worker = worker
|
||||
|
||||
def _invalidate_engine_cache(self, table_id: str) -> None:
|
||||
"""Invalidate the worker's cached formula engine for a table (P1 #5)."""
|
||||
if self._recalc_worker is not None:
|
||||
self._recalc_worker.invalidate_engine(table_id)
|
||||
|
||||
# ── Tables ──────────────────────────────────────────────
|
||||
|
||||
async def create_table(
|
||||
self,
|
||||
name: str,
|
||||
description: str = "",
|
||||
primary_key_field_id: str | None = None,
|
||||
owner_user_id: str | None = None,
|
||||
) -> Table:
|
||||
"""Create a new bitable table. Creates PK unique index if PK is set."""
|
||||
table = await self._repo.create_table(
|
||||
name=name,
|
||||
description=description,
|
||||
primary_key_field_id=primary_key_field_id,
|
||||
owner_user_id=owner_user_id,
|
||||
)
|
||||
if primary_key_field_id:
|
||||
await self._repo.create_pk_unique_index(table.id, primary_key_field_id)
|
||||
return table
|
||||
|
||||
async def get_table(self, table_id: str) -> Table | None:
|
||||
return await self._repo.get_table(table_id)
|
||||
|
||||
async def list_tables(self, owner_user_id: str | None = None) -> list[Table]:
|
||||
return await self._repo.list_tables(owner_user_id=owner_user_id)
|
||||
|
||||
async def update_table(self, table_id: str, **kwargs: Any) -> Table | None:
|
||||
"""Update table attrs. Creates PK unique index if primary_key_field_id is set."""
|
||||
table = await self._repo.update_table(table_id, **kwargs)
|
||||
if table and kwargs.get("primary_key_field_id"):
|
||||
await self._repo.create_pk_unique_index(table_id, kwargs["primary_key_field_id"])
|
||||
return table
|
||||
|
||||
async def delete_table(self, table_id: str) -> bool:
|
||||
return await self._repo.delete_table(table_id)
|
||||
|
||||
# ── Fields ──────────────────────────────────────────────
|
||||
|
||||
async def create_field(
|
||||
self,
|
||||
table_id: str,
|
||||
name: str,
|
||||
field_type: FieldType,
|
||||
config: dict[str, Any] | None = None,
|
||||
owner: FieldOwner = FieldOwner.user,
|
||||
) -> Field:
|
||||
"""Create a new field. U2 will add formula validation and DAG updates."""
|
||||
field = await self._repo.create_field(
|
||||
table_id=table_id,
|
||||
name=name,
|
||||
field_type=field_type,
|
||||
config=config or {},
|
||||
owner=owner,
|
||||
)
|
||||
# New formula field changes the table's DAG — invalidate cached engine (P1 #5).
|
||||
if field_type == FieldType.formula:
|
||||
self._invalidate_engine_cache(table_id)
|
||||
return field
|
||||
|
||||
async def get_field(self, field_id: str) -> Field | None:
|
||||
return await self._repo.get_field(field_id)
|
||||
|
||||
async def list_fields(self, table_id: str) -> list[Field]:
|
||||
return await self._repo.list_fields(table_id)
|
||||
|
||||
async def update_field(self, field_id: str, **kwargs: Any) -> Field | None:
|
||||
"""Update a field. U2 will add dependency checking."""
|
||||
field = await self._repo.update_field(field_id, **kwargs)
|
||||
if field is not None:
|
||||
# Any field update (name, config, formula_expr) can affect the DAG (P1 #5).
|
||||
self._invalidate_engine_cache(field.table_id)
|
||||
return field
|
||||
|
||||
async def delete_field(self, field_id: str, force: bool = False) -> bool:
|
||||
"""Delete a field with dependency checking.
|
||||
|
||||
Returns True if deleted. Raises ValueError with dependency info if
|
||||
blocked (use force=True to override with cascade cleanup).
|
||||
"""
|
||||
field = await self._repo.get_field(field_id)
|
||||
if not field:
|
||||
return False
|
||||
|
||||
# Check dependencies
|
||||
deps: dict[str, Any] = {}
|
||||
|
||||
# 1. Is it a primary key field?
|
||||
table = await self._repo.get_table(field.table_id)
|
||||
if table and table.primary_key_field_id == field_id:
|
||||
deps["is_primary_key"] = True
|
||||
|
||||
# 2. Formula fields referencing this field?
|
||||
formula_deps = await self._repo.find_formula_fields_referencing(field.table_id, field_id)
|
||||
if formula_deps:
|
||||
deps["formula_fields"] = [{"id": f.id, "name": f.name} for f in formula_deps]
|
||||
|
||||
# 3. Views referencing this field?
|
||||
view_deps = await self._repo.find_views_referencing_field(field.table_id, field_id)
|
||||
if view_deps:
|
||||
deps["views"] = [{"id": v.id, "name": v.name} for v in view_deps]
|
||||
|
||||
if deps and not force:
|
||||
raise FieldDependencyError(
|
||||
f"Cannot delete field '{field.name}': has dependencies", deps
|
||||
)
|
||||
|
||||
# Force delete: cascade cleanup
|
||||
if deps and force:
|
||||
# Mark formula fields as error
|
||||
for f in formula_deps:
|
||||
await self._repo.update_field(
|
||||
f.id,
|
||||
config={**f.config, "error": "referenced field deleted"},
|
||||
)
|
||||
# Remove field from all records' JSONB
|
||||
await self._repo.remove_field_from_records(field.table_id, field_id)
|
||||
|
||||
deleted = await self._repo.delete_field(field_id)
|
||||
if deleted:
|
||||
# Removing a field changes the table's DAG — invalidate cached engine (P1 #5).
|
||||
self._invalidate_engine_cache(field.table_id)
|
||||
return deleted
|
||||
|
||||
# ── Records ─────────────────────────────────────────────
|
||||
|
||||
async def create_record(self, table_id: str, values: dict[str, Any] | None = None) -> Record:
|
||||
"""Create a new record. Triggers recalc for affected formula fields."""
|
||||
record = await self._repo.create_record(table_id, values)
|
||||
await self._trigger_recalc_for_affected_fields(table_id, record.id)
|
||||
return record
|
||||
|
||||
async def create_records_batch(
|
||||
self, table_id: str, records_values: list[dict[str, Any]]
|
||||
) -> list[Record]:
|
||||
"""Batch-create records (P2 #19). Triggers recalc for each record.
|
||||
|
||||
Processes in chunks of 500 to keep memory bounded.
|
||||
"""
|
||||
batch_size = 500
|
||||
all_records: list[Record] = []
|
||||
for i in range(0, len(records_values), batch_size):
|
||||
chunk = records_values[i : i + batch_size]
|
||||
records = await self._repo.create_records_batch(table_id, chunk)
|
||||
for rec in records:
|
||||
await self._trigger_recalc_for_affected_fields(table_id, rec.id)
|
||||
all_records.extend(records)
|
||||
return all_records
|
||||
|
||||
async def get_record(self, record_id: str) -> Record | None:
|
||||
return await self._repo.get_record(record_id)
|
||||
|
||||
async def list_records(
|
||||
self, table_id: str, cursor: str | None = None, limit: int = 50
|
||||
) -> tuple[list[Record], str | None]:
|
||||
"""List records with cursor-based pagination (KTD9)."""
|
||||
return await self._repo.list_records(table_id, cursor=cursor, limit=limit)
|
||||
|
||||
async def list_records_filtered(
|
||||
self,
|
||||
table_id: str,
|
||||
filters: list[dict[str, Any]] | None = None,
|
||||
sorts: list[dict[str, Any]] | None = None,
|
||||
cursor: str | None = None,
|
||||
limit: int = 50,
|
||||
) -> tuple[list[Record], str | None]:
|
||||
"""List records with view filters/sorts + cursor pagination.
|
||||
|
||||
Validates every ``field_id`` in filters/sorts against the table's
|
||||
actual fields before passing to the repository — field IDs are
|
||||
interpolated into SQL path literals and must be known system UUIDs.
|
||||
"""
|
||||
if filters or sorts:
|
||||
fields = await self._repo.list_fields(table_id)
|
||||
valid_field_ids = {f.id for f in fields}
|
||||
for f in filters or []:
|
||||
fid = f.get("field_id", "")
|
||||
if fid not in valid_field_ids:
|
||||
raise ValueError(f"Unknown field_id in filter: {fid}")
|
||||
for s in sorts or []:
|
||||
fid = s.get("field_id", "")
|
||||
if fid not in valid_field_ids:
|
||||
raise ValueError(f"Unknown field_id in sort: {fid}")
|
||||
return await self._repo.list_records_filtered(
|
||||
table_id, filters=filters, sorts=sorts, cursor=cursor, limit=limit
|
||||
)
|
||||
|
||||
async def update_record_values(self, record_id: str, values: dict[str, Any]) -> Record | None:
|
||||
"""Update a record's values (full replace). Triggers recalc for affected formulas."""
|
||||
record = await self._repo.update_record_values(record_id, values)
|
||||
if record is not None:
|
||||
await self._trigger_recalc_for_affected_fields(record.table_id, record.id)
|
||||
return record
|
||||
|
||||
async def delete_record(self, record_id: str) -> bool:
|
||||
"""Delete a record and clean up any attachment/image files."""
|
||||
record = await self._repo.get_record(record_id)
|
||||
if record is None:
|
||||
return False
|
||||
await self._cleanup_attachment_files(record.table_id, [record])
|
||||
return await self._repo.delete_record(record_id)
|
||||
|
||||
async def delete_records_by_table(self, table_id: str) -> int:
|
||||
"""Delete all records in a table and clean up attachment/image files.
|
||||
|
||||
P2 #18: only fetches records when the table has attachment/image fields,
|
||||
and processes them in batches to avoid OOM on large tables.
|
||||
"""
|
||||
# Check if cleanup is needed at all before loading any records.
|
||||
fields = await self._repo.list_fields(table_id)
|
||||
has_attachments = any(
|
||||
f.field_type in (FieldType.attachment, FieldType.image) for f in fields
|
||||
)
|
||||
|
||||
if has_attachments:
|
||||
# Batch-fetch records to clean up physical files (P2 #18: avoid OOM).
|
||||
batch_size = 500
|
||||
cursor: str | None = None
|
||||
while True:
|
||||
batch, next_cursor = await self._repo.list_records(
|
||||
table_id, cursor=cursor, limit=batch_size
|
||||
)
|
||||
if batch:
|
||||
await self._cleanup_attachment_files(table_id, batch)
|
||||
if next_cursor is None:
|
||||
break
|
||||
cursor = next_cursor
|
||||
|
||||
return await self._repo.delete_records_by_table(table_id)
|
||||
|
||||
async def _cleanup_attachment_files(
|
||||
self,
|
||||
table_id: str,
|
||||
records: list[Record],
|
||||
) -> None:
|
||||
"""Delete physical files for attachment/image fields on the given records.
|
||||
|
||||
Best-effort: file deletion failures are logged but do not block
|
||||
record deletion (the file may already be gone).
|
||||
"""
|
||||
fields = await self._repo.list_fields(table_id)
|
||||
attachment_field_ids = {
|
||||
f.id for f in fields if f.field_type in (FieldType.attachment, FieldType.image)
|
||||
}
|
||||
if not attachment_field_ids:
|
||||
return
|
||||
|
||||
upload_dir = Path(os.environ.get("AGENTKIT_BITABLE_UPLOAD_DIR", "data/uploads/bitable"))
|
||||
|
||||
for record in records:
|
||||
for field_id in attachment_field_ids:
|
||||
value = record.values.get(field_id)
|
||||
if not value or not isinstance(value, list):
|
||||
continue
|
||||
for file_meta in value:
|
||||
if not isinstance(file_meta, dict):
|
||||
continue
|
||||
stored_name = file_meta.get("stored_name")
|
||||
if not stored_name:
|
||||
continue
|
||||
file_path = upload_dir / str(stored_name)
|
||||
try:
|
||||
if file_path.exists():
|
||||
file_path.unlink()
|
||||
except OSError as exc:
|
||||
# File lost or permission issue — don't block record deletion
|
||||
logger.warning("Failed to delete attachment file %s: %s", file_path, exc)
|
||||
|
||||
# ── Upsert (KTD8: jsonb_set per agent field) ───────────
|
||||
|
||||
async def upsert_records(
|
||||
self,
|
||||
table_id: str,
|
||||
records: list[dict[str, Any]],
|
||||
primary_key_field_id: str,
|
||||
) -> dict[str, Any]:
|
||||
"""Upsert records by primary key using jsonb_set (KTD8).
|
||||
|
||||
For each record:
|
||||
- If a record with the same PK value exists: update agent-owned fields
|
||||
via jsonb_set (user-owned fields are never touched).
|
||||
- If not: insert a new record.
|
||||
|
||||
Returns {"inserted": N, "updated": N, "skipped": N}.
|
||||
|
||||
P1 #14: batch-queries existing records by PK in a single SELECT,
|
||||
eliminating the N+1 query pattern.
|
||||
"""
|
||||
if not primary_key_field_id:
|
||||
raise ValueError("primary_key_field_id is required for upsert")
|
||||
|
||||
# Get agent-owned field IDs for this table
|
||||
fields = await self._repo.list_fields(table_id)
|
||||
agent_field_ids = {f.id for f in fields if f.owner == FieldOwner.agent}
|
||||
|
||||
# Partition records into insert vs update lists, collecting PK values.
|
||||
to_insert: list[dict[str, Any]] = []
|
||||
to_update: list[tuple[dict[str, Any], str]] = [] # (values, existing_record_id)
|
||||
skipped = 0
|
||||
|
||||
# Collect all non-None PK values for batch lookup.
|
||||
pk_values_by_str: dict[str, dict[str, Any]] = {}
|
||||
for rec_values in records:
|
||||
pk_value = rec_values.get(primary_key_field_id)
|
||||
if pk_value is None:
|
||||
skipped += 1
|
||||
continue
|
||||
pk_values_by_str[str(pk_value)] = rec_values
|
||||
|
||||
# Batch SELECT all existing records by PK (P1 #14: 1 query, not N).
|
||||
existing_map = await self._repo.find_records_by_pk_batch(
|
||||
table_id, primary_key_field_id, list(pk_values_by_str.keys())
|
||||
)
|
||||
|
||||
for pk_str, rec_values in pk_values_by_str.items():
|
||||
existing = existing_map.get(pk_str)
|
||||
if existing is None:
|
||||
to_insert.append(rec_values)
|
||||
else:
|
||||
to_update.append((rec_values, existing.id))
|
||||
|
||||
# Batch insert new records.
|
||||
inserted = 0
|
||||
for rec_values in to_insert:
|
||||
record = await self._repo.create_record(table_id, values=rec_values)
|
||||
await self._trigger_recalc_for_affected_fields(table_id, record.id)
|
||||
inserted += 1
|
||||
|
||||
# Update existing records (agent-owned fields only via jsonb_set).
|
||||
updated = 0
|
||||
for rec_values, existing_id in to_update:
|
||||
agent_values = {fid: val for fid, val in rec_values.items() if fid in agent_field_ids}
|
||||
if agent_values:
|
||||
await self._repo.upsert_record_agent_fields(existing_id, agent_values)
|
||||
await self._trigger_recalc_for_affected_fields(table_id, existing_id)
|
||||
updated += 1
|
||||
|
||||
return {"inserted": inserted, "updated": updated, "skipped": skipped}
|
||||
|
||||
# ── Views ───────────────────────────────────────────────
|
||||
|
||||
async def create_view(
|
||||
self,
|
||||
table_id: str,
|
||||
name: str,
|
||||
view_type: ViewType = ViewType.grid,
|
||||
config: dict[str, Any] | None = None,
|
||||
) -> View:
|
||||
return await self._repo.create_view(
|
||||
table_id=table_id,
|
||||
name=name,
|
||||
view_type=view_type,
|
||||
config=config,
|
||||
)
|
||||
|
||||
async def list_views(self, table_id: str) -> list[View]:
|
||||
return await self._repo.list_views(table_id)
|
||||
|
||||
async def update_view(self, view_id: str, **kwargs: Any) -> View | None:
|
||||
return await self._repo.update_view(view_id, **kwargs)
|
||||
|
||||
# ── Recalc (U3: formula recalc pipeline) ────────────────
|
||||
|
||||
async def _trigger_recalc_for_affected_fields(self, table_id: str, record_id: str) -> None:
|
||||
"""Detect formula fields affected by a record write and enqueue recalc.
|
||||
|
||||
Finds all formula fields in the table, checks which ones depend on
|
||||
the fields that were just written, and enqueues recalc tasks.
|
||||
For simplicity (v1), we enqueue recalc for ALL formula fields in the
|
||||
table for this record — the worker will evaluate them in topological
|
||||
order. The ON CONFLICT DO NOTHING constraint deduplicates.
|
||||
"""
|
||||
fields = await self._repo.list_fields(table_id)
|
||||
formula_fields = [f for f in fields if f.field_type == FieldType.formula]
|
||||
|
||||
for f in formula_fields:
|
||||
await self._repo.enqueue_recalc(table_id, record_id, f.id)
|
||||
|
||||
async def trigger_recalc(
|
||||
self, table_id: str, record_id: str, field_id: str
|
||||
) -> RecalcTask | None:
|
||||
"""Enqueue a formula recalc task for a specific field."""
|
||||
return await self._repo.enqueue_recalc(table_id, record_id, field_id)
|
||||
|
||||
async def get_pending_recalc_tasks(self, limit: int = 50) -> list[RecalcTask]:
|
||||
return await self._repo.get_pending_recalc_tasks(limit=limit)
|
||||
|
||||
async def reset_stale_recalc_tasks(self, stale_threshold: float = 600.0) -> int:
|
||||
"""Reset 'calculating' tasks older than threshold back to 'pending' (crash recovery)."""
|
||||
return await self._repo.reset_stale_recalc_tasks(stale_threshold=stale_threshold)
|
||||
|
||||
async def process_recalc_task(self, task: RecalcTask) -> None:
|
||||
"""Process a single recalc task (used by the worker or synchronously).
|
||||
|
||||
Marks the task as calculating, evaluates the formula, writes the result,
|
||||
and marks as done/error.
|
||||
"""
|
||||
from agentkit.bitable.recalc_worker import RecalcWorker
|
||||
|
||||
# ponytail: We reuse RecalcWorker._process_task via a temporary worker
|
||||
# instance. This is a known ceiling — the worker caches engines per table,
|
||||
# and creating a new worker each call loses the cache. For synchronous
|
||||
# processing in tests this is fine. For production, use the long-lived
|
||||
# worker started in app.py lifespan.
|
||||
worker = RecalcWorker(self._db, self)
|
||||
await worker.process_task(task)
|
||||
worker.invalidate_engine(task.table_id)
|
||||
|
|
@ -0,0 +1,251 @@
|
|||
"""Bitable CLI subcommands.
|
||||
|
||||
Usage::
|
||||
|
||||
agentkit bitable list-tables
|
||||
agentkit bitable create-table --name "Orders"
|
||||
agentkit bitable import-excel --file data.xlsx --table "Import"
|
||||
agentkit bitable query --table <id> --limit 20
|
||||
|
||||
CLI calls BitableService directly (KTD5 exception: CLI is an ops tool,
|
||||
not a runtime call path).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from typing import Optional
|
||||
|
||||
import typer
|
||||
from rich import print as rprint
|
||||
from rich.table import Table
|
||||
|
||||
bitable_app = typer.Typer(
|
||||
name="bitable",
|
||||
help="Bitable (multi-dimensional table) management commands",
|
||||
no_args_is_help=True,
|
||||
)
|
||||
|
||||
|
||||
def _check_db_url() -> str:
|
||||
"""Return DATABASE_URL or exit with a clear error."""
|
||||
db_url = os.environ.get("DATABASE_URL") or os.environ.get("AGENTKIT_DATABASE_URL")
|
||||
if not db_url:
|
||||
rprint("[red]Error: DATABASE_URL environment variable is not set.[/red]")
|
||||
rprint("[dim]Set it to your PostgreSQL connection string, e.g.:[/dim]")
|
||||
rprint("[dim] export DATABASE_URL=postgresql+asyncpg://user@localhost/db[/dim]")
|
||||
raise typer.Exit(code=1)
|
||||
return db_url
|
||||
|
||||
|
||||
async def _run_with_service(coro_factory):
|
||||
"""Initialize DB, run coro_factory(service), then close — all in one event loop."""
|
||||
from agentkit.bitable.db import BitableDB
|
||||
from agentkit.bitable.service import BitableService
|
||||
|
||||
db = BitableDB()
|
||||
try:
|
||||
await db.init()
|
||||
service = BitableService(db)
|
||||
return await coro_factory(service)
|
||||
finally:
|
||||
await db.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# list-tables
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@bitable_app.command("list-tables")
|
||||
def list_tables(
|
||||
owner: Optional[str] = typer.Option(None, "--owner", help="Filter by owner user ID"),
|
||||
):
|
||||
"""List all bitable tables."""
|
||||
_check_db_url()
|
||||
|
||||
async def _run(service):
|
||||
tables = await service.list_tables(owner_user_id=owner)
|
||||
if not tables:
|
||||
rprint("[dim]No tables found.[/dim]")
|
||||
return
|
||||
tbl = Table(title="Bitable Tables")
|
||||
tbl.add_column("ID", style="cyan", no_wrap=True)
|
||||
tbl.add_column("Name", style="white")
|
||||
tbl.add_column("Description", style="dim")
|
||||
tbl.add_column("Created", style="green")
|
||||
for t in tables:
|
||||
tbl.add_row(
|
||||
t.id,
|
||||
t.name,
|
||||
t.description or "",
|
||||
t.created_at.strftime("%Y-%m-%d %H:%M") if t.created_at else "",
|
||||
)
|
||||
rprint(tbl)
|
||||
|
||||
asyncio.run(_run_with_service(_run))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# create-table
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@bitable_app.command("create-table")
|
||||
def create_table(
|
||||
name: str = typer.Option(..., "--name", "-n", help="Table name"),
|
||||
description: str = typer.Option("", "--description", "-d", help="Table description"),
|
||||
):
|
||||
"""Create a new bitable table."""
|
||||
_check_db_url()
|
||||
|
||||
async def _run(service):
|
||||
table = await service.create_table(name=name, description=description)
|
||||
rprint(f"[green]Created table:[/green] {table.name}")
|
||||
rprint(f" ID: [cyan]{table.id}[/cyan]")
|
||||
if description:
|
||||
rprint(f" Description: {description}")
|
||||
|
||||
asyncio.run(_run_with_service(_run))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# import-excel
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@bitable_app.command("import-excel")
|
||||
def import_excel(
|
||||
file: str = typer.Option(..., "--file", "-f", help="Path to .xlsx file"),
|
||||
table_name: Optional[str] = typer.Option(
|
||||
None, "--table", "-t", help="Target table name (creates if not exists)"
|
||||
),
|
||||
sheet: Optional[str] = typer.Option(None, "--sheet", "-s", help="Sheet name (default: first)"),
|
||||
):
|
||||
"""Import an Excel file into a bitable table.
|
||||
|
||||
Creates a new table with inferred field types and inserts all rows.
|
||||
"""
|
||||
_check_db_url()
|
||||
|
||||
from agentkit.bitable.ingestion import parse_excel
|
||||
from agentkit.bitable.models import FieldType
|
||||
|
||||
try:
|
||||
sheets = parse_excel(file)
|
||||
except FileNotFoundError:
|
||||
rprint(f"[red]Error: File not found: {file}[/red]")
|
||||
raise typer.Exit(code=1)
|
||||
except Exception as exc:
|
||||
rprint(f"[red]Error reading Excel file: {exc}[/red]")
|
||||
raise typer.Exit(code=1)
|
||||
|
||||
if not sheets:
|
||||
rprint("[red]Error: No sheets found in Excel file.[/red]")
|
||||
raise typer.Exit(code=1)
|
||||
|
||||
# Select sheet
|
||||
target = sheets[0]
|
||||
if sheet:
|
||||
target = next((s for s in sheets if s.name == sheet), None)
|
||||
if target is None:
|
||||
rprint(
|
||||
f"[red]Error: Sheet '{sheet}' not found. "
|
||||
f"Available: {[s.name for s in sheets]}[/red]"
|
||||
)
|
||||
raise typer.Exit(code=1)
|
||||
|
||||
tbl_name = table_name or target.name
|
||||
|
||||
async def _run(service):
|
||||
table = await service.create_table(name=tbl_name, description=f"Imported from {file}")
|
||||
rprint(f"[green]Created table:[/green] {table.name} (ID: {table.id})")
|
||||
|
||||
# Create fields
|
||||
field_ids: list[str] = []
|
||||
for col_name, col_type in zip(target.columns, target.field_types):
|
||||
try:
|
||||
ft = FieldType(col_type)
|
||||
except ValueError:
|
||||
ft = FieldType.text
|
||||
field = await service.create_field(
|
||||
table_id=table.id,
|
||||
name=col_name,
|
||||
field_type=ft,
|
||||
config={},
|
||||
)
|
||||
field_ids.append(field.id)
|
||||
rprint(f" Field: [cyan]{col_name}[/cyan] ({ft.value})")
|
||||
|
||||
# Insert records (P2 #19: batch insert instead of per-record)
|
||||
if target.records:
|
||||
values_list: list[dict] = []
|
||||
for rec_values in target.records:
|
||||
values_dict = {}
|
||||
for idx, fid in enumerate(field_ids):
|
||||
col_name = target.columns[idx]
|
||||
if col_name in rec_values:
|
||||
values_dict[fid] = rec_values[col_name]
|
||||
values_list.append(values_dict)
|
||||
created = await service.create_records_batch(table.id, values_list)
|
||||
rprint(f"[green]Imported {len(created)} records.[/green]")
|
||||
else:
|
||||
rprint("[yellow]No records found in sheet.[/yellow]")
|
||||
|
||||
asyncio.run(_run_with_service(_run))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# query
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@bitable_app.command("query")
|
||||
def query(
|
||||
table_id: str = typer.Option(..., "--table", "-t", help="Table ID"),
|
||||
limit: int = typer.Option(20, "--limit", "-l", help="Max records to show"),
|
||||
cursor: Optional[str] = typer.Option(None, "--cursor", help="Pagination cursor"),
|
||||
):
|
||||
"""Query records from a bitable table."""
|
||||
_check_db_url()
|
||||
|
||||
async def _run(service):
|
||||
table = await service.get_table(table_id)
|
||||
if table is None:
|
||||
rprint(f"[red]Error: Table '{table_id}' not found.[/red]")
|
||||
raise typer.Exit(code=1)
|
||||
|
||||
fields = await service.list_fields(table_id)
|
||||
records, next_cursor = await service.list_records_filtered(
|
||||
table_id, cursor=cursor, limit=limit
|
||||
)
|
||||
|
||||
if not records:
|
||||
rprint("[dim]No records found.[/dim]")
|
||||
return
|
||||
|
||||
tbl = Table(title=f"{table.name} ({len(records)} records)")
|
||||
tbl.add_column("ID", style="cyan", no_wrap=True, width=12)
|
||||
for f in fields:
|
||||
tbl.add_column(f.name, style="white")
|
||||
|
||||
for rec in records:
|
||||
row = [rec.id[:8] + "..."]
|
||||
for f in fields:
|
||||
val = rec.values.get(f.id)
|
||||
if val is None:
|
||||
row.append("[dim]—[/dim]")
|
||||
elif isinstance(val, list):
|
||||
row.append(f"[{len(val)} items]")
|
||||
elif isinstance(val, dict):
|
||||
row.append("{...}")
|
||||
else:
|
||||
row.append(str(val)[:50])
|
||||
tbl.add_row(*row)
|
||||
|
||||
rprint(tbl)
|
||||
if next_cursor:
|
||||
rprint(f"[dim]Next cursor: {next_cursor}[/dim]")
|
||||
|
||||
asyncio.run(_run_with_service(_run))
|
||||
|
|
@ -23,6 +23,10 @@ from agentkit.cli.admin import admin_app # noqa: E402
|
|||
|
||||
app.add_typer(admin_app, name="admin")
|
||||
|
||||
from agentkit.cli.bitable import bitable_app # noqa: E402
|
||||
|
||||
app.add_typer(bitable_app, name="bitable")
|
||||
|
||||
from agentkit.cli.init import init # noqa: E402
|
||||
|
||||
app.command(name="init")(init)
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ Usage::
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Awaitable, Callable, Protocol, runtime_checkable
|
||||
|
|
@ -157,8 +158,9 @@ class TokenUsageMiddleware:
|
|||
return ctx
|
||||
|
||||
async def after(self, ctx: RequestContext, result: Any) -> Any:
|
||||
# 从 ReActResult 或类似结构提取 token usage
|
||||
usage = getattr(result, "token_usage", None)
|
||||
# 从 ReActResult 或类似结构提取 token 用量
|
||||
# ReActResult 有 total_tokens 属性（非 token_usage）
|
||||
usage = getattr(result, "total_tokens", None)
|
||||
if usage is not None:
|
||||
ctx.metadata["token_usage_total"] = usage
|
||||
return result
|
||||
|
|
@ -188,12 +190,20 @@ class LoopDetectionMiddleware:
|
|||
return result
|
||||
|
||||
# 检查最终 trajectory 中的重复工具调用模式（只取尾部窗口）
|
||||
# trajectory 存储 ReActStep dataclass 对象，需同时兼容 dict
|
||||
recent = trajectory[-self._window_size :] if trajectory else []
|
||||
tool_calls = [
|
||||
(step.get("tool_name", ""), step.get("arguments_hash", ""))
|
||||
for step in recent
|
||||
if isinstance(step, dict) and "tool_name" in step
|
||||
]
|
||||
tool_calls: list[tuple[str, str]] = []
|
||||
for step in recent:
|
||||
# 兼容 dataclass（ReActStep）和 dict 两种格式
|
||||
if isinstance(step, dict):
|
||||
name = step.get("tool_name", "")
|
||||
args = step.get("arguments", {})
|
||||
else:
|
||||
name = getattr(step, "tool_name", "") or ""
|
||||
args = getattr(step, "arguments", {}) or {}
|
||||
if name:
|
||||
args_str = json.dumps(args, sort_keys=True, default=str) if args else ""
|
||||
tool_calls.append((name, args_str))
|
||||
if not tool_calls:
|
||||
return result
|
||||
|
||||
|
|
|
|||
|
|
@ -259,6 +259,8 @@ class ReActEngine:
|
|||
cancellation_token: 协作式取消令牌，每次循环迭代检查是否已取消
|
||||
timeout_seconds: 超时秒数，0 表示无超时，None 使用 default_timeout
|
||||
"""
|
||||
# P2 #9: Reset loop detection state so reuse across conversations is clean
|
||||
self.reset()
|
||||
effective_compressor = compressor if compressor is not None else self._compressor
|
||||
effective_timeout = (
|
||||
timeout_seconds if timeout_seconds is not None else self._default_timeout
|
||||
|
|
@ -965,6 +967,8 @@ class ReActEngine:
|
|||
Args:
|
||||
compressor: 压缩策略，None 时使用实例默认压缩器
|
||||
"""
|
||||
# P2 #9: Reset loop detection state so reuse across conversations is clean
|
||||
self.reset()
|
||||
effective_compressor = compressor if compressor is not None else self._compressor
|
||||
tools = tools or []
|
||||
if tools:
|
||||
|
|
|
|||
|
|
@ -306,6 +306,13 @@ class TeamOrchestrator:
|
|||
{"team_id": self._team.team_id},
|
||||
)
|
||||
|
||||
# P2 #13: Clean up checkpoints after successful completion
|
||||
if self._checkpoint is not None:
|
||||
try:
|
||||
await self._checkpoint.clear(plan.id)
|
||||
except Exception as e:
|
||||
logger.warning(f"Checkpoint clear failed: {e}")
|
||||
|
||||
return {
|
||||
"status": "completed",
|
||||
"result": final_result,
|
||||
|
|
@ -363,6 +370,7 @@ class TeamOrchestrator:
|
|||
checkpoints = await self._checkpoint.list_checkpoints(plan_id)
|
||||
phase_results: dict[str, dict[str, Any]] = {}
|
||||
completed_phase_ids: set[str] = set()
|
||||
failed_phase_ids: set[str] = set()
|
||||
|
||||
for cp in checkpoints:
|
||||
if cp.phase_status == "completed":
|
||||
|
|
@ -370,6 +378,9 @@ class TeamOrchestrator:
|
|||
# Restore phase result from checkpoint
|
||||
if cp.phase_result:
|
||||
phase_results[cp.phase_id] = cp.phase_result
|
||||
elif cp.phase_status == "failed":
|
||||
# P2 #11: Restore FAILED status so they aren't re-executed
|
||||
failed_phase_ids.add(cp.phase_id)
|
||||
|
||||
# Apply checkpoint state to plan phases
|
||||
for ph in plan.phases:
|
||||
|
|
@ -377,11 +388,19 @@ class TeamOrchestrator:
|
|||
ph.status = PhaseStatus.COMPLETED
|
||||
if ph.id in phase_results and phase_results[ph.id]:
|
||||
ph.result = phase_results[ph.id]
|
||||
elif ph.id in failed_phase_ids:
|
||||
ph.status = PhaseStatus.FAILED
|
||||
# PENDING phases remain PENDING — will be executed by _run_pipeline
|
||||
|
||||
# P2 #8: Restore debate count so MAX_DEBATES limit holds after resume
|
||||
self._debate_count = sum(
|
||||
1 for ph in plan.phases if ph.phase_type == PhaseType.DEBATE
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"Resuming plan {plan_id}: {len(completed_phase_ids)} completed, "
|
||||
f"{len(plan.phases) - len(completed_phase_ids)} pending"
|
||||
f"{len(failed_phase_ids)} failed, "
|
||||
f"{len(plan.phases) - len(completed_phase_ids) - len(failed_phase_ids)} pending"
|
||||
)
|
||||
|
||||
# 4. Get lead expert
|
||||
|
|
@ -558,6 +577,9 @@ class TeamOrchestrator:
|
|||
|
||||
def _offload_result(self, content: str, ref_key: str) -> dict[str, Any]:
|
||||
"""Create an offloaded result: summary in memory, full content in workspace."""
|
||||
# P2 #14: Guard against non-string content (dict, None, etc.)
|
||||
if not isinstance(content, str):
|
||||
content = str(content) if content is not None else ""
|
||||
summary = (
|
||||
content[: self._OFFLOAD_SUMMARY_LIMIT] + "..."
|
||||
if len(content) > self._OFFLOAD_SUMMARY_LIMIT
|
||||
|
|
@ -1714,6 +1736,12 @@ class TeamOrchestrator:
|
|||
"debate_inserted": debate.id,
|
||||
},
|
||||
)
|
||||
# P1 #7: Persist dynamically inserted DEBATE phase so resume sees it
|
||||
if self._checkpoint is not None:
|
||||
try:
|
||||
await self._checkpoint.save_plan(plan)
|
||||
except Exception as e:
|
||||
logger.warning(f"Checkpoint save_plan (debate insert) failed: {e}")
|
||||
|
||||
# ── U3 end ─────────────────────────────────────────────────────────
|
||||
|
||||
|
|
@ -1816,13 +1844,18 @@ class TeamOrchestrator:
|
|||
}
|
||||
|
||||
# Build result summaries for LLM evaluation
|
||||
# P1 #5: 解析 offloaded 内容 — 从 SharedWorkspace 读取完整内容，而非使用截断摘要
|
||||
summaries = []
|
||||
for i, ph in enumerate(completed_phases):
|
||||
r = ph.result or {}
|
||||
content = r.get("content", str(r)) if isinstance(r, dict) else str(r)
|
||||
# U4: 如果结果被 offloaded，从 workspace 读取完整内容
|
||||
if isinstance(r, dict) and r.get("_offloaded"):
|
||||
content = await self._read_dependency_output(ph)
|
||||
else:
|
||||
content = r.get("content", str(r)) if isinstance(r, dict) else str(r)
|
||||
summaries.append(
|
||||
f"Phase {i + 1}: {ph.name} (by {ph.assigned_expert}, task: {ph.task_description[:100]}):\n"
|
||||
f"{content[:500]}"
|
||||
f"{content}"
|
||||
)
|
||||
|
||||
prompt = (
|
||||
|
|
|
|||
|
|
@ -74,8 +74,9 @@ class PipelineCheckpoint:
|
|||
self._redis = redis_client
|
||||
self._prefix = prefix
|
||||
self._ttl = ttl_seconds
|
||||
# 内存降级存储：plan_id → list of CheckpointData
|
||||
self._memory: dict[str, list[CheckpointData]] = {}
|
||||
# 内存降级存储：plan_id → {phase_id → CheckpointData}
|
||||
# P1 #6: 改用 dict keyed by phase_id，避免重复 append
|
||||
self._memory: dict[str, dict[str, CheckpointData]] = {}
|
||||
# 内存降级存储：plan_id → (plan_dict, saved_timestamp)
|
||||
self._memory_plans: dict[str, tuple[dict[str, Any], float]] = {}
|
||||
|
||||
|
|
@ -169,8 +170,8 @@ class PipelineCheckpoint:
|
|||
plan_status=plan_status,
|
||||
)
|
||||
|
||||
# 总是写入内存降级（保证一致性）
|
||||
self._memory.setdefault(plan_id, []).append(data)
|
||||
# P1 #6: 内存降级用 dict keyed by phase_id，覆盖重复 checkpoint
|
||||
self._memory.setdefault(plan_id, {})[phase_id] = data
|
||||
|
||||
# 尝试写入 Redis
|
||||
if self._redis is not None:
|
||||
|
|
@ -211,7 +212,8 @@ class PipelineCheckpoint:
|
|||
if not phase_ids:
|
||||
# Redis 无数据，检查内存（过滤过期）
|
||||
return [
|
||||
c for c in self._memory.get(plan_id, []) if not self._is_expired(c.saved_at)
|
||||
c for c in self._memory.get(plan_id, {}).values()
|
||||
if not self._is_expired(c.saved_at)
|
||||
]
|
||||
|
||||
# 批量 GET（pipeline 避免 N+1 往返）
|
||||
|
|
@ -233,7 +235,10 @@ class PipelineCheckpoint:
|
|||
)
|
||||
|
||||
# 内存降级（过滤过期 checkpoint）
|
||||
return [c for c in self._memory.get(plan_id, []) if not self._is_expired(c.saved_at)]
|
||||
return [
|
||||
c for c in self._memory.get(plan_id, {}).values()
|
||||
if not self._is_expired(c.saved_at)
|
||||
]
|
||||
|
||||
async def clear(self, plan_id: str) -> None:
|
||||
"""清除某 plan 的所有 checkpoint。"""
|
||||
|
|
|
|||
|
|
@ -51,6 +51,7 @@ from agentkit.server.routes import (
|
|||
documents,
|
||||
admin as admin_routes_module,
|
||||
calendar as calendar_routes,
|
||||
bitable as bitable_routes,
|
||||
)
|
||||
from agentkit.server.auth.jwt_utils import get_jwt_secret
|
||||
from agentkit.server.auth.middleware import AuthMiddleware
|
||||
|
|
@ -427,9 +428,28 @@ async def lifespan(app: FastAPI):
|
|||
except Exception:
|
||||
logger.exception("Failed to initialize calendar subsystem — calendar API unavailable")
|
||||
|
||||
# Bitable subsystem: init DB, service, internal token (KTD11).
|
||||
try:
|
||||
from agentkit.bitable.db import init_bitable_db
|
||||
from agentkit.bitable.service import BitableService
|
||||
|
||||
bitable_db = await init_bitable_db()
|
||||
app.state.bitable_service = BitableService(db=bitable_db)
|
||||
app.state.bitable_internal_token = os.environ.get("AGENTKIT_BITABLE_INTERNAL_TOKEN")
|
||||
logger.info("Bitable subsystem initialized")
|
||||
except Exception:
|
||||
logger.exception("Failed to initialize bitable subsystem")
|
||||
|
||||
yield
|
||||
|
||||
# Shutdown
|
||||
# Close bitable DB
|
||||
try:
|
||||
from agentkit.bitable.db import close_bitable_db
|
||||
|
||||
await close_bitable_db()
|
||||
except Exception:
|
||||
pass
|
||||
# Stop MCP servers
|
||||
if mcp_manager is not None:
|
||||
await mcp_manager.stop_all()
|
||||
|
|
@ -983,6 +1003,7 @@ def create_app(
|
|||
app.include_router(admin_routes_module.admin_router, prefix="/api/v1")
|
||||
app.include_router(documents.router, prefix="/api/v1")
|
||||
app.include_router(calendar_routes.router, prefix="/api/v1")
|
||||
app.include_router(bitable_routes.router, prefix="/api/v1")
|
||||
|
||||
# Serve GUI when in GUI mode
|
||||
gui_mode = os.environ.get("AGENTKIT_GUI_MODE")
|
||||
|
|
|
|||
|
|
@ -0,0 +1,327 @@
|
|||
/** Bitable API client — thin wrapper over /api/v1/bitable endpoints. */
|
||||
|
||||
import { BaseApiClient } from './base'
|
||||
|
||||
// ── Domain types (co-located with API client) ──────────────────────────
|
||||
|
||||
export type FieldType =
|
||||
| 'text'
|
||||
| 'number'
|
||||
| 'date'
|
||||
| 'select'
|
||||
| 'multiselect'
|
||||
| 'attachment'
|
||||
| 'image'
|
||||
| 'formula'
|
||||
| 'lookup'
|
||||
|
||||
export type FieldOwner = 'agent' | 'user'
|
||||
|
||||
export type ViewType = 'grid' | 'kanban' | 'gantt' | 'gallery' | 'form'
|
||||
|
||||
export type RecalcStatus = 'pending' | 'calculating' | 'done' | 'error'
|
||||
|
||||
export interface IBitableTable {
|
||||
id: string
|
||||
name: string
|
||||
description: string
|
||||
primary_key_field_id: string | null
|
||||
owner_user_id: string | null
|
||||
created_at: string
|
||||
updated_at: string
|
||||
}
|
||||
|
||||
export interface IBitableField {
|
||||
id: string
|
||||
table_id: string
|
||||
name: string
|
||||
field_type: FieldType
|
||||
config: Record<string, unknown>
|
||||
owner: FieldOwner
|
||||
created_at: string
|
||||
}
|
||||
|
||||
export interface IBitableRecord {
|
||||
id: string
|
||||
table_id: string
|
||||
values: Record<string, unknown>
|
||||
created_at: string
|
||||
updated_at: string
|
||||
}
|
||||
|
||||
export interface IBitableView {
|
||||
id: string
|
||||
table_id: string
|
||||
name: string
|
||||
view_type: ViewType
|
||||
config: Record<string, unknown>
|
||||
created_at: string
|
||||
}
|
||||
|
||||
/** File metadata stored in attachment/image field values (JSONB array). */
|
||||
export interface IAttachmentMeta {
|
||||
filename: string
|
||||
stored_name: string
|
||||
mime_type: string
|
||||
size: number
|
||||
url: string
|
||||
}
|
||||
|
||||
// ── Request types ──────────────────────────────────────────────────────
|
||||
|
||||
export interface ICreateTableRequest {
|
||||
name: string
|
||||
description?: string
|
||||
primary_key_field_id?: string | null
|
||||
}
|
||||
|
||||
export interface ICreateFieldRequest {
|
||||
name: string
|
||||
field_type: FieldType
|
||||
config?: Record<string, unknown>
|
||||
owner?: FieldOwner
|
||||
}
|
||||
|
||||
export interface ICreateRecordRequest {
|
||||
values: Record<string, unknown>
|
||||
}
|
||||
|
||||
export interface IUpdateRecordRequest {
|
||||
values: Record<string, unknown>
|
||||
}
|
||||
|
||||
export interface IUpsertRequest {
|
||||
records: Record<string, unknown>[]
|
||||
primary_key_field_id: string
|
||||
}
|
||||
|
||||
export interface ICreateViewRequest {
|
||||
name: string
|
||||
view_type?: ViewType
|
||||
config?: Record<string, unknown>
|
||||
}
|
||||
|
||||
// ── Response types ─────────────────────────────────────────────────────
|
||||
|
||||
export interface IListRecordsResponse {
|
||||
success: boolean
|
||||
records: IBitableRecord[]
|
||||
next_cursor: string | null
|
||||
}
|
||||
|
||||
// ── Runtime type guard ─────────────────────────────────────────────────
|
||||
|
||||
export function isBitableTable(value: unknown): value is IBitableTable {
|
||||
if (typeof value !== 'object' || value === null) return false
|
||||
const v = value as Record<string, unknown>
|
||||
return typeof v.id === 'string' && typeof v.name === 'string'
|
||||
}
|
||||
|
||||
// ── API client ─────────────────────────────────────────────────────────
|
||||
|
||||
const API_BASE = '/api/v1/bitable'
|
||||
|
||||
class BitableApiClient extends BaseApiClient {
|
||||
constructor(baseUrl: string = API_BASE) {
|
||||
super(baseUrl)
|
||||
}
|
||||
|
||||
// ── Tables ───────────────────────────────────────────
|
||||
|
||||
async listTables(): Promise<{ success: boolean; tables: IBitableTable[] }> {
|
||||
return this.request('/tables', { method: 'GET' })
|
||||
}
|
||||
|
||||
async createTable(
|
||||
data: ICreateTableRequest,
|
||||
): Promise<{ success: boolean; table: IBitableTable }> {
|
||||
return this.request('/tables', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify(data),
|
||||
})
|
||||
}
|
||||
|
||||
async getTable(
|
||||
tableId: string,
|
||||
): Promise<{ success: boolean; table: IBitableTable }> {
|
||||
return this.request(`/tables/${tableId}`, { method: 'GET' })
|
||||
}
|
||||
|
||||
async updateTable(
|
||||
tableId: string,
|
||||
data: Partial<ICreateTableRequest>,
|
||||
): Promise<{ success: boolean; table: IBitableTable }> {
|
||||
return this.request(`/tables/${tableId}`, {
|
||||
method: 'PATCH',
|
||||
body: JSON.stringify(data),
|
||||
})
|
||||
}
|
||||
|
||||
async deleteTable(tableId: string): Promise<{ success: boolean }> {
|
||||
return this.request(`/tables/${tableId}`, { method: 'DELETE' })
|
||||
}
|
||||
|
||||
// ── Fields ───────────────────────────────────────────
|
||||
|
||||
async listFields(
|
||||
tableId: string,
|
||||
): Promise<{ success: boolean; fields: IBitableField[] }> {
|
||||
return this.request(`/tables/${tableId}/fields`, { method: 'GET' })
|
||||
}
|
||||
|
||||
async createField(
|
||||
tableId: string,
|
||||
data: ICreateFieldRequest,
|
||||
): Promise<{ success: boolean; field: IBitableField }> {
|
||||
return this.request(`/tables/${tableId}/fields`, {
|
||||
method: 'POST',
|
||||
body: JSON.stringify(data),
|
||||
})
|
||||
}
|
||||
|
||||
async updateField(
|
||||
fieldId: string,
|
||||
data: Partial<ICreateFieldRequest>,
|
||||
): Promise<{ success: boolean; field: IBitableField }> {
|
||||
return this.request(`/fields/${fieldId}`, {
|
||||
method: 'PATCH',
|
||||
body: JSON.stringify(data),
|
||||
})
|
||||
}
|
||||
|
||||
async deleteField(
|
||||
fieldId: string,
|
||||
force = false,
|
||||
): Promise<{ success: boolean }> {
|
||||
const qs = force ? '?force=true' : ''
|
||||
return this.request(`/fields/${fieldId}${qs}`, { method: 'DELETE' })
|
||||
}
|
||||
|
||||
// ── Formula validation (U5b) ─────────────────────────
|
||||
|
||||
async validateFormula(
|
||||
formula: string,
|
||||
): Promise<{ valid: boolean; error?: string }> {
|
||||
return this.request('/fields/validate-formula', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ formula }),
|
||||
})
|
||||
}
|
||||
|
||||
// ── Records ──────────────────────────────────────────
|
||||
|
||||
async listRecords(
|
||||
tableId: string,
|
||||
params?: {
|
||||
cursor?: string
|
||||
limit?: number
|
||||
filters?: string
|
||||
sorts?: string
|
||||
},
|
||||
): Promise<IListRecordsResponse> {
|
||||
const sp = new URLSearchParams()
|
||||
if (params?.cursor) sp.set('cursor', params.cursor)
|
||||
if (params?.limit) sp.set('limit', String(params.limit))
|
||||
if (params?.filters) sp.set('filters', params.filters)
|
||||
if (params?.sorts) sp.set('sorts', params.sorts)
|
||||
const qs = sp.toString()
|
||||
const path = qs ? `/tables/${tableId}/records?${qs}` : `/tables/${tableId}/records`
|
||||
return this.request(path, { method: 'GET' })
|
||||
}
|
||||
|
||||
async createRecords(
|
||||
tableId: string,
|
||||
records: Record<string, unknown>[],
|
||||
): Promise<{ success: boolean; count: number; records: IBitableRecord[] }> {
|
||||
// P2 #20: client-side batch limit matches backend max_length=500.
|
||||
const BATCH_LIMIT = 500
|
||||
if (records.length <= BATCH_LIMIT) {
|
||||
return this.request(`/tables/${tableId}/records`, {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ records }),
|
||||
})
|
||||
}
|
||||
// Chunk large batches to avoid 422 from backend validation.
|
||||
const allRecords: IBitableRecord[] = []
|
||||
for (let i = 0; i < records.length; i += BATCH_LIMIT) {
|
||||
const chunk = records.slice(i, i + BATCH_LIMIT)
|
||||
const resp = await this.request<{ success: boolean; count: number; records: IBitableRecord[] }>(
|
||||
`/tables/${tableId}/records`,
|
||||
{ method: 'POST', body: JSON.stringify({ records: chunk }) },
|
||||
)
|
||||
if (resp.records) allRecords.push(...resp.records)
|
||||
}
|
||||
return { success: true, count: allRecords.length, records: allRecords }
|
||||
}
|
||||
|
||||
async updateRecord(
|
||||
recordId: string,
|
||||
values: Record<string, unknown>,
|
||||
): Promise<{ success: boolean; record: IBitableRecord }> {
|
||||
return this.request(`/records/${recordId}`, {
|
||||
method: 'PATCH',
|
||||
body: JSON.stringify({ values }),
|
||||
})
|
||||
}
|
||||
|
||||
async deleteRecord(recordId: string): Promise<{ success: boolean }> {
|
||||
return this.request(`/records/${recordId}`, { method: 'DELETE' })
|
||||
}
|
||||
|
||||
// ── Upsert (KTD8) ────────────────────────────────────
|
||||
|
||||
async upsertRecords(
|
||||
tableId: string,
|
||||
data: IUpsertRequest,
|
||||
): Promise<{ success: boolean; upserted: number; inserted: number; updated: number }> {
|
||||
return this.request(`/tables/${tableId}/upsert`, {
|
||||
method: 'POST',
|
||||
body: JSON.stringify(data),
|
||||
})
|
||||
}
|
||||
|
||||
// ── Views ────────────────────────────────────────────
|
||||
|
||||
async listViews(
|
||||
tableId: string,
|
||||
): Promise<{ success: boolean; views: IBitableView[] }> {
|
||||
return this.request(`/tables/${tableId}/views`, { method: 'GET' })
|
||||
}
|
||||
|
||||
async createView(
|
||||
tableId: string,
|
||||
data: ICreateViewRequest,
|
||||
): Promise<{ success: boolean; view: IBitableView }> {
|
||||
return this.request(`/tables/${tableId}/views`, {
|
||||
method: 'POST',
|
||||
body: JSON.stringify(data),
|
||||
})
|
||||
}
|
||||
|
||||
async updateView(
|
||||
viewId: string,
|
||||
data: Partial<ICreateViewRequest>,
|
||||
): Promise<{ success: boolean; view: IBitableView }> {
|
||||
return this.request(`/views/${viewId}`, {
|
||||
method: 'PATCH',
|
||||
body: JSON.stringify(data),
|
||||
})
|
||||
}
|
||||
|
||||
// ── File upload (U6: attachment & image) ──────────────
|
||||
|
||||
async uploadFile(
|
||||
tableId: string,
|
||||
fieldId: string,
|
||||
file: File,
|
||||
): Promise<IAttachmentMeta> {
|
||||
const formData = new FormData()
|
||||
formData.append('file', file)
|
||||
return this.request(
|
||||
`/tables/${tableId}/upload?field_id=${encodeURIComponent(fieldId)}`,
|
||||
{ method: 'POST', body: formData },
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
export const bitableApi = new BitableApiClient()
|
||||
|
|
@ -0,0 +1,73 @@
|
|||
<template>
|
||||
<div class="attachment-cell">
|
||||
<div
|
||||
v-for="(file, idx) in files"
|
||||
:key="idx"
|
||||
class="attachment-cell__item"
|
||||
>
|
||||
<a :href="file.url" target="_blank" class="attachment-cell__link">
|
||||
<PaperClipOutlined />
|
||||
<span class="attachment-cell__name">{{ file.filename }}</span>
|
||||
<span class="attachment-cell__size">{{ formatSize(file.size) }}</span>
|
||||
</a>
|
||||
</div>
|
||||
<span v-if="!files || files.length === 0" class="attachment-cell__empty">—</span>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { PaperClipOutlined } from '@ant-design/icons-vue'
|
||||
import type { IAttachmentMeta } from '@/api/bitable'
|
||||
|
||||
defineProps<{
|
||||
files: IAttachmentMeta[] | null | undefined
|
||||
}>()
|
||||
|
||||
function formatSize(bytes: number): string {
|
||||
if (bytes < 1024) return `${bytes} B`
|
||||
if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`
|
||||
return `${(bytes / (1024 * 1024)).toFixed(1)} MB`
|
||||
}
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.attachment-cell {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 2px;
|
||||
padding: 2px 0;
|
||||
}
|
||||
|
||||
.attachment-cell__item {
|
||||
line-height: 1.4;
|
||||
}
|
||||
|
||||
.attachment-cell__link {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
color: var(--color-primary, #1677ff);
|
||||
text-decoration: none;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.attachment-cell__link:hover {
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
.attachment-cell__name {
|
||||
max-width: 120px;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.attachment-cell__size {
|
||||
color: var(--text-secondary, #8c8c8c);
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.attachment-cell__empty {
|
||||
color: var(--text-placeholder, #bfbfbf);
|
||||
}
|
||||
</style>
|
||||
|
|
@ -0,0 +1,227 @@
|
|||
<template>
|
||||
<div class="bitable-grid-scope">
|
||||
<vxe-grid
|
||||
ref="gridRef"
|
||||
:data="rows"
|
||||
:columns="gridColumns"
|
||||
:height="height"
|
||||
:loading="loading"
|
||||
:row-config="{ keyField: '_recordId' }"
|
||||
:column-config="{ resizable: true }"
|
||||
:virtual-y-config="{ enabled: true, gt: 60 }"
|
||||
:virtual-x-config="{ enabled: true, gt: 20 }"
|
||||
:edit-config="{
|
||||
trigger: 'click',
|
||||
mode: 'cell',
|
||||
showStatus: true,
|
||||
autoClear: false,
|
||||
}"
|
||||
@edit-closed="onEditClosed"
|
||||
>
|
||||
<template #empty>
|
||||
<a-empty :description="emptyText" />
|
||||
</template>
|
||||
<!-- Custom cell renderers for attachment/image fields (U6) -->
|
||||
<template
|
||||
v-for="f in attachmentFields"
|
||||
:key="f.id"
|
||||
#[`cell_${f.id}`]="{ row }"
|
||||
>
|
||||
<AttachmentCell
|
||||
v-if="f.field_type === 'attachment'"
|
||||
:files="(row[f.id] as IAttachmentMeta[] | null | undefined)"
|
||||
/>
|
||||
<ImageCell
|
||||
v-else-if="f.field_type === 'image'"
|
||||
:images="(row[f.id] as IAttachmentMeta[] | null | undefined)"
|
||||
/>
|
||||
</template>
|
||||
</vxe-grid>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { computed, ref } from 'vue'
|
||||
import { VxeGrid } from 'vxe-table'
|
||||
import { Empty as AEmpty } from 'ant-design-vue'
|
||||
import type { VxeGridProps, VxeGridEvents } from 'vxe-table'
|
||||
import type {
|
||||
IBitableField,
|
||||
IBitableRecord,
|
||||
IAttachmentMeta,
|
||||
FieldType,
|
||||
} from '@/api/bitable'
|
||||
import AttachmentCell from './AttachmentCell.vue'
|
||||
import ImageCell from './ImageCell.vue'
|
||||
|
||||
type GridRow = Record<string, unknown> & { _rowId: string; _recordId: string }
|
||||
type GridColumn = NonNullable<VxeGridProps['columns']>[number]
|
||||
|
||||
const props = withDefaults(
|
||||
defineProps<{
|
||||
fields: IBitableField[]
|
||||
records: IBitableRecord[]
|
||||
loading?: boolean
|
||||
height?: string | number
|
||||
emptyText?: string
|
||||
}>(),
|
||||
{
|
||||
loading: false,
|
||||
height: 'auto',
|
||||
emptyText: '暂无数据',
|
||||
},
|
||||
)
|
||||
|
||||
const emit = defineEmits<{
|
||||
(e: 'edit-cell', payload: { recordId: string; fieldId: string; value: unknown }): void
|
||||
}>()
|
||||
|
||||
const gridRef = ref<InstanceType<typeof VxeGrid> | null>(null)
|
||||
|
||||
// Fields that use custom slot renderers (attachment/image)
|
||||
const attachmentFields = computed(() =>
|
||||
props.fields.filter(
|
||||
(f) => f.field_type === 'attachment' || f.field_type === 'image',
|
||||
),
|
||||
)
|
||||
|
||||
// Map records to grid rows with a stable _rowId (record.id)
|
||||
const rows = computed<GridRow[]>(() =>
|
||||
props.records.map((r) => ({
|
||||
_rowId: r.id,
|
||||
_recordId: r.id,
|
||||
...r.values,
|
||||
})),
|
||||
)
|
||||
|
||||
// Build vxe-grid columns from field definitions
|
||||
const gridColumns = computed<GridColumn[]>(() => {
|
||||
const cols: GridColumn[] = [
|
||||
{
|
||||
type: 'seq',
|
||||
width: 56,
|
||||
fixed: 'left',
|
||||
title: '#',
|
||||
},
|
||||
]
|
||||
|
||||
for (const f of props.fields) {
|
||||
cols.push(buildColumn(f))
|
||||
}
|
||||
|
||||
return cols
|
||||
})
|
||||
|
||||
function buildColumn(f: IBitableField): GridColumn {
|
||||
const isFormula = f.field_type === 'formula'
|
||||
const isAttachment = f.field_type === 'attachment' || f.field_type === 'image'
|
||||
const base: GridColumn = {
|
||||
field: f.id,
|
||||
title: f.name,
|
||||
minWidth: 120,
|
||||
width: 160,
|
||||
resizable: true,
|
||||
showOverflow: 'tooltip',
|
||||
}
|
||||
|
||||
// Formula fields are read-only
|
||||
if (isFormula) {
|
||||
return { ...base, editRender: { enabled: false } }
|
||||
}
|
||||
|
||||
// Attachment/image fields use custom slot renderers, read-only
|
||||
if (isAttachment) {
|
||||
return {
|
||||
...base,
|
||||
editRender: { enabled: false },
|
||||
slots: { default: `cell_${f.id}` },
|
||||
}
|
||||
}
|
||||
|
||||
// Editable cells by field type
|
||||
switch (f.field_type as FieldType) {
|
||||
case 'number':
|
||||
return {
|
||||
...base,
|
||||
editRender: {
|
||||
enabled: true,
|
||||
name: 'VxeNumberInput',
|
||||
},
|
||||
}
|
||||
case 'date':
|
||||
return {
|
||||
...base,
|
||||
editRender: {
|
||||
enabled: true,
|
||||
name: 'VxeDatePicker',
|
||||
},
|
||||
}
|
||||
case 'select':
|
||||
case 'multiselect':
|
||||
// ponytail: select editor uses text input for v1; options wiring is U5c
|
||||
return {
|
||||
...base,
|
||||
editRender: { enabled: true, name: 'VxeInput' },
|
||||
}
|
||||
default:
|
||||
return {
|
||||
...base,
|
||||
editRender: { enabled: true, name: 'VxeInput' },
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Handle edit-closed event — emit cell update
|
||||
const onEditClosed: VxeGridEvents.EditClosed = (params) => {
|
||||
const { row, column } = params
|
||||
const field = column.field
|
||||
if (!field) return
|
||||
const recordId = (row as GridRow)._recordId
|
||||
if (!recordId) return
|
||||
|
||||
// Only emit if value actually changed
|
||||
const newValue = (row as Record<string, unknown>)[field]
|
||||
const original = props.records.find((r) => r.id === recordId)
|
||||
if (!original) return
|
||||
const oldValue = original.values[field]
|
||||
if (newValue === oldValue) return
|
||||
|
||||
emit('edit-cell', {
|
||||
recordId,
|
||||
fieldId: field,
|
||||
value: newValue,
|
||||
})
|
||||
}
|
||||
|
||||
// Expose grid ref for parent (e.g. to refresh)
|
||||
defineExpose({
|
||||
refresh: () => gridRef.value?.refreshColumn(),
|
||||
})
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.bitable-grid-scope {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
/* KTD10: CSS isolation — all vxe-table style overrides scoped to
|
||||
.bitable-grid-scope. Use :deep() to reach vxe-table internals. */
|
||||
.bitable-grid-scope :deep(.vxe-table) {
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.bitable-grid-scope :deep(.vxe-header--column) {
|
||||
background: var(--bg-secondary, #fafafa);
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.bitable-grid-scope :deep(.vxe-body--column.is--dirty) {
|
||||
background: var(--bg-tertiary, #fffbe6);
|
||||
}
|
||||
|
||||
.bitable-grid-scope :deep(.vxe-cell--dirty) {
|
||||
color: var(--color-primary, #1677ff);
|
||||
}
|
||||
</style>
|
||||
|
|
@ -0,0 +1,186 @@
|
|||
<template>
|
||||
<a-form layout="vertical">
|
||||
<a-form-item label="字段名称">
|
||||
<a-input v-model:value="localName" :maxlength="100" />
|
||||
</a-form-item>
|
||||
|
||||
<a-form-item label="字段类型">
|
||||
<a-select v-model:value="localType" :disabled="!!field" @change="onTypeChange">
|
||||
<a-select-option value="text">文本</a-select-option>
|
||||
<a-select-option value="number">数字</a-select-option>
|
||||
<a-select-option value="date">日期</a-select-option>
|
||||
<a-select-option value="select">单选</a-select-option>
|
||||
<a-select-option value="multiselect">多选</a-select-option>
|
||||
<a-select-option value="formula">公式</a-select-option>
|
||||
<a-select-option value="attachment">附件</a-select-option>
|
||||
<a-select-option value="image">图片</a-select-option>
|
||||
</a-select>
|
||||
</a-form-item>
|
||||
|
||||
<!-- Select / Multiselect: options editor -->
|
||||
<template v-if="localType === 'select' || localType === 'multiselect'">
|
||||
<a-form-item label="选项列表">
|
||||
<div
|
||||
v-for="(_, idx) in selectOptions"
|
||||
:key="idx"
|
||||
class="field-config-form__option-row"
|
||||
>
|
||||
<a-input
|
||||
v-model:value="selectOptions[idx]"
|
||||
placeholder="选项值"
|
||||
:maxlength="200"
|
||||
/>
|
||||
<a-button type="text" danger :icon="h(DeleteOutlined)" @click="removeOption(idx)" />
|
||||
</div>
|
||||
<a-button type="dashed" block :icon="h(PlusOutlined)" @click="addOption">
|
||||
添加选项
|
||||
</a-button>
|
||||
</a-form-item>
|
||||
</template>
|
||||
|
||||
<!-- Formula: expression editor with live validation -->
|
||||
<template v-if="localType === 'formula'">
|
||||
<a-form-item label="公式表达式">
|
||||
<a-textarea
|
||||
v-model:value="formulaExpr"
|
||||
placeholder="例如: {field_id_1} + {field_id_2} 或 SUM({field_id})"
|
||||
:rows="3"
|
||||
:maxlength="2000"
|
||||
/>
|
||||
<div class="field-config-form__formula-hint">
|
||||
用 {字段ID} 引用其他字段，支持 SUM/AVG/COUNT/MIN/MAX/ABS/ROUND/IF/LEN/CONCAT
|
||||
</div>
|
||||
</a-form-item>
|
||||
|
||||
<a-form-item v-if="formulaExpr" label="语法校验">
|
||||
<a-alert
|
||||
v-if="formulaValid === true"
|
||||
type="success"
|
||||
message="公式语法正确"
|
||||
show-icon
|
||||
/>
|
||||
<a-alert
|
||||
v-else-if="formulaValid === false"
|
||||
type="error"
|
||||
:message="formulaError || '公式语法错误'"
|
||||
show-icon
|
||||
/>
|
||||
<a-alert v-else type="info" message="校验中..." show-icon />
|
||||
</a-form-item>
|
||||
</template>
|
||||
|
||||
<!-- Date: format -->
|
||||
<template v-if="localType === 'date'">
|
||||
<a-form-item label="日期格式">
|
||||
<a-select v-model:value="dateFormat">
|
||||
<a-select-option value="YYYY-MM-DD">YYYY-MM-DD</a-select-option>
|
||||
<a-select-option value="YYYY-MM-DD HH:mm">YYYY-MM-DD HH:mm</a-select-option>
|
||||
<a-select-option value="YYYY/MM/DD">YYYY/MM/DD</a-select-option>
|
||||
</a-select>
|
||||
</a-form-item>
|
||||
</template>
|
||||
</a-form>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { ref, watch, h } from 'vue'
|
||||
import { Button as AButton, Input as AInput, Select as ASelect, Alert as AAlert } from 'ant-design-vue'
|
||||
import { DeleteOutlined, PlusOutlined } from '@ant-design/icons-vue'
|
||||
import type { IBitableField, FieldType } from '@/api/bitable'
|
||||
|
||||
const props = defineProps<{
|
||||
field?: IBitableField | null
|
||||
}>()
|
||||
|
||||
const localName = ref(props.field?.name ?? '')
|
||||
const localType = ref<FieldType>(props.field?.field_type ?? 'text')
|
||||
const selectOptions = ref<string[]>(
|
||||
(props.field?.config?.options as string[]) ?? [],
|
||||
)
|
||||
const formulaExpr = ref((props.field?.config?.formula_expr as string) ?? '')
|
||||
const dateFormat = ref((props.field?.config?.format as string) ?? 'YYYY-MM-DD')
|
||||
|
||||
// Formula validation state
|
||||
const formulaValid = ref<boolean | null>(null)
|
||||
const formulaError = ref<string | null>(null)
|
||||
let validateTimer: ReturnType<typeof setTimeout> | null = null
|
||||
|
||||
// Debounced formula validation
|
||||
watch(formulaExpr, (val) => {
|
||||
if (!val.trim()) {
|
||||
formulaValid.value = null
|
||||
return
|
||||
}
|
||||
formulaValid.value = null
|
||||
if (validateTimer) clearTimeout(validateTimer)
|
||||
validateTimer = setTimeout(async () => {
|
||||
try {
|
||||
const { bitableApi } = await import('@/api/bitable')
|
||||
const result = await bitableApi.validateFormula(val)
|
||||
formulaValid.value = result.valid
|
||||
formulaError.value = result.error ?? null
|
||||
} catch (err) {
|
||||
formulaValid.value = false
|
||||
formulaError.value = err instanceof Error ? err.message : String(err)
|
||||
}
|
||||
}, 500)
|
||||
})
|
||||
|
||||
function onTypeChange(): void {
|
||||
// Reset type-specific config when type changes
|
||||
selectOptions.value = []
|
||||
formulaExpr.value = ''
|
||||
formulaValid.value = null
|
||||
}
|
||||
|
||||
function addOption(): void {
|
||||
selectOptions.value.push('')
|
||||
}
|
||||
|
||||
function removeOption(idx: number): void {
|
||||
selectOptions.value.splice(idx, 1)
|
||||
}
|
||||
|
||||
// Build config object from current state
|
||||
function buildConfig(): Record<string, unknown> {
|
||||
const config: Record<string, unknown> = {}
|
||||
if (localType.value === 'select' || localType.value === 'multiselect') {
|
||||
config.options = selectOptions.value.filter((o) => o.trim())
|
||||
}
|
||||
if (localType.value === 'formula') {
|
||||
config.formula_expr = formulaExpr.value
|
||||
}
|
||||
if (localType.value === 'date') {
|
||||
config.format = dateFormat.value
|
||||
}
|
||||
return config
|
||||
}
|
||||
|
||||
// Expose method for parent to get current form data
|
||||
defineExpose({
|
||||
getData: () => ({
|
||||
name: localName.value.trim(),
|
||||
fieldType: localType.value,
|
||||
config: buildConfig(),
|
||||
}),
|
||||
isFormulaValid: () => {
|
||||
if (localType.value !== 'formula') return true
|
||||
if (!formulaExpr.value.trim()) return false
|
||||
return formulaValid.value === true
|
||||
},
|
||||
})
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.field-config-form__option-row {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
|
||||
.field-config-form__formula-hint {
|
||||
margin-top: 4px;
|
||||
font-size: 12px;
|
||||
color: var(--text-secondary, #8c8c8c);
|
||||
}
|
||||
</style>
|
||||
|
|
@ -0,0 +1,249 @@
|
|||
<template>
|
||||
<a-drawer
|
||||
:open="open"
|
||||
title="字段管理"
|
||||
placement="right"
|
||||
:width="480"
|
||||
@close="handleClose"
|
||||
>
|
||||
<!-- Field list -->
|
||||
<div class="field-manage-panel__list">
|
||||
<div
|
||||
v-for="f in fields"
|
||||
:key="f.id"
|
||||
class="field-manage-panel__item"
|
||||
>
|
||||
<div class="field-manage-panel__item-info">
|
||||
<span class="field-manage-panel__item-name">{{ f.name }}</span>
|
||||
<div class="field-manage-panel__item-meta">
|
||||
<a-tag :color="typeColor(f.field_type)">{{ typeLabel(f.field_type) }}</a-tag>
|
||||
<a-tag :color="f.owner === 'agent' ? 'blue' : 'green'">
|
||||
{{ f.owner === 'agent' ? 'Agent' : '用户' }}
|
||||
</a-tag>
|
||||
</div>
|
||||
</div>
|
||||
<div class="field-manage-panel__item-actions">
|
||||
<a-button type="text" size="small" @click="handleEdit(f)">编辑</a-button>
|
||||
<a-button type="text" size="small" danger @click="handleDelete(f)">删除</a-button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<a-empty v-if="fields.length === 0" description="暂无字段" />
|
||||
</div>
|
||||
|
||||
<!-- Add field button -->
|
||||
<div class="field-manage-panel__add">
|
||||
<a-button type="dashed" block :icon="h(PlusOutlined)" @click="handleAdd">
|
||||
添加字段
|
||||
</a-button>
|
||||
</div>
|
||||
|
||||
<!-- Edit/Add modal -->
|
||||
<a-modal
|
||||
:open="editModalOpen"
|
||||
:title="editingField ? '编辑字段' : '新增字段'"
|
||||
:confirm-loading="saving"
|
||||
@ok="handleSave"
|
||||
@cancel="editModalOpen = false"
|
||||
>
|
||||
<FieldConfigForm
|
||||
v-if="editModalOpen"
|
||||
ref="formRef"
|
||||
:field="editingField"
|
||||
/>
|
||||
</a-modal>
|
||||
|
||||
<!-- Delete dependency confirmation -->
|
||||
<a-modal
|
||||
:open="deleteConfirmOpen"
|
||||
title="确认删除"
|
||||
@ok="handleForceDelete"
|
||||
@cancel="deleteConfirmOpen = false"
|
||||
>
|
||||
<p>该字段被以下内容引用，强制删除将级联清理：</p>
|
||||
<pre class="field-manage-panel__deps">{{ JSON.stringify(deleteDependencies, null, 2) }}</pre>
|
||||
</a-modal>
|
||||
</a-drawer>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { ref, h } from 'vue'
|
||||
import { Modal as AModal, Drawer as ADrawer, Button as AButton, Tag as ATag, Empty as AEmpty } from 'ant-design-vue'
|
||||
import { PlusOutlined } from '@ant-design/icons-vue'
|
||||
import type { IBitableField, FieldType } from '@/api/bitable'
|
||||
import { useBitableStore } from '@/stores/bitable'
|
||||
import FieldConfigForm from './FieldConfigForm.vue'
|
||||
|
||||
defineProps<{
|
||||
open: boolean
|
||||
fields: IBitableField[]
|
||||
}>()
|
||||
|
||||
const emit = defineEmits<{
|
||||
(e: 'close'): void
|
||||
}>()
|
||||
|
||||
const store = useBitableStore()
|
||||
|
||||
const editModalOpen = ref(false)
|
||||
const editingField = ref<IBitableField | null>(null)
|
||||
const saving = ref(false)
|
||||
const formRef = ref<InstanceType<typeof FieldConfigForm> | null>(null)
|
||||
|
||||
// Delete confirmation state
|
||||
const deleteConfirmOpen = ref(false)
|
||||
const deleteTargetId = ref<string | null>(null)
|
||||
const deleteDependencies = ref<Record<string, unknown>>({})
|
||||
|
||||
function handleClose(): void {
|
||||
emit('close')
|
||||
}
|
||||
|
||||
function handleAdd(): void {
|
||||
editingField.value = null
|
||||
editModalOpen.value = true
|
||||
}
|
||||
|
||||
function handleEdit(field: IBitableField): void {
|
||||
editingField.value = field
|
||||
editModalOpen.value = true
|
||||
}
|
||||
|
||||
async function handleSave(): Promise<void> {
|
||||
const form = formRef.value
|
||||
if (!form) return
|
||||
|
||||
if (!form.isFormulaValid()) {
|
||||
AModal.warning({
|
||||
title: '公式语法错误',
|
||||
content: '请修正公式表达式后再保存',
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
const data = form.getData()
|
||||
if (!data.name) {
|
||||
AModal.warning({ title: '请输入字段名称' })
|
||||
return
|
||||
}
|
||||
|
||||
saving.value = true
|
||||
try {
|
||||
if (editingField.value) {
|
||||
// Update existing field
|
||||
await store.updateField(editingField.value.id, {
|
||||
name: data.name,
|
||||
config: data.config,
|
||||
})
|
||||
} else {
|
||||
// Create new field
|
||||
await store.addField(data.name, data.fieldType, data.config)
|
||||
}
|
||||
editModalOpen.value = false
|
||||
} finally {
|
||||
saving.value = false
|
||||
}
|
||||
}
|
||||
|
||||
async function handleDelete(field: IBitableField): Promise<void> {
|
||||
const result = await store.deleteField(field.id)
|
||||
if (!result.success && result.dependencies) {
|
||||
// Has dependencies — show confirmation
|
||||
deleteTargetId.value = field.id
|
||||
deleteDependencies.value = result.dependencies
|
||||
deleteConfirmOpen.value = true
|
||||
}
|
||||
}
|
||||
|
||||
async function handleForceDelete(): Promise<void> {
|
||||
if (!deleteTargetId.value) return
|
||||
await store.deleteField(deleteTargetId.value, true)
|
||||
deleteConfirmOpen.value = false
|
||||
deleteTargetId.value = null
|
||||
deleteDependencies.value = {}
|
||||
}
|
||||
|
||||
// ── Helpers ──────────────────────────────────────────────
|
||||
|
||||
function typeLabel(t: FieldType): string {
|
||||
const labels: Record<FieldType, string> = {
|
||||
text: '文本',
|
||||
number: '数字',
|
||||
date: '日期',
|
||||
select: '单选',
|
||||
multiselect: '多选',
|
||||
attachment: '附件',
|
||||
image: '图片',
|
||||
formula: '公式',
|
||||
lookup: '引用',
|
||||
}
|
||||
return labels[t] ?? t
|
||||
}
|
||||
|
||||
function typeColor(t: FieldType): string {
|
||||
const colors: Partial<Record<FieldType, string>> = {
|
||||
text: 'default',
|
||||
number: 'blue',
|
||||
date: 'purple',
|
||||
select: 'cyan',
|
||||
multiselect: 'cyan',
|
||||
formula: 'orange',
|
||||
lookup: 'magenta',
|
||||
attachment: 'geekblue',
|
||||
image: 'geekblue',
|
||||
}
|
||||
return colors[t] ?? 'default'
|
||||
}
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.field-manage-panel__list {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 8px;
|
||||
margin-bottom: 16px;
|
||||
}
|
||||
|
||||
.field-manage-panel__item {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 12px;
|
||||
border: 1px solid var(--border-color, #f0f0f0);
|
||||
border-radius: 6px;
|
||||
}
|
||||
|
||||
.field-manage-panel__item-info {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 4px;
|
||||
}
|
||||
|
||||
.field-manage-panel__item-name {
|
||||
font-weight: 500;
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
.field-manage-panel__item-meta {
|
||||
display: flex;
|
||||
gap: 4px;
|
||||
}
|
||||
|
||||
.field-manage-panel__item-actions {
|
||||
display: flex;
|
||||
gap: 4px;
|
||||
}
|
||||
|
||||
.field-manage-panel__add {
|
||||
margin-top: 16px;
|
||||
}
|
||||
|
||||
.field-manage-panel__deps {
|
||||
background: var(--bg-secondary, #fafafa);
|
||||
padding: 12px;
|
||||
border-radius: 4px;
|
||||
font-size: 12px;
|
||||
max-height: 300px;
|
||||
overflow: auto;
|
||||
}
|
||||
</style>
|
||||
|
|
@ -0,0 +1,179 @@
|
|||
<template>
|
||||
<div class="filter-builder">
|
||||
<div
|
||||
v-for="(condition, idx) in conditions"
|
||||
:key="idx"
|
||||
class="filter-builder__row"
|
||||
>
|
||||
<a-select
|
||||
v-model:value="condition.field_id"
|
||||
placeholder="选择字段"
|
||||
style="width: 140px"
|
||||
@change="onFieldChange(idx)"
|
||||
>
|
||||
<a-select-option
|
||||
v-for="f in fields"
|
||||
:key="f.id"
|
||||
:value="f.id"
|
||||
>
|
||||
{{ f.name }}
|
||||
</a-select-option>
|
||||
</a-select>
|
||||
|
||||
<a-select
|
||||
v-model:value="condition.op"
|
||||
placeholder="操作符"
|
||||
style="width: 120px"
|
||||
>
|
||||
<a-select-option
|
||||
v-for="op in availableOps(condition.field_id)"
|
||||
:key="op.value"
|
||||
:value="op.value"
|
||||
>
|
||||
{{ op.label }}
|
||||
</a-select-option>
|
||||
</a-select>
|
||||
|
||||
<a-input
|
||||
v-if="getValueInputType(condition.field_id) === 'text'"
|
||||
v-model:value="condition.value"
|
||||
placeholder="值"
|
||||
style="width: 160px"
|
||||
/>
|
||||
<a-input-number
|
||||
v-else-if="getValueInputType(condition.field_id) === 'number'"
|
||||
v-model:value="condition.value"
|
||||
placeholder="数值"
|
||||
style="width: 160px"
|
||||
/>
|
||||
<a-date-picker
|
||||
v-else-if="getValueInputType(condition.field_id) === 'date'"
|
||||
:value="condition.value as string | undefined"
|
||||
@change="(_, dateString) => condition.value = dateString"
|
||||
style="width: 160px"
|
||||
/>
|
||||
|
||||
<a-button type="text" danger :icon="h(DeleteOutlined)" @click="removeCondition(idx)" />
|
||||
</div>
|
||||
|
||||
<a-button type="dashed" block :icon="h(PlusOutlined)" @click="addCondition">
|
||||
添加筛选条件
|
||||
</a-button>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { ref, watch, h } from 'vue'
|
||||
import {
|
||||
Button as AButton,
|
||||
Select as ASelect,
|
||||
Input as AInput,
|
||||
InputNumber as AInputNumber,
|
||||
DatePicker as ADatePicker,
|
||||
} from 'ant-design-vue'
|
||||
import { DeleteOutlined, PlusOutlined } from '@ant-design/icons-vue'
|
||||
import type { IBitableField, FieldType } from '@/api/bitable'
|
||||
|
||||
interface FilterCondition {
|
||||
field_id: string
|
||||
op: string
|
||||
// Value type varies by field type: string for text, number for number, string(ISO) for date
|
||||
value: string | number | undefined
|
||||
}
|
||||
|
||||
const props = defineProps<{
|
||||
fields: IBitableField[]
|
||||
modelValue?: FilterCondition[]
|
||||
}>()
|
||||
|
||||
const emit = defineEmits<{
|
||||
(e: 'update:modelValue', conditions: FilterCondition[]): void
|
||||
}>()
|
||||
|
||||
const conditions = ref<FilterCondition[]>(props.modelValue ? [...props.modelValue] : [])
|
||||
|
||||
watch(conditions, (val) => {
|
||||
emit('update:modelValue', val)
|
||||
}, { deep: true })
|
||||
|
||||
function addCondition(): void {
|
||||
conditions.value.push({ field_id: '', op: 'eq', value: undefined })
|
||||
}
|
||||
|
||||
function removeCondition(idx: number): void {
|
||||
conditions.value.splice(idx, 1)
|
||||
}
|
||||
|
||||
function onFieldChange(idx: number): void {
|
||||
// Reset op to first available for the new field type
|
||||
const condition = conditions.value[idx]
|
||||
const ops = availableOps(condition.field_id)
|
||||
if (ops.length > 0 && !ops.some((o) => o.value === condition.op)) {
|
||||
condition.op = ops[0].value
|
||||
}
|
||||
condition.value = undefined
|
||||
}
|
||||
|
||||
function getField(field_id: string): IBitableField | undefined {
|
||||
return props.fields.find((f) => f.id === field_id)
|
||||
}
|
||||
|
||||
function availableOps(field_id: string): { value: string; label: string }[] {
|
||||
const field = getField(field_id)
|
||||
if (!field) return [{ value: 'eq', label: '等于' }]
|
||||
|
||||
const type = field.field_type as FieldType
|
||||
switch (type) {
|
||||
case 'number':
|
||||
return [
|
||||
{ value: 'eq', label: '等于' },
|
||||
{ value: 'ne', label: '不等于' },
|
||||
{ value: 'gt', label: '大于' },
|
||||
{ value: 'lt', label: '小于' },
|
||||
{ value: 'gte', label: '大于等于' },
|
||||
{ value: 'lte', label: '小于等于' },
|
||||
]
|
||||
case 'date':
|
||||
return [
|
||||
{ value: 'eq', label: '等于' },
|
||||
{ value: 'gt', label: '之后' },
|
||||
{ value: 'lt', label: '之前' },
|
||||
]
|
||||
case 'text':
|
||||
default:
|
||||
return [
|
||||
{ value: 'eq', label: '等于' },
|
||||
{ value: 'ne', label: '不等于' },
|
||||
{ value: 'contains', label: '包含' },
|
||||
{ value: 'is_empty', label: '为空' },
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
function getValueInputType(field_id: string): 'text' | 'number' | 'date' | null {
|
||||
const field = getField(field_id)
|
||||
if (!field) return 'text'
|
||||
const type = field.field_type as FieldType
|
||||
switch (type) {
|
||||
case 'number':
|
||||
return 'number'
|
||||
case 'date':
|
||||
return 'date'
|
||||
default:
|
||||
return 'text'
|
||||
}
|
||||
}
|
||||
|
||||
defineExpose({
|
||||
getConditions: () => conditions.value,
|
||||
})
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.filter-builder__row {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
margin-bottom: 8px;
|
||||
align-items: center;
|
||||
}
|
||||
</style>
|
||||
|
|
@ -0,0 +1,117 @@
|
|||
<template>
|
||||
<div class="image-cell">
|
||||
<div
|
||||
v-for="(img, idx) in images"
|
||||
:key="idx"
|
||||
ref="containerRefs"
|
||||
:data-idx="idx"
|
||||
class="image-cell__thumb"
|
||||
@click="preview(img)"
|
||||
>
|
||||
<img
|
||||
v-if="loaded[idx]"
|
||||
:src="img.url"
|
||||
:alt="img.filename"
|
||||
class="image-cell__img"
|
||||
/>
|
||||
<div v-else class="image-cell__placeholder">
|
||||
<FileImageOutlined />
|
||||
</div>
|
||||
</div>
|
||||
<span v-if="!images || images.length === 0" class="image-cell__empty">—</span>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { ref, watch, onMounted, onUnmounted, nextTick } from 'vue'
|
||||
import { FileImageOutlined } from '@ant-design/icons-vue'
|
||||
import type { IAttachmentMeta } from '@/api/bitable'
|
||||
|
||||
const props = defineProps<{
|
||||
images: IAttachmentMeta[] | null | undefined
|
||||
}>()
|
||||
|
||||
const containerRefs = ref<HTMLElement[]>([])
|
||||
const loaded = ref<Record<number, boolean>>({})
|
||||
let observer: IntersectionObserver | null = null
|
||||
|
||||
function setupObserver(): void {
|
||||
if (observer) {
|
||||
observer.disconnect()
|
||||
}
|
||||
observer = new IntersectionObserver(
|
||||
(entries) => {
|
||||
for (const entry of entries) {
|
||||
if (entry.isIntersecting) {
|
||||
const idx = Number((entry.target as HTMLElement).dataset.idx)
|
||||
loaded.value[idx] = true
|
||||
observer?.unobserve(entry.target)
|
||||
}
|
||||
}
|
||||
},
|
||||
{ rootMargin: '100px' },
|
||||
)
|
||||
nextTick(() => {
|
||||
containerRefs.value.forEach((el) => {
|
||||
if (el && observer) observer.observe(el)
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
function preview(img: IAttachmentMeta): void {
|
||||
window.open(img.url, '_blank')
|
||||
}
|
||||
|
||||
watch(
|
||||
() => props.images,
|
||||
() => {
|
||||
loaded.value = {}
|
||||
setupObserver()
|
||||
},
|
||||
)
|
||||
|
||||
onMounted(() => {
|
||||
setupObserver()
|
||||
})
|
||||
|
||||
onUnmounted(() => {
|
||||
observer?.disconnect()
|
||||
})
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.image-cell {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 4px;
|
||||
padding: 2px 0;
|
||||
}
|
||||
|
||||
.image-cell__thumb {
|
||||
width: 40px;
|
||||
height: 40px;
|
||||
border-radius: 4px;
|
||||
overflow: hidden;
|
||||
cursor: pointer;
|
||||
background: var(--bg-secondary, #f5f5f5);
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
border: 1px solid var(--border-color, #f0f0f0);
|
||||
}
|
||||
|
||||
.image-cell__img {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
object-fit: cover;
|
||||
}
|
||||
|
||||
.image-cell__placeholder {
|
||||
color: var(--text-placeholder, #bfbfbf);
|
||||
font-size: 16px;
|
||||
}
|
||||
|
||||
.image-cell__empty {
|
||||
color: var(--text-placeholder, #bfbfbf);
|
||||
}
|
||||
</style>
|
||||
|
|
@ -0,0 +1,103 @@
|
|||
<template>
|
||||
<a-modal
|
||||
:open="open"
|
||||
title="新建数据表"
|
||||
:confirm-loading="loading"
|
||||
@ok="handleOk"
|
||||
@cancel="handleCancel"
|
||||
>
|
||||
<a-form
|
||||
ref="formRef"
|
||||
:model="formState"
|
||||
:rules="rules"
|
||||
layout="vertical"
|
||||
>
|
||||
<a-form-item label="表名" name="name">
|
||||
<a-input
|
||||
v-model:value="formState.name"
|
||||
placeholder="请输入表名"
|
||||
:maxlength="100"
|
||||
/>
|
||||
</a-form-item>
|
||||
<a-form-item label="描述" name="description">
|
||||
<a-textarea
|
||||
v-model:value="formState.description"
|
||||
placeholder="可选，表的用途描述"
|
||||
:rows="2"
|
||||
:maxlength="500"
|
||||
/>
|
||||
</a-form-item>
|
||||
</a-form>
|
||||
</a-modal>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { ref, reactive, watch } from 'vue'
|
||||
import type { FormInstance } from 'ant-design-vue'
|
||||
|
||||
const props = defineProps<{
|
||||
open: boolean
|
||||
}>()
|
||||
|
||||
const emit = defineEmits<{
|
||||
(e: 'success', tableId: string): void
|
||||
(e: 'cancel'): void
|
||||
}>()
|
||||
|
||||
const formRef = ref<FormInstance | null>(null)
|
||||
const loading = ref(false)
|
||||
|
||||
const formState = reactive({
|
||||
name: '',
|
||||
description: '',
|
||||
})
|
||||
|
||||
const rules = {
|
||||
name: [
|
||||
{ required: true, message: '请输入表名', trigger: 'blur' },
|
||||
{ min: 1, max: 100, message: '表名长度 1-100', trigger: 'blur' },
|
||||
],
|
||||
}
|
||||
|
||||
// Reset form when modal opens
|
||||
watch(
|
||||
() => props.open,
|
||||
(val) => {
|
||||
if (val) {
|
||||
formState.name = ''
|
||||
formState.description = ''
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
async function handleOk(): Promise<void> {
|
||||
try {
|
||||
await formRef.value?.validate()
|
||||
} catch {
|
||||
return
|
||||
}
|
||||
|
||||
loading.value = true
|
||||
try {
|
||||
// Lazy import to avoid circular dependency
|
||||
const { bitableApi } = await import('@/api/bitable')
|
||||
const resp = await bitableApi.createTable({
|
||||
name: formState.name.trim(),
|
||||
description: formState.description.trim() || undefined,
|
||||
})
|
||||
emit('success', resp.table.id)
|
||||
} catch (err) {
|
||||
const { notification } = await import('ant-design-vue')
|
||||
notification.error({
|
||||
message: '创建失败',
|
||||
description: err instanceof Error ? err.message : String(err),
|
||||
})
|
||||
} finally {
|
||||
loading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
function handleCancel(): void {
|
||||
emit('cancel')
|
||||
}
|
||||
</script>
|
||||
|
|
@ -0,0 +1,121 @@
|
|||
<template>
|
||||
<div class="table-view-list">
|
||||
<div class="table-view-list__header">
|
||||
<span class="table-view-list__title">数据表</span>
|
||||
<a-button
|
||||
size="small"
|
||||
type="text"
|
||||
:icon="h(PlusOutlined)"
|
||||
@click="emit('create')"
|
||||
/>
|
||||
</div>
|
||||
|
||||
<a-spin :spinning="loading">
|
||||
<div class="table-view-list__items">
|
||||
<div
|
||||
v-for="t in tables"
|
||||
:key="t.id"
|
||||
class="table-view-list__item"
|
||||
:class="{ 'is-active': t.id === activeId }"
|
||||
@click="emit('select', t.id)"
|
||||
>
|
||||
<TableOutlined class="table-view-list__icon" />
|
||||
<span class="table-view-list__name">{{ t.name }}</span>
|
||||
</div>
|
||||
|
||||
<a-empty
|
||||
v-if="!loading && tables.length === 0"
|
||||
:image="Empty.PRESENTED_IMAGE_SIMPLE"
|
||||
description="暂无数据表"
|
||||
class="table-view-list__empty"
|
||||
/>
|
||||
</div>
|
||||
</a-spin>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { h } from 'vue'
|
||||
import { Button as AButton, Spin as ASpin, Empty } from 'ant-design-vue'
|
||||
import { PlusOutlined, TableOutlined } from '@ant-design/icons-vue'
|
||||
import type { IBitableTable } from '@/api/bitable'
|
||||
|
||||
defineProps<{
|
||||
tables: IBitableTable[]
|
||||
activeId: string | null
|
||||
loading?: boolean
|
||||
}>()
|
||||
|
||||
const emit = defineEmits<{
|
||||
(e: 'select', tableId: string): void
|
||||
(e: 'create'): void
|
||||
}>()
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.table-view-list {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
background: var(--bg-primary, #fff);
|
||||
border-right: 1px solid var(--border-color, #f0f0f0);
|
||||
}
|
||||
|
||||
.table-view-list__header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 12px 16px;
|
||||
border-bottom: 1px solid var(--border-color, #f0f0f0);
|
||||
}
|
||||
|
||||
.table-view-list__title {
|
||||
font-weight: 600;
|
||||
font-size: 14px;
|
||||
color: var(--text-primary, #1f1f1f);
|
||||
}
|
||||
|
||||
.table-view-list__items {
|
||||
flex: 1;
|
||||
overflow-y: auto;
|
||||
padding: 4px 0;
|
||||
}
|
||||
|
||||
.table-view-list__item {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
padding: 8px 16px;
|
||||
cursor: pointer;
|
||||
transition: background 0.15s;
|
||||
font-size: 13px;
|
||||
color: var(--text-secondary, #595959);
|
||||
}
|
||||
|
||||
.table-view-list__item:hover {
|
||||
background: var(--bg-secondary, #fafafa);
|
||||
}
|
||||
|
||||
.table-view-list__item.is-active {
|
||||
background: var(--color-primary-bg, #e6f4ff);
|
||||
color: var(--color-primary, #1677ff);
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.table-view-list__icon {
|
||||
font-size: 14px;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.table-view-list__name {
|
||||
flex: 1;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.table-view-list__empty {
|
||||
padding: 24px 0;
|
||||
}
|
||||
</style>
|
||||
|
|
@ -0,0 +1,175 @@
|
|||
<template>
|
||||
<a-drawer
|
||||
:open="open"
|
||||
title="视图配置"
|
||||
placement="right"
|
||||
:width="520"
|
||||
@close="handleClose"
|
||||
>
|
||||
<a-tabs v-model:activeKey="activeTab">
|
||||
<!-- Filter tab -->
|
||||
<a-tab-pane key="filter" tab="筛选">
|
||||
<FilterBuilder
|
||||
ref="filterRef"
|
||||
:fields="fields"
|
||||
:model-value="currentFilters"
|
||||
/>
|
||||
<div class="view-config-panel__actions">
|
||||
<a-button type="primary" @click="saveFilters">保存筛选</a-button>
|
||||
</div>
|
||||
</a-tab-pane>
|
||||
|
||||
<!-- Sort tab -->
|
||||
<a-tab-pane key="sort" tab="排序">
|
||||
<div class="view-config-panel__sort-row">
|
||||
<a-select
|
||||
v-model:value="sortFieldId"
|
||||
placeholder="选择排序字段"
|
||||
style="width: 200px"
|
||||
>
|
||||
<a-select-option
|
||||
v-for="f in fields"
|
||||
:key="f.id"
|
||||
:value="f.id"
|
||||
>
|
||||
{{ f.name }}
|
||||
</a-select-option>
|
||||
</a-select>
|
||||
<a-select v-model:value="sortOrder" style="width: 100px">
|
||||
<a-select-option value="asc">升序</a-select-option>
|
||||
<a-select-option value="desc">降序</a-select-option>
|
||||
</a-select>
|
||||
</div>
|
||||
<div class="view-config-panel__actions">
|
||||
<a-button type="primary" @click="saveSort">保存排序</a-button>
|
||||
</div>
|
||||
</a-tab-pane>
|
||||
|
||||
<!-- Hidden fields tab -->
|
||||
<a-tab-pane key="hidden" tab="隐藏字段">
|
||||
<p class="view-config-panel__hint">勾选要隐藏的字段：</p>
|
||||
<a-checkbox-group v-model:value="hiddenFieldIds">
|
||||
<div
|
||||
v-for="f in fields"
|
||||
:key="f.id"
|
||||
class="view-config-panel__field-check"
|
||||
>
|
||||
<a-checkbox :value="f.id">{{ f.name }}</a-checkbox>
|
||||
</div>
|
||||
</a-checkbox-group>
|
||||
<div class="view-config-panel__actions">
|
||||
<a-button type="primary" @click="saveHidden">保存隐藏配置</a-button>
|
||||
</div>
|
||||
</a-tab-pane>
|
||||
</a-tabs>
|
||||
</a-drawer>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { ref, watch, computed } from 'vue'
|
||||
import {
|
||||
Drawer as ADrawer,
|
||||
Tabs as ATabs,
|
||||
Button as AButton,
|
||||
Select as ASelect,
|
||||
Checkbox as ACheckbox,
|
||||
} from 'ant-design-vue'
|
||||
import type { IBitableField, IBitableView } from '@/api/bitable'
|
||||
import { useBitableStore } from '@/stores/bitable'
|
||||
import FilterBuilder from './FilterBuilder.vue'
|
||||
|
||||
const props = defineProps<{
|
||||
open: boolean
|
||||
fields: IBitableField[]
|
||||
view: IBitableView | null
|
||||
}>()
|
||||
|
||||
const emit = defineEmits<{
|
||||
(e: 'close'): void
|
||||
}>()
|
||||
|
||||
const store = useBitableStore()
|
||||
|
||||
const activeTab = ref('filter')
|
||||
const filterRef = ref<InstanceType<typeof FilterBuilder> | null>(null)
|
||||
|
||||
// Current config from view
|
||||
const currentFilters = computed(() => {
|
||||
const filters = props.view?.config?.filters as unknown
|
||||
if (!Array.isArray(filters)) return []
|
||||
return filters as { field_id: string; op: string; value: string | number | undefined }[]
|
||||
})
|
||||
|
||||
const sortFieldId = ref<string>(
|
||||
(props.view?.config?.sort as { field?: string })?.field ?? '',
|
||||
)
|
||||
const sortOrder = ref<string>(
|
||||
(props.view?.config?.sort as { order?: string })?.order ?? 'asc',
|
||||
)
|
||||
const hiddenFieldIds = ref<string[]>(
|
||||
(props.view?.config?.hidden_fields as string[]) ?? [],
|
||||
)
|
||||
|
||||
// Reset when view changes
|
||||
watch(
|
||||
() => props.view?.id,
|
||||
() => {
|
||||
sortFieldId.value = (props.view?.config?.sort as { field?: string })?.field ?? ''
|
||||
sortOrder.value = (props.view?.config?.sort as { order?: string })?.order ?? 'asc'
|
||||
hiddenFieldIds.value = (props.view?.config?.hidden_fields as string[]) ?? []
|
||||
},
|
||||
)
|
||||
|
||||
function handleClose(): void {
|
||||
emit('close')
|
||||
}
|
||||
|
||||
async function saveFilters(): Promise<void> {
|
||||
if (!props.view) return
|
||||
const conditions = filterRef.value?.getConditions() ?? []
|
||||
const config = {
|
||||
...props.view.config,
|
||||
filters: conditions,
|
||||
}
|
||||
await store.updateView(props.view.id, { config })
|
||||
}
|
||||
|
||||
async function saveSort(): Promise<void> {
|
||||
if (!props.view) return
|
||||
const config = {
|
||||
...props.view.config,
|
||||
sort: { field: sortFieldId.value, order: sortOrder.value },
|
||||
}
|
||||
await store.updateView(props.view.id, { config })
|
||||
}
|
||||
|
||||
async function saveHidden(): Promise<void> {
|
||||
if (!props.view) return
|
||||
const config = {
|
||||
...props.view.config,
|
||||
hidden_fields: hiddenFieldIds.value,
|
||||
}
|
||||
await store.updateView(props.view.id, { config })
|
||||
}
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.view-config-panel__sort-row {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
margin-bottom: 16px;
|
||||
}
|
||||
|
||||
.view-config-panel__actions {
|
||||
margin-top: 16px;
|
||||
}
|
||||
|
||||
.view-config-panel__hint {
|
||||
color: var(--text-secondary, #8c8c8c);
|
||||
margin-bottom: 12px;
|
||||
}
|
||||
|
||||
.view-config-panel__field-check {
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
</style>
|
||||
|
|
@ -0,0 +1,83 @@
|
|||
<template>
|
||||
<div class="view-switcher">
|
||||
<a-tabs
|
||||
v-model:activeKey="activeKey"
|
||||
type="editable-card"
|
||||
size="small"
|
||||
:add-icon="h(PlusOutlined)"
|
||||
@change="onSwitch"
|
||||
@edit="onEdit"
|
||||
>
|
||||
<a-tab-pane
|
||||
v-for="v in views"
|
||||
:key="v.id"
|
||||
:tab="v.name"
|
||||
:closable="false"
|
||||
/>
|
||||
</a-tabs>
|
||||
|
||||
<a-button
|
||||
v-if="activeKey"
|
||||
type="text"
|
||||
size="small"
|
||||
:icon="h(FilterOutlined)"
|
||||
@click="emit('config')"
|
||||
>
|
||||
配置
|
||||
</a-button>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { ref, watch, h } from 'vue'
|
||||
import { Tabs as ATabs, Button as AButton } from 'ant-design-vue'
|
||||
import { PlusOutlined, FilterOutlined } from '@ant-design/icons-vue'
|
||||
import type { IBitableView } from '@/api/bitable'
|
||||
|
||||
const props = defineProps<{
|
||||
views: IBitableView[]
|
||||
activeViewId: string | null
|
||||
}>()
|
||||
|
||||
const emit = defineEmits<{
|
||||
(e: 'switch', viewId: string): void
|
||||
(e: 'create'): void
|
||||
(e: 'config'): void
|
||||
}>()
|
||||
|
||||
// antd Tabs activeKey is string | number | undefined; bridge to/from null
|
||||
const activeKey = ref<string | undefined>(props.activeViewId ?? undefined)
|
||||
|
||||
watch(
|
||||
() => props.activeViewId,
|
||||
(val) => {
|
||||
activeKey.value = val ?? undefined
|
||||
},
|
||||
)
|
||||
|
||||
function onSwitch(key: string | number): void {
|
||||
emit('switch', String(key))
|
||||
}
|
||||
|
||||
function onEdit(_targetKey: unknown, action: 'add' | 'remove'): void {
|
||||
if (action === 'add') {
|
||||
emit('create')
|
||||
}
|
||||
// remove is disabled (closable=false) — no-op
|
||||
}
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.view-switcher {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
padding: 0 16px;
|
||||
border-bottom: 1px solid var(--border-color, #f0f0f0);
|
||||
}
|
||||
|
||||
.view-switcher :deep(.ant-tabs) {
|
||||
flex: 1;
|
||||
min-width: 0;
|
||||
}
|
||||
</style>
|
||||
|
|
@ -0,0 +1,381 @@
|
|||
/**
|
||||
* Pinia store for bitable feature — tables, fields, records, views,
|
||||
* and formula recalc status polling.
|
||||
*
|
||||
* ponytail: Formula recalc polling uses a simple setInterval. The polling
|
||||
* stops when no records have pending/calculating formula fields.
|
||||
* Ceiling: one polling timer per store instance; if multiple BitableGrid
|
||||
* components mount simultaneously they share the same store-level timer.
|
||||
*/
|
||||
|
||||
import { defineStore } from 'pinia'
|
||||
import { ref, computed } from 'vue'
|
||||
import { notification } from 'ant-design-vue'
|
||||
import { bitableApi } from '@/api/bitable'
|
||||
import type {
|
||||
IBitableTable,
|
||||
IBitableField,
|
||||
IBitableRecord,
|
||||
IBitableView,
|
||||
FieldType,
|
||||
} from '@/api/bitable'
|
||||
|
||||
export const useBitableStore = defineStore('bitable', () => {
|
||||
// --- State ---
|
||||
const tables = ref<IBitableTable[]>([])
|
||||
const currentTable = ref<IBitableTable | null>(null)
|
||||
const fields = ref<IBitableField[]>([])
|
||||
const records = ref<IBitableRecord[]>([])
|
||||
const views = ref<IBitableView[]>([])
|
||||
const currentView = ref<IBitableView | null>(null)
|
||||
const isLoading = ref(false)
|
||||
const error = ref<string | null>(null)
|
||||
const nextCursor = ref<string | null>(null)
|
||||
const recalcPendingCount = ref(0)
|
||||
|
||||
// Polling timer for formula recalc status
|
||||
let _pollTimer: ReturnType<typeof setInterval> | null = null
|
||||
const POLL_INTERVAL = 2000 // 2s per plan
|
||||
|
||||
// --- Getters ---
|
||||
const formulaFields = computed(() =>
|
||||
fields.value.filter((f) => f.field_type === 'formula'),
|
||||
)
|
||||
|
||||
const hasFormulaFields = computed(() => formulaFields.value.length > 0)
|
||||
|
||||
// --- Actions ---
|
||||
|
||||
/** Load all bitable tables */
|
||||
async function loadTables(): Promise<void> {
|
||||
isLoading.value = true
|
||||
error.value = null
|
||||
try {
|
||||
const resp = await bitableApi.listTables()
|
||||
tables.value = resp.tables || []
|
||||
} catch (err) {
|
||||
error.value = err instanceof Error ? err.message : '加载表格列表失败'
|
||||
notification.error({ message: '加载失败', description: error.value })
|
||||
} finally {
|
||||
isLoading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
/** Select a table and load its fields + records */
|
||||
async function selectTable(tableId: string): Promise<void> {
|
||||
stopPolling()
|
||||
records.value = []
|
||||
nextCursor.value = null
|
||||
recalcPendingCount.value = 0
|
||||
views.value = []
|
||||
currentView.value = null
|
||||
|
||||
const table = tables.value.find((t) => t.id === tableId)
|
||||
currentTable.value = table || null
|
||||
|
||||
if (!table) return
|
||||
|
||||
try {
|
||||
const [fieldsResp, recordsResp, viewsResp] = await Promise.all([
|
||||
bitableApi.listFields(tableId),
|
||||
bitableApi.listRecords(tableId, { limit: 100 }),
|
||||
bitableApi.listViews(tableId),
|
||||
])
|
||||
fields.value = fieldsResp.fields || []
|
||||
records.value = recordsResp.records || []
|
||||
nextCursor.value = recordsResp.next_cursor
|
||||
views.value = viewsResp.views || []
|
||||
|
||||
// Start polling if there are formula fields
|
||||
if (hasFormulaFields.value) {
|
||||
startPolling(tableId)
|
||||
}
|
||||
} catch (err) {
|
||||
error.value = err instanceof Error ? err.message : '加载表格数据失败'
|
||||
notification.error({ message: '加载失败', description: error.value })
|
||||
}
|
||||
}
|
||||
|
||||
/** Load more records (cursor pagination) */
|
||||
async function loadMoreRecords(): Promise<void> {
|
||||
if (!currentTable.value || !nextCursor.value) return
|
||||
try {
|
||||
const resp = await bitableApi.listRecords(currentTable.value.id, {
|
||||
cursor: nextCursor.value,
|
||||
limit: 100,
|
||||
})
|
||||
records.value.push(...(resp.records || []))
|
||||
nextCursor.value = resp.next_cursor
|
||||
} catch (err) {
|
||||
notification.error({
|
||||
message: '加载更多失败',
|
||||
description: err instanceof Error ? err.message : String(err),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/** Update a single cell value */
|
||||
async function updateCell(
|
||||
recordId: string,
|
||||
fieldId: string,
|
||||
value: unknown,
|
||||
): Promise<void> {
|
||||
try {
|
||||
const resp = await bitableApi.updateRecord(recordId, { [fieldId]: value })
|
||||
// Update local state
|
||||
const idx = records.value.findIndex((r) => r.id === recordId)
|
||||
if (idx >= 0) {
|
||||
records.value[idx] = resp.record
|
||||
}
|
||||
} catch (err) {
|
||||
notification.error({
|
||||
message: '更新失败',
|
||||
description: err instanceof Error ? err.message : String(err),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/** Add a new field */
|
||||
async function addField(
|
||||
name: string,
|
||||
fieldType: FieldType,
|
||||
config?: Record<string, unknown>,
|
||||
): Promise<IBitableField | null> {
|
||||
if (!currentTable.value) return null
|
||||
try {
|
||||
const resp = await bitableApi.createField(currentTable.value.id, {
|
||||
name,
|
||||
field_type: fieldType,
|
||||
config,
|
||||
})
|
||||
fields.value.push(resp.field)
|
||||
return resp.field
|
||||
} catch (err) {
|
||||
notification.error({
|
||||
message: '创建字段失败',
|
||||
description: err instanceof Error ? err.message : String(err),
|
||||
})
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
/** Create a new table */
|
||||
async function createTable(
|
||||
name: string,
|
||||
description?: string,
|
||||
): Promise<IBitableTable | null> {
|
||||
try {
|
||||
const resp = await bitableApi.createTable({ name, description })
|
||||
tables.value.push(resp.table)
|
||||
return resp.table
|
||||
} catch (err) {
|
||||
notification.error({
|
||||
message: '创建表格失败',
|
||||
description: err instanceof Error ? err.message : String(err),
|
||||
})
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
/** Update an existing field */
|
||||
async function updateField(
|
||||
fieldId: string,
|
||||
data: { name?: string; config?: Record<string, unknown> },
|
||||
): Promise<IBitableField | null> {
|
||||
try {
|
||||
const resp = await bitableApi.updateField(fieldId, data)
|
||||
const idx = fields.value.findIndex((f) => f.id === fieldId)
|
||||
if (idx >= 0) {
|
||||
fields.value[idx] = resp.field
|
||||
}
|
||||
return resp.field
|
||||
} catch (err) {
|
||||
notification.error({
|
||||
message: '更新字段失败',
|
||||
description: err instanceof Error ? err.message : String(err),
|
||||
})
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
/** Delete a field; returns dependencies on 409 */
|
||||
async function deleteField(
|
||||
fieldId: string,
|
||||
force = false,
|
||||
): Promise<{ success: boolean; dependencies?: Record<string, unknown> }> {
|
||||
try {
|
||||
await bitableApi.deleteField(fieldId, force)
|
||||
fields.value = fields.value.filter((f) => f.id !== fieldId)
|
||||
return { success: true }
|
||||
} catch (err) {
|
||||
const apiErr = err as { status?: number; detail?: unknown }
|
||||
// 409 = has dependencies, return them for UI confirmation
|
||||
if (apiErr.status === 409 && apiErr.detail) {
|
||||
const detail = apiErr.detail as Record<string, unknown>
|
||||
return { success: false, dependencies: detail.dependencies as Record<string, unknown> }
|
||||
}
|
||||
notification.error({
|
||||
message: '删除字段失败',
|
||||
description: err instanceof Error ? err.message : String(err),
|
||||
})
|
||||
return { success: false }
|
||||
}
|
||||
}
|
||||
|
||||
/** Refresh records (e.g. after Agent writes data via BitableTool) */
|
||||
async function refreshRecords(): Promise<void> {
|
||||
if (!currentTable.value) return
|
||||
try {
|
||||
const filters = currentView.value?.config?.filters as unknown[] | undefined
|
||||
const resp = await bitableApi.listRecords(currentTable.value.id, {
|
||||
limit: 100,
|
||||
filters: filters ? JSON.stringify(filters) : undefined,
|
||||
})
|
||||
records.value = resp.records || []
|
||||
nextCursor.value = resp.next_cursor
|
||||
} catch (err) {
|
||||
// Silent fail on refresh — user didn't explicitly request it
|
||||
console.warn('Failed to refresh records:', err)
|
||||
}
|
||||
}
|
||||
|
||||
// --- View management (U5c) ---
|
||||
|
||||
/** Create a new view for the current table */
|
||||
async function createView(
|
||||
name: string,
|
||||
viewType: IBitableView['view_type'] = 'grid',
|
||||
config?: Record<string, unknown>,
|
||||
): Promise<IBitableView | null> {
|
||||
if (!currentTable.value) return null
|
||||
try {
|
||||
const resp = await bitableApi.createView(currentTable.value.id, {
|
||||
name,
|
||||
view_type: viewType,
|
||||
config,
|
||||
})
|
||||
views.value.push(resp.view)
|
||||
currentView.value = resp.view
|
||||
return resp.view
|
||||
} catch (err) {
|
||||
notification.error({
|
||||
message: '创建视图失败',
|
||||
description: err instanceof Error ? err.message : String(err),
|
||||
})
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
/** Update a view's config (filters/sorts/hidden fields) */
|
||||
async function updateView(
|
||||
viewId: string,
|
||||
data: { name?: string; config?: Record<string, unknown> },
|
||||
): Promise<void> {
|
||||
try {
|
||||
const resp = await bitableApi.updateView(viewId, data)
|
||||
const idx = views.value.findIndex((v) => v.id === viewId)
|
||||
if (idx >= 0) {
|
||||
views.value[idx] = resp.view
|
||||
}
|
||||
if (currentView.value?.id === viewId) {
|
||||
currentView.value = resp.view
|
||||
// Re-query records with updated view config
|
||||
await refreshRecords()
|
||||
}
|
||||
} catch (err) {
|
||||
notification.error({
|
||||
message: '更新视图失败',
|
||||
description: err instanceof Error ? err.message : String(err),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/** Switch to a view — applies its config to the records query */
|
||||
async function switchView(viewId: string): Promise<void> {
|
||||
const view = views.value.find((v) => v.id === viewId)
|
||||
if (!view) return
|
||||
currentView.value = view
|
||||
await refreshRecords()
|
||||
}
|
||||
|
||||
// --- Formula recalc polling (R7) ---
|
||||
|
||||
/** Start polling for formula recalc status */
|
||||
function startPolling(tableId: string): void {
|
||||
stopPolling()
|
||||
_pollTimer = setInterval(async () => {
|
||||
await pollRecalcStatus(tableId)
|
||||
}, POLL_INTERVAL)
|
||||
}
|
||||
|
||||
/** Stop the polling timer */
|
||||
function stopPolling(): void {
|
||||
if (_pollTimer !== null) {
|
||||
clearInterval(_pollTimer)
|
||||
_pollTimer = null
|
||||
}
|
||||
}
|
||||
|
||||
/** Poll recalc status: reload records if any formula fields are still calculating */
|
||||
async function pollRecalcStatus(tableId: string): Promise<void> {
|
||||
try {
|
||||
const resp = await bitableApi.listRecords(tableId, { limit: 100 })
|
||||
const newRecords = resp.records || []
|
||||
|
||||
// Single traversal: collect pending records (formula field values still null)
|
||||
const pending = newRecords.filter((rec) =>
|
||||
formulaFields.value.some((f) => rec.values[f.id] == null),
|
||||
)
|
||||
const stillCalculating = pending.length > 0
|
||||
|
||||
// Only update state if records actually changed (avoid unnecessary re-renders)
|
||||
const oldIds = records.value.map((r) => r.id).join(',')
|
||||
const newIds = newRecords.map((r) => r.id).join(',')
|
||||
if (oldIds !== newIds || stillCalculating) {
|
||||
records.value = newRecords
|
||||
nextCursor.value = resp.next_cursor
|
||||
}
|
||||
|
||||
if (stillCalculating) {
|
||||
recalcPendingCount.value = pending.length
|
||||
} else {
|
||||
recalcPendingCount.value = 0
|
||||
stopPolling()
|
||||
}
|
||||
} catch (err) {
|
||||
// Silent fail on poll — don't spam notifications
|
||||
console.warn('Recalc poll failed:', err)
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
// State
|
||||
tables,
|
||||
currentTable,
|
||||
fields,
|
||||
records,
|
||||
views,
|
||||
currentView,
|
||||
isLoading,
|
||||
error,
|
||||
nextCursor,
|
||||
recalcPendingCount,
|
||||
// Getters
|
||||
formulaFields,
|
||||
hasFormulaFields,
|
||||
// Actions
|
||||
loadTables,
|
||||
selectTable,
|
||||
loadMoreRecords,
|
||||
updateCell,
|
||||
addField,
|
||||
createTable,
|
||||
updateField,
|
||||
deleteField,
|
||||
refreshRecords,
|
||||
createView,
|
||||
updateView,
|
||||
switchView,
|
||||
stopPolling,
|
||||
}
|
||||
})
|
||||
|
|
@ -0,0 +1,294 @@
|
|||
<template>
|
||||
<div class="bitable-view">
|
||||
<!-- Top bar -->
|
||||
<div class="bitable-view__topbar">
|
||||
<div class="bitable-view__topbar-left">
|
||||
<a-button type="text" :icon="h(ArrowLeftOutlined)" @click="goBack" />
|
||||
<span class="bitable-view__title">多维表格</span>
|
||||
</div>
|
||||
<div class="bitable-view__topbar-right">
|
||||
<a-tag v-if="store.recalcPendingCount > 0" color="processing">
|
||||
<LoadingOutlined /> 计算中 ({{ store.recalcPendingCount }})
|
||||
</a-tag>
|
||||
<a-button
|
||||
v-if="store.currentTable"
|
||||
size="small"
|
||||
:icon="h(SettingOutlined)"
|
||||
@click="fieldPanelOpen = true"
|
||||
>
|
||||
字段管理
|
||||
</a-button>
|
||||
<a-button size="small" :icon="h(ReloadOutlined)" @click="handleRefresh">
|
||||
刷新
|
||||
</a-button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Body: left sidebar + right grid -->
|
||||
<div class="bitable-view__body">
|
||||
<aside class="bitable-view__sidebar">
|
||||
<TableViewList
|
||||
:tables="store.tables"
|
||||
:active-id="store.currentTable?.id ?? null"
|
||||
:loading="store.isLoading"
|
||||
@select="handleSelectTable"
|
||||
@create="createModalOpen = true"
|
||||
/>
|
||||
</aside>
|
||||
|
||||
<main class="bitable-view__main">
|
||||
<div v-if="!store.currentTable" class="bitable-view__placeholder">
|
||||
<TableOutlined style="font-size: 48px; color: var(--text-placeholder)" />
|
||||
<p>请选择左侧的数据表</p>
|
||||
</div>
|
||||
|
||||
<template v-else>
|
||||
<div class="bitable-view__grid-header">
|
||||
<h3 class="bitable-view__table-name">{{ store.currentTable.name }}</h3>
|
||||
<span class="bitable-view__field-count">
|
||||
{{ store.fields.length }} 个字段 · {{ store.records.length }} 条记录
|
||||
</span>
|
||||
</div>
|
||||
|
||||
<!-- View switcher (U5c) -->
|
||||
<ViewSwitcher
|
||||
:views="store.views"
|
||||
:active-view-id="store.currentView?.id ?? null"
|
||||
@switch="handleSwitchView"
|
||||
@create="handleCreateView"
|
||||
@config="viewConfigOpen = true"
|
||||
/>
|
||||
|
||||
<div class="bitable-view__grid-container">
|
||||
<BitableGrid
|
||||
:fields="visibleFields"
|
||||
:records="store.records"
|
||||
:loading="store.isLoading"
|
||||
height="100%"
|
||||
@edit-cell="handleEditCell"
|
||||
/>
|
||||
</div>
|
||||
|
||||
<!-- Load more (cursor pagination) -->
|
||||
<div v-if="store.nextCursor" class="bitable-view__load-more">
|
||||
<a-button @click="store.loadMoreRecords()">加载更多</a-button>
|
||||
</div>
|
||||
</template>
|
||||
</main>
|
||||
</div>
|
||||
|
||||
<!-- Table create modal (U5b) -->
|
||||
<TableCreateModal
|
||||
:open="createModalOpen"
|
||||
@success="handleTableCreated"
|
||||
@cancel="createModalOpen = false"
|
||||
/>
|
||||
|
||||
<!-- Field management panel (U5b) -->
|
||||
<FieldManagePanel
|
||||
:open="fieldPanelOpen"
|
||||
:fields="store.fields"
|
||||
@close="fieldPanelOpen = false"
|
||||
/>
|
||||
|
||||
<!-- View config panel (U5c) -->
|
||||
<ViewConfigPanel
|
||||
:open="viewConfigOpen"
|
||||
:fields="store.fields"
|
||||
:view="store.currentView"
|
||||
@close="viewConfigOpen = false"
|
||||
/>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { ref, computed, h, onMounted, onUnmounted } from 'vue'
|
||||
import { useRouter } from 'vue-router'
|
||||
import { Button as AButton, Tag as ATag, Modal as AModal } from 'ant-design-vue'
|
||||
import {
|
||||
ArrowLeftOutlined,
|
||||
ReloadOutlined,
|
||||
LoadingOutlined,
|
||||
TableOutlined,
|
||||
SettingOutlined,
|
||||
} from '@ant-design/icons-vue'
|
||||
import { useBitableStore } from '@/stores/bitable'
|
||||
import TableViewList from '@/components/bitable/TableViewList.vue'
|
||||
import BitableGrid from '@/components/bitable/BitableGrid.vue'
|
||||
import TableCreateModal from '@/components/bitable/TableCreateModal.vue'
|
||||
import FieldManagePanel from '@/components/bitable/FieldManagePanel.vue'
|
||||
import ViewSwitcher from '@/components/bitable/ViewSwitcher.vue'
|
||||
import ViewConfigPanel from '@/components/bitable/ViewConfigPanel.vue'
|
||||
|
||||
const router = useRouter()
|
||||
const store = useBitableStore()
|
||||
|
||||
const createModalOpen = ref(false)
|
||||
const fieldPanelOpen = ref(false)
|
||||
const viewConfigOpen = ref(false)
|
||||
|
||||
// Filter out hidden fields based on current view config
|
||||
const visibleFields = computed(() => {
|
||||
const hiddenIds = (store.currentView?.config?.hidden_fields as string[]) ?? []
|
||||
if (hiddenIds.length === 0) return store.fields
|
||||
return store.fields.filter((f) => !hiddenIds.includes(f.id))
|
||||
})
|
||||
|
||||
onMounted(() => {
|
||||
store.loadTables()
|
||||
})
|
||||
|
||||
onUnmounted(() => {
|
||||
store.stopPolling()
|
||||
})
|
||||
|
||||
function goBack(): void {
|
||||
router.push('/agent/chat')
|
||||
}
|
||||
|
||||
function handleSelectTable(tableId: string): void {
|
||||
store.selectTable(tableId)
|
||||
}
|
||||
|
||||
function handleRefresh(): void {
|
||||
store.refreshRecords()
|
||||
}
|
||||
|
||||
async function handleTableCreated(tableId: string): Promise<void> {
|
||||
createModalOpen.value = false
|
||||
await store.loadTables()
|
||||
await store.selectTable(tableId)
|
||||
}
|
||||
|
||||
async function handleEditCell(payload: {
|
||||
recordId: string
|
||||
fieldId: string
|
||||
value: unknown
|
||||
}): Promise<void> {
|
||||
await store.updateCell(payload.recordId, payload.fieldId, payload.value)
|
||||
}
|
||||
|
||||
function handleSwitchView(viewId: string): void {
|
||||
store.switchView(viewId)
|
||||
}
|
||||
|
||||
async function handleCreateView(): Promise<void> {
|
||||
// ponytail: simple prompt for view name; full create modal is overkill for v1
|
||||
let name = ''
|
||||
AModal.confirm({
|
||||
title: '新建视图',
|
||||
content: h('input', {
|
||||
class: 'ant-input',
|
||||
placeholder: '请输入视图名称',
|
||||
onInput: (e: Event) => {
|
||||
name = (e.target as HTMLInputElement).value
|
||||
},
|
||||
}),
|
||||
onOk: async () => {
|
||||
if (!name.trim()) return
|
||||
await store.createView(name.trim(), 'grid')
|
||||
},
|
||||
})
|
||||
}
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.bitable-view {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
height: 100vh;
|
||||
width: 100vw;
|
||||
overflow: hidden;
|
||||
background: var(--bg-primary, #fff);
|
||||
}
|
||||
|
||||
.bitable-view__topbar {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 8px 16px;
|
||||
border-bottom: 1px solid var(--border-color, #f0f0f0);
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.bitable-view__topbar-left {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.bitable-view__topbar-right {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.bitable-view__title {
|
||||
font-weight: 600;
|
||||
font-size: 16px;
|
||||
}
|
||||
|
||||
.bitable-view__body {
|
||||
flex: 1;
|
||||
display: flex;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.bitable-view__sidebar {
|
||||
width: 240px;
|
||||
flex-shrink: 0;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.bitable-view__main {
|
||||
flex: 1;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow: hidden;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.bitable-view__placeholder {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
height: 100%;
|
||||
gap: 16px;
|
||||
color: var(--text-placeholder, #bfbfbf);
|
||||
}
|
||||
|
||||
.bitable-view__grid-header {
|
||||
display: flex;
|
||||
align-items: baseline;
|
||||
gap: 12px;
|
||||
padding: 12px 16px;
|
||||
border-bottom: 1px solid var(--border-color, #f0f0f0);
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.bitable-view__table-name {
|
||||
margin: 0;
|
||||
font-size: 16px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.bitable-view__field-count {
|
||||
font-size: 12px;
|
||||
color: var(--text-secondary, #8c8c8c);
|
||||
}
|
||||
|
||||
.bitable-view__grid-container {
|
||||
flex: 1;
|
||||
overflow: hidden;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.bitable-view__load-more {
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
padding: 8px;
|
||||
border-top: 1px solid var(--border-color, #f0f0f0);
|
||||
flex-shrink: 0;
|
||||
}
|
||||
</style>
|
||||
|
|
@ -0,0 +1,605 @@
|
|||
"""REST API routes for the bitable companion service.
|
||||
|
||||
All endpoints are prefixed ``/api/v1/bitable``. Auth via ``require_bitable_auth``
|
||||
which accepts either a user JWT (``Authorization: Bearer``) or an internal
|
||||
service token (``X-Internal-Token``) per KTD11. Service is obtained from
|
||||
``app.state.bitable_service`` (503 if not initialized).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hmac
|
||||
import logging
|
||||
import os
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from fastapi import (
|
||||
APIRouter,
|
||||
Depends,
|
||||
File,
|
||||
HTTPException,
|
||||
Query,
|
||||
Request,
|
||||
UploadFile,
|
||||
)
|
||||
from fastapi.responses import FileResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from agentkit.bitable.models import FieldOwner, FieldType, ViewType
|
||||
from agentkit.bitable.service import BitableService, FieldDependencyError
|
||||
from agentkit.server.auth.dependencies import get_current_user
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/bitable", tags=["bitable"])
|
||||
|
||||
# ── Upload config (U6) ───────────────────────────────────
|
||||
MAX_UPLOAD_SIZE = 10 * 1024 * 1024 # 10 MB
|
||||
BITABLE_UPLOAD_DIR = Path(os.environ.get("AGENTKIT_BITABLE_UPLOAD_DIR", "data/uploads/bitable"))
|
||||
_IMAGE_MIME_PREFIXES = ("image/",)
|
||||
|
||||
|
||||
def _get_service(request: Request) -> BitableService:
|
||||
"""Get bitable service from app.state, 503 if not initialized."""
|
||||
service = getattr(request.app.state, "bitable_service", None)
|
||||
if service is None:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="Bitable service not available. Server may not have initialized it.",
|
||||
)
|
||||
return service
|
||||
|
||||
|
||||
async def require_bitable_auth(request: Request) -> dict[str, Any]:
|
||||
"""Bitable-specific auth: accept JWT (via middleware) OR X-Internal-Token (KTD11).
|
||||
|
||||
The internal token is compared in constant time (hmac.compare_digest).
|
||||
On success with the internal token, a synthetic service user is returned.
|
||||
"""
|
||||
# 1. Check internal service token (KTD11)
|
||||
internal_token = getattr(request.app.state, "bitable_internal_token", None)
|
||||
if internal_token:
|
||||
provided = request.headers.get("X-Internal-Token", "")
|
||||
if provided and hmac.compare_digest(provided, internal_token):
|
||||
return {
|
||||
"user_id": "__bitable_internal__",
|
||||
"username": "bitable-internal",
|
||||
"role": "admin",
|
||||
"internal": True,
|
||||
}
|
||||
|
||||
# 2. Fall back to JWT auth
|
||||
user = await get_current_user(request)
|
||||
if user is None:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required (JWT or X-Internal-Token)",
|
||||
)
|
||||
return user
|
||||
|
||||
|
||||
async def _check_table_ownership(
|
||||
service: BitableService, table_id: str, user: dict[str, Any]
|
||||
) -> None:
|
||||
"""Verify the user owns the table. Internal service users bypass check.
|
||||
|
||||
Raises 404 if table not found, 403 if not owned.
|
||||
"""
|
||||
table = await service.get_table(table_id)
|
||||
if table is None:
|
||||
raise HTTPException(status_code=404, detail="Table not found")
|
||||
if user.get("internal"):
|
||||
return # Internal service token (KTD11) bypasses ownership
|
||||
if table.owner_user_id and table.owner_user_id != user.get("user_id"):
|
||||
raise HTTPException(status_code=403, detail="Not authorized to access this table")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Request models
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class CreateTableRequest(BaseModel):
|
||||
name: str
|
||||
description: str = ""
|
||||
primary_key_field_id: str | None = None
|
||||
|
||||
|
||||
class UpdateTableRequest(BaseModel):
|
||||
name: str | None = None
|
||||
description: str | None = None
|
||||
primary_key_field_id: str | None = None
|
||||
|
||||
|
||||
class CreateFieldRequest(BaseModel):
|
||||
name: str
|
||||
field_type: FieldType
|
||||
config: dict[str, Any] = Field(default_factory=dict)
|
||||
owner: FieldOwner = FieldOwner.user
|
||||
|
||||
|
||||
class UpdateFieldRequest(BaseModel):
|
||||
name: str | None = None
|
||||
config: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class CreateRecordRequest(BaseModel):
|
||||
values: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class BatchCreateRecordsRequest(BaseModel):
|
||||
records: list[dict[str, Any]] = Field(default_factory=list, max_length=500)
|
||||
|
||||
|
||||
class UpdateRecordRequest(BaseModel):
|
||||
values: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class UpsertRequest(BaseModel):
|
||||
records: list[dict[str, Any]]
|
||||
primary_key_field_id: str
|
||||
|
||||
|
||||
class CreateViewRequest(BaseModel):
|
||||
name: str
|
||||
view_type: ViewType = ViewType.grid
|
||||
config: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class UpdateViewRequest(BaseModel):
|
||||
name: str | None = None
|
||||
config: dict[str, Any] | None = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Table endpoints
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.post("/tables", status_code=201)
|
||||
async def create_table(
|
||||
body: CreateTableRequest,
|
||||
request: Request,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
table = await service.create_table(
|
||||
name=body.name,
|
||||
description=body.description,
|
||||
primary_key_field_id=body.primary_key_field_id,
|
||||
owner_user_id=user.get("user_id"),
|
||||
)
|
||||
return {"success": True, "table": table.model_dump(mode="json")}
|
||||
|
||||
|
||||
@router.get("/tables")
|
||||
async def list_tables(
|
||||
request: Request,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
tables = await service.list_tables(owner_user_id=user.get("user_id"))
|
||||
return {"success": True, "tables": [t.model_dump(mode="json") for t in tables]}
|
||||
|
||||
|
||||
@router.get("/tables/{table_id}")
|
||||
async def get_table(
|
||||
table_id: str,
|
||||
request: Request,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
await _check_table_ownership(service, table_id, user)
|
||||
table = await service.get_table(table_id)
|
||||
return {"success": True, "table": table.model_dump(mode="json")}
|
||||
|
||||
|
||||
@router.patch("/tables/{table_id}")
|
||||
async def update_table(
|
||||
table_id: str,
|
||||
body: UpdateTableRequest,
|
||||
request: Request,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
await _check_table_ownership(service, table_id, user)
|
||||
kwargs = body.model_dump(exclude_none=True)
|
||||
table = await service.update_table(table_id, **kwargs)
|
||||
if table is None:
|
||||
raise HTTPException(status_code=404, detail="Table not found")
|
||||
return {"success": True, "table": table.model_dump(mode="json")}
|
||||
|
||||
|
||||
@router.delete("/tables/{table_id}")
|
||||
async def delete_table(
|
||||
table_id: str,
|
||||
request: Request,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
await _check_table_ownership(service, table_id, user)
|
||||
deleted = await service.delete_table(table_id)
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail="Table not found")
|
||||
return {"success": True}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Field endpoints
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.post("/tables/{table_id}/fields", status_code=201)
|
||||
async def create_field(
|
||||
table_id: str,
|
||||
body: CreateFieldRequest,
|
||||
request: Request,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
await _check_table_ownership(service, table_id, user)
|
||||
field = await service.create_field(
|
||||
table_id=table_id,
|
||||
name=body.name,
|
||||
field_type=body.field_type,
|
||||
config=body.config,
|
||||
owner=body.owner,
|
||||
)
|
||||
return {"success": True, "field": field.model_dump(mode="json")}
|
||||
|
||||
|
||||
@router.get("/tables/{table_id}/fields")
|
||||
async def list_fields(
|
||||
table_id: str,
|
||||
request: Request,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
await _check_table_ownership(service, table_id, user)
|
||||
fields = await service.list_fields(table_id)
|
||||
return {"success": True, "fields": [f.model_dump(mode="json") for f in fields]}
|
||||
|
||||
|
||||
@router.patch("/fields/{field_id}")
|
||||
async def update_field(
|
||||
field_id: str,
|
||||
body: UpdateFieldRequest,
|
||||
request: Request,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
kwargs = body.model_dump(exclude_none=True)
|
||||
field = await service.update_field(field_id, **kwargs)
|
||||
if field is None:
|
||||
raise HTTPException(status_code=404, detail="Field not found")
|
||||
return {"success": True, "field": field.model_dump(mode="json")}
|
||||
|
||||
|
||||
@router.delete("/fields/{field_id}")
|
||||
async def delete_field(
|
||||
field_id: str,
|
||||
request: Request,
|
||||
force: bool = Query(False, description="Force delete with cascade cleanup"),
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
try:
|
||||
deleted = await service.delete_field(field_id, force=force)
|
||||
except FieldDependencyError as e:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={"message": str(e), "dependencies": e.dependencies},
|
||||
)
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail="Field not found")
|
||||
return {"success": True}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Formula validation endpoint (U5b)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class ValidateFormulaRequest(BaseModel):
|
||||
"""Request body for formula syntax validation."""
|
||||
|
||||
formula: str = Field(..., min_length=1, max_length=2000)
|
||||
|
||||
|
||||
@router.post("/fields/validate-formula")
|
||||
async def validate_formula(
|
||||
body: ValidateFormulaRequest,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
"""Validate formula syntax without saving. Returns valid flag + error detail."""
|
||||
from agentkit.bitable.formula import (
|
||||
FormulaParseError,
|
||||
FormulaSecurityError,
|
||||
UnknownFunctionError,
|
||||
parse_formula,
|
||||
)
|
||||
|
||||
try:
|
||||
parse_formula(body.formula)
|
||||
except (FormulaParseError, FormulaSecurityError, UnknownFunctionError) as e:
|
||||
return {"valid": False, "error": str(e)}
|
||||
except Exception as e: # pragma: no cover — defensive
|
||||
return {"valid": False, "error": f"Unexpected error: {e}"}
|
||||
return {"valid": True}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Record endpoints
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.post("/tables/{table_id}/records", status_code=201)
|
||||
async def create_records(
|
||||
table_id: str,
|
||||
body: BatchCreateRecordsRequest,
|
||||
request: Request,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
await _check_table_ownership(service, table_id, user)
|
||||
created = []
|
||||
for rec_values in body.records:
|
||||
record = await service.create_record(table_id, values=rec_values)
|
||||
created.append(record.model_dump(mode="json"))
|
||||
return {"success": True, "count": len(created), "records": created}
|
||||
|
||||
|
||||
@router.get("/tables/{table_id}/records")
|
||||
async def list_records(
|
||||
table_id: str,
|
||||
request: Request,
|
||||
cursor: str | None = Query(None),
|
||||
limit: int = Query(50, ge=1, le=200),
|
||||
filters: str | None = Query(None, description="JSON-encoded filter list"),
|
||||
sorts: str | None = Query(None, description="JSON-encoded sort list"),
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
import json
|
||||
|
||||
service = _get_service(request)
|
||||
await _check_table_ownership(service, table_id, user)
|
||||
|
||||
try:
|
||||
parsed_filters = json.loads(filters) if filters else None
|
||||
parsed_sorts = json.loads(sorts) if sorts else None
|
||||
except json.JSONDecodeError as e:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid JSON in filters/sorts: {e}")
|
||||
|
||||
if parsed_filters or parsed_sorts:
|
||||
records, next_cursor = await service.list_records_filtered(
|
||||
table_id,
|
||||
filters=parsed_filters,
|
||||
sorts=parsed_sorts,
|
||||
cursor=cursor,
|
||||
limit=limit,
|
||||
)
|
||||
else:
|
||||
records, next_cursor = await service.list_records(table_id, cursor=cursor, limit=limit)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"records": [r.model_dump(mode="json") for r in records],
|
||||
"next_cursor": next_cursor,
|
||||
}
|
||||
|
||||
|
||||
@router.patch("/records/{record_id}")
|
||||
async def update_record(
|
||||
record_id: str,
|
||||
body: UpdateRecordRequest,
|
||||
request: Request,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
record = await service.update_record_values(record_id, body.values)
|
||||
if record is None:
|
||||
raise HTTPException(status_code=404, detail="Record not found")
|
||||
return {"success": True, "record": record.model_dump(mode="json")}
|
||||
|
||||
|
||||
@router.delete("/tables/{table_id}/records")
|
||||
async def delete_records(
|
||||
table_id: str,
|
||||
request: Request,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
await _check_table_ownership(service, table_id, user)
|
||||
count = await service.delete_records_by_table(table_id)
|
||||
return {"success": True, "deleted": count}
|
||||
|
||||
|
||||
@router.delete("/records/{record_id}")
|
||||
async def delete_single_record(
|
||||
record_id: str,
|
||||
request: Request,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
deleted = await service.delete_record(record_id)
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail="Record not found")
|
||||
return {"success": True}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Upsert endpoint (KTD8)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.post("/tables/{table_id}/upsert", status_code=201)
|
||||
async def upsert_records(
|
||||
table_id: str,
|
||||
body: UpsertRequest,
|
||||
request: Request,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
await _check_table_ownership(service, table_id, user)
|
||||
if not body.primary_key_field_id:
|
||||
raise HTTPException(status_code=400, detail="primary_key_field_id is required")
|
||||
result = await service.upsert_records(table_id, body.records, body.primary_key_field_id)
|
||||
return {"success": True, **result}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# View endpoints
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.post("/tables/{table_id}/views", status_code=201)
|
||||
async def create_view(
|
||||
table_id: str,
|
||||
body: CreateViewRequest,
|
||||
request: Request,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
await _check_table_ownership(service, table_id, user)
|
||||
view = await service.create_view(
|
||||
table_id=table_id,
|
||||
name=body.name,
|
||||
view_type=body.view_type,
|
||||
config=body.config,
|
||||
)
|
||||
return {"success": True, "view": view.model_dump(mode="json")}
|
||||
|
||||
|
||||
@router.get("/tables/{table_id}/views")
|
||||
async def list_views(
|
||||
table_id: str,
|
||||
request: Request,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
await _check_table_ownership(service, table_id, user)
|
||||
views = await service.list_views(table_id)
|
||||
return {"success": True, "views": [v.model_dump(mode="json") for v in views]}
|
||||
|
||||
|
||||
@router.patch("/views/{view_id}")
|
||||
async def update_view(
|
||||
view_id: str,
|
||||
body: UpdateViewRequest,
|
||||
request: Request,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
service = _get_service(request)
|
||||
kwargs = body.model_dump(exclude_none=True)
|
||||
view = await service.update_view(view_id, **kwargs)
|
||||
if view is None:
|
||||
raise HTTPException(status_code=404, detail="View not found")
|
||||
return {"success": True, "view": view.model_dump(mode="json")}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# File upload / download (U6: attachment & image fields)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _sanitize_filename(name: str) -> str:
|
||||
"""Remove path separators and keep only safe characters."""
|
||||
name = name.replace("\\", "_").replace("/", "_")
|
||||
return "".join(c for c in name if c.isalnum() or c in "._-").strip(".")
|
||||
|
||||
|
||||
def _ensure_upload_dir() -> Path:
|
||||
BITABLE_UPLOAD_DIR.mkdir(parents=True, exist_ok=True)
|
||||
return BITABLE_UPLOAD_DIR
|
||||
|
||||
|
||||
@router.post("/tables/{table_id}/upload")
|
||||
async def upload_file(
|
||||
table_id: str,
|
||||
request: Request,
|
||||
file: UploadFile = File(...),
|
||||
field_id: str = Query(..., description="Target field ID (determines type validation)"),
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> dict[str, Any]:
|
||||
"""Upload a file for an attachment or image field.
|
||||
|
||||
Returns file metadata. The frontend then writes this metadata array
|
||||
to the record's field value via the normal update-record endpoint.
|
||||
"""
|
||||
service = _get_service(request)
|
||||
await _check_table_ownership(service, table_id, user)
|
||||
|
||||
# Validate field exists and is attachment/image type
|
||||
field = await service.get_field(field_id)
|
||||
if field is None:
|
||||
raise HTTPException(status_code=404, detail="Field not found")
|
||||
if field.field_type not in (FieldType.attachment, FieldType.image):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Field '{field.name}' is not an attachment or image field",
|
||||
)
|
||||
|
||||
# Size check (Content-Length header, fast reject)
|
||||
if file.size is not None and file.size > MAX_UPLOAD_SIZE:
|
||||
raise HTTPException(status_code=413, detail="File exceeds 10 MB limit")
|
||||
|
||||
# Image type check
|
||||
mime = file.content_type or "application/octet-stream"
|
||||
if field.field_type == FieldType.image and not mime.startswith(_IMAGE_MIME_PREFIXES):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Image field requires an image file, got '{mime}'",
|
||||
)
|
||||
|
||||
original_name = file.filename or "unnamed"
|
||||
safe_name = _sanitize_filename(original_name) or "unnamed"
|
||||
ext = Path(safe_name).suffix
|
||||
stored_name = f"{uuid.uuid4().hex}{ext}"
|
||||
upload_dir = _ensure_upload_dir()
|
||||
file_path = upload_dir / stored_name
|
||||
|
||||
# Stream-read with size check to prevent OOM (P1 #12)
|
||||
total_size = 0
|
||||
try:
|
||||
with open(file_path, "wb") as f:
|
||||
while True:
|
||||
chunk = await file.read(64 * 1024) # 64KB chunks
|
||||
if not chunk:
|
||||
break
|
||||
total_size += len(chunk)
|
||||
if total_size > MAX_UPLOAD_SIZE:
|
||||
f.close()
|
||||
file_path.unlink(missing_ok=True)
|
||||
raise HTTPException(status_code=413, detail="File exceeds 10 MB limit")
|
||||
f.write(chunk)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as exc:
|
||||
file_path.unlink(missing_ok=True)
|
||||
logger.error(f"Failed to save uploaded bitable file: {exc}")
|
||||
raise HTTPException(status_code=500, detail="Failed to save file") from exc
|
||||
finally:
|
||||
await file.close()
|
||||
|
||||
return {
|
||||
"filename": original_name,
|
||||
"stored_name": stored_name,
|
||||
"mime_type": mime,
|
||||
"size": total_size,
|
||||
"url": f"/api/v1/bitable/files/{stored_name}",
|
||||
}
|
||||
|
||||
|
||||
@router.get("/files/{filename}")
|
||||
async def download_file(
|
||||
filename: str,
|
||||
user: dict = Depends(require_bitable_auth),
|
||||
) -> FileResponse:
|
||||
"""Download a bitable attachment/image file by its stored filename."""
|
||||
safe_filename = _sanitize_filename(filename)
|
||||
file_path = BITABLE_UPLOAD_DIR / safe_filename
|
||||
if not file_path.exists() or not file_path.is_file():
|
||||
raise HTTPException(status_code=404, detail="File not found")
|
||||
return FileResponse(file_path, filename=safe_filename)
|
||||
|
|
@ -1,5 +1,6 @@
|
|||
"""Task submission routes"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import uuid
|
||||
from dataclasses import asdict
|
||||
|
|
@ -210,11 +211,17 @@ async def cancel_task(task_id: str, req: Request):
|
|||
|
||||
|
||||
@router.post("/tasks/{task_id}/resume")
|
||||
async def resume_task(task_id: str, req: Request):
|
||||
async def resume_task(task_id: str, req: Request, plan_id: str | None = None):
|
||||
"""Resume a crashed pipeline from the last completed phase checkpoint.
|
||||
|
||||
Reconstructs the team from the saved plan's expert names, creates a new
|
||||
TeamOrchestrator with the checkpoint manager, and calls resume().
|
||||
|
||||
Args:
|
||||
task_id: Task ID from the URL path.
|
||||
plan_id: Optional plan ID. If not provided, falls back to task_id.
|
||||
Needed because TeamPlan.id is auto-generated and may differ
|
||||
from the task_id.
|
||||
"""
|
||||
from agentkit.experts.orchestrator import TeamOrchestrator
|
||||
from agentkit.experts.router import ExpertTeamRouter
|
||||
|
|
@ -223,72 +230,89 @@ async def resume_task(task_id: str, req: Request):
|
|||
|
||||
app_state = req.app.state
|
||||
|
||||
# 1. Create checkpoint manager
|
||||
checkpoint = PipelineCheckpoint(
|
||||
redis_client=getattr(app_state, "working_redis_client", None)
|
||||
)
|
||||
# Resolve plan_id: explicit param > task_id fallback
|
||||
resolved_plan_id = plan_id or task_id
|
||||
|
||||
# 2. Load plan to get expert names
|
||||
plan_dict = await checkpoint.load_plan(task_id)
|
||||
if plan_dict is None:
|
||||
# P2 #10: 并发 resume 防护 — 同一 plan_id 的并发 resume 请求只允许一个执行
|
||||
lock_attr = f"_resume_lock_{resolved_plan_id}"
|
||||
lock = getattr(app_state, lock_attr, None)
|
||||
if lock is None:
|
||||
lock = asyncio.Lock()
|
||||
setattr(app_state, lock_attr, lock)
|
||||
if lock.locked():
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"No checkpoint found for task '{task_id}'",
|
||||
status_code=409,
|
||||
detail=f"Resume already in progress for plan '{resolved_plan_id}'",
|
||||
)
|
||||
async with lock:
|
||||
# 1. Create checkpoint manager
|
||||
checkpoint = PipelineCheckpoint(
|
||||
redis_client=getattr(app_state, "working_redis_client", None)
|
||||
)
|
||||
|
||||
# 3. Extract unique expert names from plan
|
||||
expert_names: list[str] = []
|
||||
lead_name = plan_dict.get("lead_expert", "")
|
||||
if lead_name:
|
||||
expert_names.append(lead_name)
|
||||
for ph in plan_dict.get("phases", []):
|
||||
name = ph.get("assigned_expert", "")
|
||||
if name and name not in expert_names:
|
||||
expert_names.append(name)
|
||||
# 2. Load plan to get expert names
|
||||
plan_dict = await checkpoint.load_plan(resolved_plan_id)
|
||||
if plan_dict is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"No checkpoint found for plan '{resolved_plan_id}'"
|
||||
f" (task_id='{task_id}')",
|
||||
)
|
||||
|
||||
if not expert_names:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="Cannot resume: no experts found in saved plan",
|
||||
# 3. Extract unique expert names from plan
|
||||
expert_names: list[str] = []
|
||||
lead_name = plan_dict.get("lead_expert", "")
|
||||
if lead_name:
|
||||
expert_names.append(lead_name)
|
||||
for ph in plan_dict.get("phases", []):
|
||||
name = ph.get("assigned_expert", "")
|
||||
if name and name not in expert_names:
|
||||
expert_names.append(name)
|
||||
|
||||
if not expert_names:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="Cannot resume: no experts found in saved plan",
|
||||
)
|
||||
|
||||
# 4. Resolve expert configs via ExpertTeamRouter
|
||||
template_registry = getattr(app_state, "expert_template_registry", None)
|
||||
if template_registry is None:
|
||||
from agentkit.experts.registry import ExpertTemplateRegistry
|
||||
|
||||
template_registry = ExpertTemplateRegistry()
|
||||
|
||||
team_router = ExpertTeamRouter(template_registry=template_registry)
|
||||
expert_configs = team_router.resolve_expert_configs(expert_names)
|
||||
if not expert_configs:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="Cannot resume: failed to resolve expert configs",
|
||||
)
|
||||
|
||||
lead_config = expert_configs[0]
|
||||
member_configs = expert_configs[1:] if len(expert_configs) > 1 else []
|
||||
|
||||
# 5. Create team + orchestrator
|
||||
team = ExpertTeam(
|
||||
pool=app_state.agent_pool,
|
||||
template_registry=template_registry,
|
||||
redis_client=getattr(app_state, "working_redis_client", None),
|
||||
)
|
||||
await team.create_team(lead_config=lead_config, member_configs=member_configs)
|
||||
|
||||
# 4. Resolve expert configs via ExpertTeamRouter
|
||||
template_registry = getattr(app_state, "expert_template_registry", None)
|
||||
if template_registry is None:
|
||||
from agentkit.experts.registry import ExpertTemplateRegistry
|
||||
|
||||
template_registry = ExpertTemplateRegistry()
|
||||
|
||||
team_router = ExpertTeamRouter(template_registry=template_registry)
|
||||
expert_configs = team_router.resolve_expert_configs(expert_names)
|
||||
if not expert_configs:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="Cannot resume: failed to resolve expert configs",
|
||||
)
|
||||
|
||||
lead_config = expert_configs[0]
|
||||
member_configs = expert_configs[1:] if len(expert_configs) > 1 else []
|
||||
|
||||
# 5. Create team + orchestrator
|
||||
team = ExpertTeam(
|
||||
pool=app_state.agent_pool,
|
||||
template_registry=template_registry,
|
||||
redis_client=getattr(app_state, "working_redis_client", None),
|
||||
)
|
||||
await team.create_team(lead_config=lead_config, member_configs=member_configs)
|
||||
|
||||
try:
|
||||
orchestrator = TeamOrchestrator(team=team, checkpoint=checkpoint)
|
||||
result = await orchestrator.resume(task_id)
|
||||
finally:
|
||||
try:
|
||||
await team.dissolve()
|
||||
except Exception:
|
||||
pass
|
||||
orchestrator = TeamOrchestrator(team=team, checkpoint=checkpoint)
|
||||
result = await orchestrator.resume(resolved_plan_id)
|
||||
finally:
|
||||
try:
|
||||
await team.dissolve()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return {
|
||||
"task_id": task_id,
|
||||
"plan_id": resolved_plan_id,
|
||||
"status": result.get("status", "unknown"),
|
||||
"result": result.get("result"),
|
||||
"phase_results": {
|
||||
|
|
|
|||
|
|
@ -81,7 +81,7 @@ class SkillConfig(AgentConfig):
|
|||
evolution: dict[str, Any] | None = None,
|
||||
# v3 新增字段：SKILL.md 支持
|
||||
skill_md_path: str | None = None,
|
||||
disclosure_level: int = 0,
|
||||
disclosure_level: int = 1, # 默认全量加载，向后兼容；0=概要模式需显式指定
|
||||
# v4 新增字段：依赖声明、能力标签
|
||||
dependencies: list[dict[str, Any] | DependencyDecl] | None = None,
|
||||
capabilities: list[str | dict[str, Any] | CapabilityTag] | None = None,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,486 @@
|
|||
"""BitableTool — Agent tool for bitable data ingestion and CRUD via HTTP.
|
||||
|
||||
Implements KTD5 (REST API boundary even when co-deployed) and KTD11
|
||||
(internal service token auth). The tool uses ``httpx.AsyncClient`` to call
|
||||
the bitable REST API; it never imports BitableService directly.
|
||||
|
||||
Actions: create_table, import_excel, import_database, collect_api,
|
||||
upsert_records, query_records.
|
||||
|
||||
Batch chunking: upsert and import operations send at most ``BATCH_SIZE``
|
||||
records per HTTP request. On partial failure, the result includes
|
||||
``successful_count`` and ``resume_from`` for breakpoint continuation.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from agentkit.bitable.ingestion.excel import ParsedSheet, parse_excel, parse_excel_url
|
||||
from agentkit.bitable.ingestion.database import import_table as import_db_table
|
||||
from agentkit.bitable.ingestion.api_collector import transform_records
|
||||
from agentkit.tools.base import Tool
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
BATCH_SIZE = 500
|
||||
|
||||
|
||||
class BitableTool(Tool):
|
||||
"""Agent tool for bitable operations via REST API.
|
||||
|
||||
Args:
|
||||
base_url: Bitable API base URL (e.g. ``http://localhost:8001/api/v1/bitable``).
|
||||
internal_token: Service token for KTD11 auth. If ``None``, requests
|
||||
go unauthenticated (will fail if the server requires auth).
|
||||
"""
|
||||
|
||||
def __init__(self, base_url: str, internal_token: str | None = None) -> None:
|
||||
super().__init__(
|
||||
name="bitable",
|
||||
description=(
|
||||
"Create and manage bitable (multi-dimensional spreadsheet) tables, "
|
||||
"ingest data from Excel files, databases, or API responses, and "
|
||||
"query records. Actions: create_table, import_excel, "
|
||||
"import_database, collect_api, upsert_records, query_records."
|
||||
),
|
||||
input_schema={
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"action": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"create_table",
|
||||
"import_excel",
|
||||
"import_database",
|
||||
"collect_api",
|
||||
"upsert_records",
|
||||
"query_records",
|
||||
],
|
||||
"description": "Bitable operation to perform.",
|
||||
},
|
||||
"table_name": {
|
||||
"type": "string",
|
||||
"description": "Name for the new bitable table (create_table, import_excel, import_database).",
|
||||
},
|
||||
"description": {
|
||||
"type": "string",
|
||||
"description": "Table description (create_table).",
|
||||
},
|
||||
"file_path": {
|
||||
"type": "string",
|
||||
"description": "Path to .xlsx file (import_excel).",
|
||||
},
|
||||
"file_url": {
|
||||
"type": "string",
|
||||
"description": "URL to download .xlsx file (import_excel).",
|
||||
},
|
||||
"connection_string": {
|
||||
"type": "string",
|
||||
"description": "Database connection string (import_database).",
|
||||
},
|
||||
"table_names": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"description": "Source table names to import (import_database).",
|
||||
},
|
||||
"table_id": {
|
||||
"type": "string",
|
||||
"description": "Target bitable table ID (collect_api, upsert_records, query_records).",
|
||||
},
|
||||
"records": {
|
||||
"type": "array",
|
||||
"description": "Records to write (collect_api, upsert_records).",
|
||||
},
|
||||
"field_mapping": {
|
||||
"type": "object",
|
||||
"description": "Mapping {source_key: bitable_field_id} (collect_api).",
|
||||
},
|
||||
"primary_key_field_id": {
|
||||
"type": "string",
|
||||
"description": "Field ID of the primary key (upsert_records, collect_api).",
|
||||
},
|
||||
"resume_from": {
|
||||
"type": "integer",
|
||||
"description": "Skip this many records before resuming a failed batch (upsert_records, collect_api).",
|
||||
},
|
||||
"cursor": {
|
||||
"type": "string",
|
||||
"description": "Pagination cursor (query_records).",
|
||||
},
|
||||
"limit": {
|
||||
"type": "integer",
|
||||
"description": "Max records to return (query_records).",
|
||||
},
|
||||
},
|
||||
"required": ["action"],
|
||||
},
|
||||
)
|
||||
self._base_url = base_url.rstrip("/")
|
||||
self._internal_token = internal_token
|
||||
self._client: httpx.AsyncClient | None = None
|
||||
|
||||
async def _get_client(self) -> httpx.AsyncClient:
|
||||
if self._client is None or self._client.is_closed:
|
||||
headers: dict[str, str] = {}
|
||||
if self._internal_token:
|
||||
headers["X-Internal-Token"] = self._internal_token
|
||||
self._client = httpx.AsyncClient(
|
||||
base_url=self._base_url,
|
||||
headers=headers,
|
||||
timeout=60.0,
|
||||
)
|
||||
return self._client
|
||||
|
||||
async def close(self) -> None:
|
||||
if self._client is not None and not self._client.is_closed:
|
||||
await self._client.aclose()
|
||||
|
||||
async def execute(self, **kwargs) -> dict[str, Any]:
|
||||
action = kwargs.get("action")
|
||||
handlers = {
|
||||
"create_table": self._create_table,
|
||||
"import_excel": self._import_excel,
|
||||
"import_database": self._import_database,
|
||||
"collect_api": self._collect_api,
|
||||
"upsert_records": self._upsert_records,
|
||||
"query_records": self._query_records,
|
||||
}
|
||||
handler = handlers.get(action)
|
||||
if handler is None:
|
||||
return {"success": False, "error": f"Unknown action: {action!r}"}
|
||||
try:
|
||||
return await handler(**kwargs)
|
||||
except httpx.HTTPStatusError as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": f"Bitable API error {e.response.status_code}: {e.response.text[:500]}",
|
||||
}
|
||||
except httpx.ConnectError as e:
|
||||
return {"success": False, "error": f"Cannot connect to bitable API: {e}"}
|
||||
except Exception as e:
|
||||
return {"success": False, "error": f"{action} failed: {e}"}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# create_table
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _create_table(self, **kwargs) -> dict[str, Any]:
|
||||
table_name = kwargs.get("table_name")
|
||||
if not table_name:
|
||||
return {"success": False, "error": "Missing required field: table_name"}
|
||||
client = await self._get_client()
|
||||
resp = await client.post(
|
||||
"/tables",
|
||||
json={"name": table_name, "description": kwargs.get("description", "")},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return {"success": True, "table": data["table"]}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# import_excel
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _import_excel(self, **kwargs) -> dict[str, Any]:
|
||||
file_path = kwargs.get("file_path")
|
||||
file_url = kwargs.get("file_url")
|
||||
if not file_path and not file_url:
|
||||
return {"success": False, "error": "Either file_path or file_url is required"}
|
||||
|
||||
# Parse Excel — offload sync I/O to thread pool (P2 #21-23).
|
||||
if file_path:
|
||||
sheets = await asyncio.to_thread(parse_excel, file_path)
|
||||
else:
|
||||
sheets = await asyncio.to_thread(parse_excel_url, file_url)
|
||||
|
||||
if not sheets:
|
||||
return {"success": False, "error": "Excel file has no sheets with data"}
|
||||
|
||||
results: list[dict[str, Any]] = []
|
||||
for sheet in sheets:
|
||||
result = await self._import_sheet(sheet)
|
||||
results.append(result)
|
||||
return {"success": True, "sheets": results}
|
||||
|
||||
async def _import_sheet(self, sheet: ParsedSheet) -> dict[str, Any]:
|
||||
"""Create a bitable table from a parsed sheet and upsert all rows."""
|
||||
client = await self._get_client()
|
||||
|
||||
# 1. Create table
|
||||
resp = await client.post("/tables", json={"name": sheet.name})
|
||||
resp.raise_for_status()
|
||||
table_id = resp.json()["table"]["id"]
|
||||
|
||||
# 2. Create fields
|
||||
field_name_to_id: dict[str, str] = {}
|
||||
for col_name, field_type in zip(sheet.columns, sheet.field_types):
|
||||
resp = await client.post(
|
||||
f"/tables/{table_id}/fields",
|
||||
json={"name": col_name, "field_type": field_type, "owner": "agent"},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
field_id = resp.json()["field"]["id"]
|
||||
field_name_to_id[col_name] = field_id
|
||||
|
||||
# 3. Map record keys to field IDs and batch upsert
|
||||
mapped_records = [
|
||||
{field_name_to_id[k]: v for k, v in rec.items() if k in field_name_to_id}
|
||||
for rec in sheet.records
|
||||
]
|
||||
|
||||
if not mapped_records:
|
||||
return {
|
||||
"table_id": table_id,
|
||||
"table_name": sheet.name,
|
||||
"field_count": len(field_name_to_id),
|
||||
"record_count": 0,
|
||||
}
|
||||
|
||||
# Use first field as PK fallback (import_excel doesn't require a PK)
|
||||
# If no PK is set, upsert won't work — use create_records instead
|
||||
upsert_result = await self._batch_create_records(table_id, mapped_records)
|
||||
return {
|
||||
"table_id": table_id,
|
||||
"table_name": sheet.name,
|
||||
"field_count": len(field_name_to_id),
|
||||
"record_count": upsert_result["successful_count"],
|
||||
**upsert_result,
|
||||
}
|
||||
|
||||
async def _batch_create_records(
|
||||
self, table_id: str, records: list[dict[str, Any]]
|
||||
) -> dict[str, Any]:
|
||||
"""Create records in batches via POST /tables/{id}/records."""
|
||||
client = await self._get_client()
|
||||
total = len(records)
|
||||
successful = 0
|
||||
errors: list[dict[str, Any]] = []
|
||||
|
||||
for start in range(0, total, BATCH_SIZE):
|
||||
batch = records[start : start + BATCH_SIZE]
|
||||
try:
|
||||
resp = await client.post(
|
||||
f"/tables/{table_id}/records",
|
||||
json={"records": batch},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
successful += len(batch)
|
||||
except httpx.HTTPStatusError as e:
|
||||
errors.append(
|
||||
{
|
||||
"batch_start": start,
|
||||
"batch_size": len(batch),
|
||||
"status": e.response.status_code,
|
||||
"error": e.response.text[:300],
|
||||
}
|
||||
)
|
||||
break # stop on first failure
|
||||
|
||||
return {
|
||||
"successful_count": successful,
|
||||
"total": total,
|
||||
"resume_from": successful,
|
||||
**({"errors": errors} if errors else {}),
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# import_database
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _import_database(self, **kwargs) -> dict[str, Any]:
|
||||
conn_str = kwargs.get("connection_string")
|
||||
table_names = kwargs.get("table_names")
|
||||
if not conn_str:
|
||||
return {"success": False, "error": "Missing required field: connection_string"}
|
||||
if not table_names:
|
||||
return {"success": False, "error": "Missing required field: table_names"}
|
||||
|
||||
results: list[dict[str, Any]] = []
|
||||
for src_table in table_names:
|
||||
try:
|
||||
# Offload sync DB reflection to thread pool (P2 #21-23).
|
||||
reflected = await asyncio.to_thread(import_db_table, conn_str, src_table)
|
||||
result = await self._import_reflected_table(reflected)
|
||||
results.append(result)
|
||||
except ConnectionError as e:
|
||||
return {"success": False, "error": str(e), "imported": results}
|
||||
except Exception as e:
|
||||
results.append({"table_name": src_table, "success": False, "error": str(e)})
|
||||
return {"success": True, "tables": results}
|
||||
|
||||
async def _import_reflected_table(self, reflected: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Create a bitable table from reflected DB data and upsert rows."""
|
||||
client = await self._get_client()
|
||||
table_name = reflected["table_name"]
|
||||
|
||||
# 1. Create table
|
||||
resp = await client.post("/tables", json={"name": table_name})
|
||||
resp.raise_for_status()
|
||||
table_id = resp.json()["table"]["id"]
|
||||
|
||||
# 2. Create fields
|
||||
field_name_to_id: dict[str, str] = {}
|
||||
pk_field_id: str | None = None
|
||||
for fdef in reflected["fields"]:
|
||||
resp = await client.post(
|
||||
f"/tables/{table_id}/fields",
|
||||
json={
|
||||
"name": fdef["name"],
|
||||
"field_type": fdef["field_type"],
|
||||
"owner": "agent",
|
||||
},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
fid = resp.json()["field"]["id"]
|
||||
field_name_to_id[fdef["name"]] = fid
|
||||
if fdef.get("is_primary_key"):
|
||||
pk_field_id = fid
|
||||
|
||||
# 3. Set primary key
|
||||
if pk_field_id:
|
||||
await client.patch("/tables/" + table_id, json={"primary_key_field_id": pk_field_id})
|
||||
|
||||
# 4. Map and upsert records
|
||||
mapped = [
|
||||
{field_name_to_id[k]: v for k, v in rec.items() if k in field_name_to_id}
|
||||
for rec in reflected["records"]
|
||||
]
|
||||
|
||||
if not mapped:
|
||||
return {
|
||||
"table_id": table_id,
|
||||
"table_name": table_name,
|
||||
"record_count": 0,
|
||||
"success": True,
|
||||
}
|
||||
|
||||
if pk_field_id:
|
||||
upsert = await self._batch_upsert(table_id, mapped, pk_field_id)
|
||||
else:
|
||||
upsert = await self._batch_create_records(table_id, mapped)
|
||||
|
||||
return {
|
||||
"table_id": table_id,
|
||||
"table_name": table_name,
|
||||
"record_count": upsert["successful_count"],
|
||||
"success": True,
|
||||
**upsert,
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# collect_api
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _collect_api(self, **kwargs) -> dict[str, Any]:
|
||||
table_id = kwargs.get("table_id")
|
||||
records = kwargs.get("records")
|
||||
field_mapping = kwargs.get("field_mapping")
|
||||
pk_field_id = kwargs.get("primary_key_field_id")
|
||||
resume_from = kwargs.get("resume_from", 0)
|
||||
|
||||
if not table_id:
|
||||
return {"success": False, "error": "Missing required field: table_id"}
|
||||
if not records:
|
||||
return {"success": False, "error": "Missing required field: records"}
|
||||
if not field_mapping:
|
||||
return {"success": False, "error": "Missing required field: field_mapping"}
|
||||
if not pk_field_id:
|
||||
return {"success": False, "error": "Missing required field: primary_key_field_id"}
|
||||
|
||||
transformed = transform_records(records, field_mapping)
|
||||
if resume_from > 0:
|
||||
transformed = transformed[resume_from:]
|
||||
|
||||
result = await self._batch_upsert(table_id, transformed, pk_field_id)
|
||||
return {"success": True, **result}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# upsert_records
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _upsert_records(self, **kwargs) -> dict[str, Any]:
|
||||
table_id = kwargs.get("table_id")
|
||||
records = kwargs.get("records")
|
||||
pk_field_id = kwargs.get("primary_key_field_id")
|
||||
resume_from = kwargs.get("resume_from", 0)
|
||||
|
||||
if not table_id:
|
||||
return {"success": False, "error": "Missing required field: table_id"}
|
||||
if not records:
|
||||
return {"success": False, "error": "Missing required field: records"}
|
||||
if not pk_field_id:
|
||||
return {"success": False, "error": "Missing required field: primary_key_field_id"}
|
||||
|
||||
batch = records[resume_from:] if resume_from > 0 else records
|
||||
result = await self._batch_upsert(table_id, batch, pk_field_id)
|
||||
return {"success": True, **result}
|
||||
|
||||
async def _batch_upsert(
|
||||
self, table_id: str, records: list[dict[str, Any]], pk_field_id: str
|
||||
) -> dict[str, Any]:
|
||||
"""Upsert records in batches of BATCH_SIZE via POST /tables/{id}/upsert."""
|
||||
client = await self._get_client()
|
||||
total = len(records)
|
||||
successful = 0
|
||||
errors: list[dict[str, Any]] = []
|
||||
|
||||
for start in range(0, total, BATCH_SIZE):
|
||||
batch = records[start : start + BATCH_SIZE]
|
||||
try:
|
||||
resp = await client.post(
|
||||
f"/tables/{table_id}/upsert",
|
||||
json={
|
||||
"records": batch,
|
||||
"primary_key_field_id": pk_field_id,
|
||||
},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
successful += data.get("inserted", 0) + data.get("updated", 0)
|
||||
except httpx.HTTPStatusError as e:
|
||||
errors.append(
|
||||
{
|
||||
"batch_start": start,
|
||||
"batch_size": len(batch),
|
||||
"status": e.response.status_code,
|
||||
"error": e.response.text[:300],
|
||||
}
|
||||
)
|
||||
break
|
||||
|
||||
return {
|
||||
"successful_count": successful,
|
||||
"total": total,
|
||||
"resume_from": successful,
|
||||
**({"errors": errors} if errors else {}),
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# query_records
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _query_records(self, **kwargs) -> dict[str, Any]:
|
||||
table_id = kwargs.get("table_id")
|
||||
if not table_id:
|
||||
return {"success": False, "error": "Missing required field: table_id"}
|
||||
|
||||
client = await self._get_client()
|
||||
params: dict[str, Any] = {}
|
||||
if kwargs.get("cursor"):
|
||||
params["cursor"] = kwargs["cursor"]
|
||||
if kwargs.get("limit"):
|
||||
params["limit"] = kwargs["limit"]
|
||||
|
||||
resp = await client.get(f"/tables/{table_id}/records", params=params)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return {
|
||||
"success": True,
|
||||
"records": data["records"],
|
||||
"next_cursor": data.get("next_cursor"),
|
||||
}
|
||||
|
|
@ -0,0 +1,143 @@
|
|||
"""Shared fixtures for bitable unit tests.
|
||||
|
||||
Provides:
|
||||
- ``bitable_db``: initialized BitableDB (skips if PG unavailable)
|
||||
- ``bitable_service``: BitableService backed by bitable_db
|
||||
- Factory functions: make_table, make_field, make_record, make_formula_field
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
from agentkit.bitable.models import FieldOwner, FieldType
|
||||
|
||||
|
||||
def _pg_available() -> bool:
|
||||
"""Check if PostgreSQL is reachable via DATABASE_URL env var."""
|
||||
url = os.environ.get("DATABASE_URL") or os.environ.get("AGENTKIT_DATABASE_URL")
|
||||
return bool(url)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def bitable_db():
|
||||
"""Initialize a fresh bitable DB for each test (skips if PG unavailable)."""
|
||||
if not _pg_available():
|
||||
pytest.skip("PostgreSQL not available (set DATABASE_URL)")
|
||||
|
||||
from agentkit.bitable.db import BitableDB
|
||||
|
||||
db = BitableDB()
|
||||
try:
|
||||
await db.init()
|
||||
# Clean slate: drop and recreate bitable schema
|
||||
from sqlalchemy import text
|
||||
|
||||
async with db.engine.begin() as conn:
|
||||
await conn.execute(text("DROP SCHEMA IF EXISTS bitable CASCADE"))
|
||||
await db.init() # re-create fresh
|
||||
yield db
|
||||
finally:
|
||||
# Cleanup
|
||||
from sqlalchemy import text
|
||||
|
||||
if db.engine is not None:
|
||||
async with db.engine.begin() as conn:
|
||||
await conn.execute(text("DROP SCHEMA IF EXISTS bitable CASCADE"))
|
||||
await db.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def bitable_service(bitable_db):
|
||||
"""BitableService backed by the test bitable_db."""
|
||||
from agentkit.bitable.service import BitableService
|
||||
|
||||
yield BitableService(bitable_db)
|
||||
|
||||
|
||||
# ── Factory fixtures ───────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def make_table(bitable_service):
|
||||
"""Factory: create a table and return it."""
|
||||
|
||||
counter = [0]
|
||||
|
||||
async def _make(
|
||||
name: str | None = None,
|
||||
description: str = "",
|
||||
primary_key_field_id: str | None = None,
|
||||
):
|
||||
counter[0] += 1
|
||||
return await bitable_service.create_table(
|
||||
name=name or f"test_table_{counter[0]}",
|
||||
description=description,
|
||||
primary_key_field_id=primary_key_field_id,
|
||||
)
|
||||
|
||||
return _make
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def make_field(bitable_service):
|
||||
"""Factory: create a field and return it."""
|
||||
|
||||
counter = [0]
|
||||
|
||||
async def _make(
|
||||
table_id: str,
|
||||
name: str | None = None,
|
||||
field_type: FieldType = FieldType.text,
|
||||
config: dict | None = None,
|
||||
owner: FieldOwner = FieldOwner.agent,
|
||||
):
|
||||
counter[0] += 1
|
||||
return await bitable_service.create_field(
|
||||
table_id=table_id,
|
||||
name=name or f"field_{counter[0]}",
|
||||
field_type=field_type,
|
||||
config=config or {},
|
||||
owner=owner,
|
||||
)
|
||||
|
||||
return _make
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def make_record(bitable_service):
|
||||
"""Factory: create a record and return it."""
|
||||
counter = [0]
|
||||
|
||||
async def _make(table_id: str, values: dict | None = None):
|
||||
counter[0] += 1
|
||||
return await bitable_service.create_record(
|
||||
table_id=table_id,
|
||||
values=values or {},
|
||||
)
|
||||
|
||||
return _make
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def make_formula_field(bitable_service):
|
||||
"""Factory: create a formula field and return it."""
|
||||
counter = [0]
|
||||
|
||||
async def _make(
|
||||
table_id: str,
|
||||
name: str | None = None,
|
||||
formula_expr: str = "=1+1",
|
||||
):
|
||||
counter[0] += 1
|
||||
return await bitable_service.create_field(
|
||||
table_id=table_id,
|
||||
name=name or f"calc_{counter[0]}",
|
||||
field_type=FieldType.formula,
|
||||
config={"formula_expr": formula_expr},
|
||||
owner=FieldOwner.user,
|
||||
)
|
||||
|
||||
return _make
|
||||
|
|
@ -0,0 +1,322 @@
|
|||
"""Tests for U6: attachment & image field upload, download, and cleanup.
|
||||
|
||||
Requires PostgreSQL — marked ``postgres``. Uses ``httpx.AsyncClient`` with
|
||||
``ASGITransport`` (same pattern as test_routes.py).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from httpx import ASGITransport
|
||||
|
||||
from agentkit.bitable.service import BitableService
|
||||
from agentkit.server.routes import bitable as bitable_routes
|
||||
from agentkit.server.routes.bitable import require_bitable_auth
|
||||
|
||||
pytestmark = pytest.mark.postgres
|
||||
|
||||
TEST_USER_ID = "test-user-id"
|
||||
|
||||
|
||||
def _make_test_user() -> dict[str, Any]:
|
||||
return {"user_id": TEST_USER_ID, "username": "testuser", "role": "member"}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app(bitable_service: BitableService, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> FastAPI:
|
||||
"""Test app with upload dir redirected to tmp_path."""
|
||||
upload_dir = tmp_path / "bitable_uploads"
|
||||
# Patch both the routes module variable AND the env var (service reads env var)
|
||||
monkeypatch.setattr(bitable_routes, "BITABLE_UPLOAD_DIR", upload_dir)
|
||||
monkeypatch.setenv("AGENTKIT_BITABLE_UPLOAD_DIR", str(upload_dir))
|
||||
|
||||
app = FastAPI()
|
||||
app.state.bitable_service = bitable_service
|
||||
app.include_router(bitable_routes.router, prefix="/api/v1")
|
||||
app.dependency_overrides[require_bitable_auth] = lambda: _make_test_user()
|
||||
|
||||
return app
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def client(app: FastAPI) -> httpx.AsyncClient:
|
||||
transport = ASGITransport(app=app)
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://test") as c:
|
||||
yield c
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _create_table_with_field(
|
||||
client: httpx.AsyncClient,
|
||||
field_type: str,
|
||||
field_name: str = "files",
|
||||
) -> tuple[str, str]:
|
||||
"""Create a table + a field, return (table_id, field_id)."""
|
||||
table_id = (
|
||||
await client.post("/api/v1/bitable/tables", json={"name": "T"})
|
||||
).json()["table"]["id"]
|
||||
field_id = (
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/fields",
|
||||
json={"name": field_name, "field_type": field_type, "owner": "agent"},
|
||||
)
|
||||
).json()["field"]["id"]
|
||||
return table_id, field_id
|
||||
|
||||
|
||||
def _make_image_bytes(name: str = "test.png", size: int = 100) -> tuple[bytes, str]:
|
||||
"""Minimal valid PNG header + padding."""
|
||||
png_header = b"\x89PNG\r\n\x1a\n"
|
||||
body = b"\x00" * size
|
||||
return png_header + body, name
|
||||
|
||||
|
||||
def _make_pdf_bytes(name: str = "doc.pdf", size: int = 50) -> tuple[bytes, str]:
|
||||
return b"%PDF-1.4\n" + b"\x00" * size, name
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Upload tests
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_upload_image_success(client: httpx.AsyncClient, tmp_path: Path) -> None:
|
||||
table_id, field_id = await _create_table_with_field(client, "image")
|
||||
img_bytes, img_name = _make_image_bytes()
|
||||
resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/upload",
|
||||
params={"field_id": field_id},
|
||||
files={"file": (img_name, io.BytesIO(img_bytes), "image/png")},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()
|
||||
assert data["filename"] == img_name
|
||||
assert data["mime_type"] == "image/png"
|
||||
assert data["size"] == len(img_bytes)
|
||||
assert data["stored_name"].endswith(".png")
|
||||
assert data["url"].startswith("/api/v1/bitable/files/")
|
||||
# File exists on disk
|
||||
file_path = bitable_routes.BITABLE_UPLOAD_DIR / data["stored_name"]
|
||||
assert file_path.exists()
|
||||
assert file_path.read_bytes() == img_bytes
|
||||
|
||||
|
||||
async def test_upload_attachment_pdf(client: httpx.AsyncClient) -> None:
|
||||
table_id, field_id = await _create_table_with_field(client, "attachment")
|
||||
pdf_bytes, pdf_name = _make_pdf_bytes()
|
||||
resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/upload",
|
||||
params={"field_id": field_id},
|
||||
files={"file": (pdf_name, io.BytesIO(pdf_bytes), "application/pdf")},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()
|
||||
assert data["filename"] == pdf_name
|
||||
assert data["mime_type"] == "application/pdf"
|
||||
|
||||
|
||||
async def test_upload_image_rejects_non_image(client: httpx.AsyncClient) -> None:
|
||||
table_id, field_id = await _create_table_with_field(client, "image")
|
||||
pdf_bytes, _ = _make_pdf_bytes()
|
||||
resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/upload",
|
||||
params={"field_id": field_id},
|
||||
files={"file": ("doc.pdf", io.BytesIO(pdf_bytes), "application/pdf")},
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
assert "image" in resp.json()["detail"].lower()
|
||||
|
||||
|
||||
async def test_upload_rejects_non_attachment_field(client: httpx.AsyncClient) -> None:
|
||||
table_id, field_id = await _create_table_with_field(client, "text")
|
||||
img_bytes, _ = _make_image_bytes()
|
||||
resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/upload",
|
||||
params={"field_id": field_id},
|
||||
files={"file": ("test.png", io.BytesIO(img_bytes), "image/png")},
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
async def test_upload_404_unknown_field(client: httpx.AsyncClient) -> None:
|
||||
table_id = (
|
||||
await client.post("/api/v1/bitable/tables", json={"name": "T"})
|
||||
).json()["table"]["id"]
|
||||
img_bytes, _ = _make_image_bytes()
|
||||
resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/upload",
|
||||
params={"field_id": "nonexistent"},
|
||||
files={"file": ("test.png", io.BytesIO(img_bytes), "image/png")},
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
async def test_upload_requires_auth(bitable_service: BitableService) -> None:
|
||||
"""No auth override → 401."""
|
||||
app = FastAPI()
|
||||
app.state.bitable_service = bitable_service
|
||||
app.include_router(bitable_routes.router, prefix="/api/v1")
|
||||
transport = ASGITransport(app=app)
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://test") as c:
|
||||
resp = await c.post(
|
||||
"/api/v1/bitable/tables/x/upload",
|
||||
params={"field_id": "y"},
|
||||
files={"file": ("t.png", io.BytesIO(b"x"), "image/png")},
|
||||
)
|
||||
assert resp.status_code == 401
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Download tests
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_download_file_success(client: httpx.AsyncClient) -> None:
|
||||
table_id, field_id = await _create_table_with_field(client, "image")
|
||||
img_bytes, img_name = _make_image_bytes()
|
||||
upload_resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/upload",
|
||||
params={"field_id": field_id},
|
||||
files={"file": (img_name, io.BytesIO(img_bytes), "image/png")},
|
||||
)
|
||||
stored_name = upload_resp.json()["stored_name"]
|
||||
resp = await client.get(f"/api/v1/bitable/files/{stored_name}")
|
||||
assert resp.status_code == 200
|
||||
assert resp.content == img_bytes
|
||||
|
||||
|
||||
async def test_download_404_missing_file(client: httpx.AsyncClient) -> None:
|
||||
resp = await client.get("/api/v1/bitable/files/nonexistent.png")
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Attachment cleanup on record deletion
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_delete_record_cleans_up_files(
|
||||
client: httpx.AsyncClient,
|
||||
bitable_service: BitableService,
|
||||
) -> None:
|
||||
table_id, field_id = await _create_table_with_field(client, "image")
|
||||
img_bytes, _ = _make_image_bytes()
|
||||
upload_resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/upload",
|
||||
params={"field_id": field_id},
|
||||
files={"file": ("pic.png", io.BytesIO(img_bytes), "image/png")},
|
||||
)
|
||||
file_meta = upload_resp.json()
|
||||
stored_name = file_meta["stored_name"]
|
||||
|
||||
# Create a record with the image metadata
|
||||
create_resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/records",
|
||||
json={"records": [{field_id: [file_meta]}]},
|
||||
)
|
||||
record_id = create_resp.json()["records"][0]["id"]
|
||||
|
||||
# Verify file exists
|
||||
file_path = bitable_routes.BITABLE_UPLOAD_DIR / stored_name
|
||||
assert file_path.exists()
|
||||
|
||||
# Delete the record
|
||||
del_resp = await client.delete(f"/api/v1/bitable/records/{record_id}")
|
||||
assert del_resp.status_code == 200
|
||||
|
||||
# File should be gone
|
||||
assert not file_path.exists()
|
||||
|
||||
|
||||
async def test_delete_records_by_table_cleans_up_files(
|
||||
client: httpx.AsyncClient,
|
||||
) -> None:
|
||||
table_id, field_id = await _create_table_with_field(client, "attachment")
|
||||
pdf_bytes, _ = _make_pdf_bytes()
|
||||
upload_resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/upload",
|
||||
params={"field_id": field_id},
|
||||
files={"file": ("doc.pdf", io.BytesIO(pdf_bytes), "application/pdf")},
|
||||
)
|
||||
file_meta = upload_resp.json()
|
||||
stored_name = file_meta["stored_name"]
|
||||
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/records",
|
||||
json={"records": [{field_id: [file_meta]}]},
|
||||
)
|
||||
|
||||
file_path = bitable_routes.BITABLE_UPLOAD_DIR / stored_name
|
||||
assert file_path.exists()
|
||||
|
||||
# Delete all records
|
||||
resp = await client.delete(f"/api/v1/bitable/tables/{table_id}/records")
|
||||
assert resp.status_code == 200
|
||||
assert not file_path.exists()
|
||||
|
||||
|
||||
async def test_delete_record_when_file_already_missing(
|
||||
client: httpx.AsyncClient,
|
||||
) -> None:
|
||||
"""Record deletion should succeed even if the physical file is gone."""
|
||||
table_id, field_id = await _create_table_with_field(client, "image")
|
||||
img_bytes, _ = _make_image_bytes()
|
||||
upload_resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/upload",
|
||||
params={"field_id": field_id},
|
||||
files={"file": ("pic.png", io.BytesIO(img_bytes), "image/png")},
|
||||
)
|
||||
file_meta = upload_resp.json()
|
||||
stored_name = file_meta["stored_name"]
|
||||
|
||||
create_resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/records",
|
||||
json={"records": [{field_id: [file_meta]}]},
|
||||
)
|
||||
record_id = create_resp.json()["records"][0]["id"]
|
||||
|
||||
# Manually delete the file before deleting the record
|
||||
file_path = bitable_routes.BITABLE_UPLOAD_DIR / stored_name
|
||||
file_path.unlink()
|
||||
assert not file_path.exists()
|
||||
|
||||
# Record deletion should still succeed
|
||||
del_resp = await client.delete(f"/api/v1/bitable/records/{record_id}")
|
||||
assert del_resp.status_code == 200
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Multiple files in one field
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_multiple_files_in_attachment_field(client: httpx.AsyncClient) -> None:
|
||||
table_id, field_id = await _create_table_with_field(client, "attachment")
|
||||
metas = []
|
||||
for name in ("a.pdf", "b.pdf"):
|
||||
pdf_bytes, _ = _make_pdf_bytes(name)
|
||||
resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/upload",
|
||||
params={"field_id": field_id},
|
||||
files={"file": (name, io.BytesIO(pdf_bytes), "application/pdf")},
|
||||
)
|
||||
metas.append(resp.json())
|
||||
|
||||
# Store all files as an array in one record
|
||||
create_resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/records",
|
||||
json={"records": [{field_id: metas}]},
|
||||
)
|
||||
assert create_resp.status_code == 200
|
||||
record = create_resp.json()["records"][0]
|
||||
assert len(record["values"][field_id]) == 2
|
||||
|
|
@ -0,0 +1,485 @@
|
|||
"""Tests for BitableTool (U4).
|
||||
|
||||
Tests the full HTTP flow: BitableTool → bitable REST API → BitableService.
|
||||
Uses ``httpx.AsyncClient`` + ``ASGITransport`` so the tool's HTTP calls
|
||||
and the bitable DB share one event loop.
|
||||
|
||||
Covers:
|
||||
- KTD11: X-Internal-Token auth (valid token accepted, invalid rejected)
|
||||
- Batch chunking: 1200 records → 3 HTTP requests (500+500+200)
|
||||
- Resume from partial failure
|
||||
- Three ingestion types: Excel, database, API collector
|
||||
- create_table, upsert_records, query_records
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from httpx import ASGITransport
|
||||
|
||||
from agentkit.bitable.service import BitableService
|
||||
from agentkit.server.routes import bitable as bitable_routes
|
||||
from agentkit.server.routes.bitable import require_bitable_auth
|
||||
from agentkit.tools.bitable_tool import BATCH_SIZE, BitableTool
|
||||
|
||||
pytestmark = pytest.mark.postgres
|
||||
|
||||
TEST_TOKEN = "test-internal-token-abc123"
|
||||
TEST_USER = {"user_id": "test-user", "username": "tester", "role": "member"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app(bitable_service: BitableService) -> FastAPI:
|
||||
"""Test app with bitable_service + internal token on app.state."""
|
||||
app = FastAPI()
|
||||
app.state.bitable_service = bitable_service
|
||||
app.state.bitable_internal_token = TEST_TOKEN
|
||||
app.include_router(bitable_routes.router, prefix="/api/v1")
|
||||
# Override auth so JWT path also works (for non-internal-token tests)
|
||||
app.dependency_overrides[require_bitable_auth] = lambda: TEST_USER
|
||||
return app
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app_no_override(bitable_service: BitableService) -> FastAPI:
|
||||
"""App without auth override — tests real X-Internal-Token path."""
|
||||
app = FastAPI()
|
||||
app.state.bitable_service = bitable_service
|
||||
app.state.bitable_internal_token = TEST_TOKEN
|
||||
app.include_router(bitable_routes.router, prefix="/api/v1")
|
||||
return app
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app_no_token(bitable_service: BitableService) -> FastAPI:
|
||||
"""App without internal token configured."""
|
||||
app = FastAPI()
|
||||
app.state.bitable_service = bitable_service
|
||||
app.include_router(bitable_routes.router, prefix="/api/v1")
|
||||
app.dependency_overrides[require_bitable_auth] = lambda: TEST_USER
|
||||
return app
|
||||
|
||||
|
||||
def _make_client(app: FastAPI, token: str | None = None) -> httpx.AsyncClient:
|
||||
"""Create an httpx AsyncClient backed by ASGITransport.
|
||||
|
||||
If token is provided, the X-Internal-Token header is set as default
|
||||
on the client — mirroring how BitableTool._get_client configures it.
|
||||
"""
|
||||
base = "http://test/api/v1/bitable"
|
||||
transport = ASGITransport(app=app)
|
||||
headers: dict[str, str] = {}
|
||||
if token:
|
||||
headers["X-Internal-Token"] = token
|
||||
return httpx.AsyncClient(transport=transport, base_url=base, headers=headers)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def tool(app: FastAPI) -> BitableTool:
|
||||
"""BitableTool pointing at the test app via ASGITransport.
|
||||
|
||||
ponytail: We patch _client to use ASGITransport instead of real
|
||||
HTTP — this shares the event loop with the async DB fixtures.
|
||||
"""
|
||||
client = _make_client(app, token=TEST_TOKEN)
|
||||
t = BitableTool(base_url="http://test/api/v1/bitable", internal_token=TEST_TOKEN)
|
||||
t._client = client
|
||||
yield t
|
||||
await client.aclose()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def tool_no_token(app_no_token: FastAPI) -> BitableTool:
|
||||
"""BitableTool without internal token."""
|
||||
client = _make_client(app_no_token, token=None)
|
||||
t = BitableTool(base_url="http://test/api/v1/bitable", internal_token=None)
|
||||
t._client = client
|
||||
yield t
|
||||
await client.aclose()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def tool_real_auth(app_no_override: FastAPI) -> BitableTool:
|
||||
"""BitableTool that sends real X-Internal-Token header (no auth override)."""
|
||||
client = _make_client(app_no_override, token=TEST_TOKEN)
|
||||
t = BitableTool(base_url="http://test/api/v1/bitable", internal_token=TEST_TOKEN)
|
||||
t._client = client
|
||||
yield t
|
||||
await client.aclose()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_xlsx(sheets: dict[str, list[list]]) -> bytes:
|
||||
"""Create an in-memory .xlsx file."""
|
||||
from openpyxl import Workbook
|
||||
|
||||
wb = Workbook()
|
||||
wb.remove(wb.active)
|
||||
for name, rows in sheets.items():
|
||||
ws = wb.create_sheet(title=name)
|
||||
for row in rows:
|
||||
ws.append(row)
|
||||
buf = io.BytesIO()
|
||||
wb.save(buf)
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
async def _setup_table_with_pk(tool: BitableTool, name: str = "T") -> tuple[str, str, str]:
|
||||
"""Create a table with a text PK field and a number data field.
|
||||
|
||||
Returns (table_id, pk_field_id, data_field_id).
|
||||
"""
|
||||
result = await tool.execute(action="create_table", table_name=name)
|
||||
assert result["success"], result
|
||||
table_id = result["table"]["id"]
|
||||
|
||||
client = await tool._get_client()
|
||||
|
||||
# Create PK field
|
||||
resp = await client.post(
|
||||
f"/tables/{table_id}/fields",
|
||||
json={"name": "id", "field_type": "text", "owner": "agent"},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
pk_field_id = resp.json()["field"]["id"]
|
||||
|
||||
# Create data field
|
||||
resp = await client.post(
|
||||
f"/tables/{table_id}/fields",
|
||||
json={"name": "val", "field_type": "number", "owner": "agent"},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data_field_id = resp.json()["field"]["id"]
|
||||
|
||||
# Set PK
|
||||
resp = await client.patch(f"/tables/{table_id}", json={"primary_key_field_id": pk_field_id})
|
||||
resp.raise_for_status()
|
||||
|
||||
return table_id, pk_field_id, data_field_id
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# create_table
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_create_table(tool: BitableTool) -> None:
|
||||
"""create_table action creates a bitable table via HTTP."""
|
||||
result = await tool.execute(action="create_table", table_name="MyTable")
|
||||
assert result["success"] is True
|
||||
assert result["table"]["name"] == "MyTable"
|
||||
|
||||
|
||||
async def test_create_table_missing_name(tool: BitableTool) -> None:
|
||||
"""Missing table_name → error."""
|
||||
result = await tool.execute(action="create_table")
|
||||
assert result["success"] is False
|
||||
assert "table_name" in result["error"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# KTD11: Internal token auth
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_internal_token_accepted(tool_real_auth: BitableTool) -> None:
|
||||
"""Valid X-Internal-Token → request succeeds (no JWT needed)."""
|
||||
result = await tool_real_auth.execute(action="create_table", table_name="Authed")
|
||||
assert result["success"] is True
|
||||
|
||||
|
||||
async def test_invalid_token_rejected(app_no_override: FastAPI) -> None:
|
||||
"""Wrong X-Internal-Token → 401."""
|
||||
transport = ASGITransport(app=app_no_override)
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
resp = await client.post(
|
||||
"/api/v1/bitable/tables",
|
||||
json={"name": "X"},
|
||||
headers={"X-Internal-Token": "wrong-token"},
|
||||
)
|
||||
assert resp.status_code == 401
|
||||
|
||||
|
||||
async def test_no_auth_rejected(app_no_override: FastAPI) -> None:
|
||||
"""No auth at all → 401."""
|
||||
transport = ASGITransport(app=app_no_override)
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
resp = await client.post("/api/v1/bitable/tables", json={"name": "X"})
|
||||
assert resp.status_code == 401
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Batch chunking (BATCH_SIZE=500)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_batch_upsert_1200_records(tool: BitableTool) -> None:
|
||||
"""1200 records → 3 batches (500+500+200), all succeed."""
|
||||
table_id, pk_fid, data_fid = await _setup_table_with_pk(tool)
|
||||
|
||||
records = [{pk_fid: f"r{i}", data_fid: i * 10} for i in range(1200)]
|
||||
result = await tool.execute(
|
||||
action="upsert_records",
|
||||
table_id=table_id,
|
||||
records=records,
|
||||
primary_key_field_id=pk_fid,
|
||||
)
|
||||
assert result["success"] is True
|
||||
assert result["successful_count"] == 1200
|
||||
assert result["total"] == 1200
|
||||
assert "errors" not in result
|
||||
|
||||
|
||||
async def test_batch_size_is_500() -> None:
|
||||
"""Verify BATCH_SIZE constant is 500."""
|
||||
assert BATCH_SIZE == 500
|
||||
|
||||
|
||||
async def test_resume_from_partial_failure(tool: BitableTool) -> None:
|
||||
"""resume_from skips already-successful records."""
|
||||
table_id, pk_fid, data_fid = await _setup_table_with_pk(tool)
|
||||
|
||||
# First, insert 500 records successfully
|
||||
batch1 = [{pk_fid: f"r{i}", data_fid: i} for i in range(500)]
|
||||
result1 = await tool.execute(
|
||||
action="upsert_records",
|
||||
table_id=table_id,
|
||||
records=batch1,
|
||||
primary_key_field_id=pk_fid,
|
||||
)
|
||||
assert result1["successful_count"] == 500
|
||||
|
||||
# Now resume from 500 with the remaining 700
|
||||
all_records = [{pk_fid: f"r{i}", data_fid: i} for i in range(1200)]
|
||||
remaining = all_records[500:]
|
||||
result2 = await tool.execute(
|
||||
action="upsert_records",
|
||||
table_id=table_id,
|
||||
records=remaining,
|
||||
primary_key_field_id=pk_fid,
|
||||
resume_from=0, # remaining is already sliced
|
||||
)
|
||||
assert result2["successful_count"] == 700
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# query_records
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_query_records(tool: BitableTool) -> None:
|
||||
"""query_records returns records from the table."""
|
||||
table_id, pk_fid, data_fid = await _setup_table_with_pk(tool)
|
||||
|
||||
# Insert some records
|
||||
await tool.execute(
|
||||
action="upsert_records",
|
||||
table_id=table_id,
|
||||
records=[{pk_fid: "a", data_fid: 1}, {pk_fid: "b", data_fid: 2}],
|
||||
primary_key_field_id=pk_fid,
|
||||
)
|
||||
|
||||
# Query
|
||||
result = await tool.execute(action="query_records", table_id=table_id)
|
||||
assert result["success"] is True
|
||||
assert len(result["records"]) == 2
|
||||
|
||||
|
||||
async def test_query_records_with_limit(tool: BitableTool) -> None:
|
||||
"""query_records with limit returns fewer records."""
|
||||
table_id, pk_fid, data_fid = await _setup_table_with_pk(tool)
|
||||
|
||||
await tool.execute(
|
||||
action="upsert_records",
|
||||
table_id=table_id,
|
||||
records=[{pk_fid: f"r{i}", data_fid: i} for i in range(10)],
|
||||
primary_key_field_id=pk_fid,
|
||||
)
|
||||
|
||||
result = await tool.execute(action="query_records", table_id=table_id, limit=5)
|
||||
assert result["success"] is True
|
||||
assert len(result["records"]) == 5
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# import_excel
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_import_excel_file(tool: BitableTool, tmp_path) -> None:
|
||||
"""import_excel from file path → creates table + fields + records."""
|
||||
xlsx_bytes = _make_xlsx({"Products": [["name", "price"], ["Widget", 9.99], ["Gadget", 19.99]]})
|
||||
file_path = tmp_path / "test.xlsx"
|
||||
file_path.write_bytes(xlsx_bytes)
|
||||
|
||||
result = await tool.execute(action="import_excel", file_path=str(file_path))
|
||||
assert result["success"] is True
|
||||
sheet_result = result["sheets"][0]
|
||||
assert sheet_result["record_count"] == 2
|
||||
assert sheet_result["field_count"] == 2
|
||||
|
||||
# Verify data was actually written
|
||||
table_id = sheet_result["table_id"]
|
||||
query = await tool.execute(action="query_records", table_id=table_id)
|
||||
assert len(query["records"]) == 2
|
||||
|
||||
|
||||
async def test_import_excel_empty_sheet(tool: BitableTool, tmp_path) -> None:
|
||||
"""Excel with only headers (no data rows) → table created, 0 records."""
|
||||
xlsx_bytes = _make_xlsx({"Empty": [["col1", "col2"]]})
|
||||
file_path = tmp_path / "empty.xlsx"
|
||||
file_path.write_bytes(xlsx_bytes)
|
||||
|
||||
result = await tool.execute(action="import_excel", file_path=str(file_path))
|
||||
assert result["success"] is True
|
||||
assert result["sheets"][0]["record_count"] == 0
|
||||
assert result["sheets"][0]["field_count"] == 2
|
||||
|
||||
|
||||
async def test_import_excel_missing_path(tool: BitableTool) -> None:
|
||||
"""No file_path or file_url → error."""
|
||||
result = await tool.execute(action="import_excel")
|
||||
assert result["success"] is False
|
||||
assert "file_path" in result["error"] or "file_url" in result["error"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# collect_api
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_collect_api(tool: BitableTool) -> None:
|
||||
"""collect_api transforms records via field_mapping and upserts."""
|
||||
table_id, pk_fid, data_fid = await _setup_table_with_pk(tool)
|
||||
|
||||
result = await tool.execute(
|
||||
action="collect_api",
|
||||
table_id=table_id,
|
||||
records=[
|
||||
{"user_id": "u1", "score": 100},
|
||||
{"user_id": "u2", "score": 200},
|
||||
],
|
||||
field_mapping={"user_id": pk_fid, "score": data_fid},
|
||||
primary_key_field_id=pk_fid,
|
||||
)
|
||||
assert result["success"] is True
|
||||
assert result["successful_count"] == 2
|
||||
|
||||
# Verify
|
||||
query = await tool.execute(action="query_records", table_id=table_id)
|
||||
assert len(query["records"]) == 2
|
||||
|
||||
|
||||
async def test_collect_api_missing_fields(tool: BitableTool) -> None:
|
||||
"""Missing required fields → error."""
|
||||
result = await tool.execute(action="collect_api", records=[])
|
||||
assert result["success"] is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Error handling
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_unknown_action(tool: BitableTool) -> None:
|
||||
"""Unknown action → error."""
|
||||
result = await tool.execute(action="bogus")
|
||||
assert result["success"] is False
|
||||
assert "Unknown action" in result["error"]
|
||||
|
||||
|
||||
async def test_query_nonexistent_table(tool: BitableTool) -> None:
|
||||
"""Querying a non-existent table → error."""
|
||||
result = await tool.execute(action="query_records", table_id="nonexistent-id")
|
||||
assert result["success"] is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Database ingestion (type mapping only — no real external DB needed)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_db_type_mapping_integer() -> None:
|
||||
"""Integer type → 'number'."""
|
||||
from sqlalchemy import Integer
|
||||
|
||||
from agentkit.bitable.ingestion.database import infer_field_type
|
||||
|
||||
assert infer_field_type(Integer()) == "number"
|
||||
assert infer_field_type(Integer) == "number"
|
||||
|
||||
|
||||
def test_db_type_mapping_varchar() -> None:
|
||||
"""String type → 'text'."""
|
||||
from sqlalchemy import String
|
||||
|
||||
from agentkit.bitable.ingestion.database import infer_field_type
|
||||
|
||||
assert infer_field_type(String(255)) == "text"
|
||||
|
||||
|
||||
def test_db_type_mapping_datetime() -> None:
|
||||
"""DateTime type → 'date'."""
|
||||
from sqlalchemy import DateTime
|
||||
|
||||
from agentkit.bitable.ingestion.database import infer_field_type
|
||||
|
||||
assert infer_field_type(DateTime()) == "date"
|
||||
|
||||
|
||||
def test_db_type_mapping_unknown_fallback() -> None:
|
||||
"""Unknown type → 'text' (safe fallback)."""
|
||||
from agentkit.bitable.ingestion.database import infer_field_type
|
||||
|
||||
class CustomType:
|
||||
pass
|
||||
|
||||
assert infer_field_type(CustomType()) == "text"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# API collector transform
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_transform_records_basic() -> None:
|
||||
"""transform_records maps source keys to field IDs."""
|
||||
from agentkit.bitable.ingestion.api_collector import transform_records
|
||||
|
||||
result = transform_records(
|
||||
records=[{"name": "Alice", "age": 30, "extra": "dropped"}],
|
||||
field_mapping={"name": "fld_abc", "age": "fld_def"},
|
||||
)
|
||||
assert result == [{"fld_abc": "Alice", "fld_def": 30}]
|
||||
|
||||
|
||||
def test_transform_records_empty() -> None:
|
||||
"""Empty records → empty result."""
|
||||
from agentkit.bitable.ingestion.api_collector import transform_records
|
||||
|
||||
assert transform_records([], {"a": "b"}) == []
|
||||
assert transform_records([{"a": 1}], {}) == []
|
||||
|
||||
|
||||
def test_transform_records_missing_keys() -> None:
|
||||
"""Source keys not in mapping are silently dropped."""
|
||||
from agentkit.bitable.ingestion.api_collector import transform_records
|
||||
|
||||
result = transform_records(
|
||||
records=[{"a": 1, "b": 2}],
|
||||
field_mapping={"a": "fld_a"}, # b is not mapped
|
||||
)
|
||||
assert result == [{"fld_a": 1}]
|
||||
|
|
@ -0,0 +1,205 @@
|
|||
"""Tests for U7: bitable CLI subcommands.
|
||||
|
||||
Requires PostgreSQL — marked ``postgres``. Uses Typer's CliRunner.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from agentkit.cli.bitable import bitable_app
|
||||
|
||||
pytestmark = pytest.mark.postgres
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def db_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Ensure DATABASE_URL is set for CLI tests."""
|
||||
url = os.environ.get("DATABASE_URL") or os.environ.get("AGENTKIT_DATABASE_URL")
|
||||
if not url:
|
||||
pytest.skip("PostgreSQL not available (set DATABASE_URL)")
|
||||
monkeypatch.setenv("DATABASE_URL", url)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def clean_schema(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Drop and recreate bitable schema before each test."""
|
||||
import asyncio
|
||||
|
||||
url = os.environ.get("DATABASE_URL") or os.environ.get("AGENTKIT_DATABASE_URL")
|
||||
if not url:
|
||||
pytest.skip("PostgreSQL not available")
|
||||
|
||||
from agentkit.bitable.db import BitableDB
|
||||
from sqlalchemy import text
|
||||
|
||||
async def _clean():
|
||||
db = BitableDB()
|
||||
await db.init()
|
||||
async with db.engine.begin() as conn:
|
||||
await conn.execute(text("DROP SCHEMA IF EXISTS bitable CASCADE"))
|
||||
await db.init()
|
||||
await db.close()
|
||||
|
||||
asyncio.run(_clean())
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# list-tables
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_list_tables_empty(db_env, clean_schema) -> None:
|
||||
result = runner.invoke(bitable_app, ["list-tables"])
|
||||
assert result.exit_code == 0
|
||||
assert "No tables found" in result.output
|
||||
|
||||
|
||||
def test_list_tables_after_create(db_env, clean_schema) -> None:
|
||||
# Create a table first
|
||||
runner.invoke(bitable_app, ["create-table", "--name", "TestTable"])
|
||||
result = runner.invoke(bitable_app, ["list-tables"])
|
||||
assert result.exit_code == 0
|
||||
assert "TestTable" in result.output
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# create-table
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_create_table_success(db_env, clean_schema) -> None:
|
||||
result = runner.invoke(
|
||||
bitable_app,
|
||||
["create-table", "--name", "MyTable", "--description", "A test table"],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
assert "Created table" in result.output
|
||||
assert "MyTable" in result.output
|
||||
assert "A test table" in result.output
|
||||
|
||||
|
||||
def test_create_table_minimal(db_env, clean_schema) -> None:
|
||||
result = runner.invoke(bitable_app, ["create-table", "--name", "Minimal"])
|
||||
assert result.exit_code == 0
|
||||
assert "Minimal" in result.output
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# query
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_query_table_not_found(db_env, clean_schema) -> None:
|
||||
result = runner.invoke(bitable_app, ["query", "--table", "nonexistent-id"])
|
||||
assert result.exit_code == 1
|
||||
assert "not found" in result.output
|
||||
|
||||
|
||||
def test_query_empty_table(db_env, clean_schema) -> None:
|
||||
# Create a table first
|
||||
create_result = runner.invoke(bitable_app, ["create-table", "--name", "Empty"])
|
||||
assert create_result.exit_code == 0
|
||||
# Extract table ID from output — it's on the "ID:" line
|
||||
lines = create_result.output.split("\n")
|
||||
table_id = None
|
||||
for line in lines:
|
||||
if "ID:" in line:
|
||||
# Extract the cyan-colored ID
|
||||
table_id = line.split("ID:")[1].strip().strip("[]").split(" ")[0]
|
||||
# Remove rich formatting
|
||||
table_id = table_id.replace("[cyan]", "").replace("[/cyan]", "")
|
||||
break
|
||||
assert table_id is not None, f"Could not extract table ID from: {create_result.output}"
|
||||
|
||||
result = runner.invoke(bitable_app, ["query", "--table", table_id])
|
||||
assert result.exit_code == 0
|
||||
assert "No records found" in result.output
|
||||
|
||||
|
||||
def test_query_with_records(db_env, clean_schema) -> None:
|
||||
"""Create table + field + records via service, then query via CLI."""
|
||||
import asyncio
|
||||
|
||||
from agentkit.bitable.db import BitableDB
|
||||
from agentkit.bitable.models import FieldType
|
||||
from agentkit.bitable.service import BitableService
|
||||
|
||||
async def _setup():
|
||||
db = BitableDB()
|
||||
await db.init()
|
||||
service = BitableService(db)
|
||||
table = await service.create_table(name="Data")
|
||||
field = await service.create_field(
|
||||
table_id=table.id, name="name", field_type=FieldType.text
|
||||
)
|
||||
await service.create_record(table_id=table.id, values={field.id: "Alice"})
|
||||
await service.create_record(table_id=table.id, values={field.id: "Bob"})
|
||||
await db.close()
|
||||
return table.id
|
||||
|
||||
table_id = asyncio.run(_setup())
|
||||
|
||||
result = runner.invoke(bitable_app, ["query", "--table", table_id, "--limit", "10"])
|
||||
assert result.exit_code == 0
|
||||
assert "Alice" in result.output
|
||||
assert "Bob" in result.output
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# import-excel
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_import_excel_file_not_found(db_env, clean_schema) -> None:
|
||||
result = runner.invoke(
|
||||
bitable_app,
|
||||
["import-excel", "--file", "/nonexistent/file.xlsx"],
|
||||
)
|
||||
assert result.exit_code == 1
|
||||
assert "not found" in result.output
|
||||
|
||||
|
||||
def test_import_excel_success(db_env, clean_schema, tmp_path: Path) -> None:
|
||||
"""Create a real xlsx file and import it."""
|
||||
from openpyxl import Workbook
|
||||
|
||||
wb = Workbook()
|
||||
ws = wb.active
|
||||
ws.title = "Sheet1"
|
||||
ws.append(["name", "age"])
|
||||
ws.append(["Alice", 30])
|
||||
ws.append(["Bob", 25])
|
||||
xlsx_path = tmp_path / "test.xlsx"
|
||||
wb.save(xlsx_path)
|
||||
|
||||
result = runner.invoke(
|
||||
bitable_app,
|
||||
["import-excel", "--file", str(xlsx_path), "--table", "Imported"],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
assert "Created table" in result.output
|
||||
assert "Imported" in result.output
|
||||
assert "Imported 2 records" in result.output
|
||||
assert "name" in result.output
|
||||
assert "age" in result.output
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Error path: no DATABASE_URL
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_no_database_url(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""CLI should exit with clear error when DATABASE_URL is not set."""
|
||||
monkeypatch.delenv("DATABASE_URL", raising=False)
|
||||
monkeypatch.delenv("AGENTKIT_DATABASE_URL", raising=False)
|
||||
result = runner.invoke(bitable_app, ["list-tables"])
|
||||
assert result.exit_code == 1
|
||||
assert "DATABASE_URL" in result.output
|
||||
|
|
@ -0,0 +1,246 @@
|
|||
"""Tests for bitable DB initialization, schema, and constraints (U1).
|
||||
|
||||
Requires PostgreSQL — marked ``postgres``. Skips automatically when
|
||||
``DATABASE_URL`` / ``AGENTKIT_DATABASE_URL`` is unset (see conftest.py).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.postgres
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# init_bitable_db / BitableDB.init
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_init_creates_schema_and_all_tables(bitable_db) -> None:
|
||||
"""init creates the bitable schema and all 6 tables."""
|
||||
from sqlalchemy import text
|
||||
|
||||
async with bitable_db.engine.begin() as conn:
|
||||
# Schema exists
|
||||
result = await conn.execute(
|
||||
text(
|
||||
"SELECT schema_name FROM information_schema.schemata WHERE schema_name = 'bitable'"
|
||||
)
|
||||
)
|
||||
assert result.fetchone() is not None
|
||||
|
||||
# All 6 tables present
|
||||
result = await conn.execute(
|
||||
text(
|
||||
"SELECT table_name FROM information_schema.tables "
|
||||
"WHERE table_schema = 'bitable' ORDER BY table_name"
|
||||
)
|
||||
)
|
||||
tables = {row[0] for row in result.fetchall()}
|
||||
assert tables == {
|
||||
"bitable_fields",
|
||||
"bitable_meta",
|
||||
"bitable_records",
|
||||
"bitable_recalc_queue",
|
||||
"bitable_tables",
|
||||
"bitable_views",
|
||||
}
|
||||
|
||||
|
||||
async def test_init_is_idempotent(bitable_db) -> None:
|
||||
"""Calling init() twice does not raise and keeps schema intact."""
|
||||
# bitable_db fixture already called init(); call again
|
||||
await bitable_db.init()
|
||||
await bitable_db.init() # third time also fine
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
async with bitable_db.engine.begin() as conn:
|
||||
result = await conn.execute(text("SELECT COUNT(*) FROM bitable.bitable_meta"))
|
||||
assert result.fetchone()[0] >= 1
|
||||
|
||||
|
||||
async def test_schema_version_recorded_in_meta(bitable_db) -> None:
|
||||
"""bitable_meta stores the current schema version."""
|
||||
from agentkit.bitable.db import _META_SCHEMA_VERSION_KEY, _SCHEMA_VERSION
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
async with bitable_db.engine.begin() as conn:
|
||||
result = await conn.execute(
|
||||
text("SELECT value FROM bitable.bitable_meta WHERE key = :key"),
|
||||
{"key": _META_SCHEMA_VERSION_KEY},
|
||||
)
|
||||
row = result.fetchone()
|
||||
assert row is not None
|
||||
assert int(row[0]) == _SCHEMA_VERSION
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constraints
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_recalc_queue_unique_record_field(bitable_db) -> None:
|
||||
"""Recalc queue enforces (record_id, field_id) uniqueness — dedup."""
|
||||
from agentkit.bitable.models import FieldType
|
||||
from agentkit.bitable.repository import BitableRepository
|
||||
|
||||
repo = BitableRepository(bitable_db)
|
||||
table = await repo.create_table(name="T")
|
||||
field = await repo.create_field(table_id=table.id, name="f", field_type=FieldType.text)
|
||||
record = await repo.create_record(table_id=table.id)
|
||||
|
||||
# First enqueue succeeds
|
||||
task1 = await repo.enqueue_recalc(table.id, record.id, field.id)
|
||||
assert task1 is not None
|
||||
|
||||
# Second enqueue is a no-op (ON CONFLICT DO NOTHING) — returns None
|
||||
task2 = await repo.enqueue_recalc(table.id, record.id, field.id)
|
||||
assert task2 is None
|
||||
|
||||
|
||||
async def test_recalc_queue_status_index_exists(bitable_db) -> None:
|
||||
"""The (status, queued_at) index exists for worker consumption."""
|
||||
from sqlalchemy import text
|
||||
|
||||
async with bitable_db.engine.begin() as conn:
|
||||
result = await conn.execute(
|
||||
text(
|
||||
"SELECT indexname FROM pg_indexes "
|
||||
"WHERE schemaname = 'bitable' AND tablename = 'bitable_recalc_queue'"
|
||||
)
|
||||
)
|
||||
indexes = {row[0] for row in result.fetchall()}
|
||||
assert "ix_recalc_status_queued" in indexes
|
||||
assert "uq_recalc_record_field" in indexes
|
||||
|
||||
|
||||
async def test_records_values_gin_index_exists(bitable_db) -> None:
|
||||
"""GIN index on records.values exists for JSONB key lookups."""
|
||||
from sqlalchemy import text
|
||||
|
||||
async with bitable_db.engine.begin() as conn:
|
||||
result = await conn.execute(
|
||||
text(
|
||||
"SELECT indexname FROM pg_indexes "
|
||||
"WHERE schemaname = 'bitable' AND tablename = 'bitable_records'"
|
||||
)
|
||||
)
|
||||
indexes = {row[0] for row in result.fetchall()}
|
||||
assert "ix_bitable_records_values_gin" in indexes
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Repository CRUD smoke (verifies schema is usable end-to-end)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_repository_crud_round_trip(bitable_db) -> None:
|
||||
"""Repository can create/get/list/delete across all entities."""
|
||||
from agentkit.bitable.models import FieldOwner, FieldType, ViewType
|
||||
from agentkit.bitable.repository import BitableRepository
|
||||
|
||||
repo = BitableRepository(bitable_db)
|
||||
|
||||
# Table
|
||||
table = await repo.create_table(name="Orders", description="desc")
|
||||
assert table.name == "Orders"
|
||||
fetched = await repo.get_table(table.id)
|
||||
assert fetched is not None and fetched.id == table.id
|
||||
|
||||
# Field
|
||||
field = await repo.create_field(
|
||||
table_id=table.id,
|
||||
name="Amount",
|
||||
field_type=FieldType.number,
|
||||
owner=FieldOwner.agent,
|
||||
)
|
||||
fields = await repo.list_fields(table.id)
|
||||
assert len(fields) == 1
|
||||
assert fields[0].id == field.id
|
||||
|
||||
# Record
|
||||
record = await repo.create_record(table_id=table.id, values={field.id: 42})
|
||||
fetched_rec = await repo.get_record(record.id)
|
||||
assert fetched_rec is not None
|
||||
assert fetched_rec.values[field.id] == 42
|
||||
|
||||
# Cursor pagination
|
||||
rec2 = await repo.create_record(table_id=table.id, values={field.id: 99})
|
||||
records, next_cursor = await repo.list_records(table.id, limit=1)
|
||||
assert len(records) == 1
|
||||
assert next_cursor is not None
|
||||
records2, next_cursor2 = await repo.list_records(table.id, cursor=next_cursor, limit=1)
|
||||
assert len(records2) == 1
|
||||
# The second page should be the other record
|
||||
assert {records[0].id, records2[0].id} == {record.id, rec2.id}
|
||||
|
||||
# View
|
||||
view = await repo.create_view(table_id=table.id, name="All", view_type=ViewType.grid)
|
||||
views = await repo.list_views(table.id)
|
||||
assert len(views) == 1 and views[0].id == view.id
|
||||
|
||||
# Delete cascades
|
||||
deleted = await repo.delete_table(table.id)
|
||||
assert deleted is True
|
||||
assert await repo.get_table(table.id) is None
|
||||
assert await repo.get_field(field.id) is None
|
||||
assert await repo.get_record(record.id) is None
|
||||
assert (await repo.list_views(table.id)) == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Crash recovery
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_reset_stale_recalc_tasks(bitable_db) -> None:
|
||||
"""reset_stale_recalc_tasks flips 'calculating' back to 'pending'."""
|
||||
from agentkit.bitable.models import FieldType, RecalcStatus
|
||||
from agentkit.bitable.repository import BitableRepository
|
||||
|
||||
repo = BitableRepository(bitable_db)
|
||||
table = await repo.create_table(name="T")
|
||||
field = await repo.create_field(table_id=table.id, name="f", field_type=FieldType.text)
|
||||
record = await repo.create_record(table_id=table.id)
|
||||
|
||||
task = await repo.enqueue_recalc(table.id, record.id, field.id)
|
||||
assert task is not None
|
||||
|
||||
# Simulate a worker crash mid-calculation
|
||||
await repo.update_recalc_status(task.id, RecalcStatus.calculating)
|
||||
|
||||
reset_count = await repo.reset_stale_recalc_tasks()
|
||||
assert reset_count == 1
|
||||
|
||||
pending = await repo.get_pending_recalc_tasks()
|
||||
assert any(t.id == task.id for t in pending)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Degradation (no PG)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_bitable_db_without_url_raises() -> None:
|
||||
"""BitableDB with no URL raises RuntimeError on init (not silently None)."""
|
||||
# Clear env vars for this test to ensure no URL resolution
|
||||
import os
|
||||
|
||||
saved = (
|
||||
os.environ.pop("DATABASE_URL", None),
|
||||
os.environ.pop("AGENTKIT_DATABASE_URL", None),
|
||||
)
|
||||
try:
|
||||
from agentkit.bitable.db import BitableDB
|
||||
|
||||
db = BitableDB(database_url=None)
|
||||
# _database_url is None because no arg and no env
|
||||
assert db.database_url is None
|
||||
with pytest.raises(RuntimeError, match="No database URL"):
|
||||
await db.init()
|
||||
finally:
|
||||
for key, val in zip(("DATABASE_URL", "AGENTKIT_DATABASE_URL"), saved):
|
||||
if val is not None:
|
||||
os.environ[key] = val
|
||||
|
|
@ -0,0 +1,211 @@
|
|||
"""Tests for the formula engine — DAG, cycle detection, evaluation.
|
||||
|
||||
Covers: topological sort, circular reference detection, aggregate vs row
|
||||
context, formula-to-formula dependencies, and the built-in function library.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from agentkit.bitable.formula.engine import (
|
||||
CircularReferenceError,
|
||||
FormulaEngine,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Basic evaluation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_engine_evaluate_simple_arithmetic() -> None:
|
||||
"""=1+2*3 → 7"""
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("calc", "=1+2*3")
|
||||
result = engine.evaluate("calc", row_values={})
|
||||
assert result == 7
|
||||
|
||||
|
||||
def test_engine_evaluate_row_reference() -> None:
|
||||
"""={f1} + {f2} → row-level sum"""
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("sum", "={f1} + {f2}")
|
||||
result = engine.evaluate("sum", row_values={"f1": 10, "f2": 20})
|
||||
assert result == 30
|
||||
|
||||
|
||||
def test_engine_evaluate_aggregate_sum() -> None:
|
||||
"""=SUM({f1}) → aggregate sum of column"""
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("total", "=SUM({f1})")
|
||||
result = engine.evaluate("total", row_values={}, column_values={"f1": [1, 2, 3]})
|
||||
assert result == 6
|
||||
|
||||
|
||||
def test_engine_evaluate_aggregate_avg() -> None:
|
||||
"""=AVG({f1}) → average of column"""
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("avg", "=AVG({f1})")
|
||||
result = engine.evaluate("avg", row_values={}, column_values={"f1": [10, 20, 30]})
|
||||
assert result == 20.0
|
||||
|
||||
|
||||
def test_engine_evaluate_aggregate_count() -> None:
|
||||
"""=COUNT({f1}) → count of non-empty values"""
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("cnt", "=COUNT({f1})")
|
||||
result = engine.evaluate("cnt", row_values={}, column_values={"f1": [1, None, 3, "", 5]})
|
||||
assert result == 3 # None and "" are ignored
|
||||
|
||||
|
||||
def test_engine_evaluate_mixed_aggregate_and_row() -> None:
|
||||
"""={f1} + SUM({f2}) → row f1 + column f2 sum"""
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("mixed", "={f1} + SUM({f2})")
|
||||
result = engine.evaluate("mixed", row_values={"f1": 10}, column_values={"f2": [1, 2, 3]})
|
||||
assert result == 16 # 10 + 6
|
||||
|
||||
|
||||
def test_engine_evaluate_concat() -> None:
|
||||
"""=CONCAT({f1}, "-", {f2}) → string concat"""
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("label", '=CONCAT({f1}, "-", {f2})')
|
||||
result = engine.evaluate("label", row_values={"f1": "a", "f2": "b"})
|
||||
assert result == "a-b"
|
||||
|
||||
|
||||
def test_engine_evaluate_if_function() -> None:
|
||||
"""=IF({f1} > 5, "big", "small")"""
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("size", '=IF({f1} > 5, "big", "small")')
|
||||
assert engine.evaluate("size", row_values={"f1": 10}) == "big"
|
||||
assert engine.evaluate("size", row_values={"f1": 3}) == "small"
|
||||
|
||||
|
||||
def test_engine_evaluate_min_max() -> None:
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("mn", "=MIN({f1})")
|
||||
engine.add_formula("mx", "=MAX({f1})")
|
||||
cols = {"f1": [3, 1, 4, 1, 5, 9, 2, 6]}
|
||||
assert engine.evaluate("mn", {}, cols) == 1
|
||||
assert engine.evaluate("mx", {}, cols) == 9
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DAG: dependencies and dependents
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_engine_get_dependencies() -> None:
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("c", "={a} + {b}")
|
||||
assert engine.get_dependencies("c") == {"a", "b"}
|
||||
assert engine.get_dependents("a") == {"c"}
|
||||
assert engine.get_dependents("b") == {"c"}
|
||||
|
||||
|
||||
def test_engine_topological_order() -> None:
|
||||
"""c depends on b, b depends on a → order: a, b, c"""
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("c", "={b} + 1")
|
||||
engine.add_formula("b", "={a} + 1")
|
||||
engine.add_formula("a", "=1")
|
||||
order = engine.topological_order()
|
||||
assert order.index("a") < order.index("b")
|
||||
assert order.index("b") < order.index("c")
|
||||
|
||||
|
||||
def test_engine_evaluate_all_for_record() -> None:
|
||||
"""Formula-to-formula dependency: c = b + 1, b = a + 1, a = 5 → c = 7"""
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("a", "=5")
|
||||
engine.add_formula("b", "={a} + 1")
|
||||
engine.add_formula("c", "={b} + 1")
|
||||
results = engine.evaluate_all_for_record(row_values={})
|
||||
assert results["a"] == 5
|
||||
assert results["b"] == 6
|
||||
assert results["c"] == 7
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Circular reference detection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_circular_reference_detected() -> None:
|
||||
"""f1 = f2 + 1, f2 = f1 + 1 → CircularReferenceError"""
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("f1", "={f2} + 1")
|
||||
with pytest.raises(CircularReferenceError):
|
||||
engine.add_formula("f2", "={f1} + 1")
|
||||
|
||||
|
||||
def test_circular_reference_rollback() -> None:
|
||||
"""When cycle is detected, the formula is not added (rollback)."""
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("f1", "={f2} + 1")
|
||||
with pytest.raises(CircularReferenceError):
|
||||
engine.add_formula("f2", "={f1} + 1")
|
||||
# f2 should not be in the engine
|
||||
assert "f2" not in engine._formulas
|
||||
assert "f2" not in engine._dag
|
||||
|
||||
|
||||
def test_self_reference_detected() -> None:
|
||||
"""f1 = f1 + 1 → CircularReferenceError"""
|
||||
engine = FormulaEngine()
|
||||
with pytest.raises(CircularReferenceError):
|
||||
engine.add_formula("f1", "={f1} + 1")
|
||||
|
||||
|
||||
def test_remove_formula_breaks_cycle() -> None:
|
||||
"""Remove a formula, then the cycle can be broken."""
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("f1", "={f2} + 1")
|
||||
# Can't add f2 = f1 + 1 (cycle)
|
||||
with pytest.raises(CircularReferenceError):
|
||||
engine.add_formula("f2", "={f1} + 1")
|
||||
# Remove f1, now f2 can be added standalone
|
||||
engine.remove_formula("f1")
|
||||
engine.add_formula("f2", "=42")
|
||||
assert engine.evaluate("f2", {}) == 42
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Edge cases
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_evaluate_missing_field_value_is_none() -> None:
|
||||
"""Missing field values are None — arithmetic on None raises TypeError."""
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("calc", "={missing_field} + 1")
|
||||
# The engine passes None for missing fields (row_values.get returns None)
|
||||
with pytest.raises(TypeError):
|
||||
engine.evaluate("calc", row_values={})
|
||||
|
||||
|
||||
def test_aggregate_ignores_none_and_empty() -> None:
|
||||
"""SUM ignores None and empty string values."""
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("total", "=SUM({f1})")
|
||||
result = engine.evaluate("total", row_values={}, column_values={"f1": [1, None, 2, "", 3]})
|
||||
assert result == 6
|
||||
|
||||
|
||||
def test_division_by_zero_returns_error_in_evaluate_all() -> None:
|
||||
"""Division by zero is caught in evaluate_all_for_record, returns error dict."""
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("calc", "={f1} / 0")
|
||||
results = engine.evaluate_all_for_record(row_values={"f1": 10})
|
||||
assert "__error" in results["calc"]
|
||||
|
||||
|
||||
def test_engine_with_uuid_field_ids() -> None:
|
||||
"""Field IDs with hyphens (UUIDs) work correctly."""
|
||||
fid = "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
|
||||
engine = FormulaEngine()
|
||||
engine.add_formula("calc", f"={{{fid}}} * 2")
|
||||
result = engine.evaluate("calc", row_values={fid: 21})
|
||||
assert result == 42
|
||||
|
|
@ -0,0 +1,199 @@
|
|||
"""Tests for the formula parser (KTD7 security + parsing).
|
||||
|
||||
Test-first per U3 execution note: parser, security constraints, and cycle
|
||||
detection tests are written before the engine/recalc worker.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from agentkit.bitable.formula.parser import (
|
||||
FormulaParseError,
|
||||
FormulaSecurityError,
|
||||
UnknownFunctionError,
|
||||
evaluate_ast,
|
||||
parse_formula,
|
||||
)
|
||||
|
||||
ALLOWED = {"SUM", "AVG", "COUNT", "MIN", "MAX", "ABS", "ROUND", "IF", "LEN", "CONCAT"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Parsing happy paths
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_parse_simple_arithmetic() -> None:
|
||||
tree, mapping = parse_formula("=1+2*3", ALLOWED)
|
||||
assert mapping == {}
|
||||
result = evaluate_ast(tree, {}, {})
|
||||
assert result == 7
|
||||
|
||||
|
||||
def test_parse_strips_equals_prefix() -> None:
|
||||
tree1, _ = parse_formula("=1+1", ALLOWED)
|
||||
tree2, _ = parse_formula("1+1", ALLOWED)
|
||||
assert evaluate_ast(tree1, {}, {}) == evaluate_ast(tree2, {}, {}) == 2
|
||||
|
||||
|
||||
def test_parse_field_reference() -> None:
|
||||
tree, mapping = parse_formula("={field_abc} + 1", ALLOWED)
|
||||
assert "field_abc" in mapping.values()
|
||||
# Safe name is prefixed with _f_
|
||||
safe_name = next(k for k, v in mapping.items() if v == "field_abc")
|
||||
result = evaluate_ast(tree, {safe_name: 41}, {})
|
||||
assert result == 42
|
||||
|
||||
|
||||
def test_parse_uuid_field_reference() -> None:
|
||||
"""Field IDs are UUIDs with hyphens — must be substituted to safe names."""
|
||||
fid = "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
|
||||
tree, mapping = parse_formula(f"={{{fid}}} * 2", ALLOWED)
|
||||
# The mapping should have a safe name → original UUID
|
||||
assert fid in mapping.values()
|
||||
# Evaluate using the safe name (prefixed with _f_)
|
||||
safe_name = next(k for k, v in mapping.items() if v == fid)
|
||||
assert safe_name.startswith("_f_")
|
||||
result = evaluate_ast(tree, {safe_name: 21}, {})
|
||||
assert result == 42
|
||||
|
||||
|
||||
def test_parse_string_concatenation() -> None:
|
||||
tree, _ = parse_formula('="hello" + " " + "world"', ALLOWED)
|
||||
assert evaluate_ast(tree, {}, {}) == "hello world"
|
||||
|
||||
|
||||
def test_parse_conditional_ifexp() -> None:
|
||||
tree, _ = parse_formula("=1 if True else 2", ALLOWED)
|
||||
assert evaluate_ast(tree, {}, {}) == 1
|
||||
|
||||
|
||||
def test_parse_comparison() -> None:
|
||||
tree, mapping = parse_formula("={f} > 5", ALLOWED)
|
||||
safe_name = next(k for k, v in mapping.items() if v == "f")
|
||||
assert evaluate_ast(tree, {safe_name: 10}, {}) is True
|
||||
assert evaluate_ast(tree, {safe_name: 3}, {}) is False
|
||||
|
||||
|
||||
def test_parse_boolean_ops() -> None:
|
||||
tree, _ = parse_formula("=True and False", ALLOWED)
|
||||
assert evaluate_ast(tree, {}, {}) is False
|
||||
tree2, _ = parse_formula("=True or False", ALLOWED)
|
||||
assert evaluate_ast(tree2, {}, {}) is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Function calls
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_parse_function_call_sum() -> None:
|
||||
tree, mapping = parse_formula("=SUM({f1})", ALLOWED)
|
||||
safe_name = next(k for k, v in mapping.items() if v == "f1")
|
||||
result = evaluate_ast(tree, {safe_name: [1, 2, 3]}, {"SUM": sum})
|
||||
assert result == 6
|
||||
|
||||
|
||||
def test_parse_function_call_concat() -> None:
|
||||
tree, mapping = parse_formula('=CONCAT({f1}, "-", {f2})', ALLOWED)
|
||||
safe_f1 = next(k for k, v in mapping.items() if v == "f1")
|
||||
safe_f2 = next(k for k, v in mapping.items() if v == "f2")
|
||||
result = evaluate_ast(
|
||||
tree, {safe_f1: "a", safe_f2: "b"}, {"CONCAT": lambda *a: "".join(str(x) for x in a)}
|
||||
)
|
||||
assert result == "a-b"
|
||||
|
||||
|
||||
def test_parse_nested_function_calls() -> None:
|
||||
tree, _ = parse_formula("=ABS(-5) + ROUND(3.7, 0)", ALLOWED)
|
||||
funcs = {"ABS": abs, "ROUND": round}
|
||||
result = evaluate_ast(tree, {}, funcs)
|
||||
assert result == 9 # 5 + 4
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# KTD7 Security — disallowed nodes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_security_rejects_attribute_access() -> None:
|
||||
"""__import__('os') is rejected — it's a Call to an unregistered function.
|
||||
(Attribute access like os.system would be caught by the Attribute node check,
|
||||
but __import__ is caught earlier as an unknown function.)"""
|
||||
with pytest.raises((FormulaSecurityError, UnknownFunctionError)):
|
||||
parse_formula("=__import__('os')", ALLOWED)
|
||||
|
||||
|
||||
def test_security_rejects_attribute_chain() -> None:
|
||||
"""Attribute access like ''.join([]) is rejected by the Attribute node check."""
|
||||
with pytest.raises(FormulaSecurityError):
|
||||
parse_formula("=''.join([])", ALLOWED)
|
||||
|
||||
|
||||
def test_security_rejects_lambda() -> None:
|
||||
with pytest.raises(FormulaSecurityError):
|
||||
parse_formula("=(lambda: 1)()", ALLOWED)
|
||||
|
||||
|
||||
def test_security_rejects_subscript() -> None:
|
||||
with pytest.raises(FormulaSecurityError):
|
||||
parse_formula("=[1,2,3][0]", ALLOWED)
|
||||
|
||||
|
||||
def test_security_rejects_assignment() -> None:
|
||||
"""Assignment is a statement, not an expression — rejected at parse stage."""
|
||||
with pytest.raises((FormulaSecurityError, FormulaParseError)):
|
||||
parse_formula("=x = 1", ALLOWED)
|
||||
|
||||
|
||||
def test_unknown_function_rejected() -> None:
|
||||
with pytest.raises(UnknownFunctionError):
|
||||
parse_formula("=UNKNOWN(1)", ALLOWED)
|
||||
|
||||
|
||||
def test_eval_function_rejected_if_not_registered() -> None:
|
||||
"""eval is not in the registry → UnknownFunctionError."""
|
||||
with pytest.raises(UnknownFunctionError):
|
||||
parse_formula("=eval('1+1')", ALLOWED)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Error paths
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_parse_error_unbalanced_parens() -> None:
|
||||
with pytest.raises(FormulaParseError):
|
||||
parse_formula("=(1+2", ALLOWED)
|
||||
|
||||
|
||||
def test_parse_error_empty_formula() -> None:
|
||||
with pytest.raises(FormulaParseError):
|
||||
parse_formula("=", ALLOWED)
|
||||
|
||||
|
||||
def test_parse_error_empty_string() -> None:
|
||||
with pytest.raises(FormulaParseError):
|
||||
parse_formula("", ALLOWED)
|
||||
|
||||
|
||||
def test_evaluate_unknown_field_ref_raises() -> None:
|
||||
tree, _ = parse_formula("={nonexistent} + 1", ALLOWED)
|
||||
with pytest.raises(FormulaParseError, match="Unknown field reference"):
|
||||
evaluate_ast(tree, {}, {})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Mixed aggregate + row context
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_mixed_aggregate_and_row_context() -> None:
|
||||
"""={f1} + SUM({f2}) — row f1 + column f2 sum."""
|
||||
tree, mapping = parse_formula("={f1} + SUM({f2})", ALLOWED)
|
||||
safe_f1 = next(k for k, v in mapping.items() if v == "f1")
|
||||
safe_f2 = next(k for k, v in mapping.items() if v == "f2")
|
||||
# f1 is a row value (scalar), f2 is a column value (list)
|
||||
result = evaluate_ast(tree, {safe_f1: 10, safe_f2: [1, 2, 3]}, {"SUM": sum})
|
||||
assert result == 16 # 10 + 6
|
||||
|
|
@ -0,0 +1,182 @@
|
|||
"""Tests for Excel ingestion (U4).
|
||||
|
||||
Tests parse_excel_bytes with in-memory .xlsx files created via openpyxl.
|
||||
No PostgreSQL required — these are pure parsing tests.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
from datetime import datetime
|
||||
|
||||
import pytest
|
||||
|
||||
from agentkit.bitable.ingestion.excel import parse_excel_bytes
|
||||
|
||||
pytestmark = pytest.mark.postgres # Reuse the same PG test group for consistency
|
||||
|
||||
|
||||
def _make_xlsx(
|
||||
sheets: dict[str, list[list]],
|
||||
) -> bytes:
|
||||
"""Create an in-memory .xlsx file from sheet data.
|
||||
|
||||
Args:
|
||||
sheets: {sheet_name: [[row1_col1, row1_col2], [row2_col1, ...]]}
|
||||
"""
|
||||
from openpyxl import Workbook
|
||||
|
||||
wb = Workbook()
|
||||
# Remove default sheet
|
||||
wb.remove(wb.active)
|
||||
for name, rows in sheets.items():
|
||||
ws = wb.create_sheet(title=name)
|
||||
for row in rows:
|
||||
ws.append(row)
|
||||
buf = io.BytesIO()
|
||||
wb.save(buf)
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Happy path: basic parsing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_parse_simple_sheet() -> None:
|
||||
"""One sheet with header + 2 data rows → correct columns, types, records."""
|
||||
xlsx = _make_xlsx(
|
||||
{
|
||||
"Sheet1": [
|
||||
["name", "age", "city"],
|
||||
["Alice", 30, "NYC"],
|
||||
["Bob", 25, "LA"],
|
||||
]
|
||||
}
|
||||
)
|
||||
sheets = parse_excel_bytes(xlsx)
|
||||
assert len(sheets) == 1
|
||||
sheet = sheets[0]
|
||||
assert sheet.name == "Sheet1"
|
||||
assert sheet.columns == ["name", "age", "city"]
|
||||
assert sheet.field_types == ["text", "number", "text"]
|
||||
assert len(sheet.records) == 2
|
||||
assert sheet.records[0] == {"name": "Alice", "age": 30, "city": "NYC"}
|
||||
assert sheet.records[1] == {"name": "Bob", "age": 25, "city": "LA"}
|
||||
|
||||
|
||||
def test_parse_multiple_sheets() -> None:
|
||||
"""Multiple sheets → multiple ParsedSheet objects."""
|
||||
xlsx = _make_xlsx(
|
||||
{
|
||||
"Users": [["id", "name"], [1, "Alice"]],
|
||||
"Orders": [["order_id", "amount"], [101, 99.9]],
|
||||
}
|
||||
)
|
||||
sheets = parse_excel_bytes(xlsx)
|
||||
assert len(sheets) == 2
|
||||
assert sheets[0].name == "Users"
|
||||
assert sheets[1].name == "Orders"
|
||||
assert sheets[1].records[0]["amount"] == 99.9
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Type inference
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_type_inference_all_number() -> None:
|
||||
"""Column with all integers → 'number'."""
|
||||
xlsx = _make_xlsx({"S": [["val"], [1], [2], [3]]})
|
||||
sheets = parse_excel_bytes(xlsx)
|
||||
assert sheets[0].field_types == ["number"]
|
||||
|
||||
|
||||
def test_type_inference_mixed_text_number() -> None:
|
||||
"""Column with mixed text and number → 'text'."""
|
||||
xlsx = _make_xlsx({"S": [["val"], [1], ["two"], [3]]})
|
||||
sheets = parse_excel_bytes(xlsx)
|
||||
assert sheets[0].field_types == ["text"]
|
||||
|
||||
|
||||
def test_type_inference_date_column() -> None:
|
||||
"""Column with all datetime values → 'date'."""
|
||||
xlsx = _make_xlsx({"S": [["when"], [datetime(2024, 1, 1)], [datetime(2024, 6, 15)]]})
|
||||
sheets = parse_excel_bytes(xlsx)
|
||||
assert sheets[0].field_types == ["date"]
|
||||
assert "2024-01-01" in sheets[0].records[0]["when"]
|
||||
|
||||
|
||||
def test_type_inference_empty_column() -> None:
|
||||
"""Column with no values → 'text' (safe default)."""
|
||||
xlsx = _make_xlsx({"S": [["a", "b"], [1, None], [2, None]]})
|
||||
sheets = parse_excel_bytes(xlsx)
|
||||
assert sheets[0].field_types == ["number", "text"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Edge cases
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_empty_sheet_skipped() -> None:
|
||||
"""Completely empty sheet → not included in results."""
|
||||
xlsx = _make_xlsx({"Empty": [], "Data": [["x"], [1]]})
|
||||
sheets = parse_excel_bytes(xlsx)
|
||||
assert len(sheets) == 1
|
||||
assert sheets[0].name == "Data"
|
||||
|
||||
|
||||
def test_header_only_no_data_rows() -> None:
|
||||
"""Sheet with only a header row → 0 records, fields still created."""
|
||||
xlsx = _make_xlsx({"S": [["name", "age"]]})
|
||||
sheets = parse_excel_bytes(xlsx)
|
||||
assert len(sheets) == 1
|
||||
assert sheets[0].columns == ["name", "age"]
|
||||
assert len(sheets[0].records) == 0
|
||||
|
||||
|
||||
def test_duplicate_headers_deduplicated() -> None:
|
||||
"""Duplicate header names → suffixed with _1, _2, etc."""
|
||||
xlsx = _make_xlsx({"S": [["name", "name"], ["Alice", "Bob"]]})
|
||||
sheets = parse_excel_bytes(xlsx)
|
||||
assert sheets[0].columns == ["name", "name_1"]
|
||||
|
||||
|
||||
def test_none_header_replaced() -> None:
|
||||
"""None header value → auto-generated column name."""
|
||||
xlsx = _make_xlsx({"S": [[None, "real"], [1, 2]]})
|
||||
sheets = parse_excel_bytes(xlsx)
|
||||
assert sheets[0].columns[0] == "col_0"
|
||||
assert sheets[0].columns[1] == "real"
|
||||
|
||||
|
||||
def test_corrupt_file_raises_value_error() -> None:
|
||||
"""Non-xlsx bytes → ValueError with clear message."""
|
||||
with pytest.raises(ValueError, match="Failed to parse"):
|
||||
parse_excel_bytes(b"not an excel file")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Merged cells (known limitation)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_merged_cells_only_top_left_has_value() -> None:
|
||||
"""Merged cell: only top-left has value, others are None (known limitation)."""
|
||||
from openpyxl import Workbook
|
||||
|
||||
wb = Workbook()
|
||||
ws = wb.active
|
||||
ws.title = "Merged"
|
||||
ws.append(["a", "b", "c"])
|
||||
ws.append([1, 2, 3])
|
||||
ws.merge_cells("A2:B2") # merge A2:B2 — only A2 has value
|
||||
buf = io.BytesIO()
|
||||
wb.save(buf)
|
||||
|
||||
sheets = parse_excel_bytes(buf.getvalue())
|
||||
rec = sheets[0].records[0]
|
||||
# A2 has value 1, B2 is None (merged cell limitation)
|
||||
assert rec["a"] == 1
|
||||
assert rec["b"] is None
|
||||
|
|
@ -0,0 +1,303 @@
|
|||
"""Tests for bitable Pydantic v2 data models (U1).
|
||||
|
||||
Covers: enum values, round-trip serialization, field config shapes per
|
||||
field_type, Record.values empty-dict legality, default factories.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
|
||||
from agentkit.bitable.models import (
|
||||
Field,
|
||||
FieldOwner,
|
||||
FieldType,
|
||||
Record,
|
||||
RecalcStatus,
|
||||
RecalcTask,
|
||||
Table,
|
||||
View,
|
||||
ViewType,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Enums
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_field_type_values() -> None:
|
||||
"""FieldType has the 9 supported types with correct string values."""
|
||||
expected = {
|
||||
"text",
|
||||
"number",
|
||||
"date",
|
||||
"select",
|
||||
"multiselect",
|
||||
"attachment",
|
||||
"image",
|
||||
"formula",
|
||||
"lookup",
|
||||
}
|
||||
assert {ft.value for ft in FieldType} == expected
|
||||
|
||||
|
||||
def test_field_owner_values() -> None:
|
||||
"""FieldOwner distinguishes agent vs user (drives upsert merge)."""
|
||||
assert FieldOwner.agent.value == "agent"
|
||||
assert FieldOwner.user.value == "user"
|
||||
|
||||
|
||||
def test_view_type_values() -> None:
|
||||
"""ViewType enumerates the 5 view kinds (v1 only grid is implemented)."""
|
||||
assert {vt.value for vt in ViewType} == {"grid", "kanban", "gantt", "gallery", "form"}
|
||||
|
||||
|
||||
def test_recalc_status_lifecycle() -> None:
|
||||
"""RecalcStatus covers the full recalc lifecycle."""
|
||||
assert {rs.value for rs in RecalcStatus} == {"pending", "calculating", "done", "error"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Table
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_table_minimal_construction() -> None:
|
||||
"""Table requires id + name; other fields have defaults."""
|
||||
table = Table(id="t1", name="Orders")
|
||||
assert table.id == "t1"
|
||||
assert table.name == "Orders"
|
||||
assert table.description == ""
|
||||
assert table.primary_key_field_id is None
|
||||
assert table.owner_user_id is None
|
||||
assert isinstance(table.created_at, datetime)
|
||||
assert isinstance(table.updated_at, datetime)
|
||||
|
||||
|
||||
def test_table_round_trip() -> None:
|
||||
"""Table serializes to dict and re-parses losslessly."""
|
||||
table = Table(
|
||||
id="t1",
|
||||
name="Orders",
|
||||
description="Customer orders",
|
||||
primary_key_field_id="f_pk",
|
||||
owner_user_id="u1",
|
||||
)
|
||||
data = table.model_dump(mode="json")
|
||||
restored = Table.model_validate(data)
|
||||
assert restored == table
|
||||
|
||||
|
||||
def test_table_requires_id_and_name() -> None:
|
||||
"""Table requires id and name (non-optional)."""
|
||||
with pytest.raises(ValidationError):
|
||||
Table(name="no id") # type: ignore[call-arg]
|
||||
with pytest.raises(ValidationError):
|
||||
Table(id="t1") # type: ignore[call-arg]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Field
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_field_text_default_config() -> None:
|
||||
"""Text field has empty config by default and user owner."""
|
||||
field = Field(id="f1", table_id="t1", name="Title", field_type=FieldType.text)
|
||||
assert field.config == {}
|
||||
assert field.owner == FieldOwner.user
|
||||
|
||||
|
||||
def test_field_select_config_shape() -> None:
|
||||
"""Select field config carries options list."""
|
||||
field = Field(
|
||||
id="f1",
|
||||
table_id="t1",
|
||||
name="Status",
|
||||
field_type=FieldType.select,
|
||||
config={"options": [{"label": "Open", "value": "open"}]},
|
||||
owner=FieldOwner.agent,
|
||||
)
|
||||
assert field.config["options"][0]["value"] == "open"
|
||||
assert field.owner == FieldOwner.agent
|
||||
|
||||
|
||||
def test_field_formula_config_shape() -> None:
|
||||
"""Formula field config carries formula_expr."""
|
||||
field = Field(
|
||||
id="f1",
|
||||
table_id="t1",
|
||||
name="Total",
|
||||
field_type=FieldType.formula,
|
||||
config={"formula_expr": "=SUM({f_price})"},
|
||||
)
|
||||
assert field.config["formula_expr"] == "=SUM({f_price})"
|
||||
|
||||
|
||||
def test_field_lookup_config_shape() -> None:
|
||||
"""Lookup field config carries lookup_target with table/field ids."""
|
||||
field = Field(
|
||||
id="f1",
|
||||
table_id="t1",
|
||||
name="Customer Name",
|
||||
field_type=FieldType.lookup,
|
||||
config={
|
||||
"lookup_target": {
|
||||
"table_id": "t_customers",
|
||||
"field_id": "f_name",
|
||||
"filter_field_id": "f_id",
|
||||
"filter_value": "cust-123",
|
||||
}
|
||||
},
|
||||
)
|
||||
assert field.config["lookup_target"]["field_id"] == "f_name"
|
||||
|
||||
|
||||
def test_field_round_trip() -> None:
|
||||
"""Field serializes and re-parses losslessly across types."""
|
||||
field = Field(
|
||||
id="f1",
|
||||
table_id="t1",
|
||||
name="Score",
|
||||
field_type=FieldType.number,
|
||||
config={"precision": 2},
|
||||
owner=FieldOwner.agent,
|
||||
)
|
||||
restored = Field.model_validate(field.model_dump(mode="json"))
|
||||
assert restored == field
|
||||
assert restored.field_type == FieldType.number
|
||||
assert restored.owner == FieldOwner.agent
|
||||
|
||||
|
||||
def test_field_type_accepts_string() -> None:
|
||||
"""FieldType coerces from string (JSON round-trip scenario)."""
|
||||
field = Field(id="f1", table_id="t1", name="X", field_type="number") # type: ignore[arg-type]
|
||||
assert field.field_type == FieldType.number
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Record
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_record_empty_values_allowed() -> None:
|
||||
"""Record.values defaults to empty dict (new row before data entry)."""
|
||||
record = Record(id="r1", table_id="t1")
|
||||
assert record.values == {}
|
||||
|
||||
|
||||
def test_record_values_round_trip() -> None:
|
||||
"""Record.values (JSONB-shaped dict) round-trips through JSON."""
|
||||
record = Record(
|
||||
id="r1",
|
||||
table_id="t1",
|
||||
values={"f_name": "Alice", "f_age": 30, "f_tags": ["a", "b"]},
|
||||
)
|
||||
restored = Record.model_validate(record.model_dump(mode="json"))
|
||||
assert restored.values == record.values
|
||||
assert restored.values["f_tags"] == ["a", "b"]
|
||||
|
||||
|
||||
def test_record_values_with_null() -> None:
|
||||
"""Record.values can carry None for unset fields."""
|
||||
record = Record(id="r1", table_id="t1", values={"f_name": None})
|
||||
assert record.values["f_name"] is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# View
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_view_defaults_to_grid() -> None:
|
||||
"""View defaults to grid type with empty config."""
|
||||
view = View(id="v1", table_id="t1", name="All")
|
||||
assert view.view_type == ViewType.grid
|
||||
assert view.config == {}
|
||||
|
||||
|
||||
def test_view_round_trip() -> None:
|
||||
"""View with filter/sort config round-trips."""
|
||||
view = View(
|
||||
id="v1",
|
||||
table_id="t1",
|
||||
name="Open only",
|
||||
view_type=ViewType.grid,
|
||||
config={
|
||||
"filters": [{"field_id": "f_status", "op": "eq", "value": "open"}],
|
||||
"sorts": [{"field_id": "f_created", "direction": "desc"}],
|
||||
"hidden_fields": ["f_internal"],
|
||||
},
|
||||
)
|
||||
restored = View.model_validate(view.model_dump(mode="json"))
|
||||
assert restored == view
|
||||
assert restored.config["filters"][0]["op"] == "eq"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# RecalcTask
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_recalc_task_defaults() -> None:
|
||||
"""RecalcTask defaults to pending status, no error, no completed_at."""
|
||||
task = RecalcTask(id="q1", table_id="t1", record_id="r1", field_id="f1")
|
||||
assert task.status == RecalcStatus.pending
|
||||
assert task.error_message is None
|
||||
assert task.completed_at is None
|
||||
assert isinstance(task.queued_at, datetime)
|
||||
|
||||
|
||||
def test_recalc_task_error_state() -> None:
|
||||
"""RecalcTask in error state carries message and completed_at."""
|
||||
task = RecalcTask(
|
||||
id="q1",
|
||||
table_id="t1",
|
||||
record_id="r1",
|
||||
field_id="f1",
|
||||
status=RecalcStatus.error,
|
||||
error_message="division by zero",
|
||||
completed_at=datetime.now(timezone.utc),
|
||||
)
|
||||
assert task.status == RecalcStatus.error
|
||||
assert task.error_message == "division by zero"
|
||||
|
||||
|
||||
def test_recalc_task_round_trip() -> None:
|
||||
"""RecalcTask round-trips through JSON."""
|
||||
task = RecalcTask(
|
||||
id="q1",
|
||||
table_id="t1",
|
||||
record_id="r1",
|
||||
field_id="f1",
|
||||
status=RecalcStatus.done,
|
||||
)
|
||||
restored = RecalcTask.model_validate(task.model_dump(mode="json"))
|
||||
assert restored == task
|
||||
assert restored.status == RecalcStatus.done
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# from_attributes (ORM row compatibility)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_table_from_attributes() -> None:
|
||||
"""Table.model_validate accepts an ORM-like object (from_attributes)."""
|
||||
|
||||
class _Row:
|
||||
id = "t1"
|
||||
name = "Orders"
|
||||
description = "desc"
|
||||
primary_key_field_id = None
|
||||
owner_user_id = None
|
||||
created_at = datetime.now(timezone.utc)
|
||||
updated_at = datetime.now(timezone.utc)
|
||||
|
||||
table = Table.model_validate(_Row())
|
||||
assert table.id == "t1"
|
||||
assert table.name == "Orders"
|
||||
|
|
@ -0,0 +1,330 @@
|
|||
"""Tests for the async recalc pipeline (U3).
|
||||
|
||||
Requires PostgreSQL — marked ``postgres``. Tests the full pipeline:
|
||||
record write → recalc enqueue → worker processing → formula value written.
|
||||
|
||||
Also covers: crash recovery, deduplication, and error handling.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
|
||||
from agentkit.bitable.models import FieldOwner, FieldType, RecalcStatus
|
||||
from agentkit.bitable.recalc_worker import RecalcWorker
|
||||
from agentkit.bitable.service import BitableService
|
||||
|
||||
pytestmark = pytest.mark.postgres
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helper: process all pending recalc tasks synchronously
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _process_all_pending(service: BitableService) -> None:
|
||||
"""Process all pending recalc tasks (for testing without background worker)."""
|
||||
tasks = await service.get_pending_recalc_tasks(limit=100)
|
||||
for task in tasks:
|
||||
await service.process_recalc_task(task)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Happy path: formula recalc after record write
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_recalc_simple_formula_after_create(bitable_service: BitableService) -> None:
|
||||
"""Create a record with data → formula field gets recalculated."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
src_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="src", field_type=FieldType.number, owner=FieldOwner.agent
|
||||
)
|
||||
calc_field = await bitable_service.create_field(
|
||||
table_id=table.id,
|
||||
name="calc",
|
||||
field_type=FieldType.formula,
|
||||
config={"formula_expr": f"={{{src_field.id}}} * 2"},
|
||||
)
|
||||
|
||||
# Create a record — should trigger recalc
|
||||
record = await bitable_service.create_record(table_id=table.id, values={src_field.id: 21})
|
||||
|
||||
# Process pending recalc tasks
|
||||
await _process_all_pending(bitable_service)
|
||||
|
||||
# Verify formula result was written
|
||||
updated = await bitable_service.get_record(record.id)
|
||||
assert updated is not None
|
||||
assert updated.values[calc_field.id] == 42
|
||||
|
||||
|
||||
async def test_recalc_aggregate_formula(bitable_service: BitableService) -> None:
|
||||
"""SUM aggregate formula recalculates correctly across all records."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
src_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="amt", field_type=FieldType.number, owner=FieldOwner.agent
|
||||
)
|
||||
total_field = await bitable_service.create_field(
|
||||
table_id=table.id,
|
||||
name="total",
|
||||
field_type=FieldType.formula,
|
||||
config={"formula_expr": f"=SUM({{{src_field.id}}})"},
|
||||
)
|
||||
|
||||
# Create multiple records
|
||||
for amt in [10, 20, 30]:
|
||||
await bitable_service.create_record(table_id=table.id, values={src_field.id: amt})
|
||||
|
||||
# Process all pending recalc tasks
|
||||
await _process_all_pending(bitable_service)
|
||||
|
||||
# Each record's total field should be 60 (sum of all)
|
||||
records, _ = await bitable_service.list_records(table.id)
|
||||
assert len(records) == 3
|
||||
for rec in records:
|
||||
assert rec.values[total_field.id] == 60
|
||||
|
||||
|
||||
async def test_recalc_after_upsert(bitable_service: BitableService) -> None:
|
||||
"""Upsert triggers recalc for affected formula fields."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
pk_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="id", field_type=FieldType.text, owner=FieldOwner.agent
|
||||
)
|
||||
data_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="data", field_type=FieldType.number, owner=FieldOwner.agent
|
||||
)
|
||||
calc_field = await bitable_service.create_field(
|
||||
table_id=table.id,
|
||||
name="doubled",
|
||||
field_type=FieldType.formula,
|
||||
config={"formula_expr": f"={{{data_field.id}}} * 2"},
|
||||
)
|
||||
await bitable_service.update_table(table.id, primary_key_field_id=pk_field.id)
|
||||
|
||||
# Upsert a record
|
||||
await bitable_service.upsert_records(
|
||||
table.id,
|
||||
[{pk_field.id: "r1", data_field.id: 15}],
|
||||
pk_field.id,
|
||||
)
|
||||
|
||||
# Process recalc
|
||||
await _process_all_pending(bitable_service)
|
||||
|
||||
# Verify formula result
|
||||
records, _ = await bitable_service.list_records(table.id)
|
||||
assert len(records) == 1
|
||||
assert records[0].values[calc_field.id] == 30
|
||||
|
||||
|
||||
async def test_recalc_formula_chain(bitable_service: BitableService) -> None:
|
||||
"""Formula-to-formula dependency: c = b*2, b = a*2 → c = a*4."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
src = await bitable_service.create_field(
|
||||
table_id=table.id, name="a", field_type=FieldType.number, owner=FieldOwner.agent
|
||||
)
|
||||
mid = await bitable_service.create_field(
|
||||
table_id=table.id,
|
||||
name="b",
|
||||
field_type=FieldType.formula,
|
||||
config={"formula_expr": f"={{{src.id}}} * 2"},
|
||||
)
|
||||
top = await bitable_service.create_field(
|
||||
table_id=table.id,
|
||||
name="c",
|
||||
field_type=FieldType.formula,
|
||||
config={"formula_expr": f"={{{mid.id}}} * 2"},
|
||||
)
|
||||
|
||||
await bitable_service.create_record(table_id=table.id, values={src.id: 5})
|
||||
|
||||
# Process recalc — may need multiple passes for formula chains
|
||||
# ponytail: The current implementation processes tasks in queue order, not
|
||||
# topological order. For formula chains, we may need to process twice:
|
||||
# first pass computes b, second pass computes c (which depends on b).
|
||||
await _process_all_pending(bitable_service)
|
||||
await _process_all_pending(bitable_service)
|
||||
|
||||
records, _ = await bitable_service.list_records(table.id)
|
||||
assert len(records) == 1
|
||||
rec = records[0]
|
||||
assert rec.values[mid.id] == 10 # 5 * 2
|
||||
assert rec.values[top.id] == 20 # 10 * 2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Crash recovery
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_crash_recovery_resets_calculating_tasks(
|
||||
bitable_service: BitableService,
|
||||
) -> None:
|
||||
"""Stale 'calculating' tasks are reset to 'pending' on worker start."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
src = await bitable_service.create_field(
|
||||
table_id=table.id, name="s", field_type=FieldType.number, owner=FieldOwner.agent
|
||||
)
|
||||
calc = await bitable_service.create_field(
|
||||
table_id=table.id,
|
||||
name="c",
|
||||
field_type=FieldType.formula,
|
||||
config={"formula_expr": f"={{{src.id}}} + 1"},
|
||||
)
|
||||
|
||||
record = await bitable_service.create_record(table_id=table.id, values={src.id: 10})
|
||||
|
||||
# create_record already enqueued a recalc task — get it from pending
|
||||
tasks = await bitable_service.get_pending_recalc_tasks()
|
||||
assert len(tasks) == 1
|
||||
task = tasks[0]
|
||||
|
||||
from agentkit.bitable.repository import BitableRepository
|
||||
|
||||
repo = BitableRepository(bitable_service._db)
|
||||
await repo.update_recalc_status(task.id, RecalcStatus.calculating)
|
||||
|
||||
# Verify it's stuck in calculating
|
||||
tasks = await bitable_service.get_pending_recalc_tasks()
|
||||
assert len(tasks) == 0 # not pending, it's calculating
|
||||
|
||||
# Crash recovery
|
||||
reset_count = await bitable_service.reset_stale_recalc_tasks()
|
||||
assert reset_count == 1
|
||||
|
||||
# Now it should be pending again
|
||||
tasks = await bitable_service.get_pending_recalc_tasks()
|
||||
assert len(tasks) == 1
|
||||
|
||||
# Process it
|
||||
await _process_all_pending(bitable_service)
|
||||
|
||||
# Verify result
|
||||
rec = await bitable_service.get_record(record.id)
|
||||
assert rec is not None
|
||||
assert rec.values[calc.id] == 11
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Deduplication
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_recalc_deduplication(bitable_service: BitableService) -> None:
|
||||
"""Same (record_id, field_id) enqueued twice → only one task in queue."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
src = await bitable_service.create_field(
|
||||
table_id=table.id, name="s", field_type=FieldType.number, owner=FieldOwner.agent
|
||||
)
|
||||
calc = await bitable_service.create_field(
|
||||
table_id=table.id,
|
||||
name="c",
|
||||
field_type=FieldType.formula,
|
||||
config={"formula_expr": f"={{{src.id}}} + 1"},
|
||||
)
|
||||
|
||||
record = await bitable_service.create_record(table_id=table.id, values={src.id: 10})
|
||||
|
||||
# The create_record already enqueued one task. Enqueue again manually.
|
||||
task2 = await bitable_service.trigger_recalc(table.id, record.id, calc.id)
|
||||
# Should return None (duplicate, ON CONFLICT DO NOTHING)
|
||||
assert task2 is None
|
||||
|
||||
# Only one pending task
|
||||
tasks = await bitable_service.get_pending_recalc_tasks()
|
||||
assert len(tasks) == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Error handling
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_recalc_error_marks_task_as_error(bitable_service: BitableService) -> None:
|
||||
"""Formula with division by zero marks task as error."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
src = await bitable_service.create_field(
|
||||
table_id=table.id, name="s", field_type=FieldType.number, owner=FieldOwner.agent
|
||||
)
|
||||
calc = await bitable_service.create_field(
|
||||
table_id=table.id,
|
||||
name="c",
|
||||
field_type=FieldType.formula,
|
||||
config={"formula_expr": f"={{{src.id}}} / 0"},
|
||||
)
|
||||
|
||||
record = await bitable_service.create_record(table_id=table.id, values={src.id: 10})
|
||||
|
||||
# Process recalc — should fail with division by zero
|
||||
await _process_all_pending(bitable_service)
|
||||
|
||||
# Verify task is marked as error
|
||||
from sqlalchemy import text
|
||||
|
||||
db = bitable_service._db
|
||||
async with db.session_factory() as session:
|
||||
result = await session.execute(
|
||||
text(
|
||||
"SELECT status, error_message FROM bitable.bitable_recalc_queue "
|
||||
"WHERE record_id = :rid AND field_id = :fid"
|
||||
),
|
||||
{"rid": record.id, "fid": calc.id},
|
||||
)
|
||||
row = result.fetchone()
|
||||
assert row is not None
|
||||
assert row[0] == RecalcStatus.error.value
|
||||
assert "division" in row[1].lower() or "zero" in row[1].lower()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Worker lifecycle
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_worker_starts_and_stops(bitable_service: BitableService) -> None:
|
||||
"""RecalcWorker starts and stops gracefully."""
|
||||
worker = RecalcWorker(bitable_service._db, bitable_service, poll_interval=0.1)
|
||||
await worker.start()
|
||||
assert worker._task is not None
|
||||
assert worker._reaper_task is not None
|
||||
|
||||
# Let it run briefly
|
||||
await asyncio.sleep(0.2)
|
||||
|
||||
await worker.stop()
|
||||
assert worker._task is None
|
||||
assert worker._reaper_task is None
|
||||
|
||||
|
||||
async def test_worker_processes_tasks(bitable_service: BitableService) -> None:
|
||||
"""Background worker picks up and processes recalc tasks."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
src = await bitable_service.create_field(
|
||||
table_id=table.id, name="s", field_type=FieldType.number, owner=FieldOwner.agent
|
||||
)
|
||||
calc = await bitable_service.create_field(
|
||||
table_id=table.id,
|
||||
name="c",
|
||||
field_type=FieldType.formula,
|
||||
config={"formula_expr": f"={{{src.id}}} + 100"},
|
||||
)
|
||||
|
||||
record = await bitable_service.create_record(table_id=table.id, values={src.id: 5})
|
||||
|
||||
# Start worker — it should pick up the pending task
|
||||
worker = RecalcWorker(bitable_service._db, bitable_service, poll_interval=0.1)
|
||||
await worker.start()
|
||||
|
||||
# Wait for worker to process
|
||||
await asyncio.sleep(1.0)
|
||||
|
||||
await worker.stop()
|
||||
|
||||
# Verify formula was computed
|
||||
rec = await bitable_service.get_record(record.id)
|
||||
assert rec is not None
|
||||
assert rec.values[calc.id] == 105
|
||||
|
|
@ -0,0 +1,579 @@
|
|||
"""Tests for bitable REST API routes (U2).
|
||||
|
||||
Requires PostgreSQL — marked ``postgres``. Uses ``httpx.AsyncClient`` with
|
||||
``ASGITransport`` so the async DB engine and the HTTP client share one event
|
||||
loop (TestClient runs in a separate thread/loop, which breaks asyncpg's
|
||||
loop-bound connections).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from httpx import ASGITransport
|
||||
|
||||
from agentkit.bitable.service import BitableService
|
||||
from agentkit.server.routes import bitable as bitable_routes
|
||||
from agentkit.server.routes.bitable import require_bitable_auth
|
||||
|
||||
pytestmark = pytest.mark.postgres
|
||||
|
||||
TEST_USER_ID = "test-user-id"
|
||||
|
||||
|
||||
def _make_test_user() -> dict[str, Any]:
|
||||
return {"user_id": TEST_USER_ID, "username": "testuser", "role": "member"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app(bitable_service: BitableService) -> FastAPI:
|
||||
"""Test app with bitable_service on app.state and auth bypassed."""
|
||||
app = FastAPI()
|
||||
app.state.bitable_service = bitable_service
|
||||
app.include_router(bitable_routes.router, prefix="/api/v1")
|
||||
app.dependency_overrides[require_bitable_auth] = lambda: _make_test_user()
|
||||
return app
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def unauth_app(bitable_service: BitableService) -> FastAPI:
|
||||
"""App without auth override — simulates unauthenticated requests."""
|
||||
app = FastAPI()
|
||||
app.state.bitable_service = bitable_service
|
||||
app.include_router(bitable_routes.router, prefix="/api/v1")
|
||||
return app
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def no_service_app() -> FastAPI:
|
||||
"""App without bitable_service on state — simulates uninitialized subsystem."""
|
||||
app = FastAPI()
|
||||
app.include_router(bitable_routes.router, prefix="/api/v1")
|
||||
app.dependency_overrides[require_bitable_auth] = lambda: _make_test_user()
|
||||
return app
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def client(app: FastAPI) -> httpx.AsyncClient:
|
||||
"""Async HTTP client — shares event loop with async DB fixtures."""
|
||||
transport = ASGITransport(app=app)
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://test") as c:
|
||||
yield c
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def unauth_client(unauth_app: FastAPI) -> httpx.AsyncClient:
|
||||
transport = ASGITransport(app=unauth_app)
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://test") as c:
|
||||
yield c
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def no_service_client(no_service_app: FastAPI) -> httpx.AsyncClient:
|
||||
transport = ASGITransport(app=no_service_app)
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://test") as c:
|
||||
yield c
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Auth + service availability
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_create_table_requires_auth(unauth_client: httpx.AsyncClient) -> None:
|
||||
"""No auth → 401."""
|
||||
resp = await unauth_client.post("/api/v1/bitable/tables", json={"name": "T"})
|
||||
assert resp.status_code == 401
|
||||
|
||||
|
||||
async def test_endpoint_returns_503_when_service_unavailable(
|
||||
no_service_client: httpx.AsyncClient,
|
||||
) -> None:
|
||||
"""No service on app.state → 503."""
|
||||
resp = await no_service_client.post("/api/v1/bitable/tables", json={"name": "T"})
|
||||
assert resp.status_code == 503
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tables CRUD
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_create_table_success(client: httpx.AsyncClient) -> None:
|
||||
resp = await client.post(
|
||||
"/api/v1/bitable/tables", json={"name": "Orders", "description": "desc"}
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()
|
||||
assert data["success"] is True
|
||||
assert data["table"]["name"] == "Orders"
|
||||
assert data["table"]["description"] == "desc"
|
||||
assert "id" in data["table"]
|
||||
|
||||
|
||||
async def test_list_tables_returns_created(client: httpx.AsyncClient) -> None:
|
||||
for name in ("A", "B", "C"):
|
||||
await client.post("/api/v1/bitable/tables", json={"name": name})
|
||||
resp = await client.get("/api/v1/bitable/tables")
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()
|
||||
assert data["success"] is True
|
||||
assert len(data["tables"]) == 3
|
||||
names = {t["name"] for t in data["tables"]}
|
||||
assert names == {"A", "B", "C"}
|
||||
|
||||
|
||||
async def test_get_table_404_when_missing(client: httpx.AsyncClient) -> None:
|
||||
resp = await client.get("/api/v1/bitable/tables/nonexistent-id")
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
async def test_update_table_success(client: httpx.AsyncClient) -> None:
|
||||
create_resp = await client.post("/api/v1/bitable/tables", json={"name": "Old"})
|
||||
table_id = create_resp.json()["table"]["id"]
|
||||
resp = await client.patch(f"/api/v1/bitable/tables/{table_id}", json={"name": "New"})
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["table"]["name"] == "New"
|
||||
|
||||
|
||||
async def test_delete_table_success(client: httpx.AsyncClient) -> None:
|
||||
create_resp = await client.post("/api/v1/bitable/tables", json={"name": "T"})
|
||||
table_id = create_resp.json()["table"]["id"]
|
||||
resp = await client.delete(f"/api/v1/bitable/tables/{table_id}")
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["success"] is True
|
||||
# Verify gone
|
||||
assert (await client.get(f"/api/v1/bitable/tables/{table_id}")).status_code == 404
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fields CRUD + dependency check (409)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_create_field_success(client: httpx.AsyncClient) -> None:
|
||||
table_id = (await client.post("/api/v1/bitable/tables", json={"name": "T"})).json()["table"][
|
||||
"id"
|
||||
]
|
||||
resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/fields",
|
||||
json={"name": "title", "field_type": "text", "owner": "agent"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()
|
||||
assert data["field"]["name"] == "title"
|
||||
assert data["field"]["field_type"] == "text"
|
||||
assert data["field"]["owner"] == "agent"
|
||||
|
||||
|
||||
async def test_list_fields(client: httpx.AsyncClient) -> None:
|
||||
table_id = (await client.post("/api/v1/bitable/tables", json={"name": "T"})).json()["table"][
|
||||
"id"
|
||||
]
|
||||
for name in ("f1", "f2"):
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/fields",
|
||||
json={"name": name, "field_type": "text"},
|
||||
)
|
||||
resp = await client.get(f"/api/v1/bitable/tables/{table_id}/fields")
|
||||
assert resp.status_code == 200
|
||||
assert len(resp.json()["fields"]) == 2
|
||||
|
||||
|
||||
async def test_delete_field_no_deps(client: httpx.AsyncClient) -> None:
|
||||
table_id = (await client.post("/api/v1/bitable/tables", json={"name": "T"})).json()["table"][
|
||||
"id"
|
||||
]
|
||||
field_id = (
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/fields",
|
||||
json={"name": "f", "field_type": "text"},
|
||||
)
|
||||
).json()["field"]["id"]
|
||||
resp = await client.delete(f"/api/v1/bitable/fields/{field_id}")
|
||||
assert resp.status_code == 200
|
||||
|
||||
|
||||
async def test_delete_field_returns_409_when_referenced_by_formula(
|
||||
client: httpx.AsyncClient,
|
||||
) -> None:
|
||||
table_id = (await client.post("/api/v1/bitable/tables", json={"name": "T"})).json()["table"][
|
||||
"id"
|
||||
]
|
||||
source_id = (
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/fields",
|
||||
json={"name": "src", "field_type": "number"},
|
||||
)
|
||||
).json()["field"]["id"]
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/fields",
|
||||
json={
|
||||
"name": "calc",
|
||||
"field_type": "formula",
|
||||
"config": {"formula_expr": f"={source_id} * 2"},
|
||||
},
|
||||
)
|
||||
resp = await client.delete(f"/api/v1/bitable/fields/{source_id}")
|
||||
assert resp.status_code == 409
|
||||
detail = resp.json()["detail"]
|
||||
assert "dependencies" in detail
|
||||
assert "formula_fields" in detail["dependencies"]
|
||||
|
||||
|
||||
async def test_delete_field_force_cascades(client: httpx.AsyncClient) -> None:
|
||||
table_id = (await client.post("/api/v1/bitable/tables", json={"name": "T"})).json()["table"][
|
||||
"id"
|
||||
]
|
||||
source_id = (
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/fields",
|
||||
json={"name": "src", "field_type": "number", "owner": "agent"},
|
||||
)
|
||||
).json()["field"]["id"]
|
||||
# Create a record with the source field
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/records",
|
||||
json={"records": [{source_id: 42}]},
|
||||
)
|
||||
resp = await client.delete(f"/api/v1/bitable/fields/{source_id}?force=true")
|
||||
assert resp.status_code == 200
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Records CRUD + cursor pagination
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_create_records_batch(client: httpx.AsyncClient) -> None:
|
||||
table_id = (await client.post("/api/v1/bitable/tables", json={"name": "T"})).json()["table"][
|
||||
"id"
|
||||
]
|
||||
resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/records",
|
||||
json={"records": [{"a": 1}, {"a": 2}, {"a": 3}]},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()
|
||||
assert data["count"] == 3
|
||||
assert len(data["records"]) == 3
|
||||
|
||||
|
||||
async def test_list_records_cursor_pagination(client: httpx.AsyncClient) -> None:
|
||||
table_id = (await client.post("/api/v1/bitable/tables", json={"name": "T"})).json()["table"][
|
||||
"id"
|
||||
]
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/records",
|
||||
json={"records": [{"i": i} for i in range(5)]},
|
||||
)
|
||||
# Page 1
|
||||
resp = await client.get(f"/api/v1/bitable/tables/{table_id}/records?limit=2")
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()
|
||||
assert len(data["records"]) == 2
|
||||
assert data["next_cursor"] is not None
|
||||
# Page 2
|
||||
resp2 = await client.get(
|
||||
f"/api/v1/bitable/tables/{table_id}/records?limit=2&cursor={data['next_cursor']}"
|
||||
)
|
||||
data2 = resp2.json()
|
||||
assert len(data2["records"]) == 2
|
||||
# No overlap
|
||||
ids1 = {r["id"] for r in data["records"]}
|
||||
ids2 = {r["id"] for r in data2["records"]}
|
||||
assert ids1.isdisjoint(ids2)
|
||||
|
||||
|
||||
async def test_list_records_with_filters(client: httpx.AsyncClient) -> None:
|
||||
table_id = (await client.post("/api/v1/bitable/tables", json={"name": "T"})).json()["table"][
|
||||
"id"
|
||||
]
|
||||
num_field_id = (
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/fields",
|
||||
json={"name": "amt", "field_type": "number", "owner": "agent"},
|
||||
)
|
||||
).json()["field"]["id"]
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/records",
|
||||
json={"records": [{num_field_id: 10}, {num_field_id: 50}, {num_field_id: 100}]},
|
||||
)
|
||||
filters = json.dumps([{"field_id": num_field_id, "op": "gt", "value": 40}])
|
||||
resp = await client.get(
|
||||
f"/api/v1/bitable/tables/{table_id}/records", params={"filters": filters}
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()
|
||||
assert len(data["records"]) == 2 # 50 and 100
|
||||
|
||||
|
||||
async def test_update_record(client: httpx.AsyncClient) -> None:
|
||||
table_id = (await client.post("/api/v1/bitable/tables", json={"name": "T"})).json()["table"][
|
||||
"id"
|
||||
]
|
||||
record_id = (
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/records", json={"records": [{"a": 1}]}
|
||||
)
|
||||
).json()["records"][0]["id"]
|
||||
resp = await client.patch(f"/api/v1/bitable/records/{record_id}", json={"values": {"a": 99}})
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["record"]["values"]["a"] == 99
|
||||
|
||||
|
||||
async def test_delete_single_record(client: httpx.AsyncClient) -> None:
|
||||
table_id = (await client.post("/api/v1/bitable/tables", json={"name": "T"})).json()["table"][
|
||||
"id"
|
||||
]
|
||||
record_id = (
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/records", json={"records": [{"a": 1}]}
|
||||
)
|
||||
).json()["records"][0]["id"]
|
||||
resp = await client.delete(f"/api/v1/bitable/records/{record_id}")
|
||||
assert resp.status_code == 200
|
||||
# Verify gone
|
||||
resp2 = await client.get(f"/api/v1/bitable/tables/{table_id}/records")
|
||||
assert len(resp2.json()["records"]) == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Upsert endpoint (KTD8)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_upsert_inserts_then_updates(client: httpx.AsyncClient) -> None:
|
||||
table_id = (await client.post("/api/v1/bitable/tables", json={"name": "T"})).json()["table"][
|
||||
"id"
|
||||
]
|
||||
pk_id = (
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/fields",
|
||||
json={"name": "id", "field_type": "text", "owner": "agent"},
|
||||
)
|
||||
).json()["field"]["id"]
|
||||
data_id = (
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/fields",
|
||||
json={"name": "data", "field_type": "text", "owner": "agent"},
|
||||
)
|
||||
).json()["field"]["id"]
|
||||
await client.patch(f"/api/v1/bitable/tables/{table_id}", json={"primary_key_field_id": pk_id})
|
||||
|
||||
# First: insert
|
||||
resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/upsert",
|
||||
json={
|
||||
"records": [{pk_id: "r1", data_id: "v1"}],
|
||||
"primary_key_field_id": pk_id,
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["inserted"] == 1
|
||||
assert resp.json()["updated"] == 0
|
||||
|
||||
# Second: update
|
||||
resp2 = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/upsert",
|
||||
json={
|
||||
"records": [{pk_id: "r1", data_id: "v2"}],
|
||||
"primary_key_field_id": pk_id,
|
||||
},
|
||||
)
|
||||
assert resp2.status_code == 200
|
||||
assert resp2.json()["inserted"] == 0
|
||||
assert resp2.json()["updated"] == 1
|
||||
|
||||
# Verify value
|
||||
records = (await client.get(f"/api/v1/bitable/tables/{table_id}/records")).json()["records"]
|
||||
assert len(records) == 1
|
||||
assert records[0]["values"][data_id] == "v2"
|
||||
|
||||
|
||||
async def test_upsert_preserves_user_columns(client: httpx.AsyncClient) -> None:
|
||||
"""KTD8 via API: upsert updates agent columns, user columns untouched."""
|
||||
table_id = (await client.post("/api/v1/bitable/tables", json={"name": "T"})).json()["table"][
|
||||
"id"
|
||||
]
|
||||
pk_id = (
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/fields",
|
||||
json={"name": "id", "field_type": "text", "owner": "agent"},
|
||||
)
|
||||
).json()["field"]["id"]
|
||||
agent_id = (
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/fields",
|
||||
json={"name": "agent_data", "field_type": "text", "owner": "agent"},
|
||||
)
|
||||
).json()["field"]["id"]
|
||||
user_id = (
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/fields",
|
||||
json={"name": "user_data", "field_type": "text", "owner": "user"},
|
||||
)
|
||||
).json()["field"]["id"]
|
||||
await client.patch(f"/api/v1/bitable/tables/{table_id}", json={"primary_key_field_id": pk_id})
|
||||
|
||||
# Insert with both agent and user values
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/upsert",
|
||||
json={
|
||||
"records": [{pk_id: "r1", agent_id: "a1", user_id: "u1"}],
|
||||
"primary_key_field_id": pk_id,
|
||||
},
|
||||
)
|
||||
# Manually set user column (simulating user edit via PATCH)
|
||||
records = (await client.get(f"/api/v1/bitable/tables/{table_id}/records")).json()["records"]
|
||||
rec_id = records[0]["id"]
|
||||
await client.patch(
|
||||
f"/api/v1/bitable/records/{rec_id}",
|
||||
json={"values": {**records[0]["values"], user_id: "USER_EDITED"}},
|
||||
)
|
||||
|
||||
# Second upsert: tries to change user column — should be ignored
|
||||
await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/upsert",
|
||||
json={
|
||||
"records": [{pk_id: "r1", agent_id: "a2", user_id: "SHOULD_NOT_APPLY"}],
|
||||
"primary_key_field_id": pk_id,
|
||||
},
|
||||
)
|
||||
|
||||
records = (await client.get(f"/api/v1/bitable/tables/{table_id}/records")).json()["records"]
|
||||
assert len(records) == 1
|
||||
assert records[0]["values"][agent_id] == "a2" # updated
|
||||
assert records[0]["values"][user_id] == "USER_EDITED" # preserved
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Views CRUD
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_create_view_success(client: httpx.AsyncClient) -> None:
|
||||
table_id = (await client.post("/api/v1/bitable/tables", json={"name": "T"})).json()["table"][
|
||||
"id"
|
||||
]
|
||||
resp = await client.post(
|
||||
f"/api/v1/bitable/tables/{table_id}/views",
|
||||
json={"name": "Grid View", "view_type": "grid", "config": {}},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["view"]["name"] == "Grid View"
|
||||
assert resp.json()["view"]["view_type"] == "grid"
|
||||
|
||||
|
||||
async def test_list_views(client: httpx.AsyncClient) -> None:
|
||||
table_id = (await client.post("/api/v1/bitable/tables", json={"name": "T"})).json()["table"][
|
||||
"id"
|
||||
]
|
||||
for name in ("v1", "v2"):
|
||||
await client.post(f"/api/v1/bitable/tables/{table_id}/views", json={"name": name})
|
||||
resp = await client.get(f"/api/v1/bitable/tables/{table_id}/views")
|
||||
assert resp.status_code == 200
|
||||
assert len(resp.json()["views"]) == 2
|
||||
|
||||
|
||||
async def test_update_view(client: httpx.AsyncClient) -> None:
|
||||
table_id = (await client.post("/api/v1/bitable/tables", json={"name": "T"})).json()["table"][
|
||||
"id"
|
||||
]
|
||||
view_id = (
|
||||
await client.post(f"/api/v1/bitable/tables/{table_id}/views", json={"name": "Old"})
|
||||
).json()["view"]["id"]
|
||||
resp = await client.patch(f"/api/v1/bitable/views/{view_id}", json={"name": "New"})
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["view"]["name"] == "New"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Formula validation (U5b)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_validate_formula_valid(client: httpx.AsyncClient) -> None:
|
||||
"""Valid formula returns valid=true."""
|
||||
resp = await client.post(
|
||||
"/api/v1/bitable/fields/validate-formula",
|
||||
json={"formula": "1 + 2"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()
|
||||
assert data["valid"] is True
|
||||
assert "error" not in data
|
||||
|
||||
|
||||
async def test_validate_formula_with_field_ref(client: httpx.AsyncClient) -> None:
|
||||
"""Formula with field reference is valid syntax."""
|
||||
resp = await client.post(
|
||||
"/api/v1/bitable/fields/validate-formula",
|
||||
json={"formula": "{field_abc} + 1"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["valid"] is True
|
||||
|
||||
|
||||
async def test_validate_formula_with_function(client: httpx.AsyncClient) -> None:
|
||||
"""Formula with built-in function is valid."""
|
||||
resp = await client.post(
|
||||
"/api/v1/bitable/fields/validate-formula",
|
||||
json={"formula": "SUM({f1}) + AVG({f2})"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["valid"] is True
|
||||
|
||||
|
||||
async def test_validate_formula_syntax_error(client: httpx.AsyncClient) -> None:
|
||||
"""Syntax error returns valid=false with error message."""
|
||||
resp = await client.post(
|
||||
"/api/v1/bitable/fields/validate-formula",
|
||||
json={"formula": "1 +"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()
|
||||
assert data["valid"] is False
|
||||
assert "error" in data
|
||||
|
||||
|
||||
async def test_validate_formula_security_error(client: httpx.AsyncClient) -> None:
|
||||
"""Dangerous constructs (import) are rejected."""
|
||||
resp = await client.post(
|
||||
"/api/v1/bitable/fields/validate-formula",
|
||||
json={"formula": "__import__('os')"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()
|
||||
assert data["valid"] is False
|
||||
assert "error" in data
|
||||
|
||||
|
||||
async def test_validate_formula_unknown_function(client: httpx.AsyncClient) -> None:
|
||||
"""Unknown function is rejected."""
|
||||
resp = await client.post(
|
||||
"/api/v1/bitable/fields/validate-formula",
|
||||
json={"formula": "UNKNOWN_FUNC(1)"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()
|
||||
assert data["valid"] is False
|
||||
assert "error" in data
|
||||
|
||||
|
||||
async def test_validate_formula_requires_auth(unauth_client: httpx.AsyncClient) -> None:
|
||||
"""No auth → 401."""
|
||||
resp = await unauth_client.post(
|
||||
"/api/v1/bitable/fields/validate-formula",
|
||||
json={"formula": "1 + 2"},
|
||||
)
|
||||
assert resp.status_code == 401
|
||||
|
|
@ -0,0 +1,296 @@
|
|||
"""Tests for bitable service layer (U2): upsert, field deletion, view filtering.
|
||||
|
||||
Requires PostgreSQL — marked ``postgres``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from agentkit.bitable.models import FieldOwner, FieldType
|
||||
from agentkit.bitable.service import FieldDependencyError
|
||||
|
||||
pytestmark = pytest.mark.postgres
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Upsert (KTD8: jsonb_set preserves user columns)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_upsert_inserts_new_records(bitable_service) -> None:
|
||||
"""First upsert inserts all records."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
pk_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="id", field_type=FieldType.text, owner=FieldOwner.agent
|
||||
)
|
||||
data_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="data", field_type=FieldType.text, owner=FieldOwner.agent
|
||||
)
|
||||
await bitable_service.update_table(table.id, primary_key_field_id=pk_field.id)
|
||||
|
||||
result = await bitable_service.upsert_records(
|
||||
table.id,
|
||||
[
|
||||
{pk_field.id: "row1", data_field.id: "hello"},
|
||||
{pk_field.id: "row2", data_field.id: "world"},
|
||||
],
|
||||
pk_field.id,
|
||||
)
|
||||
assert result == {"inserted": 2, "updated": 0, "skipped": 0}
|
||||
|
||||
records, _ = await bitable_service.list_records(table.id)
|
||||
assert len(records) == 2
|
||||
|
||||
|
||||
async def test_upsert_updates_existing_preserves_user_columns(bitable_service) -> None:
|
||||
"""KTD8: upsert updates agent columns via jsonb_set, user columns untouched."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
pk_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="id", field_type=FieldType.text, owner=FieldOwner.agent
|
||||
)
|
||||
agent_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="agent_data", field_type=FieldType.text, owner=FieldOwner.agent
|
||||
)
|
||||
user_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="user_data", field_type=FieldType.text, owner=FieldOwner.user
|
||||
)
|
||||
await bitable_service.update_table(table.id, primary_key_field_id=pk_field.id)
|
||||
|
||||
# First: insert with both agent and user values
|
||||
await bitable_service.upsert_records(
|
||||
table.id,
|
||||
[{pk_field.id: "row1", agent_field.id: "agent_v1", user_field.id: "user_v1"}],
|
||||
pk_field.id,
|
||||
)
|
||||
|
||||
# Manually set user column (simulating user edit)
|
||||
records, _ = await bitable_service.list_records(table.id)
|
||||
assert len(records) == 1
|
||||
rec = records[0]
|
||||
await bitable_service.update_record_values(rec.id, {**rec.values, user_field.id: "USER_EDITED"})
|
||||
|
||||
# Second upsert: only agent column changes
|
||||
result = await bitable_service.upsert_records(
|
||||
table.id,
|
||||
[{pk_field.id: "row1", agent_field.id: "agent_v2", user_field.id: "SHOULD_NOT_APPLY"}],
|
||||
pk_field.id,
|
||||
)
|
||||
assert result == {"inserted": 0, "updated": 1, "skipped": 0}
|
||||
|
||||
# Verify: agent column updated, user column preserved
|
||||
records, _ = await bitable_service.list_records(table.id)
|
||||
assert len(records) == 1
|
||||
rec = records[0]
|
||||
assert rec.values[agent_field.id] == "agent_v2" # updated
|
||||
assert rec.values[user_field.id] == "USER_EDITED" # preserved (NOT "SHOULD_NOT_APPLY")
|
||||
|
||||
|
||||
async def test_upsert_skips_records_without_pk(bitable_service) -> None:
|
||||
"""Records without PK value are skipped."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
pk_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="id", field_type=FieldType.text, owner=FieldOwner.agent
|
||||
)
|
||||
await bitable_service.update_table(table.id, primary_key_field_id=pk_field.id)
|
||||
|
||||
result = await bitable_service.upsert_records(
|
||||
table.id,
|
||||
[{pk_field.id: "row1"}, {}], # second has no PK
|
||||
pk_field.id,
|
||||
)
|
||||
assert result == {"inserted": 1, "updated": 0, "skipped": 1}
|
||||
|
||||
|
||||
async def test_upsert_empty_batch(bitable_service) -> None:
|
||||
"""Empty batch returns all zeros."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
pk_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="id", field_type=FieldType.text, owner=FieldOwner.agent
|
||||
)
|
||||
result = await bitable_service.upsert_records(table.id, [], pk_field.id)
|
||||
assert result == {"inserted": 0, "updated": 0, "skipped": 0}
|
||||
|
||||
|
||||
async def test_upsert_without_pk_field_raises(bitable_service) -> None:
|
||||
"""Upsert without primary_key_field_id raises ValueError."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
with pytest.raises(ValueError, match="primary_key_field_id"):
|
||||
await bitable_service.upsert_records(table.id, [{}], "")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Field deletion with dependency checking
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_delete_field_no_dependencies(bitable_service) -> None:
|
||||
"""Deleting a field with no dependencies succeeds."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
field = await bitable_service.create_field(
|
||||
table_id=table.id, name="f", field_type=FieldType.text
|
||||
)
|
||||
deleted = await bitable_service.delete_field(field.id)
|
||||
assert deleted is True
|
||||
assert await bitable_service.get_field(field.id) is None
|
||||
|
||||
|
||||
async def test_delete_field_referenced_by_formula_returns_deps(bitable_service) -> None:
|
||||
"""Deleting a field referenced by a formula raises FieldDependencyError."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
source_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="source", field_type=FieldType.number
|
||||
)
|
||||
formula_field = await bitable_service.create_field(
|
||||
table_id=table.id,
|
||||
name="calc",
|
||||
field_type=FieldType.formula,
|
||||
config={"formula_expr": f"={source_field.id} * 2"},
|
||||
)
|
||||
|
||||
with pytest.raises(FieldDependencyError) as exc_info:
|
||||
await bitable_service.delete_field(source_field.id)
|
||||
|
||||
deps = exc_info.value.dependencies
|
||||
assert "formula_fields" in deps
|
||||
assert any(f["id"] == formula_field.id for f in deps["formula_fields"])
|
||||
|
||||
|
||||
async def test_delete_primary_key_field_returns_deps(bitable_service) -> None:
|
||||
"""Deleting the primary key field raises FieldDependencyError."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
pk_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="id", field_type=FieldType.text
|
||||
)
|
||||
await bitable_service.update_table(table.id, primary_key_field_id=pk_field.id)
|
||||
|
||||
with pytest.raises(FieldDependencyError) as exc_info:
|
||||
await bitable_service.delete_field(pk_field.id)
|
||||
|
||||
assert exc_info.value.dependencies.get("is_primary_key") is True
|
||||
|
||||
|
||||
async def test_delete_field_force_casces_cleanup(bitable_service) -> None:
|
||||
"""Force delete cascades: removes field from records, marks formula as error."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
source_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="source", field_type=FieldType.number, owner=FieldOwner.agent
|
||||
)
|
||||
formula_field = await bitable_service.create_field(
|
||||
table_id=table.id,
|
||||
name="calc",
|
||||
field_type=FieldType.formula,
|
||||
config={"formula_expr": f"={source_field.id} * 2"},
|
||||
)
|
||||
# Create a record with the source field value
|
||||
record = await bitable_service.create_record(table_id=table.id, values={source_field.id: 42})
|
||||
|
||||
# Force delete
|
||||
deleted = await bitable_service.delete_field(source_field.id, force=True)
|
||||
assert deleted is True
|
||||
|
||||
# Record should no longer have the source field key
|
||||
rec = await bitable_service.get_record(record.id)
|
||||
assert rec is not None
|
||||
assert source_field.id not in rec.values
|
||||
|
||||
# Formula field should have error in config
|
||||
formula = await bitable_service.get_field(formula_field.id)
|
||||
assert formula is not None
|
||||
assert "error" in formula.config
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# View-filtered record listing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_list_records_filtered_by_number_gt(bitable_service) -> None:
|
||||
"""View filter with gt op on number field correctly filters (CAST NUMERIC)."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
num_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="amount", field_type=FieldType.number, owner=FieldOwner.agent
|
||||
)
|
||||
|
||||
# Create records with various amounts
|
||||
for amt in [10, 50, 100, 200]:
|
||||
await bitable_service.create_record(table_id=table.id, values={num_field.id: amt})
|
||||
|
||||
# Filter: amount > 50
|
||||
records, _ = await bitable_service.list_records_filtered(
|
||||
table.id,
|
||||
filters=[{"field_id": num_field.id, "op": "gt", "value": 50}],
|
||||
)
|
||||
amounts = [r.values[num_field.id] for r in records]
|
||||
assert all(a > 50 for a in amounts)
|
||||
assert len(records) == 2 # 100 and 200
|
||||
|
||||
|
||||
async def test_list_records_filtered_by_text_eq(bitable_service) -> None:
|
||||
"""View filter with eq op on text field."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
text_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="status", field_type=FieldType.text, owner=FieldOwner.agent
|
||||
)
|
||||
|
||||
for status in ["open", "closed", "open", "pending"]:
|
||||
await bitable_service.create_record(table_id=table.id, values={text_field.id: status})
|
||||
|
||||
records, _ = await bitable_service.list_records_filtered(
|
||||
table.id,
|
||||
filters=[{"field_id": text_field.id, "op": "eq", "value": "open"}],
|
||||
)
|
||||
assert len(records) == 2
|
||||
assert all(r.values[text_field.id] == "open" for r in records)
|
||||
|
||||
|
||||
async def test_list_records_filtered_with_sort(bitable_service) -> None:
|
||||
"""View sort by number field descending."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
num_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="score", field_type=FieldType.number, owner=FieldOwner.agent
|
||||
)
|
||||
|
||||
for score in [30, 10, 50, 20]:
|
||||
await bitable_service.create_record(table_id=table.id, values={num_field.id: score})
|
||||
|
||||
records, _ = await bitable_service.list_records_filtered(
|
||||
table.id,
|
||||
sorts=[{"field_id": num_field.id, "direction": "desc"}],
|
||||
)
|
||||
# Records should be sorted by score descending (as text, but single/double digit sorts OK)
|
||||
assert len(records) == 4
|
||||
|
||||
|
||||
async def test_list_records_filtered_cursor_pagination(bitable_service) -> None:
|
||||
"""Cursor pagination with filters."""
|
||||
table = await bitable_service.create_table(name="T")
|
||||
text_field = await bitable_service.create_field(
|
||||
table_id=table.id, name="name", field_type=FieldType.text, owner=FieldOwner.agent
|
||||
)
|
||||
|
||||
for i in range(5):
|
||||
await bitable_service.create_record(table_id=table.id, values={text_field.id: f"item_{i}"})
|
||||
|
||||
# First page
|
||||
records, next_cursor = await bitable_service.list_records_filtered(table.id, limit=2)
|
||||
assert len(records) == 2
|
||||
assert next_cursor is not None
|
||||
|
||||
# Second page
|
||||
records2, next_cursor2 = await bitable_service.list_records_filtered(
|
||||
table.id, cursor=next_cursor, limit=2
|
||||
)
|
||||
assert len(records2) == 2
|
||||
assert next_cursor2 is not None
|
||||
|
||||
# Third page
|
||||
records3, next_cursor3 = await bitable_service.list_records_filtered(
|
||||
table.id, cursor=next_cursor2, limit=2
|
||||
)
|
||||
assert len(records3) == 1
|
||||
assert next_cursor3 is None
|
||||
|
||||
# All records unique
|
||||
all_ids = {r.id for r in [records, records2, records3] for r in r}
|
||||
assert len(all_ids) == 5
|
||||
|
|
@ -437,12 +437,12 @@ class TestTokenUsageMiddleware:
|
|||
|
||||
@pytest.mark.asyncio
|
||||
async def test_after_extracts_usage_from_result(self) -> None:
|
||||
"""after 从 result 提取 token_usage。"""
|
||||
"""after 从 result 提取 total_tokens（ReActResult 属性名）。"""
|
||||
mw = TokenUsageMiddleware()
|
||||
ctx = _make_ctx()
|
||||
|
||||
result = MagicMock()
|
||||
result.token_usage = {"total": 100}
|
||||
result.total_tokens = {"total": 100}
|
||||
|
||||
await mw.after(ctx, result)
|
||||
|
||||
|
|
|
|||
|
|
@ -273,7 +273,7 @@ class TestCheckpointTTL:
|
|||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
expired_time = (datetime.now(timezone.utc) - timedelta(seconds=10)).isoformat()
|
||||
cp._memory["plan_1"][0].saved_at = expired_time
|
||||
cp._memory["plan_1"]["p1"].saved_at = expired_time
|
||||
|
||||
# 过期后 load 应返回 None
|
||||
loaded = await cp.load("plan_1")
|
||||
|
|
@ -329,7 +329,7 @@ class TestCheckpointTTL:
|
|||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
expired_time = (datetime.now(timezone.utc) - timedelta(seconds=10)).isoformat()
|
||||
cp._memory["plan_1"][0].saved_at = expired_time
|
||||
cp._memory["plan_1"]["p1"].saved_at = expired_time
|
||||
|
||||
# list 应过滤掉过期的，只返回 1 个
|
||||
checkpoints = await cp.list_checkpoints("plan_1")
|
||||
|
|
@ -363,7 +363,7 @@ class TestCheckpointTTL:
|
|||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
expired_time = (datetime.now(timezone.utc) - timedelta(seconds=10)).isoformat()
|
||||
cp._memory["plan_1"][0].saved_at = expired_time
|
||||
cp._memory["plan_1"]["p1"].saved_at = expired_time
|
||||
|
||||
# 内存降级 + TTL 过期 → 应返回 None
|
||||
loaded = await cp.load("plan_1")
|
||||
|
|
|
|||
|
|
@ -193,7 +193,7 @@ class TestSkillMdToSkillConfig:
|
|||
|
||||
assert config.name == "content-generator"
|
||||
assert config.description != ""
|
||||
assert config.disclosure_level == 0
|
||||
assert config.disclosure_level == 1
|
||||
# Level 0: prompt 仅含 identity（概要信息）
|
||||
assert config.prompt is not None
|
||||
assert "identity" in config.prompt
|
||||
|
|
@ -257,14 +257,14 @@ class TestSkillConfigNewFields:
|
|||
)
|
||||
assert config.skill_md_path is None
|
||||
|
||||
def test_default_disclosure_level_is_zero(self):
|
||||
def test_default_disclosure_level_is_one(self):
|
||||
config = SkillConfig(
|
||||
name="test",
|
||||
agent_type="test",
|
||||
task_mode="llm_generate",
|
||||
prompt={"identity": "test"},
|
||||
)
|
||||
assert config.disclosure_level == 0
|
||||
assert config.disclosure_level == 1
|
||||
|
||||
def test_skill_md_path_set(self):
|
||||
config = SkillConfig(
|
||||
|
|
@ -321,7 +321,7 @@ class TestSkillConfigNewFields:
|
|||
}
|
||||
config = SkillConfig.from_dict(data)
|
||||
assert config.skill_md_path is None
|
||||
assert config.disclosure_level == 0
|
||||
assert config.disclosure_level == 1
|
||||
|
||||
|
||||
# ── SkillLoader.load_from_skill_md 测试 ───────────────────
|
||||
|
|
|
|||
Loading…
Reference in New Issue