diff --git a/docs/plans/2026-07-06-001-feat-bitable-formula-relations-v2-plan.md b/docs/plans/2026-07-06-001-feat-bitable-formula-relations-v2-plan.md new file mode 100644 index 0000000..8813cd7 --- /dev/null +++ b/docs/plans/2026-07-06-001-feat-bitable-formula-relations-v2-plan.md @@ -0,0 +1,762 @@ +--- +title: Bitable 公式与关联联动 v2 - Plan +type: feat +date: 2026-07-06 +topic: bitable-formula-relations-v2 +artifact_contract: ce-unified-plan/v1 +artifact_readiness: implementation-ready +product_contract_source: ce-brainstorm +execution: code +--- + +# Bitable 公式与关联联动 v2 - Plan + +## Goal Capsule + +**Objective:** 把 bitable 从 "agent companion" 推进到"公式+关联联动的数据建模层"，向飞书/Notion/Baserow 等成熟产品看齐。v2 聚焦公式引擎扩展、表关联（含多对多）、跨表公式与依赖追踪、轻量自动化，并以 agent 生成公式/自动化作为差异化亮点。 + +**Product authority:** 产品负责人（用户）已确认范围；技术实现细节由 ce-plan 决定。 + +**Open blockers:** 无。三个 call-out（多对多纳入 v2、agent 差异化亮点、存储模型扩展）均已确认。 + +## Product Contract + +**Product Contract unchanged** — R1-R20 / F1-F4 / AE1-AE5 来自 ce-brainstorm，未做产品范围变更。Planning Contract 在下方追加。 + +### Summary + +v2 把 bitable 的公式引擎从 10 函数扩展到 40+（含跨表引用），同步引入单向/双向/多对多表关联 + Rollup 聚合 + 存储模型扩展，轻量自动化（Webhook + 触发器 + 动作）打底，并以 agent 生成公式/自动化作为差异化亮点。协作与视图扩展留给 v3+。 + +### Problem Frame + +当前 bitable 定位为 AgentKit 的 "companion service"，已有 9 字段类型、5 视图枚举（仅 grid 实现）、10 个公式函数、3 个数据导入源、无自动化、无表关联、无实时协作。 + +竞品对比（2026 年数据）显示巨大差距： + +| 维度 | 当前 | 飞书 | Notion | Baserow | NocoDB | +|---|---|---|---|---|---| +| 公式函数 | 10 | 100+ | 80+ | 100+ | Excel 兼容 | +| 自动化 | 无 | 工作流+AI Agent | 4 触发器+4 动作 | Workflow Builder | Workflows+Scripts | +| 表关联 | lookup 仅显示 | 单/双向/跨表公式 | 单/双向/自引用/Rollup | Link/Lookup/Rollup | LTAR V2 | +| 字段类型 | 9 | 26 | 21-23 | 25+ | 40+ | + +目标是对标补齐，做完整的多人协作多维表格产品。v2 聚焦单用户深度（公式+关联），协作与视图扩展留给后续版本。 + +### Key Decisions + +**公式+关联同步做（方案 B）。** 跨表公式依赖关联基础设施，分开做会导致公式引擎返工。同步推进避免架构级重构。 + +**多对多关联纳入 v2。** 多对多需扩展存储模型（junction table 或等价方案），增加架构工作量。尽早实施避免 v3 返工——关联是数据建模的灵魂，缺多对多则复杂业务场景无法支撑。 + +**自动化轻量化。** v2 仅做 Webhook + 3 触发器 + 5 动作，不做画布式工作流编排（if/else/switch/循环）和 AI Agent 节点。优先把数据建模层做扎实，自动化深度留给 v3+。 + +**Agent 生成公式/自动化作为差异化亮点。** 飞书 AI Agent 节点很强，但我们是 agent-native 架构——agent 是"一等公民"。agent 通过自然语言为用户生成复杂公式、描述目标生成自动化规则，这是竞品没有的独特价值。 + +**视图类型全部推迟到 v3。** 牺牲短期"完整度感知"换取公式+关联的深度。kanban/gallery/gantt/form/calendar 等视图留给 v3。 + +### Actors + +- **用户（User）**：创建/编辑表、字段、记录、公式、关联、自动化规则；消费数据 +- **Agent**：通过 BitableTool REST API 读写数据；生成公式/自动化规则（差异化能力）；upsert agent-owned 字段 +- **系统（Recalc Worker）**：异步重算公式字段（单表+跨表）；处理自动化触发器 + +### Requirements + +#### 公式引擎扩展 + +R1. 公式函数从 10 扩展到 40+，覆盖日期（15+，如 DATE/DATEDIF/NETWORKDAYS/WEEKDAY/EOMONTH）、文本（10+，如 FIND/MID/REPLACE/SUBSTITUTE/UPPER/LOWER）、逻辑（10+，如 IFS/SWITCH/IFERROR/ISBLANK/AND/OR/NOT）、条件聚合（5+，如 SUMIF/COUNTIF/AVERAGEIF）、查找引用（5+，如 LOOKUP/FILTER/MATCH）五大类别。 + +R2. 支持跨表公式引用——公式可引用关联表的字段（如 `[关联表].[字段]` 整列引用或 `{关联字段}.{目标字段}` 行级引用），依赖关联基础设施（见 R5）。 + +R3. 公式引擎保持 AST 白名单安全机制（KTD7），新增函数需注册到白名单，禁止 `eval()`/`exec()`。 + +R4. 公式嵌套深度上限设为 15 层（对齐 Notion），超过则报错提示。 + +#### 表关联与数据模型 + +R5. 支持三种关联类型：单向关联（A→B，B 无反向字段）、双向关联（A↔B，自动创建反向字段）、多对多关联（通过 junction table 或等价机制）。 + +R6. 关联字段显示被关联记录的主键值；可通过 Lookup 字段拉取关联记录的其他字段；可通过 Rollup 字段对关联记录聚合（sum/count/avg/min/max/unique）。 + +R7. 支持自引用关联（关联"当前表"），用于父子层级/依赖关系建模。 + +R8. 关联关系删除时，依赖该关联的 Lookup/Rollup/跨表公式字段需级联处理（报错提示或级联删除，由 force 参数控制，对齐现有字段删除 guard 模式）。 + +R9. 存储模型扩展以支持关联——JSONB 记录存储关联引用（记录 ID 列表），junction table 存储多对多关系。具体 schema 由 ce-plan 决定。 + +#### 跨表公式与依赖追踪 + +R10. Recalc worker 从单表内重算扩展到跨表依赖追踪——当表 A 的记录变更时，表 B 中引用该记录的公式/Rollup 字段需触发重算。 + +R11. 跨表依赖图构建——维护字段级依赖关系（哪些表的哪些字段依赖当前记录），支持拓扑排序避免循环。 + +R12. 跨表重算异步执行，不阻塞原始记录写入；通过现有 recalc queue 机制（FOR UPDATE SKIP LOCKED）扩展。 + +#### 轻量自动化 + +R13. 支持 3 种触发器：记录新增、记录修改（可指定字段）、记录删除。 + +R14. 支持 5 种动作：新增记录、修改记录、发送 HTTP 请求（Webhook 出站）、发送邮件、发送消息（集成现有消息总线）。 + +R15. 支持 Webhook 入站——外部系统 POST 到指定端点触发自动化规则。 + +R16. 自动化规则配置存储在 bitable schema 中（新增 automation 表或扩展现有 meta），具体 schema 由 ce-plan 决定。 + +R17. 自动化规则执行异步，失败重试 3 次后标记错误状态；执行日志可查。 + +#### Agent 集成与差异化 + +R18. BitableTool 扩展新 action：`generate_formula`（自然语言描述 → 生成可用公式）、`generate_automation`（描述目标 → 生成自动化规则）、`create_relation`（创建关联字段）、`create_rollup`（创建 Rollup 字段）。 + +R19. Agent 生成的公式/自动化规则需经用户确认后落地（或直接落地但可撤销，由 ce-plan 决定交互模式）。 + +R20. Agent 生成公式时，公式引擎的 validate-formula 端点（已存在）复用，确保生成结果语法正确。 + +### Key Flows + +**F1. 跨表公式重算流程** +- **触发：** 用户或 agent 修改表 A 记录的字段值 +- **步骤：** 1) 识别表 A 中依赖该字段的公式字段，加入 recalc queue；2) 查跨表依赖图，找出表 B 中引用表 A 该记录的公式/Rollup 字段；3) 跨表依赖任务加入 recalc queue（标记跨表）；4) Recalc worker 按拓扑排序处理；5) 写回重算结果 +- **Covers R10, R11, R12.** + +**F2. 多对多关联创建流程** +- **触发：** 用户或 agent 在表 A 创建多对多关联字段，指向表 B +- **步骤：** 1) 创建 junction table（或等价结构）存储 A-B 关联对；2) 在表 A 创建关联字段（显示 B 记录列表）；3) 在表 B 创建反向关联字段（双向，显示 A 记录列表）；4) 更新跨表依赖图 +- **Covers R5, R9.** + +**F3. Agent 生成公式流程** +- **触发：** 用户通过 agent 描述需求（如"计算订单金额 = 单价 × 数量"） +- **步骤：** 1) Agent 调用 BitableTool `generate_formula` action；2) Agent LLM 生成公式表达式；3) 调用 validate-formula 端点验证语法；4) 验证通过 → 创建公式字段；5) Recalc worker 触发重算 +- **Covers R18, R19, R20.** + +**F4. 自动化规则执行流程** +- **触发：** 记录变更满足自动化规则的触发条件 +- **步骤：** 1) 触发器检测变更；2) 匹配自动化规则；3) 异步执行动作（新增记录/HTTP/邮件/消息）；4) 失败重试 3 次；5) 写执行日志 +- **Covers R13, R14, R15, R17.** + +### Scope Boundaries + +**Deferred for later (v3+):** +- 新视图类型：kanban、gallery、gantt、form、calendar、timeline +- 仪表盘/Dashboard、Chart 视图、15+ 图表组件 +- 画布式工作流编排：if/else/switch/循环节点、AI Agent 节点 +- 工作流画布编辑器 UI +- Formula 2.0 高级特性：`let`/`lets` 局部变量、`map`/`filter`/`find` 列表操作、富数据输出（返回 page/date/people/list） + +**Outside this product's identity (v4+ 协作阶段):** +- 实时协作（OT/CRDT 算法，多人并发编辑） +- 分享/协作权限模型（行级/列级/记录级 RBAC） +- 评论与 @ 提及 +- 视图级权限（restricted view） + +**Explicitly excluded:** +- 跨多维表格文件关联（飞书也不支持，需通过同步数据） +- 单元格内插入公式（仅公式字段支持公式，对齐飞书/Notion 模式） + +### Deferred to Follow-Up Work + +- 前端公式编辑器智能补全（函数名/字段名 autocomplete）— 后端 API 已支持，前端深度优化留给后续打磨 +- 自动化规则的可视化编排 UI — v2 仅做表单式配置，画布式 UI 推迟 +- 跨表公式的 EXPLAIN 依赖图可视化 — 后端依赖图已构建，可视化留给后续 + +### Acceptance Examples + +**AE1. 跨表公式重算（Covers R2, R10, R11）** +- **Given:** 表 Orders 有字段 `amount`，表 Customers 有 Rollup 字段 `total_amount` = SUM(Orders.amount) where Orders.customer_id = this.id +- **When:** 用户修改 Orders 表某记录的 `amount` 值 +- **Then:** Customers 表对应客户的 `total_amount` 异步重算并更新 + +**AE2. 多对多关联（Covers R5, R6, R9）** +- **Given:** 表 Students 和表 Courses +- **When:** 用户在 Students 表创建多对多关联字段指向 Courses +- **Then:** 1) junction table 创建；2) Students 显示关联的 Courses 列表；3) Courses 显示反向关联的 Students 列表；4) 在 Students 可创建 Lookup 拉取 Course 的 `credit` 字段；5) 可创建 Rollup 统计 Student 选课的总学分 + +**AE3. 公式函数扩展（Covers R1, R3）** +- **Given:** 表有 `start_date`（date）、`end_date`（date）字段 +- **When:** 用户创建公式字段 `=NETWORKDAYS({start_date}, {end_date})` +- **Then:** 公式验证通过，重算出工作日天数 + +**AE4. 自动化触发与执行（Covers R13, R14, R17）** +- **Given:** 表 Tasks 有自动化规则：当 `status` 修改为"已完成"时，调用 Webhook +- **When:** 用户将某记录 `status` 改为"已完成" +- **Then:** 1) 触发器检测到变更；2) 异步执行 Webhook 动作；3) 执行日志记录结果；4) 若 Webhook 失败，重试 3 次后标记错误 + +**AE5. Agent 生成公式（Covers R18, R19, R20）** +- **Given:** 表有 `price`、`quantity` 字段 +- **When:** 用户通过 agent 说"帮我算总价" +- **Then:** 1) Agent 调用 `generate_formula` action；2) 生成公式 `={price} * {quantity}`；3) validate-formula 验证通过；4) 创建公式字段 `total_price`；5) 重算所有记录 + +### Dependencies / Assumptions + +- **假设：** 现有 PostgreSQL JSONB 存储模型可扩展支持关联引用（记录 ID 列表嵌入 JSONB 或 junction table） +- **假设：** 现有 recalc worker（FOR UPDATE SKIP LOCKED）可扩展到跨表依赖追踪，无需替换为消息队列 +- **依赖：** 现有 BitableTool REST API 边界（KTD5）保持不变，新 action 通过 HTTP 调用 +- **依赖：** 现有公式引擎 AST 白名单（KTD7）扩展，不引入 `eval()`/`exec()` + +### Sources / Research + +**竞品调研（2026 年数据）：** +- 飞书多维表格：6 视图 + 仪表盘(15+)、26 字段类型、100+ 公式函数、工作流+AI Agent 节点、单/双向关联/跨表公式 +- Notion：11 视图、21-23 字段类型、80+ 公式（Formulas 2.0，let/lambda/map/filter）、4 触发器+4 动作自动化、单/双向/自引用/Rollup +- Baserow：6 视图、25+ 字段类型、100+ Excel 风格公式、Workflow Builder、Link/Lookup/Rollup/Count +- NocoDB：9 视图（含 Gantt/Timeline/Map）、40+ 字段类型、LTAR V2 统一 junction-table +- APITable：7 视图（含 Mindmap）、24 字段类型、OT 实时协作（ot-json0+Yjs）、Magic Link 双向关联 +- Twenty：3 视图、17+ 字段类型、CRM-as-code SDK、原生 MCP Server + +**当前实现 grounding：** +- Dossier 路径：`/tmp/compound-engineering/ce-brainstorm/bitable-1783306851/grounding.md` +- 关键文件：`src/agentkit/bitable/models.py`（FieldType/ViewType/FieldOwner 枚举）、`formula/engine.py`（DAG+拓扑排序）、`recalc_worker.py`（异步重算）、`server/routes/bitable.py`（API 端点）、`tools/bitable_tool.py`（agent REST 边界） + +--- + +## Planning Contract + +### High-Level Technical Design + +```mermaid +graph TB + subgraph "Storage Layer (Schema V3)" + DB[(bitable DB)] + JT[junction tables
多对多] + AUTO[bitable_automations
+ bitable_automation_logs] + DEP[bitable_cross_table_deps
依赖图] + end + + subgraph "Formula Engine" + FE[FormulaEngine
DAG + topo sort] + FUNC[40+ functions
date/text/logical/agg/lookup] + XCTBL[Cross-table ref resolver
relation_field.target_field] + end + + subgraph "Relation Layer" + REL[RelationService
single/bi/multi/self-ref] + LK[Lookup evaluator] + RU[Rollup evaluator
sum/count/avg/min/max/unique] + end + + subgraph "Recalc Pipeline" + RW[RecalcWorker
FOR UPDATE SKIP LOCKED] + DG[DependencyGraph
field-level cross-table] + Q[recalc_queue
+ cross_table flag] + end + + subgraph "Automation Engine" + TRIG[Trigger detector
create/update/delete] + ACT[Action executor
5 actions + retry] + WH[Webhook in/out] + LOG[Execution logs] + end + + subgraph "Agent Surface" + BT[BitableTool
REST boundary KTD5] + GF[generate_formula] + GA[generate_automation] + CR[create_relation] + CU[create_rollup] + end + + DB --> JT + DB --> AUTO + DB --> DEP + FE --> FUNC + FE --> XCTBL + REL --> LK + REL --> RU + REL --> JT + RW --> DG + RW --> Q + RW --> FE + TRIG --> ACT + ACT --> WH + ACT --> LOG + BT --> GF + BT --> GA + BT --> CR + BT --> CU + GF --> FE + CR --> REL + CU --> RU + XCTBL --> DG + RU --> DG + LK --> DG +``` + +**关键数据流：** + +1. **写入路径**：用户/agent 修改记录 → service 层识别受影响字段 → 查跨表依赖图 → enqueue 本表 + 跨表 recalc 任务 → worker 异步处理 +2. **重算路径**：worker claim 任务 → 构建引擎 → 解析跨表引用（通过 relation field → target field）→ 拉取关联记录值 → evaluate → 写回 +3. **自动化路径**：record mutation hook → trigger matcher → async action executor → retry 3x → log + +### Key Technical Decisions + +**KTD-1. Junction table 用独立表，不用 JSONB 数组。** +多对多关联用独立 `bitable_relation_links` 表存储 `(relation_field_id, source_record_id, target_record_id)` 三元组。理由：1) 支持索引和反向查询；2) JSONB 数组在记录多时性能退化；3) 对齐 NocoDB LTAR V2 / Baserow Link 表模式。一对一/一对多关联仍在记录 JSONB 中存 `["rec_id_1", "rec_id_2"]` 数组（轻量，无独立表开销）。 + +**KTD-2. 跨表公式语法用 `{relation_field}.{target_field}` 行级引用 + `[relation_field].[target_field]` 整列引用。** +对齐 Notion 的 `relation.property` 语法。行级引用返回当前记录关联的目标记录的该字段值（标量或列表）；整列引用返回关联表所有记录该字段的列表（用于 SUMIF 等）。parser 扩展 `_FIELD_REF_RE` 支持 `.` 分隔的二级路径。 + +**KTD-3. 跨表依赖图用物化表 `bitable_cross_table_deps`，不用运行时计算。** +字段创建/更新时，service 层解析公式 + relation 结构，写入 `(source_table_id, source_field_id, target_table_id, target_field_id, dep_type)` 记录。recalc worker 通过该表 O(1) 查找跨表依赖，避免每次重算都解析 AST。字段变更时同步更新该表（service 层 hook）。 + +**KTD-4. 自动化执行用独立 asyncio task，不进 recalc queue。** +自动化触发器在 service 层 record mutation hook 中同步匹配，匹配到的规则创建独立 asyncio task 执行动作。理由：1) 自动化动作（HTTP/邮件）延迟特性与公式重算不同；2) 失败重试逻辑独立；3) 避免污染 recalc queue 的 FOR UPDATE SKIP LOCKED 语义。重试用 exponential backoff（1s/2s/4s），3 次失败后标记 error。 + +**KTD-5. Agent 生成公式/自动化用"直接落地 + 可撤销"模式，不做确认弹窗。** +agent 生成结果直接创建字段/规则，但在响应中返回 `created_id` 和 `undo_token`。用户可在 30 秒内通过 `undo` action 撤销。理由：1) agent-native 架构强调低延迟；2) 确认弹窗打断 agent 流；3) 撤销比确认更符合人类直觉。`undo_token` 存 Redis（5 分钟 TTL）。 + +**KTD-6. 公式函数扩展按类别分模块，不堆在一个文件。** +当前 `functions.py` 已有 10 个函数。扩展到 40+ 后单文件过大。按类别拆分：`functions/datetime.py`、`functions/text.py`、`functions/logical.py`、`functions/aggregate.py`（含条件聚合）、`functions/lookup.py`。`functions/__init__.py` 聚合注册到 `FUNCTION_REGISTRY`。保持 `AGGREGATE_FUNCTIONS` frozenset 同步更新。 + +**KTD-7. Lookup/Rollup 作为独立 FieldType，不作为 formula 子类型。** +新增 `FieldType.lookup`（已存在，扩展 config）和 `FieldType.rollup`（新增）。理由：1) 类型明确，UI 渲染清晰；2) 重算逻辑独立（lookup 不聚合，rollup 聚合）；3) 对齐飞书/Notion/Baserow 的字段类型划分。config 形状：`lookup: {relation_field_id, target_field_id}`；`rollup: {relation_field_id, target_field_id, aggregation: "sum"|"count"|"avg"|"min"|"max"|"unique"}`。 + +**KTD-8. AST 深度限制在 parser 层而非 engine 层。** +在 `parse_formula()` 中递归计算 AST 深度，超过 15 抛 `FormulaDepthExceededError`。理由：1) 早失败，避免构建无效 DAG；2) validate-formula 端点直接返回错误；3) 不污染 engine 的拓扑排序逻辑。 + +**KTD-9. Webhook 入站用独立路由前缀 `/api/v1/bitable/webhooks/{token}`。** +不用 JWT 认证，用 webhook-specific token（每规则唯一）。理由：1) 外部系统无法获取 JWT；2) token 可在 UI 重置；3) 与现有 `/api/v1/bitable` 内部端点隔离，安全边界清晰。token 存 automation config，恒定时间比较。 + +### Implementation Units + +#### Phase 1: Storage & Model Foundation + +### U1. Schema V3 Migration + Data Models Extension + +**Goal:** 扩展 bitable 存储模型支持关联、自动化、跨表依赖追踪。新增 junction table、automation 表、依赖图表，扩展 Field/Record 模型。 + +**Requirements:** R5, R9, R16 + +**Dependencies:** 无（基础单元） + +**Files:** +- `src/agentkit/bitable/db.py` — bump `_SCHEMA_VERSION` 2→3，新增 ORM models（RelationLinkModel, AutomationModel, AutomationLogModel, CrossTableDepModel），新增 `_apply_v3_migration` +- `src/agentkit/bitable/models.py` — 新增 `FieldType.rollup`、`AutomationTrigger`、`AutomationAction`、`AutomationRule`、`AutomationLog`、`RelationLink`、`CrossTableDep` Pydantic 模型；扩展 `Field.config` 文档说明 relation/rollup 配置形状 +- `src/agentkit/bitable/repository.py` — 新增 RelationLinks / Automations / AutomationLogs / CrossTableDeps CRUD 方法 +- `tests/unit/bitable/test_db.py` — V3 migration 测试 +- `tests/unit/bitable/test_models.py` — 新模型序列化测试 + +**Approach:** +- `_apply_v3_migration(conn)` 创建 4 张新表（idempotent CREATE TABLE IF NOT EXISTS）： + - `bitable_relation_links`：`(id, relation_field_id, source_record_id, target_record_id, created_at)` + 索引 `(relation_field_id, source_record_id)` 和 `(relation_field_id, target_record_id)` 用于双向查询 + - `bitable_automations`：`(id, table_id, name, trigger_config JSONB, action_config JSONB, enabled BOOL, created_at)` + 索引 `table_id` + - `bitable_automation_logs`：`(id, automation_id, trigger_record_id, status, error_message, started_at, completed_at)` + 索引 `automation_id, started_at` + - `bitable_cross_table_deps`：`(id, source_table_id, source_field_id, target_table_id, target_field_id, dep_type)` + 唯一约束 `(source_field_id, target_field_id)` + 索引 `target_field_id`（反向查询用） +- `FieldType` 新增 `rollup = "rollup"`；`relation` 不新增（用 `lookup` field_type + `config.relation_field_id` 表示关联字段，对齐现有 lookup 设计但扩展 config） +- `Field.config` 形状扩展：relation 字段 `{"relation_type": "one_to_one"|"one_to_many"|"many_to_many", "target_table_id": "...", "is_bidirectional": bool, "reverse_field_id": "..."}`；rollup 字段 `{"relation_field_id": "...", "target_field_id": "...", "aggregation": "sum"|"count"|"avg"|"min"|"max"|"unique"}` + +**Patterns to follow:** +- `_apply_v2_migration` 的 idempotent 模式（CREATE TABLE IF NOT EXISTS + ADD COLUMN IF NOT EXISTS） +- 现有 ORM models 的 `__table_args__` schema 限定模式 +- Pydantic `model_config = ConfigDict(from_attributes=True)` 约定 + +**Test scenarios:** +- **Happy path:** 全新 DB init → V3 schema 创建 4 张新表 + V2 表保留 +- **Migration:** V2 DB → init → V3 migration 应用，新表创建，现有数据不丢失 +- **Idempotent:** V3 DB → init → 不重复创建表 +- **Covers AE2 (junction table 创建部分).** +- **Edge case:** meta 表 schema_version 正确更新为 3 +- **Integration:** repository 新增 CRUD 方法（create_relation_link, list_relation_links_by_source, etc.）读写正常 + +**Verification:** `pytest tests/unit/bitable/test_db.py tests/unit/bitable/test_models.py -x -q` 通过；V3 migration 在空 DB 和 V2 DB 上都幂等。 + +--- + +#### Phase 2: Formula Engine Expansion + +### U2. Formula Function Expansion (10 → 40+) + +**Goal:** 扩展公式函数从 10 个到 40+，覆盖日期、文本、逻辑、条件聚合、查找引用五大类别，保持 AST 白名单安全。 + +**Requirements:** R1, R3, R4 + +**Dependencies:** 无（与 U1 并行） + +**Files:** +- `src/agentkit/bitable/formula/functions/` — 新建包目录 + - `__init__.py` — 聚合所有子模块到 `FUNCTION_REGISTRY` 和 `AGGREGATE_FUNCTIONS` + - `datetime.py` — DATE, DATEDIF, NETWORKDAYS, WEEKDAY, EOMONTH, TODAY, NOW, YEAR, MONTH, DAY, HOUR, MINUTE, SECOND, DATEADD, TIMESTAMP（15 个） + - `text.py` — FIND, MID, REPLACE, SUBSTITUTE, UPPER, LOWER, TRIM, LEFT, RIGHT, REPT, TEXT, VALUE, STARTSWITH, ENDSWITH, SPLIT（15 个） + - `logical.py` — IFS, SWITCH, IFERROR, ISBLANK, AND, OR, NOT, ISNUMBER, ISTEXT, ISDATE, COALESCE, NULL, TRUE, FALSE（14 个） + - `aggregate.py` — SUMIF, COUNTIF, AVERAGEIF, SUMIFS, COUNTIFS（5 个条件聚合，迁移现有 SUM/AVG/COUNT/MIN/MAX） + - `lookup.py` — LOOKUP, FILTER, MATCH, VLOOKUP, XLOOKUP（5 个，跨表引用支持在 U3 实现） +- `src/agentkit/bitable/formula/functions.py` — 删除（迁移到包），保留兼容 re-export +- `src/agentkit/bitable/formula/parser.py` — 新增 `FormulaDepthExceededError`，`parse_formula` 中计算 AST 深度，超 15 抛错；`_ALLOWED_NODES` 扩展支持 `ast.List`（用于 FILTER/MATCH 返回列表）和 `ast.Tuple`（用于多参数） +- `src/agentkit/bitable/formula/engine.py` — `evaluate_all_for_record` 扩展支持列表类型返回值 +- `tests/unit/bitable/test_formula_parser.py` — 深度限制测试 +- `tests/unit/bitable/test_formula_engine.py` — 40+ 函数逐个测试 +- `tests/unit/bitable/test_functions_datetime.py` — 日期函数专项测试 +- `tests/unit/bitable/test_functions_text.py` — 文本函数专项测试 +- `tests/unit/bitable/test_functions_logical.py` — 逻辑函数专项测试 +- `tests/unit/bitable/test_functions_aggregate.py` — 条件聚合测试 + +**Approach:** +- 日期函数处理 ISO 8601 字符串（现有 date 字段存储格式），NETWORKDAYS 用 Python `datetime.weekday()` 排除周末（不支持节假日表，留给 v3） +- 文本函数对齐 Excel/Notion 命名（FIND/MID/REPLACE 与 Excel 一致） +- 条件聚合 `SUMIF(range, criteria, [sum_range])` — range 是字段引用（列），criteria 是字符串/数字/表达式，sum_range 可选 +- LOOKUP/FILTER/MATCH 在 U3 实现跨表引用逻辑，本单元先实现单表内查找 +- AST 深度计算：递归 `_ast_depth(node)` 返回 `1 + max(child_depths)`，在 `parse_formula` 末尾检查 +- `FUNCTION_REGISTRY` 兼容：现有代码 `from agentkit.bitable.formula.functions import FUNCTION_REGISTRY` 通过 `__init__.py` re-export 保持不变 + +**Patterns to follow:** +- 现有 `functions.py` 的 `FormulaResult: TypeAlias = str | int | float | None` 类型约定 +- 现有 `_sum`/`_avg` 的 None/空字符串过滤模式 +- 现有 `AGGREGATE_FUNCTIONS` frozenset 注册模式 + +**Test scenarios:** +- **Happy path (per function):** 每个新函数至少 1 个 happy path 测试，验证返回值正确 +- **Covers AE3:** `=NETWORKDAYS({start_date}, {end_date})` 验证通过并返回工作日天数 +- **Edge cases:** 空输入（None/空字符串）处理；日期函数对非日期字符串报错；条件聚合无匹配返回 0 +- **Error path:** 未知函数仍抛 `UnknownFunctionError`；AST 深度超 15 抛 `FormulaDepthExceededError` +- **Integration:** `parse_formula("=SUMIF({f1}, '>10')")` + `evaluate_ast` 全流程通过；validate-formula 端点对新函数返回 200 +- **Security:** 尝试 `=eval("code")` 仍被 `_SecurityVisitor` 拒绝；`=__import__('os')` 拒绝 + +**Verification:** `pytest tests/unit/bitable/test_functions_*.py tests/unit/bitable/test_formula_parser.py -x -q` 通过；40+ 函数全部在 `FUNCTION_REGISTRY` 注册。 + +--- + +### U3. Cross-Table Formula References + +**Goal:** 支持跨表公式引用 `{relation_field}.{target_field}` 行级 + `[relation_field].[target_field]` 整列，依赖 U1 的 relation 基础设施。 + +**Requirements:** R2 + +**Dependencies:** U1, U2 + +**Files:** +- `src/agentkit/bitable/formula/parser.py` — 扩展 `_FIELD_REF_RE` 支持 `.` 分隔的二级路径；`_substitute_field_refs` 返回 `field_mapping` 增加 `cross_table_refs: list[(safe_name, relation_field_id, target_field_id)]` +- `src/agentkit/bitable/formula/engine.py` — `add_formula` 扩展 `_classify_refs` 识别跨表引用；`evaluate` 扩展 `relation_context` 参数；新增 `_resolve_cross_table_ref` 方法 +- `src/agentkit/bitable/formula/functions/lookup.py` — 实现 LOOKUP/FILTER/MATCH 的跨表逻辑（依赖 relation_context） +- `src/agentkit/bitable/service.py` — `create_field` / `update_field` 中解析公式跨表引用，写入 `bitable_cross_table_deps`（KTD-3） +- `tests/unit/bitable/test_formula_parser.py` — 跨表引用解析测试 +- `tests/unit/bitable/test_formula_engine.py` — 跨表公式求值测试 + +**Approach:** +- parser 扩展正则：`_FIELD_REF_RE = re.compile(r"\{([a-zA-Z0-9_-]+)(?:\.([a-zA-Z0-9_-]+))?\}")`，第二组为可选 target_field +- 整列引用 `[rel_field].[target_field]` 用单独正则 `_\[([a-zA-Z0-9_-]+)\]\.\[([a-zA-Z0-9_-]+)\]`，替换为 `__col__` Name 节点 +- engine `evaluate` 签名扩展：新增 `relation_context: dict[str, list[dict[str, object]]]` 参数，key 为 relation_field_id，value 为关联记录列表 +- service 层在创建/更新 formula 字段时，解析公式 AST 提取跨表引用，写入 `bitable_cross_table_deps`（dep_type: `formula`） +- LOOKUP/FILTER/MATCH 函数接收 relation_context，通过 target_field_id 拉取值 + +**Patterns to follow:** +- 现有 `_substitute_field_refs` 的 safe_name 映射模式 +- 现有 `_classify_refs` 的 aggregate vs row 分类模式 + +**Test scenarios:** +- **Happy path:** `={rel_field}.{target_field}` 解析为跨表行级引用；`=[rel_field].[target_field]` 解析为整列引用 +- **Covers AE1 (跨表公式部分):** Orders.amount 修改 → Customers.total_amount（通过 rollup，U5 实现）跨表重算 +- **Edge case:** 自引用关联（rel_field 指向当前表）正常解析 +- **Error path:** relation_field 不存在 → 抛 `FormulaParseError`；target_field 不存在 → 抛 `FormulaParseError` +- **Integration:** service.create_field 写入 cross_table_deps 记录；engine.evaluate 传入 relation_context 后正确求值 + +**Verification:** `pytest tests/unit/bitable/test_formula_parser.py tests/unit/bitable/test_formula_engine.py -x -q` 通过；跨表引用解析和求值正确。 + +--- + +#### Phase 3: Relations + +### U4. Relation CRUD + Junction Table Management + +**Goal:** 实现单向/双向/多对多/自引用关联字段的创建、更新、删除，管理 junction table 生命周期。 + +**Requirements:** R5, R7, R8 + +**Dependencies:** U1 + +**Files:** +- `src/agentkit/bitable/service.py` — 新增 `create_relation_field`、`update_relation_link`（添加/删除关联记录）、`delete_relation_field`（级联处理）方法 +- `src/agentkit/bitable/repository.py` — 新增 `add_relation_link`、`remove_relation_link`、`list_relation_links`、`list_reverse_relation_links` 方法 +- `src/agentkit/server/routes/bitable.py` — 新增端点：`POST /tables/{id}/fields`（扩展支持 relation field_type）、`POST /fields/{id}/links`（添加关联）、`DELETE /fields/{id}/links/{record_id}`（删除关联） +- `tests/unit/bitable/test_service.py` — relation CRUD 测试 +- `tests/unit/bitable/test_routes.py` — relation 端点测试 + +**Approach:** +- `create_relation_field(table_id, name, target_table_id, relation_type, is_bidirectional)`： + - one_to_one / one_to_many：关联值存记录 JSONB `values[relation_field_id]` = `["rec_id"]` 或 `["rec_id1", "rec_id2"]` + - many_to_many：创建独立 junction table 引用（KTD-1），字段 config 存 `relation_type` + `target_table_id` + - is_bidirectional=True：在 target_table 自动创建反向字段，config 互相引用 `reverse_field_id` + - 自引用：target_table_id = 当前 table_id，正常流程 +- `update_relation_link(field_id, source_record_id, target_record_ids)`：many_to_many 操作 `bitable_relation_links`；其他类型更新记录 JSONB +- `delete_relation_field(field_id, force)`：扩展 `FieldDependencyError` 检查——查找依赖该 relation 的 lookup/rollup/跨表公式字段；force=True 时级联删除依赖字段的 config + +**Patterns to follow:** +- 现有 `delete_field` 的 `FieldDependencyError` + force 级联模式（service.py:234-282） +- 现有 `_check_table_ownership` IDOR 模式 +- 现有 `create_field` 的 config 验证模式 + +**Test scenarios:** +- **Happy path:** 创建单向关联（A→B）；创建双向关联（A↔B，自动建反向字段）；创建多对多关联（junction table 生成） +- **Covers AE2 (关联创建部分):** Students↔Courses 多对多创建，双向字段生成 +- **Covers F2:** 多对多关联创建流程完整验证 +- **Edge case:** 自引用关联（table A 关联自身）正常工作 +- **Error path:** target_table_id 不存在 → 400；删除有依赖的 relation 字段未 force → `FieldDependencyError` +- **Integration:** 创建关联字段 → 添加关联记录 → 查询关联记录列表 → 删除关联 + +**Verification:** `pytest tests/unit/bitable/test_service.py tests/unit/bitable/test_routes.py -x -q` 通过；三种关联类型 CRUD 完整。 + +--- + +### U5. Lookup & Rollup Fields + +**Goal:** 实现 Lookup 字段拉取关联记录字段值，Rollup 字段对关联记录聚合（sum/count/avg/min/max/unique）。 + +**Requirements:** R6, R8 + +**Dependencies:** U1, U4 + +**Files:** +- `src/agentkit/bitable/service.py` — 新增 `create_lookup_field`、`create_rollup_field` 方法；扩展 `_trigger_recalc_for_affected_fields` 识别 lookup/rollup 字段 +- `src/agentkit/bitable/recalc_worker.py` — 扩展 `process_task` 处理 lookup/rollup 字段类型；新增 `_resolve_lookup`、`_resolve_rollup` 方法 +- `src/agentkit/bitable/formula/engine.py` — 扩展支持 lookup/rollup 字段作为"虚拟公式"注册到 DAG +- `tests/unit/bitable/test_service.py` — lookup/rollup 创建测试 +- `tests/unit/bitable/test_recalc.py` — lookup/rollup 重算测试 + +**Approach:** +- lookup 字段 config: `{"relation_field_id": "...", "target_field_id": "..."}`，重算时通过 relation 找到关联记录，拉取 target_field 值 +- rollup 字段 config: `{"relation_field_id": "...", "target_field_id": "...", "aggregation": "sum"}`，重算时拉取关联记录的 target_field 列表，应用聚合函数 +- recalc_worker 扩展：`process_task` 检查 field.field_type，formula → 现有逻辑；lookup → `_resolve_lookup`；rollup → `_resolve_rollup` +- `_resolve_lookup(field, record)`：查 `bitable_relation_links`（多对多）或记录 JSONB（一对多）获取关联记录 ID 列表 → `get_records_batch` → 提取 target_field 值 → 返回（单值或列表） +- `_resolve_rollup(field, record)`：同 lookup 拉取值列表 → 应用 aggregation 函数 → 返回标量 +- 跨表依赖：lookup/rollup 字段创建时写入 `bitable_cross_table_deps`（dep_type: `lookup` / `rollup`） + +**Patterns to follow:** +- 现有 `process_task` 的 formula 处理流程（recalc_worker.py:177-235） +- 现有 `get_column_values` 批量拉取模式 +- 现有 `FieldDependencyError` 级联模式（R8） + +**Test scenarios:** +- **Happy path:** 创建 lookup 字段拉取关联记录的 text 字段；创建 rollup 字段对关联记录 number 字段求和 +- **Covers AE2 (lookup/rollup 部分):** Students 创建 lookup 拉取 Course.credit；创建 rollup 统计总学分 +- **Covers AE1:** Orders.amount 修改 → Customers.total_amount（rollup SUM）跨表重算 +- **Edge case:** 关联记录为空时 lookup 返回 None，rollup 返回 0；unique 聚合去重 +- **Error path:** relation_field 已删除 → 字段 config 标记 error；target_field 已删除 → 同上 +- **Integration:** 创建关联 → 添加关联记录 → 创建 rollup → 验证重算结果正确 + +**Verification:** `pytest tests/unit/bitable/test_service.py tests/unit/bitable/test_recalc.py -x -q` 通过；lookup/rollup 重算结果正确。 + +--- + +#### Phase 4: Cross-Table Recalc + +### U6. Cross-Table Dependency Graph + Recalc Worker Extension + +**Goal:** 构建跨表字段级依赖图，扩展 recalc worker 支持跨表重算，异步不阻塞原始写入。 + +**Requirements:** R10, R11, R12 + +**Dependencies:** U1, U3, U5 + +**Files:** +- `src/agentkit/bitable/repository.py` — 新增 `find_cross_table_dependents(target_table_id, target_field_id)` 方法 +- `src/agentkit/bitable/recalc_worker.py` — 扩展 `_run` 循环处理跨表任务；新增 `_enqueue_cross_table_recalc` 方法；`process_task` 扩展跨表上下文加载 +- `src/agentkit/bitable/service.py` — `_trigger_recalc_for_affected_fields` 扩展：查跨表依赖图，enqueue 跨表任务 +- `src/agentkit/bitable/models.py` — `RecalcTask` 新增 `is_cross_table: bool` 字段 +- `src/agentkit/bitable/db.py` — `RecalcQueueModel` 新增 `is_cross_table` 列（V3 migration 一部分） +- `tests/unit/bitable/test_recalc.py` — 跨表重算测试 + +**Approach:** +- `RecalcTask.is_cross_table` 标记跨表任务（默认 False） +- service `_trigger_recalc_for_affected_fields(table_id, record_id, changed_field_ids)`： + 1. 查本表公式字段依赖 → enqueue 本表 recalc（现有逻辑） + 2. 查 `bitable_cross_table_deps` where `target_table_id = table_id AND target_field_id IN changed_field_ids` → 对每个依赖字段，找到依赖它的记录（通过 relation_links）→ enqueue 跨表 recalc 任务 +- worker `process_task` 扩展：if `task.is_cross_table` → 加载 relation_context（关联记录列表）→ 传给 engine.evaluate +- 跨表依赖图构建在字段创建/更新时同步（U3 service hook + U5 lookup/rollup 创建 hook） +- 拓扑排序扩展：跨表依赖也加入 DAG，循环检测覆盖跨表 + +**Patterns to follow:** +- 现有 `claim_recalc_tasks` 的 `FOR UPDATE SKIP LOCKED` 模式 +- 现有 `_sort_by_topological_order` 的 Kahn 算法 +- 现有 `reset_stale_recalc_tasks` 的 crash recovery 模式 + +**Test scenarios:** +- **Happy path:** 修改表 A 记录 → 表 B 依赖该记录的 rollup 字段异步重算并更新 +- **Covers AE1 完整:** Orders.amount 修改 → Customers.total_amount 跨表重算 +- **Covers F1:** 跨表公式重算流程完整验证 +- **Edge case:** 无跨表依赖时不 enqueue 跨表任务；循环依赖检测（A 依赖 B，B 依赖 A）抛 `CircularReferenceError` +- **Error path:** 跨表任务失败 → 状态 error + error_message；stale 任务被 reaper 重置 +- **Integration:** 创建关联 → 创建 rollup → 修改源记录 → 验证目标记录异步更新（轮询 recalc 完成状态） + +**Verification:** `pytest tests/unit/bitable/test_recalc.py -x -q` 通过；跨表重算异步且正确。 + +--- + +#### Phase 5: Automation + +### U7. Automation Engine (Triggers + Actions + Webhook + Retry + Logging) + +**Goal:** 实现 3 种触发器、5 种动作、Webhook 入站、失败重试 3 次、执行日志可查。 + +**Requirements:** R13, R14, R15, R16, R17 + +**Dependencies:** U1 + +**Files:** +- `src/agentkit/bitable/automation/` — 新建包 + - `__init__.py` + - `engine.py` — `AutomationEngine` 类：`match_triggers(table_id, record_id, change_type, changed_fields)`、`execute_action(rule, trigger_record)` + - `triggers.py` — 触发器匹配逻辑（record_created / record_updated / record_deleted） + - `actions.py` — 5 种动作执行器（create_record / update_record / send_webhook / send_email / send_message） + - `webhook.py` — Webhook 入站端点处理 + 出站 HTTP 调用（含 SSRF guard，复用 `ingestion/excel.py` 的 SSRF 防护） +- `src/agentkit/bitable/service.py` — record mutation hook 调用 `AutomationEngine.match_triggers`；`create_automation_rule`、`list_automation_rules`、`update_automation_rule`、`delete_automation_rule`、`list_automation_logs` 方法 +- `src/agentkit/server/routes/bitable.py` — 新增端点：`POST /tables/{id}/automations`、`GET /tables/{id}/automations`、`PATCH /automations/{id}`、`DELETE /automations/{id}`、`GET /automations/{id}/logs`、`POST /webhooks/{token}`（入站 webhook，无 JWT 认证） +- `tests/unit/bitable/test_automation.py` — 自动化引擎测试 +- `tests/unit/bitable/test_routes.py` — 自动化端点测试 + +**Approach:** +- `AutomationEngine` 在 service record mutation hook 后同步调用 `match_triggers`，匹配到的规则创建独立 asyncio task（KTD-4） +- 触发器 config 形状：`{"type": "record_updated", "field_ids": ["f1"]}`（field_ids 可选，不指定则任何字段变更触发） +- 动作 config 形状：`{"type": "send_webhook", "url": "https://...", "method": "POST", "headers": {...}}`；`{"type": "create_record", "target_table_id": "...", "values": {...}}`；`{"type": "send_email", "to": "...", "subject": "...", "body": "..."}`；`{"type": "send_message", "channel": "...", "message": "..."}` +- Webhook 出站复用 `ingestion/excel.py` 的 `_is_private_ip` / `_validate_url` SSRF 防护 +- Webhook 入站：`POST /api/v1/bitable/webhooks/{token}` → 查 automation by token（恒定时间比较）→ 匹配 trigger → 执行 action +- 重试：exponential backoff（1s/2s/4s），3 次失败后 `AutomationLog.status = "error"` +- `send_message` 集成现有 `bus/` MemoryBus 或 RedisBus（视配置） +- `send_email` 用 `aiosmtplib`（如已安装）或 stdlib `smtplib` via `asyncio.to_thread` + +**Patterns to follow:** +- 现有 `RecalcWorker` 的 asyncio task 生命周期管理（start/stop/cancel） +- 现有 `ingestion/excel.py` 的 SSRF 防护（`_is_private_ip`, URL scheme 验证） +- 现有 `hmac.compare_digest` 恒定时间比较模式 +- 现有 route 的 IDOR ownership 检查模式 + +**Test scenarios:** +- **Happy path:** 创建自动化规则（record_updated + send_webhook）→ 修改记录 → webhook 被调用 +- **Covers AE4 完整:** Tasks.status 改为"已完成" → webhook 触发 → 失败重试 3 次 → 标记 error +- **Covers F4:** 自动化规则执行流程完整验证 +- **Edge case:** trigger 指定 field_ids 时，非指定字段变更不触发；webhook 出站 URL 为私网 IP → 拒绝（SSRF guard） +- **Error path:** 动作执行失败 → 重试 3 次 → AutomationLog 记录 error + error_message；webhook token 无效 → 404 +- **Integration:** 创建规则 → 触发 → 查询日志 → 日志显示成功/失败状态 + +**Verification:** `pytest tests/unit/bitable/test_automation.py tests/unit/bitable/test_routes.py -x -q` 通过；5 种动作 + 重试 + 日志完整。 + +--- + +#### Phase 6: Agent Integration + +### U8. BitableTool Agent Actions + REST API + Frontend + +**Goal:** BitableTool 扩展 4 个新 action，实现 agent 生成公式/自动化 + 创建关联/Rollup 的差异化能力，配套前端组件。 + +**Requirements:** R18, R19, R20 + +**Dependencies:** U1, U2, U3, U4, U5, U6, U7 + +**Files:** +- `src/agentkit/tools/bitable_tool.py` — 新增 4 个 action：`generate_formula`、`generate_automation`、`create_relation`、`create_rollup`；新增 `undo` action（撤销最近创建） +- `src/agentkit/server/routes/bitable.py` — 新增端点：`POST /tables/{id}/generate-formula`（LLM 生成 + validate + 创建）、`POST /tables/{id}/generate-automation`、`POST /undo/{undo_token}` +- `src/agentkit/server/frontend/src/api/bitable.ts` — 新增 API 方法：`generateFormula`、`generateAutomation`、`createRelation`、`createRollup`、`undoAction` +- `src/agentkit/server/frontend/src/components/bitable/RelationFieldEditor.vue` — 新建：关联字段配置组件（选择目标表、关联类型、双向开关） +- `src/agentkit/server/frontend/src/components/bitable/RollupFieldEditor.vue` — 新建：Rollup 字段配置组件（选择关联字段、目标字段、聚合函数） +- `src/agentkit/server/frontend/src/components/bitable/AutomationRuleList.vue` — 新建：自动化规则列表 + 表单式编辑器 +- `src/agentkit/server/frontend/src/components/bitable/FormulaGeneratorModal.vue` — 新建：自然语言生成公式 modal +- `src/agentkit/server/frontend/src/stores/bitable.ts` — 扩展 state：`automations`、`automationLogs`；actions：`generateFormula`、`createRelation` 等 +- `tests/unit/bitable/test_bitable_tool.py` — 新 action 测试 + +**Approach:** +- `generate_formula(table_id, description, field_name)`： + 1. 调用 LLM（通过现有 `llm/` 网关）将 description + 表 schema 转为公式表达式 + 2. 调用 `validate-formula` 端点验证语法（R20 复用） + 3. 验证通过 → `create_field(field_type="formula", config={"formula_expr": ...})` + 4. 返回 `{success, field_id, undo_token}` + 5. undo_token 存 Redis（5 分钟 TTL），undo 时删除字段 +- `generate_automation(table_id, description)`：类似流程，LLM 生成 trigger_config + action_config +- `create_relation(table_id, name, target_table_id, relation_type, is_bidirectional)`：调用 U4 的 service 方法 +- `create_rollup(table_id, name, relation_field_id, target_field_id, aggregation)`：调用 U5 的 service 方法 +- LLM prompt 工程：传入表字段 schema + 用户描述，要求 LLM 返回严格 JSON `{formula_expr: "..."}` 或 `{trigger: {...}, action: {...}}` +- 前端组件遵循现有 Ant Design Vue 4 风格，`FieldConfigForm.vue` 扩展支持 relation/rollup 类型配置 + +**Patterns to follow:** +- 现有 BitableTool 的 handler dispatch + httpx.AsyncClient 模式（bitable_tool.py:175-186） +- 现有 `_create_view` / `_update_field` 的 REST 调用模式 +- 现有 `FieldConfigForm.vue` 的表单布局 +- 现有 Pinia store 的 actions 模式 + +**Test scenarios:** +- **Happy path:** `generate_formula(description="计算总价 = 单价 × 数量")` → LLM 生成 `={price} * {quantity}` → validate 通过 → 创建字段 +- **Covers AE5 完整:** Agent 生成公式流程端到端验证 +- **Covers F3:** Agent 生成公式流程完整验证 +- **Edge case:** LLM 生成无效公式 → validate 失败 → 返回错误不创建字段；undo 在 5 分钟内有效，超时失效 +- **Error path:** LLM 调用失败 → 返回错误；表不存在 → 404 +- **Integration:** agent 调用 → 生成 → 创建字段 → recalc 触发 → 撤销 → 字段删除 +- **Frontend smoke:** RelationFieldEditor 选择目标表 + 关联类型 → 保存 → 字段创建；FormulaGeneratorModal 输入描述 → 生成 → 字段创建 + +**Verification:** `pytest tests/unit/bitable/test_bitable_tool.py -x -q` 通过；`cd src/agentkit/server/frontend && npm run typecheck` 通过；4 个新 action 端到端可用。 + +--- + +### Verification Contract + +**Per-unit verification (each U-ID):** +- 每个单元的 `pytest tests/unit/bitable/test_*.py -x -q` 必须通过 +- 涉及前端改动的单元（U8）需 `npm run typecheck` 通过 +- 涉及 DB migration 的单元（U1）需在空 DB 和 V2 DB 上都幂等 + +**Integration verification (after all units):** +- `pytest tests/unit/bitable/ -x -q` 全部通过 +- `ruff check src/agentkit/bitable/ src/agentkit/tools/bitable_tool.py && ruff format src/agentkit/bitable/` 无错误 +- 手动 smoke test（可选）：启动服务 → 创建两表 → 建立多对多关联 → 创建 rollup → 修改源记录 → 验证跨表重算 + +**Acceptance Examples enforcement:** +- AE1 (跨表重算): U5 + U6 实现并测试 +- AE2 (多对多): U1 + U4 + U5 实现并测试 +- AE3 (公式扩展): U2 实现并测试 +- AE4 (自动化): U7 实现并测试 +- AE5 (Agent 生成): U8 实现并测试 + +**Behavior change verification:** +- 公式引擎扩展是增量，现有 10 个函数行为不变（向后兼容） +- 现有 lookup 字段类型扩展 config，旧 lookup 字段（无 relation_field_id）仍可读 +- recalc worker 扩展，现有单表 recalc 流程不变 + +### Definition of Done + +- [ ] U1-U8 全部实现，单元测试通过 +- [ ] `pytest tests/unit/bitable/ -x -q` 全绿 +- [ ] `ruff check src/agentkit/bitable/ && ruff format src/agentkit/bitable/` 无错误 +- [ ] 前端 `npm run typecheck` 通过（U8 部分） +- [ ] Schema V3 migration 在空 DB 和 V2 DB 上幂等 +- [ ] 现有 bitable 功能无回归（现有测试全绿） +- [ ] AE1-AE5 验收示例对应的测试场景全部通过 +- [ ] BitableTool 新 action 通过 REST API 可用 +- [ ] 公式引擎 40+ 函数在 `FUNCTION_REGISTRY` 注册 +- [ ] 跨表依赖图正确构建和更新 +- [ ] 自动化引擎 3 触发器 + 5 动作 + 重试 + 日志完整 + +### Open Questions + +- **email 发送实现**：`aiosmtplib` 是否已在依赖中？如无，用 stdlib `smtplib` via `asyncio.to_thread`（ponytail: 不引入新依赖）。实现时检查 `pyproject.toml`。 +- **LLM 网关调用细节**：`generate_formula` 的 LLM 调用走现有 `llm/` 网关的哪个接口？实现时参考 `chat/handler.py` 的 LLM 调用模式。 +- **前端组件 SSR**：Tauri 桌面端是否需要同步更新？v2 聚焦 Web，Tauri 兼容性留给实现时验证。 + +### Risks & Dependencies + +**Risks:** +- **跨表重算性能**：跨表依赖图可能在大数据量下产生大量 recalc 任务。缓解：`is_cross_table` 标记 + 批量 enqueue + worker 限流（现有 `claim_recalc_tasks(limit=10)`）。 +- **LLM 生成公式质量**：LLM 可能生成语法正确但语义错误的公式。缓解：validate-formula 仅验证语法；agent 返回结果包含 `formula_expr` 供用户审查；undo 机制。 +- **多对多 junction table 膨胀**：高频关联操作可能产生大量 junction 记录。缓解：定期清理孤立链接（v3 优化）；索引覆盖查询路径。 +- **Schema migration 风险**：V3 migration 在生产数据上失败。缓解：idempotent CREATE IF NOT EXISTS；现有 V2 migration 模式已验证；无生产数据（grounding 确认）。 + +**Dependencies:** +- 现有 `llm/` 网关（U8 LLM 调用） +- 现有 `bus/` 消息总线（U7 send_message 动作） +- 现有 `ingestion/excel.py` SSRF 防护（U7 webhook 出站） +- 现有 `hmac.compare_digest`（U7 webhook token 比较） + +### System-Wide Impact + +- **后端**：bitable 子系统独立 schema 扩展，不影响其他模块；新增 `automation/` 包；`tools/bitable_tool.py` 扩展 +- **前端**：新增 4 个组件，扩展 store 和 API client；现有 `FieldConfigForm.vue` 扩展（向后兼容） +- **数据库**：bitable schema V3，4 张新表；不影响其他 schema +- **部署**：无新依赖（ponytail: 复用现有）；无新环境变量（webhook token 存 DB） +- **文档**：无需更新 AGENTS.md（bitable 模块自包含） + +### Phased Delivery + +| 阶段 | 单元 | 依赖 | 可交付价值 | +|---|---|---|---| +| 1 | U1 | 无 | 存储基础 + 数据模型 | +| 2 | U2, U4 | U1 | 公式 40+ 函数 + 关联 CRUD | +| 3 | U3, U5 | U1, U2, U4 | 跨表公式 + Lookup/Rollup | +| 4 | U6 | U1, U3, U5 | 跨表重算 | +| 5 | U7 | U1 | 自动化引擎 | +| 6 | U8 | 全部 | Agent 集成 + 前端 | + +**执行方向提示：** 每个单元实现后立即运行对应测试；U1 migration 优先验证幂等性；U2 函数扩展按类别分模块并行实现；U6 跨表重算需端到端集成测试。 diff --git a/src/agentkit/bitable/automation.py b/src/agentkit/bitable/automation.py new file mode 100644 index 0000000..507c5ce --- /dev/null +++ b/src/agentkit/bitable/automation.py @@ -0,0 +1,349 @@ +"""Automation engine — trigger evaluation, action execution, retry, logging (U7, R13-R17). + +Lifecycle: + engine = AutomationEngine(service) + await engine.start() # starts background task + ... + await engine.stop() + +Triggers (R13): + - record_created: fires when a record is created in the watched table + - record_updated: fires when a record is updated (optionally filtered by field_ids) + - record_deleted: fires when a record is deleted + +Actions (R14): + - create_record: insert a new record in a target table + - update_record: update fields on the triggering record (or a related record) + - send_webhook: POST to a URL with the trigger payload + - send_email: placeholder — logs the email content (SMTP integration is app-level) + - send_message: placeholder — logs the message (channel integration is app-level) + +Retry: each action is retried up to 3 times with exponential backoff (1s, 2s, 4s). +All executions are logged to ``automation_logs`` (R17). + +Webhook inbound (R16): ``handle_webhook(token, payload)`` matches a rule by its +``webhook_token`` and fires its action with the payload as trigger context. +""" + +from __future__ import annotations + +import asyncio +import json +import logging +import secrets +from datetime import datetime, timezone +from typing import TYPE_CHECKING, Any + +import httpx + +from agentkit.bitable.ingestion.excel import _assert_safe_host +from agentkit.bitable.models import ( + AutomationActionType, + AutomationStatus, + AutomationTriggerType, +) + +if TYPE_CHECKING: + from agentkit.bitable.service import BitableService + +logger = logging.getLogger(__name__) + +_MAX_RETRIES = 3 +_RETRY_DELAYS = [1.0, 2.0, 4.0] # seconds, exponential backoff +_WEBHOOK_TIMEOUT = 10.0 # seconds + + +class AutomationEngine: + """Background engine that evaluates triggers and executes actions. + + The engine itself is triggerless — it exposes ``fire_trigger`` which + the service layer calls after record writes. Actions execute in an + isolated asyncio task (KTD-4) so failures don't block the caller. + """ + + def __init__(self, service: BitableService) -> None: + self._service = service + self._repo = service.repo + self._task: asyncio.Task[None] | None = None + self._queue: asyncio.Queue[dict[str, Any]] = asyncio.Queue() + self._stop_event = asyncio.Event() + + async def start(self) -> None: + """Start the background action executor.""" + self._stop_event.clear() + self._task = asyncio.create_task(self._run(), name="automation-engine") + logger.info("AutomationEngine started") + + async def stop(self) -> None: + """Gracefully stop the engine.""" + self._stop_event.set() + if self._task is not None: + self._task.cancel() + try: + await self._task + except asyncio.CancelledError: + pass + self._task = None + logger.info("AutomationEngine stopped") + + async def _run(self) -> None: + """Main loop — process action executions from the queue.""" + while not self._stop_event.is_set(): + try: + job = await asyncio.wait_for(self._queue.get(), timeout=1.0) + await self._execute_with_retry(job) + except asyncio.TimeoutError: + continue + except asyncio.CancelledError: + break + except Exception: + logger.exception("AutomationEngine error in main loop") + + async def fire_trigger( + self, + table_id: str, + trigger_type: AutomationTriggerType, + record_id: str | None = None, + changed_field_ids: list[str] | None = None, + record_values: dict[str, object] | None = None, + ) -> int: + """Evaluate trigger rules for a table event and enqueue matching actions. + + Returns the number of rules matched (not necessarily executed yet — + actions run asynchronously). + + Args: + table_id: The table where the event occurred. + trigger_type: create / update / delete. + record_id: The affected record (None for bulk operations). + changed_field_ids: For record_updated — only fire if the rule + watches these fields (empty/None means any field). + record_values: Snapshot of the record's values (for action context). + """ + rules = await self._repo.list_automations_for_trigger( + table_id=table_id, trigger_type=trigger_type.value + ) + matched = 0 + for rule in rules: + if not rule.enabled: + continue + # For record_updated with field_ids filter, check overlap + if trigger_type == AutomationTriggerType.record_updated: + watched = rule.trigger_config.get("field_ids", []) + if watched and changed_field_ids: + if not set(watched) & set(changed_field_ids): + continue # No overlap — skip + + matched += 1 + await self._queue.put( + { + "automation_id": rule.id, + "table_id": table_id, + "trigger_type": trigger_type.value, + "record_id": record_id, + "record_values": record_values or {}, + "action_config": rule.action_config, + } + ) + return matched + + async def handle_webhook(self, token: str, payload: dict[str, Any]) -> bool: + """Handle an inbound webhook (R16). + + Returns True if a rule matched the token and was executed. + """ + rule = await self._repo.get_automation_by_webhook_token(token) + if rule is None or not rule.enabled: + return False + await self._queue.put( + { + "automation_id": rule.id, + "table_id": rule.table_id, + "trigger_type": "webhook", + "record_id": None, + "record_values": payload, + "action_config": rule.action_config, + } + ) + return True + + async def _execute_with_retry(self, job: dict[str, Any]) -> None: + """Execute an action with up to 3 retries (R15).""" + automation_id = job["automation_id"] + record_id = job.get("record_id") + last_error: str | None = None + + for attempt in range(1, _MAX_RETRIES + 1): + log_id = await self._repo.create_automation_log( + automation_id=automation_id, + trigger_record_id=record_id, + status=AutomationStatus.retrying, + attempt=attempt, + ) + try: + await self._execute_action(job) + await self._repo.update_automation_log( + log_id=log_id, + status=AutomationStatus.success, + completed_at=datetime.now(timezone.utc), + ) + return + except Exception as e: + last_error = str(e)[:500] + logger.warning( + "AutomationEngine: attempt %d failed for rule %s: %s", + attempt, + automation_id, + last_error, + ) + await self._repo.update_automation_log( + log_id=log_id, + status=AutomationStatus.error, + error_message=last_error, + completed_at=datetime.now(timezone.utc), + ) + if attempt < _MAX_RETRIES: + await asyncio.sleep(_RETRY_DELAYS[attempt - 1]) + + logger.error( + "AutomationEngine: rule %s failed after %d attempts: %s", + automation_id, + _MAX_RETRIES, + last_error, + ) + + async def _execute_action(self, job: dict[str, Any]) -> None: + """Dispatch to the correct action handler based on action_config type.""" + action_config = job["action_config"] + action_type = str(action_config.get("type", "")) + record_values = job.get("record_values", {}) + + if action_type == AutomationActionType.create_record.value: + await self._action_create_record(action_config, record_values) + elif action_type == AutomationActionType.update_record.value: + await self._action_update_record(action_config, job.get("record_id"), record_values) + elif action_type == AutomationActionType.send_webhook.value: + await self._action_send_webhook(action_config, job) + elif action_type == AutomationActionType.send_email.value: + self._action_send_email(action_config, job) + elif action_type == AutomationActionType.send_message.value: + self._action_send_message(action_config, job) + else: + raise ValueError(f"Unknown action type: {action_type}") + + async def _action_create_record( + self, config: dict[str, Any], trigger_values: dict[str, object] + ) -> None: + """Create a record in a target table (R14 create_record). + + Config: ``{target_table_id, values: {field_id: value | "@trigger.field_id"}}`` + ``@trigger.X`` placeholders are resolved from the triggering record's values. + """ + target_table_id = str(config.get("target_table_id", "")) + if not target_table_id: + raise ValueError("create_record action missing target_table_id") + raw_values: dict[str, object] = config.get("values", {}) # type: ignore[assignment] + resolved: dict[str, str | int | float | None] = {} + for k, v in raw_values.items(): + if isinstance(v, str) and v.startswith("@trigger."): + src_field = v[len("@trigger.") :] + resolved[k] = trigger_values.get(src_field) # type: ignore[assignment] + else: + resolved[k] = v # type: ignore[assignment] + await self._service.create_record(target_table_id, resolved) + + async def _action_update_record( + self, + config: dict[str, Any], + trigger_record_id: str | None, + trigger_values: dict[str, object], + ) -> None: + """Update a record's fields (R14 update_record). + + Config: ``{target_record_id (optional, defaults to trigger), values: {...}}`` + """ + target_record_id = str(config.get("target_record_id") or trigger_record_id or "") + if not target_record_id: + raise ValueError("update_record action: no target_record_id and no trigger record") + raw_values: dict[str, object] = config.get("values", {}) # type: ignore[assignment] + resolved: dict[str, str | int | float | None] = {} + for k, v in raw_values.items(): + if isinstance(v, str) and v.startswith("@trigger."): + src_field = v[len("@trigger.") :] + resolved[k] = trigger_values.get(src_field) # type: ignore[assignment] + else: + resolved[k] = v # type: ignore[assignment] + await self._service.update_record_values(target_record_id, resolved) + + async def _action_send_webhook(self, config: dict[str, Any], job: dict[str, Any]) -> None: + """POST a JSON payload to a webhook URL (R14 send_webhook). + + Config: ``{url, headers (optional), body_template (optional)}`` + + SSRF guard (KTD-9): the URL hostname is validated against + private/loopback/reserved IPs via ``_assert_safe_host`` before sending. + """ + url = str(config.get("url", "")) + if not url: + raise ValueError("send_webhook action missing url") + # SSRF protection (KTD-9): reject private/loopback/reserved targets + from urllib.parse import urlparse + + parsed = urlparse(url) + if parsed.scheme not in ("http", "https"): + raise ValueError(f"Disallowed webhook URL scheme: {parsed.scheme!r}") + if not parsed.hostname: + raise ValueError("Webhook URL has no hostname") + _assert_safe_host(parsed.hostname) + + headers = config.get("headers", {"Content-Type": "application/json"}) + body_template = config.get("body_template") + if body_template: + # ponytail: simple @trigger.X substitution in JSON body. + body = json.dumps(body_template).replace( + "@trigger.record_id", str(job.get("record_id") or "") + ) + else: + body = json.dumps( + { + "event": job.get("trigger_type"), + "table_id": job.get("table_id"), + "record_id": job.get("record_id"), + "values": job.get("record_values", {}), + } + ) + async with httpx.AsyncClient(timeout=_WEBHOOK_TIMEOUT, follow_redirects=False) as client: + resp = await client.post(url, content=body, headers=headers) + resp.raise_for_status() + + def _action_send_email(self, config: dict[str, Any], job: dict[str, Any]) -> None: + """Send email — placeholder (SMTP integration is app-level). + + Logs the email content; production deployments override this with + an actual SMTP/SES/SendGrid call. + """ + logger.info( + "AutomationEngine send_email (placeholder): to=%s subject=%s", + config.get("to"), + config.get("subject"), + ) + + def _action_send_message(self, config: dict[str, Any], job: dict[str, Any]) -> None: + """Send message — placeholder (channel integration is app-level). + + Logs the message; production deployments override this with + Slack/DingTalk/Feishu integration. + """ + logger.info( + "AutomationEngine send_message (placeholder): channel=%s text=%s", + config.get("channel"), + config.get("text"), + ) + + +def generate_webhook_token() -> str: + """Generate a cryptographically-secure webhook token (R16). + + Uses ``secrets.token_urlsafe(32)`` for 256 bits of entropy. + """ + return "whk_" + secrets.token_urlsafe(32) diff --git a/src/agentkit/bitable/db.py b/src/agentkit/bitable/db.py index bad017e..f34d4eb 100644 --- a/src/agentkit/bitable/db.py +++ b/src/agentkit/bitable/db.py @@ -21,9 +21,11 @@ from datetime import datetime, timezone from types import TracebackType from sqlalchemy import ( + Boolean, Column, DateTime, Index, + Integer, String, Text, UniqueConstraint, @@ -37,7 +39,9 @@ logger = logging.getLogger(__name__) # Current schema version — bump when adding migrations. # V1: initial schema (tables/fields/records/views/recalc/meta). # V2: add bitable_files table + file_id column on bitable_tables (R1). -_SCHEMA_VERSION = 2 +# V3: add relation_links / automations / automation_logs / cross_table_deps +# tables + is_cross_table column on bitable_recalc_queue (R5/R9/R10/R16). +_SCHEMA_VERSION = 3 _META_SCHEMA_VERSION_KEY = "schema_version" @@ -155,6 +159,7 @@ class RecalcQueueModel(BitableBase): The ``(record_id, field_id)`` unique index prevents duplicate enqueues. The ``(status, queued_at)`` index supports efficient worker consumption. + ``is_cross_table`` marks tasks triggered by a cross-table dependency (V3). """ __tablename__ = "bitable_recalc_queue" @@ -172,6 +177,7 @@ class RecalcQueueModel(BitableBase): error_message = Column(Text, nullable=True) queued_at = Column(DateTime(timezone=True), default=_utcnow) completed_at = Column(DateTime(timezone=True), nullable=True) + is_cross_table = Column(Boolean, default=False) class MetaModel(BitableBase): @@ -185,6 +191,106 @@ class MetaModel(BitableBase): updated_at = Column(DateTime(timezone=True), default=_utcnow, onupdate=_utcnow) +class RelationLinkModel(BitableBase): + """ORM model for ``bitable.bitable_relation_links`` — junction table (KTD-1, V3). + + Stores many_to_many relation pairs as + ``(relation_field_id, source_record_id, target_record_id)``. Two indexes + cover forward (source→target) and reverse (target→source) lookups for + bidirectional relations. + """ + + __tablename__ = "bitable_relation_links" + __table_args__ = ( + Index( + "ix_relation_links_field_source", + "relation_field_id", + "source_record_id", + ), + Index( + "ix_relation_links_field_target", + "relation_field_id", + "target_record_id", + ), + {"schema": "bitable"}, + ) + + id = Column(String, primary_key=True, default=_uuid_str) + relation_field_id = Column(String, nullable=False) + source_record_id = Column(String, nullable=False) + target_record_id = Column(String, nullable=False) + created_at = Column(DateTime(timezone=True), default=_utcnow) + + +class AutomationModel(BitableBase): + """ORM model for ``bitable.bitable_automations`` — automation rules (V3, R16).""" + + __tablename__ = "bitable_automations" + __table_args__ = ( + Index("ix_automations_table_id", "table_id"), + Index("ix_automations_webhook_token", "webhook_token"), + {"schema": "bitable"}, + ) + + id = Column(String, primary_key=True, default=_uuid_str) + table_id = Column(String, nullable=False) + name = Column(String, nullable=False) + trigger_config = Column(JSONB, default=dict) + action_config = Column(JSONB, default=dict) + enabled = Column(Boolean, default=True) + webhook_token = Column(String, nullable=True) + created_at = Column(DateTime(timezone=True), default=_utcnow) + updated_at = Column(DateTime(timezone=True), default=_utcnow, onupdate=_utcnow) + + +class AutomationLogModel(BitableBase): + """ORM model for ``bitable.bitable_automation_logs`` — execution logs (V3, R17).""" + + __tablename__ = "bitable_automation_logs" + __table_args__ = ( + Index("ix_automation_logs_automation_id", "automation_id", "started_at"), + {"schema": "bitable"}, + ) + + id = Column(String, primary_key=True, default=_uuid_str) + automation_id = Column(String, nullable=False) + trigger_record_id = Column(String, nullable=True) + status = Column(String, default="success") + error_message = Column(Text, nullable=True) + attempt = Column(Integer, default=1) + started_at = Column(DateTime(timezone=True), default=_utcnow) + completed_at = Column(DateTime(timezone=True), nullable=True) + + +class CrossTableDepModel(BitableBase): + """ORM model for ``bitable.bitable_cross_table_deps`` — materialized dep graph (V3, KTD-3). + + One row per ``(source_field, target_field)`` edge. ``target_field_id`` + index supports the recalc worker's reverse-edge lookup: "which fields in + other tables depend on this changed field?" + """ + + __tablename__ = "bitable_cross_table_deps" + __table_args__ = ( + UniqueConstraint( + "source_field_id", + "target_field_id", + name="uq_cross_table_dep_source_target", + ), + Index("ix_cross_table_deps_target_field", "target_field_id"), + Index("ix_cross_table_deps_target_table", "target_table_id"), + {"schema": "bitable"}, + ) + + id = Column(String, primary_key=True, default=_uuid_str) + source_table_id = Column(String, nullable=False) + source_field_id = Column(String, nullable=False) + target_table_id = Column(String, nullable=False) + target_field_id = Column(String, nullable=False) + dep_type = Column(String, default="formula") + created_at = Column(DateTime(timezone=True), default=_utcnow) + + # --------------------------------------------------------------------------- # Schema is created via BitableBase.metadata.create_all (see BitableDB.init). # The ORM models above are the single source of truth for table/index DDL. @@ -242,6 +348,132 @@ async def _apply_v2_migration(conn: object) -> None: logger.info("applied V2 migration: bitable_files table + file_id column") +async def _apply_v3_migration(conn: object) -> None: + """V3 migration: add relation_links, automations, automation_logs, + cross_table_deps tables + ``is_cross_table`` column on recalc_queue. + + Idempotent — ``create_all`` already made the V3 tables on fresh installs; + the raw SQL here ensures V2→V3 upgrades (where ``create_all`` skips + existing tables) also get the new schema. ``ADD COLUMN IF NOT EXISTS`` + covers the recalc_queue extension. + """ + # 1. bitable_relation_links (junction table for many_to_many, KTD-1) + await conn.execute( + text( + "CREATE TABLE IF NOT EXISTS bitable.bitable_relation_links (" + " id VARCHAR PRIMARY KEY," + " relation_field_id VARCHAR NOT NULL," + " source_record_id VARCHAR NOT NULL," + " target_record_id VARCHAR NOT NULL," + " created_at TIMESTAMPTZ DEFAULT NOW()" + ")" + ) + ) + await conn.execute( + text( + "CREATE INDEX IF NOT EXISTS ix_relation_links_field_source " + "ON bitable.bitable_relation_links (relation_field_id, source_record_id)" + ) + ) + await conn.execute( + text( + "CREATE INDEX IF NOT EXISTS ix_relation_links_field_target " + "ON bitable.bitable_relation_links (relation_field_id, target_record_id)" + ) + ) + + # 2. bitable_automations + await conn.execute( + text( + "CREATE TABLE IF NOT EXISTS bitable.bitable_automations (" + " id VARCHAR PRIMARY KEY," + " table_id VARCHAR NOT NULL," + " name VARCHAR NOT NULL," + " trigger_config JSONB DEFAULT '{}'::jsonb," + " action_config JSONB DEFAULT '{}'::jsonb," + " enabled BOOLEAN DEFAULT TRUE," + " webhook_token VARCHAR," + " created_at TIMESTAMPTZ DEFAULT NOW()," + " updated_at TIMESTAMPTZ DEFAULT NOW()" + ")" + ) + ) + await conn.execute( + text( + "CREATE INDEX IF NOT EXISTS ix_automations_table_id " + "ON bitable.bitable_automations (table_id)" + ) + ) + await conn.execute( + text( + "CREATE INDEX IF NOT EXISTS ix_automations_webhook_token " + "ON bitable.bitable_automations (webhook_token) " + "WHERE webhook_token IS NOT NULL" + ) + ) + + # 3. bitable_automation_logs + await conn.execute( + text( + "CREATE TABLE IF NOT EXISTS bitable.bitable_automation_logs (" + " id VARCHAR PRIMARY KEY," + " automation_id VARCHAR NOT NULL," + " trigger_record_id VARCHAR," + " status VARCHAR DEFAULT 'success'," + " error_message TEXT," + " attempt INTEGER DEFAULT 1," + " started_at TIMESTAMPTZ DEFAULT NOW()," + " completed_at TIMESTAMPTZ" + ")" + ) + ) + await conn.execute( + text( + "CREATE INDEX IF NOT EXISTS ix_automation_logs_automation_id " + "ON bitable.bitable_automation_logs (automation_id, started_at)" + ) + ) + + # 4. bitable_cross_table_deps + await conn.execute( + text( + "CREATE TABLE IF NOT EXISTS bitable.bitable_cross_table_deps (" + " id VARCHAR PRIMARY KEY," + " source_table_id VARCHAR NOT NULL," + " source_field_id VARCHAR NOT NULL," + " target_table_id VARCHAR NOT NULL," + " target_field_id VARCHAR NOT NULL," + " dep_type VARCHAR DEFAULT 'formula'," + " created_at TIMESTAMPTZ DEFAULT NOW()," + " CONSTRAINT uq_cross_table_dep_source_target " + " UNIQUE (source_field_id, target_field_id)" + ")" + ) + ) + await conn.execute( + text( + "CREATE INDEX IF NOT EXISTS ix_cross_table_deps_target_field " + "ON bitable.bitable_cross_table_deps (target_field_id)" + ) + ) + await conn.execute( + text( + "CREATE INDEX IF NOT EXISTS ix_cross_table_deps_target_table " + "ON bitable.bitable_cross_table_deps (target_table_id)" + ) + ) + + # 5. is_cross_table column on recalc_queue + await conn.execute( + text( + "ALTER TABLE bitable.bitable_recalc_queue " + "ADD COLUMN IF NOT EXISTS is_cross_table BOOLEAN DEFAULT FALSE" + ) + ) + + logger.info("applied V3 migration: relation_links / automations / logs / cross_table_deps + is_cross_table") + + def _resolve_database_url(database_url: str | None = None) -> str | None: """Resolve PostgreSQL connection URL. @@ -343,6 +575,9 @@ class BitableDB: # V2: file layer (bitable_files table + file_id column on tables) if current_version < 2: await _apply_v2_migration(conn) + # V3: relation_links + automations + logs + cross_table_deps + is_cross_table + if current_version < 3: + await _apply_v3_migration(conn) await conn.execute( text( "INSERT INTO bitable.bitable_meta (key, value, updated_at) " diff --git a/src/agentkit/bitable/formula/__init__.py b/src/agentkit/bitable/formula/__init__.py index d108666..f5bfc0b 100644 --- a/src/agentkit/bitable/formula/__init__.py +++ b/src/agentkit/bitable/formula/__init__.py @@ -16,18 +16,22 @@ from agentkit.bitable.formula.engine import ( ) from agentkit.bitable.formula.functions import FUNCTION_REGISTRY from agentkit.bitable.formula.parser import ( + FormulaDepthExceededError, FormulaParseError, FormulaSecurityError, + MAX_AST_DEPTH, UnknownFunctionError, parse_formula, ) __all__ = [ "CircularReferenceError", + "FormulaDepthExceededError", "FormulaEngine", "FormulaParseError", "FormulaSecurityError", "FUNCTION_REGISTRY", + "MAX_AST_DEPTH", "UnknownFunctionError", "parse_formula", ] diff --git a/src/agentkit/bitable/formula/engine.py b/src/agentkit/bitable/formula/engine.py index 9e06e07..7123182 100644 --- a/src/agentkit/bitable/formula/engine.py +++ b/src/agentkit/bitable/formula/engine.py @@ -19,7 +19,10 @@ from collections import deque from agentkit.bitable.formula.functions import AGGREGATE_FUNCTIONS, FUNCTION_REGISTRY from agentkit.bitable.formula.parser import ( + FormulaDepthExceededError, FormulaParseError, + FormulaSecurityError, + UnknownFunctionError, evaluate_ast, parse_formula, ) @@ -56,7 +59,9 @@ class FormulaEngine: UnknownFunctionError: Unregistered function. CircularReferenceError: Adding this formula creates a cycle. """ - tree, field_mapping = parse_formula(formula, set(FUNCTION_REGISTRY.keys())) + tree, field_mapping, cross_table_mapping = parse_formula( + formula, set(FUNCTION_REGISTRY.keys()) + ) # Classify field refs into aggregate vs row context aggregate_refs, row_refs = _classify_refs(tree, field_mapping) @@ -66,11 +71,14 @@ class FormulaEngine: field_mapping=field_mapping, aggregate_refs=aggregate_refs, row_refs=row_refs, + cross_table_mapping=cross_table_mapping, formula=formula, ) self._formulas[field_id] = entry # Update DAG: this field depends on all referenced fields - self._dag[field_id] = aggregate_refs | row_refs + # Cross-table refs depend on the relation field (local) + target field (foreign) + cross_table_local_deps = {rel_id for rel_id, _ in cross_table_mapping.values()} + self._dag[field_id] = aggregate_refs | row_refs | cross_table_local_deps # Check for cycles cycle = _detect_cycle(self._dag) @@ -92,6 +100,17 @@ class FormulaEngine: """Get the set of field IDs that ``field_id`` depends on.""" return self._dag.get(field_id, set()).copy() + def get_cross_table_mapping(self, field_id: str) -> dict[str, tuple[str, str]] | None: + """Get the cross-table reference mapping for a formula field. + + Returns ``{safe_name: (relation_field_id, target_field_id)}`` or + ``None`` if the field is not registered or has no cross-table refs. + """ + entry = self._formulas.get(field_id) + if entry is None: + return None + return entry.cross_table_mapping or None + def get_dependents(self, field_id: str) -> set[str]: """Get the set of formula field IDs that depend on ``field_id``.""" return {fid for fid, deps in self._dag.items() if field_id in deps} @@ -105,6 +124,7 @@ class FormulaEngine: field_id: str, row_values: dict[str, object], column_values: dict[str, list[object]] | None = None, + cross_table_values: dict[str, list[object]] | None = None, ) -> object: """Evaluate a formula field for a specific record. @@ -113,6 +133,9 @@ class FormulaEngine: row_values: Field ID → value for the current record (row context). column_values: Field ID → list of all values in that column (aggregate context). Required for aggregate references. + cross_table_values: Safe-name → list of values from related + records (cross-table context, U3). The service layer resolves + these before calling evaluate. Returns: The computed value. @@ -126,6 +149,7 @@ class FormulaEngine: entry = self._formulas[field_id] column_values = column_values or {} + cross_table_values = cross_table_values or {} # Build the field_values dict for the evaluator # Aggregate refs get column values (lists), row refs get row values (scalars) @@ -138,6 +162,13 @@ class FormulaEngine: else: eval_values[safe_name] = row_values.get(real_id) + # Cross-table refs: the service layer pre-resolves these to value lists + for safe_name in entry.cross_table_mapping: + if safe_name in cross_table_values: + eval_values[safe_name] = cross_table_values[safe_name] + else: + eval_values[safe_name] = [] + return evaluate_ast(entry.tree, eval_values, FUNCTION_REGISTRY) def evaluate_all_for_record( @@ -150,6 +181,9 @@ class FormulaEngine: Returns a dict of field_id → computed value. Formulas are evaluated in topological order so that formula-to-formula dependencies are resolved correctly. + + List return values (from FILTER/SPLIT/LOOKUP) are stored as-is. + Downstream formulas referencing a list-valued field receive the list. """ results: dict[str, object] = {} column_values = column_values or {} @@ -159,7 +193,15 @@ class FormulaEngine: merged_row = {**row_values, **results} try: results[field_id] = self.evaluate(field_id, merged_row, column_values) - except (FormulaParseError, ZeroDivisionError, TypeError) as e: + except ( + FormulaParseError, + FormulaSecurityError, + UnknownFunctionError, + FormulaDepthExceededError, + ZeroDivisionError, + TypeError, + ValueError, + ) as e: results[field_id] = {"__error": str(e)} return results @@ -171,7 +213,14 @@ class FormulaEngine: class _FormulaEntry: """Parsed formula metadata.""" - __slots__ = ("tree", "field_mapping", "aggregate_refs", "row_refs", "formula") + __slots__ = ( + "tree", + "field_mapping", + "aggregate_refs", + "row_refs", + "cross_table_mapping", + "formula", + ) def __init__( self, @@ -179,12 +228,14 @@ class _FormulaEntry: field_mapping: dict[str, str], aggregate_refs: set[str], row_refs: set[str], + cross_table_mapping: dict[str, tuple[str, str]], formula: str, ) -> None: self.tree = tree self.field_mapping = field_mapping self.aggregate_refs = aggregate_refs self.row_refs = row_refs + self.cross_table_mapping = cross_table_mapping self.formula = formula diff --git a/src/agentkit/bitable/formula/functions.py b/src/agentkit/bitable/formula/functions.py deleted file mode 100644 index 05009e7..0000000 --- a/src/agentkit/bitable/formula/functions.py +++ /dev/null @@ -1,110 +0,0 @@ -"""Built-in formula functions for the bitable formula engine. - -v1 implements: SUM, AVG, COUNT, MIN, MAX, CONCAT, ABS, ROUND, IF, LEN. - -Aggregate functions (SUM/AVG/COUNT/MIN/MAX) accept a list of values -(the entire column) and return a scalar. Non-aggregate functions -(ABS/ROUND/IF/LEN/CONCAT) operate on scalars. - -The engine determines whether to pass a column (list) or scalar (row value) -based on the calling context — see :mod:`agentkit.bitable.formula.engine`. -""" - -from __future__ import annotations - -from typing import Callable, TypeAlias - -# A formula evaluates to a scalar primitive: text, number, or nothing. -# bool is intentionally excluded — comparisons live in the parser layer -# and never reach the function registry. -FormulaResult: TypeAlias = str | int | float | None - -# ── Aggregate functions (operate on lists) ──────────────── - - -def _sum(values: list[FormulaResult]) -> float | int: - """Sum of numeric values, ignoring None/empty.""" - total = 0 - for v in values: - if v is None or v == "": - continue - total += v - return total - - -def _avg(values: list[FormulaResult]) -> float: - """Average of numeric values, ignoring None/empty.""" - nums = [v for v in values if v is not None and v != ""] - if not nums: - return 0.0 - return sum(nums) / len(nums) - - -def _count(values: list[FormulaResult]) -> int: - """Count of non-empty values.""" - return sum(1 for v in values if v is not None and v != "") - - -def _min(values: list[FormulaResult]) -> FormulaResult: - """Minimum of numeric values, ignoring None/empty.""" - nums = [v for v in values if v is not None and v != ""] - if not nums: - return 0 - return min(nums) - - -def _max(values: list[FormulaResult]) -> FormulaResult: - """Maximum of numeric values, ignoring None/empty.""" - nums = [v for v in values if v is not None and v != ""] - if not nums: - return 0 - return max(nums) - - -# ── Scalar functions ────────────────────────────────────── - - -def _abs(value: FormulaResult) -> FormulaResult: - return abs(value) - - -def _round(value: FormulaResult, digits: int = 0) -> float: - return round(value, digits) - - -def _if( - condition: FormulaResult, - true_val: FormulaResult, - false_val: FormulaResult = None, -) -> FormulaResult: - return true_val if condition else false_val - - -def _len(value: FormulaResult) -> int: - if value is None: - return 0 - return len(str(value)) - - -def _concat(*args: FormulaResult) -> str: - """Concatenate all arguments as strings.""" - return "".join(str(a) for a in args if a is not None) - - -# ── Registry ────────────────────────────────────────────── - -# Functions that aggregate a column (receive a list of all column values) -AGGREGATE_FUNCTIONS: frozenset[str] = frozenset({"SUM", "AVG", "COUNT", "MIN", "MAX"}) - -FUNCTION_REGISTRY: dict[str, Callable[..., FormulaResult]] = { - "SUM": _sum, - "AVG": _avg, - "COUNT": _count, - "MIN": _min, - "MAX": _max, - "ABS": _abs, - "ROUND": _round, - "IF": _if, - "LEN": _len, - "CONCAT": _concat, -} diff --git a/src/agentkit/bitable/formula/functions/__init__.py b/src/agentkit/bitable/formula/functions/__init__.py new file mode 100644 index 0000000..6960180 --- /dev/null +++ b/src/agentkit/bitable/formula/functions/__init__.py @@ -0,0 +1,64 @@ +"""Built-in formula function registry for the bitable formula engine. + +This is a package (not a single module) — functions are split by category +per KTD-6: +- :mod:`agentkit.bitable.formula.functions.aggregate` — SUM, AVG, COUNT, MIN, MAX, ABS, ROUND, SUMIF, COUNTIF, AVERAGEIF, SUMIFS, COUNTIFS +- :mod:`agentkit.bitable.formula.functions.datetime` — DATE, TODAY, NOW, YEAR, MONTH, DAY, HOUR, MINUTE, SECOND, WEEKDAY, DATEDIF, DATEADD, NETWORKDAYS, EOMONTH, TIMESTAMP +- :mod:`agentkit.bitable.formula.functions.logical` — IF, IFS, SWITCH, IFERROR, COALESCE, AND, OR, NOT, ISBLANK, ISNUMBER, ISTEXT, ISDATE, TRUE, FALSE, NULL +- :mod:`agentkit.bitable.formula.functions.text` — CONCAT, LEN, FIND, MID, LEFT, RIGHT, REPLACE, SUBSTITUTE, UPPER, LOWER, TRIM, REPT, TEXT, VALUE, STARTSWITH, ENDSWITH, SPLIT +- :mod:`agentkit.bitable.formula.functions.lookup` — LOOKUP, FILTER, MATCH, VLOOKUP, XLOOKUP + +Aggregate functions (SUM/AVG/COUNT/MIN/MAX and conditional variants) +receive a list of all column values from the engine. Non-aggregate +functions operate on scalars (row context). The engine's +``_classify_refs()`` decides context based on AGGREGATE_FUNCTIONS. +""" + +from __future__ import annotations + +from typing import Callable, TypeAlias + +# A formula evaluates to a scalar primitive: text, number, bool, or nothing. +# Lookup/FILTER may also return lists; the engine tolerates this at runtime. +# bool is included for logical functions (TRUE/FALSE/AND/OR/NOT/predicates). +FormulaResult: TypeAlias = str | int | float | bool | None + +from agentkit.bitable.formula.functions.aggregate import AGGREGATE_FUNCTIONS_DICT # noqa: E402 +from agentkit.bitable.formula.functions.datetime import DATETIME_FUNCTIONS # noqa: E402 +from agentkit.bitable.formula.functions.logical import LOGICAL_FUNCTIONS # noqa: E402 +from agentkit.bitable.formula.functions.lookup import LOOKUP_FUNCTIONS # noqa: E402 +from agentkit.bitable.formula.functions.text import TEXT_FUNCTIONS # noqa: E402 + +# ── Aggregate function names (receive column lists) ────── +#: Functions that aggregate a column (receive a list of all column values). +#: Used by engine._classify_refs() to distinguish aggregate vs row context. +AGGREGATE_FUNCTIONS: frozenset[str] = frozenset( + { + "SUM", + "AVG", + "COUNT", + "MIN", + "MAX", + "SUMIF", + "COUNTIF", + "AVERAGEIF", + "SUMIFS", + "COUNTIFS", + } +) + +# ── Unified registry ──────────────────────────────────── +#: All registered functions. Merged from category submodules. +FUNCTION_REGISTRY: dict[str, Callable[..., FormulaResult]] = { + **AGGREGATE_FUNCTIONS_DICT, + **DATETIME_FUNCTIONS, + **LOGICAL_FUNCTIONS, + **TEXT_FUNCTIONS, + **LOOKUP_FUNCTIONS, +} + +__all__ = [ + "AGGREGATE_FUNCTIONS", + "FUNCTION_REGISTRY", + "FormulaResult", +] diff --git a/src/agentkit/bitable/formula/functions/aggregate.py b/src/agentkit/bitable/formula/functions/aggregate.py new file mode 100644 index 0000000..d247b4e --- /dev/null +++ b/src/agentkit/bitable/formula/functions/aggregate.py @@ -0,0 +1,251 @@ +"""Aggregate and math formula functions (R1). + +Aggregate functions (SUM/AVG/COUNT/MIN/MAX and conditional variants) +operate on lists — the entire column. Scalar math functions (ABS, ROUND) +operate on single values. + +The engine's ``_classify_refs()`` decides whether a field reference is +aggregate (column) or row (scalar) based on whether it appears as a +direct argument to one of these functions. +""" + +from __future__ import annotations + +from typing import Callable + +from agentkit.bitable.formula.functions import FormulaResult + + +def _is_numeric(v: FormulaResult) -> bool: + return isinstance(v, (int, float)) and not isinstance(v, bool) + + +def _to_number(v: FormulaResult) -> float | None: + """Coerce a value to number. Returns None if not coercible.""" + if v is None or v == "": + return None + if isinstance(v, bool): + return 1.0 if v else 0.0 + if isinstance(v, (int, float)): + return float(v) + try: + return float(v) + except (ValueError, TypeError): + return None + + +# ── migrated aggregate functions ──────────────────────── + + +def _sum(values: list[FormulaResult]) -> float | int: + """Sum of numeric values, ignoring None/empty/non-numeric.""" + total: float | int = 0 + for v in values: + n = _to_number(v) + if n is not None: + total += n + return total + + +def _avg(values: list[FormulaResult]) -> float: + """Average of numeric values, ignoring None/empty/non-numeric.""" + nums = [n for n in (_to_number(v) for v in values) if n is not None] + if not nums: + return 0.0 + return sum(nums) / len(nums) + + +def _count(values: list[FormulaResult]) -> int: + """Count of non-empty values.""" + return sum(1 for v in values if v is not None and v != "") + + +def _min(values: list[FormulaResult]) -> FormulaResult: + """Minimum of numeric values, ignoring None/empty.""" + nums = [n for n in (_to_number(v) for v in values) if n is not None] + if not nums: + return 0 + return min(nums) + + +def _max(values: list[FormulaResult]) -> FormulaResult: + """Maximum of numeric values, ignoring None/empty.""" + nums = [n for n in (_to_number(v) for v in values) if n is not None] + if not nums: + return 0 + return max(nums) + + +# ── migrated scalar math ──────────────────────────────── + + +def _abs(value: FormulaResult) -> FormulaResult: + n = _to_number(value) + if n is None: + return None + return abs(n) + + +def _round(value: FormulaResult, digits: int = 0) -> float | None: + n = _to_number(value) + if n is None: + return None + return round(n, digits) + + +# ── conditional aggregates ────────────────────────────── + + +def _sumif(values: list[FormulaResult], criteria: FormulaResult) -> float | int: + """SUMIF(values, criteria) → sum of values matching criteria. + + criteria may be a number (equality) or string (e.g. ">10"). + ponytail: criteria parser handles >, <, >=, <=, <>, = prefixes. + """ + total: float | int = 0 + op, target = _parse_criteria(criteria) + for v in values: + n = _to_number(v) + if n is None: + continue + if _matches_criteria(n, op, target): + total += n + return total + + +def _countif(values: list[FormulaResult], criteria: FormulaResult) -> int: + """COUNTIF(values, criteria) → count of values matching criteria.""" + count = 0 + op, target = _parse_criteria(criteria) + for v in values: + n = _to_number(v) + if n is None: + continue + if _matches_criteria(n, op, target): + count += 1 + return count + + +def _averageif(values: list[FormulaResult], criteria: FormulaResult) -> float: + """AVERAGEIF(values, criteria) → average of values matching criteria.""" + nums: list[float] = [] + op, target = _parse_criteria(criteria) + for v in values: + n = _to_number(v) + if n is None: + continue + if _matches_criteria(n, op, target): + nums.append(n) + if not nums: + return 0.0 + return sum(nums) / len(nums) + + +def _sumifs( + values: list[FormulaResult], criteria_range: list[FormulaResult], criteria: FormulaResult +) -> float | int: + """SUMIFS(values, criteria_range, criteria) → sum of values where criteria_range matches. + + ponytail: single criteria range only. True SUMIFS in spreadsheets takes + multiple (range, criteria) pairs. Ceiling: variadic version would require + AST list-pair handling. + """ + if len(values) != len(criteria_range): + raise ValueError("SUMIFS requires values and criteria_range of equal length") + total: float | int = 0 + op, target = _parse_criteria(criteria) + for v, c in zip(values, criteria_range): + n = _to_number(v) + if n is None: + continue + cn = _to_number(c) + if cn is not None and _matches_criteria(cn, op, target): + total += n + return total + + +def _countifs( + values: list[FormulaResult], criteria_range: list[FormulaResult], criteria: FormulaResult +) -> int: + """COUNTIFS(values, criteria_range, criteria) → count where criteria_range matches.""" + if len(values) != len(criteria_range): + raise ValueError("COUNTIFS requires values and criteria_range of equal length") + count = 0 + op, target = _parse_criteria(criteria) + for v, c in zip(values, criteria_range): + n = _to_number(v) + if n is None: + continue + cn = _to_number(c) + if cn is not None and _matches_criteria(cn, op, target): + count += 1 + return count + + +# ── criteria helpers ──────────────────────────────────── + + +def _parse_criteria(criteria: FormulaResult) -> tuple[str, float | None]: + """Parse a criteria value into (operator, target_number). + + Supports: ">10", "<10", ">=10", "<=10", "<>10", "=10", or plain 10. + Returns ("=", None) for non-string criteria (equality against the value itself). + """ + if isinstance(criteria, str): + s = criteria.strip() + for op_prefix in (">=", "<=", "<>", ">", "<", "="): + if s.startswith(op_prefix): + target_str = s[len(op_prefix) :].strip() + try: + return op_prefix, float(target_str) + except ValueError: + return op_prefix, None + # String criteria without operator — try numeric equality + try: + return "=", float(s) + except ValueError: + return "=", None + if isinstance(criteria, (int, float)) and not isinstance(criteria, bool): + return "=", float(criteria) + return "=", None + + +def _matches_criteria(value: float, op: str, target: float | None) -> bool: + """Check if value matches the parsed criteria.""" + if target is None: + return False + if op == ">": + return value > target + if op == "<": + return value < target + if op == ">=": + return value >= target + if op == "<=": + return value <= target + if op == "<>": + return value != target + if op == "=": + return value == target + return False + + +# ── Registry ──────────────────────────────────────────── + +AGGREGATE_FUNCTIONS_LIST: frozenset[str] = frozenset( + {"SUM", "AVG", "COUNT", "MIN", "MAX", "SUMIF", "COUNTIF", "AVERAGEIF", "SUMIFS", "COUNTIFS"} +) + +AGGREGATE_FUNCTIONS_DICT: dict[str, Callable[..., FormulaResult]] = { + "SUM": _sum, + "AVG": _avg, + "COUNT": _count, + "MIN": _min, + "MAX": _max, + "ABS": _abs, + "ROUND": _round, + "SUMIF": _sumif, + "COUNTIF": _countif, + "AVERAGEIF": _averageif, + "SUMIFS": _sumifs, + "COUNTIFS": _countifs, +} diff --git a/src/agentkit/bitable/formula/functions/datetime.py b/src/agentkit/bitable/formula/functions/datetime.py new file mode 100644 index 0000000..a3269c8 --- /dev/null +++ b/src/agentkit/bitable/formula/functions/datetime.py @@ -0,0 +1,214 @@ +"""Date/time formula functions (R1). + +All functions accept ISO 8601 strings (the storage format for ``date`` fields) +or Python datetime objects. NETWORKDAYS uses Python's ``datetime.weekday()`` +to exclude Saturdays/Sundays; holiday calendars are out of scope for v2 +(ponytail: ceiling — upgrade path is a holidays JSON config). +""" + +from __future__ import annotations + +from datetime import date, datetime, timedelta, timezone +from typing import Callable + +from agentkit.bitable.formula.functions import FormulaResult + + +def _parse_date(value: FormulaResult) -> datetime | None: + """Parse an ISO 8601 string or pass through a datetime. + + Returns None if value is None/empty/invalid. Raises ValueError on + non-date strings so callers can surface a formula error. + """ + if value is None or value == "": + return None + if isinstance(value, datetime): + return value + if isinstance(value, date): + return datetime(value.year, value.month, value.day) + if isinstance(value, str): + # ISO 8601 — datetime.fromisoformat handles "2026-07-06" and + # "2026-07-06T10:30:00" and "2026-07-06T10:30:00+00:00". + return datetime.fromisoformat(value) + # Numbers aren't dates here — surface as a formula error. + raise ValueError(f"Cannot parse date from: {value!r}") + + +def _to_serializable(dt: datetime) -> str: + """Serialize a datetime to ISO 8601 string (storage format).""" + return dt.isoformat() + + +# ── Constructors ───────────────────────────────────────── + + +def _date(year: int, month: int, day: int) -> str: + """DATE(year, month, day) → ISO 8601 date string.""" + return _to_serializable(datetime(year, month, day)) + + +def _today() -> str: + """TODAY() → current date (UTC, midnight).""" + now = datetime.now(timezone.utc) + return _to_serializable(now.replace(hour=0, minute=0, second=0, microsecond=0)) + + +def _now() -> str: + """NOW() → current datetime (UTC).""" + return _to_serializable(datetime.now(timezone.utc)) + + +def _timestamp() -> str: + """TIMESTAMP() → current datetime (UTC). Alias for NOW.""" + return _now() + + +# ── Accessors ──────────────────────────────────────────── + + +def _year(value: FormulaResult) -> int | None: + dt = _parse_date(value) + return dt.year if dt else None + + +def _month(value: FormulaResult) -> int | None: + dt = _parse_date(value) + return dt.month if dt else None + + +def _day(value: FormulaResult) -> int | None: + dt = _parse_date(value) + return dt.day if dt else None + + +def _hour(value: FormulaResult) -> int | None: + dt = _parse_date(value) + return dt.hour if dt else None + + +def _minute(value: FormulaResult) -> int | None: + dt = _parse_date(value) + return dt.minute if dt else None + + +def _second(value: FormulaResult) -> int | None: + dt = _parse_date(value) + return dt.second if dt else None + + +def _weekday(value: FormulaResult) -> int | None: + """WEEKDAY(date) → day of week (1=Monday … 7=Sunday, ISO 8601).""" + dt = _parse_date(value) + if dt is None: + return None + return dt.isoweekday() + + +# ── Arithmetic ─────────────────────────────────────────── + + +def _datedif(start: FormulaResult, end: FormulaResult, unit: str = "days") -> int | float | None: + """DATEDIF(start, end, unit) → difference in days/months/years. + + unit: "days" (default), "months", "years". + """ + s = _parse_date(start) + e = _parse_date(end) + if s is None or e is None: + return None + if unit == "days": + return (e - s).days + if unit == "months": + return (e.year - s.year) * 12 + (e.month - s.month) + if unit == "years": + return e.year - s.year + raise ValueError(f"DATEDIF unit must be days/months/years, got: {unit!r}") + + +def _dateadd(value: FormulaResult, amount: int, unit: str = "days") -> str | None: + """DATEADD(date, amount, unit) → new date string. + + unit: "days" (default), "months", "years". + """ + dt = _parse_date(value) + if dt is None: + return None + if unit == "days": + return _to_serializable(dt + timedelta(days=amount)) + if unit == "months": + # ponytail: naive month arithmetic — no day-of-month overflow handling. + # Ceiling: Jan 31 + 1 month → Feb 28/29 not preserved. Upgrade path: + # use dateutil.relativedelta (would add a dependency). + new_month = dt.month + amount + new_year = dt.year + (new_month - 1) // 12 + new_month = ((new_month - 1) % 12) + 1 + # Clamp day to last day of new month + import calendar + + last_day = calendar.monthrange(new_year, new_month)[1] + new_day = min(dt.day, last_day) + return _to_serializable(dt.replace(year=new_year, month=new_month, day=new_day)) + if unit == "years": + try: + return _to_serializable(dt.replace(year=dt.year + amount)) + except ValueError: + # Feb 29 → non-leap year: clamp to Feb 28 + return _to_serializable(dt.replace(year=dt.year + amount, day=28)) + raise ValueError(f"DATEADD unit must be days/months/years, got: {unit!r}") + + +def _networkdays(start: FormulaResult, end: FormulaResult) -> int | None: + """NETWORKDAYS(start, end) → number of working days (Mon-Fri) between dates. + + Both endpoints inclusive. Holiday calendars are out of scope (v3+). + """ + s = _parse_date(start) + e = _parse_date(end) + if s is None or e is None: + return None + if s > e: + s, e = e, s + days = 0 + cur = s.date() + end_date = e.date() + while cur <= end_date: + if cur.weekday() < 5: # Mon-Fri + days += 1 + cur += timedelta(days=1) + return days + + +def _eomonth(value: FormulaResult, months: int = 0) -> str | None: + """EOMONTH(date, months) → last day of the month N months after date.""" + dt = _parse_date(value) + if dt is None: + return None + import calendar + + new_month = dt.month + months + new_year = dt.year + (new_month - 1) // 12 + new_month = ((new_month - 1) % 12) + 1 + last_day = calendar.monthrange(new_year, new_month)[1] + return _to_serializable(dt.replace(year=new_year, month=new_month, day=last_day)) + + +# ── Registry ───────────────────────────────────────────── + +#: Functions in this module that should be registered globally. +DATETIME_FUNCTIONS: dict[str, Callable[..., FormulaResult]] = { + "DATE": _date, + "DATEDIF": _datedif, + "NETWORKDAYS": _networkdays, + "WEEKDAY": _weekday, + "EOMONTH": _eomonth, + "TODAY": _today, + "NOW": _now, + "YEAR": _year, + "MONTH": _month, + "DAY": _day, + "HOUR": _hour, + "MINUTE": _minute, + "SECOND": _second, + "DATEADD": _dateadd, + "TIMESTAMP": _timestamp, +} diff --git a/src/agentkit/bitable/formula/functions/logical.py b/src/agentkit/bitable/formula/functions/logical.py new file mode 100644 index 0000000..d901652 --- /dev/null +++ b/src/agentkit/bitable/formula/functions/logical.py @@ -0,0 +1,164 @@ +"""Logical formula functions (R1). + +Boolean, conditional, and type-checking functions. All operate on +scalars (row context). Boolean returns are Python ``bool`` which the +engine treats as truthy/falsy in conditional context. +""" + +from __future__ import annotations + +from datetime import date, datetime +from typing import Callable + +from agentkit.bitable.formula.functions import FormulaResult + + +def _is_truthy(value: FormulaResult) -> bool: + """Truthiness for formula context: None/0/'' are falsy, everything else truthy.""" + if value is None: + return False + if isinstance(value, str): + return value != "" and value.lower() != "false" + if isinstance(value, (int, float)): + return value != 0 + return bool(value) + + +# ── migrated from functions.py ────────────────────────── + + +def _if( + condition: FormulaResult, + true_val: FormulaResult, + false_val: FormulaResult = None, +) -> FormulaResult: + return true_val if _is_truthy(condition) else false_val + + +# ── multi-branch conditionals ─────────────────────────── + + +def _ifs(*pairs: FormulaResult) -> FormulaResult: + """IFS(c1, v1, c2, v2, ...) → value of first truthy condition. + + Pairs must be (condition, value, condition, value, ...). + Returns None if no condition matches. + """ + if len(pairs) % 2 != 0: + raise ValueError("IFS requires pairs of (condition, value)") + for i in range(0, len(pairs), 2): + if _is_truthy(pairs[i]): + return pairs[i + 1] + return None + + +def _switch(value: FormulaResult, *pairs: FormulaResult) -> FormulaResult: + """SWITCH(value, c1, v1, c2, v2, ..., default?) → first matching value. + + Compares value against each case cN; returns vN on first match. + Optional trailing default (odd arg count) returned if no match. + """ + n = len(pairs) + has_default = n % 2 == 1 + limit = n - 1 if has_default else n + for i in range(0, limit, 2): + if value == pairs[i]: + return pairs[i + 1] + return pairs[-1] if has_default else None + + +def _iferror(value: FormulaResult, default: FormulaResult = None) -> FormulaResult: + """IFERROR(value, default=None) → value, or default if value is an error dict. + + The engine wraps exceptions as ``{"__error": "..."}`` dicts; this function + unwraps them. Non-error values pass through unchanged. + """ + if isinstance(value, dict) and "__error" in value: + return default + return value + + +def _coalesce(*values: FormulaResult) -> FormulaResult: + """COALESCE(v1, v2, ...) → first non-None/non-empty value.""" + for v in values: + if v is not None and v != "": + return v + return None + + +# ── boolean operators ─────────────────────────────────── + + +def _and(*args: FormulaResult) -> bool: + return all(_is_truthy(a) for a in args) + + +def _or(*args: FormulaResult) -> bool: + return any(_is_truthy(a) for a in args) + + +def _not(value: FormulaResult) -> bool: + return not _is_truthy(value) + + +# ── predicates ────────────────────────────────────────── + + +def _isblank(value: FormulaResult) -> bool: + return value is None or value == "" + + +def _isnumber(value: FormulaResult) -> bool: + return isinstance(value, (int, float)) and not isinstance(value, bool) + + +def _istext(value: FormulaResult) -> bool: + return isinstance(value, str) + + +def _isdate(value: FormulaResult) -> bool: + if isinstance(value, (datetime, date)): + return True + if isinstance(value, str): + try: + datetime.fromisoformat(value) + return True + except ValueError: + return False + return False + + +# ── constants ─────────────────────────────────────────── + + +def _true() -> bool: + return True + + +def _false() -> bool: + return False + + +def _null() -> None: + return None + + +# ── Registry ──────────────────────────────────────────── + +LOGICAL_FUNCTIONS: dict[str, Callable[..., FormulaResult]] = { + "IF": _if, + "IFS": _ifs, + "SWITCH": _switch, + "IFERROR": _iferror, + "COALESCE": _coalesce, + "AND": _and, + "OR": _or, + "NOT": _not, + "ISBLANK": _isblank, + "ISNUMBER": _isnumber, + "ISTEXT": _istext, + "ISDATE": _isdate, + "TRUE": _true, + "FALSE": _false, + "NULL": _null, +} diff --git a/src/agentkit/bitable/formula/functions/lookup.py b/src/agentkit/bitable/formula/functions/lookup.py new file mode 100644 index 0000000..f3cf2f2 --- /dev/null +++ b/src/agentkit/bitable/formula/functions/lookup.py @@ -0,0 +1,189 @@ +"""Lookup formula functions (R3 partial). + +LOOKUP / FILTER / MATCH / VLOOKUP / XLOOKUP operate on lists of values +and return scalars or lists. Cross-table resolution (resolving relation +fields to foreign-table columns) is deferred to U3 — these functions +expect the engine to have already expanded relation context into flat +value lists. + +ponytail: ceiling — true cross-table LOOKUP requires the U3 relation +context expansion in the engine. The functions here operate on +in-memory lists passed by the engine, keeping them usable for same-table +lookups now and cross-table later without API change. +""" + +from __future__ import annotations + +from typing import Callable + +from agentkit.bitable.formula.functions import FormulaResult + + +def _to_number(v: FormulaResult) -> float | None: + if v is None or v == "": + return None + if isinstance(v, bool): + return 1.0 if v else 0.0 + if isinstance(v, (int, float)): + return float(v) + try: + return float(v) + except (ValueError, TypeError): + return None + + +def _lookup( + lookup_value: FormulaResult, + lookup_vector: list[FormulaResult], + result_vector: list[FormulaResult], +) -> FormulaResult: + """LOOKUP(value, lookup_vector, result_vector) → first result where lookup_vector matches. + + Approximate match: lookup_vector must be sorted ascending. Returns the + last result where lookup_vector value <= lookup_value. + """ + target = _to_number(lookup_value) + if target is None: + return None + if len(lookup_vector) != len(result_vector): + raise ValueError("LOOKUP requires lookup_vector and result_vector of equal length") + last_match: FormulaResult = None + for lv, rv in zip(lookup_vector, result_vector): + n = _to_number(lv) + if n is None: + continue + if n <= target: + last_match = rv + else: + break + return last_match + + +def _filter( + values: list[FormulaResult], + criteria_range: list[FormulaResult], + criteria: FormulaResult, +) -> list[FormulaResult]: + """FILTER(values, criteria_range, criteria) → values where criteria_range matches. + + Returns a list (multi-value result). The engine treats list returns as + multi-value results — useful for downstream LOOKUP/MATCH. + """ + if len(values) != len(criteria_range): + raise ValueError("FILTER requires values and criteria_range of equal length") + target = criteria + result: list[FormulaResult] = [] + for v, c in zip(values, criteria_range): + if c == target: + result.append(v) + return result + + +def _match( + lookup_value: FormulaResult, + lookup_array: list[FormulaResult], + match_type: int = 0, +) -> int | None: + """MATCH(value, lookup_array, match_type=0) → 0-based position. + + match_type: + 0 (default) — exact match + 1 — largest value <= lookup_value (array must be sorted ascending) + -1 — smallest value >= lookup_value (array must be sorted descending) + Returns None if no match. + """ + if match_type == 0: + for i, v in enumerate(lookup_array): + if v == lookup_value: + return i + return None + target = _to_number(lookup_value) + if target is None: + return None + if match_type == 1: + last_match: int | None = None + for i, v in enumerate(lookup_array): + n = _to_number(v) + if n is None: + continue + if n <= target: + last_match = i + else: + break + return last_match + if match_type == -1: + last_match = None + for i, v in enumerate(lookup_array): + n = _to_number(v) + if n is None: + continue + if n >= target: + last_match = i + else: + break + return last_match + return None + + +def _vlookup( + lookup_value: FormulaResult, + table_vector: list[list[FormulaResult]], + col_index: int, + range_lookup: bool = True, +) -> FormulaResult: + """VLOOKUP(value, table, col_index, range_lookup=True) → value from col_index of matching row. + + table_vector is a list of rows (each row a list of values). + col_index is 1-based. + range_lookup: True (default) = approximate match (sorted); False = exact. + """ + if col_index < 1: + raise ValueError("VLOOKUP col_index must be >= 1") + target = _to_number(lookup_value) if range_lookup else lookup_value + last_match: FormulaResult = None + for row in table_vector: + if not row: + continue + first = row[0] + if range_lookup: + n = _to_number(first) + if n is None: + continue + if n <= (target or 0): + if col_index <= len(row): + last_match = row[col_index - 1] + else: + break + else: + if first == lookup_value: + if col_index <= len(row): + return row[col_index - 1] + return None + return last_match + + +def _xlookup( + lookup_value: FormulaResult, + lookup_array: list[FormulaResult], + return_array: list[FormulaResult], + if_not_found: FormulaResult = None, +) -> FormulaResult: + """XLOOKUP(value, lookup_array, return_array, if_not_found=None) → first match's return. + + Exact match by default. Returns if_not_found (default None) if no match. + """ + for lv, rv in zip(lookup_array, return_array): + if lv == lookup_value: + return rv + return if_not_found + + +# ── Registry ──────────────────────────────────────────── + +LOOKUP_FUNCTIONS: dict[str, Callable[..., FormulaResult]] = { + "LOOKUP": _lookup, + "FILTER": _filter, + "MATCH": _match, + "VLOOKUP": _vlookup, + "XLOOKUP": _xlookup, +} diff --git a/src/agentkit/bitable/formula/functions/text.py b/src/agentkit/bitable/formula/functions/text.py new file mode 100644 index 0000000..777e855 --- /dev/null +++ b/src/agentkit/bitable/formula/functions/text.py @@ -0,0 +1,212 @@ +"""Text formula functions (R1). + +All functions operate on scalars (row context). Storage format for text +fields is ``str``; numbers are coerced via ``str()`` where appropriate. +""" + +from __future__ import annotations + +from typing import Callable + +from agentkit.bitable.formula.functions import FormulaResult + + +def _to_str(value: FormulaResult) -> str: + """Coerce a formula value to string. None → empty string.""" + if value is None: + return "" + return str(value) + + +# ── migrated from functions.py ────────────────────────── + + +def _concat(*args: FormulaResult) -> str: + """Concatenate all arguments as strings, skipping None.""" + return "".join(_to_str(a) for a in args if a is not None) + + +def _len(value: FormulaResult) -> int: + """Length of value's string form. None → 0.""" + if value is None: + return 0 + return len(str(value)) + + +# ── find / extract ────────────────────────────────────── + + +def _find(text: FormulaResult, substring: FormulaResult, start: int = 0) -> int: + """FIND(text, substring, start=0) → 0-based index, -1 if not found.""" + t = _to_str(text) + s = _to_str(substring) + if not s: + return 0 + return t.find(s, max(0, start)) + + +def _mid(text: FormulaResult, start: int, length: int) -> str: + """MID(text, start, length) → substring from 0-based start for length chars.""" + t = _to_str(text) + if start < 0: + start = 0 + if length < 0: + length = 0 + return t[start : start + length] + + +def _left(text: FormulaResult, n: int = 1) -> str: + """LEFT(text, n=1) → first n characters.""" + t = _to_str(text) + if n <= 0: + return "" + return t[:n] + + +def _right(text: FormulaResult, n: int = 1) -> str: + """RIGHT(text, n=1) → last n characters.""" + t = _to_str(text) + if n <= 0: + return "" + return t[-n:] if n <= len(t) else t + + +# ── replace / substitute ──────────────────────────────── + + +def _replace(text: FormulaResult, start: int, length: int, new: FormulaResult) -> str: + """REPLACE(text, start, length, new) → text with [start:start+length] replaced by new.""" + t = _to_str(text) + n = _to_str(new) + if start < 0: + start = 0 + if length < 0: + length = 0 + return t[:start] + n + t[start + length :] + + +def _substitute( + text: FormulaResult, old: FormulaResult, new: FormulaResult, occurrence: int = -1 +) -> str: + """SUBSTITUTE(text, old, new, occurrence=-1) → text with old replaced by new. + + occurrence: -1 (default) replaces all; otherwise replaces the Nth occurrence (1-based). + """ + t = _to_str(text) + o = _to_str(old) + n = _to_str(new) + if not o: + return t + if occurrence < 0: + return t.replace(o, n) + # Replace only the Nth occurrence (1-based) + parts = t.split(o) + if len(parts) <= occurrence: + return t # not enough occurrences + return o.join(parts[:occurrence]) + n + o.join(parts[occurrence:]) + + +# ── case / trim ───────────────────────────────────────── + + +def _upper(value: FormulaResult) -> str: + return _to_str(value).upper() + + +def _lower(value: FormulaResult) -> str: + return _to_str(value).lower() + + +def _trim(value: FormulaResult) -> str: + """TRIM(text) → strip leading/trailing whitespace and collapse internal runs.""" + return " ".join(_to_str(value).split()) + + +# ── repeat / format / convert ─────────────────────────── + + +def _rept(text: FormulaResult, n: int) -> str: + """REPT(text, n) → text repeated n times.""" + if n <= 0: + return "" + return _to_str(text) * n + + +def _text(value: FormulaResult, fmt: str = "") -> str: + """TEXT(value, fmt) → value formatted as string. + + fmt is a simple format spec passed to Python's ``format()``. + Empty fmt → ``str(value)``. + """ + if value is None: + return "" + if not fmt: + return str(value) + try: + return format(value, fmt) + except (ValueError, TypeError): + return str(value) + + +def _value(text: FormulaResult) -> float | int | None: + """VALUE(text) → parse text as number. Returns None if unparseable.""" + if text is None: + return None + if isinstance(text, (int, float)): + return text + s = str(text).strip() + if not s: + return None + try: + # Prefer int if no decimal point + if "." not in s and "e" not in s.lower(): + return int(s) + return float(s) + except ValueError: + try: + return float(s) + except ValueError: + return None + + +# ── predicates / split ────────────────────────────────── + + +def _startswith(text: FormulaResult, prefix: FormulaResult) -> bool: + return _to_str(text).startswith(_to_str(prefix)) + + +def _endswith(text: FormulaResult, suffix: FormulaResult) -> bool: + return _to_str(text).endswith(_to_str(suffix)) + + +def _split(text: FormulaResult, delimiter: str = ",") -> list[str]: + """SPLIT(text, delimiter=',') → list of substrings. + + Note: returns a list. The formula engine treats list returns as + multi-value results (used by LOOKUP/FILTER context in U3). + """ + return _to_str(text).split(delimiter) if delimiter else [_to_str(text)] + + +# ── Registry ──────────────────────────────────────────── + +TEXT_FUNCTIONS: dict[str, Callable[..., FormulaResult]] = { + "CONCAT": _concat, + "LEN": _len, + "FIND": _find, + "MID": _mid, + "LEFT": _left, + "RIGHT": _right, + "REPLACE": _replace, + "SUBSTITUTE": _substitute, + "UPPER": _upper, + "LOWER": _lower, + "TRIM": _trim, + "REPT": _rept, + "TEXT": _text, + "VALUE": _value, + "STARTSWITH": _startswith, + "ENDSWITH": _endswith, + "SPLIT": _split, +} diff --git a/src/agentkit/bitable/formula/parser.py b/src/agentkit/bitable/formula/parser.py index e404e3b..c33fea6 100644 --- a/src/agentkit/bitable/formula/parser.py +++ b/src/agentkit/bitable/formula/parser.py @@ -44,30 +44,65 @@ class UnknownFunctionError(Exception): """Raised when a formula calls a function not in the registry.""" +class FormulaDepthExceededError(Exception): + """Raised when a formula AST exceeds the max depth (KTD8). + + Limits deeply nested expressions that could cause exponential + evaluation time. Default limit: 15 levels. + """ + + +# KTD8: AST depth limit — prevents exponential-time evaluation of +# deeply nested expressions (e.g. nested IF chains, arithmetic towers). +MAX_AST_DEPTH: int = 15 + + # ── Field reference substitution ────────────────────────── # Match {field_id} — field IDs are UUIDs or alphanumeric. _FIELD_REF_RE = re.compile(r"\{([a-zA-Z0-9_-]+)\}") +# Match {rel_field_id.target_field_id} — cross-table reference (KTD-2, U3). +# Must be matched BEFORE _FIELD_REF_RE to avoid stripping the rel part. +_CROSS_TABLE_REF_RE = re.compile(r"\{([a-zA-Z0-9_-]+)\.([a-zA-Z0-9_-]+)\}") -def _substitute_field_refs(formula: str) -> tuple[str, dict[str, str]]: - """Replace ``{field_id}`` with ``_f_`` (a Python Name node). - Field IDs are UUIDs that may start with a digit, which is invalid in Python - identifiers. We prefix with ``_f_`` and replace hyphens with underscores. - A reverse mapping is returned so the engine can map back to real field IDs. +def _substitute_field_refs( + formula: str, +) -> tuple[str, dict[str, str], dict[str, tuple[str, str]]]: + """Replace field references with safe Python identifiers. + + Handles two syntaxes: + - ``{field_id}`` → ``_f_`` (single-field, row or aggregate context) + - ``{rel_field_id.target_field_id}`` → ``_ct___`` + (cross-table reference, KTD-2, U3) + + Returns: + Tuple of (substituted_formula, field_mapping, cross_table_mapping) + where field_mapping maps safe_name → field_id and + cross_table_mapping maps safe_name → (rel_field_id, target_field_id). """ - mapping: dict[str, str] = {} + field_mapping: dict[str, str] = {} + cross_table_mapping: dict[str, tuple[str, str]] = {} - def _replace(match: re.Match[str]) -> str: - field_id = match.group(1) - # Convert UUID-style field_id to a valid Python identifier - safe_name = "_f_" + field_id.replace("-", "_") - mapping[safe_name] = field_id + def _replace_cross(match: re.Match[str]) -> str: + rel_field_id = match.group(1) + target_field_id = match.group(2) + safe_name = "_ct_" + rel_field_id.replace("-", "_") + "__" + target_field_id.replace("-", "_") + cross_table_mapping[safe_name] = (rel_field_id, target_field_id) return safe_name - result = _FIELD_REF_RE.sub(_replace, formula) - return result, mapping + def _replace_single(match: re.Match[str]) -> str: + field_id = match.group(1) + safe_name = "_f_" + field_id.replace("-", "_") + field_mapping[safe_name] = field_id + return safe_name + + # Cross-table refs first (they contain a dot, so won't match _FIELD_REF_RE) + result = _CROSS_TABLE_REF_RE.sub(_replace_cross, formula) + # Then single-field refs + result = _FIELD_REF_RE.sub(_replace_single, result) + return result, field_mapping, cross_table_mapping # ── AST whitelist (KTD7) ────────────────────────────────── @@ -84,6 +119,8 @@ _ALLOWED_NODES: frozenset[type[ast.AST]] = frozenset( ast.Constant, ast.IfExp, ast.Load, + ast.List, + ast.Tuple, ast.Add, ast.Sub, ast.Mult, @@ -105,6 +142,22 @@ _ALLOWED_NODES: frozenset[type[ast.AST]] = frozenset( ) +def _ast_depth(node: ast.AST) -> int: + """Compute the depth of an AST tree (KTD8). + + Uses iterative BFS to avoid recursion limits on wide trees. + Returns the maximum nesting depth. + """ + if not hasattr(node, "__dict__") and not isinstance(node, list): + return 1 + max_depth = 0 + for child in ast.iter_child_nodes(node): + child_depth = _ast_depth(child) + if child_depth > max_depth: + max_depth = child_depth + return max_depth + 1 + + class _SecurityVisitor(ast.NodeVisitor): """Visit AST nodes, reject any not in the whitelist (KTD7).""" @@ -138,7 +191,7 @@ class _SecurityVisitor(ast.NodeVisitor): def parse_formula( formula: str, allowed_functions: set[str] | None = None -) -> tuple[ast.Expression, dict[str, str]]: +) -> tuple[ast.Expression, dict[str, str], dict[str, tuple[str, str]]]: """Parse a formula string into a safe AST. Args: @@ -147,8 +200,10 @@ def parse_formula( all functions are allowed (used for syntax-only validation). Returns: - Tuple of (AST expression, field_ref_mapping) where - field_ref_mapping maps safe Python identifiers to original field IDs. + Tuple of (AST expression, field_ref_mapping, cross_table_mapping) + where field_ref_mapping maps safe Python identifiers to original + field IDs, and cross_table_mapping maps safe Python identifiers to + (relation_field_id, target_field_id) tuples for cross-table refs. Raises: FormulaParseError: Syntax error in formula. @@ -162,8 +217,8 @@ def parse_formula( if not expr: raise FormulaParseError("Empty formula") - # Substitute field references {field_id} → safe_name - substituted, field_mapping = _substitute_field_refs(expr) + # Substitute field references {field_id} → safe_name, {rel.target} → _ct_... + substituted, field_mapping, cross_table_mapping = _substitute_field_refs(expr) try: tree = ast.parse(substituted, mode="eval") @@ -179,7 +234,15 @@ def parse_formula( visitor = _SecurityVisitor(allowed) visitor.visit(tree) - return tree, field_mapping # type: ignore[return-value] + # KTD8: depth limit — reject formulas with AST deeper than MAX_AST_DEPTH + depth = _ast_depth(tree) + if depth > MAX_AST_DEPTH: + raise FormulaDepthExceededError( + f"Formula AST depth {depth} exceeds limit {MAX_AST_DEPTH}. " + "Simplify the formula (e.g. flatten nested IF/IFS chains)." + ) + + return tree, field_mapping, cross_table_mapping # type: ignore[return-value] def evaluate_ast( @@ -275,6 +338,12 @@ def _eval_node( args = [_eval_node(a, fields, functions) for a in node.args] return functions[func_name](*args) + if isinstance(node, ast.List): + return [_eval_node(e, fields, functions) for e in node.elts] + + if isinstance(node, ast.Tuple): + return tuple(_eval_node(e, fields, functions) for e in node.elts) + raise FormulaSecurityError(f"Disallowed node during evaluation: {type(node).__name__}") diff --git a/src/agentkit/bitable/models.py b/src/agentkit/bitable/models.py index c5e7d21..3c0f871 100644 --- a/src/agentkit/bitable/models.py +++ b/src/agentkit/bitable/models.py @@ -33,7 +33,9 @@ class FieldType(str, Enum): attachment = "attachment" image = "image" formula = "formula" + relation = "relation" lookup = "lookup" + rollup = "rollup" class FieldOwner(str, Enum): @@ -62,6 +64,59 @@ class RecalcStatus(str, Enum): error = "error" +class RelationType(str, Enum): + """Type of relation between two tables (R5). + + - ``one_to_one`` / ``one_to_many``: stored as record JSONB ``["rec_id"]`` + or ``["rec_id1", "rec_id2"]`` arrays (no junction table). + - ``many_to_many``: stored in ``bitable_relation_links`` junction table + (KTD-1) for indexed reverse queries. + """ + + one_to_one = "one_to_one" + one_to_many = "one_to_many" + many_to_many = "many_to_many" + + +class AutomationTriggerType(str, Enum): + """Supported automation trigger types (R13).""" + + record_created = "record_created" + record_updated = "record_updated" + record_deleted = "record_deleted" + + +class AutomationActionType(str, Enum): + """Supported automation action types (R14).""" + + create_record = "create_record" + update_record = "update_record" + send_webhook = "send_webhook" + send_email = "send_email" + send_message = "send_message" + + +class AutomationStatus(str, Enum): + """Status of an automation rule execution log (R17).""" + + success = "success" + error = "error" + retrying = "retrying" + + +class CrossTableDepType(str, Enum): + """Type of cross-table dependency (KTD-3). + + - ``formula``: a formula field references a relation_field.target_field + - ``lookup``: a lookup field pulls from a target table + - ``rollup``: a rollup field aggregates a target table column + """ + + formula = "formula" + lookup = "lookup" + rollup = "rollup" + + class BitableFile(BaseModel): """Top-level container grouping related tables (多维表格文件). @@ -153,7 +208,9 @@ class Field(BaseModel): ``config`` varies by ``field_type``: - select/multiselect: ``{"options": [{"label": "...", "value": "..."}]}`` - formula: ``{"formula_expr": "=SUM({field_abc})"}`` - - lookup: ``{"lookup_target": {"table_id": "...", "field_id": "...", "filter_field_id": "...", "filter_value": "..."}}`` + - relation: ``{"relation_type": "one_to_one|one_to_many|many_to_many", "target_table_id": "...", "is_bidirectional": bool, "reverse_field_id": "..." (optional)}`` + - lookup: ``{"relation_field_id": "...", "target_field_id": "..."}`` + - rollup: ``{"relation_field_id": "...", "target_field_id": "...", "aggregation": "sum|count|avg|min|max|unique"}`` """ model_config = ConfigDict(from_attributes=True) @@ -193,7 +250,11 @@ class View(BaseModel): class RecalcTask(BaseModel): - """An asynchronous formula recalculation task.""" + """An asynchronous formula recalculation task. + + ``is_cross_table`` marks tasks triggered by a cross-table dependency — + e.g. a record change in table A causes a rollup in table B to recalc. + """ model_config = ConfigDict(from_attributes=True) @@ -205,3 +266,81 @@ class RecalcTask(BaseModel): error_message: str | None = None queued_at: datetime = PydanticField(default_factory=_utcnow) completed_at: datetime | None = None + is_cross_table: bool = False + + +class RelationLink(BaseModel): + """A row in the ``bitable_relation_links`` junction table (KTD-1, R9). + + Used for many_to_many relations. One-to-one / one-to-many relations + store their link IDs directly in the record's JSONB ``values`` — those + do not create RelationLink rows. + """ + + model_config = ConfigDict(from_attributes=True) + + id: str + relation_field_id: str + source_record_id: str + target_record_id: str + created_at: datetime = PydanticField(default_factory=_utcnow) + + +class AutomationRule(BaseModel): + """A saved automation rule (R13-R17). + + ``trigger_config`` shape varies by trigger type: + - record_created: ``{"type": "record_created"}`` + - record_updated: ``{"type": "record_updated", "field_ids": ["f1"]}`` (field_ids optional) + - record_deleted: ``{"type": "record_deleted"}`` + + ``action_config`` shape varies by action type (see AutomationActionType). + ``webhook_token`` is set only for inbound-webhook triggered rules. + """ + + model_config = ConfigDict(from_attributes=True) + + id: str + table_id: str + name: str + trigger_config: dict[str, object] = PydanticField(default_factory=dict) + action_config: dict[str, object] = PydanticField(default_factory=dict) + enabled: bool = True + webhook_token: str | None = None + created_at: datetime = PydanticField(default_factory=_utcnow) + updated_at: datetime = PydanticField(default_factory=_utcnow) + + +class AutomationLog(BaseModel): + """Execution log entry for an automation rule (R17).""" + + model_config = ConfigDict(from_attributes=True) + + id: str + automation_id: str + trigger_record_id: str | None = None + status: AutomationStatus = AutomationStatus.success + error_message: str | None = None + attempt: int = 1 + started_at: datetime = PydanticField(default_factory=_utcnow) + completed_at: datetime | None = None + + +class CrossTableDep(BaseModel): + """A materialized cross-table field dependency edge (KTD-3, R11). + + ``source_field_id`` is in ``source_table_id`` and depends on + ``target_field_id`` in ``target_table_id``. When the target field's + value changes in any record, dependents are enqueued for cross-table + recalc (U6). + """ + + model_config = ConfigDict(from_attributes=True) + + id: str + source_table_id: str + source_field_id: str + target_table_id: str + target_field_id: str + dep_type: CrossTableDepType = CrossTableDepType.formula + created_at: datetime = PydanticField(default_factory=_utcnow) diff --git a/src/agentkit/bitable/recalc_worker.py b/src/agentkit/bitable/recalc_worker.py index d6b8497..e957f18 100644 --- a/src/agentkit/bitable/recalc_worker.py +++ b/src/agentkit/bitable/recalc_worker.py @@ -18,10 +18,11 @@ from __future__ import annotations import asyncio import logging +from decimal import Decimal from agentkit.bitable.db import BitableDB from agentkit.bitable.formula.engine import FormulaEngine -from agentkit.bitable.models import FieldType, RecalcStatus, RecalcTask +from agentkit.bitable.models import Field, FieldType, Record, RecalcStatus, RecalcTask from agentkit.bitable.repository import BitableRepository from agentkit.bitable.service import BitableService @@ -32,6 +33,22 @@ _DEFAULT_REAPER_INTERVAL = 300 # 5 minutes _DEFAULT_STALE_THRESHOLD = 600 # 10 minutes +def _to_float(v: object) -> float | None: + """Normalize JSONB numeric values to float (handles Decimal from asyncpg). + + PostgreSQL JSONB returns numbers as ``Decimal``; ``isinstance(v, (int, float))`` + misses them. ``bool`` is a subclass of ``int`` — excluded so True/False don't + pollute numeric aggregations. + """ + if isinstance(v, bool): + return None + if isinstance(v, (int, float)): + return float(v) + if isinstance(v, Decimal): + return float(v) + return None + + class RecalcWorker: """Background worker that processes formula recalc tasks. @@ -181,29 +198,25 @@ class RecalcWorker: called from the worker loop (atomic claim sets it). When called synchronously via ``service.process_recalc_task``, this method marks it calculating first (idempotent — re-marking is harmless). + + Handles three computed field types (U3/U5/U6): + - ``formula``: parsed by FormulaEngine; may reference cross-table + fields via ``{rel.target}`` syntax (resolved by _resolve_cross_table_values). + - ``lookup``: pulls a single related record's field value via a relation. + - ``rollup``: aggregates related records' field values (SUM/AVG/COUNT/MIN/MAX). """ # Idempotent: mark calculating (no-op if already calculating via claim). await self._repo.update_recalc_status(task.id, RecalcStatus.calculating) try: field = await self._repo.get_field(task.field_id) - if field is None or field.field_type != FieldType.formula: + if field is None or field.field_type not in ( + FieldType.formula, + FieldType.lookup, + FieldType.rollup, + ): await self._repo.update_recalc_status( - task.id, RecalcStatus.error, "Field not found or not a formula" - ) - return - - formula_expr = field.config.get("formula_expr", "") - if not formula_expr: - await self._repo.update_recalc_status( - task.id, RecalcStatus.error, "No formula_expr in field config" - ) - return - - engine = await self._get_or_build_engine(task.table_id) - if engine is None: - await self._repo.update_recalc_status( - task.id, RecalcStatus.error, "No formula fields in table" + task.id, RecalcStatus.error, "Field not found or not computed" ) return @@ -214,18 +227,13 @@ class RecalcWorker: ) return - deps = engine.get_dependencies(task.field_id) - column_values: dict[str, list[object]] = {} - for dep_field_id in deps: - column_values[dep_field_id] = await self._repo.get_column_values( - task.table_id, dep_field_id - ) - - result = engine.evaluate( - task.field_id, - row_values=record.values, - column_values=column_values, - ) + if field.field_type == FieldType.lookup: + result = await self._evaluate_lookup(field, record) + elif field.field_type == FieldType.rollup: + result = await self._evaluate_rollup(field, record) + else: + # Formula — may have cross-table refs (U3) + result = await self._evaluate_formula(field, record, task.table_id) await self._repo.set_formula_value(task.record_id, task.field_id, result) await self._repo.update_recalc_status(task.id, RecalcStatus.done) @@ -234,6 +242,151 @@ class RecalcWorker: logger.exception("RecalcWorker: error processing task %s", task.id) await self._repo.update_recalc_status(task.id, RecalcStatus.error, str(e)[:500]) + async def _evaluate_formula(self, field: Field, record: Record, table_id: str) -> object: + """Evaluate a formula field (U1+U3). Resolves cross-table refs.""" + formula_expr = field.config.get("formula_expr", "") + if not formula_expr: + raise ValueError("No formula_expr in field config") + + engine = await self._get_or_build_engine(table_id) + if engine is None: + raise ValueError("No formula fields in table") + + deps = engine.get_dependencies(field.id) + column_values: dict[str, list[object]] = {} + for dep_field_id in deps: + column_values[dep_field_id] = await self._repo.get_column_values(table_id, dep_field_id) + + # U3: resolve cross-table references {rel.target} + cross_table_values = await self._resolve_cross_table_values(engine, field.id, record) + + return engine.evaluate( + field.id, + row_values=record.values, + column_values=column_values, + cross_table_values=cross_table_values, + ) + + async def _evaluate_lookup(self, field: Field, record: Record) -> object: + """Evaluate a lookup field (U5, R6). + + Config: ``{relation_field_id, target_field_id}`` + Resolves the related record(s) via the relation field, then reads + the target field value from the first related record. + """ + rel_field_id = str(field.config.get("relation_field_id", "")) + target_field_id = str(field.config.get("target_field_id", "")) + if not rel_field_id or not target_field_id: + raise ValueError("lookup field missing relation_field_id or target_field_id") + + # Same forward/reverse logic as _evaluate_rollup + rel_field = await self._repo.get_field(rel_field_id) + reverse = rel_field is not None and rel_field.table_id != record.table_id + + related_ids = await self._service.list_related_records( + relation_field_id=rel_field_id, + record_id=record.id, + reverse=reverse, + ) + if not related_ids: + return None + # Lookup takes the first related record's value (one_to_one / one_to_many first) + target_record = await self._repo.get_record(related_ids[0]) + if target_record is None: + return None + raw = target_record.values.get(target_field_id) + # Normalize Decimal → float for JSON serialization (set_formula_value uses json.dumps) + n = _to_float(raw) + return n if n is not None else raw + + async def _evaluate_rollup(self, field: Field, record: Record) -> object: + """Evaluate a rollup field (U5, R8). + + Config: ``{relation_field_id, target_field_id, aggregation}`` + Aggregation is one of: sum, avg, count, min, max, concat, count_distinct. + """ + rel_field_id = str(field.config.get("relation_field_id", "")) + target_field_id = str(field.config.get("target_field_id", "")) + aggregation = str(field.config.get("aggregation", "sum")).lower() + if not rel_field_id or not target_field_id: + raise ValueError("rollup field missing relation_field_id or target_field_id") + + # Determine forward vs reverse lookup: if the record is on the same + # table as the relation field, it's a forward lookup (record is the + # source). Otherwise the record is on the target table — reverse lookup. + rel_field = await self._repo.get_field(rel_field_id) + if rel_field is not None and rel_field.table_id != record.table_id: + reverse = True + else: + reverse = False + + related_ids = await self._service.list_related_records( + relation_field_id=rel_field_id, + record_id=record.id, + reverse=reverse, + ) + if not related_ids: + return 0 if aggregation in ("sum", "count") else None + + # Fetch target field values from all related records + values: list[object] = [] + for rid in related_ids: + rec = await self._repo.get_record(rid) + if rec is not None: + values.append(rec.values.get(target_field_id)) + + # Apply aggregation + # ponytail: JSONB numbers come back as Decimal from asyncpg — _to_float + # normalizes them (and excludes bool which is an int subclass). + numeric_values = [n for v in values if (n := _to_float(v)) is not None] + non_none = [v for v in values if v is not None] + + if aggregation == "sum": + return sum(numeric_values) + if aggregation == "avg": + return sum(numeric_values) / len(numeric_values) if numeric_values else 0 + if aggregation == "count": + return len(non_none) + if aggregation == "count_distinct": + return len(set(non_none)) + if aggregation == "min": + return min(numeric_values) if numeric_values else None + if aggregation == "max": + return max(numeric_values) if numeric_values else None + if aggregation == "concat": + return "".join(str(v) for v in non_none) + raise ValueError(f"Unknown rollup aggregation: {aggregation}") + + async def _resolve_cross_table_values( + self, engine: FormulaEngine, field_id: str, record: Record + ) -> dict[str, list[object]]: + """Resolve cross-table references for a formula (U3). + + For each ``{rel_field.target_field}`` ref in the formula: + 1. Find related records via the relation field on this record. + 2. Read target_field values from those related records. + 3. Return mapping safe_name → list of values (for aggregate context). + + If the relation field has no links, returns empty lists for each ref. + """ + cross_table_mapping = engine.get_cross_table_mapping(field_id) + if not cross_table_mapping: + return {} + + result: dict[str, list[object]] = {} + for safe_name, (rel_field_id, target_field_id) in cross_table_mapping.items(): + related_ids = await self._service.list_related_records( + relation_field_id=rel_field_id, + record_id=record.id, + ) + values: list[object] = [] + for rid in related_ids: + rec = await self._repo.get_record(rid) + if rec is not None: + values.append(rec.values.get(target_field_id)) + result[safe_name] = values + return result + async def _get_or_build_engine(self, table_id: str) -> FormulaEngine | None: """Get or build a FormulaEngine for a table. diff --git a/src/agentkit/bitable/repository.py b/src/agentkit/bitable/repository.py index afaee6d..e0cbffa 100644 --- a/src/agentkit/bitable/repository.py +++ b/src/agentkit/bitable/repository.py @@ -11,27 +11,37 @@ import logging import re from datetime import datetime, timedelta, timezone -from sqlalchemy import delete, func, insert, select, text, update +from sqlalchemy import case, delete, func, insert, select, text, update from sqlalchemy.dialects.postgresql import insert as pg_insert from agentkit.bitable.db import ( + AutomationLogModel, + AutomationModel, BitableDB, + CrossTableDepModel, FieldModel, FileModel, RecordModel, RecalcQueueModel, + RelationLinkModel, TableModel, ViewModel, _uuid_str, ) from agentkit.bitable.models import ( + AutomationLog, + AutomationRule, + AutomationStatus, BitableFile, + CrossTableDep, + CrossTableDepType, Field, FieldOwner, FieldType, Record, RecalcStatus, RecalcTask, + RelationLink, Table, View, ViewType, @@ -226,13 +236,16 @@ class BitableRepository: if not re.match(r"^[a-f0-9-]{36}$", pk_field_id): raise ValueError(f"Invalid pk_field_id format: {pk_field_id}") index_name = f"ix_bitable_records_pk_{table_id.replace('-', '_')}" + # asyncpg does not support parameter binding in DDL statements (CREATE INDEX), + # so pk_field_id is interpolated directly. It is validated as a strict UUID + # (hex + hyphens, 36 chars) above, so SQL injection is not possible here. sql = text( f"CREATE UNIQUE INDEX IF NOT EXISTS {index_name} " f"ON bitable.bitable_records (table_id, (values->>'{pk_field_id}')) " - f"WHERE values ? :pk_path" + f"WHERE values ? '{pk_field_id}'" ) async with self._session_factory() as session: - await session.execute(sql, {"pk_path": pk_field_id}) + await session.execute(sql) await session.commit() async def delete_table(self, table_id: str) -> bool: @@ -411,6 +424,27 @@ class BitableRepository: await session.commit() return Record.model_validate(entity) if entity else None + async def merge_record_values( + self, record_id: str, values: dict[str, object] + ) -> Record | None: + """Shallow-merge values into a record's JSONB (``values || :new``). + + Unlike :meth:`update_record_values` (full replace), this preserves + existing keys not present in ``values``. Used by relation link + management to set a single field without clobbering the rest. + """ + async with self._session_factory() as session: + stmt = ( + update(RecordModel) + .where(RecordModel.id == record_id) + .values(values=RecordModel.values.op("||")(values)) + .returning(RecordModel) + ) + result = await session.execute(stmt) + entity = result.scalars().first() + await session.commit() + return Record.model_validate(entity) if entity else None + async def delete_record(self, record_id: str) -> bool: """Delete a record.""" async with self._session_factory() as session: @@ -518,9 +552,24 @@ class BitableRepository: # ── Recalc Queue ──────────────────────────────────────── async def enqueue_recalc( - self, table_id: str, record_id: str, field_id: str + self, + table_id: str, + record_id: str, + field_id: str, + is_cross_table: bool = False, ) -> RecalcTask | None: - """Enqueue a recalc task. Returns None if duplicate (ON CONFLICT DO NOTHING).""" + """Enqueue a recalc task. + + If a task for ``(record_id, field_id)`` already exists: + - ``pending`` / ``calculating`` → left as-is (avoid duplicate processing). + - ``done`` / ``error`` → reset to ``pending`` so a new change can + trigger a fresh recalc. Without this, the unique constraint would + silently swallow the new request and the field would show stale data. + + ``is_cross_table`` marks tasks triggered by a cross-table dependency + (U6) — distinguishing them from same-table recalc helps the worker + load the correct evaluation context. + """ async with self._session_factory() as session: stmt = ( pg_insert(RecalcQueueModel) @@ -530,8 +579,22 @@ class BitableRepository: record_id=record_id, field_id=field_id, status=RecalcStatus.pending.value, + is_cross_table=is_cross_table, + ) + .on_conflict_do_update( + constraint="uq_recalc_record_field", + set_={ + "status": case( + ( + RecalcQueueModel.status.in_( + [RecalcStatus.done.value, RecalcStatus.error.value] + ), + RecalcStatus.pending.value, + ), + else_=RecalcQueueModel.status, + ), + }, ) - .on_conflict_do_nothing(constraint="uq_recalc_record_field") .returning(RecalcQueueModel) ) result = await session.execute(stmt) @@ -951,3 +1014,381 @@ class BitableRepository: ) result = await session.execute(stmt) return [Record.model_validate(e) for e in result.scalars().all()] + + # ── Relation Links (U4, KTD-1: many_to_many junction) ─── + + async def add_relation_link( + self, + relation_field_id: str, + source_record_id: str, + target_record_id: str, + ) -> RelationLink: + """Create a junction-table link for a many_to_many relation.""" + async with self._session_factory() as session: + stmt = ( + insert(RelationLinkModel) + .values( + id=_uuid_str(), + relation_field_id=relation_field_id, + source_record_id=source_record_id, + target_record_id=target_record_id, + ) + .returning(RelationLinkModel) + ) + result = await session.execute(stmt) + entity = result.scalar_one() + await session.commit() + return RelationLink.model_validate(entity) + + async def remove_relation_link( + self, + relation_field_id: str, + source_record_id: str, + target_record_id: str, + ) -> bool: + """Remove a single junction-table link.""" + async with self._session_factory() as session: + result = await session.execute( + delete(RelationLinkModel).where( + RelationLinkModel.relation_field_id == relation_field_id, + RelationLinkModel.source_record_id == source_record_id, + RelationLinkModel.target_record_id == target_record_id, + ) + ) + await session.commit() + return result.rowcount > 0 + + async def remove_all_relation_links( + self, relation_field_id: str, source_record_id: str | None = None + ) -> int: + """Remove all links for a relation field (optionally for one source record). + + Used during cascade cleanup when a relation field is deleted, or when + a source record is deleted. + """ + async with self._session_factory() as session: + stmt = delete(RelationLinkModel).where( + RelationLinkModel.relation_field_id == relation_field_id + ) + if source_record_id is not None: + stmt = stmt.where( + RelationLinkModel.source_record_id == source_record_id + ) + result = await session.execute(stmt) + await session.commit() + return result.rowcount + + async def list_relation_links( + self, relation_field_id: str, source_record_id: str + ) -> list[str]: + """List target_record_ids linked to source_record_id via relation_field_id.""" + async with self._session_factory() as session: + stmt = ( + select(RelationLinkModel.target_record_id) + .where( + RelationLinkModel.relation_field_id == relation_field_id, + RelationLinkModel.source_record_id == source_record_id, + ) + .order_by(RelationLinkModel.created_at) + ) + result = await session.execute(stmt) + return [r[0] for r in result.fetchall()] + + async def list_reverse_relation_links( + self, relation_field_id: str, target_record_id: str + ) -> list[str]: + """List source_record_ids whose relation_field points at target_record_id. + + Used by the cross-table recalc worker to find which records in the + source table depend on a changed target record (U6). + """ + async with self._session_factory() as session: + stmt = ( + select(RelationLinkModel.source_record_id) + .where( + RelationLinkModel.relation_field_id == relation_field_id, + RelationLinkModel.target_record_id == target_record_id, + ) + .order_by(RelationLinkModel.created_at) + ) + result = await session.execute(stmt) + return [r[0] for r in result.fetchall()] + + async def list_relation_links_for_field( + self, relation_field_id: str + ) -> list[RelationLink]: + """List all links for a relation field (used during cascade delete).""" + async with self._session_factory() as session: + stmt = ( + select(RelationLinkModel) + .where(RelationLinkModel.relation_field_id == relation_field_id) + .order_by(RelationLinkModel.created_at) + ) + result = await session.execute(stmt) + return [RelationLink.model_validate(e) for e in result.scalars().all()] + + # ── Cross-table dependency graph (U6, KTD-3) ──────────── + + async def add_cross_table_dep( + self, + source_table_id: str, + source_field_id: str, + target_table_id: str, + target_field_id: str, + dep_type: CrossTableDepType = CrossTableDepType.formula, + ) -> CrossTableDep | None: + """Record a cross-table field dependency. Returns None if duplicate.""" + async with self._session_factory() as session: + stmt = ( + pg_insert(CrossTableDepModel) + .values( + id=_uuid_str(), + source_table_id=source_table_id, + source_field_id=source_field_id, + target_table_id=target_table_id, + target_field_id=target_field_id, + dep_type=dep_type.value, + ) + .on_conflict_do_nothing( + constraint="uq_cross_table_dep_source_target" + ) + .returning(CrossTableDepModel) + ) + result = await session.execute(stmt) + entity = result.scalars().first() + await session.commit() + return CrossTableDep.model_validate(entity) if entity else None + + async def remove_cross_table_deps_for_source(self, source_field_id: str) -> int: + """Remove all dep edges where source_field_id is the dependent. + + Used when a formula/lookup/rollup field is deleted or its config changes + — caller re-adds the current edges afterwards. + """ + async with self._session_factory() as session: + result = await session.execute( + delete(CrossTableDepModel).where( + CrossTableDepModel.source_field_id == source_field_id + ) + ) + await session.commit() + return result.rowcount + + async def find_cross_table_dependents( + self, target_table_id: str, target_field_id: str | None = None + ) -> list[CrossTableDep]: + """Find all (source_table, source_field) pairs that depend on target. + + Called by the recalc worker after a record write: which fields in + OTHER tables will need recalc because this field's value changed? + """ + async with self._session_factory() as session: + stmt = select(CrossTableDepModel).where( + CrossTableDepModel.target_table_id == target_table_id + ) + if target_field_id is not None: + stmt = stmt.where(CrossTableDepModel.target_field_id == target_field_id) + result = await session.execute(stmt) + return [CrossTableDep.model_validate(e) for e in result.scalars().all()] + + async def list_cross_table_deps_for_source_table( + self, source_table_id: str + ) -> list[CrossTableDep]: + """List all dep edges originating from fields in source_table_id.""" + async with self._session_factory() as session: + stmt = select(CrossTableDepModel).where( + CrossTableDepModel.source_table_id == source_table_id + ) + result = await session.execute(stmt) + return [CrossTableDep.model_validate(e) for e in result.scalars().all()] + + async def list_cross_table_deps_for_target( + self, target_table_id: str, target_field_id: str + ) -> list[CrossTableDep]: + """List all dep edges pointing at (target_table, target_field) (U6 R11). + + Used by ``trigger_cross_table_recalc`` to find downstream source + fields that must be recomputed when a target field changes. + """ + async with self._session_factory() as session: + stmt = select(CrossTableDepModel).where( + CrossTableDepModel.target_table_id == target_table_id, + CrossTableDepModel.target_field_id == target_field_id, + ) + result = await session.execute(stmt) + return [CrossTableDep.model_validate(e) for e in result.scalars().all()] + + # ── Automations (U7, R16) ─────────────────────────────── + + async def create_automation( + self, + table_id: str, + name: str, + trigger_config: dict[str, object], + action_config: dict[str, object], + enabled: bool = True, + webhook_token: str | None = None, + ) -> AutomationRule: + """Create a new automation rule.""" + async with self._session_factory() as session: + stmt = ( + insert(AutomationModel) + .values( + id=_uuid_str(), + table_id=table_id, + name=name, + trigger_config=trigger_config, + action_config=action_config, + enabled=enabled, + webhook_token=webhook_token, + ) + .returning(AutomationModel) + ) + result = await session.execute(stmt) + entity = result.scalar_one() + await session.commit() + return AutomationRule.model_validate(entity) + + async def get_automation(self, automation_id: str) -> AutomationRule | None: + async with self._session_factory() as session: + result = await session.execute( + select(AutomationModel).where(AutomationModel.id == automation_id) + ) + entity = result.scalars().first() + return AutomationRule.model_validate(entity) if entity else None + + async def get_automation_by_webhook_token(self, token: str) -> AutomationRule | None: + """Look up an automation rule by its inbound webhook token. + + Caller must use ``hmac.compare_digest`` when matching user-supplied + tokens; this method does an exact equality lookup against the DB + indexed column. + """ + async with self._session_factory() as session: + result = await session.execute( + select(AutomationModel).where(AutomationModel.webhook_token == token) + ) + entity = result.scalars().first() + return AutomationRule.model_validate(entity) if entity else None + + async def list_automations( + self, table_id: str, enabled_only: bool = False + ) -> list[AutomationRule]: + async with self._session_factory() as session: + stmt = ( + select(AutomationModel) + .where(AutomationModel.table_id == table_id) + .order_by(AutomationModel.created_at) + ) + if enabled_only: + stmt = stmt.where(AutomationModel.enabled.is_(True)) + result = await session.execute(stmt) + return [AutomationRule.model_validate(e) for e in result.scalars().all()] + + async def update_automation(self, automation_id: str, **kwargs: object) -> AutomationRule | None: + async with self._session_factory() as session: + stmt = ( + update(AutomationModel) + .where(AutomationModel.id == automation_id) + .values(**kwargs) + .returning(AutomationModel) + ) + result = await session.execute(stmt) + entity = result.scalars().first() + await session.commit() + return AutomationRule.model_validate(entity) if entity else None + + async def delete_automation(self, automation_id: str) -> bool: + async with self._session_factory() as session: + # Cascade: delete logs first, then the rule. + await session.execute( + delete(AutomationLogModel).where(AutomationLogModel.automation_id == automation_id) + ) + result = await session.execute( + delete(AutomationModel).where(AutomationModel.id == automation_id) + ) + await session.commit() + return result.rowcount > 0 + + async def list_automations_for_trigger( + self, + table_id: str, + trigger_type: str, + ) -> list[AutomationRule]: + """List enabled automations in a table that match a trigger type. + + Uses JSONB containment: ``trigger_config->>'type' = :ttype``. The + trigger_type arg is parameterized. + """ + async with self._session_factory() as session: + sql = text( + "SELECT * FROM bitable.bitable_automations " + "WHERE table_id = :table_id AND enabled = TRUE " + "AND trigger_config->>'type' = :ttype " + "ORDER BY created_at" + ) + result = await session.execute( + sql, {"table_id": table_id, "ttype": trigger_type} + ) + return [AutomationRule.model_validate(AutomationModel(**r._mapping)) for r in result.fetchall()] + + # ── Automation Logs (U7, R17) ─────────────────────────── + + async def create_automation_log( + self, + automation_id: str, + trigger_record_id: str | None, + status: AutomationStatus = AutomationStatus.success, + error_message: str | None = None, + attempt: int = 1, + ) -> AutomationLog: + async with self._session_factory() as session: + stmt = ( + insert(AutomationLogModel) + .values( + id=_uuid_str(), + automation_id=automation_id, + trigger_record_id=trigger_record_id, + status=status.value, + error_message=error_message, + attempt=attempt, + ) + .returning(AutomationLogModel) + ) + result = await session.execute(stmt) + entity = result.scalar_one() + await session.commit() + return AutomationLog.model_validate(entity) + + async def update_automation_log( + self, + log_id: str, + status: AutomationStatus, + error_message: str | None = None, + ) -> None: + async with self._session_factory() as session: + kwargs: dict[str, object] = {"status": status.value} + if error_message is not None: + kwargs["error_message"] = error_message + if status in (AutomationStatus.success, AutomationStatus.error): + kwargs["completed_at"] = func.now() + stmt = ( + update(AutomationLogModel) + .where(AutomationLogModel.id == log_id) + .values(**kwargs) + ) + await session.execute(stmt) + await session.commit() + + async def list_automation_logs( + self, automation_id: str, limit: int = 50 + ) -> list[AutomationLog]: + async with self._session_factory() as session: + stmt = ( + select(AutomationLogModel) + .where(AutomationLogModel.automation_id == automation_id) + .order_by(AutomationLogModel.started_at.desc()) + .limit(limit) + ) + result = await session.execute(stmt) + return [AutomationLog.model_validate(e) for e in result.scalars().all()] diff --git a/src/agentkit/bitable/service.py b/src/agentkit/bitable/service.py index fa72ba3..f486e30 100644 --- a/src/agentkit/bitable/service.py +++ b/src/agentkit/bitable/service.py @@ -8,6 +8,7 @@ Routes and CLI call this layer — never the repository directly. from __future__ import annotations +import asyncio import logging import os from datetime import datetime, timezone @@ -17,12 +18,15 @@ from typing import TYPE_CHECKING, TypeAlias from agentkit.bitable.db import BitableDB from agentkit.bitable.models import ( DEFAULT_FIELD_TEMPLATES, + AutomationLog, + AutomationRule, BitableFile, Field, FieldOwner, FieldType, Record, RecalcTask, + RelationType, Table, View, ViewType, @@ -37,6 +41,18 @@ if TYPE_CHECKING: def invalidate_engine(self, table_id: str) -> None: ... + class _AutomationEngine(Protocol): + """Structural type for the automation engine's trigger-dispatch surface.""" + + async def fire_trigger( + self, + table_id: str, + trigger_type: object, + record_id: str | None = None, + changed_field_ids: list[str] | None = None, + record_values: dict[str, object] | None = None, + ) -> int: ... + logger = logging.getLogger(__name__) @@ -75,6 +91,8 @@ class BitableService: self._db = db self._repo = BitableRepository(db) self._recalc_worker: _RecalcWorker | None = None # set via set_recalc_worker + self._automation_engine: _AutomationEngine | None = None # set via set_automation_engine + self._pending_trigger_tasks: set[asyncio.Task[object]] = set() @property def repo(self) -> BitableRepository: @@ -89,11 +107,53 @@ class BitableService: """ self._recalc_worker = worker + def set_automation_engine(self, engine: _AutomationEngine) -> None: + """Register the AutomationEngine for trigger dispatch (U7). + + Called after both service and engine are constructed. When unset, + record writes skip trigger evaluation — automations are silent. + """ + self._automation_engine = engine + def _invalidate_engine_cache(self, table_id: str) -> None: """Invalidate the worker's cached formula engine for a table (P1 #5).""" if self._recalc_worker is not None: self._recalc_worker.invalidate_engine(table_id) + def _fire_automation_trigger( + self, + table_id: str, + trigger_type: str, + record_id: str | None = None, + changed_field_ids: list[str] | None = None, + record_values: dict[str, object] | None = None, + ) -> None: + """Fire automation triggers (U7). Best-effort — errors are logged, not raised.""" + if self._automation_engine is None: + return + from agentkit.bitable.models import AutomationTriggerType + + try: + trig = AutomationTriggerType(trigger_type) + except ValueError: + return + # Schedule the trigger fire — don't block the record write. + # Track the task so exceptions surface in the app logger instead of + # being silently dropped as "Task exception was never retrieved". + import asyncio + + task = asyncio.create_task( + self._automation_engine.fire_trigger( + table_id=table_id, + trigger_type=trig, + record_id=record_id, + changed_field_ids=changed_field_ids, + record_values=record_values, + ) + ) + self._pending_trigger_tasks.add(task) + task.add_done_callback(self._pending_trigger_tasks.discard) + # ── Files (R1) ───────────────────────────────────────── async def create_file( @@ -212,8 +272,8 @@ class BitableService: config=config or {}, owner=owner, ) - # New formula field changes the table's DAG — invalidate cached engine (P1 #5). - if field_type == FieldType.formula: + # New computed field (formula/lookup/rollup) changes the table's DAG — invalidate cached engine (P1 #5). + if field_type in (FieldType.formula, FieldType.lookup, FieldType.rollup): self._invalidate_engine_cache(table_id) return field @@ -310,6 +370,12 @@ class BitableService: values[f.id] = now_iso record = await self._repo.create_record(table_id, values) await self._trigger_recalc_for_affected_fields(table_id, record.id) + self._fire_automation_trigger( + table_id=table_id, + trigger_type="record_created", + record_id=record.id, + record_values=dict(record.values), + ) return record async def create_records_batch( @@ -368,10 +434,22 @@ class BitableService: ) async def update_record_values(self, record_id: str, values: BitableRecord) -> Record | None: - """Update a record's values (full replace). Triggers recalc for affected formulas.""" - record = await self._repo.update_record_values(record_id, values) + """Update specific field values in a record (shallow merge, not full replace). + + Triggers recalc for affected formula fields on this table AND on tables + that reference this record via relations (cross-table rollup/lookup). + """ + record = await self._repo.merge_record_values(record_id, values) if record is not None: await self._trigger_recalc_for_affected_fields(record.table_id, record.id) + await self._trigger_cross_table_recalc(record) + self._fire_automation_trigger( + table_id=record.table_id, + trigger_type="record_updated", + record_id=record.id, + changed_field_ids=list(values.keys()), + record_values=dict(record.values), + ) return record async def delete_record(self, record_id: str) -> bool: @@ -380,7 +458,15 @@ class BitableService: if record is None: return False await self._cleanup_attachment_files(record.table_id, [record]) - return await self._repo.delete_record(record_id) + deleted = await self._repo.delete_record(record_id) + if deleted: + self._fire_automation_trigger( + table_id=record.table_id, + trigger_type="record_deleted", + record_id=record_id, + record_values=dict(record.values), + ) + return deleted async def delete_records_by_table(self, table_id: str) -> int: """Delete all records in a table and clean up attachment/image files. @@ -576,20 +662,47 @@ class BitableService: # ── Recalc (U3: formula recalc pipeline) ──────────────── async def _trigger_recalc_for_affected_fields(self, table_id: str, record_id: str) -> None: - """Detect formula fields affected by a record write and enqueue recalc. + """Detect computed fields affected by a record write and enqueue recalc. - Finds all formula fields in the table, checks which ones depend on - the fields that were just written, and enqueues recalc tasks. - For simplicity (v1), we enqueue recalc for ALL formula fields in the - table for this record — the worker will evaluate them in topological - order. The ON CONFLICT DO NOTHING constraint deduplicates. + Finds all formula/lookup/rollup fields in the table and enqueues + recalc tasks. The worker evaluates them in topological order. + ON CONFLICT DO NOTHING deduplicates. """ fields = await self._repo.list_fields(table_id) - formula_fields = [f for f in fields if f.field_type == FieldType.formula] + computed_fields = [ + f + for f in fields + if f.field_type in (FieldType.formula, FieldType.lookup, FieldType.rollup) + ] - for f in formula_fields: + for f in computed_fields: await self._repo.enqueue_recalc(table_id, record_id, f.id) + async def _trigger_cross_table_recalc(self, record: Record) -> None: + """Trigger recalc on records that depend on ``record`` via relations. + + When a record's values change, rollup/lookup fields on RELATED records + (target table) may need recalc. For each relation field on this record's + table, find the target records this record links to and trigger their + recalc. + """ + fields = await self._repo.list_fields(record.table_id) + relation_fields = [f for f in fields if f.field_type == FieldType.relation] + + for rf in relation_fields: + target_table_id = rf.config.get("target_table_id") + if not isinstance(target_table_id, str): + continue + val = record.values.get(rf.id) + target_ids: list[str] = [] + if isinstance(val, str): + target_ids = [val] + elif isinstance(val, list): + target_ids = [v for v in val if isinstance(v, str)] + + for tid in target_ids: + await self._trigger_recalc_for_affected_fields(target_table_id, tid) + async def trigger_recalc( self, table_id: str, record_id: str, field_id: str ) -> RecalcTask | None: @@ -619,3 +732,575 @@ class BitableService: worker = RecalcWorker(self._db, self) await worker.process_task(task) worker.invalidate_engine(task.table_id) + + # ── Relations (U4: R5, R7, R8) ────────────────────────── + + async def create_relation_field( + self, + table_id: str, + name: str, + target_table_id: str, + relation_type: RelationType = RelationType.one_to_many, + is_bidirectional: bool = False, + reverse_field_name: str | None = None, + ) -> Field: + """Create a relation field linking this table to a target table. + + For bidirectional relations, a reverse relation field is also created + on the target table. The reverse field's ID is stored in this field's + config as ``reverse_field_id``. + + Args: + table_id: Source table (where the field is created). + name: Field name on the source table. + target_table_id: Target table to link to. + relation_type: one_to_one, one_to_many, or many_to_many. + is_bidirectional: If True, create a reverse field on the target table. + reverse_field_name: Name for the reverse field (defaults to " ↩"). + + Returns: + The created relation Field on the source table. + """ + # Accept both RelationType enum and string for API ergonomics + if isinstance(relation_type, str): + relation_type = RelationType(relation_type) + config: dict[str, object] = { + "relation_type": relation_type.value, + "target_table_id": target_table_id, + "is_bidirectional": is_bidirectional, + } + + field = await self._repo.create_field( + table_id=table_id, + name=name, + field_type=FieldType.relation, + config=config, + owner=FieldOwner.user, + ) + + if is_bidirectional: + # Create reverse field on target table + src_table = await self._repo.get_table(table_id) + reverse_name = reverse_field_name or f"{src_table.name if src_table else 'Source'} ↩" + reverse_config: dict[str, object] = { + "relation_type": ( + RelationType.one_to_many.value + if relation_type == RelationType.one_to_many + else relation_type.value + ), + "target_table_id": table_id, + "is_bidirectional": True, + "reverse_field_id": field.id, + } + reverse_field = await self._repo.create_field( + table_id=target_table_id, + name=reverse_name, + field_type=FieldType.relation, + config=reverse_config, + owner=FieldOwner.user, + ) + # Link the two fields via reverse_field_id + await self._repo.update_field( + field.id, + config={**config, "reverse_field_id": reverse_field.id}, + ) + field = await self._repo.get_field(field.id) or field + + return field + + async def add_relation_link( + self, + relation_field_id: str, + source_record_id: str, + target_record_id: str, + ) -> bool: + """Link two records via a relation field. + + For many_to_many: creates a junction table entry (idempotent). + For one_to_one / one_to_many: appends target_record_id to the + source record's JSONB values for this field. + + Returns True if a new link was created, False if it already existed. + """ + field = await self._repo.get_field(relation_field_id) + if field is None or field.field_type != FieldType.relation: + raise ValueError(f"Field {relation_field_id} is not a relation field") + + relation_type_str = field.config.get("relation_type", "one_to_many") + relation_type = RelationType(relation_type_str) + + if relation_type == RelationType.many_to_many: + # Junction table entry (idempotent via ON CONFLICT DO NOTHING) + added = await self._repo.add_relation_link( + relation_field_id=relation_field_id, + source_record_id=source_record_id, + target_record_id=target_record_id, + ) + if added: + # Trigger recalc on both sides + src_record = await self._repo.get_record(source_record_id) + if src_record is not None: + await self._trigger_recalc_for_affected_fields( + src_record.table_id, source_record_id + ) + target_table_id = field.config.get("target_table_id") + if isinstance(target_table_id, str): + await self._trigger_recalc_for_affected_fields( + target_table_id, target_record_id + ) + return added + + # one_to_one / one_to_many: store in record JSONB + record = await self._repo.get_record(source_record_id) + if record is None: + raise ValueError(f"Source record {source_record_id} not found") + + current = record.values.get(relation_field_id) + if relation_type == RelationType.one_to_one: + new_value: object = target_record_id + else: # one_to_many + existing: list[str] = list(current) if isinstance(current, list) else [] + if target_record_id in existing: + return False + existing.append(target_record_id) + new_value = existing + + await self._repo.merge_record_values(source_record_id, {relation_field_id: new_value}) + + # For bidirectional relations, update the reverse field too + reverse_field_id = field.config.get("reverse_field_id") + if reverse_field_id and isinstance(reverse_field_id, str): + reverse_field = await self._repo.get_field(reverse_field_id) + if reverse_field is not None: + reverse_record = await self._repo.get_record(target_record_id) + if reverse_record is not None: + rev_current = reverse_record.values.get(reverse_field_id) + rev_existing: list[str] = ( + list(rev_current) if isinstance(rev_current, list) else [] + ) + if source_record_id not in rev_existing: + rev_existing.append(source_record_id) + await self._repo.merge_record_values( + target_record_id, + {reverse_field_id: rev_existing}, + ) + + # Trigger recalc on both source and target records — relation links + # affect lookup/rollup/formula fields on both sides. + await self._trigger_recalc_for_affected_fields(record.table_id, source_record_id) + target_table_id = field.config.get("target_table_id") + if isinstance(target_table_id, str): + await self._trigger_recalc_for_affected_fields(target_table_id, target_record_id) + + return True + + async def remove_relation_link( + self, + relation_field_id: str, + source_record_id: str, + target_record_id: str, + ) -> bool: + """Remove a link between two records. + + Returns True if a link was removed, False if no link existed. + For bidirectional relations, the reverse field on the target record is + also cleaned up (mirrors ``add_relation_link``). + """ + field = await self._repo.get_field(relation_field_id) + if field is None or field.field_type != FieldType.relation: + raise ValueError(f"Field {relation_field_id} is not a relation field") + + relation_type_str = field.config.get("relation_type", "one_to_many") + relation_type = RelationType(relation_type_str) + + if relation_type == RelationType.many_to_many: + removed = await self._repo.remove_relation_link( + relation_field_id=relation_field_id, + source_record_id=source_record_id, + target_record_id=target_record_id, + ) + if removed: + await self._cleanup_reverse_link(field, source_record_id, target_record_id) + return removed + + # one_to_one / one_to_many: remove from record JSONB + record = await self._repo.get_record(source_record_id) + if record is None: + return False + + current = record.values.get(relation_field_id) + if relation_type == RelationType.one_to_one: + if current == target_record_id: + await self._repo.merge_record_values(source_record_id, {relation_field_id: None}) + await self._cleanup_reverse_link(field, source_record_id, target_record_id) + return True + return False + + # one_to_many: remove from list + if not isinstance(current, list): + return False + if target_record_id not in current: + return False + new_list = [r for r in current if r != target_record_id] + await self._repo.merge_record_values(source_record_id, {relation_field_id: new_list}) + await self._cleanup_reverse_link(field, source_record_id, target_record_id) + return True + + async def _cleanup_reverse_link( + self, field: Field, source_record_id: str, target_record_id: str + ) -> None: + """Remove source_record_id from the target record's reverse field. + + Mirrors the reverse-update block in ``add_relation_link`` so removal + stays symmetric for bidirectional relations. + """ + reverse_field_id = field.config.get("reverse_field_id") + if not (reverse_field_id and isinstance(reverse_field_id, str)): + return + reverse_field = await self._repo.get_field(reverse_field_id) + if reverse_field is None: + return + target_record = await self._repo.get_record(target_record_id) + if target_record is None: + return + rev_current = target_record.values.get(reverse_field_id) + if isinstance(rev_current, list): + if source_record_id in rev_current: + rev_new = [r for r in rev_current if r != source_record_id] + await self._repo.merge_record_values(target_record_id, {reverse_field_id: rev_new}) + elif rev_current == source_record_id: + await self._repo.merge_record_values(target_record_id, {reverse_field_id: None}) + + async def list_related_records( + self, + relation_field_id: str, + record_id: str, + reverse: bool = False, + ) -> list[str]: + """Get the record IDs related to ``record_id`` via a relation field. + + Args: + relation_field_id: The relation field. + record_id: The source record ID. + reverse: If True, find records that point TO this record + (reverse lookup via junction table or reverse field). + + Returns: + List of related record IDs. + """ + field = await self._repo.get_field(relation_field_id) + if field is None or field.field_type != FieldType.relation: + raise ValueError(f"Field {relation_field_id} is not a relation field") + + relation_type_str = field.config.get("relation_type", "one_to_many") + relation_type = RelationType(relation_type_str) + + if relation_type == RelationType.many_to_many: + if reverse: + links = await self._repo.list_reverse_relation_links( + relation_field_id=relation_field_id, + target_record_id=record_id, + ) + else: + links = await self._repo.list_relation_links( + relation_field_id=relation_field_id, + source_record_id=record_id, + ) + return [ + link.target_record_id if not reverse else link.source_record_id for link in links + ] + + # one_to_one / one_to_many: read from record JSONB + if reverse: + # Find records in the SOURCE table (where the relation field lives) + # that reference this target record via the relation field. + source_table_id = field.table_id + # ponytail: O(n) scan of source table records. Ceiling: add a + # reverse index or use the reverse_field_id for direct lookup. + source_records, _ = await self._repo.list_records(source_table_id, limit=10000) + result: list[str] = [] + for rec in source_records: + val = rec.values.get(relation_field_id) + if val == record_id: + result.append(rec.id) + elif isinstance(val, list) and record_id in val: + result.append(rec.id) + return result + + record = await self._repo.get_record(record_id) + if record is None: + return [] + current = record.values.get(relation_field_id) + if current is None: + return [] + if isinstance(current, list): + return [str(r) for r in current] + return [str(current)] + + async def delete_relation_field(self, field_id: str) -> bool: + """Delete a relation field and cascade-clean junction entries. + + Removes all relation links associated with this field, then deletes + the field itself. For bidirectional relations, the reverse field's + ``reverse_field_id`` config is cleared. + """ + field = await self._repo.get_field(field_id) + if field is None or field.field_type != FieldType.relation: + return False + + # Clean up junction table entries (many_to_many only) + await self._repo.remove_all_relation_links(field_id) + + # Clean up record JSONB values (all relation types) + await self._repo.remove_field_from_records(field.table_id, field_id) + + # Clear reverse field's reference + reverse_field_id = field.config.get("reverse_field_id") + if reverse_field_id and isinstance(reverse_field_id, str): + reverse_field = await self._repo.get_field(reverse_field_id) + if reverse_field is not None: + updated_config = { + k: v for k, v in reverse_field.config.items() if k != "reverse_field_id" + } + updated_config["is_bidirectional"] = False + await self._repo.update_field(reverse_field_id, config=updated_config) + + return await self._repo.delete_field(field_id) + + # ── Lookup & Rollup (U5: R6, R8) ─────────────────────── + + async def create_lookup_field( + self, + table_id: str, + name: str, + relation_field_id: str, + target_field_id: str, + ) -> Field: + """Create a lookup field that pulls a value from a related record. + + Config: ``{relation_field_id, target_field_id}``. + The recalc worker resolves the relation → reads target_field from + the first related record. + + Args: + table_id: Table to create the field on. + name: Field name. + relation_field_id: The relation field on this table to follow. + target_field_id: The field on the target table to read. + """ + config: dict[str, object] = { + "relation_field_id": relation_field_id, + "target_field_id": target_field_id, + } + return await self._repo.create_field( + table_id=table_id, + name=name, + field_type=FieldType.lookup, + config=config, + owner=FieldOwner.user, + ) + + async def create_rollup_field( + self, + table_id: str, + name: str, + relation_field_id: str, + target_field_id: str, + aggregation: str = "sum", + ) -> Field: + """Create a rollup field that aggregates related records' values. + + Config: ``{relation_field_id, target_field_id, aggregation}``. + Supported aggregations: sum, avg, count, count_distinct, min, max, concat. + + Args: + table_id: Table to create the field on. + name: Field name. + relation_field_id: The relation field on this table to follow. + target_field_id: The field on the target table to aggregate. + aggregation: Aggregation function name. + """ + valid_aggs = {"sum", "avg", "count", "count_distinct", "min", "max", "concat"} + if aggregation not in valid_aggs: + raise ValueError(f"Invalid aggregation: {aggregation}. Must be one of {valid_aggs}") + config: dict[str, object] = { + "relation_field_id": relation_field_id, + "target_field_id": target_field_id, + "aggregation": aggregation, + } + return await self._repo.create_field( + table_id=table_id, + name=name, + field_type=FieldType.rollup, + config=config, + owner=FieldOwner.user, + ) + + # ── Cross-table dependency tracking (U6: R10, R11, R12) ─ + + async def register_cross_table_dep( + self, + source_table_id: str, + source_field_id: str, + target_table_id: str, + target_field_id: str, + dep_type: str, + ) -> None: + """Record a cross-table dependency edge (R10). + + When a target field changes, downstream formulas/lookups/rollups in + the source table must be recomputed. This materialized dep table + avoids scanning all formulas on every write. + """ + from agentkit.bitable.models import CrossTableDepType + + try: + dep_type_enum = CrossTableDepType(dep_type) + except ValueError as e: + raise ValueError(f"Invalid dep_type: {dep_type}") from e + + await self._repo.add_cross_table_dep( + source_table_id=source_table_id, + source_field_id=source_field_id, + target_table_id=target_table_id, + target_field_id=target_field_id, + dep_type=dep_type_enum, + ) + + async def trigger_cross_table_recalc( + self, target_table_id: str, target_field_id: str, target_record_ids: list[str] + ) -> int: + """Enqueue recalc for downstream fields when a target field changes (R11). + + Finds all source fields that depend on (target_table, target_field), + then for each source field, finds records in the source table whose + relation field links to any of the changed target records, and + enqueues recalc for those source records. + + Returns the number of recalc tasks enqueued. + """ + deps = await self._repo.list_cross_table_deps_for_target( + target_table_id=target_table_id, + target_field_id=target_field_id, + ) + if not deps: + return 0 + + count = 0 + for dep in deps: + # Find source records whose relation field points to changed target records + source_records = await self._find_source_records_for_targets( + dep.source_field_id, target_record_ids + ) + for record_id in source_records: + await self._repo.enqueue_recalc(dep.source_table_id, record_id, dep.source_field_id) + count += 1 + return count + + async def _find_source_records_for_targets( + self, relation_field_id: str, target_record_ids: list[str] + ) -> list[str]: + """Find source records linked to any of the target records via a relation field. + + For many_to_many: queries the junction table. + For one_to_one/one_to_many: scans source table records (O(n), ponytail ceiling). + """ + if not target_record_ids: + return [] + + field = await self._repo.get_field(relation_field_id) + if field is None or field.field_type != FieldType.relation: + return [] + + relation_type_str = field.config.get("relation_type", "one_to_many") + if relation_type_str == "many_to_many": + # Junction table: find source_record_ids for any of the target_record_ids + source_ids: list[str] = [] + for tid in target_record_ids: + links = await self._repo.list_reverse_relation_links( + relation_field_id=relation_field_id, + target_record_id=tid, + ) + source_ids.extend(link.source_record_id for link in links) + return list(set(source_ids)) + + # one_to_one / one_to_many: scan source table records for matching values + target_set = set(target_record_ids) + src_records, _ = await self._repo.list_records(field.table_id, limit=10000) + result: list[str] = [] + for rec in src_records: + val = rec.values.get(relation_field_id) + if val is None: + continue + if isinstance(val, list): + if any(str(v) in target_set for v in val): + result.append(rec.id) + elif str(val) in target_set: + result.append(rec.id) + return result + + # ── Automation rules (U7: R13-R17) ───────────────────── + + async def create_automation( + self, + table_id: str, + name: str, + trigger_config: dict[str, object], + action_config: dict[str, object], + enabled: bool = True, + generate_webhook: bool = False, + ) -> AutomationRule: + """Create an automation rule. + + Args: + table_id: Table to watch for triggers. + name: Rule name. + trigger_config: Trigger definition (type + optional field_ids). + action_config: Action definition (type + params). + enabled: Whether the rule is active. + generate_webhook: If True, generate a webhook_token for inbound triggering. + """ + webhook_token = None + if generate_webhook: + from agentkit.bitable.automation import generate_webhook_token + + webhook_token = generate_webhook_token() + return await self._repo.create_automation( + table_id=table_id, + name=name, + trigger_config=trigger_config, + action_config=action_config, + enabled=enabled, + webhook_token=webhook_token, + ) + + async def get_automation(self, automation_id: str) -> AutomationRule | None: + return await self._repo.get_automation(automation_id) + + async def list_automations( + self, table_id: str, enabled_only: bool = False + ) -> list[AutomationRule]: + """List automation rules for a table.""" + return await self._repo.list_automations(table_id=table_id, enabled_only=enabled_only) + + async def update_automation( + self, automation_id: str, **kwargs: object + ) -> AutomationRule | None: + return await self._repo.update_automation(automation_id, **kwargs) + + async def delete_automation(self, automation_id: str) -> bool: + return await self._repo.delete_automation(automation_id) + + async def list_automation_logs( + self, automation_id: str, limit: int = 50 + ) -> list[AutomationLog]: + return await self._repo.list_automation_logs(automation_id, limit=limit) + + async def handle_webhook(self, token: str, payload: dict[str, object]) -> bool: + """Handle an inbound webhook — delegates to the automation engine (R16). + + Returns True if a rule matched the token. + """ + if self._automation_engine is None: + return False + return await self._automation_engine.handle_webhook(token, payload) diff --git a/src/agentkit/llm/gateway.py b/src/agentkit/llm/gateway.py index d82a728..6a47b22 100644 --- a/src/agentkit/llm/gateway.py +++ b/src/agentkit/llm/gateway.py @@ -1,5 +1,7 @@ """LLM Gateway - 统一 LLM 调用入口""" +from __future__ import annotations + import asyncio import logging import os diff --git a/src/agentkit/server/routes/bitable.py b/src/agentkit/server/routes/bitable.py index 63072af..e194198 100644 --- a/src/agentkit/server/routes/bitable.py +++ b/src/agentkit/server/routes/bitable.py @@ -823,3 +823,309 @@ async def download_file( if not file_path.exists() or not file_path.is_file(): raise HTTPException(status_code=404, detail="File not found") return FileResponse(file_path, filename=safe_filename) + + +# ── Relation / Lookup / Rollup endpoints (U4/U5, R5-R8) ─── + + +class RelationFieldCreate(BaseModel): + """Request body for creating a relation field.""" + + name: str + target_table_id: str + relation_type: str = "one_to_many" # one_to_one | one_to_many | many_to_many + is_bidirectional: bool = False + reverse_field_name: str | None = None + + +class RelationLinkCreate(BaseModel): + """Request body for linking two records via a relation field.""" + + source_record_id: str + target_record_id: str + + +class LookupFieldCreate(BaseModel): + """Request body for creating a lookup field.""" + + name: str + relation_field_id: str + target_field_id: str + + +class RollupFieldCreate(BaseModel): + """Request body for creating a rollup field.""" + + name: str + relation_field_id: str + target_field_id: str + aggregation: str = "sum" # sum|avg|count|count_distinct|min|max|concat + + +@router.post("/tables/{table_id}/relation-fields", status_code=201) +async def create_relation_field( + table_id: str, + body: RelationFieldCreate, + user: dict = Depends(require_bitable_auth), + service: BitableService = Depends(_get_service), +) -> dict: + """Create a relation field linking this table to a target table (R5).""" + await _check_table_ownership(service, table_id, user) + # Also verify ownership of the target table to prevent IDOR via relation + await _check_table_ownership(service, body.target_table_id, user) + from agentkit.bitable.models import RelationType + + try: + rel_type = RelationType(body.relation_type) + except ValueError as e: + raise HTTPException(status_code=400, detail=f"Invalid relation_type: {e}") from e + + field = await service.create_relation_field( + table_id=table_id, + name=body.name, + target_table_id=body.target_table_id, + relation_type=rel_type, + is_bidirectional=body.is_bidirectional, + reverse_field_name=body.reverse_field_name, + ) + return {"field": field.model_dump(mode="json")} + + +@router.post("/fields/{field_id}/links", status_code=201) +async def add_relation_link( + field_id: str, + body: RelationLinkCreate, + user: dict = Depends(require_bitable_auth), + service: BitableService = Depends(_get_service), +) -> dict: + """Link two records via a relation field (R7).""" + field = await service.get_field(field_id) + if field is None: + raise HTTPException(status_code=404, detail="Field not found") + await _check_table_ownership(service, field.table_id, user) + created = await service.add_relation_link( + relation_field_id=field_id, + source_record_id=body.source_record_id, + target_record_id=body.target_record_id, + ) + return {"created": created} + + +@router.delete("/fields/{field_id}/links") +async def remove_relation_link( + field_id: str, + source_record_id: str, + target_record_id: str, + user: dict = Depends(require_bitable_auth), + service: BitableService = Depends(_get_service), +) -> dict: + """Remove a link between two records (R7).""" + field = await service.get_field(field_id) + if field is None: + raise HTTPException(status_code=404, detail="Field not found") + await _check_table_ownership(service, field.table_id, user) + removed = await service.remove_relation_link( + relation_field_id=field_id, + source_record_id=source_record_id, + target_record_id=target_record_id, + ) + return {"removed": removed} + + +@router.get("/fields/{field_id}/related-records") +async def list_related_records( + field_id: str, + record_id: str, + reverse: bool = False, + user: dict = Depends(require_bitable_auth), + service: BitableService = Depends(_get_service), +) -> dict: + """List records related to ``record_id`` via a relation field (R7).""" + field = await service.get_field(field_id) + if field is None: + raise HTTPException(status_code=404, detail="Field not found") + await _check_table_ownership(service, field.table_id, user) + ids = await service.list_related_records( + relation_field_id=field_id, + record_id=record_id, + reverse=reverse, + ) + return {"record_ids": ids} + + +@router.post("/tables/{table_id}/lookup-fields", status_code=201) +async def create_lookup_field( + table_id: str, + body: LookupFieldCreate, + user: dict = Depends(require_bitable_auth), + service: BitableService = Depends(_get_service), +) -> dict: + """Create a lookup field (R6).""" + await _check_table_ownership(service, table_id, user) + field = await service.create_lookup_field( + table_id=table_id, + name=body.name, + relation_field_id=body.relation_field_id, + target_field_id=body.target_field_id, + ) + return {"field": field.model_dump(mode="json")} + + +@router.post("/tables/{table_id}/rollup-fields", status_code=201) +async def create_rollup_field( + table_id: str, + body: RollupFieldCreate, + user: dict = Depends(require_bitable_auth), + service: BitableService = Depends(_get_service), +) -> dict: + """Create a rollup field (R8).""" + await _check_table_ownership(service, table_id, user) + try: + field = await service.create_rollup_field( + table_id=table_id, + name=body.name, + relation_field_id=body.relation_field_id, + target_field_id=body.target_field_id, + aggregation=body.aggregation, + ) + except ValueError as e: + raise HTTPException(status_code=400, detail=str(e)) from e + return {"field": field.model_dump(mode="json")} + + +# ── Automation endpoints (U7, R13-R17) ──────────────────── + + +class AutomationCreate(BaseModel): + """Request body for creating an automation rule.""" + + name: str + trigger_config: dict[str, Any] + action_config: dict[str, Any] + enabled: bool = True + generate_webhook: bool = False + + +class AutomationUpdate(BaseModel): + """Request body for updating an automation rule. + + Only these fields may be modified — ``table_id`` and ``webhook_token`` + are intentionally excluded to prevent ownership bypass via field injection. + """ + + name: str | None = None + trigger_config: dict[str, Any] | None = None + action_config: dict[str, Any] | None = None + enabled: bool | None = None + + +@router.post("/tables/{table_id}/automations", status_code=201) +async def create_automation( + table_id: str, + body: AutomationCreate, + user: dict = Depends(require_bitable_auth), + service: BitableService = Depends(_get_service), +) -> dict: + """Create an automation rule (R13-R14).""" + await _check_table_ownership(service, table_id, user) + rule = await service.create_automation( + table_id=table_id, + name=body.name, + trigger_config=body.trigger_config, + action_config=body.action_config, + enabled=body.enabled, + generate_webhook=body.generate_webhook, + ) + return {"automation": rule.model_dump(mode="json")} + + +@router.get("/tables/{table_id}/automations") +async def list_automations( + table_id: str, + enabled_only: bool = False, + user: dict = Depends(require_bitable_auth), + service: BitableService = Depends(_get_service), +) -> dict: + """List automation rules for a table.""" + await _check_table_ownership(service, table_id, user) + rules = await service.list_automations(table_id=table_id, enabled_only=enabled_only) + return {"automations": [r.model_dump(mode="json") for r in rules]} + + +@router.get("/automations/{automation_id}") +async def get_automation( + automation_id: str, + user: dict = Depends(require_bitable_auth), + service: BitableService = Depends(_get_service), +) -> dict: + """Get an automation rule by ID.""" + rule = await service.get_automation(automation_id) + if rule is None: + raise HTTPException(status_code=404, detail="Automation not found") + await _check_table_ownership(service, rule.table_id, user) + return {"automation": rule.model_dump(mode="json")} + + +@router.patch("/automations/{automation_id}") +async def update_automation( + automation_id: str, + body: AutomationUpdate, + user: dict = Depends(require_bitable_auth), + service: BitableService = Depends(_get_service), +) -> dict: + """Update an automation rule (e.g. enable/disable, change action).""" + rule = await service.get_automation(automation_id) + if rule is None: + raise HTTPException(status_code=404, detail="Automation not found") + await _check_table_ownership(service, rule.table_id, user) + updates = body.model_dump(exclude_unset=True) + updated = await service.update_automation(automation_id, **updates) + return {"automation": updated.model_dump(mode="json") if updated else None} + + +@router.delete("/automations/{automation_id}", status_code=204) +async def delete_automation( + automation_id: str, + user: dict = Depends(require_bitable_auth), + service: BitableService = Depends(_get_service), +) -> None: + """Delete an automation rule.""" + rule = await service.get_automation(automation_id) + if rule is None: + raise HTTPException(status_code=404, detail="Automation not found") + await _check_table_ownership(service, rule.table_id, user) + await service.delete_automation(automation_id) + + +@router.get("/automations/{automation_id}/logs") +async def list_automation_logs( + automation_id: str, + limit: int = Query(50, ge=1, le=500), + user: dict = Depends(require_bitable_auth), + service: BitableService = Depends(_get_service), +) -> dict: + """List execution logs for an automation rule (R17).""" + rule = await service.get_automation(automation_id) + if rule is None: + raise HTTPException(status_code=404, detail="Automation not found") + await _check_table_ownership(service, rule.table_id, user) + logs = await service.list_automation_logs(automation_id, limit=limit) + return {"logs": [log.model_dump(mode="json") for log in logs]} + + +@router.post("/webhooks/{token}") +async def inbound_webhook( + token: str, + body: dict[str, Any], + service: BitableService = Depends(_get_service), +) -> dict: + """Handle an inbound webhook trigger (R16). + + This endpoint is unauthenticated — the token itself IS the auth. + The token is a 256-bit URL-safe random string from ``secrets.token_urlsafe``. + """ + matched = await service.handle_webhook(token, body) + if not matched: + # 404 to avoid leaking valid tokens (timing-safe: same response shape) + raise HTTPException(status_code=404, detail="Not found") + return {"accepted": True} diff --git a/src/agentkit/tools/bitable_tool.py b/src/agentkit/tools/bitable_tool.py index b6d760e..f9ff37c 100644 --- a/src/agentkit/tools/bitable_tool.py +++ b/src/agentkit/tools/bitable_tool.py @@ -47,7 +47,10 @@ class BitableTool(Tool): "ingest data from Excel files, databases, or API responses, and " "query records. Actions: create_table, import_excel, " "import_database, collect_api, upsert_records, query_records, " - "create_view, update_view, update_field, delete_view." + "create_view, update_view, update_field, delete_view, " + "create_relation, add_relation_link, remove_relation_link, " + "create_lookup_field, create_rollup_field, " + "create_automation, validate_formula." ), input_schema={ "type": "object", @@ -65,6 +68,13 @@ class BitableTool(Tool): "update_view", "update_field", "delete_view", + "create_relation", + "add_relation_link", + "remove_relation_link", + "create_lookup_field", + "create_rollup_field", + "create_automation", + "validate_formula", ], "description": "Bitable operation to perform.", }, @@ -183,6 +193,13 @@ class BitableTool(Tool): "update_view": self._update_view, "update_field": self._update_field, "delete_view": self._delete_view, + "create_relation": self._create_relation, + "add_relation_link": self._add_relation_link, + "remove_relation_link": self._remove_relation_link, + "create_lookup_field": self._create_lookup_field, + "create_rollup_field": self._create_rollup_field, + "create_automation": self._create_automation, + "validate_formula": self._validate_formula, } handler = handlers.get(action) if handler is None: @@ -591,3 +608,131 @@ class BitableTool(Tool): resp.raise_for_status() # 204 No Content has an empty body; report a stable success shape. return {"success": True, "deleted": True} + + # ------------------------------------------------------------------ + # U4/U5: Relation / Lookup / Rollup actions (R5-R8) + # ------------------------------------------------------------------ + + async def _create_relation(self, **kwargs) -> dict[str, object]: + table_id = kwargs.get("table_id") + if not table_id: + return {"success": False, "error": "Missing required field: table_id"} + client = await self._get_client() + resp = await client.post( + f"/tables/{table_id}/relation-fields", + json={ + "name": kwargs.get("name", ""), + "target_table_id": kwargs.get("target_table_id", ""), + "relation_type": kwargs.get("relation_type", "one_to_many"), + "is_bidirectional": kwargs.get("is_bidirectional", False), + "reverse_field_name": kwargs.get("reverse_field_name"), + }, + ) + resp.raise_for_status() + return {"success": True, "field": resp.json()["field"]} + + async def _add_relation_link(self, **kwargs) -> dict[str, object]: + field_id = kwargs.get("field_id") + if not field_id: + return {"success": False, "error": "Missing required field: field_id"} + client = await self._get_client() + resp = await client.post( + f"/fields/{field_id}/links", + json={ + "source_record_id": kwargs.get("source_record_id", ""), + "target_record_id": kwargs.get("target_record_id", ""), + }, + ) + resp.raise_for_status() + return {"success": True, **resp.json()} + + async def _remove_relation_link(self, **kwargs) -> dict[str, object]: + field_id = kwargs.get("field_id") + if not field_id: + return {"success": False, "error": "Missing required field: field_id"} + client = await self._get_client() + resp = await client.delete( + f"/fields/{field_id}/links", + params={ + "source_record_id": kwargs.get("source_record_id", ""), + "target_record_id": kwargs.get("target_record_id", ""), + }, + ) + resp.raise_for_status() + return {"success": True, **resp.json()} + + async def _create_lookup_field(self, **kwargs) -> dict[str, object]: + table_id = kwargs.get("table_id") + if not table_id: + return {"success": False, "error": "Missing required field: table_id"} + client = await self._get_client() + resp = await client.post( + f"/tables/{table_id}/lookup-fields", + json={ + "name": kwargs.get("name", ""), + "relation_field_id": kwargs.get("relation_field_id", ""), + "target_field_id": kwargs.get("target_field_id", ""), + }, + ) + resp.raise_for_status() + return {"success": True, "field": resp.json()["field"]} + + async def _create_rollup_field(self, **kwargs) -> dict[str, object]: + table_id = kwargs.get("table_id") + if not table_id: + return {"success": False, "error": "Missing required field: table_id"} + client = await self._get_client() + resp = await client.post( + f"/tables/{table_id}/rollup-fields", + json={ + "name": kwargs.get("name", ""), + "relation_field_id": kwargs.get("relation_field_id", ""), + "target_field_id": kwargs.get("target_field_id", ""), + "aggregation": kwargs.get("aggregation", "sum"), + }, + ) + resp.raise_for_status() + return {"success": True, "field": resp.json()["field"]} + + # ------------------------------------------------------------------ + # U7: Automation actions (R13-R17) + # ------------------------------------------------------------------ + + async def _create_automation(self, **kwargs) -> dict[str, object]: + table_id = kwargs.get("table_id") + if not table_id: + return {"success": False, "error": "Missing required field: table_id"} + client = await self._get_client() + resp = await client.post( + f"/tables/{table_id}/automations", + json={ + "name": kwargs.get("name", ""), + "trigger_config": kwargs.get("trigger_config", {}), + "action_config": kwargs.get("action_config", {}), + "enabled": kwargs.get("enabled", True), + "generate_webhook": kwargs.get("generate_webhook", False), + }, + ) + resp.raise_for_status() + return {"success": True, "automation": resp.json()["automation"]} + + # ------------------------------------------------------------------ + # U3: Formula validation (agent generates formula, validates before save) + # ------------------------------------------------------------------ + + async def _validate_formula(self, **kwargs) -> dict[str, object]: + """Validate a formula expression via the server's parser. + + Agent uses this to check a generated formula before persisting it + as a field config — avoids saving broken formulas. + """ + formula_expr = kwargs.get("formula_expr") + if not formula_expr: + return {"success": False, "error": "Missing required field: formula_expr"} + client = await self._get_client() + resp = await client.post( + "/fields/validate-formula", + json={"formula_expr": formula_expr}, + ) + resp.raise_for_status() + return {"success": True, **resp.json()} diff --git a/tests/unit/bitable/test_ae1_cross_table_recalc.py b/tests/unit/bitable/test_ae1_cross_table_recalc.py new file mode 100644 index 0000000..097175f --- /dev/null +++ b/tests/unit/bitable/test_ae1_cross_table_recalc.py @@ -0,0 +1,227 @@ +"""AE1 acceptance test: cross-table rollup recalculation end-to-end. + +Validates the core v2 scenario from the plan: + Orders.amount changes -> Customers.total_amount (rollup SUM) recalculates. + +This test requires PostgreSQL (marked with @pytest.mark.postgres). +It exercises the full stack: BitableService -> repository -> recalc_worker +-> FormulaEngine -> cross-table relation resolution -> rollup aggregation. + +No mocks — real PG, real async recalc queue, real cross-table deps. +""" + +from __future__ import annotations + +import pytest +import pytest_asyncio + +from agentkit.bitable.models import FieldOwner, FieldType + +pytestmark = [pytest.mark.postgres, pytest.mark.asyncio] + + +@pytest_asyncio.fixture +async def ae1_setup(bitable_service): + """Set up Customers + Orders tables with a 1:N relation and rollup field. + + Also wires a RecalcWorker into the service so rollup/formula fields + actually recalc when records change. Returns a dict with IDs. + """ + svc = bitable_service + + # Wire a recalc worker (bitable_service fixture doesn't include one) + from agentkit.bitable.recalc_worker import RecalcWorker + + worker = RecalcWorker(svc._db, svc) + svc.set_recalc_worker(worker) + + # 1. Create Customers table + customers_table = await svc.create_table(name="Customers") + cust_id_field = await svc._repo.create_field( + table_id=customers_table.id, + name="cust_id", + field_type=FieldType.text, + config={}, + owner=FieldOwner.user, + ) + cust_name_field = await svc._repo.create_field( + table_id=customers_table.id, + name="name", + field_type=FieldType.text, + config={}, + owner=FieldOwner.user, + ) + await svc._repo.update_table(customers_table.id, primary_key_field_id=cust_id_field.id) + + # 2. Create Orders table + orders_table = await svc.create_table(name="Orders") + ord_id_field = await svc._repo.create_field( + table_id=orders_table.id, + name="ord_id", + field_type=FieldType.text, + config={}, + owner=FieldOwner.user, + ) + ord_amount_field = await svc._repo.create_field( + table_id=orders_table.id, + name="amount", + field_type=FieldType.number, + config={}, + owner=FieldOwner.user, + ) + await svc._repo.update_table(orders_table.id, primary_key_field_id=ord_id_field.id) + + # 3. Create 1:N relation field on Orders -> Customers + relation_field = await svc.create_relation_field( + table_id=orders_table.id, + name="customer", + target_table_id=customers_table.id, + relation_type="one_to_many", + ) + + # 4. Create rollup field on Customers: SUM(Orders.amount via relation) + rollup_field = await svc.create_rollup_field( + table_id=customers_table.id, + name="total_amount", + relation_field_id=relation_field.id, + target_field_id=ord_amount_field.id, + aggregation="sum", + ) + + return { + "customers_table": customers_table, + "orders_table": orders_table, + "cust_id_field": cust_id_field, + "cust_name_field": cust_name_field, + "ord_id_field": ord_id_field, + "ord_amount_field": ord_amount_field, + "relation_field": relation_field, + "rollup_field": rollup_field, + } + + +async def test_ae1_rollup_sum_recalc_on_insert(bitable_service, ae1_setup) -> None: + """AE1: Inserting Orders updates Customers.total_amount via rollup SUM.""" + svc = bitable_service + cust_fid = ae1_setup["cust_id_field"].id + ord_fid = ae1_setup["ord_id_field"].id + amt_fid = ae1_setup["ord_amount_field"].id + rel_fid = ae1_setup["relation_field"].id + rollup_fid = ae1_setup["rollup_field"].id + + # Create a customer + cust = await svc.create_record( + table_id=ae1_setup["customers_table"].id, + values={cust_fid: "c1"}, + ) + + # Create 3 orders linked to the customer + for i in range(3): + order = await svc.create_record( + table_id=ae1_setup["orders_table"].id, + values={ord_fid: f"o{i}", amt_fid: (i + 1) * 100}, + ) + await svc.add_relation_link( + relation_field_id=rel_fid, + source_record_id=order.id, + target_record_id=cust.id, + ) + + # Process recalc queue — claim and process all pending tasks + worker = svc._recalc_worker + if worker is None: + pytest.skip("recalc_worker not configured") + tasks = await worker._repo.claim_recalc_tasks(limit=50) + for task in tasks: + await worker.process_task(task) + + # Read the customer record — total_amount should be 100+200+300 = 600 + updated_cust = await svc._repo.get_record(cust.id) + assert updated_cust is not None + total = updated_cust.values.get(rollup_fid) + assert total == 600, f"Expected total_amount=600, got {total}" + + +async def test_ae1_rollup_updates_on_order_change(bitable_service, ae1_setup) -> None: + """AE1: Updating an order's amount triggers rollup recalc on customer.""" + svc = bitable_service + cust_fid = ae1_setup["cust_id_field"].id + ord_fid = ae1_setup["ord_id_field"].id + amt_fid = ae1_setup["ord_amount_field"].id + rel_fid = ae1_setup["relation_field"].id + rollup_fid = ae1_setup["rollup_field"].id + + cust = await svc.create_record( + table_id=ae1_setup["customers_table"].id, + values={cust_fid: "c1"}, + ) + order = await svc.create_record( + table_id=ae1_setup["orders_table"].id, + values={ord_fid: "o1", amt_fid: 100}, + ) + await svc.add_relation_link(rel_fid, order.id, cust.id) + + # Drain recalc queue + worker = svc._recalc_worker + if worker is None: + pytest.skip("recalc_worker not configured") + tasks = await worker._repo.claim_recalc_tasks(limit=50) + for task in tasks: + await worker.process_task(task) + + # Verify initial sum + rec = await svc._repo.get_record(cust.id) + assert rec.values.get(rollup_fid) == 100 + + # Update the order amount + await svc.update_record_values(order.id, {amt_fid: 500}) + + # Drain again + tasks = await worker._repo.claim_recalc_tasks(limit=50) + for task in tasks: + await worker.process_task(task) + + # Customer's total should now be 500 + rec = await svc._repo.get_record(cust.id) + assert rec.values.get(rollup_fid) == 500, f"Expected 500 after update, got {rec.values.get(rollup_fid)}" + + +async def test_ae1_rollup_avg_aggregation(bitable_service, ae1_setup) -> None: + """AE1: Rollup AVG aggregation across related records.""" + svc = bitable_service + cust_fid = ae1_setup["cust_id_field"].id + ord_fid = ae1_setup["ord_id_field"].id + amt_fid = ae1_setup["ord_amount_field"].id + rel_fid = ae1_setup["relation_field"].id + + # Change aggregation to avg — recreate rollup field + avg_rollup = await svc.create_rollup_field( + table_id=ae1_setup["customers_table"].id, + name="avg_amount", + relation_field_id=rel_fid, + target_field_id=amt_fid, + aggregation="avg", + ) + + cust = await svc.create_record( + table_id=ae1_setup["customers_table"].id, + values={cust_fid: "c1"}, + ) + for amt in [100, 200, 300]: + order = await svc.create_record( + table_id=ae1_setup["orders_table"].id, + values={ord_fid: f"o{amt}", amt_fid: amt}, + ) + await svc.add_relation_link(rel_fid, order.id, cust.id) + + # Drain recalc queue + worker = svc._recalc_worker + if worker is None: + pytest.skip("recalc_worker not configured") + tasks = await worker._repo.claim_recalc_tasks(limit=50) + for task in tasks: + await worker.process_task(task) + + rec = await svc._repo.get_record(cust.id) + avg = rec.values.get(avg_rollup.id) + assert avg == 200, f"Expected avg=200, got {avg}" diff --git a/tests/unit/bitable/test_automation.py b/tests/unit/bitable/test_automation.py new file mode 100644 index 0000000..b01cc82 --- /dev/null +++ b/tests/unit/bitable/test_automation.py @@ -0,0 +1,436 @@ +"""Unit tests for the automation engine (U7, R13-R17). + +Tests the trigger evaluation, action dispatch, retry logic, and webhook +handling. These tests do NOT require PostgreSQL — they mock the repo/service +interfaces to test the engine logic in isolation. +""" + +from __future__ import annotations + +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from agentkit.bitable.automation import AutomationEngine, generate_webhook_token +from agentkit.bitable.models import ( + AutomationRule, + AutomationStatus, + AutomationTriggerType, +) + + +def _make_rule( + rule_id: str = "a1", + table_id: str = "t1", + trigger_type: str = "record_created", + action_type: str = "send_webhook", + enabled: bool = True, + webhook_token: str | None = None, + field_ids: list[str] | None = None, + action_config: dict | None = None, +) -> AutomationRule: + """Build an AutomationRule for testing.""" + trig_cfg: dict = {"type": trigger_type} + if field_ids: + trig_cfg["field_ids"] = field_ids + act_cfg = action_config or {"type": action_type, "url": "https://example.com/hook"} + return AutomationRule( + id=rule_id, + table_id=table_id, + name=f"Rule {rule_id}", + trigger_config=trig_cfg, + action_config=act_cfg, + enabled=enabled, + webhook_token=webhook_token, + ) + + +def _make_mock_service() -> tuple[MagicMock, MagicMock]: + """Create a mock service with a mock repo for the engine.""" + service = MagicMock() + repo = MagicMock() + service.repo = repo + # Async methods on repo + repo.list_automations_for_trigger = AsyncMock(return_value=[]) + repo.get_automation_by_webhook_token = AsyncMock(return_value=None) + repo.create_automation_log = AsyncMock(return_value="log1") + repo.update_automation_log = AsyncMock(return_value=None) + # Async methods on service + service.create_record = AsyncMock(return_value=MagicMock(id="r_new")) + service.update_record_values = AsyncMock(return_value=None) + return service, repo + + +# --------------------------------------------------------------------------- +# Trigger evaluation (R13) +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_fire_trigger_matches_record_created() -> None: + """record_created trigger fires matching rules.""" + service, repo = _make_mock_service() + rule = _make_rule(trigger_type="record_created") + repo.list_automations_for_trigger = AsyncMock(return_value=[rule]) + + engine = AutomationEngine(service) + matched = await engine.fire_trigger( + table_id="t1", + trigger_type=AutomationTriggerType.record_created, + record_id="r1", + record_values={"f1": "value"}, + ) + assert matched == 1 + repo.list_automations_for_trigger.assert_awaited_once_with( + table_id="t1", trigger_type="record_created" + ) + + +@pytest.mark.asyncio +async def test_fire_trigger_skips_disabled_rules() -> None: + """Disabled rules don't fire.""" + service, repo = _make_mock_service() + rule = _make_rule(enabled=False) + repo.list_automations_for_trigger = AsyncMock(return_value=[rule]) + + engine = AutomationEngine(service) + matched = await engine.fire_trigger( + table_id="t1", + trigger_type=AutomationTriggerType.record_created, + ) + assert matched == 0 + + +@pytest.mark.asyncio +async def test_fire_trigger_record_updated_field_filter() -> None: + """record_updated with field_ids filter only fires on overlap.""" + service, repo = _make_mock_service() + rule_watching_f1 = _make_rule(trigger_type="record_updated", field_ids=["f1"]) + repo.list_automations_for_trigger = AsyncMock(return_value=[rule_watching_f1]) + + engine = AutomationEngine(service) + # Changed f2 only — rule watches f1 → no match + matched = await engine.fire_trigger( + table_id="t1", + trigger_type=AutomationTriggerType.record_updated, + changed_field_ids=["f2"], + ) + assert matched == 0 + + # Changed f1 → match + matched = await engine.fire_trigger( + table_id="t1", + trigger_type=AutomationTriggerType.record_updated, + changed_field_ids=["f1", "f2"], + ) + assert matched == 1 + + +@pytest.mark.asyncio +async def test_fire_trigger_no_field_filter_matches_any() -> None: + """record_updated without field_ids fires on any update.""" + service, repo = _make_mock_service() + rule_no_filter = _make_rule(trigger_type="record_updated", field_ids=[]) + repo.list_automations_for_trigger = AsyncMock(return_value=[rule_no_filter]) + + engine = AutomationEngine(service) + matched = await engine.fire_trigger( + table_id="t1", + trigger_type=AutomationTriggerType.record_updated, + changed_field_ids=["f_anything"], + ) + assert matched == 1 + + +# --------------------------------------------------------------------------- +# Webhook inbound (R16) +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_handle_webhook_matches_token() -> None: + """Inbound webhook fires the rule with matching token.""" + service, repo = _make_mock_service() + rule = _make_rule(webhook_token="whk_test123") + repo.get_automation_by_webhook_token = AsyncMock(return_value=rule) + + engine = AutomationEngine(service) + matched = await engine.handle_webhook("whk_test123", {"event": "test"}) + assert matched is True + + +@pytest.mark.asyncio +async def test_handle_webhook_unknown_token() -> None: + """Unknown webhook token returns False (no match).""" + service, repo = _make_mock_service() + repo.get_automation_by_webhook_token = AsyncMock(return_value=None) + + engine = AutomationEngine(service) + matched = await engine.handle_webhook("whk_unknown", {"event": "test"}) + assert matched is False + + +@pytest.mark.asyncio +async def test_handle_webhook_disabled_rule() -> None: + """Disabled rules don't fire via webhook.""" + service, repo = _make_mock_service() + rule = _make_rule(enabled=False, webhook_token="whk_test") + repo.get_automation_by_webhook_token = AsyncMock(return_value=rule) + + engine = AutomationEngine(service) + matched = await engine.handle_webhook("whk_test", {}) + assert matched is False + + +# --------------------------------------------------------------------------- +# Action execution (R14) +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_action_create_record_resolves_trigger_placeholders() -> None: + """create_record action resolves @trigger.X placeholders.""" + service, repo = _make_mock_service() + engine = AutomationEngine(service) + + job = { + "automation_id": "a1", + "table_id": "t1", + "trigger_type": "record_created", + "record_id": "r_trigger", + "record_values": {"f_name": "Alice"}, + "action_config": { + "type": "create_record", + "target_table_id": "t2", + "values": { + "f_target_name": "@trigger.f_name", + "f_static": "static_value", + }, + }, + } + await engine._execute_action(job) + service.create_record.assert_awaited_once() + args, kwargs = service.create_record.call_args + assert args[0] == "t2" # target_table_id + resolved = args[1] + assert resolved["f_target_name"] == "Alice" + assert resolved["f_static"] == "static_value" + + +@pytest.mark.asyncio +async def test_action_update_record_defaults_to_trigger_record() -> None: + """update_record defaults target_record_id to the trigger record.""" + service, repo = _make_mock_service() + engine = AutomationEngine(service) + + job = { + "automation_id": "a1", + "table_id": "t1", + "trigger_type": "record_updated", + "record_id": "r_trigger", + "record_values": {"f_status": "done"}, + "action_config": { + "type": "update_record", + "values": {"f_log": "completed"}, + }, + } + await engine._execute_action(job) + service.update_record_values.assert_awaited_once_with("r_trigger", {"f_log": "completed"}) + + +@pytest.mark.asyncio +async def test_action_send_webhook_posts_payload(httpx_mock, monkeypatch) -> None: + """send_webhook POSTs JSON to the configured URL.""" + # ponytail: uses httpx_mock fixture from pytest-httpx if available; + # otherwise this test is skipped. + pytest.importorskip("pytest_httpx") + # Bypass SSRF host check — this test validates the POST behavior, not the guard. + monkeypatch.setattr("agentkit.bitable.automation._assert_safe_host", lambda _host: None) + httpx_mock.add_response(url="https://example.com/hook", status_code=200) + + service, repo = _make_mock_service() + engine = AutomationEngine(service) + + job = { + "automation_id": "a1", + "table_id": "t1", + "trigger_type": "record_created", + "record_id": "r1", + "record_values": {"f1": "val"}, + "action_config": { + "type": "send_webhook", + "url": "https://example.com/hook", + }, + } + await engine._execute_action(job) + + +@pytest.mark.asyncio +async def test_action_unknown_type_raises() -> None: + """Unknown action type raises ValueError.""" + service, repo = _make_mock_service() + engine = AutomationEngine(service) + + job = { + "automation_id": "a1", + "table_id": "t1", + "trigger_type": "record_created", + "record_id": "r1", + "record_values": {}, + "action_config": {"type": "invalid_action"}, + } + with pytest.raises(ValueError, match="Unknown action type"): + await engine._execute_action(job) + + +@pytest.mark.asyncio +async def test_webhook_ssrf_guard_blocks_loopback() -> None: + """send_webhook rejects loopback URLs (SSRF protection, P1).""" + service, repo = _make_mock_service() + engine = AutomationEngine(service) + + job = { + "automation_id": "a1", + "table_id": "t1", + "trigger_type": "record_created", + "record_id": "r1", + "record_values": {"f1": "val"}, + "action_config": { + "type": "send_webhook", + "url": "http://127.0.0.1:8080/internal", + }, + } + with pytest.raises(ValueError, match="Disallowed|unsafe|private|loopback"): + await engine._execute_action(job) + + +@pytest.mark.asyncio +async def test_webhook_ssrf_guard_blocks_private_ip() -> None: + """send_webhook rejects RFC1918 private IPs (SSRF protection, P1).""" + service, repo = _make_mock_service() + engine = AutomationEngine(service) + + job = { + "automation_id": "a2", + "table_id": "t1", + "trigger_type": "record_created", + "record_id": "r1", + "record_values": {}, + "action_config": { + "type": "send_webhook", + "url": "http://10.0.0.1/admin", + }, + } + with pytest.raises(ValueError, match="Disallowed|unsafe|private"): + await engine._execute_action(job) + + +@pytest.mark.asyncio +async def test_webhook_ssrf_guard_blocks_non_http_scheme() -> None: + """send_webhook rejects non-HTTP(S) schemes (file://, gopher://, etc.).""" + service, repo = _make_mock_service() + engine = AutomationEngine(service) + + job = { + "automation_id": "a3", + "table_id": "t1", + "trigger_type": "record_created", + "record_id": "r1", + "record_values": {}, + "action_config": { + "type": "send_webhook", + "url": "file:///etc/passwd", + }, + } + with pytest.raises(ValueError, match="scheme"): + await engine._execute_action(job) + + +# --------------------------------------------------------------------------- +# Retry logic (R15) +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_retry_succeeds_on_second_attempt(monkeypatch) -> None: + """Action that fails once then succeeds is logged correctly.""" + service, repo = _make_mock_service() + engine = AutomationEngine(service) + + # Make _execute_action fail once, then succeed + call_count = [0] + + async def _flaky(job): + call_count[0] += 1 + if call_count[0] == 1: + raise RuntimeError("transient failure") + + monkeypatch.setattr(engine, "_execute_action", _flaky) + # Speed up retries + monkeypatch.setattr("agentkit.bitable.automation._RETRY_DELAYS", [0.0, 0.0, 0.0]) + + job = { + "automation_id": "a1", + "table_id": "t1", + "trigger_type": "record_created", + "record_id": "r1", + "record_values": {}, + "action_config": {"type": "send_webhook", "url": "x"}, + } + await engine._execute_with_retry(job) + assert call_count[0] == 2 # Failed once, succeeded on second + # Two log entries created (one per attempt) + assert repo.create_automation_log.await_count == 2 + # Final log status is success + repo.update_automation_log.assert_awaited_with( + log_id="log1", + status=AutomationStatus.success, + completed_at=repo.update_automation_log.await_args.kwargs["completed_at"], + ) + + +@pytest.mark.asyncio +async def test_retry_exhausted_after_3_attempts(monkeypatch) -> None: + """Action that always fails is retried 3 times then logged as error.""" + service, repo = _make_mock_service() + engine = AutomationEngine(service) + + async def _always_fail(job): + raise RuntimeError("permanent failure") + + monkeypatch.setattr(engine, "_execute_action", _always_fail) + monkeypatch.setattr("agentkit.bitable.automation._RETRY_DELAYS", [0.0, 0.0, 0.0]) + + job = { + "automation_id": "a1", + "table_id": "t1", + "trigger_type": "record_created", + "record_id": "r1", + "record_values": {}, + "action_config": {"type": "send_webhook", "url": "x"}, + } + await engine._execute_with_retry(job) + assert repo.create_automation_log.await_count == 3 # 3 attempts + # Last log status is error + last_call = repo.update_automation_log.await_args_list[-1] + assert last_call.kwargs["status"] == AutomationStatus.error + assert "permanent failure" in last_call.kwargs["error_message"] + + +# --------------------------------------------------------------------------- +# Webhook token generation (R16) +# --------------------------------------------------------------------------- + + +def test_generate_webhook_token_format() -> None: + """Webhook tokens have the whk_ prefix and sufficient entropy.""" + token = generate_webhook_token() + assert token.startswith("whk_") + # token_urlsafe(32) produces ~43 chars; with prefix, ~47 + assert len(token) > 40 + + +def test_generate_webhook_token_uniqueness() -> None: + """Two generated tokens are different (non-deterministic).""" + tokens = {generate_webhook_token() for _ in range(10)} + assert len(tokens) == 10 diff --git a/tests/unit/bitable/test_bitable_tool.py b/tests/unit/bitable/test_bitable_tool.py index 8b6b75b..0c27ea8 100644 --- a/tests/unit/bitable/test_bitable_tool.py +++ b/tests/unit/bitable/test_bitable_tool.py @@ -491,17 +491,22 @@ def test_transform_records_missing_keys() -> None: # --------------------------------------------------------------------------- -def test_action_enum_has_10_actions() -> None: - """input_schema.action.enum lists all 10 actions (6 original + 4 new).""" +def test_action_enum_has_17_actions() -> None: + """input_schema.action.enum lists all 17 actions (10 original + 7 U4/U5/U7/U3).""" tool = BitableTool(base_url="http://test/api/v1/bitable") actions = tool.input_schema["properties"]["action"]["enum"] - assert len(actions) == 10 - for new_action in ("create_view", "update_view", "update_field", "delete_view"): + assert len(actions) == 17 + for new_action in ( + "create_view", "update_view", "update_field", "delete_view", + "create_relation", "add_relation_link", "remove_relation_link", + "create_lookup_field", "create_rollup_field", + "create_automation", "validate_formula", + ): assert new_action in actions -def test_execute_handlers_dict_has_10_actions() -> None: - """execute() handlers dict contains all 10 action keys (KTD10).""" +def test_execute_handlers_dict_has_17_actions() -> None: + """execute() handlers dict contains all 17 action keys (KTD10).""" import re src = open( @@ -510,8 +515,13 @@ def test_execute_handlers_dict_has_10_actions() -> None: ).read() handlers_match = re.search(r"handlers\s*=\s*\{([^}]*)\}", src, re.DOTALL) handler_keys = re.findall(r'"([a-z_]+)":\s*self\._', handlers_match.group(1)) - assert len(handler_keys) == 10 - for new_action in ("create_view", "update_view", "update_field", "delete_view"): + assert len(handler_keys) == 17 + for new_action in ( + "create_view", "update_view", "update_field", "delete_view", + "create_relation", "add_relation_link", "remove_relation_link", + "create_lookup_field", "create_rollup_field", + "create_automation", "validate_formula", + ): assert new_action in handler_keys diff --git a/tests/unit/bitable/test_db.py b/tests/unit/bitable/test_db.py index 19026dc..3b374ef 100644 --- a/tests/unit/bitable/test_db.py +++ b/tests/unit/bitable/test_db.py @@ -15,7 +15,7 @@ import pytest async def test_init_creates_schema_and_all_tables(bitable_db) -> None: - """init creates the bitable schema and all 7 tables (V2 adds bitable_files).""" + """init creates the bitable schema and all 11 tables (V3 adds 4 new tables).""" from sqlalchemy import text async with bitable_db.engine.begin() as conn: @@ -27,7 +27,7 @@ async def test_init_creates_schema_and_all_tables(bitable_db) -> None: ) assert result.fetchone() is not None - # All 7 tables present (V2 adds bitable_files) + # All 11 tables present (V2 adds bitable_files; V3 adds 4 new tables) result = await conn.execute( text( "SELECT table_name FROM information_schema.tables " @@ -36,16 +36,85 @@ async def test_init_creates_schema_and_all_tables(bitable_db) -> None: ) tables = {row[0] for row in result.fetchall()} assert tables == { + "bitable_automation_logs", + "bitable_automations", + "bitable_cross_table_deps", "bitable_fields", "bitable_files", "bitable_meta", "bitable_records", "bitable_recalc_queue", + "bitable_relation_links", "bitable_tables", "bitable_views", } +async def test_v3_migration_adds_is_cross_table_column(bitable_db) -> None: + """V3 migration adds is_cross_table boolean column on recalc_queue.""" + from sqlalchemy import text + + async with bitable_db.engine.begin() as conn: + result = await conn.execute( + text( + "SELECT column_name, data_type FROM information_schema.columns " + "WHERE table_schema = 'bitable' AND table_name = 'bitable_recalc_queue' " + "AND column_name = 'is_cross_table'" + ) + ) + row = result.fetchone() + assert row is not None + assert row[1] == "boolean" + + +async def test_v3_migration_creates_relation_links_indexes(bitable_db) -> None: + """V3 creates the two covering indexes on bitable_relation_links.""" + from sqlalchemy import text + + async with bitable_db.engine.begin() as conn: + result = await conn.execute( + text( + "SELECT indexname FROM pg_indexes " + "WHERE schemaname = 'bitable' AND tablename = 'bitable_relation_links'" + ) + ) + indexes = {row[0] for row in result.fetchall()} + assert "ix_relation_links_field_source" in indexes + assert "ix_relation_links_field_target" in indexes + + +async def test_v3_migration_creates_cross_table_deps_unique_constraint(bitable_db) -> None: + """V3 enforces (source_field_id, target_field_id) uniqueness on cross_table_deps.""" + from sqlalchemy import text + + async with bitable_db.engine.begin() as conn: + result = await conn.execute( + text( + "SELECT conname FROM pg_constraint " + "WHERE conrelid = 'bitable.bitable_cross_table_deps'::regclass " + "AND conname = 'uq_cross_table_dep_source_target'" + ) + ) + assert result.fetchone() is not None + + +async def test_v3_migration_creates_automation_webhook_token_partial_index(bitable_db) -> None: + """V3 creates a partial index on webhook_token (WHERE NOT NULL).""" + from sqlalchemy import text + + async with bitable_db.engine.begin() as conn: + result = await conn.execute( + text( + "SELECT indexdef FROM pg_indexes " + "WHERE schemaname = 'bitable' AND indexname = 'ix_automations_webhook_token'" + ) + ) + defn = result.fetchone() + assert defn is not None + # Partial index must include WHERE webhook_token IS NOT NULL + assert "webhook_token IS NOT NULL" in defn[0] + + async def test_init_is_idempotent(bitable_db) -> None: """Calling init() twice does not raise and keeps schema intact.""" # bitable_db fixture already called init(); call again @@ -243,3 +312,218 @@ async def test_bitable_db_without_url_raises() -> None: for key, val in zip(("DATABASE_URL", "AGENTKIT_DATABASE_URL"), saved): if val is not None: os.environ[key] = val + + +# --------------------------------------------------------------------------- +# V3 repository CRUD: relation_links / cross_table_deps / automations / logs +# --------------------------------------------------------------------------- + + +async def test_relation_links_crud(bitable_db) -> None: + """RelationLink CRUD: add, list forward, list reverse, remove.""" + from agentkit.bitable.repository import BitableRepository + + repo = BitableRepository(bitable_db) + table_a = await repo.create_table(name="Students") + table_b = await repo.create_table(name="Courses") + # Relation field lives on table_a (Students). + rel_field = await repo.create_field( + table_id=table_a.id, name="Enrolled Courses", field_type="lookup" + ) + rec_a1 = await repo.create_record(table_id=table_a.id) + rec_a2 = await repo.create_record(table_id=table_a.id) + rec_b1 = await repo.create_record(table_id=table_b.id) + rec_b2 = await repo.create_record(table_id=table_b.id) + + # Add links: a1 → b1, a1 → b2, a2 → b1 + await repo.add_relation_link(rel_field.id, rec_a1.id, rec_b1.id) + await repo.add_relation_link(rel_field.id, rec_a1.id, rec_b2.id) + await repo.add_relation_link(rel_field.id, rec_a2.id, rec_b1.id) + + # Forward: a1 → [b1, b2] + forward = await repo.list_relation_links(rel_field.id, rec_a1.id) + assert set(forward) == {rec_b1.id, rec_b2.id} + + # Reverse: which sources point at b1? → [a1, a2] + reverse = await repo.list_reverse_relation_links(rel_field.id, rec_b1.id) + assert set(reverse) == {rec_a1.id, rec_a2.id} + + # Remove a1 → b1 + removed = await repo.remove_relation_link(rel_field.id, rec_a1.id, rec_b1.id) + assert removed is True + forward_after = await repo.list_relation_links(rel_field.id, rec_a1.id) + assert forward_after == [rec_b2.id] + + # remove_all for a source record cascades + await repo.add_relation_link(rel_field.id, rec_a1.id, rec_b1.id) + count = await repo.remove_all_relation_links(rel_field.id, source_record_id=rec_a1.id) + assert count >= 1 + assert await repo.list_relation_links(rel_field.id, rec_a1.id) == [] + + # remove_all for the whole field + count = await repo.remove_all_relation_links(rel_field.id) + assert count >= 1 # a2→b1 still there + + +async def test_cross_table_deps_crud(bitable_db) -> None: + """CrossTableDep CRUD: add (dedup), find dependents, remove by source.""" + from agentkit.bitable.models import CrossTableDepType + from agentkit.bitable.repository import BitableRepository + + repo = BitableRepository(bitable_db) + table_a = await repo.create_table(name="Orders") + table_b = await repo.create_table(name="Customers") + source_field = await repo.create_field( + table_id=table_b.id, name="total_amount", field_type="rollup" + ) + target_field = await repo.create_field(table_id=table_a.id, name="amount", field_type="number") + + # Add dep edge: source_field (in B) depends on target_field (in A) + dep = await repo.add_cross_table_dep( + source_table_id=table_b.id, + source_field_id=source_field.id, + target_table_id=table_a.id, + target_field_id=target_field.id, + dep_type=CrossTableDepType.rollup, + ) + assert dep is not None + + # Duplicate insert is a no-op + dup = await repo.add_cross_table_dep( + source_table_id=table_b.id, + source_field_id=source_field.id, + target_table_id=table_a.id, + target_field_id=target_field.id, + dep_type=CrossTableDepType.rollup, + ) + assert dup is None + + # Find dependents of table_a / target_field + deps = await repo.find_cross_table_dependents(table_a.id, target_field.id) + assert len(deps) == 1 + assert deps[0].source_field_id == source_field.id + assert deps[0].dep_type == CrossTableDepType.rollup + + # Remove by source + removed = await repo.remove_cross_table_deps_for_source(source_field.id) + assert removed == 1 + assert await repo.find_cross_table_dependents(table_a.id, target_field.id) == [] + + +async def test_automations_crud(bitable_db) -> None: + """AutomationRule CRUD: create, get, list, update, delete (cascade logs).""" + from agentkit.bitable.repository import BitableRepository + + repo = BitableRepository(bitable_db) + table = await repo.create_table(name="Tasks") + + rule = await repo.create_automation( + table_id=table.id, + name="Notify on done", + trigger_config={"type": "record_updated", "field_ids": ["f_status"]}, + action_config={"type": "send_webhook", "url": "https://example.com/hook"}, + ) + assert rule.enabled is True + assert rule.webhook_token is None + + fetched = await repo.get_automation(rule.id) + assert fetched is not None and fetched.id == rule.id + + rules = await repo.list_automations(table.id) + assert len(rules) == 1 + + # list_automations_for_trigger matches on JSONB trigger_config->>'type' + matching = await repo.list_automations_for_trigger(table.id, "record_updated") + assert len(matching) == 1 + non_matching = await repo.list_automations_for_trigger(table.id, "record_created") + assert non_matching == [] + + # update (disable) + updated = await repo.update_automation(rule.id, enabled=False) + assert updated is not None and updated.enabled is False + + # webhook token lookup + rule_with_token = await repo.create_automation( + table_id=table.id, + name="Inbound webhook", + trigger_config={"type": "record_created"}, + action_config={"type": "create_record"}, + webhook_token="tok_secret_123", + ) + found = await repo.get_automation_by_webhook_token("tok_secret_123") + assert found is not None and found.id == rule_with_token.id + + # delete cascades logs (no logs here, but exercise the path) + deleted = await repo.delete_automation(rule.id) + assert deleted is True + assert await repo.get_automation(rule.id) is None + + +async def test_automation_logs_crud(bitable_db) -> None: + """AutomationLog CRUD: create, update, list.""" + from agentkit.bitable.models import AutomationStatus + from agentkit.bitable.repository import BitableRepository + + repo = BitableRepository(bitable_db) + table = await repo.create_table(name="Tasks") + rule = await repo.create_automation( + table_id=table.id, + name="Auto", + trigger_config={"type": "record_created"}, + action_config={"type": "send_message"}, + ) + + log = await repo.create_automation_log( + automation_id=rule.id, + trigger_record_id="rec_1", + status=AutomationStatus.retrying, + attempt=1, + ) + assert log.status == AutomationStatus.retrying + + await repo.update_automation_log( + log.id, AutomationStatus.success, error_message=None + ) + + logs = await repo.list_automation_logs(rule.id) + assert len(logs) == 1 + assert logs[0].status == AutomationStatus.success + assert logs[0].completed_at is not None + + +async def test_enqueue_recalc_with_is_cross_table(bitable_db) -> None: + """enqueue_recalc sets is_cross_table flag when requested (U6).""" + from sqlalchemy import text + + from agentkit.bitable.models import FieldType + from agentkit.bitable.repository import BitableRepository + + repo = BitableRepository(bitable_db) + table = await repo.create_table(name="T") + field = await repo.create_field(table_id=table.id, name="f", field_type=FieldType.text) + record1 = await repo.create_record(table_id=table.id) + record2 = await repo.create_record(table_id=table.id) + + # Same-table recalc — is_cross_table defaults to False + task = await repo.enqueue_recalc(table.id, record1.id, field.id) + assert task is not None + assert task.is_cross_table is False + + # Cross-table recalc — explicit flag set on a different record + cross_task = await repo.enqueue_recalc( + table.id, record2.id, field.id, is_cross_table=True + ) + assert cross_task is not None + assert cross_task.is_cross_table is True + + # Verify the DB column actually stores the flag + async with bitable_db.engine.begin() as conn: + result = await conn.execute( + text( + "SELECT is_cross_table FROM bitable.bitable_recalc_queue " + "WHERE id = :id" + ), + {"id": cross_task.id}, + ) + row = result.fetchone() + assert row is not None and row[0] is True diff --git a/tests/unit/bitable/test_formula_engine.py b/tests/unit/bitable/test_formula_engine.py index 1179b99..6610c1e 100644 --- a/tests/unit/bitable/test_formula_engine.py +++ b/tests/unit/bitable/test_formula_engine.py @@ -209,3 +209,52 @@ def test_engine_with_uuid_field_ids() -> None: engine.add_formula("calc", f"={{{fid}}} * 2") result = engine.evaluate("calc", row_values={fid: 21}) assert result == 42 + + +# --------------------------------------------------------------------------- +# U3: Cross-table formula evaluation +# --------------------------------------------------------------------------- + + +def test_engine_evaluate_cross_table_aggregate() -> None: + """=SUM({rel.target}) — cross-table aggregate via cross_table_values.""" + engine = FormulaEngine() + engine.add_formula("cross_sum", "=SUM({f_rel.f_target})") + # The service layer pre-resolves cross-table refs to value lists + # Safe name is _ct_f_rel__f_target + cross_values = {"_ct_f_rel__f_target": [10, 20, 30]} + result = engine.evaluate( + "cross_sum", row_values={}, cross_table_values=cross_values + ) + assert result == 60 + + +def test_engine_cross_table_dep_added_to_dag() -> None: + """Cross-table refs add the local relation field to the DAG.""" + engine = FormulaEngine() + engine.add_formula("calc", "=SUM({f_rel.f_target})") + deps = engine.get_dependencies("calc") + # The relation field on this table is a dependency (so when the relation + # changes, this formula is recomputed) + assert "f_rel" in deps + + +def test_engine_mixed_cross_table_and_row() -> None: + """={f_local} + SUM({f_rel.f_target}) — row + cross-table aggregate.""" + engine = FormulaEngine() + engine.add_formula("mixed", "={f_local} + SUM({f_rel.f_target})") + cross_values = {"_ct_f_rel__f_target": [1, 2, 3]} + result = engine.evaluate( + "mixed", row_values={"f_local": 10}, cross_table_values=cross_values + ) + assert result == 16 + + +def test_engine_cross_table_missing_values_defaults_empty() -> None: + """Missing cross_table_values default to empty list (no crash).""" + engine = FormulaEngine() + engine.add_formula("calc", "=SUM({f_rel.f_target})") + # No cross_table_values provided — should default to empty list + result = engine.evaluate("calc", row_values={}) + # SUM of empty list = 0 + assert result == 0 diff --git a/tests/unit/bitable/test_formula_parser.py b/tests/unit/bitable/test_formula_parser.py index a87df4c..ee3d867 100644 --- a/tests/unit/bitable/test_formula_parser.py +++ b/tests/unit/bitable/test_formula_parser.py @@ -9,6 +9,8 @@ from __future__ import annotations import pytest from agentkit.bitable.formula.parser import ( + MAX_AST_DEPTH, + FormulaDepthExceededError, FormulaParseError, FormulaSecurityError, UnknownFunctionError, @@ -25,20 +27,20 @@ ALLOWED = {"SUM", "AVG", "COUNT", "MIN", "MAX", "ABS", "ROUND", "IF", "LEN", "CO def test_parse_simple_arithmetic() -> None: - tree, mapping = parse_formula("=1+2*3", ALLOWED) + tree, mapping, _ = parse_formula("=1+2*3", ALLOWED) assert mapping == {} result = evaluate_ast(tree, {}, {}) assert result == 7 def test_parse_strips_equals_prefix() -> None: - tree1, _ = parse_formula("=1+1", ALLOWED) - tree2, _ = parse_formula("1+1", ALLOWED) + tree1, _, _ = parse_formula("=1+1", ALLOWED) + tree2, _, _ = parse_formula("1+1", ALLOWED) assert evaluate_ast(tree1, {}, {}) == evaluate_ast(tree2, {}, {}) == 2 def test_parse_field_reference() -> None: - tree, mapping = parse_formula("={field_abc} + 1", ALLOWED) + tree, mapping, _ = parse_formula("={field_abc} + 1", ALLOWED) assert "field_abc" in mapping.values() # Safe name is prefixed with _f_ safe_name = next(k for k, v in mapping.items() if v == "field_abc") @@ -49,7 +51,7 @@ def test_parse_field_reference() -> None: def test_parse_uuid_field_reference() -> None: """Field IDs are UUIDs with hyphens — must be substituted to safe names.""" fid = "a1b2c3d4-e5f6-7890-abcd-ef1234567890" - tree, mapping = parse_formula(f"={{{fid}}} * 2", ALLOWED) + tree, mapping, _ = parse_formula(f"={{{fid}}} * 2", ALLOWED) # The mapping should have a safe name → original UUID assert fid in mapping.values() # Evaluate using the safe name (prefixed with _f_) @@ -60,26 +62,26 @@ def test_parse_uuid_field_reference() -> None: def test_parse_string_concatenation() -> None: - tree, _ = parse_formula('="hello" + " " + "world"', ALLOWED) + tree, _, _ = parse_formula('="hello" + " " + "world"', ALLOWED) assert evaluate_ast(tree, {}, {}) == "hello world" def test_parse_conditional_ifexp() -> None: - tree, _ = parse_formula("=1 if True else 2", ALLOWED) + tree, _, _ = parse_formula("=1 if True else 2", ALLOWED) assert evaluate_ast(tree, {}, {}) == 1 def test_parse_comparison() -> None: - tree, mapping = parse_formula("={f} > 5", ALLOWED) + tree, mapping, _ = parse_formula("={f} > 5", ALLOWED) safe_name = next(k for k, v in mapping.items() if v == "f") assert evaluate_ast(tree, {safe_name: 10}, {}) is True assert evaluate_ast(tree, {safe_name: 3}, {}) is False def test_parse_boolean_ops() -> None: - tree, _ = parse_formula("=True and False", ALLOWED) + tree, _, _ = parse_formula("=True and False", ALLOWED) assert evaluate_ast(tree, {}, {}) is False - tree2, _ = parse_formula("=True or False", ALLOWED) + tree2, _, _ = parse_formula("=True or False", ALLOWED) assert evaluate_ast(tree2, {}, {}) is True @@ -89,14 +91,14 @@ def test_parse_boolean_ops() -> None: def test_parse_function_call_sum() -> None: - tree, mapping = parse_formula("=SUM({f1})", ALLOWED) + tree, mapping, _ = parse_formula("=SUM({f1})", ALLOWED) safe_name = next(k for k, v in mapping.items() if v == "f1") result = evaluate_ast(tree, {safe_name: [1, 2, 3]}, {"SUM": sum}) assert result == 6 def test_parse_function_call_concat() -> None: - tree, mapping = parse_formula('=CONCAT({f1}, "-", {f2})', ALLOWED) + tree, mapping, _ = parse_formula('=CONCAT({f1}, "-", {f2})', ALLOWED) safe_f1 = next(k for k, v in mapping.items() if v == "f1") safe_f2 = next(k for k, v in mapping.items() if v == "f2") result = evaluate_ast( @@ -106,7 +108,7 @@ def test_parse_function_call_concat() -> None: def test_parse_nested_function_calls() -> None: - tree, _ = parse_formula("=ABS(-5) + ROUND(3.7, 0)", ALLOWED) + tree, _, _ = parse_formula("=ABS(-5) + ROUND(3.7, 0)", ALLOWED) funcs = {"ABS": abs, "ROUND": round} result = evaluate_ast(tree, {}, funcs) assert result == 9 # 5 + 4 @@ -179,7 +181,7 @@ def test_parse_error_empty_string() -> None: def test_evaluate_unknown_field_ref_raises() -> None: - tree, _ = parse_formula("={nonexistent} + 1", ALLOWED) + tree, _, _ = parse_formula("={nonexistent} + 1", ALLOWED) with pytest.raises(FormulaParseError, match="Unknown field reference"): evaluate_ast(tree, {}, {}) @@ -191,9 +193,117 @@ def test_evaluate_unknown_field_ref_raises() -> None: def test_mixed_aggregate_and_row_context() -> None: """={f1} + SUM({f2}) — row f1 + column f2 sum.""" - tree, mapping = parse_formula("={f1} + SUM({f2})", ALLOWED) + tree, mapping, _ = parse_formula("={f1} + SUM({f2})", ALLOWED) safe_f1 = next(k for k, v in mapping.items() if v == "f1") safe_f2 = next(k for k, v in mapping.items() if v == "f2") # f1 is a row value (scalar), f2 is a column value (list) result = evaluate_ast(tree, {safe_f1: 10, safe_f2: [1, 2, 3]}, {"SUM": sum}) assert result == 16 # 10 + 6 + + +# --------------------------------------------------------------------------- +# KTD8: AST depth limit +# --------------------------------------------------------------------------- + + +def test_depth_limit_allows_normal_formula() -> None: + """Normal formulas should pass the depth check.""" + tree, _, _ = parse_formula("=1+2+3+4+5", ALLOWED) + assert tree is not None + + +def test_depth_limit_rejects_deeply_nested() -> None: + """Deeply nested expressions should be rejected (KTD8).""" + # Build a formula deeper than MAX_AST_DEPTH: nested additions + # Each "x + " adds 1 level of BinOp depth + deep_formula = "=" + " + ".join(["1"] * (MAX_AST_DEPTH + 5)) + with pytest.raises(FormulaDepthExceededError, match="depth"): + parse_formula(deep_formula, ALLOWED) + + +def test_depth_limit_allows_at_boundary() -> None: + """Formula at exactly the limit should pass.""" + # Build a formula with depth close to but not exceeding MAX_AST_DEPTH + # A simple sum of N numbers has depth ~N, but ast groups them differently. + # A safer test: shallow formula always passes. + tree, _, _ = parse_formula("=SUM({f1}) + IF({f2}, {f3}, {f4})", ALLOWED) + assert tree is not None + + +def test_list_literal_evaluated() -> None: + """List literals [1, 2, 3] should be evaluated as Python lists.""" + tree, _, _ = parse_formula("=[1, 2, 3]", ALLOWED) + result = evaluate_ast(tree, {}, {}) + assert result == [1, 2, 3] + + +def test_tuple_literal_evaluated() -> None: + """Tuple literals (1, 2, 3) should be evaluated as Python tuples.""" + tree, _, _ = parse_formula("=(1, 2, 3)", ALLOWED) + result = evaluate_ast(tree, {}, {}) + assert result == (1, 2, 3) + + +# --------------------------------------------------------------------------- +# U3: Cross-table references {rel.target} +# --------------------------------------------------------------------------- + + +def test_parse_cross_table_reference() -> None: + """{rel_field.target_field} is parsed into a single Name node (KTD-2).""" + tree, field_mapping, cross_table_mapping = parse_formula("={f_rel.f_target}", ALLOWED) + # Single-field mapping should be empty + assert field_mapping == {} + # Cross-table mapping should have one entry + assert len(cross_table_mapping) == 1 + safe_name, (rel_id, target_id) = next(iter(cross_table_mapping.items())) + assert rel_id == "f_rel" + assert target_id == "f_target" + assert safe_name.startswith("_ct_") + assert "__" in safe_name # separator between rel and target + + +def test_parse_cross_table_with_aggregate() -> None: + """=SUM({rel.target}) — cross-table ref inside an aggregate function.""" + tree, _, cross_table_mapping = parse_formula("=SUM({f_rel.f_target})", ALLOWED) + assert len(cross_table_mapping) == 1 + safe_name, (rel_id, target_id) = next(iter(cross_table_mapping.items())) + assert rel_id == "f_rel" + assert target_id == "f_target" + # Should evaluate correctly when given a list of values + result = evaluate_ast(tree, {safe_name: [1, 2, 3]}, {"SUM": sum}) + assert result == 6 + + +def test_parse_mixed_single_and_cross_table_refs() -> None: + """={f_local} + SUM({f_rel.f_target}) — mixed row + cross-table aggregate.""" + tree, field_mapping, cross_table_mapping = parse_formula( + "={f_local} + SUM({f_rel.f_target})", ALLOWED + ) + assert "f_local" in field_mapping.values() + assert len(cross_table_mapping) == 1 + safe_local = next(k for k, v in field_mapping.items() if v == "f_local") + safe_cross = next(iter(cross_table_mapping.keys())) + result = evaluate_ast( + tree, {safe_local: 10, safe_cross: [1, 2, 3]}, {"SUM": sum} + ) + assert result == 16 + + +def test_parse_cross_table_uuid_field_ids() -> None: + """Cross-table refs work with UUID-style field IDs containing hyphens.""" + rel_id = "a1b2c3d4-e5f6-7890-abcd-ef1234567890" + target_id = "11111111-2222-3333-4444-555555555555" + tree, _, cross_table_mapping = parse_formula(f"={{{rel_id}.{target_id}}}", ALLOWED) + assert len(cross_table_mapping) == 1 + _, (parsed_rel, parsed_target) = next(iter(cross_table_mapping.items())) + assert parsed_rel == rel_id + assert parsed_target == target_id + + +def test_cross_table_ref_does_not_match_single_field() -> None: + """{rel.target} must not be partially matched by _FIELD_REF_RE.""" + # The rel part should not leak into field_mapping + tree, field_mapping, cross_table_mapping = parse_formula("={f_rel.f_target}", ALLOWED) + assert field_mapping == {} + assert len(cross_table_mapping) == 1 diff --git a/tests/unit/bitable/test_functions_aggregate.py b/tests/unit/bitable/test_functions_aggregate.py new file mode 100644 index 0000000..a50f0dc --- /dev/null +++ b/tests/unit/bitable/test_functions_aggregate.py @@ -0,0 +1,102 @@ +"""Tests for aggregate and math formula functions (R1).""" + +from __future__ import annotations + +from agentkit.bitable.formula.functions.aggregate import ( + AGGREGATE_FUNCTIONS_DICT, + AGGREGATE_FUNCTIONS_LIST, +) + + +def test_registry_has_12_functions() -> None: + assert len(AGGREGATE_FUNCTIONS_DICT) == 12 + + +def test_aggregate_set_has_10_functions() -> None: + assert len(AGGREGATE_FUNCTIONS_LIST) == 10 + assert "SUM" in AGGREGATE_FUNCTIONS_LIST + assert "SUMIF" in AGGREGATE_FUNCTIONS_LIST + assert "ABS" not in AGGREGATE_FUNCTIONS_LIST # ABS is scalar, not aggregate + assert "ROUND" not in AGGREGATE_FUNCTIONS_LIST + + +# ── migrated aggregate functions ──────────────────────── + + +def test_sum() -> None: + assert AGGREGATE_FUNCTIONS_DICT["SUM"]([1, 2, 3, 4, 5]) == 15 + assert AGGREGATE_FUNCTIONS_DICT["SUM"]([1, None, "", "3"]) == 4.0 + + +def test_avg() -> None: + assert AGGREGATE_FUNCTIONS_DICT["AVG"]([1, 2, 3, 4, 5]) == 3.0 + assert AGGREGATE_FUNCTIONS_DICT["AVG"]([]) == 0.0 + + +def test_count() -> None: + assert AGGREGATE_FUNCTIONS_DICT["COUNT"]([1, 2, None, "", "x"]) == 3 + + +def test_min() -> None: + assert AGGREGATE_FUNCTIONS_DICT["MIN"]([3, 1, 4, 1, 5]) == 1.0 + assert AGGREGATE_FUNCTIONS_DICT["MIN"]([]) == 0 + + +def test_max() -> None: + assert AGGREGATE_FUNCTIONS_DICT["MAX"]([3, 1, 4, 1, 5]) == 5.0 + assert AGGREGATE_FUNCTIONS_DICT["MAX"]([]) == 0 + + +# ── migrated scalar math ──────────────────────────────── + + +def test_abs() -> None: + assert AGGREGATE_FUNCTIONS_DICT["ABS"](-5) == 5.0 + assert AGGREGATE_FUNCTIONS_DICT["ABS"](5) == 5.0 + assert AGGREGATE_FUNCTIONS_DICT["ABS"](None) is None + + +def test_round() -> None: + assert AGGREGATE_FUNCTIONS_DICT["ROUND"](3.14159, 2) == 3.14 + assert AGGREGATE_FUNCTIONS_DICT["ROUND"](3.14159) == 3.0 + assert AGGREGATE_FUNCTIONS_DICT["ROUND"](None) is None + + +# ── conditional aggregates ────────────────────────────── + + +def test_sumif_equality() -> None: + assert AGGREGATE_FUNCTIONS_DICT["SUMIF"]([1, 2, 3, 4, 5], 3) == 3.0 + + +def test_sumif_greater_than() -> None: + assert AGGREGATE_FUNCTIONS_DICT["SUMIF"]([1, 2, 3, 4, 5], ">3") == 9.0 # 4+5 + + +def test_countif_greater_than() -> None: + assert AGGREGATE_FUNCTIONS_DICT["COUNTIF"]([1, 2, 3, 4, 5], ">3") == 2 + + +def test_averageif() -> None: + assert AGGREGATE_FUNCTIONS_DICT["AVERAGEIF"]([1, 2, 3, 4, 5], ">=3") == 4.0 # (3+4+5)/3 + + +def test_sumifs() -> None: + values = [10, 20, 30, 40] + criteria_range = [1, 2, 1, 2] + # Sum values where criteria_range == 2 → 20 + 40 = 60 + assert AGGREGATE_FUNCTIONS_DICT["SUMIFS"](values, criteria_range, 2) == 60.0 + + +def test_countifs() -> None: + values = [10, 20, 30, 40] + criteria_range = [1, 2, 1, 2] + # Count values where criteria_range == 1 → 2 + assert AGGREGATE_FUNCTIONS_DICT["COUNTIFS"](values, criteria_range, 1) == 2 + + +def test_sumifs_length_mismatch() -> None: + import pytest + + with pytest.raises(ValueError, match="equal length"): + AGGREGATE_FUNCTIONS_DICT["SUMIFS"]([1, 2, 3], [1, 2], 1) diff --git a/tests/unit/bitable/test_functions_datetime.py b/tests/unit/bitable/test_functions_datetime.py new file mode 100644 index 0000000..7f31063 --- /dev/null +++ b/tests/unit/bitable/test_functions_datetime.py @@ -0,0 +1,126 @@ +"""Tests for datetime formula functions (R1).""" + +from __future__ import annotations + +from agentkit.bitable.formula.functions.datetime import ( + DATETIME_FUNCTIONS, + _date, + _dateadd, + _datedif, + _day, + _eomonth, + _hour, + _minute, + _month, + _networkdays, + _now, + _second, + _today, + _weekday, + _year, +) + + +def test_registry_has_15_functions() -> None: + assert len(DATETIME_FUNCTIONS) == 15 + assert "DATE" in DATETIME_FUNCTIONS + assert "NETWORKDAYS" in DATETIME_FUNCTIONS + + +def test_date_constructor() -> None: + result = _date(2026, 7, 6) + assert result == "2026-07-06T00:00:00" + + +def test_year_month_day() -> None: + d = _date(2026, 7, 6) + assert _year(d) == 2026 + assert _month(d) == 7 + assert _day(d) == 6 + + +def test_hour_minute_second() -> None: + dt = "2026-07-06T14:30:45" + assert _hour(dt) == 14 + assert _minute(dt) == 30 + assert _second(dt) == 45 + + +def test_weekday() -> None: + # 2026-07-06 is a Monday → ISO weekday 1 + assert _weekday("2026-07-06") == 1 + # 2026-07-12 is a Sunday → ISO weekday 7 + assert _weekday("2026-07-12") == 7 + + +def test_datedif_days() -> None: + assert _datedif("2026-07-01", "2026-07-06", "days") == 5 + + +def test_datedif_months() -> None: + assert _datedif("2026-01-15", "2026-07-15", "months") == 6 + + +def test_datedif_years() -> None: + assert _datedif("2020-01-01", "2026-01-01", "years") == 6 + + +def test_dateadd_days() -> None: + assert _dateadd("2026-07-06", 5, "days") == "2026-07-11T00:00:00" + + +def test_dateadd_months() -> None: + assert _dateadd("2026-01-15", 1, "months") == "2026-02-15T00:00:00" + + +def test_dateadd_months_clamps_day() -> None: + # Jan 31 + 1 month → Feb 28 (2026 is not a leap year) + assert _dateadd("2026-01-31", 1, "months") == "2026-02-28T00:00:00" + + +def test_networkdays() -> None: + # 2026-07-06 (Mon) to 2026-07-10 (Fri) = 5 working days + assert _networkdays("2026-07-06", "2026-07-10") == 5 + + +def test_networkdays_includes_weekend() -> None: + # 2026-07-06 (Mon) to 2026-07-12 (Sun) = 5 working days (weekend excluded) + assert _networkdays("2026-07-06", "2026-07-12") == 5 + + +def test_networkdays_reversed_args() -> None: + # Should swap if start > end + assert _networkdays("2026-07-10", "2026-07-06") == 5 + + +def test_eomonth() -> None: + # End of July 2026 + assert _eomonth("2026-07-06", 0) == "2026-07-31T00:00:00" + + +def test_eomonth_next_month() -> None: + # End of August 2026 + assert _eomonth("2026-07-06", 1) == "2026-08-31T00:00:00" + + +def test_today_returns_iso_string() -> None: + result = _today() + assert isinstance(result, str) + assert "T00:00:00" in result + + +def test_now_returns_iso_string() -> None: + result = _now() + assert isinstance(result, str) + assert "T" in result + + +def test_none_input_returns_none_for_accessors() -> None: + assert _year(None) is None + assert _month(None) is None + assert _day(None) is None + + +def test_empty_string_returns_none_for_accessors() -> None: + assert _year("") is None + assert _datedif("", "2026-07-06") is None diff --git a/tests/unit/bitable/test_functions_logical.py b/tests/unit/bitable/test_functions_logical.py new file mode 100644 index 0000000..6081f5d --- /dev/null +++ b/tests/unit/bitable/test_functions_logical.py @@ -0,0 +1,141 @@ +"""Tests for logical formula functions (R1).""" + +from __future__ import annotations + +from agentkit.bitable.formula.functions.logical import LOGICAL_FUNCTIONS + + +def test_registry_has_15_functions() -> None: + assert len(LOGICAL_FUNCTIONS) == 15 + expected = { + "IF", "IFS", "SWITCH", "IFERROR", "COALESCE", "AND", "OR", "NOT", + "ISBLANK", "ISNUMBER", "ISTEXT", "ISDATE", "TRUE", "FALSE", "NULL", + } + assert set(LOGICAL_FUNCTIONS.keys()) == expected + + +# ── IF (migrated) ─────────────────────────────────────── + + +def test_if_true() -> None: + assert LOGICAL_FUNCTIONS["IF"](1, "yes", "no") == "yes" + + +def test_if_false() -> None: + assert LOGICAL_FUNCTIONS["IF"](0, "yes", "no") == "no" + + +def test_if_default_false() -> None: + assert LOGICAL_FUNCTIONS["IF"](0, "yes") is None + + +# ── IFS ───────────────────────────────────────────────── + + +def test_ifs_first_match() -> None: + assert LOGICAL_FUNCTIONS["IFS"](1, "a", 0, "b", 1, "c") == "a" + + +def test_ifs_second_match() -> None: + assert LOGICAL_FUNCTIONS["IFS"](0, "a", 1, "b", 0, "c") == "b" + + +def test_ifs_no_match() -> None: + assert LOGICAL_FUNCTIONS["IFS"](0, "a", 0, "b") is None + + +# ── SWITCH ────────────────────────────────────────────── + + +def test_switch_match() -> None: + assert LOGICAL_FUNCTIONS["SWITCH"]("b", "a", 1, "b", 2, "c", 3) == 2 + + +def test_switch_default() -> None: + assert LOGICAL_FUNCTIONS["SWITCH"]("x", "a", 1, "b", 2, "default") == "default" + + +def test_switch_no_match_no_default() -> None: + assert LOGICAL_FUNCTIONS["SWITCH"]("x", "a", 1, "b", 2) is None + + +# ── IFERROR ───────────────────────────────────────────── + + +def test_iferror_passes_non_error() -> None: + assert LOGICAL_FUNCTIONS["IFERROR"](42, "default") == 42 + + +def test_iferror_returns_default_for_error() -> None: + error_dict = {"__error": "division by zero"} + assert LOGICAL_FUNCTIONS["IFERROR"](error_dict, "default") == "default" + + +# ── COALESCE ──────────────────────────────────────────── + + +def test_coalesce_first_non_empty() -> None: + assert LOGICAL_FUNCTIONS["COALESCE"]("a", "b", "c") == "a" + + +def test_coalesce_skips_none_and_empty() -> None: + assert LOGICAL_FUNCTIONS["COALESCE"](None, "", "c") == "c" + + +def test_coalesce_all_empty() -> None: + assert LOGICAL_FUNCTIONS["COALESCE"](None, "", None) is None + + +# ── Boolean operators ─────────────────────────────────── + + +def test_and() -> None: + assert LOGICAL_FUNCTIONS["AND"](1, 1, 1) is True + assert LOGICAL_FUNCTIONS["AND"](1, 0, 1) is False + + +def test_or() -> None: + assert LOGICAL_FUNCTIONS["OR"](0, 0, 1) is True + assert LOGICAL_FUNCTIONS["OR"](0, 0, 0) is False + + +def test_not() -> None: + assert LOGICAL_FUNCTIONS["NOT"](1) is False + assert LOGICAL_FUNCTIONS["NOT"](0) is True + + +# ── Predicates ────────────────────────────────────────── + + +def test_isblank() -> None: + assert LOGICAL_FUNCTIONS["ISBLANK"](None) is True + assert LOGICAL_FUNCTIONS["ISBLANK"]("") is True + assert LOGICAL_FUNCTIONS["ISBLANK"]("x") is False + + +def test_isnumber() -> None: + assert LOGICAL_FUNCTIONS["ISNUMBER"](42) is True + assert LOGICAL_FUNCTIONS["ISNUMBER"](3.14) is True + assert LOGICAL_FUNCTIONS["ISNUMBER"]("42") is False + assert LOGICAL_FUNCTIONS["ISNUMBER"](True) is False # bool excluded + + +def test_istext() -> None: + assert LOGICAL_FUNCTIONS["ISTEXT"]("hello") is True + assert LOGICAL_FUNCTIONS["ISTEXT"](42) is False + + +def test_isdate() -> None: + assert LOGICAL_FUNCTIONS["ISDATE"]("2026-07-06") is True + assert LOGICAL_FUNCTIONS["ISDATE"]("2026-07-06T10:30:00") is True + assert LOGICAL_FUNCTIONS["ISDATE"]("not a date") is False + assert LOGICAL_FUNCTIONS["ISDATE"](42) is False + + +# ── Constants ─────────────────────────────────────────── + + +def test_true_false_null() -> None: + assert LOGICAL_FUNCTIONS["TRUE"]() is True + assert LOGICAL_FUNCTIONS["FALSE"]() is False + assert LOGICAL_FUNCTIONS["NULL"]() is None diff --git a/tests/unit/bitable/test_functions_text.py b/tests/unit/bitable/test_functions_text.py new file mode 100644 index 0000000..481a576 --- /dev/null +++ b/tests/unit/bitable/test_functions_text.py @@ -0,0 +1,126 @@ +"""Tests for text formula functions (R1).""" + +from __future__ import annotations + +from agentkit.bitable.formula.functions.text import TEXT_FUNCTIONS + + +# ── Registry ──────────────────────────────────────────── + + +def test_registry_has_17_functions() -> None: + assert len(TEXT_FUNCTIONS) == 17 + expected = { + "CONCAT", "LEN", "FIND", "MID", "LEFT", "RIGHT", "REPLACE", "SUBSTITUTE", + "UPPER", "LOWER", "TRIM", "REPT", "TEXT", "VALUE", "STARTSWITH", "ENDSWITH", + "SPLIT", + } + assert set(TEXT_FUNCTIONS.keys()) == expected + + +# ── migrated functions ────────────────────────────────── + + +def test_concat() -> None: + assert TEXT_FUNCTIONS["CONCAT"]("a", "b", "c") == "abc" + assert TEXT_FUNCTIONS["CONCAT"]("a", None, "c") == "ac" + + +def test_len() -> None: + assert TEXT_FUNCTIONS["LEN"]("hello") == 5 + assert TEXT_FUNCTIONS["LEN"](None) == 0 + assert TEXT_FUNCTIONS["LEN"](123) == 3 + + +# ── find / extract ────────────────────────────────────── + + +def test_find() -> None: + assert TEXT_FUNCTIONS["FIND"]("hello world", "world") == 6 + assert TEXT_FUNCTIONS["FIND"]("hello", "xyz") == -1 + assert TEXT_FUNCTIONS["FIND"]("hello", "") == 0 + + +def test_mid() -> None: + assert TEXT_FUNCTIONS["MID"]("hello", 1, 3) == "ell" + assert TEXT_FUNCTIONS["MID"]("hello", 0, 2) == "he" + assert TEXT_FUNCTIONS["MID"]("hello", -1, 2) == "he" # negative clamped + + +def test_left() -> None: + assert TEXT_FUNCTIONS["LEFT"]("hello", 3) == "hel" + assert TEXT_FUNCTIONS["LEFT"]("hello") == "h" # default n=1 + assert TEXT_FUNCTIONS["LEFT"]("hello", 0) == "" + + +def test_right() -> None: + assert TEXT_FUNCTIONS["RIGHT"]("hello", 3) == "llo" + assert TEXT_FUNCTIONS["RIGHT"]("hello") == "o" # default n=1 + + +# ── replace / substitute ──────────────────────────────── + + +def test_replace() -> None: + # REPLACE("hello", 1, 3, "XYZ") → "h" + "XYZ" + "o" = "hXYZo" + # Replaces 3 chars starting at index 1: "ell" → "XYZ" + assert TEXT_FUNCTIONS["REPLACE"]("hello", 1, 3, "XYZ") == "hXYZo" + + +def test_substitute_all() -> None: + assert TEXT_FUNCTIONS["SUBSTITUTE"]("a-b-c", "-", "+") == "a+b+c" + + +def test_substitute_nth() -> None: + # Replace only the 2nd occurrence + assert TEXT_FUNCTIONS["SUBSTITUTE"]("a-b-c", "-", "+", 2) == "a-b+c" + + +# ── case / trim ───────────────────────────────────────── + + +def test_upper_lower() -> None: + assert TEXT_FUNCTIONS["UPPER"]("hello") == "HELLO" + assert TEXT_FUNCTIONS["LOWER"]("HELLO") == "hello" + + +def test_trim() -> None: + assert TEXT_FUNCTIONS["TRIM"](" hello world ") == "hello world" + + +# ── repeat / format / convert ─────────────────────────── + + +def test_rept() -> None: + assert TEXT_FUNCTIONS["REPT"]("ab", 3) == "ababab" + assert TEXT_FUNCTIONS["REPT"]("ab", 0) == "" + + +def test_text_format() -> None: + assert TEXT_FUNCTIONS["TEXT"](3.14159, ".2f") == "3.14" + assert TEXT_FUNCTIONS["TEXT"](42) == "42" + + +def test_value() -> None: + assert TEXT_FUNCTIONS["VALUE"]("42") == 42 + assert TEXT_FUNCTIONS["VALUE"]("3.14") == 3.14 + assert TEXT_FUNCTIONS["VALUE"]("not a number") is None + assert TEXT_FUNCTIONS["VALUE"](None) is None + + +# ── predicates / split ────────────────────────────────── + + +def test_startswith() -> None: + assert TEXT_FUNCTIONS["STARTSWITH"]("hello world", "hello") is True + assert TEXT_FUNCTIONS["STARTSWITH"]("hello world", "world") is False + + +def test_endswith() -> None: + assert TEXT_FUNCTIONS["ENDSWITH"]("hello world", "world") is True + assert TEXT_FUNCTIONS["ENDSWITH"]("hello world", "hello") is False + + +def test_split() -> None: + assert TEXT_FUNCTIONS["SPLIT"]("a,b,c", ",") == ["a", "b", "c"] + assert TEXT_FUNCTIONS["SPLIT"]("hello", ",") == ["hello"] diff --git a/tests/unit/bitable/test_models.py b/tests/unit/bitable/test_models.py index cc330db..8bd4112 100644 --- a/tests/unit/bitable/test_models.py +++ b/tests/unit/bitable/test_models.py @@ -12,12 +12,21 @@ import pytest from pydantic import ValidationError from agentkit.bitable.models import ( + AutomationActionType, + AutomationLog, + AutomationRule, + AutomationStatus, + AutomationTriggerType, + CrossTableDep, + CrossTableDepType, Field, FieldOwner, FieldType, Record, RecalcStatus, RecalcTask, + RelationLink, + RelationType, Table, View, ViewType, @@ -30,7 +39,7 @@ from agentkit.bitable.models import ( def test_field_type_values() -> None: - """FieldType has the 9 supported types with correct string values.""" + """FieldType has the 11 supported types with correct string values (V3 adds relation + rollup).""" expected = { "text", "number", @@ -40,7 +49,9 @@ def test_field_type_values() -> None: "attachment", "image", "formula", + "relation", "lookup", + "rollup", } assert {ft.value for ft in FieldType} == expected @@ -61,6 +72,40 @@ def test_recalc_status_lifecycle() -> None: assert {rs.value for rs in RecalcStatus} == {"pending", "calculating", "done", "error"} +def test_relation_type_values() -> None: + """RelationType covers single/bi/multi (R5).""" + assert {rt.value for rt in RelationType} == { + "one_to_one", + "one_to_many", + "many_to_many", + } + + +def test_automation_trigger_type_values() -> None: + """AutomationTriggerType covers create/update/delete (R13).""" + assert {t.value for t in AutomationTriggerType} == { + "record_created", + "record_updated", + "record_deleted", + } + + +def test_automation_action_type_values() -> None: + """AutomationActionType covers 5 action kinds (R14).""" + assert {a.value for a in AutomationActionType} == { + "create_record", + "update_record", + "send_webhook", + "send_email", + "send_message", + } + + +def test_cross_table_dep_type_values() -> None: + """CrossTableDepType covers formula/lookup/rollup edges (KTD-3).""" + assert {d.value for d in CrossTableDepType} == {"formula", "lookup", "rollup"} + + # --------------------------------------------------------------------------- # Table # --------------------------------------------------------------------------- @@ -301,3 +346,148 @@ def test_table_from_attributes() -> None: table = Table.model_validate(_Row()) assert table.id == "t1" assert table.name == "Orders" + + +# --------------------------------------------------------------------------- +# V3 models: RecalcTask.is_cross_table, RelationLink, AutomationRule, +# AutomationLog, CrossTableDep +# --------------------------------------------------------------------------- + + +def test_recalc_task_is_cross_table_default() -> None: + """RecalcTask.is_cross_table defaults to False (same-table recalc).""" + task = RecalcTask(id="q1", table_id="t1", record_id="r1", field_id="f1") + assert task.is_cross_table is False + + +def test_recalc_task_cross_table_flag() -> None: + """RecalcTask can carry is_cross_table=True (U6).""" + task = RecalcTask( + id="q1", + table_id="t1", + record_id="r1", + field_id="f1", + is_cross_table=True, + ) + assert task.is_cross_table is True + + +def test_relation_link_construction() -> None: + """RelationLink represents a junction-table row (KTD-1).""" + link = RelationLink( + id="rl1", + relation_field_id="f_rel", + source_record_id="r_src", + target_record_id="r_tgt", + ) + assert link.relation_field_id == "f_rel" + assert link.source_record_id == "r_src" + assert link.target_record_id == "r_tgt" + assert isinstance(link.created_at, datetime) + + +def test_relation_link_round_trip() -> None: + """RelationLink round-trips through JSON.""" + link = RelationLink( + id="rl1", + relation_field_id="f_rel", + source_record_id="r_src", + target_record_id="r_tgt", + ) + restored = RelationLink.model_validate(link.model_dump(mode="json")) + assert restored == link + + +def test_automation_rule_construction() -> None: + """AutomationRule carries trigger_config + action_config + token.""" + rule = AutomationRule( + id="a1", + table_id="t1", + name="Notify on done", + trigger_config={"type": "record_updated", "field_ids": ["f_status"]}, + action_config={"type": "send_webhook", "url": "https://example.com"}, + ) + assert rule.enabled is True + assert rule.webhook_token is None + assert rule.trigger_config["type"] == "record_updated" + assert rule.action_config["url"] == "https://example.com" + + +def test_automation_rule_round_trip() -> None: + """AutomationRule round-trips through JSON.""" + rule = AutomationRule( + id="a1", + table_id="t1", + name="Auto", + trigger_config={"type": "record_created"}, + action_config={"type": "send_message", "channel": "general"}, + webhook_token="tok_123", + ) + restored = AutomationRule.model_validate(rule.model_dump(mode="json")) + assert restored == rule + assert restored.webhook_token == "tok_123" + + +def test_automation_log_construction() -> None: + """AutomationLog carries status + attempt + error_message.""" + log = AutomationLog( + id="log1", + automation_id="a1", + trigger_record_id="r1", + status=AutomationStatus.retrying, + attempt=2, + error_message="transient", + ) + assert log.status == AutomationStatus.retrying + assert log.attempt == 2 + assert log.error_message == "transient" + assert log.completed_at is None + + +def test_cross_table_dep_construction() -> None: + """CrossTableDep records a source_field → target_field edge (KTD-3).""" + dep = CrossTableDep( + id="d1", + source_table_id="t_src", + source_field_id="f_src", + target_table_id="t_tgt", + target_field_id="f_tgt", + dep_type=CrossTableDepType.rollup, + ) + assert dep.dep_type == CrossTableDepType.rollup + assert dep.target_table_id == "t_tgt" + + +def test_field_rollup_config_shape() -> None: + """Rollup field config carries relation_field_id + aggregation (KTD-7).""" + field = Field( + id="f1", + table_id="t1", + name="Total Amount", + field_type=FieldType.rollup, + config={ + "relation_field_id": "f_rel", + "target_field_id": "f_amount", + "aggregation": "sum", + }, + ) + assert field.field_type == FieldType.rollup + assert field.config["aggregation"] == "sum" + + +def test_field_relation_config_shape() -> None: + """Relation field (lookup with relation config) carries relation_type.""" + field = Field( + id="f1", + table_id="t1", + name="Customer", + field_type=FieldType.lookup, + config={ + "relation_type": "many_to_many", + "target_table_id": "t_customers", + "is_bidirectional": True, + "reverse_field_id": "f_reverse", + }, + ) + assert field.config["relation_type"] == "many_to_many" + assert field.config["is_bidirectional"] is True diff --git a/tests/unit/bitable/test_recalc.py b/tests/unit/bitable/test_recalc.py index 154593e..97e4f8a 100644 --- a/tests/unit/bitable/test_recalc.py +++ b/tests/unit/bitable/test_recalc.py @@ -192,8 +192,9 @@ async def test_crash_recovery_resets_calculating_tasks( tasks = await bitable_service.get_pending_recalc_tasks() assert len(tasks) == 0 # not pending, it's calculating - # Crash recovery - reset_count = await bitable_service.reset_stale_recalc_tasks() + # Crash recovery — use threshold=0 to reset all calculating tasks immediately + # (simulates worker startup: all calculating tasks are stale when worker was down) + reset_count = await bitable_service.reset_stale_recalc_tasks(stale_threshold=0.0) assert reset_count == 1 # Now it should be pending again @@ -230,11 +231,14 @@ async def test_recalc_deduplication(bitable_service: BitableService) -> None: record = await bitable_service.create_record(table_id=table.id, values={src.id: 10}) # The create_record already enqueued one task. Enqueue again manually. + # With ON CONFLICT DO UPDATE: a duplicate "pending" task is left as-is + # (not reset), so the existing task is returned — not None. task2 = await bitable_service.trigger_recalc(table.id, record.id, calc.id) - # Should return None (duplicate, ON CONFLICT DO NOTHING) - assert task2 is None + # task2 may be the existing task (status still "pending") — not None + assert task2 is not None + assert task2.status == RecalcStatus.pending - # Only one pending task + # Only one pending task (no duplicate created) tasks = await bitable_service.get_pending_recalc_tasks() assert len(tasks) == 1 diff --git a/tests/unit/bitable/test_relation_service.py b/tests/unit/bitable/test_relation_service.py new file mode 100644 index 0000000..58b9792 --- /dev/null +++ b/tests/unit/bitable/test_relation_service.py @@ -0,0 +1,220 @@ +"""Tests for relation field CRUD service methods (U4: R5, R7, R8). + +These tests require PostgreSQL (skipped when unavailable). They verify: +- create_relation_field creates fields with correct config +- add/remove_relation_link manages junction table entries (many_to_many) +- add/remove_relation_link manages record JSONB (one_to_one/one_to_many) +- bidirectional relations create reverse fields +- delete_relation_field cascades cleanup +""" + +from __future__ import annotations + +import pytest + +from agentkit.bitable.models import FieldType, RelationType + + +@pytest.mark.asyncio +async def test_create_relation_field_one_to_many(bitable_service) -> None: + """create_relation_field creates a relation field with correct config.""" + table_a = await bitable_service.create_table(name="TableA") + table_b = await bitable_service.create_table(name="TableB") + + field = await bitable_service.create_relation_field( + table_id=table_a.id, + name="Related B", + target_table_id=table_b.id, + relation_type=RelationType.one_to_many, + ) + + assert field.field_type == FieldType.relation + assert field.config["relation_type"] == "one_to_many" + assert field.config["target_table_id"] == table_b.id + assert field.config["is_bidirectional"] is False + + +@pytest.mark.asyncio +async def test_create_relation_field_bidirectional(bitable_service) -> None: + """Bidirectional relation creates a reverse field on the target table.""" + table_a = await bitable_service.create_table(name="TableA") + table_b = await bitable_service.create_table(name="TableB") + + field = await bitable_service.create_relation_field( + table_id=table_a.id, + name="Related B", + target_table_id=table_b.id, + relation_type=RelationType.many_to_many, + is_bidirectional=True, + reverse_field_name="Reverse A", + ) + + assert field.config["is_bidirectional"] is True + assert "reverse_field_id" in field.config + + reverse_field = await bitable_service.get_field(field.config["reverse_field_id"]) + assert reverse_field is not None + assert reverse_field.name == "Reverse A" + assert reverse_field.config["target_table_id"] == table_a.id + + +@pytest.mark.asyncio +async def test_add_relation_link_one_to_many(bitable_service) -> None: + """one_to_many relation stores links in record JSONB.""" + table_a = await bitable_service.create_table(name="TableA") + table_b = await bitable_service.create_table(name="TableB") + + field = await bitable_service.create_relation_field( + table_id=table_a.id, + name="Related B", + target_table_id=table_b.id, + relation_type=RelationType.one_to_many, + ) + + rec_a = await bitable_service.create_record(table_a.id, {}) + rec_b1 = await bitable_service.create_record(table_b.id, {}) + rec_b2 = await bitable_service.create_record(table_b.id, {}) + + # Add links + assert await bitable_service.add_relation_link(field.id, rec_a.id, rec_b1.id) is True + assert await bitable_service.add_relation_link(field.id, rec_a.id, rec_b2.id) is True + + # Verify + related = await bitable_service.list_related_records(field.id, rec_a.id) + assert set(related) == {rec_b1.id, rec_b2.id} + + # Idempotent: adding again returns False + assert await bitable_service.add_relation_link(field.id, rec_a.id, rec_b1.id) is False + + +@pytest.mark.asyncio +async def test_remove_relation_link_one_to_many(bitable_service) -> None: + """remove_relation_link removes from record JSONB.""" + table_a = await bitable_service.create_table(name="TableA") + table_b = await bitable_service.create_table(name="TableB") + + field = await bitable_service.create_relation_field( + table_id=table_a.id, + name="Related B", + target_table_id=table_b.id, + relation_type=RelationType.one_to_many, + ) + + rec_a = await bitable_service.create_record(table_a.id, {}) + rec_b = await bitable_service.create_record(table_b.id, {}) + + await bitable_service.add_relation_link(field.id, rec_a.id, rec_b.id) + assert await bitable_service.remove_relation_link(field.id, rec_a.id, rec_b.id) is True + assert await bitable_service.list_related_records(field.id, rec_a.id) == [] + + +@pytest.mark.asyncio +async def test_add_relation_link_many_to_many(bitable_service) -> None: + """many_to_many relation uses junction table.""" + table_a = await bitable_service.create_table(name="TableA") + table_b = await bitable_service.create_table(name="TableB") + + field = await bitable_service.create_relation_field( + table_id=table_a.id, + name="Tags", + target_table_id=table_b.id, + relation_type=RelationType.many_to_many, + ) + + rec_a = await bitable_service.create_record(table_a.id, {}) + rec_b1 = await bitable_service.create_record(table_b.id, {}) + rec_b2 = await bitable_service.create_record(table_b.id, {}) + + assert await bitable_service.add_relation_link(field.id, rec_a.id, rec_b1.id) is True + assert await bitable_service.add_relation_link(field.id, rec_a.id, rec_b2.id) is True + + related = await bitable_service.list_related_records(field.id, rec_a.id) + assert set(related) == {rec_b1.id, rec_b2.id} + + # Reverse lookup + reverse = await bitable_service.list_related_records(field.id, rec_b1.id, reverse=True) + assert rec_a.id in reverse + + +@pytest.mark.asyncio +async def test_delete_relation_field_cascades(bitable_service) -> None: + """delete_relation_field removes junction entries and record values.""" + table_a = await bitable_service.create_table(name="TableA") + table_b = await bitable_service.create_table(name="TableB") + + field = await bitable_service.create_relation_field( + table_id=table_a.id, + name="Related B", + target_table_id=table_b.id, + relation_type=RelationType.many_to_many, + ) + + rec_a = await bitable_service.create_record(table_a.id, {}) + rec_b = await bitable_service.create_record(table_b.id, {}) + await bitable_service.add_relation_link(field.id, rec_a.id, rec_b.id) + + assert await bitable_service.delete_relation_field(field.id) is True + assert await bitable_service.get_field(field.id) is None + + +@pytest.mark.asyncio +async def test_add_relation_link_non_relation_field_raises(bitable_service) -> None: + """add_relation_link on a non-relation field raises ValueError.""" + table_a = await bitable_service.create_table(name="TableA") + field = await bitable_service.create_field( + table_id=table_a.id, name="text_field", field_type=FieldType.text + ) + with pytest.raises(ValueError, match="not a relation field"): + await bitable_service.add_relation_link(field.id, "rec1", "rec2") + + +@pytest.mark.asyncio +async def test_bidirectional_relation_link_cleanup(bitable_service) -> None: + """remove_relation_link cleans up the reverse field on the target record. + + For bidirectional relations, add_relation_link updates BOTH sides: + - Forward: source.values[forward_field] = [target] + - Reverse: target.values[reverse_field] = [source] + + remove_relation_link must mirror this — _cleanup_reverse_link removes + source from target.values[reverse_field]. Without this, the reverse + field retains stale references (asymmetric association). + """ + table_a = await bitable_service.create_table(name="TableA") + table_b = await bitable_service.create_table(name="TableB") + + field = await bitable_service.create_relation_field( + table_id=table_a.id, + name="Related B", + target_table_id=table_b.id, + relation_type=RelationType.one_to_many, + is_bidirectional=True, + reverse_field_name="Reverse A", + ) + reverse_field_id = field.config["reverse_field_id"] + + rec_a = await bitable_service.create_record(table_a.id, {}) + rec_b = await bitable_service.create_record(table_b.id, {}) + + # Add link: forward (A→B) and reverse (B→A) should both be populated + await bitable_service.add_relation_link(field.id, rec_a.id, rec_b.id) + + rec_a_after = await bitable_service.get_record(rec_a.id) + rec_b_after = await bitable_service.get_record(rec_b.id) + assert rec_a_after.values.get(field.id) == [rec_b.id] + assert rec_b_after.values.get(reverse_field_id) == [rec_a.id] + + # Remove link: both sides should be cleaned up + assert await bitable_service.remove_relation_link(field.id, rec_a.id, rec_b.id) is True + + rec_a_final = await bitable_service.get_record(rec_a.id) + rec_b_final = await bitable_service.get_record(rec_b.id) + # Forward field: rec_b removed from list + forward_val = rec_a_final.values.get(field.id) + assert forward_val == [] or forward_val is None + # Reverse field: rec_a removed from list (this is the key assertion — + # without _cleanup_reverse_link, rec_a would still be here) + reverse_val = rec_b_final.values.get(reverse_field_id) + assert reverse_val == [] or reverse_val is None, ( + f"Reverse field not cleaned up, still contains: {reverse_val}" + )