fischer
/
fischer-agentkit
1
0
Fork 0
Code Issues Pull Requests Actions 90 Packages Projects Releases Wiki Activity
400 Commits 1 Branch 0 Tags 22 MiB
Commit Graph

22 Commits

Author SHA1 Message Date
Chiguyong 3cf29f4633 fix(enterprise): ce-code-review P1 fixes — SSO is_active + span leak + Helm chart integration 
Stage 5c applied safe P1/P2/P3 fixes from multi-agent code review:

- SEC-1 (P1, security): SSO 登录绕过 is_active 检查 — _sso_issue_token_pair
  入口加显式校验，与本地 login（auth.py:377）一致。已停用用户无法再通过
  IDP 回调拿 JWT，封堵 R1b deprovisioning 绕过。

- PERF-2/3 (P2, reliability): gateway chat() span 泄漏 + 异常未记录 — try 块
  扩展到覆盖整个 span 生命周期（含 cache 设置），新增 except 分支调用
  record_exception + set_status(ERROR)，失败调用在 trace 中不再显示为成功。

- API-3 (P1, test isolation): SCIM router 使用 import-time 捕获的
  DEFAULT_AUTH_DB_PATH — 改为 resolve_auth_db_path() 在调用时读取 env var，
  让测试 monkeypatch.setenv 可见。

- MAINT-1 (P3, dead code): 删除 BitableRepository.delete_view（被
  delete_view_if_not_last 取代，无生产调用方，仅 docs 残留引用）。

- PERF-4/5 (P3, performance): pii_filter._record_pii_metrics 的 from import
  提升至模块顶部 + except: pass 改为 logger.debug 记录失败原因。

- DEPLOY-3 (P1, chart integration): ConfigMap 不可达 — deployment.yaml
  args 增加 --config /etc/agentkit/agentkit.yaml，让应用能加载 chart 渲染
  的非密配置。

- DEPLOY-6/7 (P3, doc/chart consistency): values.yaml 注释残留旧 slot 名
  redisPgPasswords → redisPassword；slot 计数 10 → 11（5 处 chart + 3 处
  deployment docs 同步修正）。

Verification:
- ruff check + format clean (7 modified .py files)
- pytest tests/unit/{auth,bitable} + test_{pii_filter,oidc_provider,
  saml_provider,scim_router,auth_audit,telemetry,prompt_cache_layers}.py
  → 336 passed, 145 skipped
- helm lint + helm template 验证 secret key 一致性

Residual findings (deferred to follow-up):
- PERF-1: PII filter 未接入 gateway 生产路径（manual, 需配置接入决策）
- API-1/2/4/5: SCIM RFC 7644 合规性（totalResults/error format/Location
  header/SSO 预配联动）— manual, 需设计决策
- DEPLOY-1/2/4: Helm chart ↔ app 集成断裂（部署名引用/Redis 鉴权/PG 密码双源）
  — manual, 需 app+chart 联动修改
 2026-07-06 08:36:59 +08:00
Chiguyong 86bce129a4 refactor(enterprise): simplify U1-U6 — fix span leak, dedup helpers, stdlib parsing 
lfg Step 3 (ce-simplify-code) — 16 files, -89 net lines.

P1 fixes (functional):
- audit.py / scim/router.py: fix OTel span leak — start_span() returned a
  context manager but was used as a plain object, so spans never ended
  and never exported. Wrap in `with`.
- helm chart: split combined `redisPgPasswords` slot (single JSON key)
  into `redisPassword` + `postgresPassword` slots — the combined `key`
  field was dead config never referenced by any template; deployment.yaml
  hardcoded `redis-password`/`postgres-password` keys were inconsistent
  with values.yaml schema.

P2 dedup (reuse):
- Promote `resolve_auth_db_path()` to models.py (canonical); remove 3
  duplicate `_resolve_db_path` defs in oidc.py / saml.py / audit.py.
- Reuse `_now_iso` from models.py; remove 5 duplicates across audit /
  oidc / saml / scim / local providers.
- Reuse `_row_to_user` from local.py in oidc.py / saml.py (verbatim
  copies removed).
- pii_filter._redact: collapse redundant `finditer` + `sub` (2 regex
  passes + 2x sha256 per match on hot path) into a single `sub` with a
  callback that collects matches and computes hashes once.

P3 simplifications:
- RoleChangeRequest.role: `str` + manual `if not in (...)` → `Literal`
  (Pydantic validates automatically).
- Extract `_record_pii_metrics(status, redacted_count)` helper to dedup
  3 nearly-identical OTel metric emission blocks.
- Extract `LLMGateway._record_prompt_cache_metric(usage, model)` to dedup
  non-stream + stream cache hit/miss counters; switch to use the
  `TokenUsage.cache_hit` property (previously added but unused).
- middleware: split `WHITELIST_PATHS` + `PREFIX_MATCH_PATHS` (redundant
  cross-reference) into `WHITELIST_PATHS` (exact) + `PREFIX_WHITELIST_PATHS`
  (prefix) — single source of truth per kind.
- OIDC token exchange: replace f-string form body with `urlencode` to
  correctly escape `&` / `=` in code / redirect_uri.
- auth.py OIDC redirect route: replace hand-rolled `url.split("state=")`
  with `urlparse + parse_qs`.
- PII module docstring: clarify that `PIIMatch.original` is in-memory
  only (must not be logged / persisted) — the previous "仅 hash, 不含
  原文" wording was misleading.

Skipped findings (false-positive or not worth):
- OIDC/SAML `_find_or_create_user` JIT 60-line dedup — high blast
  radius across SQL + transaction boundaries; defers to dedicated refactor.
- SAML `_extract_host` hand-rolled split — already has ponytail comment;
  stdlib `urlparse` swap is behavior-equivalent for current inputs but
  changes OneLogin `http_host` field semantics; conservative keep.
- gateway.py department-quota / usage-record sequential await —
  concurrency gain unclear when dept count is typically ≤2.

Verification:
- ruff check (scoped to 12 touched files) clean
- ruff format (1 file reformatted)
- pytest (PII + OIDC + SAML + SCIM + audit + bitable + team_orchestrator):
  265 passed, 145 skipped (PG/Docker), 0 failures
- helm lint clean; helm template renders correctly for both
  sealed-secrets + external-secrets backends
 2026-07-06 08:02:40 +08:00
Chiguyong 60a58a0fdb feat(llm): U2 — Prompt Cache 全链路监控 + PII 脱敏 hook 
U2/R3 — Anthropic prompt cache 命中率监控 + 等保合规 PII 脱敏。

变更：
- protocol.py: TokenUsage 添加 cache_read/creation_input_tokens + cache_hit property
- providers/anthropic.py: 非流式 + 流式两处解析 cache 字段（cache_read_input_tokens / cache_creation_input_tokens）
- gateway.py: chat() / chat_stream() emit prompt_cache.hit / prompt_cache.miss OTel counter + span attributes
- telemetry/metrics.py: 新增 4 个 metric instruments（prompt_cache.hit/miss + pii_filter.coverage/redacted）
- llm/pii_filter.py（新文件）: 正则版 PII 脱敏（手机/邮箱/身份证/银行卡）+ ner_hook + fail-closed + sha256 hash 审计
- agentkit.yaml: 添加 fallbacks + pii_filter 配置模板（默认 disabled，等保场景按需启用）
- tests: test_prompt_cache_layers.py +4 cache metric 测试；test_pii_filter.py 新增 PII 脱敏测试（29 测试通过）

设计决策：
- PII 脱敏用 stdlib（re + hashlib），不引入 NER 依赖；ner_hook 接入客户内网 NER 模型
- 正则顺序 id_card 在 phone 之前（避免身份证号内 11 位数字子串被手机号误匹配）
- ner_hook 异常不吞掉：传播到 filter_pii 由 fail_closed 决定（True→PIIFilterError；False→降级正则版重试）
- fail-closed：脱敏失败拒绝发送（不降级到原文），等保合规要求
 2026-07-06 00:32:35 +08:00
Fischer 838a05772e refactor: follow-up tech debt cleanup (except Exception + Any 治理) (#9)
Deploy to Production / deploy (push) Waiting to run Details
Test / backend-test (push) Waiting to run Details
Test / frontend-unit (push) Waiting to run Details
Test / api-e2e (push) Waiting to run Details
Test / frontend-e2e (push) Waiting to run Details
2026-07-01 03:03:02 +08:00
chiguyong c4aaef05aa feat(U2): G2 prompt cache 双块结构 
- ReActEngine 新增 _build_system_message(stable+volatile) 双块构造
- Anthropic provider 返回 content blocks,stable 块带 cache_control
- 非 Anthropic provider 返回字符串拼接,依赖 stable 前缀命中自动前缀缓存
- execute_stream/execute 记忆注入从 system_prompt 末尾移到 volatile 层
- LLMGateway.get_provider_name_for_model 暴露 provider 检测能力
- anthropic.py _convert_messages 支持 list-type system content 透传
- ServerConfig.prompt_cache 配置项(默认 enable=True)
- ReActEngine.prompt_cache_enable 构造参数(默认 True 保当前行为)
- test_prompt_cache_layers.py 覆盖 R4-R7/R13
 2026-06-29 20:47:23 +08:00
chiguyong 31c65e01b8 fix(security): P0 安全加固 + 多实例部署一致性 (U1-U4 + U5c)
Deploy to Production / deploy (push) Has been cancelled Details 
U1: LLM gateway KB 缓存 fail-closed — 异常时默认禁用缓存防止 KB 数据泄漏
U2: MCP 危险工具黑名单过滤 — 6+1 端点覆盖，防止绕过 chat confirmation
U3: SecretsStore Redis 迁移 — 多 worker 共享凭证，内存降级保留开发模式
U4: channels webhook Redis 状态 — ZSET 滑动窗口限流 + nonce dedup + backpressure
U5c: ce-code-review 修复批次:
  - P0: 统一 MCP 黑名单与 publisher.py 一致 (terminal_execute -> terminal, +file_read)
  - P1: ZSET 限流 member 加 uuid 后缀避免同时间戳碰撞
  - P1: SecretsStore redis 参数 Any -> aioredis.Redis | None (AGENTS.md 合规)
  - P1: Redis client 添加 socket_timeout 防止单点故障请求挂死

测试: 171 scoped tests pass, ruff clean
 2026-06-26 04:05:33 +08:00
chiguyong 53faa60472 fix(review): ce-code-review P1+P2 修复 — 安全/可靠性/性能 
P1 安全与可靠性（4 项）：
- wecom: verify_signature 增加时间戳新鲜度校验（5 分钟窗口防重放）
- cache: should_cache 在 per_user_namespace 开启时拒绝 user_id=None
  匿名请求，避免跨用户缓存泄漏（安全要求 a/e）
- channels: webhook receive_message 异常兜底，防止 500 触发平台重试风暴
- app: shutdown 调用 close_all_adapters + await _pending_webhook_tasks，
  防止 httpx 连接泄漏和丢失 IM 回复

P2 效率与可维护性（5 项）：
- feishu: _TOKEN_CACHE_TTL 300 → 6900（2h 减 5min 余量，避免 24x 过频刷新）
- channels: _pending_webhook_tasks 有界化（2x 并发上限时 429 拒绝）
- gateway: quota 检查每 period 单次 get_usage，复用 summary 检查 token+cost
- cache_key: generate_cache_key 合并为单次 SHA-256（消除 8-10 次冗余哈希）
- config: ProviderConfig.get_api_key 移除未用的 secrets_store 参数

P3 去重（1 项）：
- channels: _process_inbound_message DIRECT_CHAT 路径提取 _direct_chat 辅助函数

测试：
- test_wecom: 时间戳改用 int(time.time())，新增 test_expired_timestamp_rejected
- test_cache: should_cache 测试覆盖匿名拒绝 + namespace_off 兼容
- test_config_migration: get_api_key 测试适配新签名
- channels/config_migration/quota_enforcement 测试全部通过
 2026-06-26 01:40:31 +08:00
chiguyong 1ccaf56b9a refactor: ce-simplify-code 审查修复 — 去重 + 效率 + 死代码清理 
3 个审查代理（复用/质量/效率）发现 15 个问题，全部修复：

效率与安全（6 项）：
- MCPClient 缓存 MultiServerMCPClient 单例 + aclose()，修复连接/子进程泄漏
- _rate_limits 清理空 IP 条目，修复 X-Forwarded-For 欺骗下内存泄漏
- _seen_nonces 改用 OrderedDict，O(1) 摊销过期清理
- webhook 后台任务加 Semaphore(20) + 任务引用追踪，限制无界并发
- _build_adapter 用 asyncio.gather 并行解密 secrets
- 适配器实例缓存（_adapter_cache），token TTL 缓存跨请求命中

去重（4 项）：
- header_get 提取到 channels/base.py，4 个适配器统一 import
- _get_client/close() 移入 MessageAdapter 基类，子类继承
- URLVerificationChallenge 统一到 base.py，feishu/slack/wecom 共用
- Transport ABC 添加 endpoint_url 属性，from_transport 不再访问私有字段

死代码与类型安全（5 项）：
- detect_cache_hit 死方法替换为 record_cache_result 公开 API
- execution_mode.value == "direct_chat" 改用枚举比较
- 删除 yielded_any 死变量、重复 from fastapi import Request、
  多余 getattr 防御

453 tests passed, ruff clean（预存 F841 非本次引入）
 2026-06-25 23:54:14 +08:00
chiguyong 793476cafa feat(llm): U17 — LiteLLM 语义缓存替换 + per-user/ACL scope 安全隔离 
- 新增 LitellmCacheManager：配置 litellm.cache 全局，三级后端 fallback
  (RedisSemanticCache -> RedisCache -> InMemoryCache)，redisvl lazy import
- cache_key 扩展 user_id + kb_acl_hash 参数（安全要求 a/b/e）
- gateway 集成：读取 KB caching_disabled flag（安全要求 c），构建带 scope
  的 cache_key，命中时 cost=0
- LLMResponse 新增 cache_hit 字段；LLMRequest 新增 cache 参数
- litellm_provider 透传 cache 参数 + 检测 _hidden_params 缓存命中
- 33 个新测试覆盖 13 场景（含 User A != User B 缓存隔离）
- 旧 InMemoryLLMCache/RedisLLMCache 保留向后兼容
 2026-06-25 22:49:59 +08:00
chiguyong cd371e4155 fix(review): U2 quota semantics — monthly quota + multi-dept attribution + TOCTOU docs 2026-06-22 16:30:22 +08:00
chiguyong 00c8386939 fix(review): U1 Redis quota enforcement — key construction + fail-closed + degradation recovery + async 2026-06-22 16:22:33 +08:00
chiguyong 09feca3307 feat(admin): U7 — usage dashboard + quota enforcement 
UsageRecord extended with user_id + department_id (backward compatible).
UsageStore Protocol extended: record() accepts user_id/department_id,
get_usage() accepts filters, new get_usage_by_user/department methods.
RedisUsageStore uses versioned keys (v2) for new records.

LLMGateway.chat()/chat_stream() accept user_id, department_ids, db_path.
Quota check before provider call: model whitelist + token limit + cost
limit (daily). Multi-department uses strictest-wins (any exceed → reject).
QuotaExceededError → 429 at route layer.

UsageService: summary, timeseries, by-model, top-users, export (CSV/JSON).
5 new admin endpoints under /admin/usage/*.

llm_gateway.py routes pass DepartmentContext + db_path to gateway,
catch QuotaExceededError → 429 (JSON for /chat, SSE error for /stream).

84 new tests. 441 admin+usage tests pass, no regressions.
 2026-06-21 17:23:20 +08:00
TraeAI d245f2e3d8 fix: UI/UX 修复 + 暗色主题 + async generator 防御 
- App.vue: 重构 bootstrapBackend 流程，新增 retryBootstrap 重试入口
- SplashScreen.vue: 错误状态显示「重试」按钮
- system.py: /system/resources 移除 SYSTEM_CONFIG 权限依赖，避免 dev 模式 401
- react.py + gateway.py: 新增 _ensure_async_iterable helper 防御
  'async for requires aiter, got coroutine'
- theme.ts: Ant Design colorTextLightSolid 映射到 --text-inverse
  修复暗色主题下所有 primary 按钮白底白字
- ChatSidebar.vue: 新建对话按钮兜底深色文字
- SystemMonitorPanel.vue: 服务状态区域间距优化
- chat.ts + portal.py + sqlite_conversation_store.py: 会话标题派生修复
  解决点击对话标题变成"对话"的问题
- app.py: Serve 模式自动创建 default agent
- Tauri src-tauri/: 完整 Tauri 客户端配置 (icons, capabilities, Cargo)
 2026-06-20 23:35:57 +08:00
chiguyong 840d1afd6a fix: resolve benchmark failures from root cause (LLM timeout, WebSocket, latency stats) 
U1: LLM reasoning - difficulty-based timeout (easy=20s/medium=40s/hard=60s)
    + streaming keyword detection for hard tasks with non-stream fallback
U2: GUI WebSocket - remove unreliable HTTP pre-check (FastAPI returns 404
    for HTTP GET to WS endpoints), directly test WS connection, treat
    {"type":"connected"} as pass (ping/pong is bonus info)
U3: Verification latency - exclude timeout-tagged cases from P95/p99
    percentile calculation (accuracy stats unaffected)
U4: LLM Gateway - add timeout field to LLMRequest, gateway.chat()/
    chat_stream() passthrough for provider-level timeout support

Test results: 62/63 pass (98.4%), gui-004 fixed, no regressions
pytest: 64 passed, ruff: clean
 2026-06-17 13:32:54 +08:00
chiguyong b54213b3c6 fix(review): resolve all P0/P1/P2 findings from code review 2026-06-16 09:08:03 +08:00
chiguyong 16ac592855 feat(gateway): empty response auto-retry with fallback model chain 2026-06-16 08:07:21 +08:00
chiguyong 0ccef7be5c feat: P0 production hardening — LLM cache, semantic routing, state persistence 
U1: LLM Cache Core (exact + semantic match, InMemory + Redis backends)
U2: Cache integration into LLMGateway with CacheConfig
U3: Semantic Router as Layer 1.5 in CostAwareRouter
U4: UsageStore persistence (Redis Hash + InMemory fallback)
U5: CascadeStateStore persistence (Redis INCR + InMemory TTL)
U6: EvolutionStore interface unification (Protocol + PostgreSQL backend)
U7: Configuration integration + E2E tests

Code review fixes:
- P0: date iteration bug (day>=28), semantic router index never built,
      Redis connection leak (per-call → persistent pool)
- P1: cache degradation recovery, semantic_search degradation,
      double miss counting, asyncio.Lock for PG init, LIMIT on queries,
      __import__ anti-pattern → _utcnow()
- P2: InMemory TTL cleanup, embedding preservation on put(),
      data TTL = max(exact_ttl, semantic_ttl)
 2026-06-14 15:16:00 +08:00
chiguyong 239009357a feat(telemetry): U7 OpenTelemetry integration with zero-dependency no-op pattern 
Add telemetry module with tracing (agent/tool/llm/pipeline_step spans),
metrics (5 histograms/counters), and setup with optional OTLP exporters.
Uses no-op pattern when opentelemetry not installed. GenAI Semantic
Conventions for LLM spans. Integrated into ReactEngine, LLMGateway,
ToolBase, and FastAPI app.
 2026-06-07 17:26:21 +08:00
chiguyong 6e362a8ae7 feat(agentkit): Phase 4 enterprise production upgrade — 12 Implementation Units 
Phase A (P0): EpisodicMemory pgvector search+EmbeddingCache, ReAct timeout+CancellationToken, evolution system fix (A/B test+LLMPromptOptimizer+StrategyTuner), AnthropicProvider native Messages API
Phase B (P1): RetryPolicy+CircuitBreaker, chat_stream fallback chain, WebSocket endpoint, SSE stream fix, Evolution+Memory API routes (7 endpoints), embedding cache+Enhanced Search per-KB degradation fix
Phase C (P2): GeminiProvider native generateContent API, Agent state lock+config hot-reload

Tests: 1301 passed, 18 skipped, 0 failed
 2026-06-06 21:51:04 +08:00
chiguyong 0456429beb fix(review): address all 14 P2 advisory findings 2026-06-06 18:20:46 +08:00
chiguyong 2844eeb548 feat(streaming): Phase C - LLM streaming + ReAct events + SSE endpoint 
U8: StreamChunk protocol + OpenAI chat_stream + Gateway streaming with usage tracking
U9: ReActEvent dataclass + execute_stream() yielding thinking/tool_call/tool_result/final_answer
U10: POST /tasks/stream SSE endpoint + Client SDK stream_task()

15 new tests passing, no regression.
 2026-06-06 11:54:17 +08:00
chiguyong f87b790c0f feat(agentkit): v2 Phase 1 - ReAct/LLM Gateway/Skill/Server + review fixes 
535 unit + 52 integration tests passing. README added.
 2026-06-05 23:32:16 +08:00