499 lines
30 KiB
Markdown
499 lines
30 KiB
Markdown
---
|
||
title: "feat: P0 Production Hardening — LLM Cache, Semantic Routing, State Persistence"
|
||
status: active
|
||
created_at: 2026-06-14
|
||
type: feat
|
||
origin: "行业调研与项目审视（2026-06-14）"
|
||
depth: deep
|
||
---
|
||
|
||
# P0 Production Hardening — LLM Cache, Semantic Routing, State Persistence
|
||
|
||
## Summary
|
||
|
||
Three P0 gaps identified from industry benchmarking and project audit: (1) LLM response caching to reduce 30-50% token cost, (2) embedding-based semantic routing to improve intent matching quality at zero LLM cost, (3) critical state persistence for UsageTracker, EvolutionStore, and CascadeDetector to survive restarts and enable multi-instance deployment. Each unit requires detailed architecture design and code reasoning before implementation — design-first, code-second.
|
||
|
||
## Problem Frame
|
||
|
||
AgentKit has strong differentiation in self-evolution, quality management, and multi-paradigm engines, but three production-critical gaps prevent enterprise deployment:
|
||
|
||
1. **Every LLM request hits the provider** — no caching. Identical or similar requests waste tokens and money. Competitors like Dify have built-in caching.
|
||
2. **Routing relies on keyword matching and LLM classification** — no semantic understanding. Embedding-based routing is industry standard (Agentic RAG trend) and AgentKit already has embedding infrastructure but doesn't use it for routing.
|
||
3. **Critical state lives in memory** — UsageTracker, CascadeDetector, and EvolutionStore lose data on restart. Multi-instance deployment is impossible without shared state.
|
||
|
||
These gaps are P0 because they directly impact cost (caching), quality (routing accuracy), and reliability (state persistence) — the three pillars of production readiness.
|
||
|
||
---
|
||
|
||
## Requirements
|
||
|
||
- R1. LLM cache must support exact-match (hash-based) and semantic-match (embedding-based) cache hits
|
||
- R2. LLM cache must integrate transparently into `LLMGateway.chat()` without changing the public API
|
||
- R3. LLM cache must record usage on cache hits (0 cost) to maintain usage tracking integrity
|
||
- R4. Semantic routing must insert between Layer 1 and Layer 2 in `CostAwareRouter`
|
||
- R5. Semantic routing must use existing `OpenAIEmbedder` and `compute_cosine_similarity()` infrastructure
|
||
- R6. Semantic routing must pre-compute skill embeddings at registration time, not at query time
|
||
- R7. UsageTracker must persist records to Redis with O(1) write and efficient aggregation
|
||
- R8. CascadeDetector must persist state to Redis using atomic INCR operations
|
||
- R9. EvolutionStore must support both PostgreSQL and SQLite backends with a unified interface
|
||
- R10. All three features must degrade gracefully when Redis/PG is unavailable (fallback to in-memory)
|
||
- R11. Each unit must have detailed architecture design and code reasoning documented before implementation begins
|
||
|
||
---
|
||
|
||
## Key Technical Decisions
|
||
|
||
- **KTD-1. Cache key design**: Use `SHA256(model + system_prompt_hash + messages_content_hash + temperature + tools_hash)` for exact match. For semantic match, embed the last user message and compare against cached embeddings using cosine similarity > threshold. Rationale: exact match is fast and deterministic; semantic match catches paraphrased requests. Both are needed because exact match alone misses too many hits, and semantic match alone is too slow for every request.
|
||
|
||
- **KTD-2. Cache storage backend**: Implement `LLMCache` as a Protocol with `InMemoryLLMCache` and `RedisLLMCache` backends. In-memory uses `OrderedDict` with LRU eviction (following `EmbeddingCache` pattern). Redis uses `agentkit:llm_cache:{hash}` keys with TTL. Rationale: follows existing factory pattern (`create_message_bus`, `create_session_store`); in-memory for dev/single-instance, Redis for production.
|
||
|
||
- **KTD-3. Semantic routing insertion point**: Insert as Layer 1.5 between `HeuristicClassifier` and `_classify_merged()`. When Layer 1 returns medium complexity (0.3-0.7), try semantic routing first. If similarity > 0.85, return skill match directly (skip LLM). If similarity 0.6-0.85, pass skill hint to Layer 2 LLM (reduces LLM classification tokens). If < 0.6, proceed to Layer 2 unchanged. Rationale: this placement maximizes cost savings by avoiding LLM calls when semantic match is confident, while preserving the existing fallback chain.
|
||
|
||
- **KTD-4. Skill embedding source text**: Embed `f"{skill.description} | {' '.join(skill.intent.keywords)} | {' '.join(cap.tag for cap in skill.capabilities)}"` for each skill. Cache embeddings in a dict keyed by skill name, re-embed on skill registration/update. Rationale: combines all semantic signals; description alone misses keyword intent; keywords alone misses semantic meaning.
|
||
|
||
- **KTD-5. UsageTracker persistence strategy**: Use Redis Hash for time-series data. Key pattern: `agentkit:usage:{date}` with fields `{agent}:{model}` → JSON `{tokens, cost, latency_ms, count}`. Write via `HINCRBYFLOAT` for atomic increment. Query via `HGETALL` + client-side aggregation. Rationale: O(1) write, acceptable query performance, natural TTL by date, follows Redis patterns in project.
|
||
|
||
- **KTD-6. CascadeDetector persistence strategy**: Use Redis atomic operations. Key pattern: `agentkit:cascade:{session_id}:interactions` (INCR + TTL) and `agentkit:cascade:{session_id}:depth` (SET/GET + TTL). Rationale: INCR is atomic, no race conditions across instances; TTL prevents memory leaks; matches session lifecycle.
|
||
|
||
- **KTD-7. EvolutionStore interface unification**: Extend the base `EvolutionStore` Protocol to include `skill_version` and `ab_test` methods. Make `PersistentEvolutionStore` (SQLite) implement the unified Protocol. Add a new `PostgreSQLEvolutionStore` that uses async SQLAlchemy like the existing `EvolutionStore` but with the full unified interface. Rationale: current split (sync SQLite vs async PG) creates maintenance burden; unified Protocol enables backend-agnostic usage.
|
||
|
||
- **KTD-8. Graceful degradation pattern**: All three features use the same pattern — try preferred backend, catch connection error, log warning, fall back to in-memory. Controlled by `cache.backend`, `usage_store.backend`, `cascade_store.backend` config values (`"auto"` | `"redis"` | `"memory"`). `"auto"` tries Redis, falls back to memory. Rationale: production needs persistence, but dev/testing shouldn't require Redis.
|
||
|
||
---
|
||
|
||
## High-Level Technical Design
|
||
|
||
### LLM Cache Flow
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
A[LLMGateway.chat] --> B{Cache enabled?}
|
||
B -->|no| F[Call Provider]
|
||
B -->|yes| C[Generate exact key]
|
||
C --> D{Exact match?}
|
||
D -->|hit| E[Return cached response]
|
||
D -->|miss| G[Generate embedding of last user msg]
|
||
G --> H{Semantic match? similarity > 0.92}
|
||
H -->|hit| E
|
||
H -->|miss| F
|
||
F --> I[Write to cache]
|
||
I --> J[Record usage]
|
||
E --> K[Record usage with 0 cost]
|
||
```
|
||
|
||
### Semantic Routing Flow
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
A[CostAwareRouter.route] --> B[Layer 0: Regex rules]
|
||
B -->|matched| Z[Return DIRECT_CHAT]
|
||
B -->|unmatched| C[Layer 1: HeuristicClassifier]
|
||
C -->|low complexity| Z
|
||
C -->|medium-high| D[Layer 1.5: Semantic Router NEW]
|
||
D -->|sim > 0.85| E[Return SKILL_REACT with matched skill]
|
||
D -->|sim 0.6-0.85| F[Pass skill_hint to Layer 2]
|
||
D -->|sim < 0.6| G[Layer 2: LLM classification]
|
||
F --> G
|
||
G --> H[Return routing result]
|
||
```
|
||
|
||
### State Persistence Architecture
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph "Current (In-Memory)"
|
||
UT1[UsageTracker dict]
|
||
CD1[CascadeDetector dict]
|
||
ES1[EvolutionStore SQLite]
|
||
end
|
||
subgraph "Target (Persistent)"
|
||
UT2[UsageStore Protocol]
|
||
CD2[CascadeStateStore Protocol]
|
||
ES2[UnifiedEvolutionStore Protocol]
|
||
UT2 -->|redis| R1[Redis Hash agentkit:usage:date]
|
||
UT2 -->|memory| M1[InMemoryUsageStore]
|
||
CD2 -->|redis| R2[Redis INCR agentkit:cascade:session]
|
||
CD2 -->|memory| M2[InMemoryCascadeStore]
|
||
ES2 -->|postgresql| P1[PG EvolutionEventModel + SkillVersionModel]
|
||
ES2 -->|sqlite| S1[PersistentEvolutionStore]
|
||
ES2 -->|memory| M3[InMemoryEvolutionStore]
|
||
end
|
||
```
|
||
|
||
---
|
||
|
||
## Scope Boundaries
|
||
|
||
### In Scope
|
||
|
||
- LLM response caching (exact + semantic match, in-memory + Redis backends)
|
||
- Semantic routing as Layer 1.5 in CostAwareRouter
|
||
- UsageTracker Redis persistence
|
||
- CascadeDetector Redis persistence
|
||
- EvolutionStore interface unification
|
||
- Configuration for all three features
|
||
- Architecture design documents for each unit before coding
|
||
|
||
### Deferred for Follow-Up
|
||
|
||
- Semantic cache using pgvector (current semantic match uses in-memory embedding comparison)
|
||
- Cache warming / pre-population strategies
|
||
- Routing cache (caching routing results for similar queries)
|
||
- Usage analytics dashboard (visualization of usage data)
|
||
- Multi-tenant resource quotas
|
||
- Rate limiting and concurrency control (P2)
|
||
- Distributed tracing visualization (P2)
|
||
|
||
---
|
||
|
||
## Implementation Units
|
||
|
||
### U1. LLM Cache Core
|
||
|
||
**Goal:** Implement the `LLMCache` Protocol, `InMemoryLLMCache`, and `RedisLLMCache` with exact-match and semantic-match capabilities.
|
||
|
||
**Dependencies:** None
|
||
|
||
**Files:**
|
||
- `src/agentkit/llm/cache.py` (new) — `LLMCache` Protocol, `InMemoryLLMCache`, `RedisLLMCache`, `CacheResult`, `CacheKey` generation
|
||
- `src/agentkit/llm/cache_key.py` (new) — `generate_cache_key()`, `generate_messages_hash()`, `generate_system_prompt_hash()`
|
||
- `tests/unit/llm/test_cache.py` (new) — unit tests for cache backends
|
||
|
||
**Approach:**
|
||
|
||
Architecture design before coding:
|
||
|
||
1. **CacheKey design reasoning**: The key must capture all inputs that affect LLM output. `model` determines which model responds. `system_prompt` sets behavior. `messages` carry the conversation. `temperature` affects randomness (only cache temperature=0 deterministically). `tools` affect tool_call availability. Hash each component independently so partial changes don't invalidate the entire key.
|
||
|
||
2. **Exact match implementation**: SHA-256 hash of concatenated component hashes. Store as `agentkit:llm_cache:{sha256_hex}` in Redis with TTL. In-memory uses OrderedDict keyed by hash string.
|
||
|
||
3. **Semantic match implementation**: For cache misses on exact match, embed the last user message using `OpenAIEmbedder`. Compare against cached embeddings using `compute_cosine_similarity()`. Store embeddings alongside cached responses. In-memory: linear scan of all cached embeddings. Redis: store embeddings in a separate key `agentkit:llm_cache_emb:{sha256_hex}`.
|
||
|
||
4. **Cache write policy**: Only cache responses where `temperature == 0` (deterministic). For temperature > 0, only exact-match cache applies (no semantic match, since outputs are non-deterministic).
|
||
|
||
5. **Cache invalidation**: TTL-based (configurable, default 3600s for exact, 86400s for semantic). Manual invalidation via `invalidate(pattern=None)` for admin operations.
|
||
|
||
**Patterns to follow:**
|
||
- `EmbeddingCache` in `src/agentkit/memory/embedder.py` — LRU + TTL pattern
|
||
- `create_session_store()` factory in `src/agentkit/session/store.py` — backend factory pattern
|
||
- `RedisSessionStore._get_redis()` — lazy Redis initialization
|
||
|
||
**Test scenarios:**
|
||
- Exact match: same messages + model → cache hit, returns identical response
|
||
- Exact miss: different messages → cache miss, calls provider, writes to cache
|
||
- Semantic match: paraphrased question (similarity > 0.92) → cache hit
|
||
- Semantic miss: unrelated question (similarity < 0.6) → cache miss
|
||
- Temperature > 0: only exact match attempted, no semantic match
|
||
- TTL expiry: cached entry expires after TTL, next request is a miss
|
||
- Redis unavailable: falls back to in-memory cache with warning log
|
||
- Cache with tool_calls: response containing tool_calls is cached correctly
|
||
- Concurrent access: two concurrent requests for same key don't cause double-write issues
|
||
|
||
**Verification:** Unit tests pass; cache hit rate metric is observable; no change to `LLMGateway` public API.
|
||
|
||
---
|
||
|
||
### U2. LLM Cache Integration
|
||
|
||
**Goal:** Integrate `LLMCache` into `LLMGateway.chat()` transparently, with usage tracking on cache hits.
|
||
|
||
**Dependencies:** U1
|
||
|
||
**Files:**
|
||
- `src/agentkit/llm/gateway.py` (modify) — inject cache check before provider call, cache write after provider response
|
||
- `src/agentkit/llm/config.py` (modify) — add `CacheConfig` to `LLMConfig`
|
||
- `src/agentkit/server/app.py` (modify) — pass cache config to `LLMGateway`
|
||
- `tests/unit/llm/test_gateway_cache.py` (new) — integration tests for cached gateway
|
||
|
||
**Approach:**
|
||
|
||
Architecture design before coding:
|
||
|
||
1. **Insertion point reasoning**: Cache check must happen AFTER `LLMRequest` construction (line ~79 in gateway.py) but BEFORE provider call (line ~87). This ensures all request normalization (alias resolution, model fallback list) has completed. Cache write happens AFTER response validation but BEFORE usage tracking.
|
||
|
||
2. **Cache hit usage tracking**: On cache hit, call `_usage_tracker.record()` with the original `usage` data from the cached response but with `cost=0` and `latency_ms` from cache lookup time. This preserves usage query integrity — `get_usage()` still shows all requests, just with zero cost for cached ones.
|
||
|
||
3. **Stream handling**: `chat_stream()` is NOT cached in this iteration. Streaming requires collecting all chunks before caching, which adds latency and complexity. Document this as a known limitation.
|
||
|
||
4. **Configuration integration**: Add `CacheConfig` dataclass with `enabled: bool = False`, `backend: str = "auto"`, `exact_ttl: int = 3600`, `semantic_ttl: int = 86400`, `similarity_threshold: float = 0.92`, `max_entries: int = 10000`. Nest under `LLMConfig.cache`.
|
||
|
||
**Patterns to follow:**
|
||
- `LLMConfig` dataclass + `from_dict()` pattern for config
|
||
- `LLMGateway.__init__()` dependency injection pattern
|
||
|
||
**Test scenarios:**
|
||
- Cache disabled: requests pass through to provider normally
|
||
- Cache enabled, first request: cache miss, provider called, response cached
|
||
- Cache enabled, second identical request: cache hit, provider NOT called
|
||
- Cache hit usage tracking: usage record has 0 cost, correct token counts
|
||
- Cache miss + fallback: primary model fails, fallback model response cached under fallback model key
|
||
- Config from YAML: `LLMConfig.from_dict({"cache": {"enabled": true}})` works correctly
|
||
|
||
**Verification:** Integration tests pass; `LLMGateway.chat()` returns same `LLMResponse` shape whether cached or not; usage tracking includes cache hits.
|
||
|
||
---
|
||
|
||
### U3. Semantic Router
|
||
|
||
**Goal:** Implement embedding-based semantic routing as Layer 1.5 in `CostAwareRouter`, using existing `OpenAIEmbedder` and `compute_cosine_similarity()`.
|
||
|
||
**Dependencies:** None (independent of U1/U2, uses existing embedding infrastructure)
|
||
|
||
**Files:**
|
||
- `src/agentkit/chat/semantic_router.py` (new) — `SemanticRouter` class, `SkillEmbeddingIndex`
|
||
- `src/agentkit/chat/skill_routing.py` (modify) — integrate Layer 1.5 into `CostAwareRouter.route()`
|
||
- `tests/unit/chat/test_semantic_router.py` (new) — unit tests for semantic router
|
||
|
||
**Approach:**
|
||
|
||
Architecture design before coding:
|
||
|
||
1. **SkillEmbeddingIndex design reasoning**: Pre-compute embeddings for all registered skills at initialization. Source text: `f"{description} | {' '.join(keywords)} | {' '.join(capability_tags)}"`. Store as `dict[str, tuple[list[float], str]]` (skill_name → (embedding, source_text)). On skill registration/update, re-embed only the changed skill. This avoids O(n) embedding computation per query.
|
||
|
||
2. **Query-time flow**: Embed user query → compute cosine similarity against all skill embeddings → return top match if above threshold. This is O(n) in number of skills, but with <100 skills and 1536-dim vectors, this takes <5ms on CPU. No need for approximate nearest neighbor (ANN) index at this scale.
|
||
|
||
3. **Threshold design**: Three zones:
|
||
- `similarity > 0.85`: HIGH confidence → return skill match directly, skip Layer 2 LLM
|
||
- `0.6 <= similarity <= 0.85`: MEDIUM confidence → pass skill hint to Layer 2, reducing LLM classification tokens
|
||
- `similarity < 0.6`: LOW confidence → no semantic signal, Layer 2 runs unmodified
|
||
|
||
4. **Integration into CostAwareRouter**: Modify `route()` method. After Layer 1 (`_classify_merged()`), if complexity is medium (0.3-0.7), call `semantic_router.route(query)`. Based on confidence zone, either return directly or enhance the Layer 2 prompt with skill hint.
|
||
|
||
5. **Embedding provider**: Use `OpenAIEmbedder` by default. Support `MockEmbedder` for testing. Embedder is injected via constructor, not created internally.
|
||
|
||
**Patterns to follow:**
|
||
- `OpenAIEmbedder` + `EmbeddingCache` pattern for embedding computation
|
||
- `compute_cosine_similarity()` in `src/agentkit/utils/vector_math.py`
|
||
- `CostAwareRouter` constructor injection pattern
|
||
|
||
**Test scenarios:**
|
||
- Exact skill match: query "生成一篇关于AI的文章" matches `content_generator` skill (sim > 0.85)
|
||
- Partial skill match: query "优化内容" matches `geo_optimizer` skill (sim 0.6-0.85), skill hint passed to LLM
|
||
- No skill match: query "今天天气怎么样" has sim < 0.6 for all skills, Layer 2 runs normally
|
||
- Skill registration: new skill added → embedding computed and indexed
|
||
- Skill update: skill description changed → embedding re-computed
|
||
- Empty skill registry: semantic router returns None gracefully
|
||
- Embedder failure: OpenAIEmbedder throws error → semantic router logs warning, returns None, Layer 2 runs normally
|
||
- Chinese query: "帮我写一篇文章" matches content_generator skill correctly
|
||
|
||
**Verification:** Semantic router returns correct skill matches; Layer 2 LLM calls reduced by >50% for medium-complexity queries; no regression in routing accuracy.
|
||
|
||
---
|
||
|
||
### U4. UsageStore Persistence
|
||
|
||
**Goal:** Persist UsageTracker records to Redis, with in-memory fallback and efficient aggregation queries.
|
||
|
||
**Dependencies:** None
|
||
|
||
**Files:**
|
||
- `src/agentkit/llm/usage_store.py` (new) — `UsageStore` Protocol, `InMemoryUsageStore`, `RedisUsageStore`
|
||
- `src/agentkit/llm/providers/tracker.py` (modify) — delegate to `UsageStore` backend
|
||
- `tests/unit/llm/test_usage_store.py` (new) — unit tests for usage store backends
|
||
|
||
**Approach:**
|
||
|
||
Architecture design before coding:
|
||
|
||
1. **Redis data model reasoning**: Use Redis Hash per date for time-partitioned storage. Key: `agentkit:usage:{YYYY-MM-DD}`, field: `{agent_name}:{model}`, value: JSON `{prompt_tokens, completion_tokens, total_tokens, cost, latency_ms, count}`. Write via pipeline: `HINCRBYFLOAT` for numeric fields + `HINCRBY` for count. This is O(1) per write, atomic, and naturally partitions by date.
|
||
|
||
2. **Aggregation query design**: For `get_usage(agent=None, start=None, end=None)`: scan date keys in range via `HGETALL`, filter by agent/model in application code, aggregate in memory. For single-agent queries, use field prefix matching. This is O(days × agents) which is acceptable for dashboard queries.
|
||
|
||
3. **UsageStore Protocol**: Define `record(agent, model, usage: UsageRecord) -> None`, `query(agent=None, model=None, start=None, end=None) -> list[UsageRecord]`, `get_summary(agent=None, start=None, end=None) -> UsageSummary`. Both sync and async versions (sync for backward compat, async for Redis).
|
||
|
||
4. **Migration from UsageTracker**: `UsageTracker` becomes a thin wrapper that delegates to `UsageStore`. Existing `record()` and `get_usage()` APIs preserved. Internal `_records` list replaced by store backend.
|
||
|
||
5. **TTL management**: Each date key gets TTL of 90 days (configurable). This prevents unbounded Redis memory growth while preserving 3 months of usage data.
|
||
|
||
**Patterns to follow:**
|
||
- `SessionStore` Protocol in `src/agentkit/session/store.py` — Protocol definition pattern
|
||
- `RedisSessionStore._get_redis()` — lazy Redis initialization
|
||
- `create_session_store()` — factory function pattern
|
||
- `agentkit:usage:` key namespace convention
|
||
|
||
**Test scenarios:**
|
||
- Record and query: record usage → query returns matching records
|
||
- Date partitioning: records on different dates stored in different keys
|
||
- Aggregation: multiple records for same agent/model aggregated correctly
|
||
- Agent filter: query with agent filter returns only that agent's records
|
||
- Date range filter: query with start/end returns only records in range
|
||
- TTL: date keys have correct TTL set
|
||
- Redis unavailable: falls back to in-memory store with warning
|
||
- Concurrent writes: two concurrent records for same agent/model don't lose data
|
||
- Empty query: query with no matching records returns empty list
|
||
|
||
**Verification:** Usage data survives process restart; `get_usage()` returns same shape as before; Redis memory usage bounded by TTL.
|
||
|
||
---
|
||
|
||
### U5. CascadeStateStore Persistence
|
||
|
||
**Goal:** Persist CascadeDetector state to Redis using atomic operations, enabling multi-instance cascade detection.
|
||
|
||
**Dependencies:** None
|
||
|
||
**Files:**
|
||
- `src/agentkit/quality/cascade_store.py` (new) — `CascadeStateStore` Protocol, `InMemoryCascadeStore`, `RedisCascadeStore`
|
||
- `src/agentkit/quality/cascade_detector.py` (modify) — delegate to `CascadeStateStore` backend
|
||
- `tests/unit/quality/test_cascade_store.py` (new) — unit tests for cascade store backends
|
||
|
||
**Approach:**
|
||
|
||
Architecture design before coding:
|
||
|
||
1. **Redis data model reasoning**: Use simple string keys with INCR for atomic counting. Key: `agentkit:cascade:{session_id}:interactions` (INCR + TTL), `agentkit:cascade:{session_id}:depth` (GET/SET + TTL). TTL aligned with session TTL (default 86400s). INCR is atomic — no race conditions across instances.
|
||
|
||
2. **Protocol design**: `CascadeStateStore` with `increment_interactions(session_id) -> int`, `get_interactions(session_id) -> int`, `set_depth(session_id, depth) -> None`, `get_depth(session_id) -> int`, `reset(session_id) -> None`, `get_stats(session_id) -> CascadeStats`.
|
||
|
||
3. **Integration into CascadeDetector**: Replace internal `_interaction_counts` and `_loop_depths` dicts with `CascadeStateStore` backend. All methods delegate to store. `CascadeDetector` becomes stateless — all state lives in the store.
|
||
|
||
4. **Session TTL alignment**: When `increment_interactions()` is called, refresh the key TTL to match session TTL. This ensures state is cleaned up when sessions expire.
|
||
|
||
**Patterns to follow:**
|
||
- Same Protocol + factory + fallback pattern as U4
|
||
- Redis INCR atomic operation pattern
|
||
- `agentkit:cascade:` key namespace
|
||
|
||
**Test scenarios:**
|
||
- Increment and get: increment interactions → get returns correct count
|
||
- Set and get depth: set depth → get returns correct depth
|
||
- Reset: reset session → interactions and depth both cleared
|
||
- TTL: keys have TTL set, expire after session timeout
|
||
- Multi-instance: two instances incrementing same session see consistent count
|
||
- Redis unavailable: falls back to in-memory store
|
||
- Session isolation: different sessions have independent state
|
||
|
||
**Verification:** Cascade detection state survives process restart; multi-instance deployment detects cascades correctly; no false positives from state loss.
|
||
|
||
---
|
||
|
||
### U6. EvolutionStore Interface Unification
|
||
|
||
**Goal:** Unify `EvolutionStore` and `PersistentEvolutionStore` interfaces, add PostgreSQL backend with full feature set.
|
||
|
||
**Dependencies:** None
|
||
|
||
**Files:**
|
||
- `src/agentkit/evolution/evolution_store.py` (modify) — define unified `EvolutionStoreProtocol`, refactor existing stores
|
||
- `src/agentkit/evolution/models.py` (modify) — add `SkillVersionModel` and `ABTestResultModel` to async PG models
|
||
- `src/agentkit/evolution/pg_store.py` (new) — `PostgreSQLEvolutionStore` implementing unified Protocol with async SQLAlchemy
|
||
- `tests/unit/evolution/test_unified_store.py` (new) — tests for unified interface
|
||
|
||
**Approach:**
|
||
|
||
Architecture design before coding:
|
||
|
||
1. **Protocol design reasoning**: Current `EvolutionStore` (async PG) has `record()`, `rollback()`, `list_events()`. `PersistentEvolutionStore` (sync SQLite) adds `record_skill_version()`, `list_skill_versions()`, `record_ab_test_result()`, `get_ab_test_results()`. The unified Protocol must include ALL methods from both. Each backend implements what it can; unsupported methods raise `NotImplementedError` with clear message.
|
||
|
||
2. **PostgreSQL model migration**: Add `SkillVersionModel` and `ABTestResultModel` to `src/agentkit/evolution/models.py` using async SQLAlchemy (matching `EpisodeModel` pattern in memory/models.py). These models already exist for SQLite; the PG versions use the same schema but with async engine.
|
||
|
||
3. **PostgreSQLEvolutionStore**: New class using async SQLAlchemy session (injected via constructor, same pattern as existing `EvolutionStore`). Implements all Protocol methods. Uses `run_in_executor` for any sync ORM operations if needed.
|
||
|
||
4. **Factory update**: `create_evolution_store(backend="memory"|"sqlite"|"postgresql", ...)` returns the appropriate backend. `"postgresql"` creates `PostgreSQLEvolutionStore` with async engine.
|
||
|
||
5. **Backward compatibility**: Existing `EvolutionStore` class is not removed — it becomes an internal implementation detail. The Protocol is the public interface. Code using `EvolutionStore` directly continues to work.
|
||
|
||
**Patterns to follow:**
|
||
- `EpisodeModel` in `src/agentkit/memory/models.py` — async PG model pattern
|
||
- `create_evolution_store()` factory — extend with new backend
|
||
- `PersistentEvolutionStore._run_sync()` — sync/async bridge pattern
|
||
|
||
**Test scenarios:**
|
||
- Protocol compliance: all backends implement all Protocol methods
|
||
- PG store: record event → list events returns recorded event
|
||
- PG store: record skill version → list versions returns version history
|
||
- PG store: record AB test result → get results returns test data
|
||
- SQLite store: existing functionality preserved after refactor
|
||
- Memory store: existing functionality preserved after refactor
|
||
- Factory: `create_evolution_store(backend="postgresql")` returns correct type
|
||
- PG unavailable: falls back to SQLite with warning
|
||
|
||
**Verification:** All backends pass unified Protocol compliance test; existing evolution tests pass; PG store supports skill_version and ab_test operations.
|
||
|
||
---
|
||
|
||
### U7. Configuration Integration and End-to-End Verification
|
||
|
||
**Goal:** Wire all three features into the application configuration, add `agentkit.yaml` schema support, and verify end-to-end behavior.
|
||
|
||
**Dependencies:** U1, U2, U3, U4, U5, U6
|
||
|
||
**Files:**
|
||
- `src/agentkit/server/app.py` (modify) — initialize cache, usage store, cascade store with config
|
||
- `src/agentkit/cli/main.py` (modify) — pass config to gateway and router
|
||
- `agentkit.yaml` (modify) — add cache, semantic_routing, usage_store, cascade_store config sections
|
||
- `tests/integration/test_p0_hardening.py` (new) — end-to-end integration tests
|
||
|
||
**Approach:**
|
||
|
||
1. **Configuration schema**: Add to `agentkit.yaml`:
|
||
```yaml
|
||
llm:
|
||
cache:
|
||
enabled: true
|
||
backend: "auto" # auto | redis | memory
|
||
exact_ttl: 3600
|
||
semantic_ttl: 86400
|
||
similarity_threshold: 0.92
|
||
max_entries: 10000
|
||
|
||
routing:
|
||
semantic:
|
||
enabled: true
|
||
similarity_high: 0.85 # direct match threshold
|
||
similarity_low: 0.6 # hint threshold
|
||
|
||
usage_store:
|
||
backend: "auto" # auto | redis | memory
|
||
ttl_days: 90
|
||
|
||
cascade_store:
|
||
backend: "auto" # auto | redis | memory
|
||
session_ttl: 86400
|
||
|
||
evolution_store:
|
||
backend: "auto" # auto | postgresql | sqlite | memory
|
||
```
|
||
|
||
2. **Application wiring**: In `app.py` lifespan, initialize all stores and inject into gateway/router. Follow existing pattern of creating components from config.
|
||
|
||
3. **End-to-end verification**: Integration test that exercises the full flow: user query → semantic routing → LLM cache → usage tracking → cascade detection → evolution logging.
|
||
|
||
**Test scenarios:**
|
||
- Full flow with Redis: all features use Redis backend, data persists across simulated restart
|
||
- Full flow without Redis: all features fall back to in-memory, no errors
|
||
- Config from YAML: `agentkit.yaml` parsed correctly, all features configured
|
||
- Cache + routing interaction: cached response for semantically routed query works correctly
|
||
- Usage tracking with cache: cached requests show 0 cost in usage summary
|
||
- Cascade detection across instances: simulated multi-instance scenario detects cascade correctly
|
||
|
||
**Verification:** All integration tests pass; application starts with new config; features degrade gracefully when backends unavailable.
|
||
|
||
---
|
||
|
||
## Risks & Mitigations
|
||
|
||
| Risk | Impact | Likelihood | Mitigation |
|
||
|------|--------|-----------|------------|
|
||
| Semantic cache returns stale/wrong response | High — user gets incorrect answer | Medium — embedding similarity doesn't guarantee semantic equivalence | Default to temperature=0 only for semantic cache; configurable threshold; TTL expiry; admin invalidation API |
|
||
| Redis single point of failure | High — all persistence lost | Low — Redis is typically HA | Auto-fallback to in-memory; health check in doctor command; alert on fallback activation |
|
||
| Embedding API latency adds to routing time | Medium — slower routing for first query | Medium — embedding API ~100ms | Pre-compute skill embeddings; cache query embeddings; async embedding with timeout |
|
||
| UsageStore Redis memory growth | Medium — Redis OOM | Low — TTL + date partitioning bounds growth | 90-day TTL default; monitoring on Redis memory; configurable TTL |
|
||
| EvolutionStore interface unification breaks existing code | High — evolution system stops working | Low — Protocol is backward compatible | Keep existing classes as internal implementations; comprehensive test coverage before refactor |
|
||
|
||
---
|
||
|
||
## Open Questions
|
||
|
||
- Should semantic cache also cache streaming responses (requires chunk collection)? Deferred — current plan only caches non-streaming `chat()`.
|
||
- Should UsageStore support real-time streaming of usage data (e.g., via Redis Pub/Sub)? Deferred — current plan only supports query-based access.
|
||
- What is the optimal embedding model for Chinese+English mixed text? `text-embedding-3-small` is adequate but not optimal. Consider `bge-m3` or `multilingual-e5` as alternatives. Deferred to implementation-time benchmarking.
|
||
|
||
---
|
||
|
||
## Sources & Research
|
||
|
||
- Industry benchmarking: LangChain, Dify, CrewAI, Letta, AutoGen feature comparison (2025-2026)
|
||
- Project audit: 12 core files analyzed across memory, evolution, routing, quality, and LLM subsystems
|
||
- Existing patterns: `EmbeddingCache`, `RedisSessionStore`, `create_evolution_store()`, `SessionStore` Protocol
|