a3cecd4b50
fix(review): apply P0/P2 findings from dual-agent review
...
- Dockerfile: split ENTRYPOINT/CMD to align with docker-compose serve
- test_termbase: guard jieba import with pytest.importorskip
- orchestrator: mark silent review-degradation with [DEGRADED] prefix
- chat.py: accurate ExecutionMode log message
- agentkit.yaml: document OTel exporter config
- skill_routing: replace 12 Any with object/typed (AGENTS.md compliance)
- AssistantText.vue: add aria-live/role for a11y
2026-06-30 14:27:46 +08:00
e3ae2f3a56
feat(rag_platform): U8 — TaskIQ async task integration
...
Add tasks.py: TaskManager with vectorize/batch_index tasks, per-user concurrency limits, degraded mode (sync execution without broker), WorkerSweeper for timeout detection, error message sanitization
Add taskiq>=0.11 and taskiq-redis>=0.5 to pyproject.toml
Task parameter schema validation (VectorizeTaskParams, BatchIndexTaskParams)
Tests: 41 new tests, 289 total passing
2026-06-25 12:58:51 +08:00
d026a91f43
feat(rag_platform): U6 — hit processing mode + KB settings
...
Add hit_processing.py: HitProcessor with model_opt (LLM-generated) and direct (concatenated chunks) modes, with in-process cache
Add settings.py: KBSettings/KBSettingsUpdate models, KBSettingsStore with async CRUD
Add KB settings endpoints to kb_management.py: GET/PUT /kb-management/kbs/{kb_id}/settings with owner-only modification
Tests: 43 new tests (25 hit_processing + 18 settings), 293 total passing
2026-06-25 12:44:47 +08:00
5c562dbff3
feat(rag_platform): U5 — rerank + question generation + termbase
...
Add rerank.py: Reranker with Cohere/BGE provider support, data export risk annotation, graceful degradation
Add question_gen.py: LLM-based question generation following ContextualChunker pattern, with caching
Add termbase.py: jieba custom dictionary management, add/remove/load terms
Tests: 58 new tests (14 rerank + 19 question_gen + 25 termbase), 205 total passing
2026-06-25 12:31:43 +08:00
fb9f16d6e5
feat(rag_platform): U4 — dual-index retrieval (pgvector semantic + PG fulltext jieba)
...
Add fulltext.py: jieba tokenization + tsvector write/query
Add retrieval.py: RetrievalEngine with embedding/keywords/blend modes
Update models.py: add RetrievalRequest model
Tests: 35 new tests, 147 total passing
2026-06-25 12:20:48 +08:00
b55c896794
feat(rag_platform): U3+U7 — document processing pipeline + upload security
...
U3: Document processing pipeline (document_processor.py)
- DocumentProcessor class wrapping parse → segment → vectorize
- parse() uses memory/document_loader.py for multi-format extraction
- segment() uses LlamaIndex SentenceSplitter
- preview() returns chunks for read-only preview (no vectorization)
- vectorize() embeds chunks and stores in pgvector (all-or-nothing)
- process() orchestrates full pipeline with status transitions:
pending → parsing → segmenting → vectorizing → indexed | failed
U7: Upload security & content sanitization (sanitize.py)
- ALLOWED_FILE_TYPES whitelist (pdf/docx/xlsx/pptx/txt/md/csv/html)
- MAX_FILE_SIZE 50MB limit
- validate_file_type() / validate_file_size() guards
- check_zip_bomb() for ZIP-based formats (ratio > 100:1 or > 500MB)
- check_image_bomb() for pixel count > 100MP (PNG/JPEG/GIF header parsing)
- is_safe_ip() SSRF protection (loopback/RFC1918/link-local/ULA denied)
- sanitize_markdown() removes dangerous HTML tags (script/iframe/object/embed)
- sanitize_content() main entry point for text format sanitization
- parse_xml_safe() XXE protection (forbid_dtd/forbid_entities/forbid_external)
Preview API (preview.py)
- PreviewChunk / PreviewResult Pydantic models
- generate_preview() returns read-only segmentation preview
Tests: 112 tests passing (45 new + 67 existing)
- test_sanitize.py: file type/size, markdown sanitization, SSRF, zip/image bomb
- test_document_processor.py: parse/segment, preview, vectorize, failure status
2026-06-25 11:21:42 +08:00
c1a21f57a1
feat(rag_platform): U2 — KB persistence + per-KB ACL
...
Add PostgreSQL-backed KB store replacing in-memory KnowledgeSourceStore:
- models.py: ORM models (KBModel, DocumentModel, KBAclModel) using
SQLAlchemy 2 DeclarativeBase + Mapped style
- store.py: KBStore with async CRUD for KBs and documents,
create_kb creates owner ACL in same transaction
- acl.py: filter_kb_by_user_acl(), grant_access(), revoke_access(),
list_acl() — follows filter_kb_sources_by_department pattern
Schema: rag_platform_kbs, rag_platform_documents, rag_platform_kb_acl
with FK CASCADE on kb_id. UniqueConstraint on (kb_id, user_id).
Tests: 23 unit tests covering KB CRUD, document operations, ACL
filtering, grant/revoke. All 37 rag_platform tests pass.
2026-06-25 11:01:04 +08:00
27d0184392
feat(rag_platform): U1 — RAG platform skeleton + LlamaIndex integration
...
Create src/agentkit/rag_platform/ module with:
- models.py: Pydantic domain models (KB, Document, Chunk, QueryResult)
- indexing.py: PGVectorStore wrapper with explicit table name
(rag_platform_kb_chunks) for schema isolation from episodic_memory
- pipeline.py: RAGPipeline wrapping LlamaIndex IngestionPipeline
(SentenceSplitter + embedding + vector store)
Add dependencies: llama-index-core, llama-index-vector-stores-postgres,
llama-index-embeddings-openai, pgvector, jieba.
Tests: 14 unit tests covering models, indexing (URL conversion, table
name isolation, embed_dim), and pipeline (ingest, query, chunk params).
2026-06-25 10:49:35 +08:00
chiguyong