harden code and add messenger services and ne README.md and SYSTEM.,d
This commit is contained in:
7
.env
7
.env
@@ -44,3 +44,10 @@ AI_LOG_CONTEXT=false
|
||||
# DATABASE_URL="mysql://app:!ChangeMe!@127.0.0.1:3306/app?serverVersion=10.11.2-MariaDB&charset=utf8mb4"
|
||||
DATABASE_URL="postgresql://app:!ChangeMe!@127.0.0.1:5432/app?serverVersion=16&charset=utf8"
|
||||
###< doctrine/doctrine-bundle ###
|
||||
|
||||
###> symfony/messenger ###
|
||||
# Choose one of the transports below
|
||||
# MESSENGER_TRANSPORT_DSN=amqp://guest:guest@localhost:5672/%2f/messages
|
||||
# MESSENGER_TRANSPORT_DSN=redis://localhost:6379/messages
|
||||
MESSENGER_TRANSPORT_DSN=doctrine://default?auto_setup=0
|
||||
###< symfony/messenger ###
|
||||
|
||||
604
README.md
604
README.md
@@ -1,250 +1,430 @@
|
||||
# mitho AI Agent (Alpha Version)
|
||||
**Hybrid RAG System auf Symfony-Basis mit Vektor- & Keyword-Retrieval**
|
||||
# mitho AI Agent – Developer Deep Dive
|
||||
|
||||
Enterprise Hybrid RAG System (Symfony + NDJSON + FAISS)
|
||||
|
||||
---
|
||||
|
||||
## Überblick
|
||||
# 1. System Overview
|
||||
|
||||
Der **mitho AI Agent** ist ein produktionsreifes, Symfony-basiertes RAG-System (Retrieval Augmented Generation), das KI nicht frei „raten“ lässt, sondern Antworten strikt auf Basis eines kontrollierten Wissenspools erzeugt.
|
||||
This system implements a deterministic, governance-stable Retrieval Augmented Generation (RAG) architecture based on:
|
||||
|
||||
> **Leitsatz:**
|
||||
> *„Wir nutzen KI nicht, um kreativ zu raten, sondern um verlässlich auf Basis Ihres Wissens zu antworten.“*
|
||||
- Symfony (PHP backend)
|
||||
- NDJSON-based knowledge index
|
||||
- Full FAISS vector rebuild strategy
|
||||
- Hybrid retrieval (keyword + vector)
|
||||
- Deterministic ingest pipeline
|
||||
- Strict versioning & guardrails
|
||||
- Lock-based reindex protection
|
||||
|
||||
Das System kombiniert:
|
||||
|
||||
- Large Language Model (LLM, z. B. Qwen via Ollama)
|
||||
- Keyword-basiertes Retrieval
|
||||
- FAISS-Vektor-Suche
|
||||
- Versionierte Wissensstruktur (Chunks + Index)
|
||||
- Streaming-Ausgabe via Server-Sent Events (SSE)
|
||||
- Persistente Chat-Historie pro Client
|
||||
No incremental vector mutation is allowed.
|
||||
FAISS is always rebuilt from `index.ndjson`.
|
||||
|
||||
---
|
||||
|
||||
# Architektur
|
||||
# 2. High-Level Architecture
|
||||
|
||||
## 1. Backend
|
||||
User Query
|
||||
→ Hybrid Retrieval
|
||||
→ Context Assembly
|
||||
→ Prompt Builder
|
||||
→ LLM
|
||||
→ Streaming Response (SSE)
|
||||
|
||||
**Technologie**
|
||||
|
||||
- PHP 8.2+
|
||||
- Symfony 7.4
|
||||
- Monolog Logging
|
||||
- Symfony Cache
|
||||
- Session Support
|
||||
|
||||
### Zentrale Komponenten
|
||||
|
||||
| Komponente | Aufgabe |
|
||||
|------------|----------|
|
||||
| `AgentRunner` | Orchestriert Prompt, Kontext & LLM |
|
||||
| `PromptBuilder` | Baut System- & User-Prompt |
|
||||
| `ContextService` | Historienverwaltung |
|
||||
| `ChunkKeywordRetriever` | Keyword-Scoring |
|
||||
| `VectorSearchClient` | Python-FAISS-Anbindung |
|
||||
| `KnowledgeIngestService` | Dokument → Chunks |
|
||||
| `ChunkIndexWriter` | index.json Verwaltung |
|
||||
| `CachedRetriever` | Performance-Optimierung |
|
||||
Knowledge Flow:
|
||||
Document → Version → Extract → Chunk → NDJSON → FAISS → Retrieval
|
||||
|
||||
---
|
||||
|
||||
## 2. Hybrid Retrieval (Produktionsarchitektur)
|
||||
|
||||
Das System nutzt eine **hybride Sucharchitektur**:
|
||||
|
||||
### A) Keyword-Retrieval (führend)
|
||||
|
||||
- Stopword-Filter
|
||||
- Lemma-Logik
|
||||
- Score-Berechnung
|
||||
- deterministische Gewichtung
|
||||
|
||||
### B) Vektor-Retrieval (ergänzend)
|
||||
|
||||
- SentenceTransformer: `all-MiniLM-L6-v2`
|
||||
- FAISS Index (Inner Product)
|
||||
- Normalisierte Embeddings
|
||||
- Top-K Suche
|
||||
|
||||
### Retrieval-Flow
|
||||
|
||||
1. User Prompt
|
||||
2. Keyword-Scoring
|
||||
3. FAISS-Suche
|
||||
4. Score-Fusion
|
||||
5. Top-N Chunks
|
||||
6. Kontextaufbau
|
||||
7. LLM-Antwort
|
||||
|
||||
---
|
||||
|
||||
## 3. Wissensarchitektur
|
||||
# 3. Directory Structure (Knowledge Layer)
|
||||
|
||||
```
|
||||
var/knowledge/
|
||||
├── uploads/
|
||||
├── chunks/
|
||||
├── manifest.json
|
||||
└── index.json
|
||||
├── index.ndjson
|
||||
├── index_meta.json
|
||||
├── vector.index
|
||||
└── vector_meta.json
|
||||
```
|
||||
|
||||
### Prinzipien
|
||||
|
||||
- Dokumente sind Primärquelle
|
||||
- Chunks sind abgeleitete Artefakte
|
||||
- `index.json` ist Single Source of Truth
|
||||
- Re-Ingest ist deterministisch
|
||||
- Keine manuelle Chunk-Manipulation
|
||||
|
||||
---
|
||||
|
||||
## 4. Vektor-Ingest
|
||||
# 4. NDJSON Index
|
||||
|
||||
CLI Command:
|
||||
## 4.1 index.ndjson
|
||||
|
||||
- Single Source of Truth
|
||||
- One JSON object per line
|
||||
- Streaming-readable
|
||||
- No JSON array wrapper
|
||||
- Scales beyond 200k chunks
|
||||
|
||||
Each line contains:
|
||||
|
||||
```json
|
||||
{
|
||||
"chunk_id": "uuid",
|
||||
"document_id": "uuid",
|
||||
"version": 3,
|
||||
"text": "...",
|
||||
"meta": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
NDJSON enables:
|
||||
- Append-based writes
|
||||
- Compaction per document
|
||||
- Memory-safe streaming
|
||||
- Deterministic rebuilds
|
||||
|
||||
---
|
||||
|
||||
# 5. Index Metadata
|
||||
|
||||
## index_meta.json
|
||||
|
||||
Managed by:
|
||||
|
||||
- IndexMetaManager
|
||||
- IndexConfiguration
|
||||
|
||||
Contains:
|
||||
|
||||
- index_version
|
||||
- embedding_model
|
||||
- embedding_dimension
|
||||
- chunk_size
|
||||
- overlap
|
||||
- scoring_version
|
||||
- index_format
|
||||
|
||||
If configuration changes → Global Reindex required.
|
||||
|
||||
Guarded by:
|
||||
`IndexStructureChangedException`
|
||||
|
||||
---
|
||||
|
||||
# 6. Ingest Pipeline
|
||||
|
||||
## 6.1 Core Services
|
||||
|
||||
| Service | Responsibility |
|
||||
|----------|----------------|
|
||||
| DocumentService | Document lifecycle |
|
||||
| DocumentVersionRepository | Version persistence |
|
||||
| KnowledgeIngestService | Chunk generation |
|
||||
| SimpleChunker | Deterministic splitting |
|
||||
| TextNormalizer | Text cleanup |
|
||||
| StopWords | Keyword filtering |
|
||||
| ChunkManager | NDJSON append + compaction |
|
||||
| ChunkWriter | Chunk persistence |
|
||||
| IngestFlow | Step orchestration |
|
||||
| IngestOrchestrator | Full ingest coordination |
|
||||
| IngestJobService | Job tracking |
|
||||
| LockService | Concurrency guard |
|
||||
|
||||
---
|
||||
|
||||
## 6.2 Local Ingest
|
||||
|
||||
Used when:
|
||||
- A single document version changes
|
||||
|
||||
Process:
|
||||
|
||||
1. Extract document
|
||||
2. Normalize text
|
||||
3. Chunk deterministically
|
||||
4. Remove previous chunks of document_id
|
||||
5. Append new chunks to index.ndjson
|
||||
6. Rebuild FAISS completely
|
||||
|
||||
index_version does NOT change.
|
||||
|
||||
---
|
||||
|
||||
## 6.3 Global Reindex
|
||||
|
||||
Used when:
|
||||
- Embedding model changes
|
||||
- Chunk size changes
|
||||
- Overlap changes
|
||||
- Scoring logic changes
|
||||
- index_format changes
|
||||
|
||||
Process:
|
||||
|
||||
1. Re-extract all active document versions
|
||||
2. Recreate full index.ndjson
|
||||
3. Rebuild FAISS
|
||||
4. index_version++
|
||||
|
||||
---
|
||||
|
||||
# 7. Vector Architecture
|
||||
|
||||
## 7.1 vector_ingest.py
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Stream-read index.ndjson
|
||||
- Extract text + chunk_id
|
||||
- Build embeddings
|
||||
- Normalize embeddings
|
||||
- Build FAISS IndexFlatIP
|
||||
- Write vector.index
|
||||
- Write vector.meta.json
|
||||
|
||||
Execution:
|
||||
|
||||
```bash
|
||||
python vector_ingest.py --index path/to/index.ndjson --out path/to/vector.index
|
||||
```
|
||||
|
||||
Characteristics:
|
||||
|
||||
- No partial updates
|
||||
- No incremental mutation
|
||||
- Always full rebuild
|
||||
- Batch size = 64
|
||||
- normalize_embeddings=True
|
||||
|
||||
---
|
||||
|
||||
## 7.2 vector_search.py
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Load vector.index
|
||||
- Load vector_meta.json
|
||||
- Encode query
|
||||
- Search top-K
|
||||
- Return JSON
|
||||
|
||||
Execution:
|
||||
|
||||
```bash
|
||||
python vector_search.py "query" 5
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```json
|
||||
[
|
||||
{ "chunk_id": "...", "score": 0.82 }
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7.3 VectorSearchClient (PHP)
|
||||
|
||||
- Executes Python search script
|
||||
- Parses JSON response
|
||||
- Returns structured results
|
||||
- Handles timeout + error states
|
||||
|
||||
---
|
||||
|
||||
# 8. Hybrid Retrieval
|
||||
|
||||
## 8.1 Components
|
||||
|
||||
| Class | Role |
|
||||
|--------|------|
|
||||
| NdjsonHybridRetriever | Orchestrator |
|
||||
| NdjsonKeywordSearch | Keyword scoring |
|
||||
| NdjsonChunkLookup | Chunk resolution |
|
||||
| VectorSearchClient | Vector bridge |
|
||||
| CachedRetriever | Cache layer |
|
||||
|
||||
---
|
||||
|
||||
## 8.2 Retrieval Flow
|
||||
|
||||
1. Extract terms (StopWords + normalization)
|
||||
2. Keyword scoring
|
||||
3. Vector search
|
||||
4. Score fusion
|
||||
5. Limit to N chunks
|
||||
6. Resolve chunk text
|
||||
7. Build LLM context
|
||||
|
||||
Keyword score remains primary signal.
|
||||
Vector score augments semantic similarity.
|
||||
|
||||
---
|
||||
|
||||
# 9. Document Extraction
|
||||
|
||||
Supported via:
|
||||
|
||||
- DocumentExtractorInterface
|
||||
- ExtractorResolver
|
||||
- PdfExtractor
|
||||
- DocumentLoader
|
||||
|
||||
Extraction must return clean UTF-8 text.
|
||||
Chunking must remain deterministic.
|
||||
|
||||
---
|
||||
|
||||
# 10. Admin Layer (Symfony)
|
||||
|
||||
## Controllers
|
||||
|
||||
- DashboardController
|
||||
- DocumentController
|
||||
- IngestJobController
|
||||
- SecurityController
|
||||
|
||||
## Entities
|
||||
|
||||
- Document
|
||||
- DocumentVersion
|
||||
- IngestJob
|
||||
- User
|
||||
|
||||
## Repositories
|
||||
|
||||
- DocumentVersionRepository
|
||||
- UserRepository
|
||||
|
||||
---
|
||||
|
||||
# 11. Concurrency & Locks
|
||||
|
||||
LockService ensures:
|
||||
|
||||
- No parallel reindex
|
||||
- No parallel ingest conflict
|
||||
- Controlled mutation of index.ndjson
|
||||
|
||||
File-based or service-based locking.
|
||||
|
||||
---
|
||||
|
||||
# 12. Determinism Rules
|
||||
|
||||
The system guarantees:
|
||||
|
||||
- Same documents + same config = identical index.ndjson
|
||||
- Same index.ndjson = identical FAISS
|
||||
- Same query + same index = identical results
|
||||
|
||||
No randomness.
|
||||
No adaptive mutation.
|
||||
No auto-learning.
|
||||
|
||||
---
|
||||
|
||||
# 13. LLM Integration
|
||||
|
||||
- Context strictly limited to retrieved chunks
|
||||
- PromptBuilder constructs deterministic system prompt
|
||||
- ContextService manages history
|
||||
- SSE streaming enabled
|
||||
- Model endpoint configurable
|
||||
|
||||
LLM never has direct access to full knowledge base.
|
||||
Only retrieved chunks are injected.
|
||||
|
||||
---
|
||||
|
||||
# 14. Scalability
|
||||
|
||||
Designed for:
|
||||
|
||||
- >200k chunks
|
||||
- Streaming NDJSON reads
|
||||
- Full FAISS rebuild
|
||||
- Cache layer for retrieval
|
||||
- Controlled memory usage
|
||||
|
||||
No full-array JSON loads.
|
||||
|
||||
---
|
||||
|
||||
# 15. Failure Modes
|
||||
|
||||
Handled via:
|
||||
|
||||
- Missing vector index detection
|
||||
- Structure drift detection
|
||||
- Lock collision detection
|
||||
- Embedding dependency checks
|
||||
- Python execution errors
|
||||
- Empty chunk fallback
|
||||
|
||||
---
|
||||
|
||||
# 16. Non-Goals
|
||||
|
||||
This system intentionally does NOT include:
|
||||
|
||||
- Online learning
|
||||
- Embedding mutation
|
||||
- Incremental FAISS update
|
||||
- Auto chunk merging
|
||||
- Self-modifying prompts
|
||||
|
||||
All structural changes require explicit reindex.
|
||||
|
||||
---
|
||||
|
||||
# 17. Design Philosophy
|
||||
|
||||
This is a governance-first RAG architecture:
|
||||
|
||||
- Deterministic
|
||||
- Reproducible
|
||||
- Drift-safe
|
||||
- Audit-friendly
|
||||
- Version-controlled
|
||||
|
||||
It prioritizes correctness and control over dynamic mutation.
|
||||
|
||||
---
|
||||
|
||||
# 18. Development Guidelines
|
||||
|
||||
When extending the system:
|
||||
|
||||
- Never mutate FAISS directly
|
||||
- Never edit index.ndjson manually
|
||||
- Always preserve determinism
|
||||
- Increment index_version only via Global Reindex
|
||||
- Guard all structural changes
|
||||
- Maintain streaming compatibility
|
||||
|
||||
---
|
||||
|
||||
# 19. CLI Commands (Symfony)
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
php bin/console mto:agent:vector:ingest
|
||||
```
|
||||
|
||||
Ablauf:
|
||||
|
||||
1. index.json lesen
|
||||
2. Chunk-Texte laden
|
||||
3. Embeddings erzeugen
|
||||
4. FAISS Index erstellen
|
||||
5. vector.index speichern
|
||||
6. vector_meta.json schreiben
|
||||
|
||||
---
|
||||
|
||||
## 5. LLM-Anbindung
|
||||
|
||||
Standardmäßig via Ollama.
|
||||
|
||||
Konfiguration über ENV:
|
||||
Custom commands follow namespace:
|
||||
|
||||
```
|
||||
AI_LLM_API_URL=
|
||||
AI_LLM_MODEL=
|
||||
AI_LLM_TIMEOUT=
|
||||
AI_DEBUG=
|
||||
AI_LOG_PROMPT=
|
||||
AI_LOG_CONTEXT=
|
||||
AI_HISTORY_DIR=
|
||||
mto:agent:*
|
||||
```
|
||||
|
||||
Features:
|
||||
|
||||
- Streaming-fähig
|
||||
- Konfigurierbarer Timeout
|
||||
- Denkmodus unterdrückbar
|
||||
- Historienintegration
|
||||
|
||||
---
|
||||
|
||||
## 6. Frontend
|
||||
# 20. Summary
|
||||
|
||||
Technologie:
|
||||
This system is a deterministic, enterprise-grade hybrid RAG engine with:
|
||||
|
||||
- Bootstrap
|
||||
- Marked (Markdown)
|
||||
- DOMPurify
|
||||
- SSE Streaming
|
||||
- NDJSON-based streaming index
|
||||
- Full FAISS rebuild strategy
|
||||
- Structured ingest pipeline
|
||||
- Hybrid retrieval
|
||||
- Admin governance layer
|
||||
- Strict guardrails
|
||||
|
||||
Features:
|
||||
|
||||
- Live-Streaming
|
||||
- Markdown-Rendering
|
||||
- Abbruch-Funktion
|
||||
- Chat-Verlauf
|
||||
- Client-ID per Cookie
|
||||
- Verlaufslöschung
|
||||
|
||||
---
|
||||
|
||||
## 7. Logging & Debug
|
||||
|
||||
Log-Datei:
|
||||
|
||||
```
|
||||
var/log/agent.log
|
||||
```
|
||||
|
||||
Optional aktivierbar:
|
||||
|
||||
- Prompt Logging
|
||||
- Kontext Logging
|
||||
- Debug-Modus
|
||||
|
||||
---
|
||||
|
||||
# Sicherheit & Governance
|
||||
|
||||
- Rollenmodell (Super Admin / Knowledge Admin / Redaktion)
|
||||
- Versionierte Dokumente
|
||||
- Versionierte Ingest-Profile
|
||||
- Versionierte System-Prompts
|
||||
- KI-Endpunkt abstrahiert
|
||||
- Audit-Logs
|
||||
- Lock-Mechanismen bei Reindex
|
||||
|
||||
---
|
||||
|
||||
# Produktstatus
|
||||
|
||||
Das System ist:
|
||||
|
||||
- Produktionsreif
|
||||
- Framework-neutral
|
||||
- Kundenfähig
|
||||
- Skalierbar
|
||||
- Erweiterbar (Adminbereich geplant)
|
||||
|
||||
Nicht enthalten:
|
||||
|
||||
- Autonomes Fine-Tuning
|
||||
- Live-Lernsystem
|
||||
- Self-Modifying Knowledge
|
||||
|
||||
---
|
||||
|
||||
# Unterschied zu generischen KI-Tools
|
||||
|
||||
| Generische KI | mitho AI Agent |
|
||||
|---------------|----------------|
|
||||
| trainiert auf Internet | basiert auf Ihrem Wissen |
|
||||
| keine Governance | volle Kontrolle |
|
||||
| keine Versionierung | Dokument-Versionierung |
|
||||
| nicht nachvollziehbar | transparente Wissensbasis |
|
||||
| generisch | unternehmensspezifisch |
|
||||
|
||||
---
|
||||
|
||||
# Mindestanforderungen
|
||||
|
||||
- PHP 8.2+
|
||||
- Python 3.9+
|
||||
- faiss
|
||||
- sentence-transformers
|
||||
- Ollama (oder kompatibles LLM)
|
||||
|
||||
---
|
||||
|
||||
# Vision
|
||||
|
||||
Dieses System bildet die Grundlage für:
|
||||
|
||||
- Agentic Commerce
|
||||
- Interne Wissenssysteme
|
||||
- Support-Automatisierung
|
||||
- Vertriebsassistenz
|
||||
- Technische Dokumentations-KI
|
||||
- DSGVO-konforme Unternehmens-KI
|
||||
|
||||
---
|
||||
|
||||
# Fazit
|
||||
|
||||
Der mitho AI Agent ist kein Spielzeug-Chatbot.
|
||||
|
||||
Er ist ein strukturiertes, kontrolliertes KI-System mit klarer Wissensbasis, deterministischem Retrieval und professioneller Architektur – gebaut für produktiven Unternehmenseinsatz.
|
||||
It is designed for controlled enterprise deployment, not experimental AI workflows.
|
||||
|
||||
334
SYSTEM.md
Normal file
334
SYSTEM.md
Normal file
@@ -0,0 +1,334 @@
|
||||
# mitho AI Agent
|
||||
Enterprise Hybrid RAG System (Symfony + NDJSON + FAISS)
|
||||
|
||||
---
|
||||
|
||||
## Überblick
|
||||
|
||||
Der **mitho AI Agent** ist ein deterministisches, governance-stabiles Enterprise-RAG-System auf Symfony-Basis.
|
||||
|
||||
Es kombiniert:
|
||||
|
||||
- Versionierte Dokumente
|
||||
- Streaming-NDJSON-Index
|
||||
- Deterministischen Full-Vector-Rebuild
|
||||
- Keyword- + Vektor-Hybrid-Retrieval
|
||||
- Rollen- und Governance-Modell
|
||||
- Lock- und Guardrail-Mechanismen
|
||||
- Produktionsfähige Adminoberfläche
|
||||
|
||||
> „Wir nutzen KI nicht, um kreativ zu raten, sondern um verlässlich auf Basis Ihres Wissens zu antworten.“
|
||||
|
||||
---
|
||||
|
||||
# Architektur (Enterprise-Version)
|
||||
|
||||
## 1. Kernprinzipien
|
||||
|
||||
- Dokumente sind immutable Primärquellen
|
||||
- index.ndjson ist Single Source of Truth
|
||||
- FAISS wird IMMER vollständig aus index.ndjson neu gebaut
|
||||
- Keine partiellen Vektor-Updates
|
||||
- Strukturänderungen erzwingen Global Reindex
|
||||
- Atomare Switch-Strategie (.tmp + rename)
|
||||
- >200k Chunks skalierbar
|
||||
- Kein Full-JSON-RAM-Load
|
||||
|
||||
---
|
||||
|
||||
# Wissensarchitektur
|
||||
|
||||
```
|
||||
var/knowledge/
|
||||
├── uploads/
|
||||
├── chunks/
|
||||
├── index.ndjson
|
||||
├── index_meta.json
|
||||
├── vector.index
|
||||
└── vector_meta.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Indexstruktur
|
||||
|
||||
## index.ndjson
|
||||
|
||||
- Streamingfähig
|
||||
- Eine Zeile = ein Chunk
|
||||
- JSON pro Zeile
|
||||
- Enthält:
|
||||
- chunk_id
|
||||
- document_id
|
||||
- version
|
||||
- text
|
||||
- meta
|
||||
|
||||
NDJSON ersetzt das frühere index.json (kein Array mehr).
|
||||
|
||||
---
|
||||
|
||||
# Index Meta
|
||||
|
||||
`index_meta.json` enthält:
|
||||
|
||||
- index_version
|
||||
- embedding_model
|
||||
- embedding_dimension
|
||||
- chunk_size
|
||||
- overlap
|
||||
- scoring_version
|
||||
- index_format
|
||||
|
||||
Wird verwaltet durch:
|
||||
|
||||
- `IndexMetaManager`
|
||||
- `IndexConfiguration`
|
||||
|
||||
---
|
||||
|
||||
# Ingest-Architektur
|
||||
|
||||
## Zentrale Klassen
|
||||
|
||||
| Klasse | Aufgabe |
|
||||
|--------|----------|
|
||||
| `DocumentService` | Dokumentverwaltung |
|
||||
| `DocumentVersionRepository` | Versionierung |
|
||||
| `KnowledgeIngestService` | Chunk-Erstellung |
|
||||
| `ChunkManager` | NDJSON Append + Compaction |
|
||||
| `SimpleChunker` | deterministische Text-Splittung |
|
||||
| `TextNormalizer` | Normalisierung |
|
||||
| `StopWords` | Stopword-Filter |
|
||||
| `IngestFlow` | Ablaufsteuerung |
|
||||
| `IngestOrchestrator` | Gesamtkoordination |
|
||||
| `IngestJobService` | Job-Verwaltung |
|
||||
| `LockService` | Reindex-Lock-Mechanismus |
|
||||
| `IndexMetaManager` | Index-Metadaten |
|
||||
| `IndexStructureChangedException` | Guardrail bei Strukturdrift |
|
||||
|
||||
---
|
||||
|
||||
# Ingest-Typen
|
||||
|
||||
## 1. Lokaler Ingest
|
||||
|
||||
- Neue Dokumentversion
|
||||
- Alte Chunks der document_id werden kompakt entfernt
|
||||
- Neue Chunks werden appended
|
||||
- Danach vollständiger FAISS-Rebuild
|
||||
- index_version bleibt gleich
|
||||
|
||||
## 2. Global Reindex
|
||||
|
||||
- Alle aktiven Dokumente neu ingestieren
|
||||
- index.ndjson komplett neu schreiben
|
||||
- FAISS komplett neu bauen
|
||||
- index_version++
|
||||
|
||||
---
|
||||
|
||||
# Vektor-Architektur
|
||||
|
||||
## vector_ingest.py
|
||||
|
||||
Buildet FAISS vollständig aus index.ndjson.
|
||||
|
||||
Eigenschaften:
|
||||
|
||||
- Streaming NDJSON read
|
||||
- normalize_embeddings=True
|
||||
- IndexFlatIP (Inner Product)
|
||||
- Batch-Size 64
|
||||
- Modell konfigurierbar
|
||||
|
||||
Aufruf:
|
||||
|
||||
```bash
|
||||
python vector_ingest.py --index path/to/index.ndjson --out path/to/vector.index
|
||||
```
|
||||
|
||||
Keine inkrementellen Updates. Immer Full-Rebuild.
|
||||
|
||||
---
|
||||
|
||||
## vector_search.py
|
||||
|
||||
- Nimmt Query + Limit
|
||||
- Lädt vector.index
|
||||
- Lädt vector_meta.json
|
||||
- Gibt JSON mit chunk_id + score zurück
|
||||
|
||||
Aufruf:
|
||||
|
||||
```bash
|
||||
python vector_search.py "query" 5
|
||||
```
|
||||
|
||||
Rückgabe:
|
||||
|
||||
```json
|
||||
[
|
||||
{ "chunk_id": "...", "score": 0.83 }
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Hybrid Retrieval
|
||||
|
||||
## Komponenten
|
||||
|
||||
| Klasse | Aufgabe |
|
||||
|--------|----------|
|
||||
| `NdjsonHybridRetriever` | Score-Fusion |
|
||||
| `NdjsonKeywordSearch` | Keyword-Scoring |
|
||||
| `NdjsonChunkLookup` | Chunk-Resolving |
|
||||
| `VectorSearchClient` | Python-Bridge |
|
||||
| `CachedRetriever` | Cache Layer |
|
||||
|
||||
---
|
||||
|
||||
## Retrieval Flow
|
||||
|
||||
1. Prompt
|
||||
2. Keyword-Ranking
|
||||
3. FAISS Top-K
|
||||
4. Score-Fusion
|
||||
5. Final Chunk-Auswahl
|
||||
6. Kontextaufbau
|
||||
7. LLM-Antwort
|
||||
|
||||
Keyword bleibt führend.
|
||||
Vektor ergänzt semantisch.
|
||||
|
||||
---
|
||||
|
||||
# Dokument-Extraktion
|
||||
|
||||
Unterstützt via:
|
||||
|
||||
- `DocumentExtractorInterface`
|
||||
- `ExtractorResolver`
|
||||
- `PdfExtractor`
|
||||
- `DocumentLoader`
|
||||
|
||||
PDFs werden robust extrahiert und normalisiert.
|
||||
|
||||
---
|
||||
|
||||
# Adminbereich (Symfony)
|
||||
|
||||
Controller:
|
||||
|
||||
- `DashboardController`
|
||||
- `DocumentController`
|
||||
- `IngestJobController`
|
||||
- `SecurityController`
|
||||
|
||||
Entities:
|
||||
|
||||
- `Document`
|
||||
- `DocumentVersion`
|
||||
- `IngestJob`
|
||||
- `User`
|
||||
|
||||
Repositories:
|
||||
|
||||
- `DocumentVersionRepository`
|
||||
- `UserRepository`
|
||||
|
||||
Rollenmodell:
|
||||
|
||||
- Super Admin
|
||||
- Knowledge Admin
|
||||
- Redaktion
|
||||
- Frontend User
|
||||
|
||||
---
|
||||
|
||||
# Guardrails & Sicherheit
|
||||
|
||||
- Lock bei Reindex
|
||||
- Strukturdrift-Erkennung
|
||||
- Keine Live-Änderung von Ingest-Profilen
|
||||
- Deterministische Rebuilds
|
||||
- Atomare Datei-Switches
|
||||
- NDJSON niemals vollständig im RAM
|
||||
|
||||
---
|
||||
|
||||
# LLM-Integration
|
||||
|
||||
- Ollama oder kompatibler Endpoint
|
||||
- Streaming via SSE
|
||||
- Historienverwaltung
|
||||
- Prompt-Versionierung
|
||||
- Kontextkontrolle
|
||||
|
||||
---
|
||||
|
||||
# Performance-Merkmale
|
||||
|
||||
- Streaming NDJSON
|
||||
- Kein JSON-Array-Load
|
||||
- Deterministischer FAISS-Rebuild
|
||||
- Cache-Layer
|
||||
- >200k Chunks skalierbar
|
||||
|
||||
---
|
||||
|
||||
# Mindestanforderungen
|
||||
|
||||
Backend:
|
||||
- PHP 8.2+
|
||||
- Symfony 7.x
|
||||
|
||||
Python:
|
||||
- Python 3.9+
|
||||
- faiss
|
||||
- sentence-transformers
|
||||
|
||||
Optional:
|
||||
- Ollama
|
||||
|
||||
---
|
||||
|
||||
# Produktstatus
|
||||
|
||||
Enterprise-ready:
|
||||
|
||||
- Governance-stabil
|
||||
- Drift-sicher
|
||||
- Deterministisch reproduzierbar
|
||||
- Skalierbar
|
||||
- Adminfähig
|
||||
- Rollenbasiert
|
||||
|
||||
---
|
||||
|
||||
# Positionierung
|
||||
|
||||
Dieses System ist keine generische KI.
|
||||
|
||||
Es ist:
|
||||
|
||||
- kontrolliert
|
||||
- versioniert
|
||||
- reproduzierbar
|
||||
- auditierbar
|
||||
- enterprise-tauglich
|
||||
|
||||
---
|
||||
|
||||
# Fazit
|
||||
|
||||
Der mitho AI Agent ist ein kontrolliertes Enterprise-RAG-System mit:
|
||||
|
||||
- NDJSON-Streaming-Index
|
||||
- deterministischem Vector-Rebuild
|
||||
- Hybrid-Retrieval
|
||||
- Guardrail-Mechanismen
|
||||
- Symfony-Admin-Governance
|
||||
|
||||
Er ist gebaut für produktiven, langfristigen Unternehmenseinsatz.
|
||||
@@ -16,6 +16,7 @@
|
||||
"symfony/dotenv": "^7.4",
|
||||
"symfony/flex": "^2",
|
||||
"symfony/framework-bundle": "^7.4",
|
||||
"symfony/messenger": "7.4.*",
|
||||
"symfony/monolog-bundle": "^4.0",
|
||||
"symfony/runtime": "^7.4",
|
||||
"symfony/security-bundle": "7.4.*",
|
||||
|
||||
98
composer.lock
generated
98
composer.lock
generated
@@ -4,7 +4,7 @@
|
||||
"Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies",
|
||||
"This file is @generated automatically"
|
||||
],
|
||||
"content-hash": "668be7cf3e1cc193e0f6a09448e72cf5",
|
||||
"content-hash": "6e61c35db778f34002e60f6186e53e4a",
|
||||
"packages": [
|
||||
{
|
||||
"name": "doctrine/collections",
|
||||
@@ -3113,6 +3113,100 @@
|
||||
],
|
||||
"time": "2025-12-31T08:43:57+00:00"
|
||||
},
|
||||
{
|
||||
"name": "symfony/messenger",
|
||||
"version": "v7.4.4",
|
||||
"source": {
|
||||
"type": "git",
|
||||
"url": "https://github.com/symfony/messenger.git",
|
||||
"reference": "0a39e1b256f280762293f2f441e430c8baf74f9c"
|
||||
},
|
||||
"dist": {
|
||||
"type": "zip",
|
||||
"url": "https://api.github.com/repos/symfony/messenger/zipball/0a39e1b256f280762293f2f441e430c8baf74f9c",
|
||||
"reference": "0a39e1b256f280762293f2f441e430c8baf74f9c",
|
||||
"shasum": ""
|
||||
},
|
||||
"require": {
|
||||
"php": ">=8.2",
|
||||
"psr/log": "^1|^2|^3",
|
||||
"symfony/clock": "^6.4|^7.0|^8.0",
|
||||
"symfony/deprecation-contracts": "^2.5|^3"
|
||||
},
|
||||
"conflict": {
|
||||
"symfony/console": "<7.2",
|
||||
"symfony/event-dispatcher": "<6.4",
|
||||
"symfony/event-dispatcher-contracts": "<2.5",
|
||||
"symfony/framework-bundle": "<6.4",
|
||||
"symfony/http-kernel": "<7.3",
|
||||
"symfony/lock": "<7.4",
|
||||
"symfony/serializer": "<6.4.32|>=7.3,<7.3.10|>=7.4,<7.4.4|>=8.0,<8.0.4"
|
||||
},
|
||||
"require-dev": {
|
||||
"psr/cache": "^1.0|^2.0|^3.0",
|
||||
"symfony/console": "^7.2|^8.0",
|
||||
"symfony/dependency-injection": "^6.4|^7.0|^8.0",
|
||||
"symfony/event-dispatcher": "^6.4|^7.0|^8.0",
|
||||
"symfony/http-kernel": "^7.3|^8.0",
|
||||
"symfony/lock": "^7.4|^8.0",
|
||||
"symfony/process": "^6.4|^7.0|^8.0",
|
||||
"symfony/property-access": "^6.4|^7.0|^8.0",
|
||||
"symfony/rate-limiter": "^6.4|^7.0|^8.0",
|
||||
"symfony/routing": "^6.4|^7.0|^8.0",
|
||||
"symfony/serializer": "^6.4.32|~7.3.10|^7.4.4|^8.0.4",
|
||||
"symfony/service-contracts": "^2.5|^3",
|
||||
"symfony/stopwatch": "^6.4|^7.0|^8.0",
|
||||
"symfony/validator": "^6.4|^7.0|^8.0",
|
||||
"symfony/var-dumper": "^6.4|^7.0|^8.0"
|
||||
},
|
||||
"type": "library",
|
||||
"autoload": {
|
||||
"psr-4": {
|
||||
"Symfony\\Component\\Messenger\\": ""
|
||||
},
|
||||
"exclude-from-classmap": [
|
||||
"/Tests/"
|
||||
]
|
||||
},
|
||||
"notification-url": "https://packagist.org/downloads/",
|
||||
"license": [
|
||||
"MIT"
|
||||
],
|
||||
"authors": [
|
||||
{
|
||||
"name": "Samuel Roze",
|
||||
"email": "samuel.roze@gmail.com"
|
||||
},
|
||||
{
|
||||
"name": "Symfony Community",
|
||||
"homepage": "https://symfony.com/contributors"
|
||||
}
|
||||
],
|
||||
"description": "Helps applications send and receive messages to/from other applications or via message queues",
|
||||
"homepage": "https://symfony.com",
|
||||
"support": {
|
||||
"source": "https://github.com/symfony/messenger/tree/v7.4.4"
|
||||
},
|
||||
"funding": [
|
||||
{
|
||||
"url": "https://symfony.com/sponsor",
|
||||
"type": "custom"
|
||||
},
|
||||
{
|
||||
"url": "https://github.com/fabpot",
|
||||
"type": "github"
|
||||
},
|
||||
{
|
||||
"url": "https://github.com/nicolas-grekas",
|
||||
"type": "github"
|
||||
},
|
||||
{
|
||||
"url": "https://tidelift.com/funding/github/packagist/symfony/symfony",
|
||||
"type": "tidelift"
|
||||
}
|
||||
],
|
||||
"time": "2026-01-08T14:50:10+00:00"
|
||||
},
|
||||
{
|
||||
"name": "symfony/monolog-bridge",
|
||||
"version": "v7.4.0",
|
||||
@@ -5968,5 +6062,5 @@
|
||||
"ext-iconv": "*"
|
||||
},
|
||||
"platform-dev": {},
|
||||
"plugin-api-version": "2.6.0"
|
||||
"plugin-api-version": "2.9.0"
|
||||
}
|
||||
|
||||
22
config/packages/messenger.yaml
Normal file
22
config/packages/messenger.yaml
Normal file
@@ -0,0 +1,22 @@
|
||||
framework:
|
||||
messenger:
|
||||
# Uncomment this (and the failed transport below) to send failed messages to this transport for later handling.
|
||||
# failure_transport: failed
|
||||
|
||||
transports:
|
||||
# https://symfony.com/doc/current/messenger.html#transport-configuration
|
||||
# async: '%env(MESSENGER_TRANSPORT_DSN)%'
|
||||
# failed: 'doctrine://default?queue_name=failed'
|
||||
sync: 'sync://'
|
||||
|
||||
routing:
|
||||
# Route your messages to the transports
|
||||
# 'App\Message\YourMessage': async
|
||||
|
||||
# when@test:
|
||||
# framework:
|
||||
# messenger:
|
||||
# transports:
|
||||
# # replace with your transport name here (e.g., my_transport: 'in-memory://')
|
||||
# # For more Messenger testing tools, see https://github.com/zenstruck/messenger-test
|
||||
# async: 'in-memory://'
|
||||
@@ -426,7 +426,7 @@ use Symfony\Component\Config\Loader\ParamConfigurator as Param;
|
||||
* resources?: array<string, scalar|null|Param>,
|
||||
* },
|
||||
* messenger?: bool|array{ // Messenger configuration
|
||||
* enabled?: bool|Param, // Default: false
|
||||
* enabled?: bool|Param, // Default: true
|
||||
* routing?: array<string, array{ // Default: []
|
||||
* senders?: list<scalar|null|Param>,
|
||||
* }>,
|
||||
|
||||
12
symfony.lock
12
symfony.lock
@@ -87,6 +87,18 @@
|
||||
"ref": "fadbfe33303a76e25cb63401050439aa9b1a9c7f"
|
||||
}
|
||||
},
|
||||
"symfony/messenger": {
|
||||
"version": "7.4",
|
||||
"recipe": {
|
||||
"repo": "github.com/symfony/recipes",
|
||||
"branch": "main",
|
||||
"version": "6.0",
|
||||
"ref": "d8936e2e2230637ef97e5eecc0eea074eecae58b"
|
||||
},
|
||||
"files": [
|
||||
"config/packages/messenger.yaml"
|
||||
]
|
||||
},
|
||||
"symfony/monolog-bundle": {
|
||||
"version": "4.0",
|
||||
"recipe": {
|
||||
|
||||
Reference in New Issue
Block a user