add new md files
This commit is contained in:
team 1
368
TECHNICAL_ARCHITECTURE.md
Normal file
368
TECHNICAL_ARCHITECTURE.md
Normal file
@@ -0,0 +1,368 @@
|
||||
# TECHNICAL_ARCHITECTURE.md
|
||||
mitho AI Agent – Enterprise Hybrid RAG System
|
||||
Stand: Februar 2026
|
||||
|
||||
---
|
||||
|
||||
# 1. Zielsetzung
|
||||
|
||||
Diese Dokumentation beschreibt die vollständige technische Architektur des
|
||||
mitho AI Agent RAG-Systems.
|
||||
|
||||
Ziele:
|
||||
|
||||
- Deterministisches Retrieval
|
||||
- Governance-stabile Index-Struktur
|
||||
- Versionierte Wissensbasis
|
||||
- Reproduzierbare Builds
|
||||
- Keine inkrementellen Vektor-Mutationen
|
||||
- Job-basierte, asynchrone Ingest-Architektur
|
||||
- Skalierbarkeit >200k Chunks
|
||||
|
||||
---
|
||||
|
||||
# 2. Architektur-Übersicht
|
||||
|
||||
Systemebenen:
|
||||
|
||||
1. Admin Layer (Symfony)
|
||||
2. Ingest Layer
|
||||
3. Storage Layer (NDJSON)
|
||||
4. Vector Layer (FAISS)
|
||||
5. Retrieval Layer (Hybrid)
|
||||
6. Application Layer (LLM + SSE)
|
||||
|
||||
---
|
||||
|
||||
# 3. Core Architekturprinzipien
|
||||
|
||||
## 3.1 Determinismus
|
||||
|
||||
Gleiche Eingabe + gleiche Konfiguration → identisches Ergebnis.
|
||||
|
||||
- index.ndjson ist deterministisch
|
||||
- FAISS wird vollständig neu gebaut
|
||||
- Retrieval ist stabil reproduzierbar
|
||||
- Keine impliziten Indexveränderungen
|
||||
|
||||
## 3.2 Governance
|
||||
|
||||
- Genau eine aktive Version pro Dokument
|
||||
- Keine Inline-Rebuilds im HTTP-Request
|
||||
- Strukturänderungen erzwingen Global Reindex
|
||||
- Locking verhindert Parallel-Ingest
|
||||
|
||||
## 3.3 Skalierbarkeit
|
||||
|
||||
- NDJSON statt JSON-Array
|
||||
- Streaming-Append
|
||||
- Streaming-Compaction
|
||||
- Kein RAM-Full-Load
|
||||
|
||||
---
|
||||
|
||||
# 4. Domain Model
|
||||
|
||||
## 4.1 Document
|
||||
|
||||
Primäre Wissenseinheit.
|
||||
|
||||
Eigenschaften:
|
||||
- id (UUID)
|
||||
- title
|
||||
- createdAt
|
||||
- createdBy
|
||||
- archived (bool)
|
||||
- currentVersion (ManyToOne → DocumentVersion)
|
||||
|
||||
## 4.2 DocumentVersion
|
||||
|
||||
Immutable Inhaltsversion.
|
||||
|
||||
Eigenschaften:
|
||||
- id (UUID)
|
||||
- document (ManyToOne)
|
||||
- versionNumber
|
||||
- filePath
|
||||
- checksum
|
||||
- active (bool)
|
||||
- ingestStatus (PENDING, RUNNING, INDEXED, FAILED)
|
||||
|
||||
Regel:
|
||||
Nur eine Version pro Dokument darf active = true sein.
|
||||
|
||||
---
|
||||
|
||||
# 5. Ingest Architektur
|
||||
|
||||
## 5.1 Grundsatz
|
||||
|
||||
Kein Ingest läuft synchron im HTTP-Request.
|
||||
|
||||
Jede Indexänderung läuft über:
|
||||
|
||||
IngestJob → CLI Runner → IngestOrchestrator → IngestFlow
|
||||
|
||||
---
|
||||
|
||||
## 5.2 IngestJob
|
||||
|
||||
Entity mit:
|
||||
|
||||
- id
|
||||
- type
|
||||
- status
|
||||
- documentId
|
||||
- documentVersionId
|
||||
- startedAt
|
||||
- finishedAt
|
||||
- errorMessage
|
||||
|
||||
Job-Typen:
|
||||
|
||||
DOCUMENT_VERSION_ACTIVATE
|
||||
DOCUMENT
|
||||
GLOBAL_REINDEX
|
||||
|
||||
Status:
|
||||
|
||||
QUEUED
|
||||
RUNNING
|
||||
COMPLETED
|
||||
FAILED
|
||||
ABORTED
|
||||
|
||||
---
|
||||
|
||||
## 5.3 Job-Execution
|
||||
|
||||
Start:
|
||||
|
||||
php bin/console mto:agent:ingest:run <jobId>
|
||||
|
||||
Execution-Flow:
|
||||
|
||||
1. LockService.acquire()
|
||||
2. Job → RUNNING
|
||||
3. IngestFlow ausführen
|
||||
4. Job → COMPLETED / FAILED
|
||||
5. LockService.release()
|
||||
|
||||
---
|
||||
|
||||
# 6. IngestFlow Details
|
||||
|
||||
## 6.1 Local Ingest (Version aktivieren oder neue Datei)
|
||||
|
||||
Ablauf:
|
||||
|
||||
1. Extract (PDF/Text)
|
||||
2. Normalize
|
||||
3. Chunk deterministisch
|
||||
4. Streaming Compaction:
|
||||
- Entferne alle Chunks mit document_id
|
||||
5. Append neue Chunks
|
||||
6. Full FAISS Rebuild
|
||||
|
||||
Wichtig:
|
||||
Es werden immer alle alten Chunks des Dokuments entfernt.
|
||||
|
||||
---
|
||||
|
||||
## 6.2 Global Reindex
|
||||
|
||||
Wird ausgelöst bei:
|
||||
|
||||
- embedding_model Änderung
|
||||
- embedding_dimension Änderung
|
||||
- chunk_size Änderung
|
||||
- scoring_version Änderung
|
||||
- index_format Änderung
|
||||
|
||||
Ablauf:
|
||||
|
||||
1. Alle aktiven Versionen extrahieren
|
||||
2. Komplettes index.ndjson neu schreiben
|
||||
3. FAISS neu bauen
|
||||
4. index_version++
|
||||
|
||||
---
|
||||
|
||||
# 7. Storage Layer
|
||||
|
||||
## 7.1 index.ndjson
|
||||
|
||||
Single Source of Truth.
|
||||
|
||||
Eigenschaften:
|
||||
|
||||
- 1 JSON-Objekt pro Zeile
|
||||
- Streaming-lesbar
|
||||
- Append-only
|
||||
- Compaction by document_id
|
||||
|
||||
Beispiel:
|
||||
|
||||
{
|
||||
"chunk_id": "...",
|
||||
"document_id": "...",
|
||||
"document_version_id": "...",
|
||||
"text": "...",
|
||||
"meta": {...}
|
||||
}
|
||||
|
||||
---
|
||||
|
||||
## 7.2 index_meta.json
|
||||
|
||||
Enthält:
|
||||
|
||||
- index_version
|
||||
- embedding_model
|
||||
- embedding_dimension
|
||||
- chunk_size
|
||||
- chunk_overlap
|
||||
- scoring_version
|
||||
- index_format
|
||||
- vector_backend
|
||||
|
||||
Bei strukturellem Mismatch:
|
||||
→ Global Reindex zwingend.
|
||||
|
||||
---
|
||||
|
||||
# 8. Vector Layer (FAISS)
|
||||
|
||||
## 8.1 Build
|
||||
|
||||
vector_ingest.py:
|
||||
|
||||
- stream index.ndjson
|
||||
- Embeddings berechnen
|
||||
- FAISS IndexFlatIP bauen
|
||||
- vector.index schreiben
|
||||
- vector_meta.json schreiben
|
||||
|
||||
Immer vollständiger Rebuild.
|
||||
Keine Partial Updates.
|
||||
|
||||
## 8.2 Search
|
||||
|
||||
vector_search.py:
|
||||
|
||||
Input:
|
||||
- Query
|
||||
- Top-K
|
||||
|
||||
Output:
|
||||
- chunk_id
|
||||
- score
|
||||
|
||||
---
|
||||
|
||||
# 9. Hybrid Retrieval
|
||||
|
||||
Ablauf:
|
||||
|
||||
1. Keyword Retrieval (NDJSON Keyword Search)
|
||||
2. Vector Retrieval (FAISS)
|
||||
3. Score Fusion
|
||||
4. Chunk Lookup
|
||||
5. Context Builder
|
||||
6. Prompt Builder
|
||||
7. LLM
|
||||
|
||||
Keyword bleibt Primärsignal.
|
||||
Vector ergänzt semantische Nähe.
|
||||
|
||||
---
|
||||
|
||||
# 10. Locking
|
||||
|
||||
LockService schützt:
|
||||
|
||||
- NDJSON Mutationen
|
||||
- FAISS Rebuild
|
||||
- Global Reindex
|
||||
- Aktivierungs-Parallelität
|
||||
|
||||
Keine zwei Ingest-Jobs gleichzeitig erlaubt.
|
||||
|
||||
---
|
||||
|
||||
# 11. Admin Flows
|
||||
|
||||
## 11.1 Neue Datei hochladen
|
||||
|
||||
1. Document + Version 1 erstellen
|
||||
2. Version aktiv
|
||||
3. DOCUMENT_VERSION_ACTIVATE Job anlegen
|
||||
4. Async Runner starten
|
||||
5. Redirect auf Job-Seite
|
||||
|
||||
## 11.2 Version aktivieren
|
||||
|
||||
1. DB-Status ändern
|
||||
2. IngestStatus → PENDING
|
||||
3. DOCUMENT_VERSION_ACTIVATE Job anlegen
|
||||
4. Async Runner starten
|
||||
|
||||
## 11.3 Manuelles Ingest
|
||||
|
||||
1. DOCUMENT Job anlegen
|
||||
2. Async Runner starten
|
||||
|
||||
---
|
||||
|
||||
# 12. Error Handling
|
||||
|
||||
Typische Fehler:
|
||||
|
||||
- exec deaktiviert → Async Start nicht möglich
|
||||
- Lock aktiv → paralleler Ingest blockiert
|
||||
- index_meta mismatch → Reindex erforderlich
|
||||
- Vector-Dateien fehlen → Rebuild ausführen
|
||||
|
||||
---
|
||||
|
||||
# 13. Security & Stability
|
||||
|
||||
- Keine User-Uploads direkt ins Retrieval
|
||||
- Kein Self-Learning
|
||||
- Kein unkontrollierter Index-Mutate
|
||||
- Strict Separation:
|
||||
- Admin Layer
|
||||
- Ingest Layer
|
||||
- Retrieval Layer
|
||||
- LLM Layer
|
||||
|
||||
---
|
||||
|
||||
# 14. System Guarantees
|
||||
|
||||
Dieses System garantiert:
|
||||
|
||||
- Reproduzierbarkeit
|
||||
- Drift-Freiheit
|
||||
- Auditierbarkeit
|
||||
- Versionierte Wissensbasis
|
||||
- Deterministische Retrieval-Ergebnisse
|
||||
- Enterprise-taugliche Skalierbarkeit
|
||||
|
||||
---
|
||||
|
||||
# 15. Zusammenfassung
|
||||
|
||||
Das mitho AI Agent System ist eine:
|
||||
|
||||
- deterministische Hybrid-RAG Engine
|
||||
- NDJSON-basierte Wissensplattform
|
||||
- Full-Rebuild-FAISS Architektur
|
||||
- Job-basierte Ingest-Pipeline
|
||||
- Versionierte, governance-stabile Wissensstruktur
|
||||
|
||||
Keine Inline-Rebuilds.
|
||||
Keine inkrementellen Vektor-Updates.
|
||||
Keine impliziten Strukturänderungen.
|
||||
|
||||
Alles läuft kontrolliert über das Job-System.
|
||||
Reference in New Issue
Block a user