Spaces:
Sleeping
Sleeping
Remove RAG system: Clean up all RAG-related files and dependencies
Browse files- Remove rag/ module (embedder, vector_store, retriever)
- Remove data/raw_pdfs/ (FEI PDFs)
- Remove scripts/index_documents.py
- Remove RAG_SETUP.md documentation
- Clean main.py: remove RAG imports and endpoints
- Clean requirements.txt: remove RAG dependencies (PyPDF2, sentence-transformers, chromadb, numpy<2.0, tqdm)
- Keep only fine-tuned Mistral-7B without RAG augmentation
- RAG_SETUP.md +0 -192
- data/raw_pdfs/CCE/.gitkeep +0 -1
- data/raw_pdfs/CCE/2026 FEI Eventing Rules_Clean - no Tack.pdf +0 -3
- data/raw_pdfs/CCE/Tack Equipment Requirements -Eventing 2026_17 December 2025.pdf +0 -3
- data/raw_pdfs/CSO/.gitkeep +0 -1
- data/raw_pdfs/CSO/FEI Jumping Rules 2026_1.pdf +0 -3
- data/raw_pdfs/Dressage/.gitkeep +0 -1
- data/raw_pdfs/Dressage/FEI_Dressage_Rules_2026_Clean_Version_6.pdf +0 -3
- data/raw_pdfs/Dressage/Tack Equipment Requirements - Dressage_4.pdf +0 -3
- data/raw_pdfs/README.md +0 -119
- main.py +8 -239
- rag/__init__.py +0 -24
- rag/embedder.py +0 -139
- rag/retriever.py +0 -201
- rag/vector_store.py +0 -267
- requirements.txt +0 -7
- scripts/index_documents.py +0 -355
RAG_SETUP.md
DELETED
|
@@ -1,192 +0,0 @@
|
|
| 1 |
-
# 🎯 Guide RAG - Coach Équestre AI
|
| 2 |
-
|
| 3 |
-
## 📚 Système RAG Intégré
|
| 4 |
-
|
| 5 |
-
Ce Space HuggingFace contient un système RAG (Retrieval-Augmented Generation) pour les règlements FEI.
|
| 6 |
-
|
| 7 |
-
## 🗂️ Structure
|
| 8 |
-
|
| 9 |
-
```
|
| 10 |
-
coach-equestre-ai/
|
| 11 |
-
├── rag/ # Modules RAG
|
| 12 |
-
│ ├── embedder.py # Embeddings multilingual-e5-base
|
| 13 |
-
│ ├── vector_store.py # ChromaDB
|
| 14 |
-
│ └── retriever.py # Pipeline RAG
|
| 15 |
-
├── scripts/
|
| 16 |
-
│ └── index_documents.py # Indexation des PDFs
|
| 17 |
-
├── data/
|
| 18 |
-
│ └── raw_pdfs/ # PDFs FEI (5 fichiers)
|
| 19 |
-
│ ├── CSO/
|
| 20 |
-
│ ├── Dressage/
|
| 21 |
-
│ └── CCE/
|
| 22 |
-
└── main.py # API FastAPI
|
| 23 |
-
```
|
| 24 |
-
|
| 25 |
-
## 🚀 Première Installation
|
| 26 |
-
|
| 27 |
-
### 1. Activer le GPU T4 (OBLIGATOIRE)
|
| 28 |
-
|
| 29 |
-
⚠️ **Le RAG nécessite un GPU pour les embeddings**
|
| 30 |
-
|
| 31 |
-
1. Va dans **Settings** de ton Space
|
| 32 |
-
2. **Resource configuration** → **Space hardware**
|
| 33 |
-
3. Sélectionne **T4 Small** (gratuit)
|
| 34 |
-
4. Clique **Save**
|
| 35 |
-
5. Attends 2-3 minutes que le Space redémarre
|
| 36 |
-
|
| 37 |
-
### 2. Indexer les PDFs FEI
|
| 38 |
-
|
| 39 |
-
Une fois le Space redémarré avec GPU:
|
| 40 |
-
|
| 41 |
-
```bash
|
| 42 |
-
# Se connecter au terminal du Space
|
| 43 |
-
# (ou utiliser les logs pour voir l'exécution)
|
| 44 |
-
|
| 45 |
-
# Indexer tous les PDFs (5-10 minutes)
|
| 46 |
-
python scripts/index_documents.py --all
|
| 47 |
-
|
| 48 |
-
# Ou indexer discipline par discipline
|
| 49 |
-
python scripts/index_documents.py --discipline CSO
|
| 50 |
-
python scripts/index_documents.py --discipline Dressage
|
| 51 |
-
python scripts/index_documents.py --discipline CCE
|
| 52 |
-
```
|
| 53 |
-
|
| 54 |
-
**Ce que fait l'indexation:**
|
| 55 |
-
1. Lit chaque PDF page par page (5 PDFs FEI)
|
| 56 |
-
2. Découpe en chunks de 500 caractères avec overlap 50
|
| 57 |
-
3. Génère embeddings avec multilingual-e5-base (768D)
|
| 58 |
-
4. Stocke dans ChromaDB avec métadonnées (discipline, source, page)
|
| 59 |
-
|
| 60 |
-
**Temps estimé:** 5-10 minutes sur GPU T4
|
| 61 |
-
|
| 62 |
-
### 3. Vérifier l'Indexation
|
| 63 |
-
|
| 64 |
-
```python
|
| 65 |
-
from rag import Retriever
|
| 66 |
-
|
| 67 |
-
retriever = Retriever(top_k=5)
|
| 68 |
-
stats = retriever.get_stats()
|
| 69 |
-
|
| 70 |
-
print(f"Documents indexés: {stats['total_documents']}")
|
| 71 |
-
print(f"Disciplines: {stats['disciplines']}")
|
| 72 |
-
# Attendu: 500-1500 chunks selon la taille des PDFs
|
| 73 |
-
```
|
| 74 |
-
|
| 75 |
-
## 🧪 Tester le RAG
|
| 76 |
-
|
| 77 |
-
### Test Simple
|
| 78 |
-
|
| 79 |
-
```python
|
| 80 |
-
from rag import Retriever
|
| 81 |
-
|
| 82 |
-
retriever = Retriever(top_k=3)
|
| 83 |
-
|
| 84 |
-
# Recherche
|
| 85 |
-
results = retriever.search("Quelle est la hauteur maximale pour CSO Amateur 1?")
|
| 86 |
-
|
| 87 |
-
# Afficher les résultats
|
| 88 |
-
for i, result in enumerate(results, 1):
|
| 89 |
-
print(f"\n{i}. Score: {result['score']:.3f}")
|
| 90 |
-
print(f" Source: {result['metadata']['source_file']} (page {result['metadata']['page_number']})")
|
| 91 |
-
print(f" Texte: {result['text'][:150]}...")
|
| 92 |
-
|
| 93 |
-
# Formater pour le LLM
|
| 94 |
-
context = retriever.format_context(results, max_results=3)
|
| 95 |
-
print(f"\n📝 Contexte pour LLM:\n{context}")
|
| 96 |
-
```
|
| 97 |
-
|
| 98 |
-
### Questions de Test Recommandées
|
| 99 |
-
|
| 100 |
-
✅ **Questions factuelles (nécessitent RAG):**
|
| 101 |
-
- "Quelle hauteur maximale pour CSO Amateur 1?"
|
| 102 |
-
- "Combien de pénalités pour un refus en CSO?"
|
| 103 |
-
- "Quel équipement obligatoire en dressage Grand Prix?"
|
| 104 |
-
- "Vitesse minimale en cross CCE 3 étoiles?"
|
| 105 |
-
|
| 106 |
-
❌ **Questions pratiques (LLM seul):**
|
| 107 |
-
- "Comment améliorer mon galop à droite?"
|
| 108 |
-
- "Mon cheval a peur des obstacles colorés"
|
| 109 |
-
- "Comment travailler les transitions?"
|
| 110 |
-
|
| 111 |
-
## 🔧 Configuration
|
| 112 |
-
|
| 113 |
-
### Modèle d'Embedding
|
| 114 |
-
|
| 115 |
-
- **Modèle**: `intfloat/multilingual-e5-base`
|
| 116 |
-
- **Taille**: 560MB
|
| 117 |
-
- **Dimensions**: 768
|
| 118 |
-
- **Langues**: Français + Anglais (parfait pour règlements FEI bilingues)
|
| 119 |
-
- **Device**: GPU T4 (auto-détecté)
|
| 120 |
-
|
| 121 |
-
### Vector Database
|
| 122 |
-
|
| 123 |
-
- **Type**: ChromaDB
|
| 124 |
-
- **Stockage**: `data/chroma_db/` (persistence)
|
| 125 |
-
- **Métadonnées**: discipline, source_file, page_number, chunk_index
|
| 126 |
-
- **Metric**: Cosine similarity
|
| 127 |
-
|
| 128 |
-
## 📊 PDFs FEI Inclus
|
| 129 |
-
|
| 130 |
-
- ✅ **CSO**: FEI Jumping Rules 2026 (1.3MB)
|
| 131 |
-
- ✅ **CCE**: FEI Eventing Rules 2026 + Tack & Equipment (1.2MB)
|
| 132 |
-
- ✅ **Dressage**: FEI Dressage Rules 2026 + Tack & Equipment (600KB)
|
| 133 |
-
|
| 134 |
-
**Total:** 5 PDFs, ~3.5MB
|
| 135 |
-
|
| 136 |
-
## 🔄 Ré-indexation
|
| 137 |
-
|
| 138 |
-
Si tu mets à jour les PDFs:
|
| 139 |
-
|
| 140 |
-
```bash
|
| 141 |
-
# Réinitialiser la base
|
| 142 |
-
python scripts/index_documents.py --all --reset
|
| 143 |
-
|
| 144 |
-
# Le script supprime automatiquement les anciens chunks avant de ré-indexer
|
| 145 |
-
```
|
| 146 |
-
|
| 147 |
-
## 🚨 Dépannage
|
| 148 |
-
|
| 149 |
-
### Erreur "No module named 'rag'"
|
| 150 |
-
|
| 151 |
-
```bash
|
| 152 |
-
# Vérifie que les dossiers sont présents
|
| 153 |
-
ls -la rag/
|
| 154 |
-
ls -la scripts/
|
| 155 |
-
ls -la data/raw_pdfs/
|
| 156 |
-
```
|
| 157 |
-
|
| 158 |
-
### Erreur "CUDA not available"
|
| 159 |
-
|
| 160 |
-
Le GPU T4 n'est pas activé. Va dans Settings → Hardware → T4 Small.
|
| 161 |
-
|
| 162 |
-
### Base ChromaDB vide
|
| 163 |
-
|
| 164 |
-
```bash
|
| 165 |
-
# L'indexation n'a pas été faite
|
| 166 |
-
python scripts/index_documents.py --all
|
| 167 |
-
```
|
| 168 |
-
|
| 169 |
-
### Recherche lente (> 5 secondes)
|
| 170 |
-
|
| 171 |
-
Tu es probablement en mode CPU. Active le GPU T4 dans Settings.
|
| 172 |
-
|
| 173 |
-
## 🎯 Prochaines Étapes (Phase 2)
|
| 174 |
-
|
| 175 |
-
- [ ] Intégrer le RAG dans l'endpoint `/api/chat`
|
| 176 |
-
- [ ] Ajouter un classifier (RAG vs LLM seul)
|
| 177 |
-
- [ ] Ajouter reranking (cross-encoder) pour améliorer précision
|
| 178 |
-
- [ ] Tester avec questions réelles utilisateurs
|
| 179 |
-
|
| 180 |
-
## 📚 Documentation Complète
|
| 181 |
-
|
| 182 |
-
Voir `backend/RAG_GUIDE.md` dans le repo principal pour:
|
| 183 |
-
- Architecture détaillée
|
| 184 |
-
- Pipeline complet
|
| 185 |
-
- Métriques de performance
|
| 186 |
-
- Guide d'intégration API
|
| 187 |
-
|
| 188 |
-
---
|
| 189 |
-
|
| 190 |
-
**Status:** ✅ RAG Phase 1 prêt à indexer
|
| 191 |
-
**GPU:** T4 Small requis (gratuit)
|
| 192 |
-
**Temps d'indexation:** 5-10 minutes
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
data/raw_pdfs/CCE/.gitkeep
DELETED
|
@@ -1 +0,0 @@
|
|
| 1 |
-
# Placer ici les PDFs des règlements CCE (Concours Complet)
|
|
|
|
|
|
data/raw_pdfs/CCE/2026 FEI Eventing Rules_Clean - no Tack.pdf
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:71b4e529ba5ef36edf9ee186cafa3c7f2db0fb208844ab09d3ef6f647fda2fc0
|
| 3 |
-
size 717006
|
|
|
|
|
|
|
|
|
|
|
|
data/raw_pdfs/CCE/Tack Equipment Requirements -Eventing 2026_17 December 2025.pdf
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:27ed1109b0302e86238b430e8d6d7e1ae2bbc19f3e93dc0cca73000b4514fe37
|
| 3 |
-
size 532118
|
|
|
|
|
|
|
|
|
|
|
|
data/raw_pdfs/CSO/.gitkeep
DELETED
|
@@ -1 +0,0 @@
|
|
| 1 |
-
# Placer ici les PDFs des règlements CSO (Saut d'Obstacles)
|
|
|
|
|
|
data/raw_pdfs/CSO/FEI Jumping Rules 2026_1.pdf
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:2217afd42268252625762085892042d889343f5b52d517c4e91b4b269918a90b
|
| 3 |
-
size 1415956
|
|
|
|
|
|
|
|
|
|
|
|
data/raw_pdfs/Dressage/.gitkeep
DELETED
|
@@ -1 +0,0 @@
|
|
| 1 |
-
# Placer ici les PDFs des règlements Dressage
|
|
|
|
|
|
data/raw_pdfs/Dressage/FEI_Dressage_Rules_2026_Clean_Version_6.pdf
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:1258dce1aeac23f769c716b155d2d860a2af0e673c5dd41b1e0991026dc2384e
|
| 3 |
-
size 511345
|
|
|
|
|
|
|
|
|
|
|
|
data/raw_pdfs/Dressage/Tack Equipment Requirements - Dressage_4.pdf
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:6bc72874ea71f096b16909b5eff728187ad54a2d6a366f04d0692a53d0e30cb2
|
| 3 |
-
size 97516
|
|
|
|
|
|
|
|
|
|
|
|
data/raw_pdfs/README.md
DELETED
|
@@ -1,119 +0,0 @@
|
|
| 1 |
-
# 📚 Règlements FEI - Documentation Source
|
| 2 |
-
|
| 3 |
-
Ce dossier contient les PDFs officiels des règlements de la Fédération Équestre Internationale (FEI) qui seront indexés par le système RAG.
|
| 4 |
-
|
| 5 |
-
## 📁 Structure des Dossiers
|
| 6 |
-
|
| 7 |
-
```
|
| 8 |
-
raw_pdfs/
|
| 9 |
-
├── CSO/ # Concours de Saut d'Obstacles
|
| 10 |
-
├── Dressage/ # Compétitions de Dressage
|
| 11 |
-
└── CCE/ # Concours Complet d'Équitation
|
| 12 |
-
```
|
| 13 |
-
|
| 14 |
-
## 📥 Comment Ajouter des PDFs
|
| 15 |
-
|
| 16 |
-
### 1. Télécharger les Règlements FEI
|
| 17 |
-
|
| 18 |
-
Rendez-vous sur le site officiel FEI:
|
| 19 |
-
- **Site FEI**: https://inside.fei.org/fei/regulations/
|
| 20 |
-
- **Jumping Rules**: https://inside.fei.org/fei/regulations/jumping
|
| 21 |
-
- **Dressage Rules**: https://inside.fei.org/fei/regulations/dressage
|
| 22 |
-
- **Eventing Rules**: https://inside.fei.org/fei/regulations/eventing
|
| 23 |
-
|
| 24 |
-
### 2. Organiser les PDFs par Discipline
|
| 25 |
-
|
| 26 |
-
**CSO (Saut d'Obstacles):**
|
| 27 |
-
```bash
|
| 28 |
-
backend/data/raw_pdfs/CSO/
|
| 29 |
-
├── FEI_Jumping_Rules_2024.pdf
|
| 30 |
-
├── FEI_Jumping_World_Cup_Rules.pdf
|
| 31 |
-
└── FEI_Jumping_Nations_Cup_Rules.pdf
|
| 32 |
-
```
|
| 33 |
-
|
| 34 |
-
**Dressage:**
|
| 35 |
-
```bash
|
| 36 |
-
backend/data/raw_pdfs/Dressage/
|
| 37 |
-
├── FEI_Dressage_Rules_2024.pdf
|
| 38 |
-
├── FEI_Dressage_Tests.pdf
|
| 39 |
-
└── FEI_Freestyle_Music_Requirements.pdf
|
| 40 |
-
```
|
| 41 |
-
|
| 42 |
-
**CCE (Concours Complet):**
|
| 43 |
-
```bash
|
| 44 |
-
backend/data/raw_pdfs/CCE/
|
| 45 |
-
├── FEI_Eventing_Rules_2024.pdf
|
| 46 |
-
├── FEI_Cross_Country_Specifications.pdf
|
| 47 |
-
└── FEI_Eventing_Risk_Management.pdf
|
| 48 |
-
```
|
| 49 |
-
|
| 50 |
-
## 🔍 Format des Fichiers
|
| 51 |
-
|
| 52 |
-
- **Format accepté**: `.pdf` uniquement
|
| 53 |
-
- **Langue**: Français et/ou Anglais (le système RAG supporte les deux)
|
| 54 |
-
- **Taille**: Aucune limite (le chunking gérera les gros documents)
|
| 55 |
-
|
| 56 |
-
## ⚡ Indexation
|
| 57 |
-
|
| 58 |
-
Une fois les PDFs ajoutés, lancer le script d'indexation:
|
| 59 |
-
|
| 60 |
-
```bash
|
| 61 |
-
cd backend
|
| 62 |
-
python scripts/index_documents.py --discipline CSO
|
| 63 |
-
python scripts/index_documents.py --discipline Dressage
|
| 64 |
-
python scripts/index_documents.py --discipline CCE
|
| 65 |
-
```
|
| 66 |
-
|
| 67 |
-
Ou indexer tout d'un coup:
|
| 68 |
-
```bash
|
| 69 |
-
python scripts/index_documents.py --all
|
| 70 |
-
```
|
| 71 |
-
|
| 72 |
-
## 📊 Métadonnées Automatiques
|
| 73 |
-
|
| 74 |
-
Chaque chunk de texte extrait sera enrichi avec:
|
| 75 |
-
- `discipline`: CSO / Dressage / CCE
|
| 76 |
-
- `source_file`: Nom du PDF original
|
| 77 |
-
- `page_number`: Numéro de page dans le PDF
|
| 78 |
-
- `chunk_index`: Index du chunk dans le document
|
| 79 |
-
- `language`: Détection automatique FR/EN
|
| 80 |
-
|
| 81 |
-
## 🔄 Mise à Jour des Règlements
|
| 82 |
-
|
| 83 |
-
Pour mettre à jour un règlement:
|
| 84 |
-
1. Supprimer l'ancien PDF du dossier
|
| 85 |
-
2. Ajouter le nouveau PDF avec le même nom
|
| 86 |
-
3. Re-lancer l'indexation pour cette discipline
|
| 87 |
-
4. L'ancien contenu sera automatiquement écrasé dans ChromaDB
|
| 88 |
-
|
| 89 |
-
## 📝 Exemples de Règlements Utiles
|
| 90 |
-
|
| 91 |
-
### CSO
|
| 92 |
-
- Hauteurs et barèmes par catégorie (Amateur 1/2/3, Pro, GP)
|
| 93 |
-
- Pénalités (refus, chute, dépassement de temps)
|
| 94 |
-
- Équipement autorisé (éperons, cravache, protections)
|
| 95 |
-
- Règlement du barème en Table A/C
|
| 96 |
-
|
| 97 |
-
### Dressage
|
| 98 |
-
- Reprises imposées par niveau (Préliminaire → Grand Prix)
|
| 99 |
-
- Critères de notation par mouvement
|
| 100 |
-
- Équipement réglementaire (filet, selle, guêtres)
|
| 101 |
-
- Temps de reprise autorisés
|
| 102 |
-
|
| 103 |
-
### CCE
|
| 104 |
-
- Normes cross-country (hauteur, largeur obstacles)
|
| 105 |
-
- Vitesse minimale/maximale par niveau
|
| 106 |
-
- Pénalités en dressage/cross/CSO
|
| 107 |
-
- Équipement de sécurité obligatoire
|
| 108 |
-
|
| 109 |
-
## ⚠️ Important
|
| 110 |
-
|
| 111 |
-
- Les PDFs doivent être **lisibles** (pas de scans de mauvaise qualité)
|
| 112 |
-
- Éviter les documents protégés par mot de passe
|
| 113 |
-
- Les images dans les PDFs seront ignorées (extraction texte uniquement)
|
| 114 |
-
|
| 115 |
-
---
|
| 116 |
-
|
| 117 |
-
**Status**: ⏳ En attente de PDFs
|
| 118 |
-
**Dernière indexation**: Jamais
|
| 119 |
-
**Nombre de documents**: 0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
main.py
CHANGED
|
@@ -15,18 +15,6 @@ from peft import AutoPeftModelForCausalLM
|
|
| 15 |
|
| 16 |
import config
|
| 17 |
|
| 18 |
-
# Import RAG modules
|
| 19 |
-
try:
|
| 20 |
-
from rag import Retriever
|
| 21 |
-
RAG_AVAILABLE = True
|
| 22 |
-
print("✅ Modules RAG chargés avec succès")
|
| 23 |
-
except ImportError as e:
|
| 24 |
-
RAG_AVAILABLE = False
|
| 25 |
-
print(f"⚠️ Modules RAG non disponibles: {str(e)}")
|
| 26 |
-
import traceback
|
| 27 |
-
print("Traceback complet:")
|
| 28 |
-
traceback.print_exc()
|
| 29 |
-
|
| 30 |
# Désactiver le parallélisme des tokenizers
|
| 31 |
os.environ["TOKENIZERS_PARALLELISM"] = "false"
|
| 32 |
|
|
@@ -57,7 +45,6 @@ class ChatResponse(BaseModel):
|
|
| 57 |
model = None
|
| 58 |
tokenizer = None
|
| 59 |
text_generator = None
|
| 60 |
-
retriever = None # Retriever RAG (chargé après indexation)
|
| 61 |
|
| 62 |
def load_model():
|
| 63 |
"""
|
|
@@ -78,29 +65,26 @@ def load_model():
|
|
| 78 |
print("📥 Téléchargement du modèle fine-tuné et adapters depuis le Hub...")
|
| 79 |
print("⏳ Cela peut prendre 30-60 secondes...")
|
| 80 |
|
| 81 |
-
|
| 82 |
if torch.cuda.is_available():
|
| 83 |
-
print("🎮 Mode GPU: Chargement en float16
|
| 84 |
model = AutoPeftModelForCausalLM.from_pretrained(
|
| 85 |
config.LORA_MODEL_PATH,
|
| 86 |
-
|
| 87 |
-
device_map=
|
| 88 |
trust_remote_code=True,
|
| 89 |
low_cpu_mem_usage=True,
|
| 90 |
cache_dir="/tmp/transformers_cache"
|
| 91 |
)
|
| 92 |
-
|
| 93 |
else:
|
| 94 |
-
print("💻 Mode CPU: Chargement
|
| 95 |
print("⚠️ Inference sera plus lente (~30-60s par réponse)")
|
| 96 |
-
|
| 97 |
-
# Sur CPU: charger sans quantization (bitsandbytes ne fonctionne pas sur CPU)
|
| 98 |
model = AutoPeftModelForCausalLM.from_pretrained(
|
| 99 |
config.LORA_MODEL_PATH,
|
| 100 |
-
|
| 101 |
-
device_map={"": "cpu"}, # Forcer CPU
|
| 102 |
trust_remote_code=True,
|
| 103 |
-
low_cpu_mem_usage=True, # Optimisation mémoire CPU
|
| 104 |
cache_dir="/tmp/transformers_cache"
|
| 105 |
)
|
| 106 |
|
|
@@ -166,221 +150,6 @@ async def health_check():
|
|
| 166 |
"gpu_available": torch.cuda.is_available()
|
| 167 |
}
|
| 168 |
|
| 169 |
-
# Nouveaux modèles Pydantic pour RAG
|
| 170 |
-
class RAGSearchRequest(BaseModel):
|
| 171 |
-
query: str
|
| 172 |
-
top_k: int = 5
|
| 173 |
-
discipline: str = None
|
| 174 |
-
|
| 175 |
-
class RAGSearchResponse(BaseModel):
|
| 176 |
-
results: list
|
| 177 |
-
stats: dict
|
| 178 |
-
|
| 179 |
-
@app.post("/api/rag/index")
|
| 180 |
-
async def index_documents():
|
| 181 |
-
"""
|
| 182 |
-
Endpoint pour indexer les PDFs FEI
|
| 183 |
-
Lance le script d'indexation
|
| 184 |
-
"""
|
| 185 |
-
if not RAG_AVAILABLE:
|
| 186 |
-
raise HTTPException(status_code=503, detail="Modules RAG non disponibles")
|
| 187 |
-
|
| 188 |
-
try:
|
| 189 |
-
import subprocess
|
| 190 |
-
print("🚀 Lancement de l'indexation des PDFs...")
|
| 191 |
-
result = subprocess.run(
|
| 192 |
-
["python", "scripts/index_documents.py", "--all"],
|
| 193 |
-
capture_output=True,
|
| 194 |
-
text=True,
|
| 195 |
-
timeout=600 # 10 minutes max
|
| 196 |
-
)
|
| 197 |
-
|
| 198 |
-
if result.returncode == 0:
|
| 199 |
-
# Initialiser le retriever après indexation
|
| 200 |
-
global retriever
|
| 201 |
-
retriever = Retriever(top_k=5)
|
| 202 |
-
|
| 203 |
-
return {
|
| 204 |
-
"status": "success",
|
| 205 |
-
"message": "Indexation terminée avec succès",
|
| 206 |
-
"output": result.stdout,
|
| 207 |
-
"stats": retriever.get_stats()
|
| 208 |
-
}
|
| 209 |
-
else:
|
| 210 |
-
raise HTTPException(
|
| 211 |
-
status_code=500,
|
| 212 |
-
detail=f"Erreur d'indexation: {result.stderr}"
|
| 213 |
-
)
|
| 214 |
-
|
| 215 |
-
except subprocess.TimeoutExpired:
|
| 216 |
-
raise HTTPException(status_code=504, detail="Timeout: indexation > 10 minutes")
|
| 217 |
-
except Exception as e:
|
| 218 |
-
raise HTTPException(status_code=500, detail=f"Erreur: {str(e)}")
|
| 219 |
-
|
| 220 |
-
@app.post("/api/rag/search", response_model=RAGSearchResponse)
|
| 221 |
-
async def search_rag(request: RAGSearchRequest):
|
| 222 |
-
"""
|
| 223 |
-
Endpoint pour tester la recherche RAG
|
| 224 |
-
|
| 225 |
-
Args:
|
| 226 |
-
request: RAGSearchRequest avec query, top_k, discipline
|
| 227 |
-
|
| 228 |
-
Returns:
|
| 229 |
-
RAGSearchResponse avec les résultats et statistiques
|
| 230 |
-
"""
|
| 231 |
-
if not RAG_AVAILABLE:
|
| 232 |
-
raise HTTPException(status_code=503, detail="Modules RAG non disponibles")
|
| 233 |
-
|
| 234 |
-
global retriever
|
| 235 |
-
|
| 236 |
-
# Initialiser le retriever si pas encore fait
|
| 237 |
-
if retriever is None:
|
| 238 |
-
try:
|
| 239 |
-
retriever = Retriever(top_k=request.top_k)
|
| 240 |
-
except Exception as e:
|
| 241 |
-
raise HTTPException(
|
| 242 |
-
status_code=503,
|
| 243 |
-
detail=f"Base RAG non indexée. Lancez /api/rag/index d'abord. Erreur: {str(e)}"
|
| 244 |
-
)
|
| 245 |
-
|
| 246 |
-
try:
|
| 247 |
-
# Recherche RAG
|
| 248 |
-
results = retriever.search(
|
| 249 |
-
query=request.query,
|
| 250 |
-
top_k=request.top_k,
|
| 251 |
-
discipline=request.discipline
|
| 252 |
-
)
|
| 253 |
-
|
| 254 |
-
# Formater les résultats pour JSON
|
| 255 |
-
formatted_results = []
|
| 256 |
-
for r in results:
|
| 257 |
-
formatted_results.append({
|
| 258 |
-
"text": r["text"],
|
| 259 |
-
"score": float(r["score"]),
|
| 260 |
-
"metadata": r["metadata"],
|
| 261 |
-
"id": r["id"]
|
| 262 |
-
})
|
| 263 |
-
|
| 264 |
-
return RAGSearchResponse(
|
| 265 |
-
results=formatted_results,
|
| 266 |
-
stats=retriever.get_stats()
|
| 267 |
-
)
|
| 268 |
-
|
| 269 |
-
except Exception as e:
|
| 270 |
-
raise HTTPException(status_code=500, detail=f"Erreur de recherche: {str(e)}")
|
| 271 |
-
|
| 272 |
-
@app.get("/api/rag/stats")
|
| 273 |
-
async def rag_stats():
|
| 274 |
-
"""
|
| 275 |
-
Endpoint pour obtenir les statistiques de la base RAG
|
| 276 |
-
"""
|
| 277 |
-
if not RAG_AVAILABLE:
|
| 278 |
-
raise HTTPException(status_code=503, detail="Modules RAG non disponibles")
|
| 279 |
-
|
| 280 |
-
global retriever
|
| 281 |
-
|
| 282 |
-
if retriever is None:
|
| 283 |
-
try:
|
| 284 |
-
retriever = Retriever()
|
| 285 |
-
except:
|
| 286 |
-
return {"indexed": False, "message": "Base RAG non indexée"}
|
| 287 |
-
|
| 288 |
-
return {
|
| 289 |
-
"indexed": True,
|
| 290 |
-
"stats": retriever.get_stats()
|
| 291 |
-
}
|
| 292 |
-
|
| 293 |
-
@app.get("/api/rag/debug")
|
| 294 |
-
async def rag_debug():
|
| 295 |
-
"""
|
| 296 |
-
Endpoint de diagnostic pour tester les imports RAG
|
| 297 |
-
"""
|
| 298 |
-
results = {
|
| 299 |
-
"rag_available": RAG_AVAILABLE,
|
| 300 |
-
"imports": {}
|
| 301 |
-
}
|
| 302 |
-
|
| 303 |
-
# Test sentence-transformers
|
| 304 |
-
try:
|
| 305 |
-
import sentence_transformers
|
| 306 |
-
results["imports"]["sentence_transformers"] = {
|
| 307 |
-
"success": True,
|
| 308 |
-
"version": sentence_transformers.__version__
|
| 309 |
-
}
|
| 310 |
-
except Exception as e:
|
| 311 |
-
results["imports"]["sentence_transformers"] = {
|
| 312 |
-
"success": False,
|
| 313 |
-
"error": str(e)
|
| 314 |
-
}
|
| 315 |
-
|
| 316 |
-
# Test chromadb
|
| 317 |
-
try:
|
| 318 |
-
import chromadb
|
| 319 |
-
results["imports"]["chromadb"] = {
|
| 320 |
-
"success": True,
|
| 321 |
-
"version": chromadb.__version__
|
| 322 |
-
}
|
| 323 |
-
except Exception as e:
|
| 324 |
-
results["imports"]["chromadb"] = {
|
| 325 |
-
"success": False,
|
| 326 |
-
"error": str(e)
|
| 327 |
-
}
|
| 328 |
-
|
| 329 |
-
# Test PyPDF2
|
| 330 |
-
try:
|
| 331 |
-
import PyPDF2
|
| 332 |
-
results["imports"]["PyPDF2"] = {
|
| 333 |
-
"success": True,
|
| 334 |
-
"version": PyPDF2.__version__
|
| 335 |
-
}
|
| 336 |
-
except Exception as e:
|
| 337 |
-
results["imports"]["PyPDF2"] = {
|
| 338 |
-
"success": False,
|
| 339 |
-
"error": str(e)
|
| 340 |
-
}
|
| 341 |
-
|
| 342 |
-
# Test rag module files
|
| 343 |
-
import os
|
| 344 |
-
rag_path = os.path.join(os.path.dirname(__file__), "rag")
|
| 345 |
-
results["rag_directory"] = {
|
| 346 |
-
"exists": os.path.exists(rag_path),
|
| 347 |
-
"files": []
|
| 348 |
-
}
|
| 349 |
-
if os.path.exists(rag_path):
|
| 350 |
-
results["rag_directory"]["files"] = os.listdir(rag_path)
|
| 351 |
-
|
| 352 |
-
# Test rag.embedder import
|
| 353 |
-
try:
|
| 354 |
-
from rag import embedder
|
| 355 |
-
results["imports"]["rag.embedder"] = {"success": True}
|
| 356 |
-
except Exception as e:
|
| 357 |
-
results["imports"]["rag.embedder"] = {
|
| 358 |
-
"success": False,
|
| 359 |
-
"error": str(e)
|
| 360 |
-
}
|
| 361 |
-
|
| 362 |
-
# Test rag.vector_store import
|
| 363 |
-
try:
|
| 364 |
-
from rag import vector_store
|
| 365 |
-
results["imports"]["rag.vector_store"] = {"success": True}
|
| 366 |
-
except Exception as e:
|
| 367 |
-
results["imports"]["rag.vector_store"] = {
|
| 368 |
-
"success": False,
|
| 369 |
-
"error": str(e)
|
| 370 |
-
}
|
| 371 |
-
|
| 372 |
-
# Test rag.retriever import
|
| 373 |
-
try:
|
| 374 |
-
from rag import retriever as ret_module
|
| 375 |
-
results["imports"]["rag.retriever"] = {"success": True}
|
| 376 |
-
except Exception as e:
|
| 377 |
-
results["imports"]["rag.retriever"] = {
|
| 378 |
-
"success": False,
|
| 379 |
-
"error": str(e)
|
| 380 |
-
}
|
| 381 |
-
|
| 382 |
-
return results
|
| 383 |
-
|
| 384 |
@app.post("/api/chat", response_model=ChatResponse)
|
| 385 |
async def chat(request: ChatRequest):
|
| 386 |
"""
|
|
|
|
| 15 |
|
| 16 |
import config
|
| 17 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
# Désactiver le parallélisme des tokenizers
|
| 19 |
os.environ["TOKENIZERS_PARALLELISM"] = "false"
|
| 20 |
|
|
|
|
| 45 |
model = None
|
| 46 |
tokenizer = None
|
| 47 |
text_generator = None
|
|
|
|
| 48 |
|
| 49 |
def load_model():
|
| 50 |
"""
|
|
|
|
| 65 |
print("📥 Téléchargement du modèle fine-tuné et adapters depuis le Hub...")
|
| 66 |
print("⏳ Cela peut prendre 30-60 secondes...")
|
| 67 |
|
| 68 |
+
# Détecter GPU vs CPU
|
| 69 |
if torch.cuda.is_available():
|
| 70 |
+
print("🎮 Mode GPU: Chargement en float16 avec device_map=auto")
|
| 71 |
model = AutoPeftModelForCausalLM.from_pretrained(
|
| 72 |
config.LORA_MODEL_PATH,
|
| 73 |
+
dtype=torch.float16,
|
| 74 |
+
device_map="auto",
|
| 75 |
trust_remote_code=True,
|
| 76 |
low_cpu_mem_usage=True,
|
| 77 |
cache_dir="/tmp/transformers_cache"
|
| 78 |
)
|
|
|
|
| 79 |
else:
|
| 80 |
+
print("💻 Mode CPU: Chargement en 8-bit pour réduire la mémoire")
|
| 81 |
print("⚠️ Inference sera plus lente (~30-60s par réponse)")
|
| 82 |
+
# Sur CPU: forcer device_map explicite + 8bit pour éviter offloading vers disk
|
|
|
|
| 83 |
model = AutoPeftModelForCausalLM.from_pretrained(
|
| 84 |
config.LORA_MODEL_PATH,
|
| 85 |
+
load_in_8bit=True, # Quantization 8-bit pour CPU
|
| 86 |
+
device_map={"": "cpu"}, # Forcer CPU sans offloading
|
| 87 |
trust_remote_code=True,
|
|
|
|
| 88 |
cache_dir="/tmp/transformers_cache"
|
| 89 |
)
|
| 90 |
|
|
|
|
| 150 |
"gpu_available": torch.cuda.is_available()
|
| 151 |
}
|
| 152 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 153 |
@app.post("/api/chat", response_model=ChatResponse)
|
| 154 |
async def chat(request: ChatRequest):
|
| 155 |
"""
|
rag/__init__.py
DELETED
|
@@ -1,24 +0,0 @@
|
|
| 1 |
-
"""
|
| 2 |
-
Module RAG pour le Coach Équestre AI
|
| 3 |
-
|
| 4 |
-
Ce module fournit les fonctionnalités de Retrieval-Augmented Generation
|
| 5 |
-
pour enrichir les réponses du coach avec des connaissances factuelles
|
| 6 |
-
issues des règlements FEI.
|
| 7 |
-
|
| 8 |
-
Phase 1 (Fondation):
|
| 9 |
-
- embedder: Conversion texte → vecteurs (multilingual-e5-base)
|
| 10 |
-
- vector_store: Stockage et recherche vectorielle (ChromaDB)
|
| 11 |
-
- retriever: Pipeline RAG simple (vector search only)
|
| 12 |
-
|
| 13 |
-
Usage:
|
| 14 |
-
from rag import Retriever
|
| 15 |
-
|
| 16 |
-
retriever = Retriever()
|
| 17 |
-
results = retriever.search("Quelle est la hauteur pour CSO Amateur 1?")
|
| 18 |
-
"""
|
| 19 |
-
|
| 20 |
-
from .embedder import Embedder
|
| 21 |
-
from .vector_store import VectorStore
|
| 22 |
-
from .retriever import Retriever
|
| 23 |
-
|
| 24 |
-
__all__ = ["Embedder", "VectorStore", "Retriever"]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
rag/embedder.py
DELETED
|
@@ -1,139 +0,0 @@
|
|
| 1 |
-
"""
|
| 2 |
-
Embedder - Conversion texte → vecteurs avec multilingual-e5-base
|
| 3 |
-
|
| 4 |
-
Ce module encapsule le modèle sentence-transformer multilingual-e5-base
|
| 5 |
-
pour générer des embeddings vectoriels de texte en français et anglais.
|
| 6 |
-
|
| 7 |
-
Modèle: intfloat/multilingual-e5-base
|
| 8 |
-
- Dimensions: 768
|
| 9 |
-
- Taille: ~560MB
|
| 10 |
-
- Langues: 100+ (dont FR et EN)
|
| 11 |
-
- Usage: Questions + Documents FEI
|
| 12 |
-
"""
|
| 13 |
-
|
| 14 |
-
from typing import List, Union
|
| 15 |
-
import torch
|
| 16 |
-
from sentence_transformers import SentenceTransformer
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
class Embedder:
|
| 20 |
-
"""
|
| 21 |
-
Wrapper pour le modèle d'embedding multilingual-e5-base
|
| 22 |
-
|
| 23 |
-
Convertit du texte (questions ou documents) en vecteurs 768D
|
| 24 |
-
pour la recherche de similarité sémantique.
|
| 25 |
-
"""
|
| 26 |
-
|
| 27 |
-
def __init__(
|
| 28 |
-
self,
|
| 29 |
-
model_name: str = "intfloat/multilingual-e5-base",
|
| 30 |
-
device: str = None,
|
| 31 |
-
cache_dir: str = "/tmp/sentence_transformers_cache"
|
| 32 |
-
):
|
| 33 |
-
"""
|
| 34 |
-
Initialise le modèle d'embedding
|
| 35 |
-
|
| 36 |
-
Args:
|
| 37 |
-
model_name: Nom du modèle HuggingFace
|
| 38 |
-
device: "cuda", "cpu", ou None (auto-détection)
|
| 39 |
-
cache_dir: Dossier de cache pour le modèle
|
| 40 |
-
"""
|
| 41 |
-
# Auto-détection du device si non spécifié
|
| 42 |
-
if device is None:
|
| 43 |
-
device = "cuda" if torch.cuda.is_available() else "cpu"
|
| 44 |
-
|
| 45 |
-
self.device = device
|
| 46 |
-
self.model_name = model_name
|
| 47 |
-
|
| 48 |
-
print(f"📥 Chargement du modèle d'embedding: {model_name}")
|
| 49 |
-
print(f"💻 Device: {device}")
|
| 50 |
-
|
| 51 |
-
# Charger le modèle sentence-transformer
|
| 52 |
-
self.model = SentenceTransformer(
|
| 53 |
-
model_name,
|
| 54 |
-
device=device,
|
| 55 |
-
cache_folder=cache_dir
|
| 56 |
-
)
|
| 57 |
-
|
| 58 |
-
print(f"✅ Modèle chargé (dimensions: {self.model.get_sentence_embedding_dimension()})")
|
| 59 |
-
|
| 60 |
-
def embed_query(self, query: str) -> List[float]:
|
| 61 |
-
"""
|
| 62 |
-
Génère l'embedding d'une question utilisateur
|
| 63 |
-
|
| 64 |
-
Args:
|
| 65 |
-
query: Question de l'utilisateur
|
| 66 |
-
|
| 67 |
-
Returns:
|
| 68 |
-
Vecteur d'embedding (768 dimensions)
|
| 69 |
-
"""
|
| 70 |
-
# E5 models require "query: " prefix for queries
|
| 71 |
-
prefixed_query = f"query: {query}"
|
| 72 |
-
|
| 73 |
-
embedding = self.model.encode(
|
| 74 |
-
prefixed_query,
|
| 75 |
-
convert_to_numpy=True,
|
| 76 |
-
normalize_embeddings=True # Normalisation pour similarité cosinus
|
| 77 |
-
)
|
| 78 |
-
|
| 79 |
-
return embedding.tolist()
|
| 80 |
-
|
| 81 |
-
def embed_documents(self, documents: List[str]) -> List[List[float]]:
|
| 82 |
-
"""
|
| 83 |
-
Génère les embeddings de plusieurs documents
|
| 84 |
-
|
| 85 |
-
Args:
|
| 86 |
-
documents: Liste de textes/chunks à embedder
|
| 87 |
-
|
| 88 |
-
Returns:
|
| 89 |
-
Liste de vecteurs d'embedding
|
| 90 |
-
"""
|
| 91 |
-
# E5 models require "passage: " prefix for documents
|
| 92 |
-
prefixed_docs = [f"passage: {doc}" for doc in documents]
|
| 93 |
-
|
| 94 |
-
embeddings = self.model.encode(
|
| 95 |
-
prefixed_docs,
|
| 96 |
-
convert_to_numpy=True,
|
| 97 |
-
normalize_embeddings=True,
|
| 98 |
-
show_progress_bar=len(documents) > 10 # Progress bar si > 10 docs
|
| 99 |
-
)
|
| 100 |
-
|
| 101 |
-
return embeddings.tolist()
|
| 102 |
-
|
| 103 |
-
def embed_single_document(self, document: str) -> List[float]:
|
| 104 |
-
"""
|
| 105 |
-
Génère l'embedding d'un seul document
|
| 106 |
-
|
| 107 |
-
Args:
|
| 108 |
-
document: Texte/chunk à embedder
|
| 109 |
-
|
| 110 |
-
Returns:
|
| 111 |
-
Vecteur d'embedding (768 dimensions)
|
| 112 |
-
"""
|
| 113 |
-
return self.embed_documents([document])[0]
|
| 114 |
-
|
| 115 |
-
def get_embedding_dimension(self) -> int:
|
| 116 |
-
"""Retourne la dimension des embeddings (768 pour E5-base)"""
|
| 117 |
-
return self.model.get_sentence_embedding_dimension()
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
if __name__ == "__main__":
|
| 121 |
-
# Test du module
|
| 122 |
-
print("🧪 Test du module Embedder\n")
|
| 123 |
-
|
| 124 |
-
embedder = Embedder()
|
| 125 |
-
|
| 126 |
-
# Test embedding query
|
| 127 |
-
query = "Quelle est la hauteur maximale pour CSO Amateur 1?"
|
| 128 |
-
query_embedding = embedder.embed_query(query)
|
| 129 |
-
print(f"\n✅ Query embedding: {len(query_embedding)} dimensions")
|
| 130 |
-
print(f" Premiers éléments: {query_embedding[:5]}")
|
| 131 |
-
|
| 132 |
-
# Test embedding documents
|
| 133 |
-
docs = [
|
| 134 |
-
"La hauteur maximale pour le CSO Amateur 1 est de 1,15m.",
|
| 135 |
-
"Les pénalités en CSO sont de 4 points par barre tombée."
|
| 136 |
-
]
|
| 137 |
-
doc_embeddings = embedder.embed_documents(docs)
|
| 138 |
-
print(f"\n✅ Document embeddings: {len(doc_embeddings)} vecteurs")
|
| 139 |
-
print(f" Dimensions: {len(doc_embeddings[0])}")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
rag/retriever.py
DELETED
|
@@ -1,201 +0,0 @@
|
|
| 1 |
-
"""
|
| 2 |
-
Retriever - Pipeline RAG Simple (Phase 1)
|
| 3 |
-
|
| 4 |
-
Ce module orchestre le pipeline RAG complet:
|
| 5 |
-
1. Embedding de la question
|
| 6 |
-
2. Recherche vectorielle dans ChromaDB
|
| 7 |
-
3. Retour des passages pertinents
|
| 8 |
-
|
| 9 |
-
Phase 1: Vector search only (pas encore de reranking)
|
| 10 |
-
"""
|
| 11 |
-
|
| 12 |
-
from typing import List, Dict, Any, Optional
|
| 13 |
-
from .embedder import Embedder
|
| 14 |
-
from .vector_store import VectorStore
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
class Retriever:
|
| 18 |
-
"""
|
| 19 |
-
Pipeline RAG simple pour récupérer des passages pertinents
|
| 20 |
-
|
| 21 |
-
Combine Embedder + VectorStore pour rechercher dans les règlements FEI
|
| 22 |
-
"""
|
| 23 |
-
|
| 24 |
-
def __init__(
|
| 25 |
-
self,
|
| 26 |
-
embedder: Optional[Embedder] = None,
|
| 27 |
-
vector_store: Optional[VectorStore] = None,
|
| 28 |
-
top_k: int = 5
|
| 29 |
-
):
|
| 30 |
-
"""
|
| 31 |
-
Initialise le retriever avec embedder et vector store
|
| 32 |
-
|
| 33 |
-
Args:
|
| 34 |
-
embedder: Instance d'Embedder (créé par défaut si None)
|
| 35 |
-
vector_store: Instance de VectorStore (créé par défaut si None)
|
| 36 |
-
top_k: Nombre de résultats à retourner (default: 5)
|
| 37 |
-
"""
|
| 38 |
-
self.top_k = top_k
|
| 39 |
-
|
| 40 |
-
print("🚀 Initialisation du Retriever (Pipeline RAG Phase 1)")
|
| 41 |
-
|
| 42 |
-
# Créer embedder si non fourni
|
| 43 |
-
if embedder is None:
|
| 44 |
-
print("📥 Création de l'Embedder...")
|
| 45 |
-
self.embedder = Embedder()
|
| 46 |
-
else:
|
| 47 |
-
self.embedder = embedder
|
| 48 |
-
|
| 49 |
-
# Créer vector store si non fourni
|
| 50 |
-
if vector_store is None:
|
| 51 |
-
print("📂 Création du VectorStore...")
|
| 52 |
-
self.vector_store = VectorStore()
|
| 53 |
-
else:
|
| 54 |
-
self.vector_store = vector_store
|
| 55 |
-
|
| 56 |
-
# Afficher les stats de la base
|
| 57 |
-
stats = self.vector_store.get_stats()
|
| 58 |
-
print(f"✅ Retriever prêt!")
|
| 59 |
-
print(f" 📊 Base: {stats['total_documents']} documents indexés")
|
| 60 |
-
print(f" 🏷️ Disciplines: {list(stats['disciplines'].keys())}")
|
| 61 |
-
print(f" 📚 Sources: {stats['unique_sources']} fichiers")
|
| 62 |
-
|
| 63 |
-
def search(
|
| 64 |
-
self,
|
| 65 |
-
query: str,
|
| 66 |
-
top_k: Optional[int] = None,
|
| 67 |
-
discipline: Optional[str] = None
|
| 68 |
-
) -> List[Dict[str, Any]]:
|
| 69 |
-
"""
|
| 70 |
-
Recherche les passages les plus pertinents pour une question
|
| 71 |
-
|
| 72 |
-
Args:
|
| 73 |
-
query: Question de l'utilisateur
|
| 74 |
-
top_k: Nombre de résultats (utilise self.top_k si None)
|
| 75 |
-
discipline: Filtrer par discipline ("CSO", "Dressage", "CCE")
|
| 76 |
-
|
| 77 |
-
Returns:
|
| 78 |
-
Liste de dictionnaires avec:
|
| 79 |
-
- text: Le passage de texte
|
| 80 |
-
- metadata: Métadonnées (discipline, source, page)
|
| 81 |
-
- score: Score de similarité (1 - distance cosinus)
|
| 82 |
-
"""
|
| 83 |
-
if top_k is None:
|
| 84 |
-
top_k = self.top_k
|
| 85 |
-
|
| 86 |
-
print(f"\n🔍 Question: {query}")
|
| 87 |
-
|
| 88 |
-
# 1. Générer l'embedding de la question
|
| 89 |
-
print(" 📝 Génération embedding de la question...")
|
| 90 |
-
query_embedding = self.embedder.embed_query(query)
|
| 91 |
-
|
| 92 |
-
# 2. Filtrer par discipline si spécifié
|
| 93 |
-
filter_metadata = None
|
| 94 |
-
if discipline:
|
| 95 |
-
filter_metadata = {"discipline": discipline}
|
| 96 |
-
print(f" 🏷️ Filtre: discipline={discipline}")
|
| 97 |
-
|
| 98 |
-
# 3. Recherche dans ChromaDB
|
| 99 |
-
print(f" 🔍 Recherche dans la base (top_k={top_k})...")
|
| 100 |
-
results = self.vector_store.search(
|
| 101 |
-
query_embedding=query_embedding,
|
| 102 |
-
top_k=top_k,
|
| 103 |
-
filter_metadata=filter_metadata
|
| 104 |
-
)
|
| 105 |
-
|
| 106 |
-
# 4. Formater les résultats
|
| 107 |
-
formatted_results = []
|
| 108 |
-
|
| 109 |
-
for i in range(len(results["ids"])):
|
| 110 |
-
# Convertir distance cosinus en score de similarité (1 - distance)
|
| 111 |
-
# Distance = 0 → Score = 1 (parfait match)
|
| 112 |
-
# Distance = 2 → Score = -1 (opposé)
|
| 113 |
-
similarity_score = 1 - results["distances"][i]
|
| 114 |
-
|
| 115 |
-
formatted_results.append({
|
| 116 |
-
"text": results["documents"][i],
|
| 117 |
-
"metadata": results["metadatas"][i],
|
| 118 |
-
"score": similarity_score,
|
| 119 |
-
"id": results["ids"][i]
|
| 120 |
-
})
|
| 121 |
-
|
| 122 |
-
print(f" ✅ Trouvé {len(formatted_results)} résultats")
|
| 123 |
-
|
| 124 |
-
# Afficher les meilleurs résultats
|
| 125 |
-
if formatted_results:
|
| 126 |
-
print(f"\n 🏆 Top 3 résultats:")
|
| 127 |
-
for i, result in enumerate(formatted_results[:3], 1):
|
| 128 |
-
metadata = result["metadata"]
|
| 129 |
-
text_preview = result["text"][:100].replace("\n", " ")
|
| 130 |
-
print(f" {i}. Score: {result['score']:.3f}")
|
| 131 |
-
print(f" Source: {metadata.get('source_file', 'N/A')} (page {metadata.get('page_number', '?')})")
|
| 132 |
-
print(f" Texte: {text_preview}...")
|
| 133 |
-
|
| 134 |
-
return formatted_results
|
| 135 |
-
|
| 136 |
-
def format_context(
|
| 137 |
-
self,
|
| 138 |
-
results: List[Dict[str, Any]],
|
| 139 |
-
max_results: int = 3
|
| 140 |
-
) -> str:
|
| 141 |
-
"""
|
| 142 |
-
Formate les résultats de recherche en contexte pour le LLM
|
| 143 |
-
|
| 144 |
-
Args:
|
| 145 |
-
results: Résultats de la recherche
|
| 146 |
-
max_results: Nombre maximum de passages à inclure
|
| 147 |
-
|
| 148 |
-
Returns:
|
| 149 |
-
Contexte formaté avec sources
|
| 150 |
-
"""
|
| 151 |
-
if not results:
|
| 152 |
-
return ""
|
| 153 |
-
|
| 154 |
-
context_parts = []
|
| 155 |
-
|
| 156 |
-
for i, result in enumerate(results[:max_results], 1):
|
| 157 |
-
metadata = result["metadata"]
|
| 158 |
-
source = metadata.get("source_file", "Source inconnue")
|
| 159 |
-
page = metadata.get("page_number", "?")
|
| 160 |
-
text = result["text"]
|
| 161 |
-
|
| 162 |
-
context_parts.append(
|
| 163 |
-
f"[Source {i}: {source}, page {page}]\n{text}"
|
| 164 |
-
)
|
| 165 |
-
|
| 166 |
-
return "\n\n".join(context_parts)
|
| 167 |
-
|
| 168 |
-
def get_stats(self) -> Dict[str, Any]:
|
| 169 |
-
"""Retourne les statistiques de la base vectorielle"""
|
| 170 |
-
return self.vector_store.get_stats()
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
if __name__ == "__main__":
|
| 174 |
-
# Test du module
|
| 175 |
-
print("🧪 Test du module Retriever\n")
|
| 176 |
-
|
| 177 |
-
# Créer un retriever (utilise les instances par défaut)
|
| 178 |
-
retriever = Retriever(top_k=3)
|
| 179 |
-
|
| 180 |
-
# Vérifier que la base est vide
|
| 181 |
-
stats = retriever.get_stats()
|
| 182 |
-
|
| 183 |
-
if stats["total_documents"] == 0:
|
| 184 |
-
print("\n⚠️ La base est vide. Utilisez index_documents.py pour indexer les PDFs FEI d'abord.")
|
| 185 |
-
else:
|
| 186 |
-
# Test de recherche
|
| 187 |
-
test_queries = [
|
| 188 |
-
"Quelle est la hauteur maximale pour CSO Amateur 1?",
|
| 189 |
-
"Quelles sont les pénalités en cas de chute?",
|
| 190 |
-
"Équipement obligatoire en dressage"
|
| 191 |
-
]
|
| 192 |
-
|
| 193 |
-
for query in test_queries:
|
| 194 |
-
results = retriever.search(query, top_k=3)
|
| 195 |
-
|
| 196 |
-
if results:
|
| 197 |
-
print(f"\n✅ Contexte formaté pour LLM:")
|
| 198 |
-
context = retriever.format_context(results)
|
| 199 |
-
print(context[:300] + "...")
|
| 200 |
-
|
| 201 |
-
print("\n" + "="*80)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
rag/vector_store.py
DELETED
|
@@ -1,267 +0,0 @@
|
|
| 1 |
-
"""
|
| 2 |
-
VectorStore - Stockage et recherche vectorielle avec ChromaDB
|
| 3 |
-
|
| 4 |
-
Ce module gère le stockage persistant des embeddings et la recherche
|
| 5 |
-
par similarité sémantique dans les règlements FEI.
|
| 6 |
-
|
| 7 |
-
Base de données: ChromaDB
|
| 8 |
-
- Persistence: backend/data/chroma_db/
|
| 9 |
-
- Metadata filtering: discipline, source_file, page_number
|
| 10 |
-
- Distance metric: Cosine similarity
|
| 11 |
-
"""
|
| 12 |
-
|
| 13 |
-
from typing import List, Dict, Any, Optional
|
| 14 |
-
import chromadb
|
| 15 |
-
from chromadb.config import Settings
|
| 16 |
-
from chromadb.utils import embedding_functions
|
| 17 |
-
import os
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
class VectorStore:
|
| 21 |
-
"""
|
| 22 |
-
Client ChromaDB pour stocker et rechercher des embeddings
|
| 23 |
-
|
| 24 |
-
Gère une collection par discipline (CSO, Dressage, CCE) avec
|
| 25 |
-
persistence sur disque et métadonnées enrichies.
|
| 26 |
-
"""
|
| 27 |
-
|
| 28 |
-
def __init__(
|
| 29 |
-
self,
|
| 30 |
-
persist_directory: str = "backend/data/chroma_db",
|
| 31 |
-
collection_name: str = "fei_rules"
|
| 32 |
-
):
|
| 33 |
-
"""
|
| 34 |
-
Initialise le client ChromaDB avec persistence
|
| 35 |
-
|
| 36 |
-
Args:
|
| 37 |
-
persist_directory: Dossier de persistence ChromaDB
|
| 38 |
-
collection_name: Nom de la collection (default: "fei_rules")
|
| 39 |
-
"""
|
| 40 |
-
self.persist_directory = os.path.abspath(persist_directory)
|
| 41 |
-
self.collection_name = collection_name
|
| 42 |
-
|
| 43 |
-
# Créer le dossier si nécessaire
|
| 44 |
-
os.makedirs(self.persist_directory, exist_ok=True)
|
| 45 |
-
|
| 46 |
-
print(f"📂 Initialisation ChromaDB")
|
| 47 |
-
print(f" Persistence: {self.persist_directory}")
|
| 48 |
-
print(f" Collection: {collection_name}")
|
| 49 |
-
|
| 50 |
-
# Initialiser le client ChromaDB avec persistence
|
| 51 |
-
self.client = chromadb.PersistentClient(
|
| 52 |
-
path=self.persist_directory,
|
| 53 |
-
settings=Settings(
|
| 54 |
-
anonymized_telemetry=False,
|
| 55 |
-
allow_reset=True
|
| 56 |
-
)
|
| 57 |
-
)
|
| 58 |
-
|
| 59 |
-
# Créer ou récupérer la collection
|
| 60 |
-
# Note: On utilise pas d'embedding function car on fournit les embeddings manuellement
|
| 61 |
-
try:
|
| 62 |
-
self.collection = self.client.get_collection(name=collection_name)
|
| 63 |
-
doc_count = self.collection.count()
|
| 64 |
-
print(f"✅ Collection existante chargée ({doc_count} documents)")
|
| 65 |
-
except:
|
| 66 |
-
self.collection = self.client.create_collection(
|
| 67 |
-
name=collection_name,
|
| 68 |
-
metadata={"description": "Règlements FEI pour RAG Coach Équestre"}
|
| 69 |
-
)
|
| 70 |
-
print(f"✅ Nouvelle collection créée")
|
| 71 |
-
|
| 72 |
-
def add_documents(
|
| 73 |
-
self,
|
| 74 |
-
documents: List[str],
|
| 75 |
-
embeddings: List[List[float]],
|
| 76 |
-
metadatas: List[Dict[str, Any]],
|
| 77 |
-
ids: Optional[List[str]] = None
|
| 78 |
-
) -> None:
|
| 79 |
-
"""
|
| 80 |
-
Ajoute des documents avec leurs embeddings à la collection
|
| 81 |
-
|
| 82 |
-
Args:
|
| 83 |
-
documents: Liste de textes/chunks
|
| 84 |
-
embeddings: Liste de vecteurs d'embedding (768D)
|
| 85 |
-
metadatas: Liste de métadonnées (discipline, source_file, page_number, etc.)
|
| 86 |
-
ids: IDs optionnels (générés automatiquement si None)
|
| 87 |
-
"""
|
| 88 |
-
# Générer des IDs si non fournis
|
| 89 |
-
if ids is None:
|
| 90 |
-
start_id = self.collection.count()
|
| 91 |
-
ids = [f"doc_{start_id + i}" for i in range(len(documents))]
|
| 92 |
-
|
| 93 |
-
print(f"➕ Ajout de {len(documents)} documents à ChromaDB...")
|
| 94 |
-
|
| 95 |
-
# Ajouter à la collection
|
| 96 |
-
self.collection.add(
|
| 97 |
-
documents=documents,
|
| 98 |
-
embeddings=embeddings,
|
| 99 |
-
metadatas=metadatas,
|
| 100 |
-
ids=ids
|
| 101 |
-
)
|
| 102 |
-
|
| 103 |
-
print(f"✅ Documents ajoutés (total: {self.collection.count()})")
|
| 104 |
-
|
| 105 |
-
def search(
|
| 106 |
-
self,
|
| 107 |
-
query_embedding: List[float],
|
| 108 |
-
top_k: int = 10,
|
| 109 |
-
filter_metadata: Optional[Dict[str, Any]] = None
|
| 110 |
-
) -> Dict[str, Any]:
|
| 111 |
-
"""
|
| 112 |
-
Recherche les documents les plus similaires à la query
|
| 113 |
-
|
| 114 |
-
Args:
|
| 115 |
-
query_embedding: Vecteur d'embedding de la question (768D)
|
| 116 |
-
top_k: Nombre de résultats à retourner
|
| 117 |
-
filter_metadata: Filtres optionnels (ex: {"discipline": "CSO"})
|
| 118 |
-
|
| 119 |
-
Returns:
|
| 120 |
-
Dictionnaire avec ids, documents, metadatas, distances
|
| 121 |
-
"""
|
| 122 |
-
print(f"🔍 Recherche dans ChromaDB (top_k={top_k})")
|
| 123 |
-
|
| 124 |
-
# Construire la where clause pour le filtering
|
| 125 |
-
where = filter_metadata if filter_metadata else None
|
| 126 |
-
|
| 127 |
-
# Recherche par similarité
|
| 128 |
-
results = self.collection.query(
|
| 129 |
-
query_embeddings=[query_embedding],
|
| 130 |
-
n_results=top_k,
|
| 131 |
-
where=where,
|
| 132 |
-
include=["documents", "metadatas", "distances"]
|
| 133 |
-
)
|
| 134 |
-
|
| 135 |
-
# Format des résultats ChromaDB: listes imbriquées car batch queries supportés
|
| 136 |
-
# On retourne juste le premier (et seul) résultat de query
|
| 137 |
-
formatted_results = {
|
| 138 |
-
"ids": results["ids"][0],
|
| 139 |
-
"documents": results["documents"][0],
|
| 140 |
-
"metadatas": results["metadatas"][0],
|
| 141 |
-
"distances": results["distances"][0]
|
| 142 |
-
}
|
| 143 |
-
|
| 144 |
-
print(f"✅ Trouvé {len(formatted_results['ids'])} résultats")
|
| 145 |
-
|
| 146 |
-
return formatted_results
|
| 147 |
-
|
| 148 |
-
def delete_by_source(self, source_file: str) -> None:
|
| 149 |
-
"""
|
| 150 |
-
Supprime tous les documents d'un fichier source
|
| 151 |
-
|
| 152 |
-
Utile pour re-indexer un PDF mis à jour
|
| 153 |
-
|
| 154 |
-
Args:
|
| 155 |
-
source_file: Nom du fichier source (ex: "FEI_Jumping_Rules_2024.pdf")
|
| 156 |
-
"""
|
| 157 |
-
print(f"🗑️ Suppression des documents de: {source_file}")
|
| 158 |
-
|
| 159 |
-
# ChromaDB where clause pour filtrer par source
|
| 160 |
-
self.collection.delete(
|
| 161 |
-
where={"source_file": source_file}
|
| 162 |
-
)
|
| 163 |
-
|
| 164 |
-
print(f"✅ Documents supprimés (total restant: {self.collection.count()})")
|
| 165 |
-
|
| 166 |
-
def reset(self) -> None:
|
| 167 |
-
"""
|
| 168 |
-
Supprime TOUS les documents de la collection
|
| 169 |
-
|
| 170 |
-
⚠️ Utiliser avec précaution! Toutes les données seront perdues.
|
| 171 |
-
"""
|
| 172 |
-
print("⚠️ Reset de la collection (suppression de tous les documents)")
|
| 173 |
-
|
| 174 |
-
count_before = self.collection.count()
|
| 175 |
-
self.client.delete_collection(name=self.collection_name)
|
| 176 |
-
|
| 177 |
-
# Re-créer la collection vide
|
| 178 |
-
self.collection = self.client.create_collection(
|
| 179 |
-
name=self.collection_name,
|
| 180 |
-
metadata={"description": "Règlements FEI pour RAG Coach Équestre"}
|
| 181 |
-
)
|
| 182 |
-
|
| 183 |
-
print(f"✅ Collection reset ({count_before} documents supprimés)")
|
| 184 |
-
|
| 185 |
-
def get_stats(self) -> Dict[str, Any]:
|
| 186 |
-
"""
|
| 187 |
-
Retourne des statistiques sur la collection
|
| 188 |
-
|
| 189 |
-
Returns:
|
| 190 |
-
Dictionnaire avec count, disciplines, sources
|
| 191 |
-
"""
|
| 192 |
-
total_docs = self.collection.count()
|
| 193 |
-
|
| 194 |
-
# Pour les stats détaillées, on doit fetch tous les metadatas
|
| 195 |
-
# (ChromaDB ne supporte pas les aggregations directement)
|
| 196 |
-
if total_docs > 0:
|
| 197 |
-
all_docs = self.collection.get(include=["metadatas"])
|
| 198 |
-
|
| 199 |
-
# Compter par discipline
|
| 200 |
-
disciplines = {}
|
| 201 |
-
sources = set()
|
| 202 |
-
|
| 203 |
-
for metadata in all_docs["metadatas"]:
|
| 204 |
-
disc = metadata.get("discipline", "unknown")
|
| 205 |
-
disciplines[disc] = disciplines.get(disc, 0) + 1
|
| 206 |
-
sources.add(metadata.get("source_file", "unknown"))
|
| 207 |
-
|
| 208 |
-
return {
|
| 209 |
-
"total_documents": total_docs,
|
| 210 |
-
"disciplines": disciplines,
|
| 211 |
-
"unique_sources": len(sources),
|
| 212 |
-
"sources": sorted(list(sources))
|
| 213 |
-
}
|
| 214 |
-
else:
|
| 215 |
-
return {
|
| 216 |
-
"total_documents": 0,
|
| 217 |
-
"disciplines": {},
|
| 218 |
-
"unique_sources": 0,
|
| 219 |
-
"sources": []
|
| 220 |
-
}
|
| 221 |
-
|
| 222 |
-
|
| 223 |
-
if __name__ == "__main__":
|
| 224 |
-
# Test du module
|
| 225 |
-
print("🧪 Test du module VectorStore\n")
|
| 226 |
-
|
| 227 |
-
# Créer un vector store de test
|
| 228 |
-
vector_store = VectorStore(
|
| 229 |
-
persist_directory="/tmp/test_chroma",
|
| 230 |
-
collection_name="test_collection"
|
| 231 |
-
)
|
| 232 |
-
|
| 233 |
-
# Reset pour partir de zéro
|
| 234 |
-
vector_store.reset()
|
| 235 |
-
|
| 236 |
-
# Ajouter des documents de test
|
| 237 |
-
test_docs = [
|
| 238 |
-
"La hauteur maximale pour le CSO Amateur 1 est de 1,15m.",
|
| 239 |
-
"Les pénalités en CSO sont de 4 points par barre tombée.",
|
| 240 |
-
"En dressage, la reprise imposée dure 5 minutes maximum."
|
| 241 |
-
]
|
| 242 |
-
|
| 243 |
-
# Embeddings factices (768D de zéros pour le test)
|
| 244 |
-
test_embeddings = [[0.0] * 768 for _ in range(len(test_docs))]
|
| 245 |
-
|
| 246 |
-
test_metadatas = [
|
| 247 |
-
{"discipline": "CSO", "source_file": "test.pdf", "page_number": 1},
|
| 248 |
-
{"discipline": "CSO", "source_file": "test.pdf", "page_number": 2},
|
| 249 |
-
{"discipline": "Dressage", "source_file": "test2.pdf", "page_number": 1}
|
| 250 |
-
]
|
| 251 |
-
|
| 252 |
-
vector_store.add_documents(test_docs, test_embeddings, test_metadatas)
|
| 253 |
-
|
| 254 |
-
# Test recherche
|
| 255 |
-
query_embedding = [0.0] * 768
|
| 256 |
-
results = vector_store.search(query_embedding, top_k=2)
|
| 257 |
-
|
| 258 |
-
print(f"\n✅ Résultats de recherche:")
|
| 259 |
-
for i, doc in enumerate(results["documents"]):
|
| 260 |
-
print(f" {i+1}. {doc[:60]}... (distance: {results['distances'][i]:.4f})")
|
| 261 |
-
|
| 262 |
-
# Test statistiques
|
| 263 |
-
stats = vector_store.get_stats()
|
| 264 |
-
print(f"\n✅ Statistiques:")
|
| 265 |
-
print(f" Total: {stats['total_documents']} documents")
|
| 266 |
-
print(f" Disciplines: {stats['disciplines']}")
|
| 267 |
-
print(f" Sources: {stats['sources']}")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
requirements.txt
CHANGED
|
@@ -13,13 +13,6 @@ peft==0.18.0 # Version utilisée pour l'entraînement
|
|
| 13 |
accelerate==1.2.0
|
| 14 |
# bitsandbytes retiré: ne fonctionne pas sur CPU (nécessite GPU CUDA)
|
| 15 |
|
| 16 |
-
# RAG System Dependencies
|
| 17 |
-
PyPDF2==3.0.1 # PDF text extraction
|
| 18 |
-
sentence-transformers==2.3.1 # Multilingual embeddings (legacy-resolver gère le conflit avec transformers 5.x)
|
| 19 |
-
chromadb==0.4.22 # Vector database
|
| 20 |
-
numpy<2.0 # ChromaDB 0.4.22 incompatible avec NumPy 2.0 (np.float_ removed)
|
| 21 |
-
tqdm==4.66.1 # Progress bars
|
| 22 |
-
|
| 23 |
# Utilities
|
| 24 |
python-dotenv==1.0.1
|
| 25 |
loguru==0.7.2
|
|
|
|
| 13 |
accelerate==1.2.0
|
| 14 |
# bitsandbytes retiré: ne fonctionne pas sur CPU (nécessite GPU CUDA)
|
| 15 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
# Utilities
|
| 17 |
python-dotenv==1.0.1
|
| 18 |
loguru==0.7.2
|
scripts/index_documents.py
DELETED
|
@@ -1,355 +0,0 @@
|
|
| 1 |
-
"""
|
| 2 |
-
Script d'indexation des PDFs FEI
|
| 3 |
-
|
| 4 |
-
Ce script:
|
| 5 |
-
1. Lit les PDFs des règlements FEI depuis backend/data/raw_pdfs/
|
| 6 |
-
2. Extrait et chunke le texte (500 tokens, overlap 50)
|
| 7 |
-
3. Génère les embeddings avec multilingual-e5-base
|
| 8 |
-
4. Stocke dans ChromaDB pour recherche RAG
|
| 9 |
-
|
| 10 |
-
Usage:
|
| 11 |
-
# Indexer une discipline spécifique
|
| 12 |
-
python index_documents.py --discipline CSO
|
| 13 |
-
|
| 14 |
-
# Indexer toutes les disciplines
|
| 15 |
-
python index_documents.py --all
|
| 16 |
-
|
| 17 |
-
# Réinitialiser la base avant indexation
|
| 18 |
-
python index_documents.py --all --reset
|
| 19 |
-
"""
|
| 20 |
-
|
| 21 |
-
import argparse
|
| 22 |
-
import os
|
| 23 |
-
import sys
|
| 24 |
-
from pathlib import Path
|
| 25 |
-
from typing import List, Dict, Any
|
| 26 |
-
import PyPDF2
|
| 27 |
-
from tqdm import tqdm
|
| 28 |
-
|
| 29 |
-
# Fixer les variables d'environnement OpenMP (problème HuggingFace Spaces GPU T4)
|
| 30 |
-
# Kubernetes injecte OMP_NUM_THREADS='3500m' (millicores) qui crash les libs Python
|
| 31 |
-
# Solution: forcer chaîne vide pour désactiver OpenMP complètement
|
| 32 |
-
os.environ['OMP_NUM_THREADS'] = '' # DOIT être avant imports torch/transformers
|
| 33 |
-
os.environ['TOKENIZERS_PARALLELISM'] = 'false'
|
| 34 |
-
|
| 35 |
-
# Ajouter le dossier parent au path pour importer les modules RAG
|
| 36 |
-
sys.path.insert(0, str(Path(__file__).parent.parent))
|
| 37 |
-
|
| 38 |
-
from rag.embedder import Embedder
|
| 39 |
-
from rag.vector_store import VectorStore
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
def extract_text_from_pdf(pdf_path: str) -> List[Dict[str, Any]]:
|
| 43 |
-
"""
|
| 44 |
-
Extrait le texte d'un PDF page par page
|
| 45 |
-
|
| 46 |
-
Args:
|
| 47 |
-
pdf_path: Chemin vers le fichier PDF
|
| 48 |
-
|
| 49 |
-
Returns:
|
| 50 |
-
Liste de dicts avec {text, page_number}
|
| 51 |
-
"""
|
| 52 |
-
pages_text = []
|
| 53 |
-
|
| 54 |
-
print(f"📄 Extraction: {os.path.basename(pdf_path)}")
|
| 55 |
-
|
| 56 |
-
try:
|
| 57 |
-
with open(pdf_path, 'rb') as file:
|
| 58 |
-
pdf_reader = PyPDF2.PdfReader(file)
|
| 59 |
-
total_pages = len(pdf_reader.pages)
|
| 60 |
-
|
| 61 |
-
print(f" 📖 {total_pages} pages détectées")
|
| 62 |
-
|
| 63 |
-
for page_num in range(total_pages):
|
| 64 |
-
page = pdf_reader.pages[page_num]
|
| 65 |
-
text = page.extract_text()
|
| 66 |
-
|
| 67 |
-
# Ne garder que les pages avec du texte
|
| 68 |
-
if text.strip():
|
| 69 |
-
pages_text.append({
|
| 70 |
-
"text": text.strip(),
|
| 71 |
-
"page_number": page_num + 1 # Page 1-indexed
|
| 72 |
-
})
|
| 73 |
-
|
| 74 |
-
print(f" ✅ {len(pages_text)} pages avec texte extraites")
|
| 75 |
-
|
| 76 |
-
except Exception as e:
|
| 77 |
-
print(f" ❌ Erreur lors de l'extraction: {e}")
|
| 78 |
-
return []
|
| 79 |
-
|
| 80 |
-
return pages_text
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
def chunk_text(text: str, chunk_size: int = 500, overlap: int = 50) -> List[str]:
|
| 84 |
-
"""
|
| 85 |
-
Découpe un texte en chunks avec overlap
|
| 86 |
-
|
| 87 |
-
Args:
|
| 88 |
-
text: Texte à découper
|
| 89 |
-
chunk_size: Taille du chunk en caractères (~100 tokens)
|
| 90 |
-
overlap: Overlap entre chunks en caractères
|
| 91 |
-
|
| 92 |
-
Returns:
|
| 93 |
-
Liste de chunks de texte
|
| 94 |
-
"""
|
| 95 |
-
# Simple chunking par caractères
|
| 96 |
-
# TODO: Améliorer avec RecursiveCharacterTextSplitter de langchain
|
| 97 |
-
|
| 98 |
-
if len(text) <= chunk_size:
|
| 99 |
-
return [text]
|
| 100 |
-
|
| 101 |
-
chunks = []
|
| 102 |
-
start = 0
|
| 103 |
-
|
| 104 |
-
while start < len(text):
|
| 105 |
-
end = start + chunk_size
|
| 106 |
-
chunk = text[start:end]
|
| 107 |
-
|
| 108 |
-
# Essayer de couper à la fin d'une phrase
|
| 109 |
-
if end < len(text):
|
| 110 |
-
last_period = chunk.rfind('.')
|
| 111 |
-
last_newline = chunk.rfind('\n')
|
| 112 |
-
cut_point = max(last_period, last_newline)
|
| 113 |
-
|
| 114 |
-
if cut_point > chunk_size * 0.5: # Si on trouve un bon point de coupe
|
| 115 |
-
chunk = chunk[:cut_point + 1]
|
| 116 |
-
end = start + cut_point + 1
|
| 117 |
-
|
| 118 |
-
chunks.append(chunk.strip())
|
| 119 |
-
start = end - overlap # Overlap
|
| 120 |
-
|
| 121 |
-
return chunks
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
def process_pdf(
|
| 125 |
-
pdf_path: str,
|
| 126 |
-
discipline: str,
|
| 127 |
-
chunk_size: int = 500,
|
| 128 |
-
overlap: int = 50
|
| 129 |
-
) -> List[Dict[str, Any]]:
|
| 130 |
-
"""
|
| 131 |
-
Traite un PDF: extraction + chunking + métadonnées
|
| 132 |
-
|
| 133 |
-
Args:
|
| 134 |
-
pdf_path: Chemin vers le PDF
|
| 135 |
-
discipline: CSO, Dressage ou CCE
|
| 136 |
-
chunk_size: Taille des chunks
|
| 137 |
-
overlap: Overlap entre chunks
|
| 138 |
-
|
| 139 |
-
Returns:
|
| 140 |
-
Liste de dicts avec {text, metadata}
|
| 141 |
-
"""
|
| 142 |
-
# Extraire le texte page par page
|
| 143 |
-
pages = extract_text_from_pdf(pdf_path)
|
| 144 |
-
|
| 145 |
-
if not pages:
|
| 146 |
-
return []
|
| 147 |
-
|
| 148 |
-
# Chunker chaque page
|
| 149 |
-
all_chunks = []
|
| 150 |
-
source_file = os.path.basename(pdf_path)
|
| 151 |
-
|
| 152 |
-
print(f" ✂️ Chunking (chunk_size={chunk_size}, overlap={overlap})...")
|
| 153 |
-
|
| 154 |
-
for page in pages:
|
| 155 |
-
page_chunks = chunk_text(page["text"], chunk_size, overlap)
|
| 156 |
-
|
| 157 |
-
for chunk_idx, chunk in enumerate(page_chunks):
|
| 158 |
-
all_chunks.append({
|
| 159 |
-
"text": chunk,
|
| 160 |
-
"metadata": {
|
| 161 |
-
"discipline": discipline,
|
| 162 |
-
"source_file": source_file,
|
| 163 |
-
"page_number": page["page_number"],
|
| 164 |
-
"chunk_index": chunk_idx
|
| 165 |
-
}
|
| 166 |
-
})
|
| 167 |
-
|
| 168 |
-
print(f" ✅ {len(all_chunks)} chunks créés")
|
| 169 |
-
|
| 170 |
-
return all_chunks
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
def index_discipline(
|
| 174 |
-
discipline: str,
|
| 175 |
-
raw_pdfs_dir: str,
|
| 176 |
-
embedder: Embedder,
|
| 177 |
-
vector_store: VectorStore,
|
| 178 |
-
chunk_size: int = 500,
|
| 179 |
-
overlap: int = 50
|
| 180 |
-
) -> int:
|
| 181 |
-
"""
|
| 182 |
-
Indexe tous les PDFs d'une discipline
|
| 183 |
-
|
| 184 |
-
Args:
|
| 185 |
-
discipline: CSO, Dressage ou CCE
|
| 186 |
-
raw_pdfs_dir: Dossier contenant les PDFs
|
| 187 |
-
embedder: Instance d'Embedder
|
| 188 |
-
vector_store: Instance de VectorStore
|
| 189 |
-
chunk_size: Taille des chunks
|
| 190 |
-
overlap: Overlap entre chunks
|
| 191 |
-
|
| 192 |
-
Returns:
|
| 193 |
-
Nombre total de chunks indexés
|
| 194 |
-
"""
|
| 195 |
-
print(f"\n{'='*80}")
|
| 196 |
-
print(f"🏇 Indexation: {discipline}")
|
| 197 |
-
print(f"{'='*80}\n")
|
| 198 |
-
|
| 199 |
-
discipline_dir = os.path.join(raw_pdfs_dir, discipline)
|
| 200 |
-
|
| 201 |
-
# Vérifier que le dossier existe
|
| 202 |
-
if not os.path.exists(discipline_dir):
|
| 203 |
-
print(f"⚠️ Dossier introuvable: {discipline_dir}")
|
| 204 |
-
return 0
|
| 205 |
-
|
| 206 |
-
# Lister les PDFs
|
| 207 |
-
pdf_files = [f for f in os.listdir(discipline_dir) if f.endswith('.pdf')]
|
| 208 |
-
|
| 209 |
-
if not pdf_files:
|
| 210 |
-
print(f"⚠️ Aucun PDF trouvé dans {discipline_dir}")
|
| 211 |
-
return 0
|
| 212 |
-
|
| 213 |
-
print(f"📚 {len(pdf_files)} PDF(s) trouvé(s):\n")
|
| 214 |
-
for pdf in pdf_files:
|
| 215 |
-
print(f" - {pdf}")
|
| 216 |
-
print()
|
| 217 |
-
|
| 218 |
-
total_chunks = 0
|
| 219 |
-
|
| 220 |
-
# Traiter chaque PDF
|
| 221 |
-
for pdf_file in pdf_files:
|
| 222 |
-
pdf_path = os.path.join(discipline_dir, pdf_file)
|
| 223 |
-
|
| 224 |
-
# Supprimer l'ancien index si le fichier existe déjà
|
| 225 |
-
vector_store.delete_by_source(pdf_file)
|
| 226 |
-
|
| 227 |
-
# Extraire et chunker le PDF
|
| 228 |
-
chunks_data = process_pdf(pdf_path, discipline, chunk_size, overlap)
|
| 229 |
-
|
| 230 |
-
if not chunks_data:
|
| 231 |
-
continue
|
| 232 |
-
|
| 233 |
-
# Extraire textes et métadonnées
|
| 234 |
-
texts = [chunk["text"] for chunk in chunks_data]
|
| 235 |
-
metadatas = [chunk["metadata"] for chunk in chunks_data]
|
| 236 |
-
|
| 237 |
-
# Générer les embeddings
|
| 238 |
-
print(f" 🧮 Génération des embeddings...")
|
| 239 |
-
embeddings = embedder.embed_documents(texts)
|
| 240 |
-
|
| 241 |
-
# Ajouter à ChromaDB
|
| 242 |
-
print(f" 💾 Ajout à ChromaDB...")
|
| 243 |
-
vector_store.add_documents(
|
| 244 |
-
documents=texts,
|
| 245 |
-
embeddings=embeddings,
|
| 246 |
-
metadatas=metadatas
|
| 247 |
-
)
|
| 248 |
-
|
| 249 |
-
total_chunks += len(chunks_data)
|
| 250 |
-
print(f" ✅ {len(chunks_data)} chunks indexés\n")
|
| 251 |
-
|
| 252 |
-
return total_chunks
|
| 253 |
-
|
| 254 |
-
|
| 255 |
-
def main():
|
| 256 |
-
parser = argparse.ArgumentParser(
|
| 257 |
-
description="Indexer les PDFs des règlements FEI pour le système RAG"
|
| 258 |
-
)
|
| 259 |
-
|
| 260 |
-
parser.add_argument(
|
| 261 |
-
"--discipline",
|
| 262 |
-
type=str,
|
| 263 |
-
choices=["CSO", "Dressage", "CCE"],
|
| 264 |
-
help="Discipline à indexer (CSO, Dressage ou CCE)"
|
| 265 |
-
)
|
| 266 |
-
|
| 267 |
-
parser.add_argument(
|
| 268 |
-
"--all",
|
| 269 |
-
action="store_true",
|
| 270 |
-
help="Indexer toutes les disciplines"
|
| 271 |
-
)
|
| 272 |
-
|
| 273 |
-
parser.add_argument(
|
| 274 |
-
"--reset",
|
| 275 |
-
action="store_true",
|
| 276 |
-
help="Réinitialiser la base ChromaDB avant indexation"
|
| 277 |
-
)
|
| 278 |
-
|
| 279 |
-
parser.add_argument(
|
| 280 |
-
"--chunk-size",
|
| 281 |
-
type=int,
|
| 282 |
-
default=500,
|
| 283 |
-
help="Taille des chunks en caractères (default: 500)"
|
| 284 |
-
)
|
| 285 |
-
|
| 286 |
-
parser.add_argument(
|
| 287 |
-
"--overlap",
|
| 288 |
-
type=int,
|
| 289 |
-
default=50,
|
| 290 |
-
help="Overlap entre chunks en caractères (default: 50)"
|
| 291 |
-
)
|
| 292 |
-
|
| 293 |
-
args = parser.parse_args()
|
| 294 |
-
|
| 295 |
-
# Validation des arguments
|
| 296 |
-
if not args.discipline and not args.all:
|
| 297 |
-
parser.error("Vous devez spécifier --discipline ou --all")
|
| 298 |
-
|
| 299 |
-
# Initialiser les composants RAG
|
| 300 |
-
print("🚀 Initialisation du système RAG\n")
|
| 301 |
-
|
| 302 |
-
embedder = Embedder()
|
| 303 |
-
vector_store = VectorStore()
|
| 304 |
-
|
| 305 |
-
# Reset si demandé
|
| 306 |
-
if args.reset:
|
| 307 |
-
print("\n⚠️ Reset de la base ChromaDB...")
|
| 308 |
-
vector_store.reset()
|
| 309 |
-
print()
|
| 310 |
-
|
| 311 |
-
# Dossier des PDFs
|
| 312 |
-
script_dir = Path(__file__).parent
|
| 313 |
-
raw_pdfs_dir = script_dir.parent / "data" / "raw_pdfs"
|
| 314 |
-
|
| 315 |
-
# Indexer les disciplines
|
| 316 |
-
disciplines = []
|
| 317 |
-
if args.all:
|
| 318 |
-
disciplines = ["CSO", "Dressage", "CCE"]
|
| 319 |
-
else:
|
| 320 |
-
disciplines = [args.discipline]
|
| 321 |
-
|
| 322 |
-
total_indexed = 0
|
| 323 |
-
|
| 324 |
-
for discipline in disciplines:
|
| 325 |
-
count = index_discipline(
|
| 326 |
-
discipline=discipline,
|
| 327 |
-
raw_pdfs_dir=str(raw_pdfs_dir),
|
| 328 |
-
embedder=embedder,
|
| 329 |
-
vector_store=vector_store,
|
| 330 |
-
chunk_size=args.chunk_size,
|
| 331 |
-
overlap=args.overlap
|
| 332 |
-
)
|
| 333 |
-
total_indexed += count
|
| 334 |
-
|
| 335 |
-
# Afficher les statistiques finales
|
| 336 |
-
print(f"\n{'='*80}")
|
| 337 |
-
print("📊 STATISTIQUES FINALES")
|
| 338 |
-
print(f"{'='*80}\n")
|
| 339 |
-
|
| 340 |
-
stats = vector_store.get_stats()
|
| 341 |
-
|
| 342 |
-
print(f"✅ Indexation terminée!")
|
| 343 |
-
print(f" 📚 Total: {stats['total_documents']} chunks indexés")
|
| 344 |
-
print(f" 🏷️ Disciplines: {stats['disciplines']}")
|
| 345 |
-
print(f" 📁 Sources: {stats['unique_sources']} fichiers")
|
| 346 |
-
print(f"\n Fichiers:")
|
| 347 |
-
for source in stats['sources']:
|
| 348 |
-
print(f" - {source}")
|
| 349 |
-
|
| 350 |
-
print(f"\n💡 La base vectorielle est prête pour le RAG!")
|
| 351 |
-
print(f" Testez avec: python -m rag.retriever")
|
| 352 |
-
|
| 353 |
-
|
| 354 |
-
if __name__ == "__main__":
|
| 355 |
-
main()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|