Hugging Face's logo

mayoula
/
RAMTUNET_VLM

English
medical-imaging
brain-tumor
segmentation
vlm
glioblastoma
ucsf-pdgm
Model card Files
xet
 Community
RAMTUNET_VLM
File size: 3,511 Bytes
331ead4
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
---
language: en
license: mit
tags:
  - medical-imaging
  - brain-tumor
  - segmentation
  - vlm
  - glioblastoma
  - ucsf-pdgm
datasets:
  - UCSF-PDGM-v5
metrics:
  - bertscore
  - rouge
  - bleu
---

# RCMTUNetV4-VLM — Brain Tumor Segmentation & Report Generation

## Description
Multimodal pipeline for brain tumor segmentation and automated neuro-oncology
report generation. Architecture: **RCMTUNetV4** segmentation + **RAG** (40 WHO CNS 2021 chunks, FAISS) + **LLaVA-Med** report generation.

## 📂 Fichiers dans ce dépôt

| Fichier | Description |
|---|---|
| `rcmt_unet_v4_final.pth` | Poids segmentation RCMTUNetV4 |
| `pipeline.py` | **Architecture complète** (toutes classes) |
| `rag_who_chunks.json` | **40 chunks RAG** WHO CNS 2021 |
| `rag_faiss.index` | **Index FAISS** pré-calculé |
| `rag_embeddings.npy` | Embeddings numpy (backup) |
| `prompts.json` | **Tous les prompts** v3.2 + profils ACP |
| `config.json` | Configuration et métriques |
| `evaluation_metrics.json` | Résultats détaillés |

## Performance (n=50, seed=42, UCSF-PDGM)

| Métrique | Score | vs Baseline |
|---|---|---|
| BERTScore-F | 0.814 | > MediVLM (0.616) ✅ |
| TBFact | 0.922 | > BTReport (0.353) ✅ |
| RadGraph-F1 | 0.871 | > AutoRG (0.380) ✅ |
| Anti-hallucination | 1.000 | UNIQUE ✅ |
| Cross-validation | 1.000 | UNIQUE ✅ |
| Global score | 0.852 | Classe A ✅ |

## 🚀 Usage — Chargement complet en 10 lignes

```python
import torch, json, faiss, numpy as np
from huggingface_hub import hf_hub_download, snapshot_download
from sentence_transformers import SentenceTransformer

# 1. Télécharger tous les fichiers
local_dir = snapshot_download(repo_id="mayoula/RAMTUNET_VLM")

# 2. Charger l'architecture
import sys; sys.path.insert(0, local_dir)
from pipeline import RCMTUNetV4

# 3. Charger les poids segmentation
seg_model = RCMTUNetV4(in_channels=4, out_channels=4, features=(24,48,96,192))
seg_model.load_state_dict(torch.load(f"{local_dir}/rcmt_unet_v4_final.pth", map_location="cpu"))
seg_model.eval()

# 4. Charger le RAG
with open(f"{local_dir}/rag_who_chunks.json") as f:
    rag_data = json.load(f)
WHO_CHUNKS = rag_data["chunks"]
faiss_idx  = faiss.read_index(f"{local_dir}/rag_faiss.index")
embedder   = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

# 5. Charger les prompts
with open(f"{local_dir}/prompts.json") as f:
    prompts = json.load(f)

# 6. Charger le VLM (LLaVA-Med — non fine-tuné, rechargé depuis HF)
from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration, BitsAndBytesConfig
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16,
                          bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True)
vlm_processor = LlavaNextProcessor.from_pretrained("microsoft/llava-med-v1.5-mistral-7b")
vlm_model     = LlavaNextForConditionalGeneration.from_pretrained(
    "microsoft/llava-med-v1.5-mistral-7b", quantization_config=bnb, device_map="auto")

# 7. Fonction RAG retrieve
def rag_retrieve(query, top_k=4):
    emb  = embedder.encode([query], normalize_embeddings=True)
    D, I = faiss_idx.search(emb.astype(np.float32), top_k)
    refs = [f"[REF-{r+1}] {WHO_CHUNKS[i]}" for r, (d, i) in enumerate(zip(D[0], I[0])) if d > 0.10]
    return "\n".join(refs) if refs else "Standard glioma protocol (WHO CNS 2021)."
```

## Dataset
Trained and evaluated on [UCSF-PDGM](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=119705830) (n=50 patients, seed=42).