Update RAG eval numbers
Browse files
README.md
CHANGED
|
@@ -28,14 +28,14 @@ We work with biotech, healthcare, defense, legal and fintech teams. Founded in 2
|
|
| 28 |
| [class1-router-synthetic](https://huggingface.co/datasets/SolvrenAI/class1-router-synthetic) | Router training data, no real PII or PHI | 1,536 template rows + 50 hand-written test rows |
|
| 29 |
| [bge-base-en-v1.5-hipaa](https://huggingface.co/SolvrenAI/bge-base-en-v1.5-hipaa) | HIPAA retrieval embeddings | NDCG@10 0.907 vs 0.834 for the base model, 99 held-out queries |
|
| 30 |
| [hipaa-retrieval-pairs](https://huggingface.co/datasets/SolvrenAI/hipaa-retrieval-pairs) | Query-passage pairs over 45 CFR Parts 160 and 164 | 912 pairs over 304 regulation chunks |
|
| 31 |
-
| [hipaa-fda-rag-eval](https://huggingface.co/datasets/SolvrenAI/hipaa-fda-rag-eval) | Synthetic RAG eval QA over HIPAA 45 CFR 164 and FDA device-cybersecurity guidance, plus a 345-chunk corpus |
|
| 32 |
|
| 33 |
Read the numbers with care:
|
| 34 |
|
| 35 |
- These are demos trained on synthetic or LLM-generated data.
|
| 36 |
- The test sets are small (n=50 and n=99), so one error moves the router's accuracy by 2 points.
|
| 37 |
- Router latency was measured on one shared laptop that was running other jobs, so it is approximate.
|
| 38 |
-
- In the RAG eval set, about
|
| 39 |
- On the HIPAA eval the fine-tuned embedding model also scores ahead of bge-large-en-v1.5 (0.864) and snowflake-arctic-embed-m-v1.5 (0.865), with no meaningful change on BEIR SciFact.
|
| 40 |
|
| 41 |
Each model card covers the eval setup and its limitations.
|
|
|
|
| 28 |
| [class1-router-synthetic](https://huggingface.co/datasets/SolvrenAI/class1-router-synthetic) | Router training data, no real PII or PHI | 1,536 template rows + 50 hand-written test rows |
|
| 29 |
| [bge-base-en-v1.5-hipaa](https://huggingface.co/SolvrenAI/bge-base-en-v1.5-hipaa) | HIPAA retrieval embeddings | NDCG@10 0.907 vs 0.834 for the base model, 99 held-out queries |
|
| 30 |
| [hipaa-retrieval-pairs](https://huggingface.co/datasets/SolvrenAI/hipaa-retrieval-pairs) | Query-passage pairs over 45 CFR Parts 160 and 164 | 912 pairs over 304 regulation chunks |
|
| 31 |
+
| [hipaa-fda-rag-eval](https://huggingface.co/datasets/SolvrenAI/hipaa-fda-rag-eval) | Synthetic RAG eval QA over HIPAA 45 CFR 164 and FDA device-cybersecurity guidance, plus a 345-chunk corpus | 162 QA rows; reference answers are noisy (see below) |
|
| 32 |
|
| 33 |
Read the numbers with care:
|
| 34 |
|
| 35 |
- These are demos trained on synthetic or LLM-generated data.
|
| 36 |
- The test sets are small (n=50 and n=99), so one error moves the router's accuracy by 2 points.
|
| 37 |
- Router latency was measured on one shared laptop that was running other jobs, so it is approximate.
|
| 38 |
+
- In the RAG eval set, about 28% of spot-checked reference answers are partial or wrong, and two-thirds of questions are yes/no with answers skewed heavily to "Yes". Use it mainly for retrieval recall; the card has details.
|
| 39 |
- On the HIPAA eval the fine-tuned embedding model also scores ahead of bge-large-en-v1.5 (0.864) and snowflake-arctic-embed-m-v1.5 (0.865), with no meaningful change on BEIR SciFact.
|
| 40 |
|
| 41 |
Each model card covers the eval setup and its limitations.
|