nitya-gupta commited on
Commit
8bc260b
·
verified ·
1 Parent(s): 0cb1685

Update RAG eval numbers

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -28,14 +28,14 @@ We work with biotech, healthcare, defense, legal and fintech teams. Founded in 2
28
  | [class1-router-synthetic](https://huggingface.co/datasets/SolvrenAI/class1-router-synthetic) | Router training data, no real PII or PHI | 1,536 template rows + 50 hand-written test rows |
29
  | [bge-base-en-v1.5-hipaa](https://huggingface.co/SolvrenAI/bge-base-en-v1.5-hipaa) | HIPAA retrieval embeddings | NDCG@10 0.907 vs 0.834 for the base model, 99 held-out queries |
30
  | [hipaa-retrieval-pairs](https://huggingface.co/datasets/SolvrenAI/hipaa-retrieval-pairs) | Query-passage pairs over 45 CFR Parts 160 and 164 | 912 pairs over 304 regulation chunks |
31
- | [hipaa-fda-rag-eval](https://huggingface.co/datasets/SolvrenAI/hipaa-fda-rag-eval) | Synthetic RAG eval QA over HIPAA 45 CFR 164 and FDA device-cybersecurity guidance, plus a 345-chunk corpus | 133 QA rows; reference answers are noisy (see below) |
32
 
33
  Read the numbers with care:
34
 
35
  - These are demos trained on synthetic or LLM-generated data.
36
  - The test sets are small (n=50 and n=99), so one error moves the router's accuracy by 2 points.
37
  - Router latency was measured on one shared laptop that was running other jobs, so it is approximate.
38
- - In the RAG eval set, about 23% of spot-checked reference answers are wrong or partial and 71% of questions are yes/no. Use it mainly for retrieval recall; the card has details.
39
  - On the HIPAA eval the fine-tuned embedding model also scores ahead of bge-large-en-v1.5 (0.864) and snowflake-arctic-embed-m-v1.5 (0.865), with no meaningful change on BEIR SciFact.
40
 
41
  Each model card covers the eval setup and its limitations.
 
28
  | [class1-router-synthetic](https://huggingface.co/datasets/SolvrenAI/class1-router-synthetic) | Router training data, no real PII or PHI | 1,536 template rows + 50 hand-written test rows |
29
  | [bge-base-en-v1.5-hipaa](https://huggingface.co/SolvrenAI/bge-base-en-v1.5-hipaa) | HIPAA retrieval embeddings | NDCG@10 0.907 vs 0.834 for the base model, 99 held-out queries |
30
  | [hipaa-retrieval-pairs](https://huggingface.co/datasets/SolvrenAI/hipaa-retrieval-pairs) | Query-passage pairs over 45 CFR Parts 160 and 164 | 912 pairs over 304 regulation chunks |
31
+ | [hipaa-fda-rag-eval](https://huggingface.co/datasets/SolvrenAI/hipaa-fda-rag-eval) | Synthetic RAG eval QA over HIPAA 45 CFR 164 and FDA device-cybersecurity guidance, plus a 345-chunk corpus | 162 QA rows; reference answers are noisy (see below) |
32
 
33
  Read the numbers with care:
34
 
35
  - These are demos trained on synthetic or LLM-generated data.
36
  - The test sets are small (n=50 and n=99), so one error moves the router's accuracy by 2 points.
37
  - Router latency was measured on one shared laptop that was running other jobs, so it is approximate.
38
+ - In the RAG eval set, about 28% of spot-checked reference answers are partial or wrong, and two-thirds of questions are yes/no with answers skewed heavily to "Yes". Use it mainly for retrieval recall; the card has details.
39
  - On the HIPAA eval the fine-tuned embedding model also scores ahead of bge-large-en-v1.5 (0.864) and snowflake-arctic-embed-m-v1.5 (0.865), with no meaningful change on BEIR SciFact.
40
 
41
  Each model card covers the eval setup and its limitations.