Spaces:
Sleeping
RAGBench: Help Me Benchmark RAG Systems Better
RAGBench โ Help me benchmark RAG systems better
Hi folks! I've been building RAGBench, an evaluation framework for comparing RAG pipelines, and I'd love to get feedback from the community.
Why I built it
There's a lot of discussion around RAG chunking strategies, but a simple question often gets overlooked:
Which strategy actually performs better for a given workload?
Instead of choosing a strategy because it "feels right," I wanted to make it easier to compare them using the same documents, queries, and evaluation metrics.
What I've done
- Built a benchmark comparing semantic, document-level, and parent-child chunking
- Evaluated them using retrieval metrics such as Hit@1, Hit@3, and MRR
- Added LLM-based evaluation for faithfulness and answer relevancy
- Created an interactive Hugging Face Space so the evaluation can be explored interactively
- Made the datasets available so the experiments can be reproduced
What I'm looking for
I'd really like feedback from people working with RAG:
What's been the hardest part of evaluating your RAG system?
I'm especially interested in:
- Domain-specific or multilingual datasets
- Evaluation metrics you actually find useful
- Different chunking strategies I should test
- Feedback on the benchmark methodology
- Real-world RAG failure cases
Try RAGBench
๐ Interactive benchmark:
https://huggingface.co/spaces/Gul55555/ragbench
I'd appreciate any feedback, criticism, or suggestions for what I should test next.
What's your biggest RAG pain point right now?