RAGBench: Help Me Benchmark RAG Systems Better

#1
by Gul55555 - opened
Owner

RAGBench โ€” Help me benchmark RAG systems better

Hi folks! I've been building RAGBench, an evaluation framework for comparing RAG pipelines, and I'd love to get feedback from the community.

Why I built it

There's a lot of discussion around RAG chunking strategies, but a simple question often gets overlooked:

Which strategy actually performs better for a given workload?

Instead of choosing a strategy because it "feels right," I wanted to make it easier to compare them using the same documents, queries, and evaluation metrics.

What I've done

  • Built a benchmark comparing semantic, document-level, and parent-child chunking
  • Evaluated them using retrieval metrics such as Hit@1, Hit@3, and MRR
  • Added LLM-based evaluation for faithfulness and answer relevancy
  • Created an interactive Hugging Face Space so the evaluation can be explored interactively
  • Made the datasets available so the experiments can be reproduced

What I'm looking for

I'd really like feedback from people working with RAG:

What's been the hardest part of evaluating your RAG system?

I'm especially interested in:

  • Domain-specific or multilingual datasets
  • Evaluation metrics you actually find useful
  • Different chunking strategies I should test
  • Feedback on the benchmark methodology
  • Real-world RAG failure cases

Try RAGBench

๐Ÿ”— Interactive benchmark:
https://huggingface.co/spaces/Gul55555/ragbench

I'd appreciate any feedback, criticism, or suggestions for what I should test next.

What's your biggest RAG pain point right now?

Gul55555 changed discussion status to closed

Sign up or log in to comment