Spaces:
Running
Open source multi-metric LLM evaluation: accuracy + cost + hallucination
Hi AllenAI team and RewardBench community 👋
RewardBench is fantastic for evaluating reward models. For teams who also need end-to-end LLM evaluation beyond reward scoring, I built an open source framework.
LLM Evaluation Framework measures 5 dimensions simultaneously:
→ 🎯 Accuracy — 4-strategy cascade: exact, normalized, MC letter, fuzzy (≥0.85)
→ ⚡ Latency — p50/p95/p99 percentiles for SLA planning
→ 💰 Cost per 1K tokens — 15+ models priced from real token counts
→ 🔍 Hallucination Rate — linguistic signal analysis, runs offline
→ 🧠 Reasoning Quality — chain-of-thought depth score 1-10
Pairs naturally with reward model evaluation — use ours for the task metric side, RewardBench for the alignment side.
Live demo (no API key): https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework
Open source. 71 tests. Free forever.