Cost-per-token benchmarking tool to complement quantized model accuracy tracking
Hi Intel + community đź‘‹
Quantized models are all about the accuracy-vs-efficiency tradeoff. I built an open source framework that makes that tradeoff measurable:
LLM Evaluation Framework tracks:
- 💰 Cost per 1K tokens — exact pricing across 15+ models including quantized variants
- 🎯 Accuracy — so you can plot accuracy vs cost directly
- ⚡ Latency p95 — real-world latency, not just throughput
- 🔍 Hallucination Rate — quantized models sometimes trade accuracy for hallucination rate differently than full-precision
The goal: give teams a single report that answers "is this quantized model good enough for my use case at my budget?"
Live demo: https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework
Open source, 71 tests, full CI/CD. Feedback welcome!
Looks like you've built a solid framework for evaluating quantized models! Tracking cost per token, accuracy, latency, and hallucination rates is crucial for making informed decisions. The live demo sounds interesting—would love to see how it handles different model variants. What specific challenges did you encounter while building this?