Spaces:
Running
Running
| title: README | |
| emoji: 📊 | |
| colorFrom: blue | |
| colorTo: indigo | |
| sdk: static | |
| pinned: false | |
| # Quantiles | |
| **Open-source, local-first evaluation infrastructure for applied AI systems, designed for developer and coding-agent workflows.** | |
| Quantiles gives developers a CLI and Python SDK for creating, running, analyzing, and comparing AI evaluations from their existing development environment. It records metrics, sample-level outputs, execution traces, and run history locally, making it easier to measure system behavior, detect regressions, validate changes, and ship higher-quality, more reliable AI systems. | |
| With Quantiles, developers can: | |
| - Run built-in benchmarks or create custom evaluations from configuration or Python. | |
| - Debug individual samples with step-level inputs, outputs, metrics, and errors. | |
| - Resume interrupted runs and reuse completed steps. | |
| - Compare models, prompts, datasets, and configurations from the CLI. | |
| - Run evaluation workflows with coding agents using the Quantiles agent skill. | |
| ## Quickstart | |
| Install the CLI and run SimpleQA Verified: | |
| ```bash | |
| curl -fsSL https://cli.quantiles.io/install.sh | bash | |
| qt run simpleqa-verified | |
| qt show 1 --json | |
| ``` | |
| This first run uses the built-in demo model, so it requires no model API key and incurs no inference cost. Use it to verify the workflow, then configure a provider-backed model for a real evaluation. | |
| ## Documentation & Resources | |
| - [Documentation](https://quantiles.io/documentation) — Everything needed to use Quantiles. | |
| - [Quickstart](https://quantiles.io/documentation/quickstart) — Install Quantiles and run your first evaluation. | |
| - [GitHub](https://github.com/quantiles-evals/quantiles) — Explore the source code and contribute. | |
| - [Agent skill](https://github.com/quantiles-evals/skill) — Run and analyze AI evaluations with a coding agent. | |
| ### Articles | |
| - **Oct 1, 2026** | [Does your SKILL.md help coding agents?](https://huggingface.co/blog/phranzia/does-your-skill-md-help-coding-agents) | |
| - **Sept 24, 2026** | [Evaluating AI Agent Tool Use](https://huggingface.co/blog/phranzia/evaluating-ai-agent-tool-use) | |
| - **Sept 17, 2026** | [Measuring Agent Recovery](https://huggingface.co/blog/phranzia/measuring-agent-recovery) | |
| - **Sept 14, 2026** | [Measure, Understand, and Improve Agent Latency](https://huggingface.co/blog/phranzia/measure-understand-and-improve-agent-latency) | |
| - **Sept 4, 2026** | [Using Execution Traces to Evaluate AI Agent Behavior](https://huggingface.co/blog/phranzia/using-execution-traces-to-evaluate-agent-behavior) | |
| - **Aug 27, 2026** | [How to Evaluate AI Agents](https://huggingface.co/blog/phranzia/how-to-evaluate-ai-agents) | |
| - **Aug 4, 2026** | [Build an AI Evaluation from a Hugging Face Dataset Without Writing Python](https://huggingface.co/blog/phranzia/build-an-ai-evaluation-from-a-hugging-face-dataset) | |
| Quantiles is available under the [Apache License 2.0](https://github.com/quantiles-evals/quantiles/blob/main/LICENSE). |