Quantiles

Open-source, local-first evaluation infrastructure for applied AI systems, designed for developer and coding-agent workflows.

Quantiles gives developers a CLI and Python SDK for creating, running, analyzing, and comparing AI evaluations from their existing development environment. It records metrics, sample-level outputs, execution traces, and run history locally, making it easier to measure system behavior, detect regressions, validate changes, and ship higher-quality, more reliable AI systems.

With Quantiles, developers can:

• Run built-in benchmarks or create custom evaluations from configuration or Python.
• Debug individual samples with step-level inputs, outputs, metrics, and errors.
• Resume interrupted runs and reuse completed steps.
• Compare models, prompts, datasets, and configurations from the CLI.
• Run evaluation workflows with coding agents using the Quantiles agent skill.

Quickstart

Install the CLI and run SimpleQA Verified:

curl -fsSL https://cli.quantiles.io/install.sh | bash
qt run simpleqa-verified
qt show 1 --json

This first run uses the built-in demo model, so it requires no model API key and incurs no inference cost. Use it to verify the workflow, then configure a provider-backed model for a real evaluation.

Quantiles on Hugging Face

Follow this organization for technical articles on benchmarking, evaluating, and improving AI systems.

Documentation · Quickstart · GitHub · Benchmark Hub · Agent skill

Quantiles is available under the Apache License 2.0.