--- license: mit title: AI Stupid Level sdk: static colorFrom: green colorTo: green short_description: Continuous, independent evaluation of production LLMs --- # AI Stupid Level (ASL) [![Website](https://img.shields.io/badge/Website-aistupidlevel.info-blue)](https://aistupidlevel.info) [![Methodology](https://img.shields.io/badge/Methodology-public-green)](https://aistupidlevel.info/methodology) [![Data API](https://img.shields.io/badge/Data%20API-v1-orange)](https://aistupidlevel.info/api-docs) [![LinkedIn](https://img.shields.io/badge/LinkedIn-ASL-0A66C2)](https://www.linkedin.com/company/asl-aistupidlevel-info) [![X](https://img.shields.io/badge/X-@aistupidlevel-black)](https://x.com/aistupidlevel) **Continuous, independent evaluation of the language models people actually ship on.** We re-measure a fixed fleet of production models several times a day, on the same tasks, through the vendors' own APIs — and publish when their behaviour changes. The question we exist to answer is not "which model is best on a leaderboard" but **"is the model you deployed last month still the model you are calling today?"** **Website:** [aistupidlevel.info](https://aistupidlevel.info) · **Operated by:** Studio Platforms (Romania) --- ## What is measured **18 models across 6 providers**, re-measured continuously: | Provider | Models | |---|---| | OpenAI | gpt-5.5, gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, gpt-6-astra | | Anthropic | claude-opus-5, claude-opus-4-8, claude-opus-4-6, claude-sonnet-5, claude-sonnet-4-6, claude-fable-5-1 | | Google | gemini-3.8-flash, gemini-3.5-flash-lite | | DeepSeek | deepseek-v4-pro, deepseek-v4-flash | | Moonshot AI | kimi-k3, kimi-k2.7-code | | Zhipu AI | glm-5.2 | The fleet changes as vendors ship and retire models. Retired models keep their history — nothing is deleted, because a comparable past is the entire point. --- ## The measurement suites Five suites run on independent schedules. Each has its own tasks, its own axes and its own drift baseline. **They are never blended into a single number without saying so.** ### Coding — every 4 hours 7 Python tasks, **7 trials each**. Six of the seven are repository-repair tasks: the model is given a real codebase and a bug report, and graded on whether it fixed the actual defect or silenced the symptom it was shown. Scored on 9 axes: correctness, stability, edge cases, debugging, code quality, efficiency, format, safety, complexity. ### Tool calling — daily 9 sandboxed agent tasks (3 easy, 3 medium, 3 hard) covering file operations, multi-step workflows and tool chaining. Scored on 7 axes: task completion, tool selection, parameter accuracy, efficiency, error handling, context awareness, safety compliance. ### Deep reasoning — daily 4 multi-turn tasks that run to completion rather than a single exchange: IDE assistance, specification following, document memory, project refactoring. Scored on 6 axes: correctness, memory retention, plan coherence, context window handling, debugging, stability. ### Canary — hourly 2 tasks, 2 trials, deliberately small. A fast probe whose only job is to notice change between the heavier runs. ### Calibration and known-unknowns — every 6 hours Does the model know what it does not know? Each model is asked questions from a stratified corpus; some have a verifiable answer, some have none — a library function that does not exist, a real company with an attribute it cannot have, a fact it has no way to observe. Declining the second group is the correct behaviour and is scored as such. Reported per model: **fabrication rate**, accuracy, abstention rate, **expected calibration error**, and over/under-confidence. These figures are published separately and never enter the composite score. Being well calibrated does not make a model better at coding, and blending the two would make both numbers mean less. --- ## Drift detection A leaderboard tells you where a model stands today. Drift detection tells you when it moved. - **Page-Hinkley change detection**, run **per suite** on that suite's own daily series — coding, tool use and reasoning are never blended, because each has a different scale and noise floor. - A suite needs **10 measured days** on its current benchmark configuration before it may fire. The statistic is visible from day one, but a reading during warm-up is an early indication, not a verdict. - **Any scoring or corpus change mints a new configuration id** and resets the baseline. Scores either side of such a change are not comparable and are not compared. - **Task-level regressions** are detected separately, from execution records rather than composite scores. A single task collapsing is close to invisible in a composite — it moves the overall score by less than the alert threshold — so it is reported on its own. --- ## How the numbers are kept honest This is the part we would want to read first about someone else's benchmark. - **A provider that is not answering is skipped, not scored.** A dead API key or an outage must never be published as a model getting worse. - **Refusals are recorded, not imputed.** If a model declines a task, that task drops out and the run is marked partial. A refusal is not a zero. - **A missing suite drops out and the remaining weights are renormalised.** No suite is ever backfilled with a placeholder value. - **Provenance is labelled.** Where a figure is not backed by a live measurement, the interface says so rather than presenting it as one. - **Every task grades only what was asked.** Hidden requirements that were never stated in the prompt are not graded. - **Ranks close together are statistical ties** and are presented as such, with confidence intervals rather than false precision. The full methodology, including the scoring weights and the task design, is published at [aistupidlevel.info/methodology](https://aistupidlevel.info/methodology). --- ## Smart router An OpenAI-compatible endpoint that picks the model for each request from current benchmark results instead of a hard-coded name. ```bash curl -X POST https://aistupidlevel.info/v1/chat/completions \ -H "Authorization: Bearer aism_your_key_here" \ -H "Content-Type: application/json" \ -d '{ "model": "auto-coding", "messages": [{"role": "user", "content": "Write a binary search in Python"}] }' ``` **Eight strategies:** `auto` (best overall), `auto-coding`, `auto-reasoning`, `auto-creative`, `auto-tooling`, `auto-agentic`, `auto-cheapest`, `auto-fastest`. **You bring your own provider keys.** They are encrypted at rest with AES-256-GCM, and the providers bill you directly — we never resell tokens or mark them up. A provider outage falls through to the next best model rather than failing your request. Works as a drop-in replacement in tools that accept an OpenAI base URL, such as Cline, Continue and Cursor. --- ## Data API Programmatic access to scores, rankings, history, drift signals and incidents, at `https://aistupidlevel.info/api/v1` with `asl_` keys. | Tier | Requests / day | Requests / minute | |---|---|---| | Free | 10 | 1 | | Pro | 10,000 | 60 | | Teams | 100,000 | 300 | | Enterprise | 250,000 | 1,000 | The free tier is an evaluation allowance — enough to build against, not to run on. Documentation: [aistupidlevel.info/api-docs](https://aistupidlevel.info/api-docs). --- ## Who this is for - **Teams running models in production** — know within hours when a model you depend on starts behaving differently, instead of finding out from your users. - **Researchers** — a public methodology, published per-task results, and history that survives corpus changes because configuration boundaries are recorded rather than smoothed over. - **Developers choosing a model** — per-suite comparisons, cost per million tokens, and measured latency, rather than a single aggregate that hides the trade-off you care about. - **Anyone tired of vibes** — every published figure traces back to a run we can point at. Current scores, every category ranking, seven days of history, provider status and the full methodology are **free and require no account**. Paid plans add longer comparable history, the drift curves, calibration data, exports and the routing volume. --- ## Technical notes - **API:** TypeScript, Fastify, SQLite with Drizzle ORM, Redis caching - **Web:** Next.js 14, React 18, Recharts, NextAuth - **Infrastructure:** a single VPS under systemd, behind nginx with SSL - **Frontend source:** [github.com/StudioPlatforms/aistupidmeter-web](https://github.com/StudioPlatforms/aistupidmeter-web) The web frontend is public. The benchmark engine is not open source — the task corpus is deliberately unpublished, because a benchmark whose answers a model has been trained on measures nothing. The methodology, the scoring weights and the per-task outcomes are public instead, which is what makes a result checkable without making it gameable. --- ## Status and transparency A live [status page](https://aistupidlevel.info/status) reports provider availability and separates three things that look identical from outside and mean different things: the provider being down, our own account or key failing, and our probe failing. When a suite could not run, the site says which and why rather than quietly publishing a lower number. --- ## Contact - **Contact form:** [aistupidlevel.info/contact](https://aistupidlevel.info/contact) - **LinkedIn:** [ASL — aistupidlevel.info](https://www.linkedin.com/company/asl-aistupidlevel-info) - **X / Twitter:** [@aistupidlevel](https://x.com/aistupidlevel) - **Website:** [aistupidlevel.info](https://aistupidlevel.info) · also reachable at [asl.info](https://asl.info) --- **AI Stupid Level is an independent evaluation platform built by Ionut Adrian Visan. It is not affiliated with, endorsed by, or funded by any of the model providers it measures — which is the only reason its numbers are worth anything.**