Spaces:
Running
Running
| license: mit | |
| title: AI Stupid Level | |
| sdk: static | |
| colorFrom: green | |
| colorTo: green | |
| short_description: Continuous, independent evaluation of production LLMs | |
| # AI Stupid Level (ASL) | |
| [](https://aistupidlevel.info) | |
| [](https://aistupidlevel.info/methodology) | |
| [](https://aistupidlevel.info/api-docs) | |
| [](https://www.linkedin.com/company/asl-aistupidlevel-info) | |
| [](https://x.com/aistupidlevel) | |
| **Continuous, independent evaluation of the language models people actually ship on.** | |
| We re-measure a fixed fleet of production models several times a day, on the same tasks, | |
| through the vendors' own APIs — and publish when their behaviour changes. The question we | |
| exist to answer is not "which model is best on a leaderboard" but **"is the model you | |
| deployed last month still the model you are calling today?"** | |
| **Website:** [aistupidlevel.info](https://aistupidlevel.info) · **Operated by:** Studio Platforms (Romania) | |
| --- | |
| ## What is measured | |
| **18 models across 6 providers**, re-measured continuously: | |
| | Provider | Models | | |
| |---|---| | |
| | OpenAI | gpt-5.5, gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, gpt-6-astra | | |
| | Anthropic | claude-opus-5, claude-opus-4-8, claude-opus-4-6, claude-sonnet-5, claude-sonnet-4-6, claude-fable-5-1 | | |
| | Google | gemini-3.8-flash, gemini-3.5-flash-lite | | |
| | DeepSeek | deepseek-v4-pro, deepseek-v4-flash | | |
| | Moonshot AI | kimi-k3, kimi-k2.7-code | | |
| | Zhipu AI | glm-5.2 | | |
| The fleet changes as vendors ship and retire models. Retired models keep their history — | |
| nothing is deleted, because a comparable past is the entire point. | |
| --- | |
| ## The measurement suites | |
| Five suites run on independent schedules. Each has its own tasks, its own axes and its own | |
| drift baseline. **They are never blended into a single number without saying so.** | |
| ### Coding — every 4 hours | |
| 7 Python tasks, **7 trials each**. Six of the seven are repository-repair tasks: the model | |
| is given a real codebase and a bug report, and graded on whether it fixed the actual defect | |
| or silenced the symptom it was shown. Scored on 9 axes: correctness, stability, edge cases, | |
| debugging, code quality, efficiency, format, safety, complexity. | |
| ### Tool calling — daily | |
| 9 sandboxed agent tasks (3 easy, 3 medium, 3 hard) covering file operations, multi-step | |
| workflows and tool chaining. Scored on 7 axes: task completion, tool selection, parameter | |
| accuracy, efficiency, error handling, context awareness, safety compliance. | |
| ### Deep reasoning — daily | |
| 4 multi-turn tasks that run to completion rather than a single exchange: IDE assistance, | |
| specification following, document memory, project refactoring. Scored on 6 axes: | |
| correctness, memory retention, plan coherence, context window handling, debugging, | |
| stability. | |
| ### Canary — hourly | |
| 2 tasks, 2 trials, deliberately small. A fast probe whose only job is to notice change | |
| between the heavier runs. | |
| ### Calibration and known-unknowns — every 6 hours | |
| Does the model know what it does not know? Each model is asked questions from a stratified | |
| corpus; some have a verifiable answer, some have none — a library function that does not | |
| exist, a real company with an attribute it cannot have, a fact it has no way to observe. | |
| Declining the second group is the correct behaviour and is scored as such. Reported per | |
| model: **fabrication rate**, accuracy, abstention rate, **expected calibration error**, and | |
| over/under-confidence. | |
| These figures are published separately and never enter the composite score. Being well | |
| calibrated does not make a model better at coding, and blending the two would make both | |
| numbers mean less. | |
| --- | |
| ## Drift detection | |
| A leaderboard tells you where a model stands today. Drift detection tells you when it moved. | |
| - **Page-Hinkley change detection**, run **per suite** on that suite's own daily series — | |
| coding, tool use and reasoning are never blended, because each has a different scale and | |
| noise floor. | |
| - A suite needs **10 measured days** on its current benchmark configuration before it may | |
| fire. The statistic is visible from day one, but a reading during warm-up is an early | |
| indication, not a verdict. | |
| - **Any scoring or corpus change mints a new configuration id** and resets the baseline. | |
| Scores either side of such a change are not comparable and are not compared. | |
| - **Task-level regressions** are detected separately, from execution records rather than | |
| composite scores. A single task collapsing is close to invisible in a composite — it | |
| moves the overall score by less than the alert threshold — so it is reported on its own. | |
| --- | |
| ## How the numbers are kept honest | |
| This is the part we would want to read first about someone else's benchmark. | |
| - **A provider that is not answering is skipped, not scored.** A dead API key or an outage | |
| must never be published as a model getting worse. | |
| - **Refusals are recorded, not imputed.** If a model declines a task, that task drops out | |
| and the run is marked partial. A refusal is not a zero. | |
| - **A missing suite drops out and the remaining weights are renormalised.** No suite is | |
| ever backfilled with a placeholder value. | |
| - **Provenance is labelled.** Where a figure is not backed by a live measurement, the | |
| interface says so rather than presenting it as one. | |
| - **Every task grades only what was asked.** Hidden requirements that were never stated in | |
| the prompt are not graded. | |
| - **Ranks close together are statistical ties** and are presented as such, with confidence | |
| intervals rather than false precision. | |
| The full methodology, including the scoring weights and the task design, is published at | |
| [aistupidlevel.info/methodology](https://aistupidlevel.info/methodology). | |
| --- | |
| ## Smart router | |
| An OpenAI-compatible endpoint that picks the model for each request from current benchmark | |
| results instead of a hard-coded name. | |
| ```bash | |
| curl -X POST https://aistupidlevel.info/v1/chat/completions \ | |
| -H "Authorization: Bearer aism_your_key_here" \ | |
| -H "Content-Type: application/json" \ | |
| -d '{ | |
| "model": "auto-coding", | |
| "messages": [{"role": "user", "content": "Write a binary search in Python"}] | |
| }' | |
| ``` | |
| **Eight strategies:** `auto` (best overall), `auto-coding`, `auto-reasoning`, | |
| `auto-creative`, `auto-tooling`, `auto-agentic`, `auto-cheapest`, `auto-fastest`. | |
| **You bring your own provider keys.** They are encrypted at rest with AES-256-GCM, and the | |
| providers bill you directly — we never resell tokens or mark them up. A provider outage | |
| falls through to the next best model rather than failing your request. Works as a drop-in | |
| replacement in tools that accept an OpenAI base URL, such as Cline, Continue and Cursor. | |
| --- | |
| ## Data API | |
| Programmatic access to scores, rankings, history, drift signals and incidents, at | |
| `https://aistupidlevel.info/api/v1` with `asl_` keys. | |
| | Tier | Requests / day | Requests / minute | | |
| |---|---|---| | |
| | Free | 10 | 1 | | |
| | Pro | 10,000 | 60 | | |
| | Teams | 100,000 | 300 | | |
| | Enterprise | 250,000 | 1,000 | | |
| The free tier is an evaluation allowance — enough to build against, not to run on. | |
| Documentation: [aistupidlevel.info/api-docs](https://aistupidlevel.info/api-docs). | |
| --- | |
| ## Who this is for | |
| - **Teams running models in production** — know within hours when a model you depend on | |
| starts behaving differently, instead of finding out from your users. | |
| - **Researchers** — a public methodology, published per-task results, and history that | |
| survives corpus changes because configuration boundaries are recorded rather than | |
| smoothed over. | |
| - **Developers choosing a model** — per-suite comparisons, cost per million tokens, and | |
| measured latency, rather than a single aggregate that hides the trade-off you care about. | |
| - **Anyone tired of vibes** — every published figure traces back to a run we can point at. | |
| Current scores, every category ranking, seven days of history, provider status and the full | |
| methodology are **free and require no account**. Paid plans add longer comparable history, | |
| the drift curves, calibration data, exports and the routing volume. | |
| --- | |
| ## Technical notes | |
| - **API:** TypeScript, Fastify, SQLite with Drizzle ORM, Redis caching | |
| - **Web:** Next.js 14, React 18, Recharts, NextAuth | |
| - **Infrastructure:** a single VPS under systemd, behind nginx with SSL | |
| - **Frontend source:** [github.com/StudioPlatforms/aistupidmeter-web](https://github.com/StudioPlatforms/aistupidmeter-web) | |
| The web frontend is public. The benchmark engine is not open source — the task corpus is | |
| deliberately unpublished, because a benchmark whose answers a model has been trained on | |
| measures nothing. The methodology, the scoring weights and the per-task outcomes are public | |
| instead, which is what makes a result checkable without making it gameable. | |
| --- | |
| ## Status and transparency | |
| A live [status page](https://aistupidlevel.info/status) reports provider availability and | |
| separates three things that look identical from outside and mean different things: the | |
| provider being down, our own account or key failing, and our probe failing. When a suite | |
| could not run, the site says which and why rather than quietly publishing a lower number. | |
| --- | |
| ## Contact | |
| - **Contact form:** [aistupidlevel.info/contact](https://aistupidlevel.info/contact) | |
| - **LinkedIn:** [ASL — aistupidlevel.info](https://www.linkedin.com/company/asl-aistupidlevel-info) | |
| - **X / Twitter:** [@aistupidlevel](https://x.com/aistupidlevel) | |
| - **Website:** [aistupidlevel.info](https://aistupidlevel.info) · also reachable at | |
| [asl.info](https://asl.info) | |
| --- | |
| **AI Stupid Level is an independent evaluation platform built by Ionut Adrian Visan. It is not affiliated with, endorsed by, or funded by any of the model providers | |
| it measures — which is the only reason its numbers are worth anything.** | |