Blame for README.md

---
license: mit
title: AI Stupid Level
sdk: static
colorFrom: green
colorTo: green
short_description: Continuous, independent evaluation of production LLMs
---

# AI Stupid Level (ASL)

[![Website](https://img.shields.io/badge/Website-aistupidlevel.info-blue)](https://aistupidlevel.info)
[![Methodology](https://img.shields.io/badge/Methodology-public-green)](https://aistupidlevel.info/methodology)
[![Data API](https://img.shields.io/badge/Data%20API-v1-orange)](https://aistupidlevel.info/api-docs)
[![LinkedIn](https://img.shields.io/badge/LinkedIn-ASL-0A66C2)](https://www.linkedin.com/company/asl-aistupidlevel-info)
[![X](https://img.shields.io/badge/X-@aistupidlevel-black)](https://x.com/aistupidlevel)

**Continuous, independent evaluation of the language models people actually ship on.**

We re-measure a fixed fleet of production models several times a day, on the same tasks,
through the vendors' own APIs — and publish when their behaviour changes. The question we
exist to answer is not "which model is best on a leaderboard" but **"is the model you
deployed last month still the model you are calling today?"**

**Website:** [aistupidlevel.info](https://aistupidlevel.info) · **Operated by:** Studio Platforms (Romania)

---

## What is measured

**18 models across 6 providers**, re-measured continuously:

| Provider | Models |
|---|---|
| OpenAI | gpt-5.5, gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, gpt-6-astra |
| Anthropic | claude-opus-5, claude-opus-4-8, claude-opus-4-6, claude-sonnet-5, claude-sonnet-4-6, claude-fable-5-1 |
| Google | gemini-3.8-flash, gemini-3.5-flash-lite |
| DeepSeek | deepseek-v4-pro, deepseek-v4-flash |
| Moonshot AI | kimi-k3, kimi-k2.7-code |
| Zhipu AI | glm-5.2 |

The fleet changes as vendors ship and retire models. Retired models keep their history —
nothing is deleted, because a comparable past is the entire point.

---

## The measurement suites

Five suites run on independent schedules. Each has its own tasks, its own axes and its own
drift baseline. **They are never blended into a single number without saying so.**

### Coding — every 4 hours
7 Python tasks, **7 trials each**. Six of the seven are repository-repair tasks: the model
is given a real codebase and a bug report, and graded on whether it fixed the actual defect
or silenced the symptom it was shown. Scored on 9 axes: correctness, stability, edge cases,
debugging, code quality, efficiency, format, safety, complexity.

### Tool calling — daily
9 sandboxed agent tasks (3 easy, 3 medium, 3 hard) covering file operations, multi-step
workflows and tool chaining. Scored on 7 axes: task completion, tool selection, parameter
accuracy, efficiency, error handling, context awareness, safety compliance.

### Deep reasoning — daily
4 multi-turn tasks that run to completion rather than a single exchange: IDE assistance,
specification following, document memory, project refactoring. Scored on 6 axes:
correctness, memory retention, plan coherence, context window handling, debugging,
stability.

### Canary — hourly
2 tasks, 2 trials, deliberately small. A fast probe whose only job is to notice change
between the heavier runs.

### Calibration and known-unknowns — every 6 hours
Does the model know what it does not know? Each model is asked questions from a stratified
corpus; some have a verifiable answer, some have none — a library function that does not
exist, a real company with an attribute it cannot have, a fact it has no way to observe.
Declining the second group is the correct behaviour and is scored as such. Reported per
model: **fabrication rate**, accuracy, abstention rate, **expected calibration error**, and
over/under-confidence.

These figures are published separately and never enter the composite score. Being well
calibrated does not make a model better at coding, and blending the two would make both
numbers mean less.

---

## Drift detection

A leaderboard tells you where a model stands today. Drift detection tells you when it moved.

- **Page-Hinkley change detection**, run **per suite** on that suite's own daily series —
  coding, tool use and reasoning are never blended, because each has a different scale and
  noise floor.
- A suite needs **10 measured days** on its current benchmark configuration before it may
  fire. The statistic is visible from day one, but a reading during warm-up is an early
  indication, not a verdict.
- **Any scoring or corpus change mints a new configuration id** and resets the baseline.
  Scores either side of such a change are not comparable and are not compared.
- **Task-level regressions** are detected separately, from execution records rather than
  composite scores. A single task collapsing is close to invisible in a composite — it
  moves the overall score by less than the alert threshold — so it is reported on its own.

---

## How the numbers are kept honest

This is the part we would want to read first about someone else's benchmark.

- **A provider that is not answering is skipped, not scored.** A dead API key or an outage
  must never be published as a model getting worse.
- **Refusals are recorded, not imputed.** If a model declines a task, that task drops out
  and the run is marked partial. A refusal is not a zero.
- **A missing suite drops out and the remaining weights are renormalised.** No suite is
  ever backfilled with a placeholder value.
- **Provenance is labelled.** Where a figure is not backed by a live measurement, the
  interface says so rather than presenting it as one.
- **Every task grades only what was asked.** Hidden requirements that were never stated in
  the prompt are not graded.
- **Ranks close together are statistical ties** and are presented as such, with confidence
  intervals rather than false precision.

The full methodology, including the scoring weights and the task design, is published at
[aistupidlevel.info/methodology](https://aistupidlevel.info/methodology).

---

## Smart router

An OpenAI-compatible endpoint that picks the model for each request from current benchmark
results instead of a hard-coded name.

```bash
curl -X POST https://aistupidlevel.info/v1/chat/completions \
  -H "Authorization: Bearer aism_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto-coding",
    "messages": [{"role": "user", "content": "Write a binary search in Python"}]
  }'
```

**Eight strategies:** `auto` (best overall), `auto-coding`, `auto-reasoning`,
`auto-creative`, `auto-tooling`, `auto-agentic`, `auto-cheapest`, `auto-fastest`.

**You bring your own provider keys.** They are encrypted at rest with AES-256-GCM, and the
providers bill you directly — we never resell tokens or mark them up. A provider outage
falls through to the next best model rather than failing your request. Works as a drop-in
replacement in tools that accept an OpenAI base URL, such as Cline, Continue and Cursor.

---

## Data API

Programmatic access to scores, rankings, history, drift signals and incidents, at
`https://aistupidlevel.info/api/v1` with `asl_` keys.

| Tier | Requests / day | Requests / minute |
|---|---|---|
| Free | 10 | 1 |
| Pro | 10,000 | 60 |
| Teams | 100,000 | 300 |
| Enterprise | 250,000 | 1,000 |

The free tier is an evaluation allowance — enough to build against, not to run on.
Documentation: [aistupidlevel.info/api-docs](https://aistupidlevel.info/api-docs).

---

## Who this is for

- **Teams running models in production** — know within hours when a model you depend on
  starts behaving differently, instead of finding out from your users.
- **Researchers** — a public methodology, published per-task results, and history that
  survives corpus changes because configuration boundaries are recorded rather than
  smoothed over.
- **Developers choosing a model** — per-suite comparisons, cost per million tokens, and
  measured latency, rather than a single aggregate that hides the trade-off you care about.
- **Anyone tired of vibes** — every published figure traces back to a run we can point at.

Current scores, every category ranking, seven days of history, provider status and the full
methodology are **free and require no account**. Paid plans add longer comparable history,
the drift curves, calibration data, exports and the routing volume.

---

## Technical notes

- **API:** TypeScript, Fastify, SQLite with Drizzle ORM, Redis caching
- **Web:** Next.js 14, React 18, Recharts, NextAuth
- **Infrastructure:** a single VPS under systemd, behind nginx with SSL
- **Frontend source:** [github.com/StudioPlatforms/aistupidmeter-web](https://github.com/StudioPlatforms/aistupidmeter-web)

The web frontend is public. The benchmark engine is not open source — the task corpus is
deliberately unpublished, because a benchmark whose answers a model has been trained on
measures nothing. The methodology, the scoring weights and the per-task outcomes are public
instead, which is what makes a result checkable without making it gameable.

---

## Status and transparency

A live [status page](https://aistupidlevel.info/status) reports provider availability and
separates three things that look identical from outside and mean different things: the
provider being down, our own account or key failing, and our probe failing. When a suite
could not run, the site says which and why rather than quietly publishing a lower number.

---

## Contact

- **Contact form:** [aistupidlevel.info/contact](https://aistupidlevel.info/contact)
- **LinkedIn:** [ASL — aistupidlevel.info](https://www.linkedin.com/company/asl-aistupidlevel-info)
- **X / Twitter:** [@aistupidlevel](https://x.com/aistupidlevel)
- **Website:** [aistupidlevel.info](https://aistupidlevel.info) · also reachable at
  [asl.info](https://asl.info)

---

**AI Stupid Level is an independent evaluation platform built by Ionut Adrian Visan. It is not affiliated with, endorsed by, or funded by any of the model providers
it measures — which is the only reason its numbers are worth anything.**