AI Humanizer Benchmark

A monthly, independently verifiable benchmark of AI humanizers. Every tool rewrites the same freshly generated texts on its default settings, and every output is scored by a panel of commercial AI detectors (GPTZero, Originality.ai, Copyleaks, Pangram, ZeroGPT, QuillBot, and Grammarly), alongside meaning preservation, readability, and consistency. Every input, every output, every detector verdict, and the frozen scoring code are published, so anyone can recompute the rankings from scratch without trusting us.

A new cycle runs every month on freshly generated texts. Published cycles are never edited; corrections land as new cycles with a note in the changelog.

Current rankings

The live leaderboard, with per-detector and per-category breakdowns for every tool, is at aihumanizerbenchmark.com/leaderboard. The same numbers, cycle by cycle, are in the dataset below.

How a cycle works

1. Commit. A random 32-byte nonce is generated and only its SHA-256 hash is published, before anything runs, together with the list of models that will write the source texts. The nonce itself stays private until the cycle closes.
2. Select. That nonce seeds which values fill the prompt templates and which model writes each one, so the prompt set is fixed before any tool sees it, and could not have been chosen to favour anyone.
3. Write the corpus. One AI-written text per prompt, from the model assigned to it, spanning a fixed set of writing categories (academic essays, application essays, blog posts, business emails, discussion-board posts, marketing copy, news articles). Every humanizer gets the identical set.
4. Run. Each tool is exercised automatically on its default settings, unmodified. The exact settings used are recorded and published per tool.
5. Score. Every output goes to every detector on the panel, plus meaning-preservation and readability measurement.
6. Publish. The nonce is revealed, the full data bundle is written, and a manifest pins every file under SHA-256 hashes.

The ordering is the point: the commitment is public before the run, so the prompt set cannot be reselected after seeing which tools it flatters, not even by us.

How the score is built

The composite is 0–100: detector bypass 42%, meaning preservation 32%, readability 16%, consistency 10%, minus penalties.

• Bypass — per test, the median verdict across the detector panel: a tool must fool more than half the panel on a text for it to count, so a single lenient detector can never carry a tool. Per tool, the mean across every test. This is also why single-detector marketing claims ("passes GPTZero") tell you very little: detectors disagree widely, and the strictest and most lenient differ by roughly 2× in how much humanized text they pass.
• Meaning preservation — embedding similarity between input and output, rescaled so realistic differences spread across the full scale. A humanizer that changes what the text says hasn't humanized it.
• Readability — a writing-quality rating of the output, frozen per test in the published data.
• Consistency — how evenly a tool performs across the writing categories. A tool that beats detectors on blog posts but folds on business emails scores poorly here.
• Penalties — deducted for quality failures: severe meaning drift, length inflation (>1.4× the input) or deflation (<0.6×), refusals, and output returned unchanged. Each occurrence is counted and published.

A tool that fails to return output on at least half its attempts is marked unavailable and excluded rather than scored: a service being down is not evidence about its quality.

Fully open data and code

Everything needed to re-derive the leaderboard is public:

• Every input text, every humanized output, every detector verdict — one row per tool per text, including the exact settings each tool ran with.
• The scoring code, frozen per cycle. Each cycle directory carries the scoring.js that actually scored it, so an old cycle still verifies under the rules it was scored under, and changing today's scoring can't retroactively touch old results.
• The commit–reveal record (commit.json, nonce.txt) and a SHA-256 manifest of every file.

Recomputing the entire leaderboard is one command, npm run verify in the GitHub repository, and it runs in CI on every push. The verifier checks the whole chain: nonce → prompts → samples → tests → scores → leaderboard. Spot-checking a single verdict needs no code at all: paste any published output into the detector and compare.

The same data ships as analysis-ready tables in the Hugging Face dataset:

from datasets import load_dataset
tests = load_dataset("ai-humanizer-benchmark/ai-humanizer-benchmark", "tests")
leaderboard = load_dataset("ai-humanizer-benchmark/ai-humanizer-benchmark", "leaderboard")

Licensing: cycle data is CC BY 4.0: use it freely, including commercially, with attribution. Verifier and scoring code are MIT.

What this benchmark is not

• Not affiliate media. No tool pays for placement, there are no affiliate links, and no number comes from a vendor. Every tool is paid for and run through the identical pipeline, including UndetectedGPT, whose team operates this benchmark (how conflicts are handled).
• Not a one-off study. Detectors and humanizers both update constantly; a tool that cleared detectors last month may not clear them today. That's why the benchmark re-runs monthly instead of publishing once, and why old cycles are kept as historical record, never edited.
• Not a black box. If you believe a result is wrong, open an issue or use the dispute form and point at the test id: every row ships with its input and output, so disagreements are settled by looking at the data.

FAQ

How often is this updated? Monthly, on the first of each month.

Can I reproduce the rankings myself? Yes: clone the GitHub repo and run npm run verify, or load the dataset above and recompute from the raw rows. No API keys or network access needed; the verifier uses only the published files.

Why a panel of detectors? Detectors disagree widely. Scoring against a panel with a median rule measures what survives most detectors, not what slips past one. Per-detector breakdowns are published for every tool.

Links

• 🌐 Website & leaderboard: aihumanizerbenchmark.com
• 📊 Raw data: ai-humanizer-benchmark dataset
• 🔍 Per-prompt evidence: aihumanizerbenchmark.com/evidence
• 💻 Data + scoring code: GitHub
• 🤖 Machine-readable summary: llms.txt