Upload 4 files

data.js

@@ -51,7 +51,7 @@ window.BENCH_DATA = {
     title: "September 2026 Benchmarks",
     badge: "Published September 28, 2026",
     summary:
-      "Seven public benchmarks, measured on Aligned AI's production pipeline and scored by each benchmark's official grader.",
+      "Nine public benchmarks, measured on Aligned AI's production pipeline and scored by each benchmark's official grader.",
     measured: "Sep 2026",
     preview: false,
     headlineBoards: ["graphwalks", "simpleqa", "gsm8k"],
@@ -68,17 +68,17 @@ window.BENCH_DATA = {
       { id: "truthfulqa", why: "Questions designed to draw out common misconceptions. Aligned AI ranks third among 8 models." }
     ],
     methodology: [
-      "All Aligned AI scores come from Aligned AI's production pipeline at joinaligned.ai.",
-      "Questions are labeled samples drawn from each benchmark's official test set. Every card states the sample size (e.g. \u201c150 of 1,172 questions\u201d). Scores are graded with the benchmark's official grader or reference scoring method.",
-      "Every card links the benchmark's official dataset and, where one exists, its original paper and shows its run ID as plain-text provenance. Scores are from Aligned AI's production pipeline, measured in September 2026. Dates are fixed and never auto-generated.",
-      "Competitor scores come from Aligned's competitor dataset compiled on September 28, 2026 (the top 15 recent models per benchmark, released July 2025 or later). Ranks are computed from those entries only.",
-      "Disclaimers. All Aligned AI scores on this page are self-reported. Aligned ran every benchmark itself on its own infrastructure; no third party has independently run or verified these results.",
-      "Aligned AI is a compound system (a production pipeline that can combine multiple models, tools and retrieval), not a single standalone model. Competitor scores generally describe individual models under their own publishers' settings, so comparisons are indicative, not like-for-like.",
-      "Aligned AI scores come from labeled samples of each official test set (sample sizes are stated on each card), not the full test set. Scores from samples carry statistical uncertainty and may differ on a full run.",
-      "Competitor scores are a point-in-time snapshot of publicly reported results, collected on September 28, 2026. They were not re-run by Aligned, and publishers may have since updated, corrected or removed them, so current figures may differ. Evaluation settings (prompting, number of attempts, tool use, grader version) may differ between sources and from Aligned's runs.",
+      "Source of results. All Aligned AI scores were produced by the Aligned AI production pipeline available at joinaligned.ai, as measured in September 2026.",
+      "Evaluation method. Aligned AI was evaluated on samples drawn from each benchmark’s official test set and scored using the benchmark’s official grader or reference scoring method. Sample sizes are stated on the relevant board where recorded. Each board references the benchmark’s official dataset and, where available, its original publication.",
+      "Competitor data. Competitor scores are drawn from a dataset compiled by Aligned on September 28, 2026, covering leading models released in or after July 2025. Rankings reflect Aligned AI’s position relative to the models in that dataset only and are not intended as a comprehensive ranking of all available models.",
+      "Self-reported results. All Aligned AI scores on this page are self-reported. Each evaluation was conducted by Aligned on its own infrastructure, and no independent third party has reproduced, audited or verified these results.",
+      "System comparison. Aligned AI is a compound system that may combine multiple models, tools and retrieval within a single production pipeline; it is not a single standalone model. Competitor scores generally describe individual models evaluated under their publishers’ own settings. Comparisons are therefore indicative only and should not be construed as like-for-like.",
+      "Sampling. Aligned AI scores are based on samples of each official test set rather than the complete test set. Sample-based results are subject to statistical uncertainty and may differ from results on a full evaluation.",
+      "Preliminary results. Certain scores, including MATH-500, MBPP and TriviaQA, were obtained from preliminary evaluation runs of approximately 50 questions on an earlier version of the Aligned AI pipeline. Results from limited samples are subject to substantial variance, should be regarded as indicative estimates only, and may be materially higher or lower in subsequent or full-scale evaluations.",
+      "Third-party figures. Competitor scores represent a point-in-time snapshot of publicly reported results as of September 28, 2026. They were not independently re-run by Aligned and may since have been updated, corrected or withdrawn by their publishers. Evaluation conditions, including prompting, number of attempts, tool use and grader version, may vary between sources and from those used by Aligned. Aligned makes no representation as to the accuracy or completeness of third-party figures.",
       "SimpleQA: Aligned AI answered with web search enabled.",
-      "[2] GraphWalks: the run reached its time limit, so the 100% score covers only the 92 questions that completed out of 150 sampled.",
-      "Results reflect the Aligned AI production pipeline as measured in September 2026 and may change as the system is updated. Model names and trademarks belong to their respective owners; their inclusion does not imply endorsement. This page is provided for informational purposes only and makes no warranty of performance on any particular task."
+      "[2] GraphWalks: the evaluation reached its time limit, and the reported score reflects the 92 of 150 sampled questions that completed.",
+      "No warranty. Results reflect the Aligned AI pipeline at the time of measurement and may change as the system is updated. All information on this page is provided for informational purposes only, on an “as is” basis, without representation or warranty of any kind, express or implied, including as to accuracy, completeness or future performance. Benchmark results do not guarantee performance on any particular task, use or deployment. Model names and trademarks are the property of their respective owners, and their inclusion does not imply affiliation with or endorsement by those owners."
     ]
   },
 
@@ -163,19 +163,26 @@ window.BENCH_DATA = {
     },
     {
       id: "humaneval", name: "HumanEval", category: "Code", fieldSize: 19,
-      description: "Code-generation benchmark from OpenAI (Chen et al., 2021). 164 hand-written Python programming problems, each with a function signature, docstring, and unit tests. A solution counts as correct only if it passes every test (pass@1).",
-      unit: "pass@1 %", higherIsBetter: true,
+      description: "Code-generation benchmark from OpenAI (Chen et al., 2021). 164 hand-written Python programming problems, each with a function signature, docstring, and unit tests. A solution counts as correct only if it passes every test.",
+      unit: "accuracy %", higherIsBetter: true,
       links: { dataset: "https://huggingface.co/datasets/openai/openai_humaneval", datasetLabel: "Official dataset (OpenAI)", paper: "https://arxiv.org/abs/2107.03374", paperLabel: "Paper (Chen et al., 2021)" },
       aligned: { rank: 16, score: 92.0, sample: "138 of 150 problems passed", runId: "36425671251" },
       competitors: [{"model": "Claude Sonnet 4.5", "vendor": "anthropic", "lab": "Anthropic", "rank": 1, "score": 97.6, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Anthropic (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Grok 4", "vendor": "xai", "lab": "xAI", "rank": 2, "score": 97.0, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "xAI (Aligned competitor dataset, Sep 28 2026)"}, {"model": "GLM-5", "vendor": "other", "lab": "Zhipu AI", "rank": 2, "score": 97.0, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Zhipu AI (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Claude Opus 4.6", "vendor": "anthropic", "lab": "Anthropic", "rank": 2, "score": 97.0, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Anthropic (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Claude Opus 4.5", "vendor": "anthropic", "lab": "Anthropic", "rank": 2, "score": 97.0, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Anthropic (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Gemini 3 Pro", "vendor": "google", "lab": "Google", "rank": 2, "score": 97.0, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Google (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Claude Haiku 4.5", "vendor": "anthropic", "lab": "Anthropic", "rank": 7, "score": 96.3, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Anthropic (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Qwen3.5 397B A17B", "vendor": "qwen", "lab": "Alibaba", "rank": 7, "score": 96.3, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Alibaba (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Claude Opus 4.1", "vendor": "anthropic", "lab": "Anthropic", "rank": 9, "score": 95.7, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Anthropic (Aligned competitor dataset, Sep 28 2026)"}, {"model": "INTELLECT-3", "vendor": "other", "lab": "Prime Intellect", "rank": 10, "score": 95.1, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Prime Intellect (Aligned competitor dataset, Sep 28 2026)"}, {"model": "GPT-5.1", "vendor": "openai", "lab": "OpenAI", "rank": 11, "score": 94.5, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "OpenAI (Aligned competitor dataset, Sep 28 2026)"}, {"model": "DeepSeek V3.2", "vendor": "deepseek", "lab": "DeepSeek", "rank": 12, "score": 93.9, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "DeepSeek (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Nemotron 3 Nano 30B A3B", "vendor": "other", "lab": "NVIDIA", "rank": 13, "score": 93.3, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "NVIDIA (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Qwen3.5 Plus", "vendor": "qwen", "lab": "Alibaba", "rank": 13, "score": 93.3, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Alibaba (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Qwen3 Coder 480B A35B", "vendor": "qwen", "lab": "Alibaba", "rank": 15, "score": 92.7, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Alibaba (Aligned competitor dataset, Sep 28 2026)"}]
     },
     {
-      // Rerunning now. Hidden until score + runId are filled in and pending is set to false.
-      id: "math-500", name: "MATH-500", category: "Math", fieldSize: 114, pending: true,
+      id: "mbpp", name: "MBPP", category: "Code", fieldSize: 16,
+      description: "Mostly Basic Python Problems, a code-generation benchmark from Google Research (Austin et al., 2021). Entry-level Python programming tasks, each with a short description and three test cases. A solution counts as correct only if it passes every test.",
+      unit: "accuracy %", higherIsBetter: true,
+      links: { dataset: "https://huggingface.co/datasets/google-research-datasets/mbpp", datasetLabel: "Official dataset (Google Research)", paper: "https://arxiv.org/abs/2108.07732", paperLabel: "Paper (Austin et al., 2021)" },
+      aligned: { rank: 4, score: 84.0, sample: "", runId: "" },
+      competitors: [{"model": "Sarvam-30B", "vendor": "other", "lab": "Sarvam AI", "rank": 1, "score": 92.7, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Sarvam AI (Aligned competitor dataset, Sep 28 2026)"}, {"model": "MiniCPM-SALA", "vendor": "other", "lab": "OpenBMB", "rank": 2, "score": 89.1, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "OpenBMB (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Gemini 3 Pro", "vendor": "google", "lab": "Google", "rank": 3, "score": 86.2, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Google (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Stable-DiffCoder-8B (base)", "vendor": "other", "lab": "ByteDance", "rank": 5, "score": 83.6, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "ByteDance (Aligned competitor dataset, Sep 28 2026)"}, {"model": "GPT-5 (high)", "vendor": "openai", "lab": "OpenAI", "rank": 6, "score": 83.1, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "OpenAI (Aligned competitor dataset, Sep 28 2026)"}, {"model": "ERNIE 5.0", "vendor": "other", "lab": "Baidu", "rank": 7, "score": 82.5, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Baidu (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Kimi K2 Instruct", "vendor": "other", "lab": "Moonshot AI", "rank": 8, "score": 81.8, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Moonshot AI (Aligned competitor dataset, Sep 28 2026)"}, {"model": "DeepSeek-V3.2 (thinking)", "vendor": "deepseek", "lab": "DeepSeek", "rank": 9, "score": 81.5, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "DeepSeek (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Seed-OSS-36B (base)", "vendor": "other", "lab": "ByteDance", "rank": 10, "score": 80.6, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "ByteDance (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Qwen3-235B-A22B-Instruct-2507", "vendor": "qwen", "lab": "Alibaba", "rank": 11, "score": 79.9, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Alibaba (Aligned competitor dataset, Sep 28 2026)"}, {"model": "DeepSeek-V3.1", "vendor": "deepseek", "lab": "DeepSeek", "rank": 11, "score": 79.9, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "DeepSeek (Aligned competitor dataset, Sep 28 2026)"}, {"model": "LongCat-Flash-Chat", "vendor": "other", "lab": "Meituan", "rank": 13, "score": 79.6, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Meituan (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Kimi K2 (base)", "vendor": "other", "lab": "Moonshot AI", "rank": 14, "score": 79.4, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Moonshot AI (Aligned competitor dataset, Sep 28 2026)"}, {"model": "ERNIE 5.0 (base)", "vendor": "other", "lab": "Baidu", "rank": 15, "score": 79.1, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Baidu (Aligned competitor dataset, Sep 28 2026)"}, {"model": "DeepSeek-V3.2-Exp (base)", "vendor": "deepseek", "lab": "DeepSeek", "rank": 16, "score": 78.8, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "DeepSeek (Aligned competitor dataset, Sep 28 2026)"}]
+    },
+    {
+      id: "math-500", name: "MATH-500", category: "Math", fieldSize: 114,
       description: "Competition mathematics benchmark. A 500-problem subset, selected by OpenAI (Lightman et al., 2023), of the MATH dataset of high-school competition problems (Hendrycks et al., 2021). Scored on the final boxed answer.",
       unit: "accuracy %", higherIsBetter: true,
       links: { dataset: "https://huggingface.co/datasets/HuggingFaceH4/MATH-500", paper: "https://arxiv.org/abs/2103.03874", paperLabel: "Paper (Hendrycks et al., 2021)", extra: [{ label: "Subset paper (Lightman et al., 2023)", url: "https://arxiv.org/abs/2305.20050" }] },
-      aligned: { rank: 15, score: null, sample: "", runId: "" },
+      aligned: { rank: 15, score: 97.9, sample: "", runId: "" },
       competitors: [{"model": "Qwen3 235B A22B Thinking 2507", "vendor": "qwen", "lab": "Alibaba", "rank": 1, "score": 99.6, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Alibaba (Aligned competitor dataset, Sep 28 2026)"}, {"model": "GPT-5", "vendor": "openai", "lab": "OpenAI", "rank": 2, "score": 99.4, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "OpenAI (Aligned competitor dataset, Sep 28 2026)"}, {"model": "LongCat-Flash-Thinking", "vendor": "other", "lab": "Meituan", "rank": 3, "score": 99.2, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Meituan (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Gemini 3.8 Flash", "vendor": "google", "lab": "Google", "rank": 4, "score": 99.1, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Google (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Gemini 3.7 Flash", "vendor": "google", "lab": "Google", "rank": 5, "score": 99.0, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Google (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Grok 4", "vendor": "xai", "lab": "xAI", "rank": 5, "score": 99.0, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "xAI (Aligned competitor dataset, Sep 28 2026)"}, {"model": "DeepSeek V3.1", "vendor": "deepseek", "lab": "DeepSeek", "rank": 7, "score": 98.8, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "DeepSeek (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Llama 3.3 Nemotron Super 49B v1.5", "vendor": "other", "lab": "NVIDIA", "rank": 8, "score": 98.3, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "NVIDIA (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Grok 4.6", "vendor": "xai", "lab": "xAI", "rank": 9, "score": 98.2, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "xAI (Aligned competitor dataset, Sep 28 2026)"}, {"model": "DeepSeek V4 Pro 0813", "vendor": "deepseek", "lab": "DeepSeek", "rank": 9, "score": 98.2, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "DeepSeek (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Qwen3 235B A22B Instruct 2507", "vendor": "qwen", "lab": "Alibaba", "rank": 11, "score": 98.0, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Alibaba (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Qwen3.5 Flash", "vendor": "qwen", "lab": "Alibaba", "rank": 11, "score": 98.0, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Alibaba (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Qwen3.6 Max Preview", "vendor": "qwen", "lab": "Alibaba", "rank": 11, "score": 98.0, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Alibaba (Aligned competitor dataset, Sep 28 2026)"}, {"model": "Gemini 3.6 Flash", "vendor": "google", "lab": "Google", "rank": 11, "score": 98.0, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Google (Aligned competitor dataset, Sep 28 2026)"}, {"model": "GLM-4.5", "vendor": "other", "lab": "Zhipu AI", "rank": 15, "score": 97.9, "source": "aligned-competitor-data-2026-09-28", "sourceLabel": "Zhipu AI (Aligned competitor dataset, Sep 28 2026)"}]
     }
   ]