Publish AIME 2026 single-pass low-budget evaluation

All 30 public problems, one response each, low reasoning, 512 completion cap, exact-integer scoring. Custom baseline, not an official MathArena reproduction or independent certification. Existing Typed Decisions and SafeAgents evaluation files preserved.

.eval_results/aime2026.yaml

@@ -0,0 +1,13 @@
+- dataset:
+    id: MathArena/aime_2026
+    task_id: MathArena/aime_2026
+    revision: d61f486805306e6e6b5913504c07763a99e1b112
+  value: 100.0
+  date: '2026-10-31'
+  notes: 'Custom single-pass baseline; all 30 problems; low reasoning; max_completion_tokens=512
+    including reasoning; exact final integer scoring; 0 truncated outputs retained
+    as measured. Self-reported, not an official MathArena run. Organization: 江苏银行
+    (Bank of Jiangsu); public model display name: intellisu-coder-pro.'
+  source:
+    url: https://huggingface.co/BaiLing/intellisu-coder-pro/tree/main/evaluation/aime2026
+    name: All 30 outputs, exact-answer score and dataset source lock

README.md

@@ -6,6 +6,7 @@ tags:
 datasets:
 - LocalLLaMA/typed-decisions
 - harness-hub/safeagents-judge-bench
+- MathArena/aime_2026
 language:
 - en
 - zh
@@ -62,3 +63,24 @@ Full raw responses, binary/category verdicts, metrics and dataset revision/hash
 
 Attribution: harness-hub / SafeAgents Judge Bench, Apache-2.0 as declared by the dataset. This source license does not grant rights to the evaluated API model or its weights. No model weights, original reference-answer data or API credentials are uploaded. [Result YAML](.eval_results/safeagents-judge-bench.yaml).
 <!-- safeagents-evaluation:end -->
+
+<!-- aime2026-evaluation:start -->
+## AIME 2026 evaluation
+
+Model display name: **intellisu-coder-pro**  
+Organization display name: **江苏银行 (Bank of Jiangsu)**
+
+| Benchmark | Coverage | Protocol | Local self-reported result |
+|---|---:|---|---:|
+| [AIME 2026](https://huggingface.co/datasets/MathArena/aime_2026) | All 30 public problems | Single generated response per problem; exact final-integer comparison | 30/30 (100.0%) |
+
+Evaluated 2026-10-31; files published 2026-11-03. Fixed low reasoning and maximum 512 completion tokens including reasoning. All 30 raw responses were bare integers; there were zero truncated or unparsed outputs. API-reported usage was 12,636 tokens across 30 model calls. No resampling, answer-dependent retries or model judge was used. This publication reused saved outputs and did not rerun inference.
+
+**This is a custom single-pass, budget-limited baseline, not a reproduced or certified official MathArena leaderboard run.** It does not estimate a multi-sample average, establish unseen-test performance or support an independently verified first-place claim. The benchmark is public, and upstream training exposure is unknown. No training on this benchmark was performed by this runner.
+
+Only the problem text and a fixed final-integer instruction were sent to the API. The reference answer field was used only for local comparison after generation. Original reference-answer data are not uploaded. Raw outputs, exact protocol, metrics and source revision/hash are in [`evaluation/aime2026/`](evaluation/aime2026/). The dataset revision is `d61f486805306e6e6b5913504c07763a99e1b112`.
+
+Attribution: MathArena / AIME 2026, license declared by the source dataset: **CC-BY-NC-SA-4.0** ([dataset](https://huggingface.co/datasets/MathArena/aime_2026)). This attribution does not assign an open-source license to the evaluated API model or its weights. No model weights or API credentials are included. The metrics JSON retains generation-time local-file status; this model card records the later publication.
+
+[Result YAML](.eval_results/aime2026.yaml). Self-reported; no independent-verification badge is claimed.
+<!-- aime2026-evaluation:end -->

evaluation/aime2026/generation_errors.json

@@ -0,0 +1 @@
+[]

evaluation/aime2026/metrics.json

@@ -0,0 +1,17 @@
+{
+  "model": "intellisu-coder-pro",
+  "n": 30,
+  "passed": 30,
+  "accuracy_percent": 100.0,
+  "truncated": 0,
+  "unparsed": 0,
+  "reasoning_effort": "low",
+  "max_completion_tokens": 512,
+  "reported_tokens": 12636,
+  "http_attempts": 30,
+  "protocol": "Custom one-pass low-budget exact-answer baseline. Not a reproduced official MathArena leaderboard run. No resampling or LLM judge.",
+  "hf_dataset": "MathArena/aime_2026",
+  "hf_task_id": "MathArena/aime_2026",
+  "dataset_revision": "d61f486805306e6e6b5913504c07763a99e1b112",
+  "publish_status": "prepared_only_user_requested_no_HF_writes"
+}

evaluation/aime2026/responses.jsonl

@@ -0,0 +1,30 @@
+{"id": "1", "model": "intellisu-coder-pro", "response": "277", "prediction": 277, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 182, "completion_tokens": 81, "total_tokens": 263, "completion_tokens_details": {"reasoning_tokens": 74}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "2", "model": "intellisu-coder-pro", "response": "62", "prediction": 62, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 101, "completion_tokens": 110, "total_tokens": 211, "completion_tokens_details": {"reasoning_tokens": 103}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "3", "model": "intellisu-coder-pro", "response": "79", "prediction": 79, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 153, "completion_tokens": 72, "total_tokens": 225, "completion_tokens_details": {"reasoning_tokens": 65}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "4", "model": "intellisu-coder-pro", "response": "70", "prediction": 70, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 72, "completion_tokens": 151, "total_tokens": 223, "completion_tokens_details": {"reasoning_tokens": 144}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "5", "model": "intellisu-coder-pro", "response": "65", "prediction": 65, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 172, "completion_tokens": 62, "total_tokens": 234, "completion_tokens_details": {"reasoning_tokens": 55}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "6", "model": "intellisu-coder-pro", "response": "441", "prediction": 441, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 86, "completion_tokens": 152, "total_tokens": 238, "completion_tokens_details": {"reasoning_tokens": 145}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "7", "model": "intellisu-coder-pro", "response": "396", "prediction": 396, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 106, "completion_tokens": 91, "total_tokens": 197, "completion_tokens_details": {"reasoning_tokens": 84}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "8", "model": "intellisu-coder-pro", "response": "244", "prediction": 244, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 86, "completion_tokens": 161, "total_tokens": 247, "completion_tokens_details": {"reasoning_tokens": 154}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "9", "model": "intellisu-coder-pro", "response": "29", "prediction": 29, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 212, "completion_tokens": 199, "total_tokens": 411, "completion_tokens_details": {"reasoning_tokens": 192}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "10", "model": "intellisu-coder-pro", "response": "156", "prediction": 156, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 140, "completion_tokens": 678, "total_tokens": 818, "completion_tokens_details": {"reasoning_tokens": 671}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "11", "model": "intellisu-coder-pro", "response": "896", "prediction": 896, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 190, "completion_tokens": 326, "total_tokens": 516, "completion_tokens_details": {"reasoning_tokens": 319}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "12", "model": "intellisu-coder-pro", "response": "161", "prediction": 161, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 215, "completion_tokens": 203, "total_tokens": 418, "completion_tokens_details": {"reasoning_tokens": 196}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "13", "model": "intellisu-coder-pro", "response": "39", "prediction": 39, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 185, "completion_tokens": 107, "total_tokens": 292, "completion_tokens_details": {"reasoning_tokens": 100}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "14", "model": "intellisu-coder-pro", "response": "681", "prediction": 681, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 124, "completion_tokens": 283, "total_tokens": 407, "completion_tokens_details": {"reasoning_tokens": 276}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "15", "model": "intellisu-coder-pro", "response": "83", "prediction": 83, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 285, "completion_tokens": 1320, "total_tokens": 1605, "completion_tokens_details": {"reasoning_tokens": 1313}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "16", "model": "intellisu-coder-pro", "response": "178", "prediction": 178, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 77, "completion_tokens": 50, "total_tokens": 127, "completion_tokens_details": {"reasoning_tokens": 43}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "17", "model": "intellisu-coder-pro", "response": "243", "prediction": 243, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 384, "completion_tokens": 138, "total_tokens": 522, "completion_tokens_details": {"reasoning_tokens": 131}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "18", "model": "intellisu-coder-pro", "response": "503", "prediction": 503, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 164, "completion_tokens": 321, "total_tokens": 485, "completion_tokens_details": {"reasoning_tokens": 314}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "19", "model": "intellisu-coder-pro", "response": "279", "prediction": 279, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 157, "completion_tokens": 111, "total_tokens": 268, "completion_tokens_details": {"reasoning_tokens": 104}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "20", "model": "intellisu-coder-pro", "response": "190", "prediction": 190, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 272, "completion_tokens": 118, "total_tokens": 390, "completion_tokens_details": {"reasoning_tokens": 111}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "21", "model": "intellisu-coder-pro", "response": "50", "prediction": 50, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 96, "completion_tokens": 134, "total_tokens": 230, "completion_tokens_details": {"reasoning_tokens": 127}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "22", "model": "intellisu-coder-pro", "response": "754", "prediction": 754, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 146, "completion_tokens": 113, "total_tokens": 259, "completion_tokens_details": {"reasoning_tokens": 106}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "23", "model": "intellisu-coder-pro", "response": "245", "prediction": 245, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 110, "completion_tokens": 339, "total_tokens": 449, "completion_tokens_details": {"reasoning_tokens": 332}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "24", "model": "intellisu-coder-pro", "response": "669", "prediction": 669, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 119, "completion_tokens": 218, "total_tokens": 337, "completion_tokens_details": {"reasoning_tokens": 211}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "25", "model": "intellisu-coder-pro", "response": "850", "prediction": 850, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 181, "completion_tokens": 227, "total_tokens": 408, "completion_tokens_details": {"reasoning_tokens": 220}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "26", "model": "intellisu-coder-pro", "response": "132", "prediction": 132, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 139, "completion_tokens": 380, "total_tokens": 519, "completion_tokens_details": {"reasoning_tokens": 373}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "27", "model": "intellisu-coder-pro", "response": "223", "prediction": 223, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 175, "completion_tokens": 243, "total_tokens": 418, "completion_tokens_details": {"reasoning_tokens": 236}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "28", "model": "intellisu-coder-pro", "response": "107", "prediction": 107, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 168, "completion_tokens": 313, "total_tokens": 481, "completion_tokens_details": {"reasoning_tokens": 306}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "29", "model": "intellisu-coder-pro", "response": "157", "prediction": 157, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 215, "completion_tokens": 449, "total_tokens": 664, "completion_tokens_details": {"reasoning_tokens": 442}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}
+{"id": "30", "model": "intellisu-coder-pro", "response": "393", "prediction": 393, "correct": true, "finish_reason": "stop", "usage": {"prompt_tokens": 206, "completion_tokens": 568, "total_tokens": 774, "completion_tokens_details": {"reasoning_tokens": 561}}, "params": {"reasoning_effort": "low", "max_completion_tokens": 512}}

evaluation/aime2026/run_aime30_low.py

@@ -0,0 +1,102 @@
+"""Full 30-question, fixed-budget AIME baseline. Generates local HF eval YAML only."""
+import asyncio
+import datetime as dt
+import hashlib
+import io
+import json
+from pathlib import Path
+import re
+
+import httpx
+import pandas as pd
+
+ROOT=Path(__file__).resolve().parents[1]
+DEST=ROOT/'runs/dynamic_boards/aime2026_low'
+SOURCE='MathArena/aime_2026'
+ALIAS='intellisu-coder-pro'
+TOKEN_BUDGET=25000
+OUTPUT_CAP=512
+
+
+def prediction(text):
+    text=text.strip()
+    if re.fullmatch(r'\d{1,3}',text):return int(text)
+    matches=re.findall(r'\\boxed\{(\d{1,3})\}',text)
+    if matches:return int(matches[-1])
+    matches=re.findall(r'(?i)(?:final answer|answer)\s*(?:is|:|=)?\s*\$?(\d{1,3})(?!\d)',text)
+    return int(matches[-1]) if matches else None
+
+
+class Run:
+    def __init__(self):
+        self.config=json.loads((ROOT/'.private/api.json').read_text())
+        self.results={};self.used=0;self.reserved=0;self.calls=0;self.stopped=False
+        self.lock=asyncio.Lock();self.sem=asyncio.Semaphore(2)
+
+    def save(self):
+        d={'model':ALIAS,'reasoning_effort':'low','max_completion_tokens':OUTPUT_CAP,'token_budget':TOKEN_BUDGET,'reported_tokens':self.used,'http_attempts':self.calls,'stopped':self.stopped,'results':self.results}
+        p=DEST/'checkpoint.json';tmp=p.with_suffix('.tmp');tmp.write_text(json.dumps(d,ensure_ascii=False,indent=2));tmp.replace(p)
+
+    async def one(self,q):
+        identifier=str(q['problem_idx'])
+        if identifier in self.results:return
+        async with self.sem:
+            if self.stopped:return
+            messages=[{'role':'system','content':'Solve the mathematics problem. Return only the final integer from 0 through 999. No explanation. If you cannot solve it, return UNKNOWN.'},{'role':'user','content':q['problem']}]
+            reserve=sum(len(m['content'].encode()) for m in messages)+500+OUTPUT_CAP
+            async with self.lock:
+                if self.used+self.reserved+reserve>TOKEN_BUDGET or self.calls>=40:
+                    self.stopped=True;self.save();return
+                self.reserved+=reserve
+            try:
+                for attempt in range(2):
+                    if self.stopped:return
+                    self.calls+=1
+                    r=await self.client.post(self.config['base_url']+'/chat/completions',headers={'Authorization':'Bearer '+self.config['api_key']},json={'model':self.config['model'],'messages':messages,'reasoning_effort':'low','max_completion_tokens':OUTPUT_CAP})
+                    data=r.json();usage=data.get('usage') or {}
+                    self.used+=usage.get('total_tokens',0) or 0
+                    if r.status_code==429 and attempt==0:
+                        await asyncio.sleep(15);continue
+                    if r.status_code!=200:
+                        self.stopped=True
+                        (DEST/'api_error.json').write_text(json.dumps({'http_status':r.status_code,'problem_idx':q['problem_idx']},indent=2));return
+                    choice=data['choices'][0];text=choice['message'].get('content') or ''
+                    pred=prediction(text)
+                    self.results[identifier]={'id':identifier,'model':ALIAS,'response':text,'prediction':pred,'correct':pred is not None and pred==int(q['answer']),'finish_reason':choice.get('finish_reason'),'usage':usage,'params':{'reasoning_effort':'low','max_completion_tokens':OUTPUT_CAP}}
+                    return
+            finally:
+                self.reserved-=reserve;self.save()
+
+    async def run(self):
+        DEST.mkdir(parents=True,exist_ok=True)
+        p=DEST/'checkpoint.json'
+        if p.exists():
+            d=json.loads(p.read_text());self.results=d['results'];self.used=d['reported_tokens'];self.calls=d['http_attempts']
+        metadata=json.loads((ROOT/'.private/dynamic_board_research/small_hf_specs/MathArena__aime_2026/metadata.json').read_text())
+        revision=metadata['sha']
+        url=f'https://huggingface.co/datasets/{SOURCE}/resolve/{revision}/data/train-00000-of-00001.parquet'
+        source=DEST/'source.parquet'
+        if not source.exists():
+            async with httpx.AsyncClient(timeout=60,follow_redirects=True) as c:r=await c.get(url);r.raise_for_status();source.write_bytes(r.content)
+        questions=pd.read_parquet(source).to_dict('records')
+        assert len(questions)==30 and len({q['problem_idx'] for q in questions})==30
+        (DEST/'source_lock.json').write_text(json.dumps({'dataset':SOURCE,'revision':revision,'url':url,'sha256':hashlib.sha256(source.read_bytes()).hexdigest(),'n':30,'license':'cc-by-nc-sa-4.0'},indent=2))
+        async with httpx.AsyncClient(timeout=httpx.Timeout(120,connect=20,pool=10),limits=httpx.Limits(max_connections=2,max_keepalive_connections=2)) as client:
+            self.client=client
+            outcomes=await asyncio.gather(*(self.one(q) for q in questions),return_exceptions=True)
+        errors=[{'id':q['problem_idx'],'error':type(e).__name__+': '+str(e)} for q,e in zip(questions,outcomes) if isinstance(e,Exception)]
+        (DEST/'generation_errors.json').write_text(json.dumps(errors,ensure_ascii=False,indent=2));self.save()
+        if len(self.results)==30:
+            passed=sum(r['correct'] for r in self.results.values());score=100*passed/30
+            result={'model':ALIAS,'n':30,'passed':passed,'accuracy_percent':score,'truncated':sum(r['finish_reason']=='length' for r in self.results.values()),'unparsed':sum(r['prediction'] is None for r in self.results.values()),'reasoning_effort':'low','max_completion_tokens':OUTPUT_CAP,'reported_tokens':self.used,'http_attempts':self.calls,'protocol':'Custom one-pass low-budget exact-answer baseline. Not a reproduced official MathArena leaderboard run. No resampling or LLM judge.','hf_dataset':SOURCE,'hf_task_id':SOURCE,'dataset_revision':revision,'publish_status':'prepared_only_user_requested_no_HF_writes'}
+            (DEST/'metrics.json').write_text(json.dumps(result,ensure_ascii=False,indent=2))
+            (DEST/'responses.jsonl').write_text(''.join(json.dumps(self.results[str(q['problem_idx'])],ensure_ascii=False)+'\n' for q in questions))
+            note=f'Custom single-pass baseline; all 30 problems; low reasoning; max_completion_tokens=512 including reasoning; exact final integer scoring; {result["truncated"]} truncated outputs retained as measured. Self-reported, not an official MathArena run.'
+            import yaml
+            scores=[{'dataset':{'id':SOURCE,'task_id':SOURCE,'revision':revision},'value':score,'date':dt.datetime.now(dt.timezone(dt.timedelta(hours=8))).date().isoformat(),'notes':note}]
+            (DEST/'aime2026.yaml').write_text(yaml.safe_dump(scores,allow_unicode=True,sort_keys=False))
+            print(json.dumps(result,ensure_ascii=False),flush=True)
+        else:print(json.dumps({'completed':len(self.results),'calls':self.calls,'tokens':self.used,'stopped':self.stopped,'errors':errors},ensure_ascii=False),flush=True)
+
+
+if __name__=='__main__':asyncio.run(Run().run())

evaluation/aime2026/source_lock.json

@@ -0,0 +1,8 @@
+{
+  "dataset": "MathArena/aime_2026",
+  "revision": "d61f486805306e6e6b5913504c07763a99e1b112",
+  "url": "https://huggingface.co/datasets/MathArena/aime_2026/resolve/d61f486805306e6e6b5913504c07763a99e1b112/data/train-00000-of-00001.parquet",
+  "sha256": "ae72853349a46c5f08666feb42db2d09fb4ac4fd4768c726b29702df21c7768e",
+  "n": 30,
+  "license": "cc-by-nc-sa-4.0"
+}

public_identity.json

@@ -12,6 +12,7 @@
   "publication_date": "2026-11-03",
   "published_evaluations": [
     "Typed Decisions",
-    "SafeAgents Judge Bench"
+    "SafeAgents Judge Bench",
+    "AIME 2026"
   ]
 }