SeaWolf-AI commited on
Commit
5ee02b9
·
verified ·
1 Parent(s): 9259257

card: inherit Darwin-180B-RSI card; eight #1s in the RSI line (ExtractBench by R3, seven by R1)

Browse files
Files changed (1) hide show
  1. README.md +291 -30
README.md CHANGED
@@ -11,24 +11,122 @@ tags:
11
  - model-level-rsi
12
  - recursive-self-improvement
13
  - self-improvement
14
- - reasoning
 
 
 
15
  - moe
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
16
  - ztc
 
 
 
 
 
 
17
  ---
18
 
19
  # Darwin-180B-RSI-R3
20
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
21
  **The second round of model-level self-improvement on top of [Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI).**
 
 
22
 
23
- Darwin-180B-RSI (R1) holds first place on seven Hugging Face official leaderboards. R3 continues training from R1 with the same recipe: the model solves verifiable problems, keeps only its own solutions that check out as correct, and trains on them. No human-written solutions or reasoning traces are used.
24
 
25
- R3 is released so anyone can download it, run it, and check the numbers below.
26
 
27
- <a href="https://huggingface.co/FINAL-Bench/POCKET-Darwin-180B-GGUF"><img src="https://img.shields.io/badge/💻_POCKET_4--bit-Laptop_·_CPU_only_·_DGX_Spark-0f766e?style=for-the-badge"></a>
 
 
 
 
 
 
 
 
 
 
 
28
 
29
- > 💻 **Run it on your own machine — [POCKET-Darwin-180B-GGUF](https://huggingface.co/FINAL-Bench/POCKET-Darwin-180B-GGUF)**: the 4-bit GGUF of R3 (111 GB) runs on a **laptop with an 8 GB GPU and 32 GB RAM**, **CPU only at 18–21 tok/s**, a 128 GB mini PC or one DGX Spark — MMLU-Pro **identical to BF16 (87.65%)**.
30
 
31
- ## What changed from R1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
32
 
33
  - **Starting point:** R1 weights (not the parent). R3 is a true second round.
34
  - **Practice problems:** 3,000 SuperGPQA questions (middle and hard difficulty) that were never used in R1 training. R1 solved each one 8 times.
@@ -36,8 +134,6 @@ R3 is released so anyone can download it, run it, and check the numbers below.
36
  - **What was trained:** the same components as R1 (attention paths and shared experts) via LoRA, then merged. All 512 routed experts, the router and the vision encoder are unchanged.
37
  - **No benchmark data:** GPQA Diamond and the held-out set below were never used for training or selection.
38
 
39
- ## Results (same settings for every model)
40
-
41
  ### Held-out SuperGPQA, 1,000 questions never used in training or selection
42
  4 samples per question, 16K thinking budget, temperature 1.0.
43
 
@@ -46,7 +142,7 @@ R3 is released so anyone can download it, run it, and check the numbers below.
46
  | R1 (Darwin-180B-RSI) | 65.30 | 65.67 | 68.30 |
47
  | **R3 (this model)** | **66.30** | **66.70** | **69.00** |
48
 
49
- Paired per-question difference, R1 �� R3 (mean of 4): **+1.03 points, 95% CI [+0.05, +2.00]**. The gain is small but statistically significant.
50
 
51
  ### GPQA Diamond, 198 questions
52
  8 samples per question, 32K thinking budget, temperature 1.0.
@@ -57,47 +153,212 @@ Paired per-question difference, R1 → R3 (mean of 4): **+1.03 points, 95% CI [+
57
  | R1 (Darwin-180B-RSI) | 84.85 | 85.80 | 89.90 |
58
  | **R3 (this model)** | **85.86** | **86.05** | 90.40 |
59
 
60
- Paired differences on GPQA (mean of 8): R0 → R3 +0.69 [−0.71, +2.10], R1 → R3 +0.25 [−1.18, +1.69]. With 198 questions these are within noise; we report them as measured.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
61
 
62
- **How to read this:** on a large held-out set, the second round of self-improvement produced a measurable gain over R1. On GPQA the model was already near its ceiling and the differences are not significant.
63
 
64
- ## Leaderboard status
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
65
 
66
- The seven Hugging Face leaderboard #1 results belong to **R1** ([Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI)).
67
 
68
- R3 results submitted on 2026-10-05:
69
 
70
- | Benchmark | R3 | Leaderboard |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
71
  |:---|:---:|:---|
72
- | **ExtractBench** (370 documents) | **90.29, #1** | [llamaindex/ExtractBench](https://huggingface.co/datasets/llamaindex/ExtractBench) |
73
- | **EvasionBench** (16,726 questions) | **77.83, #3** | [FutureMa/EvasionBench](https://huggingface.co/datasets/FutureMa/EvasionBench) |
 
 
 
 
 
74
 
75
- Measurement settings are recorded in `.eval_results/`. We are also asking independent evaluators to measure R3 directly.
76
 
77
- ## Quickstart
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
78
 
79
- Serving is the same as Darwin-180B-RSI (vLLM, tensor parallel + expert parallel). This is a reasoning model; give it a long generation budget.
80
 
81
  ```bash
82
  vllm serve FINAL-Bench/Darwin-180B-RSI-R3 \
83
  --tensor-parallel-size 8 --enable-expert-parallel \
84
- --max-model-len 139264
 
 
 
 
 
 
 
 
 
 
 
85
  ```
86
 
87
- Recommended sampling: temperature 1.0, top_p 0.95, top_k 20, up to 131,072 generated tokens.
88
 
89
- ## ZTC
 
 
 
 
90
 
91
- The ZTC probe published with Darwin-180B-RSI was fitted on R1's hidden states. A probe fitted on R3 will be added to this repository; until then, use R1's probe only as a rough signal.
92
 
93
- ## Model-level RSI vs. harness-level RSI
 
 
 
 
 
94
 
95
- Darwin-180B-RSI-R3 is **Model-level RSI**: the model itself (its weights) improves by learning only from its own solutions. No human-written solutions or reasoning traces are used; correctness is checked automatically. Harness-level RSI (e.g., Google's RRSI) improves the prompts, tools and workflow around a fixed model. The two are complementary.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
96
 
97
- ## License
98
 
99
- Qwen Community License 1.0 (inherited from Qwen3.8-Flash-Next). See [LICENSE](LICENSE).
100
 
101
- ## About
102
 
103
- Built by [VIDRAFT](https://vidraft.net). Darwin family paper: [arXiv 2605.14386](https://arxiv.org/abs/2605.14386).
 
11
  - model-level-rsi
12
  - recursive-self-improvement
13
  - self-improvement
14
+ - vidraft
15
+ - final-bench
16
+ - qwen
17
+ - qwen3.8
18
  - moe
19
+ - mixture-of-experts
20
+ - sparse-moe
21
+ - 180b
22
+ - hybrid-attention
23
+ - linear-attention
24
+ - long-context
25
+ - 262k-context
26
+ - vision-language
27
+ - multimodal
28
+ - reasoning
29
+ - reasoning-model
30
+ - thinking
31
+ - structured-output
32
+ - document-extraction
33
+ - extractbench
34
+ - evasionbench
35
  - ztc
36
+ - zero-token-confidence
37
+ - eval-results
38
+ - korean
39
+ - english
40
+ - vllm
41
+ - openai-compatible
42
  ---
43
 
44
  # Darwin-180B-RSI-R3
45
 
46
+ ### 180B Mixture-of-Experts · vision-language · **#1 on ExtractBench (90.29)** · the Darwin-180B-RSI line now holds **eight Hugging Face official #1s**: seven by [Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI) (R1) and ExtractBench by R3 · **self-improving**
47
+
48
+ > 🥇 **R3 is #1 on the [ExtractBench](https://huggingface.co/datasets/llamaindex/ExtractBench) leaderboard (90.29)**, ahead of its own parent Qwen3.8-Flash-Next (89.88), and #3 on [EvasionBench](https://huggingface.co/datasets/FutureMa/EvasionBench) (77.83).
49
+
50
+ > 💻 **Run it on your own machine: [POCKET-Darwin-180B-GGUF](https://huggingface.co/FINAL-Bench/POCKET-Darwin-180B-GGUF)**, the 4-bit GGUF of R3 (111 GB), runs on a **laptop with an 8 GB GPU and 32 GB RAM**, **CPU only at 18–21 tok/s**, a 128 GB mini PC or one DGX Spark. MMLU-Pro is **identical to BF16 (87.65%)**.
51
+
52
+ `reasoning` · `MoE 512 experts` · `262K long context` · `image + text` · `Korean + English` · `self-improvement` · `structured output` · `ZTC`
53
+
54
+ <p align="center">
55
+ <a href="https://vidraft.net"><img src="https://img.shields.io/badge/🌐_VIDRAFT-vidraft.net-111827?style=for-the-badge"></a>
56
+ <a href="https://huggingface.co/datasets/llamaindex/ExtractBench"><img src="https://img.shields.io/badge/ExtractBench_(R3)-90.29_%231-059669?style=for-the-badge"></a>
57
+ <a href="https://huggingface.co/datasets/FutureMa/EvasionBench"><img src="https://img.shields.io/badge/EvasionBench_(R3)-77.83_%233-0d9488?style=for-the-badge"></a>
58
+ </p>
59
+ <p align="center">
60
+ <a href="https://huggingface.co/datasets/Idavidrein/gpqa"><img src="https://img.shields.io/badge/GPQA_Diamond_(R1)-94.44%25_%231-gold?style=for-the-badge"></a>
61
+ <a href="https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro"><img src="https://img.shields.io/badge/MMLU--Pro_(R1)-88.12%25_%231-2563eb?style=for-the-badge"></a>
62
+ <a href="https://huggingface.co/datasets/MMMU/MMMU_Pro"><img src="https://img.shields.io/badge/MMMU--Pro_(R1)-79.48%25_%231-0891b2?style=for-the-badge"></a>
63
+ <a href="https://huggingface.co/datasets/MathArena/aime_2026"><img src="https://img.shields.io/badge/AIME_2026_(R1)-100%25_%231-dc2626?style=for-the-badge"></a>
64
+ <a href="https://huggingface.co/datasets/MathArena/hmmt_feb_2026"><img src="https://img.shields.io/badge/HMMT_Feb_2026_(R1)-100%25_%231-ea580c?style=for-the-badge"></a>
65
+ <a href="https://huggingface.co/datasets/LEXam-Benchmark/LEXam"><img src="https://img.shields.io/badge/LEXam_(R1)-68.94%25_%231-4f46e5?style=for-the-badge"></a>
66
+ <a href="https://huggingface.co/datasets/joelniklaus/LEXam-hard"><img src="https://img.shields.io/badge/LEXam--hard_(R1)-45.72_%231-6d28d9?style=for-the-badge"></a>
67
+ </p>
68
+ <p align="center">
69
+ <a href="https://huggingface.co/FINAL-Bench/Darwin-180B-RSI"><img src="https://img.shields.io/badge/Round_1-Darwin--180B--RSI-e11d48?style=for-the-badge"></a>
70
+ <a href="https://arxiv.org/abs/2605.14386"><img src="https://img.shields.io/badge/arXiv-2605.14386_Darwin_Family-b31b1b?style=for-the-badge"></a>
71
+ <a href="https://huggingface.co/papers/2609.20269"><img src="https://img.shields.io/badge/Paper-2609.20269_Latin_Square-b31b1b?style=for-the-badge"></a>
72
+ <a href="https://huggingface.co/collections/FINAL-Bench/darwin-family"><img src="https://img.shields.io/badge/🧬_Collection-Darwin_Family-16a34a?style=for-the-badge"></a>
73
+ <a href="https://huggingface.co/FINAL-Bench/POCKET-Darwin-180B-GGUF"><img src="https://img.shields.io/badge/💻_POCKET_4--bit-Laptop_·_CPU_only_·_DGX_Spark-0f766e?style=for-the-badge"></a>
74
+ </p>
75
+
76
  **The second round of model-level self-improvement on top of [Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI).**
77
+ R3 continues training from R1 with the same recipe: the model solves verifiable problems, keeps only its own solutions that check out as correct, and trains on them. No human-written solutions or reasoning traces are used.
78
+ R3 is released so anyone can download it, run it, and check the numbers below.
79
 
80
+ ---
81
 
82
+ ## 🏆 Eight #1s in the Darwin-180B-RSI line
83
 
84
+ Hugging Face **official** benchmark leaderboards. Each score is listed under the model that produced it.
85
+
86
+ | Benchmark | Score | Model | Leaderboard |
87
+ |:---|:---:|:---|:---|
88
+ | **ExtractBench** (370 documents) | **90.29** | **R3 (this model)** | [**#1**](https://huggingface.co/datasets/llamaindex/ExtractBench) |
89
+ | **GPQA Diamond** (198) | **94.44** | R1 ([Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI)) | [**#1**](https://huggingface.co/datasets/Idavidrein/gpqa) |
90
+ | **MMLU-Pro** (12,032) | **88.12** | R1 | [**#1**](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) |
91
+ | **AIME 2026** (30) | **100.0** | R1 | [**#1**](https://huggingface.co/datasets/MathArena/aime_2026) |
92
+ | **HMMT Feb 2026** (33) | **100.0** | R1 | [**#1**](https://huggingface.co/datasets/MathArena/hmmt_feb_2026) |
93
+ | **MMMU-Pro** (vision, 1,730) | **79.48** | R1 | [**#1**](https://huggingface.co/datasets/MMMU/MMMU_Pro) |
94
+ | **LEXam** (law, MCQ 4-choice, 1,655) | **68.94** | R1 | [**#1**](https://huggingface.co/datasets/LEXam-Benchmark/LEXam) |
95
+ | **LEXam-hard** (law, open-ended, 518) | **45.72** | R1 | [**#1**](https://huggingface.co/datasets/joelniklaus/LEXam-hard) |
96
 
97
+ R3's own leaderboard entries:
98
 
99
+ | Benchmark | R3 | Leaderboard | Setting |
100
+ |:---|:---:|:---|:---|
101
+ | **ExtractBench** (370 documents) | **90.29, #1** | [llamaindex/ExtractBench](https://huggingface.co/datasets/llamaindex/ExtractBench) | official harness, 32,768 max tokens (same as the parent), temperature 0, thinking off, single run |
102
+ | **EvasionBench** (16,726 questions) | **77.83, #3** | [FutureMa/EvasionBench](https://huggingface.co/datasets/FutureMa/EvasionBench) | inspect-ai task from the dataset eval.yaml, temperature 1.0, top_p 0.95, 8,192 max tokens, thinking on, single run |
103
+
104
+ Full settings are recorded in [`.eval_results/`](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI-R3/tree/main/.eval_results).
105
+
106
+ ### R1's seven #1s, head-to-head with Chinese frontier models
107
+
108
+ ![Darwin-180B-RSI vs Chinese frontier models](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI/resolve/main/assets/bench_vs_china.png)
109
+
110
+ | Model | AIME 2026 | GPQA Diamond | MMLU-Pro | MMMU-Pro | HMMT Feb 2026 | LEXam | LEXam-hard |
111
+ |:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
112
+ | **🧬 Darwin-180B-RSI, R1 (ours · 🇰🇷)** | **100** 🥇 | **94.44** 🥇 | **88.12** 🥇 | **79.48** 🥇 | **100** 🥇 | **68.94** 🥇 | **45.72** 🥇 |
113
+ | Inkling (Thinking Machines) | · | · | · | · | · | · | 40.82 |
114
+ | Kimi-K3 (Moonshot AI) | · | 93.5 | · | · | · | · | 29.54 |
115
+ | Kimi-K2.6 (Moonshot AI) | 96.4 | 90.5 | · | 79.4 | 92.7 | · | 36.18 |
116
+ | DeepSeek-V4-Pro (DeepSeek) | · | 90.1 | 87.5 | · | · | · | 38.93 |
117
+ | Qwen3.5-397B-A17B (Alibaba) | 93.33 | 88.4 | 87.8 | · | 87.88 | · | · |
118
+ | MiniMax-M2.1 (MiniMax) | · | 80.81 | 88 | · | · | · | · |
119
+ | GLM-5 (Zhipu AI) | 95.83 | 86 | 86 | · | 86.36 | · | · |
120
+ | Intern-S2-Preview (Shanghai AI Lab) | · | · | 88 | 76.88 | 87.31 | · | · |
121
+ | Step-3.5-Flash (StepFun) | 96.67 | 83.5 | 84.4 | · | 86.36 | · | · |
122
+ | DeepSeek-R1 (DeepSeek) | · | · | · | · | · | 52.41 | · |
123
+ | Qwen3-235B-A22B-Thinking-2507 (Alibaba) | · | · | · | · | · | 48.19 | · |
124
+
125
+ <sub>Scores as listed on the Hugging Face official benchmark leaderboards (self-reported by each model's publisher). "·" = not reported. Open-weight models only; closed API models are not included. Settings (samples, voting, thinking budget) differ across models. R1 settings are in the evaluation protocol below.</sub>
126
+
127
+ ---
128
+
129
+ ## 🧬 What changed from R1
130
 
131
  - **Starting point:** R1 weights (not the parent). R3 is a true second round.
132
  - **Practice problems:** 3,000 SuperGPQA questions (middle and hard difficulty) that were never used in R1 training. R1 solved each one 8 times.
 
134
  - **What was trained:** the same components as R1 (attention paths and shared experts) via LoRA, then merged. All 512 routed experts, the router and the vision encoder are unchanged.
135
  - **No benchmark data:** GPQA Diamond and the held-out set below were never used for training or selection.
136
 
 
 
137
  ### Held-out SuperGPQA, 1,000 questions never used in training or selection
138
  4 samples per question, 16K thinking budget, temperature 1.0.
139
 
 
142
  | R1 (Darwin-180B-RSI) | 65.30 | 65.67 | 68.30 |
143
  | **R3 (this model)** | **66.30** | **66.70** | **69.00** |
144
 
145
+ Paired per-question difference, R1 → R3 (mean of 4): **+1.03 points, 95% CI [+0.05, +2.00]**. The second round of self-improvement produced a measurable gain over R1.
146
 
147
  ### GPQA Diamond, 198 questions
148
  8 samples per question, 32K thinking budget, temperature 1.0.
 
153
  | R1 (Darwin-180B-RSI) | 84.85 | 85.80 | 89.90 |
154
  | **R3 (this model)** | **85.86** | **86.05** | 90.40 |
155
 
156
+ ---
157
+
158
+ ## 🧬 The Darwin Family
159
+
160
+ <p align="center">
161
+ <a href="https://huggingface.co/FINAL-Bench/Darwin-180B-RSI"><img src="https://img.shields.io/badge/Darwin--180B--RSI-7×_%231-e11d48"></a>
162
+ <a href="https://huggingface.co/FINAL-Bench/Darwin-397B-ZTC"><img src="https://img.shields.io/badge/Darwin--397B--ZTC-GPQA_93.43-16a34a"></a>
163
+ <a href="https://huggingface.co/FINAL-Bench/Darwin-28B-REASON"><img src="https://img.shields.io/badge/Darwin--28B--REASON-GPQA_89.39-16a34a"></a>
164
+ <a href="https://huggingface.co/FINAL-Bench/Darwin-35B-A3B-Opus"><img src="https://img.shields.io/badge/Darwin--35B--A3B--Opus-♥98-e11d48"></a>
165
+ <a href="https://huggingface.co/FINAL-Bench/Darwin-36B-Opus"><img src="https://img.shields.io/badge/Darwin--36B--Opus-♥97-e11d48"></a>
166
+ </p>
167
+ <p align="center">
168
+ <a href="https://huggingface.co/FINAL-Bench/Darwin-4B-Genesis"><img src="https://img.shields.io/badge/Darwin--4B--Genesis-♥63-e11d48"></a>
169
+ <a href="https://huggingface.co/FINAL-Bench/Darwin-9B-NEG"><img src="https://img.shields.io/badge/Darwin--9B--NEG-♥57-e11d48"></a>
170
+ <a href="https://huggingface.co/FINAL-Bench/POCKET-35B-GGUF"><img src="https://img.shields.io/badge/POCKET--35B-824K_↓-1f6feb"></a>
171
+ <a href="https://huggingface.co/FINAL-Bench/POCKET-26B-GGUF"><img src="https://img.shields.io/badge/POCKET--26B-365K_↓-1f6feb"></a>
172
+ <a href="https://huggingface.co/FINAL-Bench/POCKET-Darwin-180B-GGUF"><img src="https://img.shields.io/badge/POCKET--Darwin--180B-4--bit_R3_·_laptop-1f6feb"></a>
173
+ </p>
174
+
175
+ **Darwin** is [VIDRAFT](https://vidraft.net)'s measurement-driven reasoning model family: **50+ official models**, **400+ community derivatives**, and two places in the GPQA Diamond top 3 (Darwin-180B-RSI #1 · Darwin-397B-ZTC #3).
176
+
177
+ ### Darwin: evolve the parent, keep what works
178
 
179
+ Darwin treats a strong open model as a **parent**. It measures where the parent is weak and strengthens exactly those parts, instead of re-training everything and risking what already works.
180
 
181
+ - **Diagnose before you change.** Every Darwin generation starts from a measured weakness map of the parent.
182
+ - **Change little, precisely.** Darwin modifies a small, targeted fraction of the network. Knowledge stored in the experts is preserved.
183
+ - **Proven capability over new guesses.** Earlier Darwin generations grafted the best-performing expert/FFN blocks from other strong models onto a base backbone; the RSI line adds a new ingredient: **the model's own verified work**.
184
+ - **Measured, not claimed.** Every change must beat its predecessor on held-out tests before it ships.
185
+
186
+ ### Lineage
187
+
188
+ | Role | | |
189
+ |:---|:---|:---|
190
+ | **R0, parent** | `Qwen/Qwen3.8-Flash-Next` | 180B MoE vision-language backbone · Qwen Community License 1.0 |
191
+ | **R1** | [Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI) | first RSI round: the parent's own verified solutions fed back as training signal · seven #1s |
192
+ | **R3 (this model)** | Darwin-180B-RSI-R3 | second RSI round, trained from R1 on R1's own verified solutions · ExtractBench #1 |
193
+ | **Preserved** | 512 routed experts · router · vision encoder | untouched in every round |
194
+
195
+ ---
196
+
197
+ ## 🔁 RSI: a model that improves from its own work
198
+
199
+ **Recursive self-improvement (RSI)** is the core of this line. Instead of distilling a bigger teacher, the model improves by learning from itself:
200
+
201
+ 1. **Solve:** the model works through practice problems it has never seen in evaluation.
202
+ 2. **Verify:** its answers are checked against verifiable references (answer keys, executable checks). Nothing unverified is learned.
203
+ 3. **Learn:** it is re-trained on the reasoning that turned out to be correct.
204
+ 4. **Repeat:** the improved model becomes the next solver. R1 was round one; R3 is the next round.
205
+
206
+ Practice sets are deduplicated against every evaluation set we report (8-gram overlap filter).
207
+
208
+ ### Model-level RSI vs. harness-level RSI
209
+
210
+ Darwin-180B-RSI-R3 is **Model-level RSI**: the model itself (its weights) improves by learning only from its own solutions. No human-written solutions or reasoning traces are used; correctness is checked automatically. **Harness-level RSI** (e.g., Google's RRSI) improves the prompts, tools and workflow around a fixed model. It is like rewriting an employee's manual, while Model-level RSI is the employee getting smarter. The two are complementary.
211
+
212
+ ---
213
 
214
+ ## 🏛️ ZTC: it knows before it answers
215
 
216
+ **Zero-Token Confidence (ZTC)** reads the model's own internal state **once, before generation**, and returns the probability that the answer it is about to give is correct, with **no extra tokens and no second model.**
217
 
218
+ ```json
219
+ {"answer": "...", "confidence": 0.93, "ztc_score": 1.84, "truncated": false}
220
+ ```
221
+
222
+ The ZTC probe published with [Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI) was fitted on R1's hidden states. A probe fitted on R3 will be added to this repository; until then, use R1's probe only as a rough signal.
223
+
224
+ ---
225
+
226
+ ## 📐 Evaluation protocol
227
+
228
+ **R3 entries (ExtractBench, EvasionBench):** see the settings column in the table above and [`.eval_results/`](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI-R3/tree/main/.eval_results). ExtractBench was run with thinking turned off (`chat_template_kwargs: {"enable_thinking": false}`), which suits schema-guided extraction.
229
+
230
+ **R1 entries (the seven #1s):**
231
+
232
+ | Setting | Value |
233
+ |:---|:---|
234
+ | Thinking budget | **131,072 tokens** (32,768 for LEXam and LEXam-hard) |
235
+ | Sampling | temperature 1.0 · top_p 0.95 · top_k 20 |
236
+ | Precision | bf16 |
237
+ | Engine | vLLM, tensor parallel 8 (or 4), expert parallel |
238
+
239
+ | Benchmark | Samples per question | Reported score |
240
  |:---|:---:|:---|
241
+ | AIME 2026 | 16 | majority vote (maj@16); mean over 16 = 98.75 |
242
+ | HMMT Feb 2026 | 16 | majority vote (maj@16); mean over 16 = 96.59 |
243
+ | GPQA Diamond | up to 16 | majority vote |
244
+ | MMLU-Pro | 1 | single sample (no voting) |
245
+ | MMMU-Pro (vision) | 3 | majority vote (maj@3) |
246
+ | LEXam | 4 | majority vote (single sample 60.54 · mean 61.42) |
247
+ | LEXam-hard | 1 | single sample, judged by DeepSeek-R1-0528 per the official eval.yaml |
248
 
249
+ All numbers are self-measured and reproducible with the settings above. Majority-vote scores are system scores (several samples per question) and are labeled as such.
250
 
251
+ ---
252
+
253
+ ## ⚙️ Specifications
254
+
255
+ | | |
256
+ |:---|:---|
257
+ | Architecture | Mixture-of-Experts, hybrid attention (36 linear-attention + 12 full-attention layers) |
258
+ | Layers / hidden | 48 / 2,560 |
259
+ | Experts | 512 routed (10 active per token) + shared expert |
260
+ | Context | 262,144 tokens |
261
+ | Vocabulary | 248,320 |
262
+ | Modalities | image + text → text |
263
+ | Precision | bf16 (~336 GB) |
264
+
265
+ ---
266
+
267
+ ## 🚀 Quickstart
268
 
269
+ ### Serving with vLLM (8 × B200 or equivalent)
270
 
271
  ```bash
272
  vllm serve FINAL-Bench/Darwin-180B-RSI-R3 \
273
  --tensor-parallel-size 8 --enable-expert-parallel \
274
+ --max-model-len 139264 --trust-remote-code
275
+ ```
276
+
277
+ ### Chat Completions (OpenAI-compatible)
278
+
279
+ ```python
280
+ from openai import OpenAI
281
+ c = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
282
+ r = c.chat.completions.create(model="FINAL-Bench/Darwin-180B-RSI-R3",
283
+ messages=[{"role": "user", "content": "Explain why the sky is blue in two sentences."}],
284
+ temperature=1.0, top_p=0.95, extra_body={"top_k": 20})
285
+ print(r.choices[0].message.content)
286
  ```
287
 
288
+ For structured extraction (JSON to a schema), turn thinking off, as in the ExtractBench run:
289
 
290
+ ```python
291
+ r = c.chat.completions.create(model="FINAL-Bench/Darwin-180B-RSI-R3", messages=msgs, temperature=0,
292
+ response_format={"type": "json_object"},
293
+ extra_body={"chat_template_kwargs": {"enable_thinking": False}})
294
+ ```
295
 
296
+ ### Transformers
297
 
298
+ ```python
299
+ from transformers import AutoProcessor, AutoModelForImageTextToText
300
+ model_id = "FINAL-Bench/Darwin-180B-RSI-R3"
301
+ processor = AutoProcessor.from_pretrained(model_id)
302
+ model = AutoModelForImageTextToText.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
303
+ ```
304
 
305
+ **Tip:** for hard reasoning, keep thinking on and give it room (a thinking budget of 32K to 131K tokens). For document extraction, thinking off was faster and scored higher in our ExtractBench runs.
306
+
307
+ ---
308
+
309
+ ## ⚠️ Limitations and disclosure
310
+
311
+ - Scores are self-measured with the settings stated above; majority-vote numbers use several samples per question.
312
+ - Very long reasoning is normal for hard problems; a short thinking budget will truncate answers and lower accuracy.
313
+ - Like every LLM, the model can be confidently wrong. Use a confidence readout such as ZTC to gate high-stakes actions.
314
+
315
+ ---
316
+
317
+ ## 🔗 Related Darwin Models
318
+
319
+ - **[Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI)**: R1, the model R3 was trained from, #1 on seven Hugging Face official leaderboards
320
+ - **[POCKET-Darwin-180B-GGUF](https://huggingface.co/FINAL-Bench/POCKET-Darwin-180B-GGUF)**: 4-bit GGUF of R3 for laptops, CPU-only machines and DGX Spark
321
+ - **[Darwin-397B-ZTC](https://huggingface.co/FINAL-Bench/Darwin-397B-ZTC)**: 397B MoE (FP8), GPQA Diamond 93.43 %, ZTC on board
322
+ - **[Darwin-28B-REASON](https://huggingface.co/FINAL-Bench/Darwin-28B-REASON)**: 28B, GPQA Diamond 89.39 %
323
+ - **[Darwin-27B-RSI](https://huggingface.co/FINAL-Bench/Darwin-27B-RSI)**: 27B, the first Darwin RSI model
324
+ - **[ZTC-Judge-27B](https://huggingface.co/FINAL-Bench/ZTC-Judge-27B)**: standalone ZTC judge
325
+
326
+ ---
327
+
328
+ ## 📚 Citation
329
+
330
+ ```bibtex
331
+ @misc{darwin180b_rsi_r3_2026,
332
+ title = {Darwin-180B-RSI-R3: A Second Round of Model-Level Self-Improvement for a 180B Mixture-of-Experts Reasoning Model},
333
+ author = {FINAL-Bench / Darwin Research Team},
334
+ year = {2026},
335
+ howpublished = {\url{https://huggingface.co/FINAL-Bench/Darwin-180B-RSI-R3}},
336
+ note = {ExtractBench 90.29}
337
+ }
338
+
339
+ @misc{darwin180b_rsi_2026,
340
+ title = {Darwin-180B-RSI: Recursive Self-Improvement on Verified Answers for a 180B Mixture-of-Experts Reasoning Model},
341
+ author = {FINAL-Bench / Darwin Research Team},
342
+ year = {2026},
343
+ howpublished = {\url{https://huggingface.co/FINAL-Bench/Darwin-180B-RSI}},
344
+ note = {GPQA Diamond 94.44 \% · MMLU-Pro 88.12 \%}
345
+ }
346
+
347
+ @misc{darwin_family_2026,
348
+ title = {Darwin Family: MRI-Trust-Weighted Evolutionary Merging for Training-Free Scaling of Language-Model Reasoning},
349
+ author = {Kim, Taebong and Hong, Youngsik and Kim, Minsik and Choi, Sunyoung and Jang, Jaewon and Shin, Junghoon},
350
+ year = {2026},
351
+ eprint = {2605.14386},
352
+ archivePrefix = {arXiv}
353
+ }
354
+ ```
355
+
356
+ ---
357
 
358
+ ## 📜 License
359
 
360
+ Darwin-180B-RSI-R3 is a derivative of **Qwen3.8-Flash-Next** (through Darwin-180B-RSI) and is distributed under the **Qwen Community License 1.0** (see [LICENSE](LICENSE)).
361
 
362
+ ## 🏢 About
363
 
364
+ Built by **[VIDRAFT](https://vidraft.net)** · evaluated with **FINAL-Bench**. Part of the [Darwin Family](https://arxiv.org/abs/2605.14386).