SeaWolf-AI commited on
Commit
102dd91
·
verified ·
1 Parent(s): 5ee02b9

card: nine #1s in the RSI line (IFStruct by R1)

Browse files
Files changed (1) hide show
  1. README.md +10 -7
README.md CHANGED
@@ -43,7 +43,7 @@ tags:
43
 
44
  # Darwin-180B-RSI-R3
45
 
46
- ### 180B Mixture-of-Experts · vision-language · **#1 on ExtractBench (90.29)** · the Darwin-180B-RSI line now holds **eight Hugging Face official #1s**: seven by [Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI) (R1) and ExtractBench by R3 · **self-improving**
47
 
48
  > 🥇 **R3 is #1 on the [ExtractBench](https://huggingface.co/datasets/llamaindex/ExtractBench) leaderboard (90.29)**, ahead of its own parent Qwen3.8-Flash-Next (89.88), and #3 on [EvasionBench](https://huggingface.co/datasets/FutureMa/EvasionBench) (77.83).
49
 
@@ -64,6 +64,7 @@ tags:
64
  <a href="https://huggingface.co/datasets/MathArena/hmmt_feb_2026"><img src="https://img.shields.io/badge/HMMT_Feb_2026_(R1)-100%25_%231-ea580c?style=for-the-badge"></a>
65
  <a href="https://huggingface.co/datasets/LEXam-Benchmark/LEXam"><img src="https://img.shields.io/badge/LEXam_(R1)-68.94%25_%231-4f46e5?style=for-the-badge"></a>
66
  <a href="https://huggingface.co/datasets/joelniklaus/LEXam-hard"><img src="https://img.shields.io/badge/LEXam--hard_(R1)-45.72_%231-6d28d9?style=for-the-badge"></a>
 
67
  </p>
68
  <p align="center">
69
  <a href="https://huggingface.co/FINAL-Bench/Darwin-180B-RSI"><img src="https://img.shields.io/badge/Round_1-Darwin--180B--RSI-e11d48?style=for-the-badge"></a>
@@ -79,7 +80,7 @@ R3 is released so anyone can download it, run it, and check the numbers below.
79
 
80
  ---
81
 
82
- ## 🏆 Eight #1s in the Darwin-180B-RSI line
83
 
84
  Hugging Face **official** benchmark leaderboards. Each score is listed under the model that produced it.
85
 
@@ -93,6 +94,7 @@ Hugging Face **official** benchmark leaderboards. Each score is listed under the
93
  | **MMMU-Pro** (vision, 1,730) | **79.48** | R1 | [**#1**](https://huggingface.co/datasets/MMMU/MMMU_Pro) |
94
  | **LEXam** (law, MCQ 4-choice, 1,655) | **68.94** | R1 | [**#1**](https://huggingface.co/datasets/LEXam-Benchmark/LEXam) |
95
  | **LEXam-hard** (law, open-ended, 518) | **45.72** | R1 | [**#1**](https://huggingface.co/datasets/joelniklaus/LEXam-hard) |
 
96
 
97
  R3's own leaderboard entries:
98
 
@@ -103,7 +105,7 @@ R3's own leaderboard entries:
103
 
104
  Full settings are recorded in [`.eval_results/`](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI-R3/tree/main/.eval_results).
105
 
106
- ### R1's seven #1s, head-to-head with Chinese frontier models
107
 
108
  ![Darwin-180B-RSI vs Chinese frontier models](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI/resolve/main/assets/bench_vs_china.png)
109
 
@@ -158,7 +160,7 @@ Paired per-question difference, R1 → R3 (mean of 4): **+1.03 points, 95% CI [+
158
  ## 🧬 The Darwin Family
159
 
160
  <p align="center">
161
- <a href="https://huggingface.co/FINAL-Bench/Darwin-180B-RSI"><img src="https://img.shields.io/badge/Darwin--180B--RSI-7×_%231-e11d48"></a>
162
  <a href="https://huggingface.co/FINAL-Bench/Darwin-397B-ZTC"><img src="https://img.shields.io/badge/Darwin--397B--ZTC-GPQA_93.43-16a34a"></a>
163
  <a href="https://huggingface.co/FINAL-Bench/Darwin-28B-REASON"><img src="https://img.shields.io/badge/Darwin--28B--REASON-GPQA_89.39-16a34a"></a>
164
  <a href="https://huggingface.co/FINAL-Bench/Darwin-35B-A3B-Opus"><img src="https://img.shields.io/badge/Darwin--35B--A3B--Opus-♥98-e11d48"></a>
@@ -188,7 +190,7 @@ Darwin treats a strong open model as a **parent**. It measures where the parent
188
  | Role | | |
189
  |:---|:---|:---|
190
  | **R0, parent** | `Qwen/Qwen3.8-Flash-Next` | 180B MoE vision-language backbone · Qwen Community License 1.0 |
191
- | **R1** | [Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI) | first RSI round: the parent's own verified solutions fed back as training signal · seven #1s |
192
  | **R3 (this model)** | Darwin-180B-RSI-R3 | second RSI round, trained from R1 on R1's own verified solutions · ExtractBench #1 |
193
  | **Preserved** | 512 routed experts · router · vision encoder | untouched in every round |
194
 
@@ -227,7 +229,7 @@ The ZTC probe published with [Darwin-180B-RSI](https://huggingface.co/FINAL-Benc
227
 
228
  **R3 entries (ExtractBench, EvasionBench):** see the settings column in the table above and [`.eval_results/`](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI-R3/tree/main/.eval_results). ExtractBench was run with thinking turned off (`chat_template_kwargs: {"enable_thinking": false}`), which suits schema-guided extraction.
229
 
230
- **R1 entries (the seven #1s):**
231
 
232
  | Setting | Value |
233
  |:---|:---|
@@ -245,6 +247,7 @@ The ZTC probe published with [Darwin-180B-RSI](https://huggingface.co/FINAL-Benc
245
  | MMMU-Pro (vision) | 3 | majority vote (maj@3) |
246
  | LEXam | 4 | majority vote (single sample 60.54 · mean 61.42) |
247
  | LEXam-hard | 1 | single sample, judged by DeepSeek-R1-0528 per the official eval.yaml |
 
248
 
249
  All numbers are self-measured and reproducible with the settings above. Majority-vote scores are system scores (several samples per question) and are labeled as such.
250
 
@@ -316,7 +319,7 @@ model = AutoModelForImageTextToText.from_pretrained(model_id, torch_dtype="auto"
316
 
317
  ## 🔗 Related Darwin Models
318
 
319
- - **[Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI)**: R1, the model R3 was trained from, #1 on seven Hugging Face official leaderboards
320
  - **[POCKET-Darwin-180B-GGUF](https://huggingface.co/FINAL-Bench/POCKET-Darwin-180B-GGUF)**: 4-bit GGUF of R3 for laptops, CPU-only machines and DGX Spark
321
  - **[Darwin-397B-ZTC](https://huggingface.co/FINAL-Bench/Darwin-397B-ZTC)**: 397B MoE (FP8), GPQA Diamond 93.43 %, ZTC on board
322
  - **[Darwin-28B-REASON](https://huggingface.co/FINAL-Bench/Darwin-28B-REASON)**: 28B, GPQA Diamond 89.39 %
 
43
 
44
  # Darwin-180B-RSI-R3
45
 
46
+ ### 180B Mixture-of-Experts · vision-language · **#1 on ExtractBench (90.29)** · the Darwin-180B-RSI line now holds **nine Hugging Face official #1s**: eight by [Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI) (R1) and ExtractBench by R3 · **self-improving**
47
 
48
  > 🥇 **R3 is #1 on the [ExtractBench](https://huggingface.co/datasets/llamaindex/ExtractBench) leaderboard (90.29)**, ahead of its own parent Qwen3.8-Flash-Next (89.88), and #3 on [EvasionBench](https://huggingface.co/datasets/FutureMa/EvasionBench) (77.83).
49
 
 
64
  <a href="https://huggingface.co/datasets/MathArena/hmmt_feb_2026"><img src="https://img.shields.io/badge/HMMT_Feb_2026_(R1)-100%25_%231-ea580c?style=for-the-badge"></a>
65
  <a href="https://huggingface.co/datasets/LEXam-Benchmark/LEXam"><img src="https://img.shields.io/badge/LEXam_(R1)-68.94%25_%231-4f46e5?style=for-the-badge"></a>
66
  <a href="https://huggingface.co/datasets/joelniklaus/LEXam-hard"><img src="https://img.shields.io/badge/LEXam--hard_(R1)-45.72_%231-6d28d9?style=for-the-badge"></a>
67
+ <a href="https://huggingface.co/datasets/LiquidAI/ifstruct-v1.0"><img src="https://img.shields.io/badge/IFStruct_(R1)-98.95%25_%231-b45309?style=for-the-badge"></a>
68
  </p>
69
  <p align="center">
70
  <a href="https://huggingface.co/FINAL-Bench/Darwin-180B-RSI"><img src="https://img.shields.io/badge/Round_1-Darwin--180B--RSI-e11d48?style=for-the-badge"></a>
 
80
 
81
  ---
82
 
83
+ ## 🏆 Nine #1s in the Darwin-180B-RSI line
84
 
85
  Hugging Face **official** benchmark leaderboards. Each score is listed under the model that produced it.
86
 
 
94
  | **MMMU-Pro** (vision, 1,730) | **79.48** | R1 | [**#1**](https://huggingface.co/datasets/MMMU/MMMU_Pro) |
95
  | **LEXam** (law, MCQ 4-choice, 1,655) | **68.94** | R1 | [**#1**](https://huggingface.co/datasets/LEXam-Benchmark/LEXam) |
96
  | **LEXam-hard** (law, open-ended, 518) | **45.72** | R1 | [**#1**](https://huggingface.co/datasets/joelniklaus/LEXam-hard) |
97
+ | **IFStruct** (structured output, 2,000 prompts) | **98.95** | R1 | [**#1**](https://huggingface.co/datasets/LiquidAI/ifstruct-v1.0) |
98
 
99
  R3's own leaderboard entries:
100
 
 
105
 
106
  Full settings are recorded in [`.eval_results/`](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI-R3/tree/main/.eval_results).
107
 
108
+ ### R1 head-to-head with Chinese frontier models
109
 
110
  ![Darwin-180B-RSI vs Chinese frontier models](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI/resolve/main/assets/bench_vs_china.png)
111
 
 
160
  ## 🧬 The Darwin Family
161
 
162
  <p align="center">
163
+ <a href="https://huggingface.co/FINAL-Bench/Darwin-180B-RSI"><img src="https://img.shields.io/badge/Darwin--180B--RSI-8×_%231-e11d48"></a>
164
  <a href="https://huggingface.co/FINAL-Bench/Darwin-397B-ZTC"><img src="https://img.shields.io/badge/Darwin--397B--ZTC-GPQA_93.43-16a34a"></a>
165
  <a href="https://huggingface.co/FINAL-Bench/Darwin-28B-REASON"><img src="https://img.shields.io/badge/Darwin--28B--REASON-GPQA_89.39-16a34a"></a>
166
  <a href="https://huggingface.co/FINAL-Bench/Darwin-35B-A3B-Opus"><img src="https://img.shields.io/badge/Darwin--35B--A3B--Opus-♥98-e11d48"></a>
 
190
  | Role | | |
191
  |:---|:---|:---|
192
  | **R0, parent** | `Qwen/Qwen3.8-Flash-Next` | 180B MoE vision-language backbone · Qwen Community License 1.0 |
193
+ | **R1** | [Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI) | first RSI round: the parent's own verified solutions fed back as training signal · eight #1s |
194
  | **R3 (this model)** | Darwin-180B-RSI-R3 | second RSI round, trained from R1 on R1's own verified solutions · ExtractBench #1 |
195
  | **Preserved** | 512 routed experts · router · vision encoder | untouched in every round |
196
 
 
229
 
230
  **R3 entries (ExtractBench, EvasionBench):** see the settings column in the table above and [`.eval_results/`](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI-R3/tree/main/.eval_results). ExtractBench was run with thinking turned off (`chat_template_kwargs: {"enable_thinking": false}`), which suits schema-guided extraction.
231
 
232
+ **R1 entries (its #1s):**
233
 
234
  | Setting | Value |
235
  |:---|:---|
 
247
  | MMMU-Pro (vision) | 3 | majority vote (maj@3) |
248
  | LEXam | 4 | majority vote (single sample 60.54 · mean 61.42) |
249
  | LEXam-hard | 1 | single sample, judged by DeepSeek-R1-0528 per the official eval.yaml |
250
+ | IFStruct | 1 | single run, temperature 0, 16,000 max tokens (official harness defaults) |
251
 
252
  All numbers are self-measured and reproducible with the settings above. Majority-vote scores are system scores (several samples per question) and are labeled as such.
253
 
 
319
 
320
  ## 🔗 Related Darwin Models
321
 
322
+ - **[Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI)**: R1, the model R3 was trained from, #1 on eight Hugging Face official leaderboards
323
  - **[POCKET-Darwin-180B-GGUF](https://huggingface.co/FINAL-Bench/POCKET-Darwin-180B-GGUF)**: 4-bit GGUF of R3 for laptops, CPU-only machines and DGX Spark
324
  - **[Darwin-397B-ZTC](https://huggingface.co/FINAL-Bench/Darwin-397B-ZTC)**: 397B MoE (FP8), GPQA Diamond 93.43 %, ZTC on board
325
  - **[Darwin-28B-REASON](https://huggingface.co/FINAL-Bench/Darwin-28B-REASON)**: 28B, GPQA Diamond 89.39 %