Update README.md

README.md

@@ -35,56 +35,91 @@ The end result is a model that is especially capable at reasoning and agentic ta
 US open-weight models with more than 10x the number of total parameters.
 At only 65B total parameters, the model can fit on consumer hardware such as the NVIDIA DGX Spark. 
 To encourage local model development, we provide an NVFP4 checkpoint and colab to help you get started. 
-The model is also fast at token generation when running on datacenter GPUs, easily reaching 250+ tok/s with DFlash2 on 4xB200 prior to quantization. 
+The model is also fast at token generation when running on datacenter GPUs, easily reaching 250+ tok/s with DFlash2 on 4xB200 (batch=16) prior to quantization. 
 
 **Pre-training.** Ripple-65B Preview is pretrained from scratch on a 12T text-only corpus at 8K sequence length. 
 Ripple-65B Base Preview is a **pretrain-only checkpoint** without any midtraining or synthetic QA data.
 
 **Post-training.** We use a standard post-training recipe consisting of midtraining, long context, SFT, and mixed domain RL. 
 
-## Instruct Model
+## Instruct Model Benchmarks
 
+We compare Ripple-65B Preview against currently-available US open-weight frontier models.
+All models are tested at full BF16 precision with maximum effort levels.
+
+Each lab typically only provides a limited subset of available benchmarks.
+Our table below shows official model card scores where available, and we use our own in-house eval flow for missing evals (italized).
+In cases where the harness is not specified, we use OpenCode.
+A higher number is better for all benchmarks.
+
+Replicating benchmark scores has been challenging, particularly for modern agentic evals, due to the number of factors which can cause 
+eval scores to shift. 
+For example, we found that Terminal-Bench 2.1 scores were universally lower (with the exception of Muse Spark Open) 
+when using the official Terminus 2 harness at 128K sequence length with summarization.
+Therefore, our results represent a best-effort attempt at a fair comparison.
 
 <div align="center">
 
-| Benchmark (Metric) | Opus-5.0 | GPT-5.6 Sol | K3 | GLM-5.3 | DS-V4-Pro | DS-V4-Flash | DS-V4.1-Flash |
-| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
-| **Reasoning** | | | | | | | |
-| GPQA Diamond (Pass@1) | 93.4 | **94.1** | 92.9 | 88.1 | 92.4 | 89.9 | 90.9 |
-| HLE (Pass@1) | **56.3** | 44.5 | 43.5 | 42.0† | 42.7† | 37.8† | 36.8 (39.1†) |
-| Codeforces (Rating) | — | — | — | — | 3348 | 3289 | **3471** |
-| MathArena Apex (Pass@1) | — | — | **65.6** | — | 65.3 | 58.6 | **65.6** |
-| **Agentic** | | | | | | | |
-| Terminal-Bench 2.1 (Pass@1) | 89.1 | 88.8 | 88.3 | 88.2 | 87.9 | 82.7 | **90.6** |
-| Terminal-Bench 3.0 (Pass@1) | **43.3** | 34.4 | 17.7 | 28.3 | 11.8 | 7.6 | 30.0 |
-| Terminal-Bench 4.0 (Pass@1) | **51.8** | 39.9 | 12.6 | 37.9 | 12.4 | 7.0 | 31.2 |
-| DeepSWE v1.1 (Resolved) | 74.0 | 73.0 | 67.5 | 66.9 | 62.7 | 54.4 | **74.2** |
-| ProgramBench (Almost@1) | **37.0** | 23.0 | 17.5 | 19.0 | 15.5 | — | 20.3 |
-| NL2Repo-Bench (Score) | **75.3** | 56.8 | 58.0 | 58.0 | 61.5 | 54.2 | 64.0 |
-| CyberGym (Pass@1) | — | 84.5 | 80.0 | 84.5 | 83.3 | 76.7 | **88.1** |
-| SEC-Bench Pro (Pass@1) | — | **74.3** | — | — | 56.4 | 30.9 | 62.8 |
-| ExploitGym (Pass@1) | 22.1 | **33.7** | — | 15.0 | 5.4 | 1.8 | 15.3 |
-| HLE w/ tools (Pass@1) | 63.6 | — | 59.8 | 62.5 | 60.0 | 51.5 | **63.9** |
-| AutomationBench (Pass@1) | 50.3 | 45.8 | 46.7 | 48.8 | 43.2 | 37.7 | **54.8** |
-| Agent's Last Exam (Pass@1) | 28.6 | 26.7 | 27.6 | 28.5 | 25.7 | 25.2 | **31.8** |
-| Chartography w/ tools (Pass@1) | **84.0** | 79.9 | 68.1 | — | — | — | 78.9 |
-| BabyVision w/ tools (Pass@1) | **94.1** | 88.9 | 85.7 | — | — | — | 89.6 |
-| ZeroBench-main w/ tools (Pass@5) | 52.0 | **53.0** | 41.0 | — | — | — | 49.0 |
+  | Benchmark | Ripple-65B<br>Preview | Nemotron 3<br>Ultra | Inkling | Muse Spark<br>Open | Reflection<br>Beam |
+  |---|---|---|---|---|---|
+  | **Model** | | | | | |
+  | Architecture | Dense | MoE | MoE | MoE | MoE |
+  | Total params | 65B | 550B | 975B | 816B | 501B |
+  | Active params | 65B | 55B | 41B | 42B | 23B |
+  | Pre-training tokens | 12T | 20T | 45T | N/A | 23.8T |
+  | Training FLOPs (6ND) | 4.7e24 | 6.6e24 | 1.1e25 | N/A | 3.3e24 |
+  | **Knowledge** | | | | | |
+  | MMLU | — | *92.0* | *91.3* | *92.6* | *92.4* |
+  | MMLU-Pro | — | 86.8 | *87.3* | *87.95* | *76.5* |
+  | GPQA Diamond | — | 87.0 | 87.2[^h] | 89.0 | **90.6** |
+  | **Math** | | | | | |
+  | AIME 2024 | — | *94.17* | *97.5* | *97.92* | *96.67* | 
+  | AIME 2025 | — | *94.17* | *91.67* | *100.0* | *99.58* |
+  | AIME 2026 | — | *95.42* | 97.1 | *98.33* | **97.7** |
+  | HMMT Feb 2026 | — | *80.3* | *78.79* | *93.94* | *93.94* | 
+  | **Coding & Agentic** | | | | | |
+  | Terminal-Bench 2.1 (reported) | — | 56.4 | 63.8 | 76.4 | **80.1** |
+  | Terminal-Bench 2.1 (Terminus 2) | — | *51.31* | *55.43* | *79.78* | *67.79* |
+  | SWE-bench Verified | — | 70.7[^b] | 77.6[^f] | *78.67* | **81.6**[^n] |
+  | SWE-bench Pro v1.0 | — | *46.92* | 54.3[^g] | — | **66.1** |
+  | SWE-bench Pro v2.0 Hard | — | *30.72* | *1.96* | *52.29* | 76.5 |
+  | DeepSWE 1.1 | — | *0.22* | *1.96* | **60**[^j] | 43.3 |
+  | SciCode | — | 44.6[^c] | *39.18* | **58.5** | 49.2 | 
+  | τ³-bench banking | — | 22.6 | 23.7[^h] | *14.43* | **36.1** |
+  | AutomationBench (Public) | — | *8.39* | *14.56* | *42.44* | 39.3 | 
+  | MCP Atlas | — | *62.2* | 76.0[^i] | **89.2** | 80.8 |
+  | **Safety** | | | | | |
+  | FORTRESS Adversarial | — | *65.80* | 78.0 | *89.07* | *81.33* |
+  | FORTRESS Benign | — | *89.13* | 95.9 | *88.13* | *86.40* |
+  | StrongREJECT (direct prompts) | — | *100.0* | 98.6 | *96.59* | *100.0* |
+  | **Instruction Following**  | | | | | |
+  | IFBench | — | **81.7**[^d] | 79.8 | *78.56* | 79.6 |
+  | **Long Context**  | | | | | |
+  | AA-LCR | — | 65.4 | *78.33* | *80.67* | **78.2** |
 
 </div>
 
 
-## Base Model
+## Base Model Benchmarks
+
+We show traditional pretraining evals comparing Ripple-65B Base Preview against a selection of open-weight base models. Although we would have liked to 
+compare against the same set of models throughout the model card, many companies are now choosing to only open-source their instruction-tuned models.
+
+With the rise of the importance of posttraining, many modern LLM recipes now blend posttraining data earlier into the training process, including during pretraining.
+For our first model, we kept our pretraining data clean with minimal synthetic data, focusing on high quality natural data rather than aggressively targeting evals.
+We instead added significantly more synthetic data during midtraining, which allowed us to quickly iterate on our synthetic data recipe without blocking pretraining.
 
-Ripple-65B Base Preview is a pretrain-only checkpoint and deliberately does not include synthetic QA data.
-Nemotron 3 Ultra Base uses a two-phase pretraining data mix approach in which both phases include substantial synthetic data, including QA and SFT data.
-We show official eval scores for all provided evals; for missing evals, we run the evals ourselves and show the results in italics.
+Ripple-65B Base Preview is a pretrain-only checkpoint. The model overall compares favorably with Laguna M.1, and is comparable with DeepSeek V3.1 and V4 Flash on 
+reasoning, math, and code.
+Nemotron 3 Ultra dominates on traditional pretraining evals, likely due to their heavy use of [task-seeded synthetic Q&A data](https://huggingface.co/blog/nvidia/task-seeded-sdg).
+However, its pretraining eval performance did not translate to modern agentic tasks, likely due to Q&A being out of distribution relative to these tasks.
 
-Evaluation is performed on an internal fork of [Olmes](https://github.com/allenai/olmes).
+We report official model results where available.
+For our own model and for missing evals, we use our internal fork of [Olmes](https://github.com/allenai/olmes).
 
 <div align="center">
 
-  | Benchmark (Metric) | # Shots | Ripple-65B | Laguna-M.1 | DeepSeek V3.1 Base | DeepSeek V4 Flash Base | Nemotron 3 Ultra |
+  | Benchmark (Metric) | # Shots | Ripple-65B<br>Preview | Laguna<br>M.1 | DeepSeek<br>V3.1 | DeepSeek<br>V4 Flash | Nemotron 3<br>Ultra |
   |---|---|---|---|---|---|---|
   | **Model** | | | | | | |
   | Architecture | — | Dense | MoE | MoE | MoE | MoE |
@@ -108,13 +143,12 @@ Evaluation is performed on an internal fork of [Olmes](https://github.com/allena
   | HellaSwag (acc_norm) | 0-shot | *84.6* | *86.2* | *86.8* | 85.7 | ***89.1*** |
   | HellaSwag (acc_norm) | 10-shot | *87.2* | *88.1* | *89.4* | *88.8* | **90.5** |
   | ARC-Challenge (MC acc) | 25-shot | *95.7* | *90.8* | *95.1* | *95.8* | **97.4**[^d] |
-  | ARC-Challenge (acc_norm) | 25-shot | *72.3* | *69.0* | *71.8* | *73.5* | ***74.1*** |
   | OpenBookQA (acc_norm) | 0-shot | *47.4* | *48.6* | ***50.4*** | *48.0* | 48.6 |
   | PIQA (acc_norm) | 0-shot | *83.6* | *84.8* | ***85.2*** | *84.7* | 83.8 |
   | WinoGrande (acc) | 5-shot | ***83.4*** | *83.2* | *83.2* | *81.4*[^a] | 79.3 |
   | **Code & Math** | | | | | | |
   | HumanEval (pass@1, n=20, T=0.2) | 0-shot | *70.9* | *83.4* | *69.8* | 69.5 | **83.8**[^b] |
-  | MBPP sanitized-378 (pass@1, n=32, T=0.2) | 3-shot | *81.5* | *81.1* | *75.2* | *74.6* | **86.0**[^c] |
+  | MBPP Sanitized (pass@1, n=32, T=0.2) | 3-shot | *81.5* | *81.1* | *75.2* | *74.6* | **86.0**[^c] |
   | BigCodeBench (pass@1, n=5, T=0.8) | 3-shot | *49.2* | *50.5* | ***59.9*** | 56.8 | *55.6* |
   | GSM8K (CoT EM) | 8-shot | *88.5* | *80.4* | *79.6*[^e] | **90.8** | 88.1 |
   | GSM1K (CoT EM) | 8-shot | *88.4* | *79.7* | *81.8*[^e] | *83.2* | ***90.2*** |