Update README.md

README.md

@@ -12,10 +12,10 @@
 </p>
 
 We introduce **Ripple-65B Preview**, a 65B parameter dense Transformer model pretrained on 12 trillion tokens that scores [PLACEHOLDER] on TerminalBench 2.1,
-surpassing significantly-larger US open-weight models such as Muse Spark Open 816B (76.4), Inkling-975B (63.8), and Nemotron 3 Ultra 550B (56.4). 
+surpassing significantly-larger US open-weight models such as Reflection Beam 501B (80.1), Muse Spark Open 816B (76.4), Inkling-975B (63.8), and Nemotron 3 Ultra 550B (56.4). 
 
 Starting from an initial Olmo 3 baseline, we iteratively improved all aspects of the training recipe, including the pretraining scaling laws, model architecture, and data.
-Our final Ripple model ladder improves Compute Efficiency (CE) on code by **16x** relative to the Olmo 3 ladder, with the advantage further increasing with scale up to 
+Our final Ripple v1 model ladder improves Compute Efficiency (CE) on code by **16x** relative to the Olmo 3 ladder, with the advantage further increasing with scale up to 
 **55x** for the full hero run. 
 
 Ripple-65B was codesigned to be trained and served efficiently on NVIDIA hardware.
@@ -38,6 +38,7 @@ To encourage local model development, we provide an NVFP4 checkpoint and colab t
 The model is also fast at token generation when running on datacenter GPUs, easily reaching 250+ tok/s with DFlash2 on 4xB200 prior to quantization. 
 
 **Pre-training.** Ripple-65B Preview is pretrained from scratch on a 12T text-only corpus at 8K sequence length. 
+Ripple-65B Base Preview is a **pretrain-only checkpoint** without any midtraining or synthetic QA data.
 
 **Post-training.** We use a standard post-training recipe consisting of midtraining, long context, SFT, and mixed domain RL. 
 
@@ -75,40 +76,51 @@ The model is also fast at token generation when running on datacenter GPUs, easi
 
 ## Base Model
 
-We use an internal fork of [Olmes](https://github.com/allenai/olmes) for our evals.
+Ripple-65B Base Preview is a pretrain-only checkpoint and deliberately does not include synthetic QA data.
+Nemotron 3 Ultra Base uses a two-phase pretraining data mix approach in which both phases include substantial synthetic data, including QA and SFT data.
+We show official eval scores for all provided evals; for missing evals, we run the evals ourselves and show the results in italics.
+
+Evaluation is performed on an internal fork of [Olmes](https://github.com/allenai/olmes).
 
 <div align="center">
 
-| Benchmark (Metric) | # Shots | DeepSeek-V4-Flash-Base | DeepSeek-V4-Pro-Base | DeepSeek-V4.1-Flash-Base |
-| :--- | :---: | :---: | :---: | :---: |
-| Architecture | — | MoE | MoE | MoE |
-| # Backbone Params | — | 284B | 1.6T | 552B |
-| # Activated Params | — | 13B | 49B | 8B / 16B |
-| **World Knowledge** | | | | |
-| AGIEval (EM) | 3–5-shot | 83.9 | **84.4** | 83.4 |
-| MMLU-Pro (EM) | 5-shot | 68.3 | 73.5 | **74.1** |
-| C-Eval (EM) | 5-shot | 92.1 | **93.1** | 92.1 |
-| MultiLoKo (LLM-Judge) | 5-shot | 42.6 | **50.9** | 45.5 |
-| SimpleQA-Verified (EM) | 25-shot | 30.1 | **55.2** | 42.3 |
-| SuperGPQA (EM) | 5-shot | 46.5 | **53.9** | 53.1 |
-| **Language & Reasoning** | | | | |
-| BBH (EM) | 3-shot | 86.9 | **87.5** | 86.1 |
-| BBEH (EM) | 1-shot | 25.4 | **29.8** | 27.2 |
-| DROP (F1) | 1-shot | **88.6** | **88.7** | 87.9 |
-| HellaSwag (EM) | 0-shot | 85.7 | **88.0** | 87.2 |
-| **Code & Math** | | | | |
-| BigCodeBench (Pass@1) | 3-shot | 56.8 | 59.2 | **60.6** |
-| HumanEval (Pass@1) | 0-shot | 69.5 | 76.8 | **79.4** |
-| GSM8K (EM) | 8-shot | 90.8 | 92.6 | **93.0** |
-| MATH (EM) | 4-shot | 57.4 | **64.5** | 61.1 |
-| MGSM (EM) | 8-shot | **85.7** | 84.4 | 80.2 |
-| **Long Context** | | | | |
-| LongBench-V2 (EM) | 1-shot | 44.7 | **51.5** | 45.2 |
-| **Multimodal** | | | | |
-| MMMU-Pro (EM) | 4-shot | — | — | 56.5 |
-| CVBench (EM) | 4-shot | — | — | 77.9 |
-| DocVQA (LLM-Judge) | 4-shot | — | — | 95.6 |
-| RefCOCO-avg (Acc@0.5) | 0-shot | — | — | 86.0 |
+  | Benchmark (Metric) | # Shots | Ripple-65B | Laguna-M.1 | DeepSeek V3.1 Base | DeepSeek V4 Flash Base | Nemotron 3 Ultra |
+  |---|---|---|---|---|---|---|
+  | **Model** | | | | | | |
+  | Architecture | — | Dense | MoE | MoE | MoE | MoE |
+  | Total params | — | 65B | 225B | 671B | 284B | 550B |
+  | Active params | — | 65B | 23B | 37B | 13B | 55B |
+  | Training tokens (T) | — | 12T | 30T+ | 15.6T | 32T+ | 20T |
+  | Training FLOPs (6ND) | — | 4.7e24 | 4.2e24 | 3.5e24 | 2.5e24 | 6.6e24 |
+  | **World Knowledge** | | | | | | |
+  | MMLU (MC acc) | 5-shot | *83.7* | *76.6* | *88.3* | 88.7 | **89.1** |
+  | MMLU-Pro (CoT EM) | 5-shot | *61.5* | *52.2* | *57.8* | 68.3 | **79.1** |
+  | AGIEval-En (acc) | 3-5-shot | *74.4* | *66.2* | *81.9* | **83.9** | 78.7 |
+  | GPQA main (CoT EM) | 5-shot | *37.7* | *31.7* | *30.8* | *44.0* | ***49.6*** |
+  | GPQA-Diamond (CoT EM) | 5-shot | *38.4* | *27.8* | *37.4* | *44.4* | ***53.0*** |
+  | C-Eval (MC acc) | 5-shot | *69.6* | *59.3* | *90.7* | **92.1** | *82.2* |
+  | SuperGPQA (MC acc) | 5-shot | *34.2* | *24.5* | *43.1* | 46.5 | ***52.9*** |
+  | SimpleQA-Verified (EM) | 25-shot | *10.1* | *10.8* | *17.3* | **30.1** | *27.1* |
+  | **Language & Reasoning** | | | | | | | 
+  | BBH (CoT EM) | 3-shot | *85.6* | *79.6* | *87.2* | 86.9 | ***88.0*** | 
+  | BBEH (EM) | 1-shot | *17.9* | *14.2* | *18.4* | **25.4** | *21.2* |
+  | DROP (F1) | 1-shot | *77.8* | *66.8* | *81.2* | **88.6** | *87.6* |
+  | HellaSwag (acc_norm) | 0-shot | *84.6* | *86.2* | *86.8* | 85.7 | ***89.1*** |
+  | HellaSwag (acc_norm) | 10-shot | *87.2* | *88.1* | *89.4* | *88.8* | **90.5** |
+  | ARC-Challenge (MC acc) | 25-shot | *95.7* | *90.8* | *95.1* | *95.8* | **97.4**[^d] |
+  | ARC-Challenge (acc_norm) | 25-shot | *72.3* | *69.0* | *71.8* | *73.5* | ***74.1*** |
+  | OpenBookQA (acc_norm) | 0-shot | *47.4* | *48.6* | ***50.4*** | *48.0* | 48.6 |
+  | PIQA (acc_norm) | 0-shot | *83.6* | *84.8* | ***85.2*** | *84.7* | 83.8 |
+  | WinoGrande (acc) | 5-shot | ***83.4*** | *83.2* | *83.2* | *81.4*[^a] | 79.3 |
+  | **Code & Math** | | | | | | |
+  | HumanEval (pass@1, n=20, T=0.2) | 0-shot | *70.9* | *83.4* | *69.8* | 69.5 | **83.8**[^b] |
+  | MBPP sanitized-378 (pass@1, n=32, T=0.2) | 3-shot | *81.5* | *81.1* | *75.2* | *74.6* | **86.0**[^c] |
+  | BigCodeBench (pass@1, n=5, T=0.8) | 3-shot | *49.2* | *50.5* | ***59.9*** | 56.8 | *55.6* |
+  | GSM8K (CoT EM) | 8-shot | *88.5* | *80.4* | *79.6*[^e] | **90.8** | 88.1 |
+  | GSM1K (CoT EM) | 8-shot | *88.4* | *79.7* | *81.8*[^e] | *83.2* | ***90.2*** |
+  | MATH (flex EM) | 4-shot | *59.1* | *52.5* | *59.7* | 57.4 | **82.0** |
+  | MGSM (CoT EM, 11-lang avg) | 8-shot | *78.2* | *65.4* | *81.7* | 85.7 | **87.7** |
+
 
 </div>