Update README.md
Browse files
README.md
CHANGED
|
@@ -1,201 +0,0 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: apache-2.0
|
| 3 |
-
---
|
| 4 |
-
# Ripple-65B Preview
|
| 5 |
-
|
| 6 |
-
<div align="center">
|
| 7 |
-
<picture>
|
| 8 |
-
<source media="(prefers-color-scheme: dark)" srcset="https://framerusercontent.com/images/rSaJGoPP1Saln5kpScZNGCw.png" width="30%">
|
| 9 |
-
<source media="(prefers-color-scheme: light)" srcset="https://framerusercontent.com/images/3rFANSmoZuSBhjRwxZolyEU5guo.png" width="30%">
|
| 10 |
-
<img alt="Ricursive Logo" src="https://framerusercontent.com/images/3rFANSmoZuSBhjRwxZolyEU5guo.png" width="30%">
|
| 11 |
-
</picture>
|
| 12 |
-
</div>
|
| 13 |
-
|
| 14 |
-
<p align="center">
|
| 15 |
-
<a href="https://huggingface.co/ricursiveAI/Ripple-65B-Preview">BF16</a> |
|
| 16 |
-
<a href="https://huggingface.co/ricursiveAI/Ripple-65B-Preview-NVFP4">NVFP4</a> |
|
| 17 |
-
<a href="https://huggingface.co/ricursiveAI/Ripple-65B-Base-Preview">Base Model</a> |
|
| 18 |
-
<a href="https://huggingface.co/ricursiveAI/Ripple-65B-Preview-DFlash2">DFlash2</a> |
|
| 19 |
-
<a href="https://www.ricursive.com/ripple">Launch Blog</a> |
|
| 20 |
-
License: <a href="https://huggingface.co/ricursiveAI/Ripple-65B-Preview/blob/main/LICENSE">Apache 2.0</a>
|
| 21 |
-
</p>
|
| 22 |
-
|
| 23 |
-
Ricursive Intelligence introduces **Ripple-65B Preview**, a 65B parameter dense Transformer model pretrained on 12 trillion tokens with support for up to 1M context length.
|
| 24 |
-
Ripple-65B Preview delivers strong results on agentic coding, scoring **80.52 on TerminalBench 2.1** and
|
| 25 |
-
surpassing significantly-larger US open-weight models such as Reflection Beam 501B (**80.1**), Meta Muse Spark Open 816B (**76.4**),
|
| 26 |
-
Thinking Machines Inkling 975B (**63.8**), and NVIDIA Nemotron 3 Ultra 550B (**56.4**).
|
| 27 |
-
|
| 28 |
-
Our Preview release includes an instruction-tuned model in BF16 and NVFP4 numerics, and a BF16 base model. **This repo contains the BF16 instruction-tuned model.**
|
| 29 |
-
|
| 30 |
-
Starting from an initial Olmo 3 baseline, we iteratively improved all aspects of the training recipe including pretraining scaling laws, model architecture, data, and posttraining stack.
|
| 31 |
-
Our final Ripple v1 model ladder improves Compute Efficiency (CE) on code by **16x** relative to the Olmo 3 ladder, with the advantage further increasing with scale up to
|
| 32 |
-
**55x** for the full hero run.
|
| 33 |
-
|
| 34 |
-
Ripple-65B was codesigned to be trained and served efficiently on NVIDIA hardware.
|
| 35 |
-
For our first model, a key objective was to make design decisions which reduce pretraining and posttraining
|
| 36 |
-
infrastructure complexity while still enabling strong reasoning and agentic performance.
|
| 37 |
-
Therefore, we chose to train a modernized dense Transformer model rather than use Mixture-of-Experts (MoE) like other labs.
|
| 38 |
-
This decision quickly paid off, and we were able to train a strong model from scratch with a small team that scaled from 1 to 5 people during development,
|
| 39 |
-
while maintaining solid MFU (40%) throughout pretraining and posttraining with simple sharding, no custom kernels, and no quantization (BF16).
|
| 40 |
-
|
| 41 |
-
Ripple-65B uses a **thin-and-deep** model aspect ratio, with a narrow d_model (5120) and 80 total layers.
|
| 42 |
-
We switch from the Olmo 3 llama-like architecture to a Gemma 3N-like architecture to improve training stability at scale.
|
| 43 |
-
To improve reasoning performance, we **overprovision the number of attention heads** while increasing the SWA-to-global layer ratio
|
| 44 |
-
and reducing SWA window size to compensate for the increased cost.
|
| 45 |
-
In addition, we added FFN activation clipping, gated attention, global NoPE, and value residuals.
|
| 46 |
-
|
| 47 |
-
The end result is a model that is especially capable at reasoning and agentic tasks given its model size, matching or exceeding other
|
| 48 |
-
US open-weight models with more than 10x the number of total parameters.
|
| 49 |
-
At only 65B total parameters, the model can fit on consumer hardware such as the NVIDIA DGX Spark.
|
| 50 |
-
To encourage local model development, we provide an NVFP4 checkpoint and colab to help you get started.
|
| 51 |
-
The model is also fast at token generation when running on datacenter GPUs, reaching 250 tok/s on 2xB200 batch=8 prior to quantization or custom kernels.
|
| 52 |
-
|
| 53 |
-
**Pre-training.** Ripple-65B Preview is pretrained from scratch on our 12T text-only predominantly-English and code corpus at 8K sequence length.
|
| 54 |
-
Ripple-65B Base Preview is a **pretrain-only checkpoint** without any midtraining or synthetic QA data.
|
| 55 |
-
|
| 56 |
-
**Post-training.** We use a standard post-training recipe consisting of midtraining, long context, SFT, long context SFT, and mixed domain RL.
|
| 57 |
-
Our Preview release has gone through relatively light posttraining, with a focus on agentic coding.
|
| 58 |
-
|
| 59 |
-
We show a brief overview of each stage below:
|
| 60 |
-
|
| 61 |
-
<div align="center">
|
| 62 |
-
|
| 63 |
-
| Stage | Tokens | Step | Seqlen | Batch Size |
|
| 64 |
-
| ----- | ------ | ------ | ---- | ----- |
|
| 65 |
-
| Pretrain | 12T | 877,500 | 8K | 1408 -> 3072 |
|
| 66 |
-
| Midtrain | 200B | 8,000 | 64K | 384 |
|
| 67 |
-
| Long Context | 6.5B | 389 | 2M | 8 |
|
| 68 |
-
| SFT | 183B | 7,274 | 128K | 192 |
|
| 69 |
-
| LC SFT | 13.5B | 802 | 2M | 8 |
|
| 70 |
-
| RL | N/A | 700 | 128K | 2048 |
|
| 71 |
-
|
| 72 |
-
</div>
|
| 73 |
-
|
| 74 |
-
## Instruct Model Benchmarks
|
| 75 |
-
|
| 76 |
-
We compare Ripple-65B Preview against currently-available US open-weight frontier models.
|
| 77 |
-
All models are tested at full BF16 precision with maximum effort levels.
|
| 78 |
-
|
| 79 |
-
Each lab typically only provides a limited subset of available benchmarks.
|
| 80 |
-
Our table below shows official model card scores where available, and we use our own in-house eval flow for missing evals (italized).
|
| 81 |
-
For SWE-bench Pro v2.0 Hard and DeepSWE 1.1, we use Terminus 2 for Ripple-65B Preview.
|
| 82 |
-
In other cases where the harness is not specified, we use OpenCode.
|
| 83 |
-
A higher number is better for all benchmarks.
|
| 84 |
-
|
| 85 |
-
Replicating benchmark scores has been challenging, particularly for modern agentic evals, due to the number of factors which can cause
|
| 86 |
-
eval scores to shift.
|
| 87 |
-
For example, we found that Terminal-Bench 2.1 scores were universally lower (with the exception of Muse Spark Open)
|
| 88 |
-
when using the official Terminus 2 harness at 128K sequence length with summarization.
|
| 89 |
-
Therefore, our results represent a best-effort attempt at a fair comparison.
|
| 90 |
-
|
| 91 |
-
<div align="center">
|
| 92 |
-
|
| 93 |
-
| Benchmark | Ripple-65B<br>Preview | Nemotron 3<br>Ultra 550B | Inkling<br>975B | Muse Spark<br>Open 816B | Reflection<br>Beam 501B |
|
| 94 |
-
|---|---|---|---|---|---|
|
| 95 |
-
| **Model** | | | | | |
|
| 96 |
-
| Architecture | Dense | MoE | MoE | MoE | MoE |
|
| 97 |
-
| Total params | 65B | 550B | **975B** | 816B | 501B |
|
| 98 |
-
| Active params | **65B** | 55B | 41B | 42B | 23B |
|
| 99 |
-
| Pre-training tokens | 12T | 20T | **45T** | N/A | 23.8T |
|
| 100 |
-
| **Knowledge** | | | | | |
|
| 101 |
-
| MMLU | 89.6 | *92.0* | *91.3* | ***92.6*** | *92.4* |
|
| 102 |
-
| MMLU-Pro | 84.2 | 86.8 | *87.3* | ***87.9*** | *76.5* |
|
| 103 |
-
| GPQA Diamond | 85.2 | 87.0 | 87.2 | 89.0 | **90.6** |
|
| 104 |
-
| **Math** | | | | | |
|
| 105 |
-
| AIME 2024 | 97.1 | *94.2* | *97.5* | ***97.9*** | *96.7* |
|
| 106 |
-
| AIME 2025 | 98.3 | *94.2* | *91.7* | ***100.0*** | *99.6* |
|
| 107 |
-
| AIME 2026 | 95.0 | *95.4* | 97.1 | ***98.3*** | 97.7 |
|
| 108 |
-
| HMMT Feb 2026 | 90.5 | *80.3* | 86.3 | ***93.9*** | ***93.9*** |
|
| 109 |
-
| **Coding & Agentic** | | | | | |
|
| 110 |
-
| Terminal-Bench 2.1 (self-reported) | **80.5** | 56.4 | 63.8 | 76.4 | 80.1 |
|
| 111 |
-
| Terminal-Bench 2.1 (Terminus 2) | **80.5** | *55.1* | *56.2* | *78.7* | *70.8* |
|
| 112 |
-
| SWE-bench Verified | 76.5 | 71.9 | 77.6 | *78.7* | **81.6** |
|
| 113 |
-
| SWE-bench Pro v1.0 | 61.42 | *46.9* | 54.3 | *58.2* | **66.1** |
|
| 114 |
-
| SWE-bench Pro v2.0 Hard | **79.1** | *30.7* | *2.0* | *52.3* | 76.5 |
|
| 115 |
-
| DeepSWE 1.1 | 44.3 | *0.2* | *1.8* | **60.0** | 43.3 |
|
| 116 |
-
| SciCode (Inspect AI) | 39.4 | 44.6 | 46.1 | **58.5** | 49.2 |
|
| 117 |
-
| τ³-bench banking | 31.27 | 22.6 | 23.7 | *14.4* | **36.1** |
|
| 118 |
-
| AutomationBench (Public) | 36.4 | *8.4* | *14.6* | *42.4* | 39.3 |
|
| 119 |
-
| MCP Atlas | 71.67 | *62.2* | 76.0 | **89.2** | 80.8 |
|
| 120 |
-
| **Safety** | | | | | |
|
| 121 |
-
| FORTRESS Adversarial | 83.8 | *79.3* | 78.0 | **94.2** | *92.1* |
|
| 122 |
-
| FORTRESS Benign | 95.5 | *89.1* | **95.9** | *82.6* | *86.4* |
|
| 123 |
-
| StrongREJECT | 97.5 | *98.2* | **98.6** | *94.7* | *95.8* |
|
| 124 |
-
| **Instruction Following** | | | | | |
|
| 125 |
-
| IFBench | 79.2 | **81.7** | 79.8 | *78.6* | 79.6 |
|
| 126 |
-
| **Long Context** | | | | | |
|
| 127 |
-
| AA-LCR | 75.0 | 65.4 | *78.3* | ***80.7*** | 78.2 |
|
| 128 |
-
|
| 129 |
-
</div>
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
## Base Model Benchmarks
|
| 133 |
-
|
| 134 |
-
We show traditional pretraining evals comparing Ripple-65B Base Preview against a selection of open-weight base models. Although we would have liked to
|
| 135 |
-
compare against the same set of models throughout the model card, many companies are now choosing to only open-source their instruction-tuned models.
|
| 136 |
-
|
| 137 |
-
With the rise of the importance of posttraining, many modern LLM recipes now blend posttraining data earlier into the training process, including during pretraining.
|
| 138 |
-
For our first model, we kept our pretraining data clean with minimal synthetic data, focusing on high quality natural data rather than aggressively targeting evals.
|
| 139 |
-
We instead added significantly more synthetic data during midtraining, which allowed us to quickly iterate on our synthetic data recipe without blocking pretraining.
|
| 140 |
-
|
| 141 |
-
Ripple-65B Base Preview is a pretrain-only checkpoint. The base model overall beats Poolside Laguna M.1,
|
| 142 |
-
and is comparable with DeepSeek V3.1 and V4 Flash on reasoning, math, and code.
|
| 143 |
-
Nemotron 3 Ultra dominates on traditional pretraining evals, likely due to their heavy use of [task-seeded synthetic data](https://huggingface.co/blog/nvidia/task-seeded-sdg).
|
| 144 |
-
However, its pretraining eval performance did not translate to modern agentic tasks, likely due to their synthetic data being out of distribution relative to these tasks.
|
| 145 |
-
|
| 146 |
-
We report official model results where available.
|
| 147 |
-
For our own model and for missing evals, we use our internal fork of [Olmes](https://github.com/allenai/olmes).
|
| 148 |
-
|
| 149 |
-
<div align="center">
|
| 150 |
-
|
| 151 |
-
| Benchmark (Metric) | # Shots | Ripple-65B<br>Preview | Laguna<br>M.1 225B | DeepSeek<br>V3.1 671B | DeepSeek<br>V4 Flash 284B | Nemotron 3<br>Ultra 550B |
|
| 152 |
-
|---|---|---|---|---|---|---|
|
| 153 |
-
| **Model** | | | | | | |
|
| 154 |
-
| Architecture | — | Dense | MoE | MoE | MoE | MoE |
|
| 155 |
-
| Total params | — | 65B | 225B | **671B** | 284B | 550B |
|
| 156 |
-
| Active params | — | **65B** | 23B | 37B | 13B | 55B |
|
| 157 |
-
| Training tokens (T) | — | 12T | 30T+ | 15.6T | **32T+** | 20T |
|
| 158 |
-
| **World Knowledge** | | | | | | |
|
| 159 |
-
| MMLU (MC acc) | 5-shot | *83.7* | *76.6* | *88.3* | 88.7 | **89.1** |
|
| 160 |
-
| MMLU-Pro (CoT EM) | 5-shot | *61.5* | *52.2* | *57.8* | 68.3 | **79.1** |
|
| 161 |
-
| AGIEval-En (acc) | 3-5-shot | *74.4* | *66.2* | *81.9* | **83.9** | 78.7 |
|
| 162 |
-
| GPQA main (CoT EM) | 5-shot | *37.7* | *31.7* | *30.8* | *44.0* | ***49.6*** |
|
| 163 |
-
| GPQA-Diamond (CoT EM) | 5-shot | *38.4* | *27.8* | *37.4* | *44.4* | ***53.0*** |
|
| 164 |
-
| C-Eval (MC acc) | 5-shot | *69.6* | *59.3* | *90.7* | **92.1** | *82.2* |
|
| 165 |
-
| SuperGPQA (MC acc) | 5-shot | *34.2* | *24.5* | *43.1* | 46.5 | ***52.9*** |
|
| 166 |
-
| SimpleQA-Verified (EM) | 25-shot | *10.1* | *10.8* | *17.3* | **30.1** | *27.1* |
|
| 167 |
-
| **Language & Reasoning** | | | | | | |
|
| 168 |
-
| BBH (CoT EM) | 3-shot | *85.6* | *79.6* | *87.2* | 86.9 | ***88.0*** |
|
| 169 |
-
| BBEH (EM) | 1-shot | *17.9* | *14.2* | *18.4* | **25.4** | *21.2* |
|
| 170 |
-
| DROP (F1) | 1-shot | *77.8* | *66.8* | *81.2* | **88.6** | *87.6* |
|
| 171 |
-
| HellaSwag (acc_norm) | 0-shot | *84.6* | *86.2* | *86.8* | 85.7 | ***89.1*** |
|
| 172 |
-
| HellaSwag (acc_norm) | 10-shot | *87.2* | *88.1* | *89.4* | *88.8* | **90.5** |
|
| 173 |
-
| ARC-Challenge (MC acc) | 25-shot | *95.7* | *90.8* | *95.1* | *95.8* | **97.4** |
|
| 174 |
-
| OpenBookQA (acc_norm) | 0-shot | *47.4* | *48.6* | ***50.4*** | *48.0* | 48.6 |
|
| 175 |
-
| PIQA (acc_norm) | 0-shot | *83.6* | *84.8* | ***85.2*** | *84.7* | 83.8 |
|
| 176 |
-
| WinoGrande (acc) | 5-shot | ***83.4*** | *83.2* | *83.2* | *81.4* | 79.3 |
|
| 177 |
-
| **Code & Math** | | | | | | |
|
| 178 |
-
| HumanEval (pass@1, n=20, T=0.2) | 0-shot | *70.9* | *83.4* | *69.8* | 69.5 | **83.8** |
|
| 179 |
-
| MBPP Sanitized (pass@1, n=32, T=0.2) | 3-shot | *81.5* | *81.1* | *75.2* | *74.6* | **86.0** |
|
| 180 |
-
| BigCodeBench (pass@1, n=5, T=0.8) | 3-shot | *49.2* | *50.5* | ***59.9*** | 56.8 | *55.6* |
|
| 181 |
-
| GSM8K (CoT EM) | 8-shot | *88.5* | *80.4* | *79.6* | **90.8** | 88.1 |
|
| 182 |
-
| GSM1K (CoT EM) | 8-shot | *88.4* | *79.7* | *81.8* | *83.2* | ***90.2*** |
|
| 183 |
-
| MATH (flex EM) | 4-shot | *59.1* | *52.5* | *59.7* | 57.4 | **82.0** |
|
| 184 |
-
| MGSM (CoT EM, 11-lang avg) | 8-shot | *78.2* | *65.4* | *81.7* | 85.7 | **87.7** |
|
| 185 |
-
|
| 186 |
-
|
| 187 |
-
</div>
|
| 188 |
-
|
| 189 |
-
## Citation
|
| 190 |
-
|
| 191 |
-
```bibtex
|
| 192 |
-
@misc{ricursive2026ripple65b,
|
| 193 |
-
title={Ripple-65B: Pushing the Frontier of Local Intelligence},
|
| 194 |
-
author={Ricursive Intelligence},
|
| 195 |
-
year={2026},
|
| 196 |
-
}
|
| 197 |
-
```
|
| 198 |
-
|
| 199 |
-
## Contact
|
| 200 |
-
|
| 201 |
-
For questions and comments, please contact us at [info@ricursive.com](mailto:info@ricursive.com). To learn more, please visit our [website](https://www.ricursive.com).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|