xsingh commited on
Commit
4b91e27
·
verified ·
1 Parent(s): 314d309

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -201
README.md CHANGED
@@ -1,201 +0,0 @@
1
- ---
2
- license: apache-2.0
3
- ---
4
- # Ripple-65B Preview
5
-
6
- <div align="center">
7
- <picture>
8
- <source media="(prefers-color-scheme: dark)" srcset="https://framerusercontent.com/images/rSaJGoPP1Saln5kpScZNGCw.png" width="30%">
9
- <source media="(prefers-color-scheme: light)" srcset="https://framerusercontent.com/images/3rFANSmoZuSBhjRwxZolyEU5guo.png" width="30%">
10
- <img alt="Ricursive Logo" src="https://framerusercontent.com/images/3rFANSmoZuSBhjRwxZolyEU5guo.png" width="30%">
11
- </picture>
12
- </div>
13
-
14
- <p align="center">
15
- <a href="https://huggingface.co/ricursiveAI/Ripple-65B-Preview">BF16</a> |
16
- <a href="https://huggingface.co/ricursiveAI/Ripple-65B-Preview-NVFP4">NVFP4</a> |
17
- <a href="https://huggingface.co/ricursiveAI/Ripple-65B-Base-Preview">Base Model</a> |
18
- <a href="https://huggingface.co/ricursiveAI/Ripple-65B-Preview-DFlash2">DFlash2</a> |
19
- <a href="https://www.ricursive.com/ripple">Launch Blog</a> |
20
- License: <a href="https://huggingface.co/ricursiveAI/Ripple-65B-Preview/blob/main/LICENSE">Apache 2.0</a>
21
- </p>
22
-
23
- Ricursive Intelligence introduces **Ripple-65B Preview**, a 65B parameter dense Transformer model pretrained on 12 trillion tokens with support for up to 1M context length.
24
- Ripple-65B Preview delivers strong results on agentic coding, scoring **80.52 on TerminalBench 2.1** and
25
- surpassing significantly-larger US open-weight models such as Reflection Beam 501B (**80.1**), Meta Muse Spark Open 816B (**76.4**),
26
- Thinking Machines Inkling 975B (**63.8**), and NVIDIA Nemotron 3 Ultra 550B (**56.4**).
27
-
28
- Our Preview release includes an instruction-tuned model in BF16 and NVFP4 numerics, and a BF16 base model. **This repo contains the BF16 instruction-tuned model.**
29
-
30
- Starting from an initial Olmo 3 baseline, we iteratively improved all aspects of the training recipe including pretraining scaling laws, model architecture, data, and posttraining stack.
31
- Our final Ripple v1 model ladder improves Compute Efficiency (CE) on code by **16x** relative to the Olmo 3 ladder, with the advantage further increasing with scale up to
32
- **55x** for the full hero run.
33
-
34
- Ripple-65B was codesigned to be trained and served efficiently on NVIDIA hardware.
35
- For our first model, a key objective was to make design decisions which reduce pretraining and posttraining
36
- infrastructure complexity while still enabling strong reasoning and agentic performance.
37
- Therefore, we chose to train a modernized dense Transformer model rather than use Mixture-of-Experts (MoE) like other labs.
38
- This decision quickly paid off, and we were able to train a strong model from scratch with a small team that scaled from 1 to 5 people during development,
39
- while maintaining solid MFU (40%) throughout pretraining and posttraining with simple sharding, no custom kernels, and no quantization (BF16).
40
-
41
- Ripple-65B uses a **thin-and-deep** model aspect ratio, with a narrow d_model (5120) and 80 total layers.
42
- We switch from the Olmo 3 llama-like architecture to a Gemma 3N-like architecture to improve training stability at scale.
43
- To improve reasoning performance, we **overprovision the number of attention heads** while increasing the SWA-to-global layer ratio
44
- and reducing SWA window size to compensate for the increased cost.
45
- In addition, we added FFN activation clipping, gated attention, global NoPE, and value residuals.
46
-
47
- The end result is a model that is especially capable at reasoning and agentic tasks given its model size, matching or exceeding other
48
- US open-weight models with more than 10x the number of total parameters.
49
- At only 65B total parameters, the model can fit on consumer hardware such as the NVIDIA DGX Spark.
50
- To encourage local model development, we provide an NVFP4 checkpoint and colab to help you get started.
51
- The model is also fast at token generation when running on datacenter GPUs, reaching 250 tok/s on 2xB200 batch=8 prior to quantization or custom kernels.
52
-
53
- **Pre-training.** Ripple-65B Preview is pretrained from scratch on our 12T text-only predominantly-English and code corpus at 8K sequence length.
54
- Ripple-65B Base Preview is a **pretrain-only checkpoint** without any midtraining or synthetic QA data.
55
-
56
- **Post-training.** We use a standard post-training recipe consisting of midtraining, long context, SFT, long context SFT, and mixed domain RL.
57
- Our Preview release has gone through relatively light posttraining, with a focus on agentic coding.
58
-
59
- We show a brief overview of each stage below:
60
-
61
- <div align="center">
62
-
63
- | Stage | Tokens | Step | Seqlen | Batch Size |
64
- | ----- | ------ | ------ | ---- | ----- |
65
- | Pretrain | 12T | 877,500 | 8K | 1408 -> 3072 |
66
- | Midtrain | 200B | 8,000 | 64K | 384 |
67
- | Long Context | 6.5B | 389 | 2M | 8 |
68
- | SFT | 183B | 7,274 | 128K | 192 |
69
- | LC SFT | 13.5B | 802 | 2M | 8 |
70
- | RL | N/A | 700 | 128K | 2048 |
71
-
72
- </div>
73
-
74
- ## Instruct Model Benchmarks
75
-
76
- We compare Ripple-65B Preview against currently-available US open-weight frontier models.
77
- All models are tested at full BF16 precision with maximum effort levels.
78
-
79
- Each lab typically only provides a limited subset of available benchmarks.
80
- Our table below shows official model card scores where available, and we use our own in-house eval flow for missing evals (italized).
81
- For SWE-bench Pro v2.0 Hard and DeepSWE 1.1, we use Terminus 2 for Ripple-65B Preview.
82
- In other cases where the harness is not specified, we use OpenCode.
83
- A higher number is better for all benchmarks.
84
-
85
- Replicating benchmark scores has been challenging, particularly for modern agentic evals, due to the number of factors which can cause
86
- eval scores to shift.
87
- For example, we found that Terminal-Bench 2.1 scores were universally lower (with the exception of Muse Spark Open)
88
- when using the official Terminus 2 harness at 128K sequence length with summarization.
89
- Therefore, our results represent a best-effort attempt at a fair comparison.
90
-
91
- <div align="center">
92
-
93
- | Benchmark | Ripple-65B<br>Preview | Nemotron 3<br>Ultra 550B | Inkling<br>975B | Muse Spark<br>Open 816B | Reflection<br>Beam 501B |
94
- |---|---|---|---|---|---|
95
- | **Model** | | | | | |
96
- | Architecture | Dense | MoE | MoE | MoE | MoE |
97
- | Total params | 65B | 550B | **975B** | 816B | 501B |
98
- | Active params | **65B** | 55B | 41B | 42B | 23B |
99
- | Pre-training tokens | 12T | 20T | **45T** | N/A | 23.8T |
100
- | **Knowledge** | | | | | |
101
- | MMLU | 89.6 | *92.0* | *91.3* | ***92.6*** | *92.4* |
102
- | MMLU-Pro | 84.2 | 86.8 | *87.3* | ***87.9*** | *76.5* |
103
- | GPQA Diamond | 85.2 | 87.0 | 87.2 | 89.0 | **90.6** |
104
- | **Math** | | | | | |
105
- | AIME 2024 | 97.1 | *94.2* | *97.5* | ***97.9*** | *96.7* |
106
- | AIME 2025 | 98.3 | *94.2* | *91.7* | ***100.0*** | *99.6* |
107
- | AIME 2026 | 95.0 | *95.4* | 97.1 | ***98.3*** | 97.7 |
108
- | HMMT Feb 2026 | 90.5 | *80.3* | 86.3 | ***93.9*** | ***93.9*** |
109
- | **Coding & Agentic** | | | | | |
110
- | Terminal-Bench 2.1 (self-reported) | **80.5** | 56.4 | 63.8 | 76.4 | 80.1 |
111
- | Terminal-Bench 2.1 (Terminus 2) | **80.5** | *55.1* | *56.2* | *78.7* | *70.8* |
112
- | SWE-bench Verified | 76.5 | 71.9 | 77.6 | *78.7* | **81.6** |
113
- | SWE-bench Pro v1.0 | 61.42 | *46.9* | 54.3 | *58.2* | **66.1** |
114
- | SWE-bench Pro v2.0 Hard | **79.1** | *30.7* | *2.0* | *52.3* | 76.5 |
115
- | DeepSWE 1.1 | 44.3 | *0.2* | *1.8* | **60.0** | 43.3 |
116
- | SciCode (Inspect AI) | 39.4 | 44.6 | 46.1 | **58.5** | 49.2 |
117
- | τ³-bench banking | 31.27 | 22.6 | 23.7 | *14.4* | **36.1** |
118
- | AutomationBench (Public) | 36.4 | *8.4* | *14.6* | *42.4* | 39.3 |
119
- | MCP Atlas | 71.67 | *62.2* | 76.0 | **89.2** | 80.8 |
120
- | **Safety** | | | | | |
121
- | FORTRESS Adversarial | 83.8 | *79.3* | 78.0 | **94.2** | *92.1* |
122
- | FORTRESS Benign | 95.5 | *89.1* | **95.9** | *82.6* | *86.4* |
123
- | StrongREJECT | 97.5 | *98.2* | **98.6** | *94.7* | *95.8* |
124
- | **Instruction Following** | | | | | |
125
- | IFBench | 79.2 | **81.7** | 79.8 | *78.6* | 79.6 |
126
- | **Long Context** | | | | | |
127
- | AA-LCR | 75.0 | 65.4 | *78.3* | ***80.7*** | 78.2 |
128
-
129
- </div>
130
-
131
-
132
- ## Base Model Benchmarks
133
-
134
- We show traditional pretraining evals comparing Ripple-65B Base Preview against a selection of open-weight base models. Although we would have liked to
135
- compare against the same set of models throughout the model card, many companies are now choosing to only open-source their instruction-tuned models.
136
-
137
- With the rise of the importance of posttraining, many modern LLM recipes now blend posttraining data earlier into the training process, including during pretraining.
138
- For our first model, we kept our pretraining data clean with minimal synthetic data, focusing on high quality natural data rather than aggressively targeting evals.
139
- We instead added significantly more synthetic data during midtraining, which allowed us to quickly iterate on our synthetic data recipe without blocking pretraining.
140
-
141
- Ripple-65B Base Preview is a pretrain-only checkpoint. The base model overall beats Poolside Laguna M.1,
142
- and is comparable with DeepSeek V3.1 and V4 Flash on reasoning, math, and code.
143
- Nemotron 3 Ultra dominates on traditional pretraining evals, likely due to their heavy use of [task-seeded synthetic data](https://huggingface.co/blog/nvidia/task-seeded-sdg).
144
- However, its pretraining eval performance did not translate to modern agentic tasks, likely due to their synthetic data being out of distribution relative to these tasks.
145
-
146
- We report official model results where available.
147
- For our own model and for missing evals, we use our internal fork of [Olmes](https://github.com/allenai/olmes).
148
-
149
- <div align="center">
150
-
151
- | Benchmark (Metric) | # Shots | Ripple-65B<br>Preview | Laguna<br>M.1 225B | DeepSeek<br>V3.1 671B | DeepSeek<br>V4 Flash 284B | Nemotron 3<br>Ultra 550B |
152
- |---|---|---|---|---|---|---|
153
- | **Model** | | | | | | |
154
- | Architecture | — | Dense | MoE | MoE | MoE | MoE |
155
- | Total params | — | 65B | 225B | **671B** | 284B | 550B |
156
- | Active params | — | **65B** | 23B | 37B | 13B | 55B |
157
- | Training tokens (T) | — | 12T | 30T+ | 15.6T | **32T+** | 20T |
158
- | **World Knowledge** | | | | | | |
159
- | MMLU (MC acc) | 5-shot | *83.7* | *76.6* | *88.3* | 88.7 | **89.1** |
160
- | MMLU-Pro (CoT EM) | 5-shot | *61.5* | *52.2* | *57.8* | 68.3 | **79.1** |
161
- | AGIEval-En (acc) | 3-5-shot | *74.4* | *66.2* | *81.9* | **83.9** | 78.7 |
162
- | GPQA main (CoT EM) | 5-shot | *37.7* | *31.7* | *30.8* | *44.0* | ***49.6*** |
163
- | GPQA-Diamond (CoT EM) | 5-shot | *38.4* | *27.8* | *37.4* | *44.4* | ***53.0*** |
164
- | C-Eval (MC acc) | 5-shot | *69.6* | *59.3* | *90.7* | **92.1** | *82.2* |
165
- | SuperGPQA (MC acc) | 5-shot | *34.2* | *24.5* | *43.1* | 46.5 | ***52.9*** |
166
- | SimpleQA-Verified (EM) | 25-shot | *10.1* | *10.8* | *17.3* | **30.1** | *27.1* |
167
- | **Language & Reasoning** | | | | | | |
168
- | BBH (CoT EM) | 3-shot | *85.6* | *79.6* | *87.2* | 86.9 | ***88.0*** |
169
- | BBEH (EM) | 1-shot | *17.9* | *14.2* | *18.4* | **25.4** | *21.2* |
170
- | DROP (F1) | 1-shot | *77.8* | *66.8* | *81.2* | **88.6** | *87.6* |
171
- | HellaSwag (acc_norm) | 0-shot | *84.6* | *86.2* | *86.8* | 85.7 | ***89.1*** |
172
- | HellaSwag (acc_norm) | 10-shot | *87.2* | *88.1* | *89.4* | *88.8* | **90.5** |
173
- | ARC-Challenge (MC acc) | 25-shot | *95.7* | *90.8* | *95.1* | *95.8* | **97.4** |
174
- | OpenBookQA (acc_norm) | 0-shot | *47.4* | *48.6* | ***50.4*** | *48.0* | 48.6 |
175
- | PIQA (acc_norm) | 0-shot | *83.6* | *84.8* | ***85.2*** | *84.7* | 83.8 |
176
- | WinoGrande (acc) | 5-shot | ***83.4*** | *83.2* | *83.2* | *81.4* | 79.3 |
177
- | **Code & Math** | | | | | | |
178
- | HumanEval (pass@1, n=20, T=0.2) | 0-shot | *70.9* | *83.4* | *69.8* | 69.5 | **83.8** |
179
- | MBPP Sanitized (pass@1, n=32, T=0.2) | 3-shot | *81.5* | *81.1* | *75.2* | *74.6* | **86.0** |
180
- | BigCodeBench (pass@1, n=5, T=0.8) | 3-shot | *49.2* | *50.5* | ***59.9*** | 56.8 | *55.6* |
181
- | GSM8K (CoT EM) | 8-shot | *88.5* | *80.4* | *79.6* | **90.8** | 88.1 |
182
- | GSM1K (CoT EM) | 8-shot | *88.4* | *79.7* | *81.8* | *83.2* | ***90.2*** |
183
- | MATH (flex EM) | 4-shot | *59.1* | *52.5* | *59.7* | 57.4 | **82.0** |
184
- | MGSM (CoT EM, 11-lang avg) | 8-shot | *78.2* | *65.4* | *81.7* | 85.7 | **87.7** |
185
-
186
-
187
- </div>
188
-
189
- ## Citation
190
-
191
- ```bibtex
192
- @misc{ricursive2026ripple65b,
193
- title={Ripple-65B: Pushing the Frontier of Local Intelligence},
194
- author={Ricursive Intelligence},
195
- year={2026},
196
- }
197
- ```
198
-
199
- ## Contact
200
-
201
- For questions and comments, please contact us at [info@ricursive.com](mailto:info@ricursive.com). To learn more, please visit our [website](https://www.ricursive.com).