Create README.md

README.md

@@ -0,0 +1,23 @@
+# Ripple-65B Base Preview
+
+We introduce **Ripple-65B Preview**, a 65B parameter dense Transformer model pretrained on 12 trillion tokens that scores [PLACEHOLDER] on TerminalBench 2.1,
+surpassing significantly-larger US open-weight models such as Muse Spark Open 816B (76.4), Inkling-975B (63.8), and Nemotron 3 Ultra 550B (56.4). 
+
+Starting from an initial Olmo 3 baseline, we iteratively improved all aspects of the training recipe, including the pretraining scaling laws, model architecture, and data.
+Our final Ripple model ladder improves Compute Efficiency (CE) on code by **16x** relative to the Olmo 3 ladder, with the advantage further increasing with scale up to 
+**55x** for the full hero run. 
+
+Ripple-65B was codesigned to be trained and served efficiently on NVIDIA hardware.
+For our first model, a key design objective was to make model architecture decisions which reduce pretraining and posttraining
+infrastructure complexity while still enabling strong reasoning and agentic performance.
+Therefore, we chose to train a modernized dense Transformer model rather than use Mixture-of-Experts (MoE).
+This decision quickly paid off, and we were able to train a strong model from scratch with a small team with only 5 people.
+
+Starting from the Olmo llama-like baseline, we switch to a more Gemma-like architecture to improve training stability at scale. 
+Ripple-65B uses a **thin-and-deep** model aspect ratio, with a narrow d_model (5120) and 80 total layers.
+To improve reasoning performance, we **overprovision the number of attention heads** while increasing the SWA-to-global ratio and reducing SWA window size to compensate.
+In addition, we added FFN activation clipping, gated attention, global NoPE, and value residuals.
+
+The end result is a model that is especially capable at reasoning given its model size.
+At 65B total parameters, the model can fit on consumer hardware, such as the NVIDIA DGX Spark.
+The model is also fast at datacenter token generation, easily reaching 250+ tok/s with DFlash2 on 4xB200.