|
Download README.md from xsingh/SGI-Preview: direct link, hf CLI and curl.
- Browser
- Download file 8.71 kB
-
https://huggingface.co/xsingh/SGI-Preview/resolve/main/README.md
- Command line
-
hf download hf://xsingh/SGI-Preview/README.md
-
curl -L -o README.md https://huggingface.co/xsingh/SGI-Preview/resolve/main/README.md
8.71 kB
| license: apache-2.0 | |
| # SGI-Preview | |
| <div align="center"> | |
| <picture> | |
| <source media="(prefers-color-scheme: dark)" srcset="https://placeholder.url/logo-dark.png" width="30%"> | |
| <source media="(prefers-color-scheme: light)" srcset="https://placeholder.url/logo-light.png" width="30%"> | |
| <img alt="SGI Logo" src="https://placeholder.url/logo-light.png" width="30%"> | |
| </picture> | |
| </div> | |
| <p align="center"> | |
| <a href="https://huggingface.co/xsingh/SGI-Preview">BF16</a> | | |
| <a href="https://huggingface.co/xsingh/SGI-Preview-NVFP4">NVFP4</a> | | |
| <a href="https://huggingface.co/xsingh/SGI-Base-Preview">Base Model</a> | | |
| <a href="https://huggingface.co/xsingh/SGI-Preview-DFlash2">DFlash2</a> | | |
| <a href="https://example.com/sgi-preview">Launch Blog</a> | | |
| License: <a href="https://huggingface.co/xsingh/SGI-Preview/blob/main/LICENSE">Apache 2.0</a> | |
| </p> | |
| We introduce **SGI-Preview**, a 70B parameter dense Transformer model pretrained on 12 trillion tokens with support for up to 1M context length. | |
| SGI-Preview delivers strong results on agentic coding and reasoning, scoring **80.52 on TerminalBench 2.1** and | |
| surpassing significantly larger open-weight models including Reflection Beam 501B (**80.1**), Meta Muse Spark Open 816B (**76.4**), | |
| Thinking Machines Inkling 975B (**63.8**), and NVIDIA Nemotron 3 Ultra 550B (**56.4**). | |
| Our Preview release includes an instruction-tuned model in BF16 and NVFP4 numerics, along with a pretrain-only BF16 base model. **This repository contains the BF16 instruction-tuned model.** | |
| Starting from an initial Olmo 3 baseline, we iteratively improved all aspects of the training recipe including scaling laws, architecture, data curation, and the post-training stack. | |
| Our final model ladder improves Compute Efficiency (CE) on code by **16x** relative to the baseline ladder, with advantages reaching up to **55x** at full scale. | |
| SGI-Preview was codesigned for high-efficiency training and inference on NVIDIA hardware. | |
| A core objective was reducing infrastructure complexity while maximizing reasoning and agentic performance. | |
| We selected a modernized dense Transformer architecture over Mixture-of-Experts (MoE), enabling training from scratch with a lean team | |
| while maintaining a consistent ~40% Model Flops Utilization (MFU) across pretraining and posttraining using standard sharding (BF16), without custom kernels or early quantization. | |
| ### Architectural Highlights | |
| - **Aspect Ratio:** Thin-and-deep design with a narrow `d_model` (5120) and 80 total layers. | |
| - **Backbone Stability:** Transitioned from an Olmo 3-style setup to a Gemma 3N-like architecture for enhanced large-scale stability. | |
| - **Attention Design:** Overprovisioned attention heads paired with an increased SWA-to-global layer ratio and compact SWA window sizes. | |
| - **Stabilization Features:** FFN activation clipping, gated attention, global NoPE, and value residuals. | |
| At 70B dense parameters, the model is compact enough to run on workstation setups (e.g., NVIDIA DGX Spark via the NVFP4 checkpoint) while sustaining generation throughputs of ~240 tok/s on datacenter nodes (2x B200, batch=8). | |
| --- | |
| ## Training Overview | |
| - **Pre-training:** Trained from scratch on a 12T token text and code corpus (predominantly English) at an 8K sequence length. | |
| - **Post-training:** Multi-stage pipeline covering midtraining, context window extension, SFT, long-context SFT, and mixed-domain RL. | |
| <div align="center"> | |
| | Stage | Tokens | Steps | Seqlen | Batch Size | | |
| | :--- | :--- | :--- | :--- | :--- | | |
| | **Pretrain** | 12T | 877,500 | 8K | 1,408 → 3,072 | | |
| | **Midtrain** | 200B | 8,000 | 64K | 384 | | |
| | **Long Context** | 6.5B | 389 | 2M | 8 | | |
| | **SFT** | 183B | 7,274 | 128K | 192 | | |
| | **LC SFT** | 13.5B | 802 | 2M | 8 | | |
| | **RL** | N/A | 700 | 128K | 2,048 | | |
| </div> | |
| --- | |
| ## Instruct Model Benchmarks | |
| Evaluated at full BF16 precision with maximum effort levels. Official model card results are shown where available; values in italics denote in-house benchmark reproductions. Higher is better across all metrics. | |
| <div align="center"> | |
| | Benchmark | SGI-Preview<br>(70B Dense) | Nemotron 3<br>Ultra 550B | Inkling<br>975B | Muse Spark<br>Open 816B | Reflection<br>Beam 501B | | |
| | :--- | :---: | :---: | :---: | :---: | :---: | | |
| | **Architecture** | **Dense** | MoE | MoE | MoE | MoE | | |
| | **Total Params** | **70B** | 550B | 975B | 816B | 501B | | |
| | **Active Params** | **70B** | 55B | 41B | 42B | 23B | | |
| | **Pretrain Tokens** | **12T** | 20T | 45T | N/A | 23.8T | | |
| | **Knowledge** | | | | | | | |
| | MMLU | 89.6 | *92.0* | *91.3* | **92.6** | *92.4* | | |
| | MMLU-Pro | 84.2 | 86.8 | *87.3* | **87.9** | *76.5* | | |
| | GPQA Diamond | 85.2 | 87.0 | 87.2 | 89.0 | **90.6** | | |
| | **Math** | | | | | | | |
| | AIME 2024 | 97.1 | *94.2* | *97.5* | **97.9** | *96.7* | | |
| | AIME 2025 | 98.3 | *94.2* | *91.7* | **100.0** | *99.6* | | |
| | AIME 2026 | 95.0 | *95.4* | 97.1 | **98.3** | 97.7 | | |
| | HMMT Feb 2026 | 90.5 | *80.3* | 86.3 | **93.9** | **93.9** | | |
| | **Coding & Agentic** | | | | | | | |
| | Terminal-Bench 2.1 (Self-reported) | **80.5** | 56.4 | 63.8 | 76.4 | 80.1 | | |
| | Terminal-Bench 2.1 (Terminus 2) | **80.5** | *55.1* | *56.2* | *78.7* | *70.8* | | |
| | SWE-bench Verified | 76.5 | 71.9 | 77.6 | *78.7* | **81.6** | | |
| | SWE-bench Pro v1.0 | 61.4 | *46.9* | 54.3 | *58.2* | **66.1** | | |
| | SWE-bench Pro v2.0 Hard | **79.1** | *30.7* | *2.0* | *52.3* | 76.5 | | |
| | DeepSWE 1.1 | 44.3 | *0.2* | *1.8* | **60.0** | 43.3 | | |
| | SciCode (Inspect AI) | 39.4 | 44.6 | 46.1 | **58.5** | 49.2 | | |
| | τ³-bench Banking | 31.3 | 22.6 | 23.7 | *14.4* | **36.1** | | |
| | AutomationBench (Public) | 36.4 | *8.4* | *14.6* | *42.4* | 39.3 | | |
| | MCP Atlas | 71.7 | *62.2* | 76.0 | **89.2** | 80.8 | | |
| | **Safety & Alignment** | | | | | | | |
| | FORTRESS Adversarial | 83.8 | *79.3* | 78.0 | **94.2** | *92.1* | | |
| | FORTRESS Benign | 95.5 | *89.1* | **95.9** | *82.6* | *86.4* | | |
| | StrongREJECT | 97.5 | *98.2* | **98.6** | *94.7* | *95.8* | | |
| | **Instruction & Context** | | | | | | | |
| | IFBench | 79.2 | **81.7** | 79.8 | *78.6* | 79.6 | | |
| | AA-LCR (Long Context) | 75.0 | 65.4 | *78.3* | **80.7** | 78.2 | | |
| </div> | |
| --- | |
| ## Base Model Benchmarks | |
| Evaluation results for **SGI-Base-Preview** (pretrain-only checkpoint) against representative open-weight base models using standardized multi-shot setups: | |
| <div align="center"> | |
| | Benchmark (Metric) | Shots | SGI-Base-Preview | Laguna M.1 225B | DeepSeek V3.1 671B | DeepSeek V4 Flash 284B | Nemotron 3 Ultra 550B | | |
| | :--- | :---: | :---: | :---: | :---: | :---: | | |
| | **Architecture** | — | **Dense** | MoE | MoE | MoE | MoE | | |
| | **Total Params** | — | **70B** | 225B | 671B | 284B | 550B | | |
| | **Active Params** | — | **70B** | 23B | 37B | 13B | 55B | | |
| | **Tokens (T)** | — | **12T** | 30T+ | 15.6T | 32T+ | 20T | | |
| | **World Knowledge** | | | | | | | |
| | MMLU (MC acc) | 5 | *83.7* | *76.6* | *88.3* | 88.7 | **89.1** | | |
| | MMLU-Pro (CoT EM) | 5 | *61.5* | *52.2* | *57.8* | 68.3 | **79.1** | | |
| | AGIEval-En (acc) | 3–5 | *74.4* | *66.2* | *81.9* | **83.9** | 78.7 | | |
| | GPQA Main (CoT EM) | 5 | *37.7* | *31.7* | *30.8* | *44.0* | **49.6** | | |
| | GPQA Diamond (CoT EM) | 5 | *38.4* | *27.8* | *37.4* | *44.4* | **53.0** | | |
| | C-Eval (MC acc) | 5 | *69.6* | *59.3* | *90.7* | **92.1** | *82.2* | | |
| | SuperGPQA (MC acc) | 5 | *34.2* | *24.5* | *43.1* | 46.5 | **52.9** | | |
| | SimpleQA-Verified (EM) | 25 | *10.1* | *10.8* | *17.3* | **30.1** | *27.1* | | |
| | **Language & Reasoning** | | | | | | | |
| | BBH (CoT EM) | 3 | *85.6* | *79.6* | *87.2* | 86.9 | **88.0** | | |
| | BBEH (EM) | 1 | *17.9* | *14.2* | *18.4* | **25.4** | *21.2* | | |
| | DROP (F1) | 1 | *77.8* | *66.8* | *81.2* | **88.6** | *87.6* | | |
| | HellaSwag (acc_norm) | 0 | *84.6* | *86.2* | *86.8* | 85.7 | **89.1** | | |
| | HellaSwag (acc_norm) | 10 | *87.2* | *88.1* | *89.4* | *88.8* | **90.5** | | |
| | ARC-Challenge (MC acc) | 25 | *95.7* | *90.8* | *95.1* | *95.8* | **97.4** | | |
| | OpenBookQA (acc_norm) | 0 | *47.4* | *48.6* | **50.4** | *48.0* | 48.6 | | |
| | PIQA (acc_norm) | 0 | *83.6* | *84.8* | **85.2** | *84.7* | 83.8 | | |
| | WinoGrande (acc) | 5 | **83.4** | *83.2* | *83.2* | *81.4* | 79.3 | | |
| | **Code & Math** | | | | | | | |
| | HumanEval (pass@1) | 0 | *70.9* | *83.4* | *69.8* | 69.5 | **83.8** | | |
| | MBPP Sanitized (pass@1) | 3 | *81.5* | *81.1* | *75.2* | *74.6* | **86.0** | | |
| | BigCodeBench (pass@1) | 3 | *49.2* | *50.5* | **59.9** | 56.8 | *55.6* | | |
| | GSM8K (CoT EM) | 8 | *88.5* | *80.4* | *79.6* | **90.8** | 88.1 | | |
| | GSM1K (CoT EM) | 8 | *88.4* | *79.7* | *81.8* | *83.2* | **90.2** | | |
| | MATH (flex EM) | 4 | *59.1* | *52.5* | *59.7* | 57.4 | **82.0** | | |
| | MGSM (11-lang avg) | 8 | *78.2* | *65.4* | *81.7* | 85.7 | **87.7** | | |
| </div> | |
| --- | |
| ## Citation | |
| ```bibtex | |
| @misc{sgi2026preview, | |
| title={SGI-Preview: Pushing the Frontier of Dense Model Efficiency}, | |
| author={SGI Team}, | |
| year={2026}, | |
| } |