---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- base-model
- small-language-model
- causal-lm
datasets:
- karpathy/climbmix-400b-shuffle
- HuggingFaceTB/finemath
- openbmb/UltraData-Math
- microsoft/orca-math-word-problems-200k
- meta-math/MetaMathQA
- MU-NLPC/Calc-ape210k
- MU-NLPC/Calc-mawps
- HuggingFaceTB/cosmopedia
- HuggingFaceTB/smollm-corpus
- mlfoundations/dclm-baseline-1.0
model-index:
- name: Boris-2
results:
- task:
type: text-generation
dataset:
name: HellaSwag
type: Rowan/hellaswag
split: validation
metrics:
- type: acc_norm
value: 50.26
name: acc_norm
source:
url: https://huggingface.co/opencerebral/Boris-2
name: Model card
- task:
type: text-generation
dataset:
name: ARC-Easy
type: allenai/ai2_arc
config: ARC-Easy
split: test
metrics:
- type: acc_norm
value: 59.81
name: acc_norm
source:
url: https://huggingface.co/opencerebral/Boris-2
name: Model card
- task:
type: text-generation
dataset:
name: ARC-Challenge
type: allenai/ai2_arc
config: ARC-Challenge
split: test
metrics:
- type: acc_norm
value: 32.0
name: acc_norm
source:
url: https://huggingface.co/opencerebral/Boris-2
name: Model card
- task:
type: text-generation
dataset:
name: PIQA
type: baber/piqa
split: validation
metrics:
- type: acc_norm
value: 72.91
name: acc_norm
source:
url: https://huggingface.co/opencerebral/Boris-2
name: Model card
- task:
type: text-generation
dataset:
name: ArithMark-3
type: AxiomicLabs/ArithMark-3.0
metrics:
- type: acc_norm
value: 45.1
name: acc_norm
source:
url: https://huggingface.co/opencerebral/Boris-2
name: Model card
---
Hugging Face |
Wiki |
Code
License: Apache 2.0 |
Authors: OpenCerebral
# Boris-2
Boris-2 is a 144M-parameter base language model from OpenCerebral, trained from scratch on a single NVIDIA V100. It was trained for 200B tokens, followed by two short final stages (40M tokens in total). It follows [Boris-2-Preview-1](https://huggingface.co/opencerebral/Boris-2-Preview-1) and [Boris-2-Preview-2](https://huggingface.co/opencerebral/Boris-2-Preview-2).
This is a base model. It continues text; it does not follow instructions or chat.
## Results
Nine-task comparison with other small models; the numbers are in [benchmarks_comparison.csv](./benchmarks_comparison.csv).
Zero-shot, float32 weights. ArithMark-3 uses the leaderboard's own `bencharithmark-3.py` (unmodified, default settings).
| Benchmark | Metric | Preview-2 | Boris-2 @200B | **Boris-2** |
|---|---|---:|---:|---:|
| HellaSwag | acc_norm | 46.60 | 50.36 | **50.26** |
| ARC-Easy | acc_norm | 59.13 | 60.10 | **59.81** |
| ARC-Challenge | acc_norm | 30.20 | 32.25 | **32.00** |
| PIQA | acc_norm | 70.84 | 72.91 | **72.91** |
| ArithMark-3 | acc_norm | 42.20 | 42.00 | **45.10** |
| **Open SLM Index** | | 30.58 | 33.59 | **34.19** |
The Index is the Open SLM Leaderboard formula: `(N(HellaSwag,25) + N(mean(ARC-E, ARC-C),25) + N(PIQA,50) + 0.65 * N(ArithMark-3,25)) / 3.65`, with `N(v,c) = 100(v-c)/(100-c)`.
Every benchmark run of Boris-2 (all 10B-token checkpoints of the main run, the previews, the 200B checkpoint and this release) is in [`benchmarks.csv`](./benchmarks.csv).
**How this checkpoint was chosen.** "Boris-2 @200B" is the end of the main run. The released weights add two short low-learning-rate stages on top of it (see Training). About ten such short final-stage candidates were tried (different data, seeds and learning rates, plus a math-neuron amplification experiment that was not kept), and this one was selected by its score on these five benchmarks. Single-checkpoint scores move by roughly ±0.8 Index, so part of the margin over the 200B checkpoint may be selection on noise. Almost all of the measured gain is ArithMark-3 (+3.1); the other four benchmarks changed by 0.3 points or less.
## Model
| Field | Value |
|---|---:|
| Parameters | 143.7M |
| Transformer blocks | 111.9M |
| Token embedding (tied with output) | 16.8M |
| N-gram table | 15.0M |
| Layers | 31 |
| Hidden size | 512 |
| Attention | 8 heads of 64, full multi-head |
| MLP | SwiGLU, inner 1664 |
| Context | 2048 tokens |
| Position encoding | RoPE, theta 10,000 |
| Vocabulary | 32,768 (byte-level BPE, trained for this model) |
Two components differ from a standard decoder:
- **Canon layers.** Each block applies a causal 3-tap depthwise convolution to the fused query/key/value projection, so every position mixes in the two previous positions before attention.
- **N-gram table.** An exact lookup on the last two and three token ids (210,938 bigrams and 23,437 trigrams, no hashing) retrieves a learned rank-64 vector that is added to the residual stream before block 2.
Training packed several documents per sequence with attention, Canon and the n-gram lookup all masked at document boundaries; the modeling code reproduces this (`eos_resets_context` in the config).
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "opencerebral/Boris-2"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
x = tok("The water cycle begins when", return_tensors="pt")
y = model.generate(**x, max_new_tokens=40, do_sample=False)
print(tok.decode(y[0], skip_special_tokens=True))
```
Plain PyTorch custom code, no extra dependencies, no KV cache. Run in float32 or bfloat16; the residual stream is kept in float32 internally.
## Training
| Field | Value |
|---|---:|
| Tokens | 200.0B main run + 40M final stages |
| Hardware | 1x NVIDIA V100 32GB, about 59,500 tokens per second (hand-written Volta attention and fused kernels) |
| Precision | float16 mixed precision, float32 residual stream |
| Batch | 524,288 tokens per step to 170B, then 1,048,576 (doubled after a pilot showed lower loss) |
| Sequence length | 2048 |
| Optimizer | Muon (matrices) and AdamW (embedding, n-gram table, norms, Canon) |
| Schedule | warmup, constant learning rate to 174B, then 1-sqrt decay to zero over 26B tokens |
Main run data: up to 55B tokens about 89% ClimbMix and 11% FineMath; from 55B about 90% ClimbMix, 10% UltraData-Math L3 and 0.15% Orca-Math. From 150B to 174B the mix ramps linearly to the decay mix.
Decay mix (174B to 196.5B), by tokens:
| Source | Share |
|---|---:|
| ClimbMix | 73.92% |
| UltraData-Math L3 | 12% |
| Cosmopedia v2 WikiHow-style tutorials | 10.9% |
| Cosmopedia v1 tutorials (HellaSwag-title filtered) | 1.1% |
| DCLM-Baseline | 1% |
| MetaMathQA (GSM8K-derived rows) | 0.75% |
| Calc-ape210k / Calc-mawps (training splits) | 0.18% |
| Orca-Math | 0.15% |
The last 3.5B tokens shift to Preview-2's decay mix (about 5% MetaMathQA and 1.5% elementary word problems).
Final stages, starting from the 200B checkpoint:
1. 30M tokens at 0.1x peak learning rate (1-sqrt to zero): mostly ClimbMix and Cosmopedia tutorials, plus UltraData-Math, MetaMathQA and Orca-Math.
2. 10M tokens at 0.02x peak learning rate (1-sqrt to zero): LibreMath v1 only, an OpenCerebral synthetic math word-problem set (not yet public).
## Contamination
No benchmark training, validation or test split was trained on. Every source except ClimbMix was filtered against every evaluation item (HellaSwag, ARC, PIQA, ArithMark-3) with a 13-gram overlap check; LibreMath additionally carries 8-gram bans against ArithMark-3, SVAMP and ASDiv, and the how-to data excludes HellaSwag's WikiHow titles. ClimbMix, the bulk of the data, has not been filtered.
## Limitations
A 144M-parameter base model. It makes factual and arithmetic mistakes, loses track of longer arguments, and repeats itself under greedy decoding. It has no safety tuning.
## Copyright & License
*Copyright 2026 OpenCerebral*
This project and all associated files (the "Work") are licensed under the Apache
License, Version 2.0 (the "License"); you may not use this project except in
compliance with the License. You may obtain a copy of the License at:
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.