File size: 6,828 Bytes
f1b1d76 7f15382 f1b1d76 7f15382 f1b1d76 0568838 11111ab 0568838 11111ab 0568838 11111ab 7f15382 11111ab 7f15382 11111ab 0568838 11111ab 893c1c2 11111ab 0568838 11111ab f1b1d76 11111ab f1b1d76 11111ab f1b1d76 11111ab f1b1d76 11111ab | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 | ---
library_name: transformers
tags:
- weevil
- recurrent
- gated-deltanet
language:
- en
pipeline_tag: text-generation
datasets:
- HuggingFaceFW/fineweb-edu
- OptimalScale/ClimbMix
- IFM/TxT360-v2
- HuggingFaceTB/finemath
- HuggingFaceTB/smollm-corpus
- nvidia/OpenMathInstruct-2
- codeparrot/codeparrot-clean
---
# Weevil-Recurrent-16M-Base
A 16,085,568-parameter recurrent decoder with post-decay pretraining weights. [Weevil-Recurrent-16M-QA](https://huggingface.co/ELBennett/Weevil-Recurrent-16M-QA) is this model fine-tuned for multiple-choice questions.
## Checkpoints
| Model | Description |
|---|---|
| [ELBennett/Weevil-Recurrent-16M-Base](https://huggingface.co/ELBennett/Weevil-Recurrent-16M-Base) | Post-decay pretraining checkpoint. |
| [ELBennett/Weevil-Recurrent-16M-QA](https://huggingface.co/ELBennett/Weevil-Recurrent-16M-QA) | The Base checkpoint fine-tuned for multiple-choice questions. |
## Model details
| Property | Value |
|---|---|
| Parameters | 16,085,568 |
| Architecture | Causal Gated DeltaNet-2 |
| Vocabulary | 50,368 |
| Factorized embedding | 128 |
| Hidden size | 256 |
| Stack | 2-layer prelude + 4-layer core + 2-layer coda |
| Pretraining | 9.75B first-mix tokens + the entire 750M decay stage = 10.5B content tokens |
| Precision | Float32 stored weights; bfloat16 linears |
| Pause token | `[pause]`, token id 50285 |
## How to use
```bash
pip install torch transformers
```
### Generate
Pass unpadded rows: right-padding is consumed by the convolution.
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"ELBennett/Weevil-Recurrent-16M-Base", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("ELBennett/Weevil-Recurrent-16M-Base")
model.eval()
prompt = "The capital of France is"
prompt_ids = torch.tensor([tokenizer.encode(prompt, add_special_tokens=False)])
output_ids = model.generate(prompt_ids, max_new_tokens=16)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
```
### Score choices
Continue with the model and tokenizer above.
```python
choices = ["cat", "dog"]
prompt = """Which animal barks?
A. cat
B. dog
Answer:"""
pause_id = tokenizer.convert_tokens_to_ids("[pause]")
passes, n_pause = 16, 10
prompt_ids = torch.tensor(
[tokenizer.encode(prompt, add_special_tokens=False) + [pause_id] * n_pause]
)
answers = [tokenizer.encode(" " + choice, add_special_tokens=False) for choice in choices]
with torch.no_grad():
scores = model.score_continuations(prompt_ids, answers, core_passes=passes)
print(choices[int(scores.argmax())], [round(score, 3) for score in scores.tolist()])
```
The prompt and pauses are prefilled once. Each choice is scored from its own copy of the recurrent state.
### Reuse the recurrent state
This continues the `prompt_ids` from the scoring example. `generate` prefills the GDN2 state and convolution windows, then advances them one token at a time. `past_key_values` and `recurrent_state` refer to the same object: one state matrix and one convolution window per mixer, with a separate copy for every core pass.
```python
out = model(prompt_ids, use_cache=True)
next_id = out.logits[:, -1:].argmax(dim=-1)
nxt = model(next_id, past_key_values=out.recurrent_state, use_cache=True)
print(out.recurrent_state.seq_length, nxt.logits.shape)
```
Use `recurrent_state.clone()` or `recurrent_state.repeat(n)` to copy the state when computing token log-probabilities yourself.
### Inference speed
On an H100 at 16 core passes, after the prompt is prefilled, one new token takes 2.52 ms at batch 1. Compiling each block and replaying the token step from a CUDA graph reaches 259,000 tokens/s at batch 16,384 (100% GPU utilization, about 532 W, 31 GB). Prefill of a long prompt still runs the chunk schedule; decode does not recompute it.
## Pretraining mix
This checkpoint trained **9.75 billion tokens** on the first mix. Ten percent of the positions in each pretraining sequence were `[pause]` tokens, and those positions are not part of the content count.
Weight is the probability of drawing the next document from a dataset. Tokens are that weight times 9.75 billion. Each packed document also contributes one end-of-text token toward the count.
| Dataset | Subset | Weight | Tokens |
|---|---|---:|---:|
| [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) | | 25% | 2.4375B |
| [ClimbMix](https://huggingface.co/datasets/OptimalScale/ClimbMix) | | 30% | 2.925B |
| [TxT360-v2](https://huggingface.co/datasets/IFM/TxT360-v2) | `web-high-medium` | 20% | 1.95B |
| [FineMath](https://huggingface.co/datasets/HuggingFaceTB/finemath) | `finemath-4plus` | 10% | 975M |
| [Cosmopedia-v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | `cosmopedia-v2` | 8% | 780M |
| [OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2) | problem and generated solution | 5% | 487.5M |
| [codeparrot-clean](https://huggingface.co/datasets/codeparrot/codeparrot-clean) | Python source, autogenerated files skipped | 2% | 195M |
| **First mix** | | **100%** | **9.75B** |
## Decay mix
The entire decay stage trained **750 million tokens**, shifting weight toward ClimbMix, WikiHow-style tutorials, and math. Tokens are the weight times 750 million.
WikiHow and math rows are filters of [Cosmopedia-v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus): `format = wikihow`, and `seed_data = auto_math_text`.
| Dataset | Subset | Weight | Tokens |
|---|---|---:|---:|
| [ClimbMix](https://huggingface.co/datasets/OptimalScale/ClimbMix) | | 40% | 300M |
| [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) | | 15% | 112.5M |
| [Cosmopedia-v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | WikiHow tutorials | 15% | 112.5M |
| [OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2) | problem and generated solution | 12% | 90M |
| [FineMath](https://huggingface.co/datasets/HuggingFaceTB/finemath) | `finemath-4plus` | 10% | 75M |
| [Cosmopedia-v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | math (`auto_math_text`) | 8% | 60M |
| **Decay stage** | | **100%** | **750M** |
Together that is **10.5 billion** content tokens: 9.75 billion from the first mix and 750 million from the decay stage.
## Architecture
`core_passes` repeats only the core. Each core pass adds `input_scale * prelude_output` back onto the residual. Each mixer and MLP branch has a 1-lane read/write gate. Next-token logits go back through the factorized embedding. Linears run in bfloat16.
`[pause]` is token id 50285. It is a special token, so the word "pause" in ordinary text is not rewritten to it. The model does not learn to emit pauses. Put them in the input.
|