Pulse-10M v1
Pulse-10M v1 is an approximately 9.7M-parameter causal language model trained from scratch.
Architecture
Pulse uses a sequential hierarchical recurrent architecture:
Token embedding → Stem → recurrent Fast → Medium → Slow hierarchy → Exit → LM head
Within each recurrent cycle, the model computes:
Fast → optionally Medium → optionally Slow
Two learned depth-routing probabilities control how much of the hierarchy is used.
The three possible cycle depths are blended during training:
- Fast only
- Fast → Medium
- Fast → Medium → Slow
An ACT-style halting controller determines how many recurrent cycles are used, with a maximum of 4 cycles.
The embedding matrix and LM output matrix are weight tied.
Core dimensions
| Parameter | Value |
|---|---|
d_model |
288 |
| Attention heads | 6 |
d_ff |
768 |
| Maximum recurrent cycles | 4 |
| Context length | 512 |
| Vocabulary | ~16,000 |
| Dropout | 0 |
| Normalization | RMSNorm |
| Position encoding | RoPE |
| FFN | SwiGLU |
First pretraining phase
The first training phase contains exactly:
1,000,000,000 next-token targets
Training is one-pass. Packed training blocks are consumed once rather than repeated over conventional epochs.
Sequence length is 512 prediction targets using non-overlapping 513-token raw blocks.
Data mixture
| Source | Target tokens |
|---|---|
| Ultra-FineWeb | 550M |
| FinePDFs Edu | 180M |
| peS2o | 80M |
| FineMath | 60M |
| Stack v2 | 60M |
| Wikipedia | 40M |
| Synthetic algorithmic data | 30M |
Documents reserved for tokenizer training and validation are excluded from training.
A persistent exact-document deduplication ledger is also used.
Architecture curriculum
The adaptive architecture is introduced progressively.
0–50M tokens
- Full Fast → Medium → Slow depth
- Fixed 2 recurrent cycles
50–150M
- Depth floor decreases from 1.0 → 0.25
- Recurrent count remains fixed
150–300M
- Depth floor decreases from 0.25 → 0
- Adaptive halting progressively activates
300M+
- Depth routing fully learned
- Halting fully adaptive
300–500M
The compute penalty ramps to its full training strength.
Learning-rate schedule
Peak learning rate:
2e-3
Warmup:
0–10M tokens
Repeated WSD-S-style cooldowns:
| Interval | LR floor |
|---|---|
| 225M → 250M | 2e-4 |
| 475M → 500M | 2e-4 |
| 725M → 750M | 2e-4 |
| 925M → 1B | 4e-5 |
After each intermediate cooldown, the learning rate returns directly to the 2e-3 stable learning rate.
The final cooldown does not restart.
Repository contents
This repository contains the full first-billion-token experiment, potentially including:
- final model weights
- full PyTorch checkpoint
- tokenizer
- model configuration
- training history
- boundary evaluations
- training summary
- periodic checkpoints
- boundary checkpoints
- streamed corpus state
- validation state
- data source schedule
- deduplication ledger
- training code
The .safetensors file contains model weights only.
The .pt checkpoints additionally contain optimizer/training state and are
intended for continuation/resume.
Loading
Pulse-10M v1 is currently a custom PyTorch architecture rather than a standard Transformers architecture.
Use the included Pulse architecture/training code to instantiate the model and
load either the state dictionary from the .safetensors file or the complete
training checkpoint from the .pt file.
Training philosophy
The first billion-token phase was designed around:
- unique-data consumption
- no intentional epoch replay
- exact cross-source deduplication
- progressive streaming
- adaptive recurrent computation
- periodic LR cooldowns
- checkpointed continuation