Ultron-Small Baseline

Recurrent-Depth Transformer pretrained on FineWeb-Edu

Architecture

Ultron is a Recurrent-Depth Transformer (RDT) that loops a small set of transformer layers multiple times per forward pass, enabling "thinking in latent space" β€” implicit chain-of-thought reasoning without generating intermediate tokens.

Input β†’ [Embedding + RoPE] β†’ [Prelude (2 layers)] β†’ [LN(e)] β†’ [RecurrentBlock Γ—8 loops (4 layers/loop)] β†’ [CΒ·h] β†’ [Coda (2 layers)] β†’ [LM Head]
Property Value
Total Parameters 88.9M
Non-embedding Parameters 50.4M
Effective Depth 36 layers (8 loops Γ— 4 layers + 2 prelude + 2 coda)
Attention GQA (12 heads, 4 KV heads)
Hidden Dimension 768
Sequence Length 1024
MoE No (dense FFN)
Spectral Radius ρ(A) < 1 by construction (LTI stability)

Training

  • Dataset: FineWeb-Edu (sample-10BT)
  • Optimizer: AdamW (Ξ²1=0.9, Ξ²2=0.95, Ξ΅=1e-8, weight decay=0.1)
  • LR Schedule: Linear warmup (1000 steps) + cosine decay (3e-4 β†’ 3e-5)
  • Batch Size: 8 Γ— 8 gradient accumulation = 64 effective
  • Precision: bf16
  • Depth Sampling: Parcae geometric per-sequence (ΞΌ=8)
  • Gradient Checkpointing: Enabled

Key Research Papers

Usage

from ultron.model import Ultron, UltronConfig
import torch

# Load from checkpoint
ckpt = torch.load("ultron_final.pt", weights_only=False)
cfg = UltronConfig(**ckpt["config"])
model = Ultron(cfg)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()

# Inference with depth extrapolation
logits = model(input_ids, n_loops=8)   # standard
logits = model(input_ids, n_loops=16)  # deeper reasoning

Repository

trojan0x/ultron β€” Full source code, training scripts, and notebook

Downloads last month
8
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train trojan0x/ultron-small-baseline

Papers for trojan0x/ultron-small-baseline