README.md

---
pipeline_tag: text-generation
language:
  - en
tags:
  - lm-engine
  - mamba2
  - moe
  - hybrid
  - base-model
---

Rigel is a 2.3B-total, 360M-active-parameter hybrid Mamba-2 + attention mixture-of-experts base model, trained with [LM Engine](https://github.com/open-lm-engine/lm-engine). It used about 117× fewer pretraining FLOPs than Llama-3.2-1B/3B and 53× fewer than Granite-4.2-3B, while landing within a few points of Granite-4.2-3B and Llama-3.2-3B on zero-shot averages and above Llama-3.2-1B.

Read the full write-up, including architecture, data curriculum, the multi-chip training infrastructure, and evaluations: **[Rigel Base: Reaching Llama-3.2 Quality with <1% of its Compute](https://open-lm-engine.github.io/blog/rigel/)**.

This repo is the 4,096-token-context pretrained checkpoint. For the 294,912-token long-context extension, see [`open-lm-engine/rigel-base-long`](https://huggingface.co/open-lm-engine/rigel-base-long). Both are in the [Rigel collection](https://huggingface.co/collections/open-lm-engine/rigel).

## Model summary

| | |
|---|---|
| Total / active parameters | 2.3B / 360M (260M active non-embedding) |
| Layers | 40 (30 Mamba-2, 10 grouped-query attention) |
| Attention | GQA (16 heads, 4 KV heads), 4,096-token sliding window, exclusive self attention (XSA), no positional embeddings |
| Feed-forward | Fine-grained MoE: 128 SwiGLU experts, intermediate size 128, top-2 routing |
| Hidden size | 1,024 |
| Vocabulary | 100,352 tokens, tied input/output embeddings |
| Pretraining context | 4,096 tokens |

This is a base model: it has no chat template and is not instruction-tuned.

## Usage with LM Engine

Rigel's architecture lives in LM Engine. Installing it and importing `lm_engine.training` registers the architecture with `transformers`' `Auto` classes, after which the usual Hugging Face API works.

```bash
git clone https://github.com/open-lm-engine/lm-engine.git && cd lm-engine && uv sync --extra cuda
```

```python
import torch
import lm_engine.training  # registers Rigel's architecture with transformers' Auto classes
from lm_engine.training.kernels import Kernel, enable_kernels
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "open-lm-engine/rigel-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map={"": "cuda"}).eval()

inputs = tokenizer("Rigel is a blue supergiant star in the constellation", return_tensors="pt").to("cuda")
with torch.no_grad(), enable_kernels([Kernel.mamba2_ssm, Kernel.causal_conv1d, Kernel.sonicmoe, Kernel.flash_attention_3]):
    output_ids = model.generate(**inputs, max_new_tokens=64, do_sample=False)

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
```

Any kernel left out of the list falls back to a pure-PyTorch implementation, so the snippet also runs with an empty list, just slower.

To finetune or continue pretraining, point an LM Engine training config at this checkpoint; see the [LM Engine README](https://github.com/open-lm-engine/lm-engine#quick-start).

## Citation

```bibtex
@misc{mishra2026rigel,
  title  = {Rigel Base: Reaching Llama-3.2 Quality with <1% of its Compute},
  author = {Mishra, Mayank and Runwal, Bharat and Stoica, Ion and Dao, Tri and Gonzalez, Joseph E.},
  year   = {2026},
  url    = {https://open-lm-engine.github.io/blog/rigel/}
}
```