---
license: cc-by-nc-4.0
language:
  - en
pipeline_tag: text-generation
tags:
  - gguf
  - causal-lm
  - mixture-of-experts
  - recurrent
  - custom-architecture
  - preview
---

# Sabaki Pretrain 20B

This release continues pretraining from the earlier Sabaki model, adding **14.32B training tokens** to its original **8B**, for a total of **22.32B tokens**. That is about **2.8 times the original pretraining budget**, using the same 221.5M-parameter architecture. Sabaki remains a research preview intended for experimentation, evaluation, and further post-training.

Sabaki is our first attempt at a small, experimental language model. It was trained from scratch with a custom architecture. 

This upload is a research preview, not a polished assistant. It is here for people who want to inspect the design, run local experiments, or continue post-training. Expect rough edges, including uneven instruction following, repetition, factual mistakes, and sensitivity to prompts.

## At a glance

| | |
| --- | --- |
| Model | Decoder-only causal language model |
| Size | 221.5M trainable parameters |
| Active per token | 130.7M parameters, excluding the tied vocabulary matrix |
| Layers | 16 total: 12 recurrent and 4 global-attention layers |
| Hidden size | 768 |
| Attention heads | 12 x 64 |
| Experts | 8 routed experts (Top-2) and 2 shared experts |
| Vocabulary | 49,152-token lossless byte-level BPE |
| Configured context | Up to 16,384 tokens |
| Pretraining | 22,320,971,776 lifetime tokens (22.32B) |
| Checkpoint | Pretrained weights, step 559,111 |
| Formats | FP32 PyTorch checkpoint (`.pt`) and FP16 GGUF |
| License | CC BY-NC 4.0 |

The 16,384-token context is a configuration limit, not a promise of reliable recall across the full window. Initial training used a curriculum of 512, 2,048, and 4,096 tokens; the continued pretraining checkpoint records a sequence length of 1,024. Behavior beyond 4K has not been established.

## How it works

Most layers use Kimi Delta Attention (KDA), which carries a fixed-size recurrent state forward as it reads. Every fourth layer uses Multi-head Latent Attention (MLA) with rotary position embeddings, giving the model periodic access to the full preceding context. This keeps most sequence processing linear while still allowing global attention.

The feed-forward side is sparse in all but the first block. For each token, the router selects two of eight routed experts and combines them with two shared experts. Persistent routing biases help keep expert use balanced without adding a separate balancing loss.

The complete layer schedule is:

| Blocks | Sequence mixer | Feed-forward path |
| --- | --- | --- |
| 1 | KDA | Dense SiTU-GLU, width 2,048 |
| 2-3 | KDA | Top-2 routed MoE + 2 shared experts |
| 4 | Rotary MLA | Top-2 routed MoE + 2 shared experts |
| 5-16 | Three KDA blocks followed by one rotary MLA block, repeated | Top-2 routed MoE + 2 shared experts |

Input and output embeddings are tied, and every block uses pre-normalization and ordinary residual connections.

## Running the model

Public filenames and display metadata use **Sabaki**. Internal `kurokiba` architecture, checkpoint-format, and tokenizer identifiers are retained for compatibility with the existing loader; the model weights and tokenizer are unchanged. The `.pt` contains model weights and release metadata, without optimizer state.

The model repository includes the model-only FP32 PyTorch checkpoint (`Sabaki-Pretrain-20B.pt`) and an FP16 GGUF export (`Sabaki-Pretrain-20B-F16.gguf`). The GGUF file contains custom tensors and requires the [Sabaki Preview llama.cpp runtime](https://github.com/Local-Axiom-AI/Sabaki-Preview-Llama-cpp). It does not currently run in stock `transformers` or upstream `llama.cpp`.

Clone and build the companion runtime, then start its launcher with `Sabaki-Pretrain-20B-F16.gguf` as the model. The launcher provides a browser UI and an OpenAI-compatible local API. See the runtime repository for platform-specific build and launch instructions.

This is a pretrained completion model. A role-label prompt can be tried, but instruction following is not established for this checkpoint:

```text
USER: Explain why the sky appears blue.
ASSISTANT:
```

A conservative place to start is temperature `0.1`, top-k `50`, and repeat penalty `1.15`. These are practical defaults, not universally optimal settings.

## Evaluation

Sabaki was pretrained on **22.32B tokens**. The released model achieves an Intelligence Index of **14.6283** on the Open SLM benchmarks.

| Benchmark | Questions | Zero-shot normalized accuracy |
| --- | ---: | ---: |
| HellaSwag | 10,042 | 32.04% |
| ARC-Easy | 2,376 | 42.26% |
| ARC-Challenge | 1,172 | 26.37% |
| PIQA | 1,838 | 60.77% |
| ArithMark-3 | 1,000 | 36.60% |

The evaluation used an unquantized FP16 export and a custom native likelihood evaluator. These are not independently verified leaderboard results. The prompts were zero-shot completion prompts rather than the chat format shown above. ArithMark-3 also has a documented middle-answer shortcut that should be kept in mind when reading its score.

### Leaderboard context

The table below compares Sabaki with several models from the [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard). It is a comparison, not a claimed leaderboard placement: the leaderboard requires fewer than 150M total parameters, while Sabaki has 221.5M total parameters and 130.7M active non-embedding parameters per token.

| Model | Parameters | HellaSwag | ARC-Easy | ARC-Challenge | PIQA | ArithMark-3 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| **Sabaki Pretrain 20B** | **221.5M total / 130.7M active** | **32.04%** | **42.26%** | **26.37%** | **60.77%** | **36.60%** |
| [GPT-2](https://huggingface.co/openai-community/gpt2) | 124M | 31.26% | 39.35% | 22.35% | 62.08% | 35.70% |
| [OPT-125M](https://huggingface.co/facebook/opt-125m) | 125M | 31.31% | 40.28% | 22.70% | 62.24% | 35.20% |
| [GTM-v2-chat](https://huggingface.co/OPENGCM/GTM-v2-chat) | 119.5M | 30.70% | 40.49% | 25.17% | 63.98% | 36.60% |
| [SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) | 135M | 43.22% | 58.63% | 29.69% | 68.44% | 39.20% |

The released checkpoint has 22.32B lifetime pretraining tokens. Comparisons are not controlled for model design, training data, tokenization, objectives, or evaluation implementation.

## Good uses for this preview

Sabaki is a good fit for:

- studying compact recurrent/attention hybrids;
- experimenting with sparse mixture-of-experts models;
- local, non-commercial text-generation projects;
- evaluation and further post-training by researchers comfortable with custom runtimes.

It is not a good fit for autonomous decisions or situations where a wrong answer could cause medical, legal, financial, physical, or other serious harm. The model has not received comprehensive safety evaluation or preference optimization, and it can invent facts, citations, code, and tool results. English dominates its training data; other languages have not been evaluated separately.

## License

The model weights are available under the **Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)** license. Attribution is required, and commercial use is not permitted. Software and third-party datasets keep their own licenses; the model license does not replace those terms.

## Citation

If Sabaki helps your research, please cite this model repository. A formal citation will be added with a future full release.

```bibtex
@software{sabaki_preview_2026,
  author = {{Local-Axiom-AI}},
  title  = {Sabaki Preview},
  year   = {2026},
  note   = {A compact hybrid recurrent, latent-attention, and sparse mixture-of-experts language model}
}
```