Update README.md

README.md

@@ -1,3 +1,137 @@
 ---
 license: cc-by-nc-4.0
+language:
+  - en
+pipeline_tag: text-generation
+tags:
+  - gguf
+  - causal-lm
+  - mixture-of-experts
+  - recurrent
+  - custom-architecture
+  - preview
 ---
+
+# Sabaki Preview
+
+Sabaki is a small, experimental language model from **Local-Axiom-AI**. It was trained from scratch to explore a simple question: how much can a compact model do when recurrent memory, occasional global attention, and sparse experts work together?
+
+This upload is a research preview, not a polished assistant. It is here for people who want to inspect the design, run local experiments, or continue post-training. Expect rough edges, including uneven instruction following, repetition, factual mistakes, and sensitivity to prompts.
+
+## At a glance
+
+| | |
+| --- | --- |
+| Model | Decoder-only causal language model |
+| Size | 221.5M trainable parameters |
+| Active per token | 130.7M parameters, excluding the tied vocabulary matrix |
+| Layers | 16 total: 12 recurrent and 4 global-attention layers |
+| Hidden size | 768 |
+| Attention heads | 12 x 64 |
+| Experts | 8 routed experts (Top-2) and 2 shared experts |
+| Vocabulary | 49,152-token lossless byte-level BPE |
+| Configured context | Up to 16,384 tokens |
+| Pretraining | 8.0B tokens |
+| Post-training | About 23.9M assistant-target tokens |
+| Formats | FP16 PyTorch checkpoint (`.pt`) and FP16 GGUF |
+| License | CC BY-NC 4.0 |
+
+The 16,384-token context is a configuration limit, not a promise of reliable recall across the full window. Training used a curriculum of 512, 2,048, and 4,096 tokens, so behavior beyond 4K has not been established.
+
+## Known issue: runaway generation
+
+Sabaki has a serious tendency to answer a prompt and then continue generating unrelated or nonsensical text instead of stopping. In some cases it will keep going until it reaches the generation limit. This is a model behavior issue, not just a sampling quirk.
+
+Always set a maximum token limit when running the preview, and do not rely on it to stop cleanly on its own. A repetition penalty can help, but it does not eliminate the problem.
+
+## How it works
+
+Most layers use Kimi Delta Attention (KDA), which carries a fixed-size recurrent state forward as it reads. Every fourth layer uses Multi-head Latent Attention (MLA) with rotary position embeddings, giving the model periodic access to the full preceding context. This keeps most sequence processing linear while still allowing global attention.
+
+The feed-forward side is sparse in all but the first block. For each token, the router selects two of eight routed experts and combines them with two shared experts. Persistent routing biases help keep expert use balanced without adding a separate balancing loss.
+
+![Diagram of Sabaki's decoder blocks and layer schedule](./sabaki-architecture.png)
+
+The complete layer schedule is:
+
+| Blocks | Sequence mixer | Feed-forward path |
+| --- | --- | --- |
+| 1 | KDA | Dense SiTU-GLU, width 2,048 |
+| 2-3 | KDA | Top-2 routed MoE + 2 shared experts |
+| 4 | Rotary MLA | Top-2 routed MoE + 2 shared experts |
+| 5-16 | Three KDA blocks followed by one rotary MLA block, repeated | Top-2 routed MoE + 2 shared experts |
+
+Input and output embeddings are tied, and every block uses pre-normalization and ordinary residual connections.
+
+## Running the model
+
+The model repository includes both the original FP16 PyTorch checkpoint (`.pt`) and an FP16 GGUF export. The GGUF file contains custom tensors and requires the [Sabaki Preview llama.cpp runtime](https://github.com/Local-Axiom-AI/Sabaki-Preview-Llama-cpp). It does not currently run in stock `transformers` or upstream `llama.cpp`.
+
+Clone and build the companion runtime, then start its launcher with `Sabaki-Preview-F16.gguf` as the model. The launcher provides a browser UI and an OpenAI-compatible local API. See the runtime repository for platform-specific build and launch instructions.
+
+For a simple conversation, use plain role labels:
+
+```text
+USER: Explain why the sky appears blue.
+ASSISTANT:
+```
+
+A conservative place to start is temperature `0.1`, top-k `50`, and repeat penalty `1.15`. These are practical defaults, not universally optimal settings.
+
+## Evaluation
+
+The table below comes from the nearest fully evaluated training snapshot. That snapshot had completed the same 8,000,045,056 pretraining tokens and 20,016,551 assistant-target post-training tokens. The weights in this upload are from a slightly later post-training checkpoint, so treat these numbers as a useful reference rather than exact scores for this file.
+
+| Benchmark | Questions | Zero-shot normalized accuracy |
+| --- | ---: | ---: |
+| HellaSwag | 10,042 | 31.34% |
+| ARC-Easy | 2,376 | 40.49% |
+| ARC-Challenge | 1,172 | 25.34% |
+| PIQA | 1,838 | 60.72% |
+| ArithMark-3 | 1,000 | 36.60% |
+
+The evaluation used an unquantized FP16 export and a custom native likelihood evaluator. These are not independently verified leaderboard results. The prompts were zero-shot completion prompts rather than the chat format shown above. ArithMark-3 also has a documented middle-answer shortcut that should be kept in mind when reading its score.
+
+### Leaderboard context
+
+Taken as the first end-to-end training run of a new model design, this is a strong result. The table below places Sabaki beside several familiar models from the [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard). It is a comparison, not a claimed leaderboard placement: the leaderboard requires fewer than 150M total parameters, while Sabaki has 221.5M total parameters and 130.7M active non-embedding parameters per token.
+
+| Model | Parameters | HellaSwag | ARC-Easy | ARC-Challenge | PIQA | ArithMark-3 |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| **Sabaki Preview** | **221.5M total / 130.7M active** | **31.34%** | **40.49%** | **25.34%** | **60.72%** | **36.60%** |
+| [GPT-2](https://huggingface.co/openai-community/gpt2) | 124M | 31.26% | 39.35% | 22.35% | 62.08% | 35.70% |
+| [OPT-125M](https://huggingface.co/facebook/opt-125m) | 125M | 31.31% | 40.28% | 22.70% | 62.24% | 35.20% |
+| [GTM-v2-chat](https://huggingface.co/OPENGCM/GTM-v2-chat) | 119.5M | 30.70% | 40.49% | 25.17% | 63.98% | 36.60% |
+| [SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) | 135M | 43.22% | 58.63% | 29.69% | 68.44% | 39.20% |
+
+Sabaki scores higher than GPT-2 and OPT-125M on four of these five benchmarks. Against GTM-v2-chat, it is ahead on HellaSwag and ARC-Challenge, tied on ARC-Easy and ArithMark-3, and behind on PIQA. SmolLM2-135M remains clearly stronger across the board and is a useful picture of how much headroom remains.
+
+The training budget makes that result notable. Sabaki saw 8B pretraining tokens, compared with 180B reported for OPT and 2T for SmolLM2-135M, or about 22 times and 250 times as many, respectively. Its post-training was also small at roughly 23.9M assistant-target tokens; for context, GTM-v2-chat reports about 1B SFT token exposures over several passes. These are not perfectly controlled comparisons because model design, data quality, tokenization, training objectives, and evaluation implementations all differ. Even so, the results suggest the core design learned useful capabilities unusually early.
+
+## Good uses for this preview
+
+Sabaki is a good fit for:
+
+- studying compact recurrent/attention hybrids;
+- experimenting with sparse mixture-of-experts models;
+- local, non-commercial text-generation projects;
+- evaluation and further post-training by researchers comfortable with custom runtimes.
+
+It is not a good fit for autonomous decisions or situations where a wrong answer could cause medical, legal, financial, physical, or other serious harm. The model has not received comprehensive safety evaluation or preference optimization, and it can invent facts, citations, code, and tool results. English dominates its training data; other languages have not been evaluated separately.
+
+## License
+
+The model weights are available under the **Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)** license. Attribution is required, and commercial use is not permitted. Software and third-party datasets keep their own licenses; the model license does not replace those terms.
+
+## Citation
+
+If Sabaki helps your research, please cite this model repository. A formal citation will be added with a future full release.
+
+```bibtex
+@software{sabaki_preview_2026,
+  author = {{Local-Axiom-AI}},
+  title  = {Sabaki Preview},
+  year   = {2026},
+  note   = {A compact hybrid recurrent, latent-attention, and sparse mixture-of-experts language model}
+}
+```