| --- |
| license: apache-2.0 |
| language: |
| - en |
| pipeline_tag: text-generation |
| --- |
| |
|  |
|
|
| Sekai-1 is the first model in the Sekai series: a small language model trained from scratch on a single RTX 5060 Ti. |
|
|
| The model has 95.6M parameters, with 23 layers, a hidden size of 576 and a 2,048-token context window. It uses a Qwen3-style decoder-only architecture (grouped-query attention with 9 query heads and 3 key-value heads, a SwiGLU MLP, RMSNorm, RoPE, tied embeddings) and its own 24,576-token BPE tokenizer, trained on Sekai’s data mix, that splits numbers into single digits. |
|
|
| Sekai-1 is pretrained on 58.5B tokens of web, educational, math, code and conversation data, using the Muon and AdamW optimizers with a warmup-stable-decay learning-rate schedule. |
|
|
| ## Evaluation |
|
|
|  |
|
|
| The model scores number two in the under 100M parameters, just behind [KLM-1.5](https://huggingface.co/MinimaLabs/KLM-1.5) on the [SLM leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard) as seen in the image above. |
| All scores are zero-shot. Sekai-1's scores were measured locally on the same tasks the leaderboard uses; the other models' scores are taken from the leaderboard (November 7, 2026). |
|
|
| ## Training |
|
|
| **Pretraining:** 58.5B tokens of web, educational, math, code and conversation data, with a warmup-stable-decay schedule. The final ~12B tokens were a decay phase on a higher-quality mix. |
| **Chat fine-tuning (SFT):** about 400M tokens of instruction following, everyday chat, short and long answers, summarizing and rewriting, code writing and code fixing, plus a small set of identity examples, so the model knows it is Sekai-1. It does not use chain-of-thought (a model this size doesn't benefit from it), and refusals were filtered out of the data. |
|
|
| ## How to use |
|
|
| ```bash |
| pip install transformers torch |
| ``` |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "8BitStudio/Sekai-1" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id) |
| |
| messages = [{"role": "user", "content": "Who are you?"}] |
| inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_dict=True, return_tensors="pt") |
| outputs = model.generate(**inputs) |
| print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) |
| ``` |
|
|
| The model is small enough to run on a CPU. The default sampling settings (temperature 0.5, top-p 0.9, top-k 40, repetition penalty 1.1) are stored in `generation_config.json`; pass `do_sample=False` to `generate()` for deterministic answers. |
|
|
| Sekai-1 uses the ChatML format: |
|
|
| ``` |
| <|im_start|>user |
| What is the capital of France?<|im_end|> |
| <|im_start|>assistant |
| Paris<|im_end|> |
| ``` |
|
|
| ## Limitations |
|
|
| Sekai-1 is a 95.6M-parameter model. It makes factual mistakes, loses track in long conversations, and is weak at multi-step reasoning and math beyond simple arithmetic. |
| It has no safety tuning or refusal behaviour, so don't use it anywhere that needs either. It understands English only. Check responses for the things that matter. |
|
|
| ## |
