8BitStudio commited on
Commit
fa03aac
·
verified ·
1 Parent(s): c13d1ec

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +30 -0
README.md CHANGED
@@ -1,3 +1,33 @@
1
  ---
2
  license: apache-2.0
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ language:
4
+ - en
5
+ pipeline_tag: text-generation
6
  ---
7
+
8
+ ![Sekai-1](Sekai-1.png)
9
+
10
+ Sekai-1 is the first model in the Sekai series: a small language model trained from scratch on a single RTX 5060 Ti.
11
+
12
+ The model has 95.6M parameters, with 23 layers, a hidden size of 576 and a 2,048-token context window. It uses a Qwen3-style decoder-only architecture (grouped-query attention with 9 query heads and 3 key-value heads, a SwiGLU MLP, RMSNorm, RoPE, tied embeddings) and its own 24,576-token BPE tokenizer, trained on Sekai’s data mix, that splits numbers into single digits.
13
+
14
+ Sekai-1 is pretrained on 58.5B tokens of web, educational, math, code and conversation data, using the Muon and AdamW optimizers with a warmup-stable-decay learning-rate schedule.
15
+
16
+ ## Evaluation
17
+
18
+ ![Benchmark](Leaderboard.png)
19
+
20
+ The model scores number two in the under 100M parameters, just behind [KLM-1.5](https://huggingface.co/MinimaLabs/KLM-1.5) on the [SLM leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard) as seen in the image above.
21
+ All scores are zero-shot. Sekai-1's scores were measured locally on the same tasks the leaderboard uses; the other models' scores are taken from the leaderboard (November 7, 2026).
22
+
23
+ ## Training
24
+
25
+ **Pretraining:** 58.5B tokens of web, educational, math, code and conversation data, with a warmup-stable-decay schedule. The final ~12B tokens were a decay phase on a higher-quality mix.
26
+ **Chat fine-tuning (SFT):** about 400M tokens of instruction following, everyday chat, short and long answers, summarizing and rewriting, code writing and code fixing, plus a small set of identity examples, so the model knows it is Sekai-1. It does not use chain-of-thought (a model this size doesn't benefit from it), and refusals were filtered out of the data.
27
+
28
+ ## Limitations
29
+
30
+ Sekai-1 is a 95.6M-parameter model. It makes factual mistakes, loses track in long conversations, and is weak at multi-step reasoning and math beyond simple arithmetic.
31
+ It has no safety tuning or refusal behaviour, so don't use it anywhere that needs either. It understands English only. Check responses for the things that matter.
32
+
33
+ ##