Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+little_kedi.jpg filter=lfs diff=lfs merge=lfs -text
+LittleKedi-tiny-instruct-f16.gguf filter=lfs diff=lfs merge=lfs -text
+LittleKedi-tiny-instruct-q8_0.gguf filter=lfs diff=lfs merge=lfs -text

LittleKedi-tiny-instruct-f16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77369781414494db5fcb4c258e794eea7d071e30ad79b2bd049ba75345f2f4e0
+size 6367232

LittleKedi-tiny-instruct-q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a7ce65ee25544f185ca8119a2491fcd2e05e3d1909a7e75165938650bd79ad7e
+size 3528512

README.md

@@ -0,0 +1,295 @@
+---
+license: apache-2.0
+language:
+- en
+pipeline_tag: text-generation
+datasets:
+- HuggingFaceFW/fineweb-edu
+- HuggingFaceTB/cosmopedia
+- HuggingFaceTB/finemath
+- HuggingFaceTB/smoltalk
+- codeparrot/github-code-clean
+base_model: edededdy/LittleKedi-tiny-base
+base_model_relation: finetune
+tags:
+- mixture-of-experts
+- moe
+- deepseek
+- multi-head-latent-attention
+- mla
+- from-scratch
+- tiny
+- littlekedi
+- chat
+- instruction-tuned
+---
+
+<p align="center"><img src="little_kedi.jpg" alt="LittleKedi logo" width="640"></p>
+
+# LittleKedi-tiny-instruct
+
+`LittleKedi-tiny-instruct` is the **instruction-tuned (chat) version of [LittleKedi-tiny-base](https://huggingface.co/edededdy/LittleKedi-tiny-base)**, a **small DeepSeek-V3-style Mixture-of-Experts language model trained from scratch**. It has
+**2.0M parameters** in total, of which **1.5M are active per token**
+(0.50M excluding the embedding table). It was pretrained on a 1,000M-token mix (FineWeb-Edu, Cosmopedia (Stanford), Cosmopedia (OpenStax), Cosmopedia (WikiHow), Cosmopedia (Khan Academy), FineMath 4+, SmolTalk, github-code-clean).
+
+*Part of the LittleKedi family. Not affiliated with DeepSeek.*
+
+This is the **instruction-tuned (chat) model**. It answers in a chat format and stops when it's done, but it's still a very small model: answers are fluent and on-topic but often **wrong or vague**, and it **invents facts confidently**. It's an educational and research artifact; don't use it for anything that matters.
+
+## Chat format
+
+```
+<|system|>You are a helpful assistant.<|end|><|user|>What causes the seasons?<|end|><|assistant|>
+```
+
+The model writes its reply and ends it with `<|end|>`. The system turn is optional. The GGUF files embed this
+template, and `chat_template.jinja` / `tokenizer_config.json` carry it for other tools.
+
+## The LittleKedi family
+
+LittleKedi is a family of small DeepSeek-V3-style Mixture-of-Experts language models trained from
+scratch. Every size comes in three variants that differ in **how sparse** they are: how many routed experts
+they have, and what share of their parameters a token actually passes through.
+
+| Variant | Routed experts |
+|---|---|
+| **standard** (`<size>`) | a few wide experts; the densest variant |
+| **sparse** (`<size>-sparse`) | many narrower, fine-grained experts, more of them active per token |
+| **wide-sparse** (`<size>-wide-sparse`) | more experts again; the largest total and smallest active share |
+
+More experts means more total capacity to store knowledge, but each expert
+sees fewer tokens (tokens × top-k ÷ experts per layer), so sparser variants need more training data before
+their experts are well trained. The variants of a size share a tokenizer and dataset, so they can be
+compared directly. Shapes change as the family scales; the table below is generated from the current
+presets.
+
+This card is for **`tiny`**, the **standard** tiny model: 1.5M of its 2.0M parameters (78%) are active per token. Its siblings are `tiny-sparse` and `tiny-wide-sparse`.
+
+| Model | Variant | Vocab | Total | Active | Active non-emb. | Active % | Routed experts |
+|---|---|---|---|---|---|---|---|
+| **`tiny` ← this model** | **standard** | **8K** | **2.0M** | **1.5M** | **0.5M** | **78%** | **8 × 64, top-2** |
+| `tiny-sparse` | sparse | 8K | 2.6M | 1.6M | 0.5M | 60% | 32 × 32, top-4 |
+| `tiny-wide-sparse` | wide-sparse | 8K | 3.8M | 1.6M | 0.5M | 42% | 64 × 32, top-4 |
+| `mini` | standard | 8K | 9.4M | 4.8M | 3.2M | 51% | 12 × 128, top-3 |
+| `mini-sparse` | sparse | 8K | 19.8M | 4.8M | 3.2M | 24% | 64 × 64, top-6 |
+| `mini-wide-sparse` | wide-sparse | 8K | 36.4M | 4.9M | 3.3M | 13% | 128 × 64, top-6 |
+| `small` | standard | 8K | 12.2M | 6.3M | 4.2M | 52% | 16 × 128, top-4 |
+| `small-sparse` | sparse | 8K | 24.1M | 6.4M | 4.3M | 27% | 80 × 64, top-8 |
+| `small-wide-sparse` | wide-sparse | 8K | 43.9M | 6.5M | 4.4M | 15% | 160 × 64, top-8 |
+| `moderate` | standard | 16K | 49.8M | 18.8M | 12.5M | 38% | 24 × 192, top-4 |
+| `moderate-sparse` | sparse | 16K | 105.8M | 19.1M | 12.8M | 18% | 120 × 96, top-8 |
+| `moderate-wide-sparse` | wide-sparse | 16K | 199.0M | 19.4M | 13.1M | 10% | 240 × 96, top-8 |
+| `medium` | standard | 16K | 85.9M | 30.8M | 22.4M | 36% | 24 × 256, top-4 |
+| `medium-sparse` | sparse | 16K | 185.3M | 31.2M | 22.8M | 17% | 120 × 128, top-8 |
+| `medium-wide-sparse` | wide-sparse | 16K | 350.9M | 31.6M | 23.2M | 9% | 240 × 128, top-8 |
+| `base` | standard | 32K | 258.7M | 105.4M | 80.2M | 41% | 32 × 256, top-6 |
+| `base-sparse` | sparse | 32K | 542.8M | 106.3M | 81.2M | 20% | 160 × 128, top-12 |
+| `base-wide-sparse` | wide-sparse | 32K | 1015.9M | 107.6M | 82.4M | 11% | 320 × 128, top-12 |
+
+"Active" counts everything a token passes through: embeddings, attention, the dense layer, shared
+experts and its top-k routed experts. "Active non-embedding" leaves out the embedding table (tied with the
+output head), which costs memory but almost no compute; it's the fair number for comparing compute
+between models.
+
+## Architecture
+
+| Component | This model |
+|---|---|
+| Parameters | 1.99M total, 1.55M active per token (78%), 0.50M active non-embedding |
+| Layers / hidden size | 4 / 128 (layer 0 has a dense SwiGLU FFN of 352; layers 1–3 are MoE) |
+| Attention | **Multi-head Latent Attention (MLA)**: 4 heads, KV latent 32, decoupled RoPE dim 16, head dims 16+16 (q/k) and 16 (v). The KV cache stores only the 32-dim latent plus a 16-dim RoPE key per token |
+| MoE | **DeepSeekMoE**: 8 routed experts (SwiGLU, 64 hidden), top-2, plus 1 shared expert; sigmoid gating |
+| Load balancing | **Auxiliary-loss-free** bias balancing (update speed 0.001) plus a small sequence-wise loss (α = 0.0001). Over the second half of training the median batch put 1.07× the mean load on the busiest expert (1.08× in the worst layer), and 0.00% of expert assignments were dropped for capacity. |
+| Expert dispatch | Training: fixed capacity (capacity factor 2.0; overflow assignments are dropped). Inference is always dropless |
+| Embeddings | Input embedding and output head **tied** |
+| Context | 1,024 tokens |
+
+## Training
+
+| | |
+|---|---|
+| Data | 1,000M tokens, 801,792 documents (~502 tokens per parameter, ~646 per active parameter), one pass, sources interleaved throughout (mix below) |
+| Tokenizer | Byte-level BPE, 8,196 tokens (~3.75 bytes/token on this mix); digits not split (v1 tokenizer); chat special tokens built in |
+| Steps | 15,258 × 65,536 tokens (batch 16 × 1024 × 4 accumulation) |
+| Optimizer | **Muon** (momentum 0.95, Nesterov, 5 Newton-Schulz steps; each expert's matrix orthogonalized separately) for hidden matrices, AdamW (β = 0.9, 0.95) for embeddings, norms and the router; weight decay 0.1, gradient clipping 1.0 |
+| Learning rate | 2e-3 peak; **WSD**: 200 warm-up steps, constant, then linear decay to 0 over the last 20% of steps (schedule set for 15,258 steps) |
+| Precision | bf16 autocast, `torch.compile` |
+| Compute | ~3e15 FLOPs (6 × active non-embedding parameters × tokens) |
+| Hardware | One AMD Radeon 890M integrated GPU (Ryzen AI 300 laptop), ROCm on Windows: about 8.9 hours at a median ~31.1K tokens/s |
+
+| Source | What it is | Tokens | Share |
+|---|---|---|---|
+| FineWeb-Edu `sample/10BT` | educational web pages | 700M | 70% |
+| Cosmopedia (Stanford) | synthetic textbooks | 70M | 7% |
+| Cosmopedia (OpenStax) | synthetic textbooks | 30M | 3% |
+| Cosmopedia (WikiHow) | synthetic how-to articles | 30M | 3% |
+| Cosmopedia (Khan Academy) | synthetic lessons | 20M | 2% |
+| FineMath 4+ | maths web pages | 80M | 8% |
+| SmolTalk | chat conversations | 50M | 5% |
+| github-code-clean | source code | 20M | 2% |
+
+## Fine-tuning (SFT)
+
+| | |
+|---|---|
+| Base model | [LittleKedi-tiny-base](https://huggingface.co/edededdy/LittleKedi-tiny-base) |
+| Conversations | 6,730 (6,730 train / 451 validation), 611.6K assistant tokens trained on |
+| Data | ~3,950 single-turn question/answer pairs (~28% of assistant tokens) written by **Gemma 4 E4B** from 1,499 documents sampled from **this model's own pretraining data**, so every question concerns text it saw; filtered to drop questions that refer back to the source document. Plus ~2,780 general conversations from **HuggingFaceTB/smol-smoltalk** (everyday small talk in full, plus capped samples of openhermes-50k, smol-constraints and smol-magpie-ultra-short), filtered to remove code and to fit 1,024 tokens. 301 grounded pairs (whole documents) and 150 conversations held out as the test set. |
+| Loss | On assistant replies and their closing `<|end|>` only; prompts and padding are masked |
+| Schedule | 842 steps, batch 16, 2 epochs, Muon + AdamW, peak LR 2e-4, cosine decay |
+| MoE routing | Load-balancing biases **frozen** at their pretrained values, with padding excluded from expert-load statistics; experts run dropless |
+
+Validation loss on 451 held-out conversations (assistant tokens only): **4.154 → 3.303**.
+
+## Evaluation
+
+### Zero-shot benchmarks (`evals.py`)
+
+| Task | acc_norm | acc | Items | Chance |
+|---|---|---|---|---|
+| SciQ | 55.9% ± 1.6% | 57.5% | 1,000 | 25% |
+| ARC-Easy | 32.2% ± 1.0% | 32.8% | 2,376 | 25% |
+| HellaSwag | 26.8% ± 0.4% | 26.5% | 10,042 | 25% |
+
+Each answer choice is scored by the log-probability of its text after the prompt, using lm-evaluation-harness
+prompts (SciQ includes its supporting passage). **acc_norm** divides by the answer's length in characters;
+**acc** doesn't. Splits: SciQ test, ARC-Easy test, HellaSwag validation.
+
+### MMLU (`mmlu_eval.py`)
+
+| Format | Accuracy | Questions scored | Average few-shot examples |
+|---|---|---|---|
+| letter | 25.6% | 14,037 (skipped 5) | 4.5 |
+| cloze | 25.5% | 14,037 (skipped 5) | 0.0 |
+
+| Category | Subjects | Questions | Letter | Cloze |
+|---|---|---|---|---|
+| STEM | 19 | 3,153 | 27.1% | 24.0% |
+| Humanities | 13 | 4,705 | 25.2% | 25.4% |
+| Social Sciences | 12 | 3,077 | 22.7% | 26.0% |
+| Other | 13 | 3,102 | 27.6% | 26.7% |
+
+**Biology & medicine (cloze): 27.1% on 2,089 questions**, +2.1 points against 25% chance (about 2.2 standard errors, significant). This pools the 10 life-science and health subjects; it isn't an official MMLU category.
+
+In the **letter** format the model compares " A"–" D", with same-subject examples added while they fit the
+context. In the **cloze** format each answer's text is scored by log-probability per byte; at this size the
+cloze results are the meaningful ones. Chance is 25%.
+
+### Base vs instruct
+
+Both are scored on the same raw-text prompts, without the chat template, so the numbers are directly comparable. SFT mainly teaches format, turn-taking and when to stop, so large changes in knowledge benchmarks aren't expected. Chance on the multiple-choice tasks is 25%.
+
+| Metric | Base | **Instruct** |
+|---|---|---|
+| SciQ (acc_norm) | 56.3% | 55.9% |
+| ARC-Easy (acc_norm) | 32.7% | 32.2% |
+| HellaSwag (acc_norm) | 26.7% | 26.8% |
+| MMLU (letter) | 25.4% | 25.6% |
+| MMLU (cloze) | 25.8% | 25.5% |
+| MMLU biology & medicine (cloze) | 28.4% | 27.1% |
+
+### Compared with its siblings and other small models
+
+| | **This model** | `tiny-wide-sparse` | [Pythia-70M](https://huggingface.co/EleutherAI/pythia-70m) | [GPT-2](https://huggingface.co/openai-community/gpt2) | [Supra-50M](https://huggingface.co/SupraLabs/Supra-50M-Base) |
+|---|---|---|---|---|---|
+| Parameters (total) | 2.0M | 3.8M | 70.4M | 124.4M | 51.8M |
+| Active non-embedding parameters | 0.5M | 0.5M | 18.9M | 85.1M | 35.4M |
+| Training tokens | 1,000M | 1,000M | 300B (the Pile) | undisclosed (WebText) | 20B (FineWeb-Edu) |
+| SciQ (acc_norm) | 55.9% | 53.9% | 55.2% | 53.2% | 77.2% |
+| ARC-Easy (acc_norm) | 32.2% | 32.4% | 35.0% | 42.0% | 52.2% |
+| HellaSwag (acc_norm) | 26.8% | 26.3% | – | 29.5% | 31.8% |
+| MMLU (cloze) | 25.5% | 25.7% | – | – | – |
+| MMLU biology & medicine (cloze) | 27.1% | 27.8% | – | – | – |
+| Validation loss (held-out mix) | – | 3.736 | – | – | – |
+
+LittleKedi columns are measured with this repo's `evals.py` and `mmlu_eval.py` on the checkpoints named. Validation
+losses are only comparable between models that share a tokenizer and validation set. The other columns are
+**published figures**: Pythia-70M from EleutherAI's evaluations as quoted on the
+[Wisp-15M card](https://huggingface.co/DedeProGames/Wisp-15M), GPT-2 and Supra-50M from the
+[Supra-50M-Base card](https://huggingface.co/SupraLabs/Supra-50M-Base). Their prompts may differ from ours,
+so treat gaps of a few points as noise.
+
+## What it can and can't do
+
+This is a **0.5M active non-embedding parameter chat model**: fine-tuning taught it the shape of a conversation,
+but it has almost no knowledge to put in it. It's a research artifact for studying instruction tuning at the
+smallest scale, not an assistant.
+
+**What it does**
+
+- ✅ Replies in the chat format: one short paragraph, then it stops by emitting `<|end|>` (it did on every one of
+  our test prompts).
+- ✅ Writes grammatical English in the register of an answer ("This process can help…", "It also reduces the
+  risk of…").
+- ✅ Occasionally picks up the topic's vocabulary (asked about stress, it mentions reducing stress and anxiety).
+
+**What it can't do**
+
+- ❌ **Answer the question asked.** Most replies are unrelated to the prompt: *"Hi! How are you today?"* →
+  *"Analyzing aids in food is crucial for maintaining healthy eating habits…"*.
+- ❌ **Facts.** Even on questions about text from its own pretraining data, answers are invented. Treat every name,
+  date and number as made up.
+- ❌ **Arithmetic.** *"What is 7 + 6?"* → *"Yes, it's about 8."*
+- ❌ **Creative requests, instructions, multi-turn follow-up, code.**
+
+**Benchmarks:** fine-tuning leaves them where the base model was (SciQ 55.9%, ARC-Easy 32.2%), as expected:
+instruction tuning changes the format, not what the model knows.
+
+**Decoding:** temperature ≈ 0.7, top-p ≈ 0.9, repetition penalty ≈ 1.15 (the GGUF files carry these as defaults).
+
+## Usage
+
+**llama.cpp** (GGUF, llama.cpp's built-in `deepseek2` architecture; the chat template is embedded):
+
+```sh
+llama-cli -m LittleKedi-tiny-instruct-q8_0.gguf -cnv --temp 0.7 --top-p 0.9 --repeat-penalty 1.15
+```
+
+**Ollama** (uses the bundled `template` and `params` files): `ollama run hf.co/edededdy/LittleKedi-tiny-instruct:Q8_0`
+
+**PyTorch** (needs the `littlekedi` package from [the training code](https://github.com/EdmundMartin/LittleKedi)):
+
+```python
+import json, torch
+from safetensors.torch import load_file
+from littlekedi import LittleKediModel, ModelConfig
+from littlekedi.chat import END, render_prompt
+from littlekedi.tokenizer import BPETokenizer
+
+model = LittleKediModel(ModelConfig.from_dict(json.load(open("config.json")))).eval()
+model.load_state_dict(load_file("model.safetensors"))
+tok = BPETokenizer.from_file("tokenizer.json")
+ids = torch.tensor([render_prompt(tok, [{"role": "user", "content": "What causes the seasons?"}])])
+out = model.generate(ids, 200, temperature=0.7, top_p=0.9, repetition_penalty=1.15, stop_ids={tok.token_to_id(END)})
+print(tok.decode(out[0, ids.shape[1]:].tolist()).replace(END, "").strip())
+```
+
+Or in a terminal: `python chat.py --ckpt <checkpoint>.pt`.
+
+## Files
+
+| File | Contents |
+|---|---|
+| `model.safetensors` | PyTorch weights (embedding tied with the output head), `littlekedi` parameter names |
+| `config.json` | `ModelConfig` for `littlekedi` |
+| `tokenizer.json` | Byte-level BPE tokenizer (Hugging Face `tokenizers` format) |
+| `chat_template.jinja` | Chat template (Jinja), the same one embedded in the GGUF files |
+| `tokenizer_config.json` | Special tokens (`<|end|>` = end of turn), chat template, context length |
+| `template` | Ollama chat template |
+| `params` | Ollama stop tokens and sampling defaults |
+| `LittleKedi-tiny-instruct-f16.gguf` | GGUF (f16) for llama.cpp / Ollama / LM Studio |
+| `LittleKedi-tiny-instruct-q8_0.gguf` | GGUF (q8_0) for llama.cpp / Ollama / LM Studio |
+| `little_kedi.jpg` | The LittleKedi logo |
+
+## Acknowledgements
+
+- The architecture follows **DeepSeek-V3** (DeepSeek-AI, 2024): MLA, DeepSeekMoE and auxiliary-loss-free balancing.
+- **Muon** (Keller Jordan et al., 2024) with Moonlight's update scaling (Liu et al., 2025).
+- Pretraining data: **FineWeb-Edu** (ODC-BY 1.0).
+- Pretraining data: **Cosmopedia** (Apache 2.0).
+- Pretraining data: **FineMath 4+** (ODC-BY 1.0).
+- Pretraining data: **SmolTalk** (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).
+- Pretraining data: **github-code-clean** (Apache 2.0, with each file under its own open-source license).
+- Chat data: see the SFT table above for its sources, each under its own license.
+
+*Card generated 2026-10-13 by `prepare_release.py` from `tiny-sft.pt` (step 15257).*

chat_template.jinja

@@ -0,0 +1 @@
+{% for m in messages %}{% if m['role'] == 'system' %}<|system|>{{ m['content'] }}<|end|>{% elif m['role'] == 'user' %}<|user|>{{ m['content'] }}<|end|>{% elif m['role'] == 'assistant' %}<|assistant|>{{ m['content'] }}<|end|>{% endif %}{% endfor %}{% if add_generation_prompt %}<|assistant|>{% endif %}

config.json

@@ -0,0 +1,29 @@
+{
+  "vocab_size": 8196,
+  "max_seq_len": 1024,
+  "dim": 128,
+  "n_layers": 4,
+  "n_dense_layers": 1,
+  "dense_hidden_dim": 352,
+  "norm_eps": 1e-06,
+  "init_std": 0.02,
+  "tie_embeddings": true,
+  "n_heads": 4,
+  "q_lora_rank": 0,
+  "kv_lora_rank": 32,
+  "qk_nope_head_dim": 16,
+  "qk_rope_head_dim": 16,
+  "v_head_dim": 16,
+  "rope_theta": 10000.0,
+  "n_routed_experts": 8,
+  "n_shared_experts": 1,
+  "n_activated_experts": 2,
+  "moe_hidden_dim": 64,
+  "n_expert_groups": 1,
+  "n_limited_groups": 1,
+  "route_scale": 1.0,
+  "bias_update_speed": 0.001,
+  "seq_aux_loss_alpha": 0.0001,
+  "moe_backend": "capacity",
+  "capacity_factor": 2.0
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49496e1e999c2dcedcb31e2b7ceef0b5d43c60e601b3b2c92496f84f1d14936e
+size 7971200

params

@@ -0,0 +1,11 @@
+{
+  "stop": [
+    "<|end|>",
+    "<|user|>",
+    "<|endoftext|>"
+  ],
+  "temperature": 0.7,
+  "top_p": 0.9,
+  "repeat_penalty": 1.15,
+  "num_ctx": 1024
+}

template

@@ -0,0 +1,6 @@
+{{- range .Messages }}
+{{- if eq .Role "system" }}<|system|>{{ .Content }}<|end|>
+{{- else if eq .Role "user" }}<|user|>{{ .Content }}<|end|>
+{{- else if eq .Role "assistant" }}<|assistant|>{{ .Content }}<|end|>
+{{- end }}
+{{- end }}<|assistant|>

tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "chat_template": "{% for m in messages %}{% if m['role'] == 'system' %}<|system|>{{ m['content'] }}<|end|>{% elif m['role'] == 'user' %}<|user|>{{ m['content'] }}<|end|>{% elif m['role'] == 'assistant' %}<|assistant|>{{ m['content'] }}<|end|>{% endif %}{% endfor %}{% if add_generation_prompt %}<|assistant|>{% endif %}",
+  "bos_token": null,
+  "add_bos_token": false,
+  "eos_token": "<|end|>",
+  "pad_token": "<|endoftext|>",
+  "additional_special_tokens": [
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>"
+  ],
+  "model_max_length": 1024
+}