Update README.md

README.md

@@ -10,5 +10,166 @@ tags:
 - qwen3
 - 100m
 library_name: transformers
+datasets:
+- HuggingFaceTB/smollm-corpus
+- HuggingFaceTB/smol-smoltalk
+- HuggingFaceTB/smoltalk
 ---
 
+# CalmaCatCoder-Next-96M
+
+A tiny **experimental** chat model (96.6M parameters) trained completely **from scratch on one consumer GPU**
+(RTX 5060 Ti, 16 GB) by an enthusiast from ViorikaAI. It is a hobby project: a small calm, cat-themed
+Python helper that knows who it is, speaks ChatML and writes very simple Python.
+
+> **Please read the limitations below.** This is a learning project, not a useful assistant.
+> It memorised a handful of templates and will confidently produce wrong code and wrong facts.
+
+## Model details
+
+| | |
+|---|---|
+| Architecture | `Qwen3ForCausalLM` (Qwen3 architecture, **no Qwen weights**, random init) |
+| Parameters | 96.6M (tied input/output embeddings) |
+| Layers / hidden / FFN | 14 / 768 / 2304 |
+| Attention | 12 heads, 4 KV heads, head_dim 64 |
+| Context length | **1024 bytes** (not words) |
+| Vocabulary | **256 tokens: one token = one UTF-8 byte** (no learned tokenizer) |
+| Precision | bf16 weights |
+| Language | English only |
+
+### The tokenizer is byte-level
+
+Token id equals the byte value, so `"def "` becomes `[100, 101, 102, 32]` and one word costs several tokens.
+This is why the context is short. `<|im_start|>` and `<|im_end|>` are **plain text made of bytes**, not special
+tokens, so the model ends its answer by writing the literal text `<|im_end|>`.
+Stop generation on that string (see the example below).
+
+Chat format (the same text the model was trained on, available as `tokenizer.apply_chat_template`):
+
+```
+<|im_start|>system
+You are CalmaCatCoder-Next, a calm and friendly cat-themed coding assistant. You explain things step by step and say so when you are not sure.<|im_end|>
+<|im_start|>user
+Who are you?<|im_end|>
+<|im_start|>assistant
+```
+
+If you pass no system message, the one above is inserted automatically. Other system prompts were never seen
+in training, so expect them to be mostly ignored.
+
+## Quick start
+
+Requires `transformers>=4.51` (Qwen3 support). No `trust_remote_code` needed.
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM, StoppingCriteria, StoppingCriteriaList
+
+repo = "YOUR_USERNAME/CalmaCatCoder-Next-96M"      # or a local folder
+device = "cuda" if torch.cuda.is_available() else "cpu"
+tok = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForCausalLM.from_pretrained(repo).to(device, dtype=torch.bfloat16).eval()
+
+END = tok.encode("<|im_end|>")                      # the end marker is ordinary bytes
+class StopOnEnd(StoppingCriteria):
+    def __call__(self, input_ids, scores, **kwargs):
+        return torch.tensor([input_ids[0, -len(END):].tolist() == END] * input_ids.shape[0],
+                            device=input_ids.device)
+
+messages = [{"role": "user", "content": "Who are you?"}]
+prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = tok(prompt, return_tensors="pt").to(device)
+with torch.no_grad():
+    out = model.generate(**inputs, max_new_tokens=300,
+                         stopping_criteria=StoppingCriteriaList([StopOnEnd()]))
+print(tok.decode(out[0, inputs["input_ids"].shape[1]:]).replace("<|im_end|>", "").strip())
+```
+
+Sampling defaults live in `generation_config.json` (temperature 0.7, top-k 40). Keep prompt + answer under
+1024 bytes; the model never saw longer sequences.
+
+### GGUF / llama.cpp
+
+The tokenizer has no BPE merges and a pre-tokenizer unknown to the stock converter, so
+`convert_hf_to_gguf.py` fails on it. The `make_gguf.py` wrapper from the code repository patches this
+at conversion time without modifying llama.cpp (details in its header):
+
+```bash
+git clone https://github.com/ggml-org/llama.cpp
+python make_gguf.py llama.cpp CalmaCatCoder-Next-96M calmacat-96m-f16.gguf f16     # or q8_0
+```
+
+Because `<|im_end|>` is plain text, llama.cpp does not stop on it by itself. Save the ChatML prompt to a file
+and use a reverse prompt:
+
+```bash
+llama-completion -m calmacat-96m-f16.gguf -no-cnv --no-escape -f prompt.txt -r "<|im_end|>" -n 300 --temp 0.7 --top-k 40
+```
+
+A warning `special_eos_id is not in special_eog_ids` is expected: the model has no EOS token.
+Prefer f16/bf16 or q8_0; this model is so small that aggressive quantization will likely hurt it.
+
+## How it was trained
+
+All of it ran on a single RTX 5060 Ti (16 GB) under Windows with a PyTorch nightly build.
+
+**Stage 1: pre-training from scratch (1 epoch, ~7.6 hours).**
+29,908 steps × 32,768 byte-tokens ≈ 980M tokens, about 36k tokens/s.
+AdamW (lr 6e-4, betas 0.9/0.95, weight decay 0.1), 300 warm-up steps, cosine decay to 10%,
+gradient clipping 1.0, bf16 autocast, `torch.compile`.
+Final validation loss 0.508 nats/byte on a held-out 2% of the mix. This number is **not comparable** with other
+models: it is per byte, and the held-out part contains near-copies of the templated persona dialogues.
+
+Data mix (~1 GB, target shares; the exact proportions depend on what each source could supply):
+
+- Python code: `codeparrot/codeparrot-clean`, the largest share (about 60%)
+- English text: `cosmopedia-v2` from `HuggingFaceTB/smollm-corpus` (15%)
+- Math: `meta-math/MetaMathQA` (5%)
+- ChatML dialogues: `HuggingFaceTB/smol-smoltalk`, only dialogues shorter than 1000 bytes (up to 20%)
+- About 5 MB of synthetic persona dialogues generated by a script (identity, creator, greetings,
+  a few Python tasks)
+
+**Stage 2: supervised fine-tuning (1 epoch, ~13 minutes).**
+About 28k short ChatML dialogues (~17M tokens), loss computed on assistant tokens only, peak lr 1e-4.
+Mixture of programmatically generated dialogues (identity and creator questions in many phrasings, greetings,
+18 small Python tasks, follow-ups such as "add a docstring", remembering a name within a chat),
+`HuggingFaceTB/smoltalk` (`everyday-conversations`) and a subset of `smol-smoltalk`.
+SFT validation loss was 0.123, which mostly reflects how templated that data is.
+
+## Intended use
+
+Learning and curiosity: look at how a tiny model trained from scratch behaves, reuse the training recipe,
+test tokenizers or inference tooling on a very small model.
+
+**Not intended for** real coding help, factual questions, any decision that matters, or production use.
+
+## Limitations (observed, not hypothetical)
+
+- **English only, almost only Python.** Other languages and programming languages are outside what it learned.
+- **It memorised templates.** For a coding request it picks the closest of the ~18 tasks it knows and adapts it.
+  Real examples from testing:
+  - "give me a random python code" produced a function named `random_state` that simply reverses a string,
+    followed by an explanation copied from a different task.
+  - "Write a function that multiplies all numbers in a list" produced a list comprehension that filters even numbers.
+  - After "My name is Zorro." and one more turn, "What is my name?" was answered with a different name (Sam).
+- **Facts are unreliable**, and it often confidently continues with plausible-looking nonsense.
+- **Short memory.** 1024 bytes of context covers a couple of turns at most.
+- **No safety training** of any kind. It is too small to be dangerous, but it has no guardrails either.
+- Very low training loss does not mean good behaviour here: it mostly measures memorised templates.
+
+## Training data notes and license
+
+Part of the training data (`cosmopedia-v2`, `smol-smoltalk`, `smoltalk` and the synthetic persona dialogues)
+was generated by other language models, so this model inherits their style and any of their errors.
+The code corpus consists of public repositories with mixed licenses.
+Check each dataset card for its license before reusing the weights or the data mix.
+
+License: TODO (choose after checking the dataset licenses above; `license: other` is set in the metadata until then).
+
+## Reproducing
+
+Scripts (data preparation, pre-training, SFT, Hugging Face export, GGUF conversion): TODO link.
+The weights were exported with `export_hf.py` (byte tokenizer, chat template and generation config included).
+
+