| --- |
| license: other |
| license_name: ccpl-1.0 |
| license_link: LICENSE |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - code |
| - qwen3 |
| - 100m |
| library_name: transformers |
| datasets: |
| - HuggingFaceTB/smollm-corpus |
| - HuggingFaceTB/smol-smoltalk |
| - HuggingFaceTB/smoltalk |
| --- |
| |
| # CalmaCatCoder-Next-Mini |
|
|
| A tiny **experimental** chat model (96.6M parameters) trained completely **from scratch on one consumer GPU** |
| (RTX 5060 Ti, 16 GB) by ViorikaAI. It is a hobby project: a small calm, cat-themed |
| Python helper that knows who it is, speaks ChatML and writes very simple Python. |
|
|
| > **Please read the limitations below.** This is a learning project, not a useful assistant. |
| > It memorised a handful of templates and will confidently produce wrong code and wrong facts. |
|
|
| ## Model details |
|
|
| | | | |
| |---|---| |
| | Architecture | `Qwen3ForCausalLM` (Qwen3 architecture, **no Qwen weights**, random init) | |
| | Parameters | 96.6M (tied input/output embeddings) | |
| | Layers / hidden / FFN | 14 / 768 / 2304 | |
| | Attention | 12 heads, 4 KV heads, head_dim 64 | |
| | Context length | **1024 bytes** (not words) | |
| | Vocabulary | **256 tokens: one token = one UTF-8 byte** (no learned tokenizer) | |
| | Precision | bf16 weights | |
| | Language | English only | |
| |
| ### The tokenizer is byte-level |
| |
| Token id equals the byte value, so `"def "` becomes `[100, 101, 102, 32]` and one word costs several tokens. |
| This is why the context is short. `<|im_start|>` and `<|im_end|>` are **plain text made of bytes**, not special |
| tokens, so the model ends its answer by writing the literal text `<|im_end|>`. |
| Stop generation on that string (see the example below). |
|
|
| Chat format (the same text the model was trained on, available as `tokenizer.apply_chat_template`): |
|
|
| ``` |
| <|im_start|>system |
| You are CalmaCatCoder-Next, a calm and friendly cat-themed coding assistant. You explain things step by step and say so when you are not sure.<|im_end|> |
| <|im_start|>user |
| Who are you?<|im_end|> |
| <|im_start|>assistant |
| ``` |
|
|
| If you pass no system message, the one above is inserted automatically. Other system prompts were never seen |
| in training, so expect them to be mostly ignored. |
|
|
| ## Quick start |
|
|
| Requires `transformers>=4.51` (Qwen3 support). No `trust_remote_code` needed. |
|
|
| ```python |
| import torch |
| from transformers import AutoTokenizer, AutoModelForCausalLM, StoppingCriteria, StoppingCriteriaList |
| |
| repo = "ViorikaAI-org/CalmaCatCoder-Next-mini" # or a local folder |
| device = "cuda" if torch.cuda.is_available() else "cpu" |
| tok = AutoTokenizer.from_pretrained(repo) |
| model = AutoModelForCausalLM.from_pretrained(repo).to(device, dtype=torch.bfloat16).eval() |
| |
| END = tok.encode("<|im_end|>") # the end marker is ordinary bytes |
| class StopOnEnd(StoppingCriteria): |
| def __call__(self, input_ids, scores, **kwargs): |
| return torch.tensor([input_ids[0, -len(END):].tolist() == END] * input_ids.shape[0], |
| device=input_ids.device) |
| |
| messages = [{"role": "user", "content": "Who are you?"}] |
| prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = tok(prompt, return_tensors="pt").to(device) |
| with torch.no_grad(): |
| out = model.generate(**inputs, max_new_tokens=300, |
| stopping_criteria=StoppingCriteriaList([StopOnEnd()])) |
| print(tok.decode(out[0, inputs["input_ids"].shape[1]:]).replace("<|im_end|>", "").strip()) |
| ``` |
|
|
| Sampling defaults live in `generation_config.json` (temperature 0.7, top-k 40). Keep prompt + answer under |
| 1024 bytes; the model never saw longer sequences. |
|
|
| ## How it was trained |
|
|
| All of it ran on a single RTX 5060 Ti (16 GB) under Windows with a PyTorch nightly build. |
|
|
| **Stage 1: pre-training from scratch (1 epoch, ~7.6 hours).** |
| 29,908 steps × 32,768 byte-tokens ≈ 980M tokens, about 36k tokens/s. |
| AdamW (lr 6e-4, betas 0.9/0.95, weight decay 0.1), 300 warm-up steps, cosine decay to 10%, |
| gradient clipping 1.0, bf16 autocast, `torch.compile`. |
| Final validation loss 0.508 nats/byte on a held-out 2% of the mix. This number is **not comparable** with other |
| models: it is per byte, and the held-out part contains near-copies of the templated persona dialogues. |
|
|
| Data mix (~1 GB, target shares; the exact proportions depend on what each source could supply): |
|
|
| - Python code: `codeparrot/codeparrot-clean`, the largest share (about 60%) |
| - English text: `cosmopedia-v2` from `HuggingFaceTB/smollm-corpus` (15%) |
| - Math: `meta-math/MetaMathQA` (5%) |
| - ChatML dialogues: `HuggingFaceTB/smol-smoltalk`, only dialogues shorter than 1000 bytes (up to 20%) |
| - About 5 MB of synthetic persona dialogues generated by a script (identity, creator, greetings, |
| a few Python tasks) |
|
|
| **Stage 2: supervised fine-tuning (1 epoch, ~13 minutes).** |
| About 28k short ChatML dialogues (~17M tokens), loss computed on assistant tokens only, peak lr 1e-4. |
| Mixture of programmatically generated dialogues (identity and creator questions in many phrasings, greetings, |
| 18 small Python tasks, follow-ups such as "add a docstring", remembering a name within a chat), |
| `HuggingFaceTB/smoltalk` (`everyday-conversations`) and a subset of `smol-smoltalk`. |
| SFT validation loss was 0.123, which mostly reflects how templated that data is. |
|
|
| ## Intended use |
|
|
| Learning and curiosity: look at how a tiny model trained from scratch behaves, reuse the training recipe, |
| test tokenizers or inference tooling on a very small model. |
|
|
| **Not intended for** real coding help, factual questions, any decision that matters, or production use. |
|
|
| ## Limitations (observed, not hypothetical) |
|
|
| - **English only, almost only Python.** Other languages and programming languages are outside what it learned. |
| - **It memorised templates.** For a coding request it picks the closest of the ~18 tasks it knows and adapts it. |
| Real examples from testing: |
| - "give me a random python code" produced a function named `random_state` that simply reverses a string, |
| followed by an explanation copied from a different task. |
| - "Write a function that multiplies all numbers in a list" produced a list comprehension that filters even numbers. |
| - After "My name is Zorro." and one more turn, "What is my name?" was answered with a different name (Sam). |
| - **Facts are unreliable**, and it often confidently continues with plausible-looking nonsense. |
| - **Short memory.** 1024 bytes of context covers a couple of turns at most. |
| - **No safety training** of any kind. It is too small to be dangerous, but it has no guardrails either. |
| - Very low training loss does not mean good behaviour here: it mostly measures memorised templates. |
| |
| ## Training data notes and license |
|
|
| Part of the training data (`cosmopedia-v2`, `smol-smoltalk`, `smoltalk` and the synthetic persona dialogues) |
| was generated by other language models, so this model inherits their style and any of their errors. |
| The code corpus consists of public repositories with mixed licenses. |
| Check each dataset card for its license before reusing the weights or the data mix. |
|
|
| License: CalmaCat Open License (CCPL-1.0) |
|
|
|
|
|
|
