Upload README.md

README.md

@@ -0,0 +1,236 @@
+---
+license: cdla-sharing-1.0
+language:
+- en
+pipeline_tag: text-generation
+tags:
+- gguf
+- llama
+- tinystories
+- small-language-model
+- trained-from-scratch
+- educational
+datasets:
+- roneneldan/TinyStories
+---
+
+# JunkyAI
+
+**A 15.7-million-parameter language model trained from scratch.**
+
+JunkyAI is a small English language model built end to end without any
+pretrained weights: its own byte-level BPE vocabulary, its own transformer
+implementation, its own training loop, and its own GGUF writer. It speaks
+simple, fluent English — roughly the register of a children's book — and
+answers short questions. It does nothing else, by design and by capacity.
+
+It exists to be understood completely rather than to be capable. Every
+component is a few hundred lines of readable Python with no ML framework above
+PyTorch tensors: no `transformers`, no `tokenizers`, no `peft`, no `gguf`.
+
+---
+
+## At a glance
+
+| | |
+|---|---|
+| Parameters | 15,735,168 (~15.7M) |
+| Architecture | Decoder-only transformer, Llama-shaped |
+| Layers / hidden dim | 8 / 384 |
+| Attention | 6 query heads, 2 KV heads (grouped-query) |
+| Feed-forward | SwiGLU, 1024 inner |
+| Normalisation | RMSNorm, pre-norm |
+| Positions | RoPE (interleaved pairs), θ=10000 |
+| Context length | 512 tokens |
+| Vocabulary | 8,192 byte-level BPE, trained on the same corpus |
+| Precision | bfloat16 training, Q8_0 export |
+| File size | 20.4 MB (Q8_0 GGUF) |
+| Language | English only |
+
+The architecture is deliberately Llama-shaped so the exported GGUF loads in
+unmodified llama.cpp, Ollama and LM Studio. One detail differs from the usual
+HuggingFace implementation: RoPE is applied to interleaved pairs rather than a
+rotate-half split, matching ggml's `GGML_ROPE_TYPE_NORM` so weights export with
+no permutation step.
+
+---
+
+## Training
+
+### Stage 1 — Pretraining
+
+| | |
+|---|---|
+| Corpus | TinyStories (`roneneldan/TinyStories`), 300,000 stories |
+| Tokens | 65.5M train / 329k validation |
+| Steps | 9,000 (batch 24 × grad-accum 2 × seq 512 = 24,576 tokens/step) |
+| Total tokens seen | 221M (~3.4 epochs) |
+| Optimiser | AdamW, lr 3e-4, cosine decay to 10%, 300 warmup, weight decay 0.1 |
+| Final validation loss | **1.4786** (perplexity 4.4) |
+| Hardware / time | RTX 4060 8GB, ~55 minutes at 67,000 tokens/sec |
+
+TinyStories is a corpus of short children's stories written with a small
+vocabulary. It was chosen because its distribution is close to what a model
+this size can actually represent — a general web corpus would exceed its
+capacity and produce worse English, not better.
+
+### Stage 2 — Instruction tuning
+
+| | |
+|---|---|
+| Data | 2,216 single-turn instruction pairs + 179 multi-turn conversations |
+| Source | Synthetic, generated with Groq-hosted `openai/gpt-oss-120b` |
+| Held-out eval | 120 examples on topics absent from the training set |
+| Steps | 300 (batch 16, ~2 epochs), lr 1e-4 |
+| Final validation loss | **2.5164** |
+| Time | ~2 minutes |
+
+Only assistant tokens contribute to the loss; prompt tokens are masked out.
+Full fine-tuning was used rather than LoRA — at this size the model occupies
+under 4 GB of VRAM during training, so parameter-efficient methods save nothing
+and cost quality. LoRA and NF4 QLoRA are implemented in the repository for
+larger base models.
+
+---
+
+## What it can do
+
+- Hold a short, grammatical conversation in simple present and past tense
+- Answer "What is X?" for everyday concrete nouns — animals, food, weather,
+  household objects, places
+- Greet, thank, say goodbye, and answer questions about itself
+- Continue a short story coherently for several sentences
+- Decline questions beyond its ability, with varied wording, and offer an
+  alternative
+
+Representative outputs:
+
+```
+How are you?               -> I am good, thank you.
+hello                      -> Hi! I am happy to see you.
+What is a dog?             -> A dog is a pet. It wags its tail and barks.
+Tell me what rain is.      -> Rain is wet and cold.
+What is 847 times 293?     -> That is too much for a small model like me.
+                              But I can tell you about food or play.
+Write me a Python program. -> I cannot work that out. I am sorry.
+                              I can talk about simple things instead.
+```
+
+---
+
+## What it cannot do
+
+These are capacity limits, not bugs, and no additional training fixes them at
+this parameter count.
+
+- **No arithmetic.** It cannot add, multiply or count reliably.
+- **No code.** It cannot read or write any programming language.
+- **No world knowledge.** It knows nothing about real people, places, dates or
+  events. Anything it says about them is invented.
+- **No reasoning.** It cannot follow multi-step logic or answer "why" questions
+  with genuine explanation.
+- **Long-range logic drifts.** Within a few sentences it stays coherent; across
+  a paragraph, causality breaks down. It may describe a character wanting
+  something that belongs to a different character.
+- **Weak multi-turn.** With only 179 conversations in training, it tends to lock
+  onto the first topic and answer later, unrelated questions as if still on it.
+  Single-turn use is markedly more reliable.
+- **Narrow vocabulary.** Words outside the TinyStories range (~1,500–3,000 common
+  English words) have near-random representations. It will produce them as noise.
+- **English only.**
+
+It will sometimes answer confidently and incorrectly. The graceful refusals
+cover common cases but are not a reliable safety property.
+
+---
+
+## Intended use
+
+Educational and experimental. JunkyAI is a complete, legible example of the
+full language-model pipeline at a scale that trains in an hour on one consumer
+GPU — useful for understanding tokenisation, transformer internals,
+pretraining, instruction tuning, quantisation and the GGUF format.
+
+**Not suitable for**: production systems, factual reference, advice of any kind
+(medical, legal, financial), decisions affecting people, content moderation, or
+any setting where an incorrect answer carries cost. It has received no safety
+tuning or red-teaming beyond the refusal examples described above.
+
+---
+
+## Data provenance and licensing
+
+- **Pretraining**: TinyStories, by Eldan and Li (arXiv:2305.07759), distributed
+  on the Hugging Face Hub as `roneneldan/TinyStories` under
+  **CDLA-Sharing-1.0**. That is a share-alike data license: if you publish the
+  data or data derived from it, you must do so under the same terms. Whether
+  model weights count as derived data under CDLA is not settled law, so this
+  card carries the same license as the safer assumption. TinyStories was itself
+  generated by GPT-3.5 and GPT-4, so OpenAI's terms may also apply upstream.
+- **Instruction data**: synthetic, generated through the Groq API using
+  `openai/gpt-oss-120b`. Model outputs may be subject to the provider's terms;
+  the data inherits whatever biases and errors that model has.
+- **Code**: the training pipeline in this repository is separate from the
+  weights and carries whatever license you choose for it. The license above
+  applies to the released model, because of the corpus it was trained on.
+
+The instruction data was generated and filtered automatically, and was not
+reviewed example by example.
+
+---
+
+## Running it
+
+```bash
+llama-cli -m junkyai-q8_0.gguf -cnv
+```
+
+The chat template is embedded in the GGUF, so llama.cpp formats turns itself.
+Ollama, via a `Modelfile`:
+
+```
+FROM ./junkyai-q8_0.gguf
+PARAMETER temperature 0.8
+PARAMETER stop "<|eos|>"
+```
+
+Suggested sampling: temperature 0.7–0.8, top-k 40, top-p 0.95, repetition
+penalty 1.1, and 40–60 new tokens. Longer generations drift.
+
+**Ask one question at a time.** Clearing the conversation between unrelated
+questions noticeably improves answers.
+
+---
+
+## Reproducing
+
+```bash
+python scripts/bootstrap_pretrain_data.py     # ~10 min
+python scripts/train_tokenizer.py             # ~20 s
+python -m junkyai.train --stage pretrain --steps 9000
+python scripts/generate_sft_data.py           # needs a Groq API key
+python -m junkyai.train --stage sft --steps 300 --init-from checkpoints/junkyai-tiny
+python -m junkyai.gguf_export --checkpoint checkpoints/junkyai-tiny-sft --quant q8_0
+```
+
+Verify the exported file against the checkpoint it came from:
+
+```bash
+python scripts/verify_gguf.py junkyai-q8_0.gguf --checkpoint checkpoints/junkyai-tiny-sft
+```
+
+---
+
+## Quantisation
+
+Round-trip error measured against the source weights:
+
+| Format | Size | Mean relative error |
+|---|---|---|
+| F16 | 38 MB | 0.018% |
+| **Q8_0** (released) | **20.4 MB** | **0.56%** |
+| Q4_0 | 11 MB | 9.0% |
+
+Q8_0 is the released format. Quantisation error is absorbed by redundant
+weights, and a 15.7M model has none to spare — Q4_0 saves 9 MB and does
+sixteen times the damage. Norm tensors are kept at F32 in all formats.