CoolDip00 commited on
Commit
b11100b
·
verified ·
1 Parent(s): a715b23

tiny-llm 30M TinyStories model (4.8b export, checked)

Browse files
README.md ADDED
@@ -0,0 +1,100 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ language:
4
+ - en
5
+ library_name: transformers
6
+ pipeline_tag: text-generation
7
+ datasets:
8
+ - roneneldan/TinyStories
9
+ tags:
10
+ - llama
11
+ - tinystories
12
+ - from-scratch
13
+ - story-generation
14
+ ---
15
+
16
+ # tiny-llm TinyStories 30M
17
+
18
+ A 29.9M-parameter Llama-style language model, trained from scratch on a laptop GPU as part of
19
+ [tiny-llm](https://github.com/cooldip00/tiny-llm): an LLM training and inference project written from scratch.
20
+
21
+ It is a **base model for story continuation**. Give it the opening of a short children's story and
22
+ it writes the rest, in the simple English of the TinyStories dataset. It does not follow
23
+ instructions or chat, and it knows nothing outside that kind of story.
24
+
25
+ ## Use it
26
+
27
+ With Hugging Face Transformers (the weights are a standard `LlamaForCausalLM`):
28
+
29
+ ```python
30
+ from transformers import AutoModelForCausalLM, AutoTokenizer
31
+
32
+ repo = "CoolDip00/tiny-llm-tinystories-30m"
33
+ tok = AutoTokenizer.from_pretrained(repo)
34
+ model = AutoModelForCausalLM.from_pretrained(repo)
35
+
36
+ ids = tok("Once upon a time, a little dog", return_tensors="pt").input_ids
37
+ out = model.generate(ids, max_new_tokens=120, do_sample=True, temperature=0.8, top_k=50)
38
+ print(tok.decode(out[0]))
39
+ ```
40
+
41
+ Or with the project's own inference server (OpenAI-compatible `/v1/completions`), on CPU with Docker:
42
+
43
+ ```bash
44
+ docker build -t tinyllm https://github.com/cooldip00/tiny-llm.git
45
+ docker run --rm -p 7860:7860 tinyllm
46
+ curl localhost:7860/v1/completions -H 'content-type: application/json' \
47
+ -d '{"prompt": "Once upon a time", "max_tokens": 80}'
48
+ ```
49
+
50
+ Sample, prompted with "Once upon a time": *"…a little girl named Lily. She was very curious… she
51
+ found a shiny rock that looked like it was magic… she saw a butterfly that was also very shiny…"*
52
+
53
+ ## Model
54
+
55
+ | | |
56
+ |---|---|
57
+ | Architecture | Llama-style decoder: RMSNorm (pre-norm), RoPE (θ = 10,000), SwiGLU MLP, multi-head attention, tied input / output embeddings |
58
+ | Size | 29,893,120 parameters: 8 layers, d_model 512, 8 heads (head dim 64), MLP hidden 1,408 |
59
+ | Context | 512 tokens |
60
+ | Tokenizer | Byte-level BPE, 8,192 tokens, trained on TinyStories. No BOS token; `<|endoftext|>` (id 0) ends a story |
61
+ | Weights | `model.safetensors`, fp32, as trained |
62
+
63
+ ## Training
64
+
65
+ | | |
66
+ |---|---|
67
+ | Data | [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) train split, cleaned (broken encodings fixed, non-ASCII stories dropped); 996M tokens, ~2.1 epochs |
68
+ | Steps | 7,600 × 131,072 tokens (batch 32 × 512, 8 gradient-accumulation steps) |
69
+ | Optimizer | AdamW (β 0.9 / 0.95, weight decay 0.1 on matrices and embeddings), peak LR 1e-3 after 200 warmup steps, cosine decay to 1e-4, gradient clipping 1.0 |
70
+ | Precision | bf16 autocast, fp32 master weights, `torch.compile` |
71
+ | Hardware | One laptop RTX 3070 Ti (8 GB): 2 h 27 min, ~115K tokens/s, 56% MFU |
72
+
73
+ ## Evaluation
74
+
75
+ | | |
76
+ |---|---|
77
+ | Validation loss | **1.2825** nats / token on the TinyStories validation split (perplexity 3.6); train loss 1.280, so no overfitting |
78
+
79
+ This is the only metric: the model is too small for general benchmarks to mean anything.
80
+
81
+ **Export check** (`scripts/export_hf.py` in the project): the files in this repo were loaded back and compared on CPU in fp32.
82
+ - HF Transformers' logits match the project's own model to within 1.4e-5 over 2,741 positions, both as one full forward and as prefill plus cached decode.
83
+ - Greedy generation matches token for token on 8 prompts, each run to EOS or 200 tokens.
84
+ - `AutoTokenizer` gives the project tokenizer's ids on every validation story.
85
+
86
+ ## Limitations
87
+
88
+ - Story continuation only, in TinyStories' vocabulary and style. Other prompts (questions, code, other languages) give story-like or meaningless text.
89
+ - Small and trained on synthetic stories: its text can be repetitive or contradict itself, and none of it is factual.
90
+ - Inherits whatever biases the TinyStories stories (generated by GPT-3.5 and GPT-4) contain.
91
+ - Not for any real-world or safety-relevant use; it is a learning and demo model.
92
+
93
+ ## License
94
+
95
+ The weights and code are MIT. The training data, TinyStories, is published under CDLA-Sharing-1.0.
96
+
97
+ ## Links
98
+
99
+ - Project, code and write-ups: <https://github.com/cooldip00/tiny-llm>
100
+ - TinyStories paper: Eldan & Li, 2023, [arXiv:2305.07759](https://arxiv.org/abs/2305.07759)
config.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LlamaForCausalLM"
4
+ ],
5
+ "model_type": "llama",
6
+ "hidden_act": "silu",
7
+ "vocab_size": 8192,
8
+ "max_position_embeddings": 512,
9
+ "hidden_size": 512,
10
+ "num_hidden_layers": 8,
11
+ "num_attention_heads": 8,
12
+ "num_key_value_heads": 8,
13
+ "intermediate_size": 1408,
14
+ "rms_norm_eps": 1e-05,
15
+ "rope_theta": 10000.0,
16
+ "initializer_range": 0.02,
17
+ "tie_word_embeddings": true,
18
+ "attention_bias": false,
19
+ "rope_scaling": null,
20
+ "dtype": "float32",
21
+ "bos_token_id": null,
22
+ "eos_token_id": 0,
23
+ "pad_token_id": null
24
+ }
generation_config.json ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": null,
3
+ "eos_token_id": 0,
4
+ "pad_token_id": 0
5
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4efa5367619f5f01cf903e659fc0e234d7315d18fde58eb1d25f720db3b13931
3
+ size 119580576
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "tokenizer_class": "PreTrainedTokenizerFast",
3
+ "eos_token": "<|endoftext|>",
4
+ "bos_token": null,
5
+ "unk_token": null,
6
+ "pad_token": "<|endoftext|>",
7
+ "add_bos_token": false,
8
+ "add_eos_token": false,
9
+ "clean_up_tokenization_spaces": false,
10
+ "model_max_length": 512
11
+ }