DedeProGames commited on
Commit
a8695e3
·
verified ·
1 Parent(s): fddf996

Model card

Browse files
Files changed (2) hide show
  1. README.md +138 -0
  2. loss_curve.png +0 -0
README.md ADDED
@@ -0,0 +1,138 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ library_name: transformers
6
+ pipeline_tag: text-generation
7
+ tags:
8
+ - llama
9
+ - tiny
10
+ - small-language-model
11
+ - pretrained
12
+ - wisp
13
+ datasets:
14
+ - HuggingFaceFW/fineweb-edu
15
+ - mlfoundations/dclm-baseline-1.0
16
+ - HuggingFaceCode/stack-v3-train
17
+ - HuggingFaceTB/finemath
18
+ ---
19
+
20
+ # Wisp-5M
21
+
22
+ **Wisp-5M** is a tiny decoder-only language model from the **Wisp** family, trained from scratch on
23
+ **3.00B tokens** of English web text, code and math. It uses a plain LLaMA architecture, so it
24
+ loads with `transformers` (`LlamaForCausalLM`) with no custom code.
25
+
26
+ It is a **base model** (not instruction-tuned). Its size makes it useful for research, education,
27
+ speculative decoding drafts, on-device experiments and as a starting point for fine-tuning. Don't
28
+ expect factual accuracy.
29
+
30
+ ![loss curve](loss_curve.png)
31
+
32
+ ## The Wisp family
33
+
34
+ | Model | Params | Non-embedding | Hidden | Layers | Heads | KV heads | FFN | Tokens |
35
+ |---|---|---|---|---|---|---|---|---|
36
+ | [Wisp-5M](https://huggingface.co/DedeProGames/Wisp-5M) | 5.1M | 3.5M | 192 | 9 | 6 | 2 | 512 | 3B |
37
+ | [Wisp-15M](https://huggingface.co/DedeProGames/Wisp-15M) | 15.5M | 12.9M | 320 | 12 | 10 | 2 | 864 | 3B |
38
+ | Wisp-60M *(planned)* | 60.6M | 56.4M | 512 | 20 | 8 | 2 | 1408 | — |
39
+
40
+ All models share the same tokenizer and saw the **same data in the same order**: one fixed global
41
+ shuffle of 2048-token windows. That makes the family directly comparable across sizes.
42
+
43
+ ## Architecture
44
+
45
+ - LLaMA-style decoder: pre-norm RMSNorm, rotary position embeddings (θ = 10,000), SwiGLU MLP,
46
+ grouped-query attention, no biases, **tied input/output embeddings**
47
+ - Deep-and-thin shapes (following MobileLLM), which work better than wide-and-shallow at this scale
48
+ - Context length: **2048 tokens**
49
+ - Tokenizer: Wisp byte-level BPE, **8,192 tokens**, trained on the same data mix; digits are split
50
+ individually (better for arithmetic); `<|endoftext|>` is used as BOS/EOS and document separator.
51
+ Compression is ~3.8 characters/token on English web text and ~3.0 on code.
52
+
53
+ ## Training
54
+
55
+ | | |
56
+ |---|---|
57
+ | Tokens | 3.00B (11,444 steps × 262,144 tokens) |
58
+ | Sequence length | 2048 |
59
+ | Optimizer | Muon (hidden matrices, Moonlight RMS-matched update) + AdamW (embeddings, norms) |
60
+ | Peak LR / weight decay | 0.01 / 0.1 |
61
+ | Schedule | Warmup-Stable-Decay: 1% warmup, linear decay to 0 over the last 20% |
62
+ | Precision | mixed precision (bf16/fp16 autocast, fp32 master weights), `torch.compile` |
63
+ | Hardware | 1× NVIDIA Tesla T4 (Hugging Face Jobs), 340 min |
64
+
65
+ - **Compute cost:** ~US$2.28 on Hugging Face Jobs
66
+
67
+ ### Data mix
68
+
69
+ An 8B-token pool was pretokenized with the mix below and shuffled once as 2048-token windows. Each
70
+ model trained on the first 3.00B tokens of that shuffle, which keeps the same proportions.
71
+
72
+ | Source | Subset / filter | Share | Tokens in pool |
73
+ |---|---|---|---|
74
+ | [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) | sample-100BT | 45% | 3.6B |
75
+ | [DCLM-Baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) | global-shard_01 | 35% | 2.8B |
76
+ | [The Stack v3](https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train) | 40 popular languages, no vendor files, <=64KB | 12% | 0.96B |
77
+ | [FineMath](https://huggingface.co/datasets/HuggingFaceTB/finemath) | finemath-4plus | 8% | 0.64B |
78
+
79
+ ## Evaluation
80
+
81
+ Held-out cross-entropy (nats/token, lower is better) on documents never seen in training:
82
+
83
+ | FineWeb-Edu | DCLM | Stack v3 | FineMath | Average |
84
+ |---|---|---|---|---|
85
+ | 3.228 | 3.456 | 1.768 | 2.575 | 2.757 |
86
+
87
+ The average is unweighted across the four sources. Code is much more predictable than prose, so it
88
+ sits below the training loss, which follows the 45/35/12/8 training mix. Weighted by that mix, the
89
+ held-out loss is 3.080.
90
+
91
+ ### Zero-shot benchmarks
92
+
93
+ | Task | Score |
94
+ |---|---|
95
+ | HellaSwag | 27.1 |
96
+ | ARC-Easy | 32.7 |
97
+ | ARC-Challenge | 22.7 |
98
+ | PIQA | 54.2 |
99
+ | WinoGrande | 48.6 |
100
+ | OpenBookQA | 24.6 |
101
+ | SciQ | 60.2 |
102
+ | LAMBADA | 16.8 |
103
+ | LAMBADA ppl | 266.9 |
104
+
105
+ Greedy samples right after training:
106
+ - `'The meaning of life is the same as the meaning of life. The meaning of life is the same as the meaning of life.\nThe meaning of life is the same as the meaning of life. It means that we know what we are doing and how we can do'`
107
+ - `'Once upon a time, the world was not as much of an issue. The world had to be more than just a single day and a week.\nThe world was very different from what we have today. We are all in our own homes. It is a great'`
108
+ - `'def is_prime(n): 10\n# 2. The number of digits in the first digit is 3, and the last digit is 4.\n# 3. The number of digits in the second digit is 5, and the last digit'`
109
+ - `'The derivative of x^2 is the product of its value. The derivative of y^2 is the product of its value.\n\n## 1. What are the derivatives?\n\n### 3. How do you find the derivative of x^2?\n\n### '`
110
+
111
+ ## Usage
112
+
113
+ ```python
114
+ from transformers import AutoTokenizer, AutoModelForCausalLM
115
+
116
+ repo = "DedeProGames/Wisp-5M"
117
+ tok = AutoTokenizer.from_pretrained(repo)
118
+ model = AutoModelForCausalLM.from_pretrained(repo)
119
+
120
+ inputs = tok("The water cycle is", return_tensors="pt")
121
+ out = model.generate(**inputs, max_new_tokens=60, do_sample=True, temperature=0.8, top_p=0.95,
122
+ repetition_penalty=1.1)
123
+ print(tok.decode(out[0], skip_special_tokens=True))
124
+ ```
125
+
126
+ The tokenizer prepends `<|endoftext|>` as BOS, which matches how documents were laid out during
127
+ training.
128
+
129
+ ## Limitations
130
+
131
+ A model this small has very limited knowledge and reasoning. It will produce fluent-looking but often
132
+ wrong or incoherent text, and it can reproduce biases present in web data. It is English-centric and
133
+ was not aligned or safety-tuned. Don't use it for anything where correctness matters.
134
+
135
+ ## License
136
+
137
+ Apache-2.0 for the model weights. Training data licenses: FineWeb-Edu, FineMath and The Stack v3 are
138
+ ODC-By; DCLM-Baseline is CC-BY-4.0. Code in The Stack v3 comes with its original licenses.
loss_curve.png ADDED