User01110 commited on
Commit
dcd51f5
·
verified ·
1 Parent(s): 4bcbe71

Upload tinyLM-8M-exp final novelty-gated Qwen3 checkpoint

Browse files
Files changed (3) hide show
  1. README.md +31 -16
  2. config.json +2 -2
  3. model.safetensors +2 -2
README.md CHANGED
@@ -11,17 +11,17 @@ tags:
11
 
12
  # tinyLM-8M-exp
13
 
14
- Tiny 5M-class parameter Qwen3-config causal LM with math-only novelty-gated GQA.
15
 
16
  ## Architecture
17
 
18
  | Item | Value |
19
  | --- | ---: |
20
  | Config type | `qwen3` |
21
- | Parameters | 8.919M |
22
- | Layers | 10 |
23
  | Hidden size | 256 |
24
- | MLP size | 768 |
25
  | Query heads | 8 |
26
  | KV heads | 4 |
27
  | Head dim | 32 |
@@ -40,13 +40,13 @@ Tiny 5M-class parameter Qwen3-config causal LM with math-only novelty-gated GQA.
40
  | --- | --- |
41
  | Tokenizer | `AxiomicLabs/GPT-S2-5M` |
42
  | Sequence length | 512 |
43
- | Microbatch size | 512 |
44
  | Gradient accumulation | 4 |
45
- | Effective batch size | 2048 |
46
- | Steps | 20,000 |
47
  | Validation cadence | every 1,000 steps |
48
- | Raw MC eval cadence | every 2,000 steps on ARC-Easy, ARC-Challenge, PIQA, HellaSwag |
49
- | LR schedule | warmup, cosine to min by 10,000, hold min to 15,000, cosine tail to zero by 20,000 |
50
  | Optimizer | Muon for middle 2D weights, AdamW for the rest |
51
  | Special-token policy | BOS/EOS are document-level; `<|im_start|>`/`<|im_end|>` are sequence-level |
52
 
@@ -62,9 +62,9 @@ Tiny 5M-class parameter Qwen3-config causal LM with math-only novelty-gated GQA.
62
  | --- | ---: |
63
  | Dataset | `Salesforce/wikitext`, `wikitext-103-raw-v1`, validation |
64
  | Context / stride | 512 / 256 |
65
- | Loss | 3.1546 |
66
- | Perplexity | 23.44 |
67
- | UTF-8 BPB | 1.4433 |
68
  | Scored tokens | 365,258 |
69
  | UTF-8 bytes | 1,151,766 |
70
 
@@ -76,13 +76,28 @@ import torch
76
 
77
  repo = "User01110/tinyLM-8M-exp"
78
  tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
79
- model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
 
 
 
 
 
80
 
81
- inputs = tokenizer("Once upon a time", return_tensors="pt").to(model.device)
82
- print(inputs.input_ids[0][:2].tolist()) # [<|im_start|>, <|bos|>]
 
83
 
84
  with torch.no_grad():
85
- output = model.generate(**inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_k=40)
 
 
 
 
 
 
 
 
 
86
 
87
  print(tokenizer.decode(output[0], skip_special_tokens=True))
88
  ```
 
11
 
12
  # tinyLM-8M-exp
13
 
14
+ Tiny 8M-class Qwen3-config causal LM with math-only novelty-gated GQA.
15
 
16
  ## Architecture
17
 
18
  | Item | Value |
19
  | --- | ---: |
20
  | Config type | `qwen3` |
21
+ | Parameters | 8.132M |
22
+ | Layers | 8 |
23
  | Hidden size | 256 |
24
+ | MLP size | 896 |
25
  | Query heads | 8 |
26
  | KV heads | 4 |
27
  | Head dim | 32 |
 
40
  | --- | --- |
41
  | Tokenizer | `AxiomicLabs/GPT-S2-5M` |
42
  | Sequence length | 512 |
43
+ | Microbatch size | 1024 |
44
  | Gradient accumulation | 4 |
45
+ | Effective batch size | 4096 |
46
+ | Steps | 10,000 |
47
  | Validation cadence | every 1,000 steps |
48
+ | Official lm-eval | after final Hub upload on ARC-Easy, ARC-Challenge, PIQA, HellaSwag |
49
+ | LR schedule | warmup, cosine to min by 10,000 |
50
  | Optimizer | Muon for middle 2D weights, AdamW for the rest |
51
  | Special-token policy | BOS/EOS are document-level; `<|im_start|>`/`<|im_end|>` are sequence-level |
52
 
 
62
  | --- | ---: |
63
  | Dataset | `Salesforce/wikitext`, `wikitext-103-raw-v1`, validation |
64
  | Context / stride | 512 / 256 |
65
+ | Loss | 3.2769 |
66
+ | Perplexity | 26.49 |
67
+ | UTF-8 BPB | 1.4992 |
68
  | Scored tokens | 365,258 |
69
  | UTF-8 bytes | 1,151,766 |
70
 
 
76
 
77
  repo = "User01110/tinyLM-8M-exp"
78
  tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
79
+ model = AutoModelForCausalLM.from_pretrained(
80
+ repo,
81
+ trust_remote_code=True,
82
+ torch_dtype="auto",
83
+ device_map="auto",
84
+ )
85
 
86
+ prompt = "The future of AI is"
87
+ inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
88
+ print(inputs.input_ids[0][:2].tolist()) # auto-prefix: [<|im_start|>, <bos>]
89
 
90
  with torch.no_grad():
91
+ output = model.generate(
92
+ **inputs,
93
+ max_new_tokens=512,
94
+ do_sample=True,
95
+ temperature=0.65,
96
+ top_k=30,
97
+ repetition_penalty=1.2,
98
+ pad_token_id=tokenizer.pad_token_id,
99
+ eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
100
+ )
101
 
102
  print(tokenizer.decode(output[0], skip_special_tokens=True))
103
  ```
config.json CHANGED
@@ -8,8 +8,8 @@
8
  },
9
  "vocab_size": 4098,
10
  "hidden_size": 256,
11
- "intermediate_size": 768,
12
- "num_hidden_layers": 10,
13
  "num_attention_heads": 8,
14
  "num_key_value_heads": 4,
15
  "head_dim": 32,
 
8
  },
9
  "vocab_size": 4098,
10
  "hidden_size": 256,
11
+ "intermediate_size": 896,
12
+ "num_hidden_layers": 8,
13
  "num_attention_heads": 8,
14
  "num_key_value_heads": 4,
15
  "head_dim": 32,
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:1ef0aff6c3442e38c986a8c47f88e9e06b7bb37ac884a3e5595228e8b97b17ec
3
- size 35688600
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:96aa6a05a7d1de5562ce71a856395c929cb16e3d25eefce03a7964e52300faad
3
+ size 32536112