KitsuVp commited on
Commit
3d28735
·
verified ·
1 Parent(s): 35255d6

Model save

Browse files
Files changed (4) hide show
  1. README.md +43 -177
  2. config.json +6 -1
  3. model.safetensors +2 -2
  4. training_args.bin +1 -1
README.md CHANGED
@@ -1,197 +1,63 @@
1
  ---
2
- language: en
3
- license: apache-2.0
4
  tags:
5
- - causal-lm
6
- - research
7
- - fp8
8
- - attention
9
- - normalization
10
- - neollm
11
- datasets:
12
- - HuggingFaceFW/fineweb-edu
13
  ---
14
 
15
- # NeoLLM
16
-
17
- NeoLLM is a **135 M parameter** decoder-only language model trained from scratch on
18
- [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) in **FP8**
19
- precision, completing training in approximately **6 hours** on a single NVIDIA RTX 5090.
20
- It integrates a collection of recently published attention and normalization techniques
21
- into a single architecture, with the goal of studying how they interact during
22
- pretraining. The model is actively being developed and the current checkpoint represents
23
- an intermediate training state.
24
-
25
- > **Author / contact:** [@Kyokopom](https://x.com/Kyokopom) on X
26
- > **Repository:** [KitsuVp/NeoLLM](https://huggingface.co/KitsuVp/NeoLLM)
27
-
28
- ---
29
-
30
- ## Architecture
31
-
32
- NeoLLM is a decoder-only transformer with the following configuration:
33
-
34
- | Parameter | Value |
35
- |---|---|
36
- | Hidden size | 512 |
37
- | Layers | 12 |
38
- | Attention heads | 8 |
39
- | KV heads (GQA) | 4 |
40
- | Head dim | 64 |
41
- | Intermediate size | 1536 |
42
- | Vocabulary | Qwen3 tokenizer (64,402 tokens) |
43
- | Context length | 512 tokens |
44
-
45
- ### Parameter breakdown
46
-
47
- | Parameter bucket | Count |
48
- |---|---|
49
- | **Total parameters** | 116.22M (116,216,184) |
50
- | **Embedding parameters** (tied) | 32.97M (32,973,824) |
51
- | **Non-embedding parameters** | 83.24M (83,242,360) |
52
- | **Effective trainable parameters** | 116.22M (116,216,184) |
53
-
54
- > Weight tying is **enabled**: the input embedding matrix and the language-model head
55
- > share the same parameters, so the effective trainable budget is
56
- > `total − embed = 83.24M`.
57
-
58
- ### Integrated techniques
59
-
60
- Each layer combines the following mechanisms simultaneously.
61
-
62
- **Normalization and residual stream**
63
-
64
- - **SeeDNorm** ([arXiv:2510.22777](https://arxiv.org/abs/2510.22777)) — Applied to Q and K
65
- projections. Dynamically rescales the normalization based on the input's own statistics,
66
- making the attention geometry more stable across varying input distributions.
67
- - **PolyNorm** ([arXiv:2602.04902](https://arxiv.org/abs/2602.04902)) — Replaces the standard
68
- MLP activation with three branches: linear (x), quadratic (x²), and cubic (x³) — each
69
- normalized and combined with learned weights. This allows the MLP to express both linear
70
- and non-linear relationships simultaneously.
71
- - **GPAS** ([arXiv:2506.22049](https://arxiv.org/abs/2506.22049)) — Gradient-Preserving
72
- Activation Scaling. Applied to residual connections between sublayers; helps gradients
73
- flow more cleanly during training without distorting the residual stream.
74
- - **LayerNorm Scaling / LNS** ([arXiv:2502.05795](https://arxiv.org/abs/2502.05795)) — Each
75
- layer's output is scaled by 1/√ℓ where ℓ is the layer index. Directly addresses the
76
- "Curse of Depth" in Pre-LN transformers.
77
-
78
- **Attention mechanisms**
79
-
80
- - **FAN** ([arXiv:2502.21309](https://arxiv.org/abs/2502.21309)) — Fourier Analysis Networks.
81
- A portion of the input projection channels are dedicated to representing periodic patterns
82
- (cosine/sine pairs), while the remainder handle standard linear content.
83
- - **MEA** ([arXiv:2601.19611](https://arxiv.org/abs/2601.19611)) — Explicit Multi-head
84
- Attention. Adds small learnable interaction matrices between attention heads for K and V.
85
- - **LUCID** ([arXiv:2602.10410](https://arxiv.org/abs/2602.10410)) — Applies a learned
86
- lower-triangular preconditioner to V before attention, decorrelating value representations
87
- across positions.
88
- - **Affine-Scaled Attention** ([arXiv:2602.23057](https://arxiv.org/abs/2602.23057)) — Adds
89
- two learnable per-head scalars (α and β) to the softmax weights:
90
- `[α·softmax(QKᵀ) + β]·V`.
91
- - **XSA** ([arXiv:2603.09078](https://arxiv.org/abs/2603.09078)) — Exclusive Self Attention.
92
- After computing attention, removes the component of the output aligned with the token's
93
- own value vector.
94
- - **Directional Routing** ([arXiv:2603.14923](https://arxiv.org/abs/2603.14923)) — Each head
95
- learns K=4 directions in the output space; a learned router suppresses the attention output
96
- along each direction per input.
97
- - **Gated Attention** ([arXiv:2505.06708](https://arxiv.org/abs/2505.06708)) — A sigmoid gate
98
- is applied to the attention output before the output projection, introducing non-linearity
99
- and preventing attention sinks.
100
- - **Momentum Attention** ([arXiv:2411.03884](https://arxiv.org/abs/2411.03884)) — Modifies Q
101
- and K by subtracting a fraction of the previous position's Q and K values (causal
102
- first-difference).
103
-
104
- **MLP**
105
-
106
- - **Learnable Multipliers** ([arXiv:2601.04890](https://arxiv.org/abs/2601.04890)) — Adds
107
- per-row and per-column learnable scalar parameters to each linear layer.
108
- - **SimpleGPT** ([arXiv:2602.01212](https://arxiv.org/abs/2602.01212)) — A normalization
109
- strategy derived from second-order geometry analysis, applied inside MLP projections to
110
- improve optimization stability.
111
-
112
- ---
113
 
114
- ## Training
115
-
116
- | Setting | Value |
117
- |---|---|
118
- | Dataset | FineWeb-Edu (sample-10BT) |
119
- | Tokens seen | ~0.51B (15,625 steps × batch 64 × length 512) |
120
- | Precision | FP8 native (E4M3 weights/activations, E5M2 gradients) + BF16 fallback |
121
- | Optimizer | Conda (Column-Normalized Adam) |
122
- | Learning rate | 6e-04 with linear warmup (10 % of steps) |
123
- | Weight decay | 0.1 |
124
- | Training time | ~1h 25m |
125
- | Hardware | NVIDIA RTX 5090 (single GPU) |
126
-
127
- ### Training curve
128
 
129
- | Step | Train Loss | Val Loss |
130
- |---|---|---|
131
- | 5,000 | 4.098 | 4.043 |
132
- | 10,000 | 3.811 | 3.783 |
133
- | 15,000 | 3.637 | 3.596 |
134
- | 15,625 | — | 3.584 |
135
 
136
- ---
137
 
138
- ## Limitations
139
 
140
- - **Token budget** — ~1.5 B tokens seen; below estimated optimum. Knowledge-intensive tasks
141
- will improve with more training.
142
- - **Gradient spike at step 40k** — Reorganized the attention pattern in layer 9 that
143
- previously captured long-range token correlations. A checkpoint from ~step 38k is expected
144
- to have better aggregate benchmark scores.
145
- - **PolyNorm exclusivity** — The quadratic branch has become partially redundant with the
146
- linear branch. Will be corrected in the next training run.
147
- - **Base model only** — Not instruction-tuned or aligned; purely a next-token-prediction
148
- base model.
149
 
150
- ---
151
 
152
- ## References
153
-
154
- All papers whose techniques are integrated into NeoLLM's architecture:
155
-
156
- | Technique | Paper title | arXiv |
157
- |---|---|---|
158
- | SeeDNorm | Self-Rescaled Dynamic Normalization | [2510.22777](https://arxiv.org/abs/2510.22777) |
159
- | MEA | Explicit Multi-head Attention | [2601.19611](https://arxiv.org/abs/2601.19611) |
160
- | Learnable Multipliers | Freeing the Scale of Language Model Matrix Layers | [2601.04890](https://arxiv.org/abs/2601.04890) |
161
- | Directional Routing | Directional Routing in Transformers | [2603.14923](https://arxiv.org/abs/2603.14923) |
162
- | XSA | Exclusive Self Attention | [2603.09078](https://arxiv.org/abs/2603.09078) |
163
- | Gated Attention | Gated Attention for LLMs | [2505.06708](https://arxiv.org/abs/2505.06708) |
164
- | Affine-Scaled Attention | Affine-Scaled Attention | [2602.23057](https://arxiv.org/abs/2602.23057) |
165
- | LNS | The Curse of Depth in LLMs | [2502.05795](https://arxiv.org/abs/2502.05795) |
166
- | LUCID | Attention with Preconditioned Representations | [2602.10410](https://arxiv.org/abs/2602.10410) |
167
- | FAN | Fourier Analysis Networks | [2502.21309](https://arxiv.org/abs/2502.21309) |
168
- | SimpleGPT | SimpleGPT | [2602.01212](https://arxiv.org/abs/2602.01212) |
169
- | GPAS | Gradient-Preserving Activation Scaling | [2506.22049](https://arxiv.org/abs/2506.22049) |
170
- | PolyNorm | PolyNorm / PolyCom | [2602.04902](https://arxiv.org/abs/2602.04902) |
171
- | Momentum Attention | Momentum Attention | [2411.03884](https://arxiv.org/abs/2411.03884) |
172
- | TWEO (analysis ref.) | Transformers Without Extreme Outliers | [2511.23225](https://arxiv.org/abs/2511.23225) |
173
 
174
- ---
175
 
176
- ## Citation
177
 
178
- ```bibtex
179
- @misc{neollm2026,
180
- title = {NeoLLM: A Research Language Model Integrating Recent Attention and Normalization Techniques},
181
- author = {KitsuVp},
182
- year = {2026},
183
- url = {https://huggingface.co/KitsuVp/NeoLLM}
184
- }
185
- ```
186
 
187
- ---
 
 
 
 
 
 
 
 
188
 
189
- ## Author
190
 
191
- [@Kyokopom](https://x.com/Kyokopom) on X
 
 
 
 
 
192
 
193
- ---
194
 
195
- ## License
196
 
197
- Apache 2.0
 
 
 
 
1
  ---
2
+ library_name: transformers
 
3
  tags:
4
+ - generated_from_trainer
5
+ model-index:
6
+ - name: NeoLLM
7
+ results: []
 
 
 
 
8
  ---
9
 
10
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
11
+ should probably proofread and complete it, then remove this comment. -->
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
 
13
+ # NeoLLM
 
 
 
 
 
 
 
 
 
 
 
 
 
14
 
15
+ This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
16
+ It achieves the following results on the evaluation set:
17
+ - Loss: 3.4825
18
+ - Ntp Loss: 3.3540
19
+ - Tweo Loss: 0.0073
20
+ - Total Model Loss: 3.3541
21
 
22
+ ## Model description
23
 
24
+ More information needed
25
 
26
+ ## Intended uses & limitations
 
 
 
 
 
 
 
 
27
 
28
+ More information needed
29
 
30
+ ## Training and evaluation data
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
 
32
+ More information needed
33
 
34
+ ## Training procedure
35
 
36
+ ### Training hyperparameters
 
 
 
 
 
 
 
37
 
38
+ The following hyperparameters were used during training:
39
+ - learning_rate: 0.0006
40
+ - train_batch_size: 64
41
+ - eval_batch_size: 64
42
+ - seed: 42
43
+ - optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
44
+ - lr_scheduler_type: linear
45
+ - lr_scheduler_warmup_steps: 0.1
46
+ - num_epochs: 1
47
 
48
+ ### Training results
49
 
50
+ | Training Loss | Epoch | Step | Validation Loss | Loss | Model Loss |
51
+ |:-------------:|:-----:|:-----:|:---------------:|:------:|:----------:|
52
+ | 3.9978 | 0.32 | 5000 | 3.9580 | 0.0026 | 3.8440 |
53
+ | 3.7170 | 0.64 | 10000 | 3.6885 | 0.0064 | 3.5680 |
54
+ | 3.5323 | 0.96 | 15000 | 3.4943 | 0.0073 | 3.3645 |
55
+ | 3.5161 | 1.0 | 15625 | 3.4825 | 0.0073 | 3.3541 |
56
 
 
57
 
58
+ ### Framework versions
59
 
60
+ - Transformers 5.12.1
61
+ - Pytorch 2.12.1+cu130
62
+ - Datasets 5.0.0
63
+ - Tokenizers 0.22.2
config.json CHANGED
@@ -82,6 +82,10 @@
82
  "stack_slots": 16,
83
  "tie_word_embeddings": false,
84
  "transformers_version": "5.12.1",
 
 
 
 
85
  "use_affine_scaled_attention": false,
86
  "use_attn_res": false,
87
  "use_cache": false,
@@ -103,7 +107,7 @@
103
  "use_mea_attention": false,
104
  "use_momentum_attention": true,
105
  "use_nextlat": false,
106
- "use_nitp": true,
107
  "use_repo": true,
108
  "use_repo_goat_prior": false,
109
  "use_repo_grape": true,
@@ -111,6 +115,7 @@
111
  "use_spelling_bee_embeddings": true,
112
  "use_stack_memory": false,
113
  "use_token_generator": false,
 
114
  "use_xsa": true,
115
  "vocab_size": 64402,
116
  "xsa_eps": 1e-06
 
82
  "stack_slots": 16,
83
  "tie_word_embeddings": false,
84
  "transformers_version": "5.12.1",
85
+ "tweo_eps": 1e-06,
86
+ "tweo_loss_weight": 0.01,
87
+ "tweo_power": 4.0,
88
+ "tweo_tau": 3.0,
89
  "use_affine_scaled_attention": false,
90
  "use_attn_res": false,
91
  "use_cache": false,
 
107
  "use_mea_attention": false,
108
  "use_momentum_attention": true,
109
  "use_nextlat": false,
110
+ "use_nitp": false,
111
  "use_repo": true,
112
  "use_repo_goat_prior": false,
113
  "use_repo_grape": true,
 
115
  "use_spelling_bee_embeddings": true,
116
  "use_stack_memory": false,
117
  "use_token_generator": false,
118
+ "use_tweo": true,
119
  "use_xsa": true,
120
  "vocab_size": 64402,
121
  "xsa_eps": 1e-06
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:5c22d654dec3eb8efeade4d92d9b37cf943b54a774165cdbfdbefe824d4a7877
3
- size 240855132
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3323b1097851ef0a226e686324653a6ecd8c4940c7a34b785edf1910fdde2504
3
+ size 234563356
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:d1787bd21e8c93589c29727a05fd8361a56fc3df0647648f42ecb72e0299331a
3
  size 5329
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6afaa7827f84934199e32f295ded5465c960bff6dd6849e234fdcd3a45b99fa2
3
  size 5329