KitsuVp commited on
Commit
caa2359
·
verified ·
1 Parent(s): 16ee283

Model save

Browse files
Files changed (3) hide show
  1. README.md +72 -519
  2. model.safetensors +1 -1
  3. training_args.bin +1 -1
README.md CHANGED
@@ -1,525 +1,78 @@
1
  ---
2
- language: en
3
- license: apache-2.0
4
  tags:
5
- - causal-lm
6
- - research
7
- - bf16
8
- - attention
9
- - normalization
10
- - neollm
11
- datasets:
12
- - HuggingFaceFW/fineweb-edu
13
  ---
14
 
15
- # NeoLLM
16
-
17
- NeoLLM is a **84.57 M parameter** decoder-only language model trained from scratch on
18
- [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) with
19
- **BF16** compute, completing training in approximately
20
- **~0h 46m** on **NVIDIA RTX PRO 6000 Blackwell Server Edition**.
21
- It integrates a collection of recently published attention and normalization techniques
22
- into a single architecture, with the goal of studying how they interact during
23
- pretraining. The model is actively being developed and the current checkpoint represents
24
- an intermediate training state.
25
-
26
- > **Author / contact:** [@Kyokopom](https://x.com/Kyokopom) on X
27
- > **Repository:** [KitsuVp/NeoLLM](https://huggingface.co/KitsuVp/NeoLLM)
28
-
29
- ---
30
-
31
- ## Architecture
32
-
33
- NeoLLM is a decoder-only transformer with the following configuration:
34
-
35
- | Parameter | Value |
36
- |---|---|
37
- | Hidden size | 512 |
38
- | Layers | 12 |
39
- | Attention heads | 8 |
40
- | KV heads (GQA) | 4 |
41
- | Head dim | 64 |
42
- | Intermediate size | 1536 |
43
- | Vocabulary | LiquidAI/LFM2.5-1.2B-Thinking tokenizer (64,402 tokens) |
44
- | Context length | 512 tokens |
45
- | Leviathan | enabled=True; d_seed=128, modes=8, knots=16 |
46
- | JTok / JTok-M | enabled=False / False; Torch continuous surfaces, experts=5, top_k=2 |
47
- | DAGFormer | enabled=False; {'predictor_dim': 256, 'predictor_layers': 2, 'predictor_heads': 4, 'local_dim': 128, 'input_source': 'tokens'} |
48
-
49
- ### Parameter breakdown
50
-
51
- | Parameter bucket | Count |
52
- |---|---|
53
- | **Total parameters** | 84.57M (84,567,992) |
54
- | **Embedding parameters** (tied) | 32.97M (32,973,824) |
55
- | **Non-embedding parameters** | 51.59M (51,594,168) |
56
- | **Effective trainable parameters** | 84.57M (84,567,992) |
57
-
58
- > Weight tying is **enabled**: the input embedding matrix and the language-model head
59
- > share the same parameters, so the effective trainable budget is
60
- > `total − embed = 51.59M`.
61
-
62
- ### Integrated techniques
63
-
64
- NeoLLM combines architecture modules, optional auxiliary objectives, and
65
- training-time optimizer/stability components from the following papers.
66
-
67
- **Embedding and token representation**
68
-
69
- - **Learnable Multipliers** ([arXiv:2601.04890](https://arxiv.org/abs/2601.04890)) — Adds
70
- per-row and per-column learnable scalar parameters to selected matrix layers and, when
71
- enabled, embeddings.
72
- - **Leviathan** ([arXiv:2601.22040](https://arxiv.org/abs/2601.22040)) — Optional
73
- continuous token embedding generator that can replace the discrete input lookup table.
74
- - **Leviathan-JTok / JTok-M** ([arXiv:2602.00800](https://arxiv.org/abs/2602.00800)) —
75
- Optional continuous adaptation of JTok over the shared Leviathan coordinate. JTok
76
- applies one normalized surface as a multiplicative MLP gate; JTok-M routes a
77
- Sigmoid+TopK mixture of continuous surfaces and adds the paper's depth-scaled
78
- residual. Runtime ``jtok_kernel_backend`` selects the PyTorch reference or the
79
- CCE Triton adapter; the JTok surfaces remain outside MXFP8 conversion.
80
- - **KHRONOS** ([arXiv:2505.13315](https://arxiv.org/abs/2505.13315)) — Kernel/basis
81
- reference used by the Leviathan continuous token generator implementation.
82
- - **Spelling Bee Embeddings** ([arXiv:2601.18030](https://arxiv.org/abs/2601.18030)) —
83
- Augments token embeddings with character-level spelling information.
84
- - **Token Embedding Manifold analysis** ([arXiv:2504.01002](https://arxiv.org/abs/2504.01002)) —
85
- Reference motivation for treating token embeddings as structured objects rather than
86
- unconstrained lookup rows.
87
-
88
- **Attention, positions, and output projection**
89
-
90
- - **FAN** ([arXiv:2502.21309](https://arxiv.org/abs/2502.21309)) — Fourier Analysis Networks.
91
- A portion of the projection channels are dedicated to periodic cosine/sine features.
92
- - **MEA** ([arXiv:2601.19611](https://arxiv.org/abs/2601.19611)) — Explicit Multi-head
93
- Attention. Adds small learnable interaction matrices between attention heads for K and V.
94
- - **LUCID** ([arXiv:2602.10410](https://arxiv.org/abs/2602.10410)) — Applies a learned
95
- lower-triangular preconditioner to V before attention, decorrelating value representations
96
- across positions.
97
- - **Affine-Scaled Attention** ([arXiv:2602.23057](https://arxiv.org/abs/2602.23057)) — Adds
98
- two learnable per-head scalars (α and β) to the softmax weights:
99
- `[α·softmax(QKᵀ) + β]·V`.
100
- - **XSA** ([arXiv:2603.09078](https://arxiv.org/abs/2603.09078)) — Exclusive Self Attention.
101
- After computing attention, removes the component of the output aligned with the token's
102
- own value vector.
103
- - **Directional Routing** ([arXiv:2603.14923](https://arxiv.org/abs/2603.14923)) — Each head
104
- learns K=4 directions in the output space; a learned router suppresses the attention output
105
- along each direction per input.
106
- - **Gated Attention** ([arXiv:2505.06708](https://arxiv.org/abs/2505.06708)) — A sigmoid gate
107
- is applied to the attention output before the output projection, introducing non-linearity
108
- and preventing attention sinks.
109
- - **Momentum Attention** ([arXiv:2411.03884](https://arxiv.org/abs/2411.03884)) — Modifies Q
110
- and K by subtracting a fraction of the previous position's Q and K values (causal
111
- first-difference).
112
- - **Interleaved Head Attention / IHA** ([arXiv:2602.21371](https://arxiv.org/abs/2602.21371)) —
113
- Builds pseudo-heads from learned cross-head mixtures to create multiple attention patterns
114
- per original head.
115
- - **REPO** ([arXiv:2512.14391](https://arxiv.org/abs/2512.14391)) — Context re-positioning
116
- module that learns contextual position coordinates above a configurable start layer.
117
- - **GRAPE** ([arXiv:2512.07805](https://arxiv.org/abs/2512.07805)) — Group representational
118
- position encoding used by the REPO-GRAPE positional path.
119
- - **GOAT priors** ([arXiv:2601.15380](https://arxiv.org/abs/2601.15380)) — Optional
120
- factorized attention log-prior channels inspired by trainable attention priors.
121
- - **Hadamard output projection** ([arXiv:2603.08343](https://arxiv.org/abs/2603.08343)) —
122
- Replaces dense attention output projection with a structured Hadamard transform plus
123
- lightweight scaling.
124
-
125
- **Normalization, residual flow, and MLP**
126
-
127
- - **SeeDNorm** ([arXiv:2510.22777](https://arxiv.org/abs/2510.22777)) — Applied to Q and K
128
- projections. Dynamically rescales normalization from the input's own statistics.
129
- - **LayerNorm Scaling / LNS** ([arXiv:2502.05795](https://arxiv.org/abs/2502.05795)) — Each
130
- layer's output is scaled by 1/√ℓ where ℓ is the layer index.
131
- - **GPAS** ([arXiv:2506.22049](https://arxiv.org/abs/2506.22049)) — Gradient-Preserving
132
- Activation Scaling for residual junctions.
133
- - **PolyNorm** ([arXiv:2602.04902](https://arxiv.org/abs/2602.04902)) — Replaces the standard
134
- MLP activation with normalized linear, quadratic, and cubic branches.
135
- - **SimpleGPT** ([arXiv:2602.01212](https://arxiv.org/abs/2602.01212)) — Second-order
136
- geometry-inspired normalization strategy applied inside MLP projections.
137
- - **StackMemory / STACKTRANS** ([NeurIPS 2025](https://openreview.net/forum?id=2bbDg587uh)) —
138
- Optional differentiable hidden-state stack between decoder layers.
139
- - **Attention Residuals / AttnRes** ([arXiv:2603.15031](https://arxiv.org/abs/2603.15031)) —
140
- Optional learned depth-wise aggregation over previous layer outputs or block summaries.
141
- - **LAUREL** ([arXiv:2411.07501](https://arxiv.org/abs/2411.07501)) — Optional learned
142
- augmented residual layer with residual-weight and low-rank variants.
143
-
144
- **Training objectives and training-time regularizers**
145
-
146
- - **Cut Cross Entropy** ([Apple repository](https://github.com/apple/ml-cross-entropy)) —
147
- Memory-efficient next-token loss that avoids materializing the full token-by-vocabulary
148
- logits tensor. NeoLLM remains compatible with the upstream package when the extensions
149
- below are disabled.
150
- - **MiLe Loss** ([arXiv:2310.19531](https://arxiv.org/abs/2310.19531)) — Optional detached,
151
- mean-normalized predictive-entropy weighting of token losses, implemented inside the
152
- extended CCE path.
153
- - **Output Embedding Centering / mu-loss**
154
- ([arXiv:2601.02031](https://arxiv.org/abs/2601.02031)) — Optional
155
- `lambda * ||mean(output_embeddings)||^2` regularizer for output-logit stability.
156
- - **MEAP** ([arXiv:2502.07490](https://arxiv.org/abs/2502.07490)) — Optional training-only
157
- input corruption that masks a fixed fraction of eligible tokens while preserving clean
158
- next-token labels, causal attention, and the inference path.
159
- - **TWEO** ([arXiv:2511.23225](https://arxiv.org/abs/2511.23225)) — Optional
160
- Transformers Without Extreme Outliers activation regularizer for FP8/low-bit-friendly
161
- training.
162
- - **NITP** ([arXiv:2605.24956](https://arxiv.org/abs/2605.24956)) — Optional Next Implicit
163
- Token Prediction auxiliary objective using shallow-layer implicit token targets and a
164
- cosine loss.
165
- - **NextLat** ([arXiv:2511.05963](https://arxiv.org/abs/2511.05963)) — Optional next-latent
166
- prediction objective using latent dynamics, Smooth L1 supervision, and frozen-head KL.
167
-
168
- ### Optional extended-CCE configuration
169
-
170
- | Feature | Enabled | Value |
171
- |---|---:|---:|
172
- | MiLe Loss | True | gamma=1.0 |
173
- | mu-loss | True | lambda=0.0001 |
174
- | MEAP | True | ratio=0.15 |
175
-
176
- MiLe, mu-loss, and MEAP require the extended
177
- [`Kitsunp/ml-cross-entropy`](https://github.com/Kitsunp/ml-cross-entropy) package only when
178
- their corresponding flags are enabled. With all flags disabled, NeoLLM calls upstream CCE
179
- without extension-specific arguments. When any extension is active, CCE reports three compact
180
- scalars: unweighted NTP cross entropy, the MiLe reweighting delta, and the mu-loss penalty.
181
- Their sum reconstructs `ntp_loss` exactly. MEAP reports its eligible and selected counts from
182
- the masking kernel; the trainer logs the selected count and exact fraction. No diagnostic path
183
- materializes full-vocabulary logits or a token mask outside the kernels.
184
-
185
- **Optimizer and training stability**
186
-
187
- - **Adam-family optimizer mode** — Active variant: **AdEMAMix**.
188
- **AdEMAMix** ([arXiv:2409.03137](https://arxiv.org/abs/2409.03137)) contributes the optional
189
- beta3 slow-memory branch when `ADEMAMIX=True`. **DeltaMomentum**
190
- ([arXiv:2608.19491](https://arxiv.org/abs/2608.19491)) optionally replaces only the fast beta1
191
- first-moment update on hidden Linear weights when `DELTA=True`; beta2 remains Adam's
192
- ordinary squared-gradient EMA in every mode.
193
- - **BF16 stochastic rounding** ([arXiv:2502.20566](https://arxiv.org/abs/2502.20566)) —
194
- Optional master-free BF16 model update with stochastic rather than nearest rounding; the same
195
- write-back path is shared by AdamW, DeltaAdamW, AdEMAMix, and DeltaAdEMAMix.
196
- - **Cautious Weight Decay** ([arXiv:2510.12402](https://arxiv.org/abs/2510.12402)) —
197
- Sign-selective weight decay variant used by the custom optimizer logic.
198
- - **Correction of Decoupled Weight Decay** ([arXiv:2512.08217](https://arxiv.org/abs/2512.08217)) —
199
- Adapts decoupled weight decay during learning-rate decay.
200
- - **AdamHD** ([arXiv:2511.14721](https://arxiv.org/abs/2511.14721)) —
201
- Decoupled Huber decay regularization reference used by the optimizer.
202
- - **Gradient conditioning** — CAST active; history-adaptive spectral/norm policies by optimizer group; 15625 committed updates.
203
- References: [CAST](https://arxiv.org/abs/2610.19761),
204
- [GradientStabilizer](https://arxiv.org/abs/2502.17055).
205
- ---
206
-
207
- ## Training
208
-
209
- | Setting | Value |
210
- |---|---|
211
- | Dataset | FineWeb-Edu (sample-10BT) |
212
- | Tokens seen | ~0.51B (15,625 steps × batch 64 × length 512) |
213
- | Precision | BF16 model state (bfloat16: 84,565,816 elements, float32: 2,176 elements) + AdEMAMix FP32 master active; stochastic rounding disabled |
214
- | Optimizer | AdEMAMix |
215
- | Gradient transform | CAST active; history-adaptive spectral/norm policies by optimizer group; 15625 committed updates |
216
- | Learning rate | 1e-03 with 4,000-step linear warmup |
217
- | Weight decay | coefficients [0.005, 0.01, 0.1]; WD 5/5 groups; CWD 2; Huber 0; Correction 0 |
218
- | Training time | ~0h 46m |
219
- | Hardware | NVIDIA RTX PRO 6000 Blackwell Server Edition |
220
-
221
 
222
- ### Training curve — total objective
223
-
224
- Total includes MiLe when enabled, plus all active auxiliary objectives and regularizers. Without MiLe means model total minus its MiLe delta on the same forward; this is not a separate training run without MiLe.
225
-
226
- | Step | Train total | Val total | Train without MiLe | Val without MiLe |
227
- |---|---|---|---|---|
228
- | 1 | 12.1388 | — | 12.1388 | — |
229
- | 250 | 7.98512 | — | 7.97511 | — |
230
- | 500 | 7.8254 | — | 7.80911 | — |
231
- | 750 | 7.69152 | — | 7.67302 | — |
232
- | 1,000 | 7.60931 | — | 7.58873 | — |
233
- | 1,250 | 7.55672 | — | 7.53484 | — |
234
- | 1,500 | 7.52617 | — | 7.50214 | — |
235
- | 1,750 | 7.49842 | — | 7.47334 | — |
236
- | 2,000 | 7.47787 | — | 7.45123 | — |
237
- | 2,250 | 7.47165 | — | 7.4431 | — |
238
- | 2,500 | 7.44367 | — | 7.41406 | — |
239
- | 2,750 | 7.42427 | — | 7.39387 | — |
240
- | 3,000 | 7.40542 | — | 7.37407 | — |
241
- | 3,250 | 7.38995 | — | 7.35702 | — |
242
- | 3,500 | 7.37661 | — | 7.34258 | — |
243
- | 3,750 | 7.3544 | — | 7.31952 | — |
244
- | 4,000 | 7.34423 | — | 7.30894 | — |
245
- | 4,250 | 7.32617 | — | 7.28946 | — |
246
- | 4,500 | 7.30903 | — | 7.27187 | — |
247
- | 4,750 | 7.28931 | — | 7.2511 | — |
248
- | 5,000 | 7.26439 | 7.42393 | 7.22555 | 7.36892 |
249
- | 5,250 | 7.24687 | — | 7.20728 | — |
250
- | 5,500 | 7.23375 | — | 7.19312 | — |
251
- | 5,750 | 7.21333 | — | 7.17154 | — |
252
- | 6,000 | 7.1986 | — | 7.15614 | — |
253
- | 6,250 | 7.18516 | — | 7.14238 | — |
254
- | 6,500 | 7.1682 | — | 7.12551 | — |
255
- | 6,750 | 7.15914 | — | 7.11515 | — |
256
- | 7,000 | 7.1406 | — | 7.09638 | — |
257
- | 7,250 | 7.13519 | — | 7.09033 | — |
258
- | 7,500 | 7.12815 | — | 7.08269 | — |
259
- | 7,750 | 7.1143 | — | 7.06872 | — |
260
- | 8,000 | 7.10478 | — | 7.05756 | — |
261
- | 8,250 | 7.09332 | — | 7.04579 | — |
262
- | 8,500 | 7.09115 | — | 7.04364 | — |
263
- | 8,750 | 7.09173 | — | 7.04375 | — |
264
- | 9,000 | 7.08177 | — | 7.03372 | — |
265
- | 9,250 | 7.06866 | — | 7.01952 | — |
266
- | 9,500 | 7.06379 | — | 7.01455 | — |
267
- | 9,750 | 7.06571 | — | 7.01678 | — |
268
- | 10,000 | 7.05078 | 7.20018 | 7.00036 | 7.13484 |
269
- | 10,250 | 7.05115 | — | 7.00069 | — |
270
- | 10,500 | 6.84575 | — | 6.73927 | — |
271
- | 10,750 | 5.93191 | — | 5.6053 | — |
272
- | 11,000 | 5.76889 | — | 5.40413 | — |
273
- | 11,250 | 5.68452 | — | 5.29233 | — |
274
- | 11,500 | 5.62871 | — | 5.21546 | — |
275
- | 11,750 | 5.58288 | — | 5.15794 | — |
276
- | 12,000 | 5.55091 | — | 5.11768 | — |
277
- | 12,250 | 5.51622 | — | 5.07728 | — |
278
- | 12,500 | 5.48044 | — | 5.03584 | — |
279
- | 12,750 | 5.46397 | — | 5.01222 | — |
280
- | 13,000 | 5.43349 | — | 4.97721 | — |
281
- | 13,250 | 5.415 | — | 4.95451 | — |
282
- | 13,500 | 5.39183 | — | 4.92584 | — |
283
- | 13,750 | 5.38363 | — | 4.91604 | — |
284
- | 14,000 | 5.35952 | — | 4.88958 | — |
285
- | 14,250 | 5.34301 | — | 4.87144 | — |
286
- | 14,500 | 5.32907 | — | 4.8535 | — |
287
- | 14,750 | 5.31257 | — | 4.83203 | — |
288
- | 15,000 | 5.31041 | 4.92658 | 4.83019 | 4.34564 |
289
- | 15,250 | 5.28783 | — | 4.80606 | — |
290
- | 15,500 | 5.27966 | — | 4.79588 | — |
291
- | 15,625 | 5.26531 | 4.89987 | 4.78016 | 4.30866 |
292
-
293
- ### Cross-entropy with and without MiLe
294
-
295
- CE without MiLe is unweighted CE; CE with MiLe adds the reweighting delta. Both exclude mu and auxiliary terms. `ntp_loss` includes mu when enabled.
296
-
297
- | Step | Train CE without MiLe | Val CE without MiLe | Train CE with MiLe | Val CE with MiLe |
298
- |---|---|---|---|---|
299
- | 1 | 11.1797 | — | 11.1797 | — |
300
- | 250 | 7.70254 | — | 7.71255 | — |
301
- | 500 | 7.37642 | — | 7.39271 | — |
302
- | 750 | 7.23382 | — | 7.25233 | — |
303
- | 1,000 | 7.1488 | — | 7.16938 | — |
304
- | 1,250 | 7.08809 | — | 7.10998 | — |
305
- | 1,500 | 7.04248 | — | 7.06651 | — |
306
- | 1,750 | 6.99841 | — | 7.02348 | — |
307
- | 2,000 | 6.96604 | — | 6.99268 | — |
308
- | 2,250 | 6.94518 | — | 6.97373 | — |
309
- | 2,500 | 6.91234 | — | 6.94194 | — |
310
- | 2,750 | 6.88875 | — | 6.91915 | — |
311
- | 3,000 | 6.86703 | — | 6.89838 | — |
312
- | 3,250 | 6.84514 | — | 6.87807 | — |
313
- | 3,500 | 6.82863 | — | 6.86265 | — |
314
- | 3,750 | 6.80252 | — | 6.8374 | — |
315
- | 4,000 | 6.7882 | — | 6.8235 | — |
316
- | 4,250 | 6.76929 | — | 6.80601 | — |
317
- | 4,500 | 6.75193 | — | 6.78909 | — |
318
- | 4,750 | 6.73117 | — | 6.76937 | — |
319
- | 5,000 | 6.70779 | 6.66266 | 6.74662 | 6.71767 |
320
- | 5,250 | 6.69276 | — | 6.73236 | — |
321
- | 5,500 | 6.68077 | — | 6.72141 | — |
322
- | 5,750 | 6.66076 | — | 6.70255 | — |
323
- | 6,000 | 6.64853 | — | 6.69098 | — |
324
- | 6,250 | 6.6371 | — | 6.67987 | — |
325
- | 6,500 | 6.62383 | — | 6.66651 | — |
326
- | 6,750 | 6.61443 | — | 6.65842 | — |
327
- | 7,000 | 6.5967 | — | 6.64092 | — |
328
- | 7,250 | 6.59131 | — | 6.63617 | — |
329
- | 7,500 | 6.58502 | — | 6.63047 | — |
330
- | 7,750 | 6.57292 | — | 6.6185 | — |
331
- | 8,000 | 6.56308 | — | 6.61029 | — |
332
- | 8,250 | 6.55154 | — | 6.59907 | — |
333
- | 8,500 | 6.54732 | — | 6.59483 | — |
334
- | 8,750 | 6.54686 | — | 6.59484 | — |
335
- | 9,000 | 6.53598 | — | 6.58403 | — |
336
- | 9,250 | 6.52115 | — | 6.57028 | — |
337
- | 9,500 | 6.51535 | — | 6.56459 | — |
338
- | 9,750 | 6.51733 | — | 6.56626 | — |
339
- | 10,000 | 6.50188 | 6.44648 | 6.55229 | 6.51183 |
340
- | 10,250 | 6.50001 | — | 6.55046 | — |
341
- | 10,500 | 6.22545 | — | 6.33193 | — |
342
- | 10,750 | 5.14674 | — | 5.47334 | — |
343
- | 11,000 | 4.9484 | — | 5.31316 | — |
344
- | 11,250 | 4.8344 | — | 5.2266 | — |
345
- | 11,500 | 4.75564 | — | 5.1689 | — |
346
- | 11,750 | 4.69681 | — | 5.12174 | — |
347
- | 12,000 | 4.65506 | — | 5.0883 | — |
348
- | 12,250 | 4.61458 | — | 5.05351 | — |
349
- | 12,500 | 4.57247 | — | 5.01707 | — |
350
- | 12,750 | 4.54935 | — | 5.0011 | — |
351
- | 13,000 | 4.51444 | — | 4.97073 | — |
352
- | 13,250 | 4.49163 | — | 4.95212 | — |
353
- | 13,500 | 4.46322 | — | 4.92921 | — |
354
- | 13,750 | 4.45248 | — | 4.92008 | — |
355
- | 14,000 | 4.4259 | — | 4.89584 | — |
356
- | 14,250 | 4.40784 | — | 4.87941 | — |
357
- | 14,500 | 4.38954 | — | 4.86512 | — |
358
- | 14,750 | 4.36861 | — | 4.84916 | — |
359
- | 15,000 | 4.3654 | 3.90764 | 4.84562 | 4.48858 |
360
- | 15,250 | 4.342 | — | 4.82377 | — |
361
- | 15,500 | 4.33173 | — | 4.8155 | — |
362
- | 15,625 | 4.31639 | 3.87037 | 4.80155 | 4.46158 |
363
-
364
- ### Observed loss components — weighted contributions
365
-
366
- Only contributions actually added to the objective are shown; raw values and subcomponents remain in training metrics. `regularization_loss` is the subtotal of mu, weighted TWeo, JTok-M and StackMemory entropy, excluding NITP/NextLat; do not add it again to those components. JTok-M auxiliary loss is training-only.
367
-
368
- | Step | Phase | mu | TWeo | NITP | Reg. subtotal |
369
- |---|---|---|---|---|---|
370
- | 1 | train | 3.18928e-10 | 8.85304e-07 | 0.959145 | 8.85623e-07 |
371
- | 250 | train | 5.68541e-06 | 0.000889012 | 0.271676 | 0.000894698 |
372
- | 500 | train | 1.39317e-05 | 0.00048898 | 0.43219 | 0.000502912 |
373
- | 750 | train | 2.39475e-05 | 0.000592826 | 0.438579 | 0.000616773 |
374
- | 1,000 | train | 3.58659e-05 | 0.000741196 | 0.439151 | 0.000777062 |
375
- | 1,250 | train | 4.88121e-05 | 0.000991387 | 0.445703 | 0.0010402 |
376
- | 1,500 | train | 6.18302e-05 | 0.00132114 | 0.458276 | 0.00138297 |
377
- | 1,750 | train | 7.31645e-05 | 0.00163189 | 0.47323 | 0.00170506 |
378
- | 2,000 | train | 8.32728e-05 | 0.00189371 | 0.483209 | 0.00197699 |
379
- | 2,250 | train | 9.34909e-05 | 0.00216144 | 0.49566 | 0.00225493 |
380
- | 2,500 | train | 0.000104415 | 0.00233031 | 0.499293 | 0.00243473 |
381
- | 2,750 | train | 0.000115916 | 0.00246703 | 0.502539 | 0.00258294 |
382
- | 3,000 | train | 0.000127831 | 0.00244716 | 0.504458 | 0.002575 |
383
- | 3,250 | train | 0.000141259 | 0.00240863 | 0.509329 | 0.00254989 |
384
- | 3,500 | train | 0.0001555 | 0.00235224 | 0.51145 | 0.00250774 |
385
- | 3,750 | train | 0.000170426 | 0.00234662 | 0.51449 | 0.00251704 |
386
- | 4,000 | train | 0.000185567 | 0.00222962 | 0.518318 | 0.00241518 |
387
- | 4,250 | train | 0.000203734 | 0.00213374 | 0.51783 | 0.00233748 |
388
- | 4,500 | train | 0.00022362 | 0.00194741 | 0.517777 | 0.00217103 |
389
- | 4,750 | train | 0.000244492 | 0.00176364 | 0.517932 | 0.00200813 |
390
- | 5,000 | train | 0.000269174 | 0.00161852 | 0.515877 | 0.00188769 |
391
- | 5,000 | validation | 0.000281882 | 0.0219972 | 0.683979 | 0.0222791 |
392
- | 5,250 | train | 0.0002948 | 0.00152122 | 0.512702 | 0.00181602 |
393
- | 5,500 | train | 0.000320074 | 0.00141181 | 0.510612 | 0.00173188 |
394
- | 5,750 | train | 0.000344429 | 0.00129077 | 0.50915 | 0.0016352 |
395
- | 6,000 | train | 0.000369502 | 0.00119281 | 0.506049 | 0.00156231 |
396
- | 6,250 | train | 0.00039563 | 0.00114157 | 0.503747 | 0.0015372 |
397
- | 6,500 | train | 0.000424725 | 0.00111766 | 0.500143 | 0.00154238 |
398
- | 6,750 | train | 0.000454083 | 0.00109027 | 0.499179 | 0.00154435 |
399
- | 7,000 | train | 0.00048038 | 0.00103857 | 0.498165 | 0.00151895 |
400
- | 7,250 | train | 0.00050527 | 0.000984053 | 0.497533 | 0.00148932 |
401
- | 7,500 | train | 0.000530754 | 0.000993553 | 0.49615 | 0.00152431 |
402
- | 7,750 | train | 0.000560454 | 0.000970533 | 0.494275 | 0.00153099 |
403
- | 8,000 | train | 0.000594531 | 0.000919749 | 0.492966 | 0.00151428 |
404
- | 8,250 | train | 0.000624764 | 0.000917213 | 0.492705 | 0.00154198 |
405
- | 8,500 | train | 0.000650141 | 0.00152388 | 0.494143 | 0.00217402 |
406
- | 8,750 | train | 0.000673515 | 0.000843965 | 0.495374 | 0.00151748 |
407
- | 9,000 | train | 0.000697932 | 0.000824539 | 0.496217 | 0.00152247 |
408
- | 9,250 | train | 0.000722145 | 0.000783098 | 0.496869 | 0.00150524 |
409
- | 9,500 | train | 0.000745877 | 0.000778402 | 0.497681 | 0.00152428 |
410
- | 9,750 | train | 0.000765973 | 0.000764489 | 0.497919 | 0.00153046 |
411
- | 10,000 | train | 0.000788166 | 0.000781291 | 0.496911 | 0.00156946 |
412
- | 10,000 | validation | 0.00080187 | 0.00311829 | 0.684435 | 0.00392016 |
413
- | 10,250 | train | 0.000816356 | 0.000781903 | 0.499086 | 0.00159826 |
414
- | 10,500 | train | 0.000843664 | 0.00125388 | 0.511724 | 0.00209754 |
415
- | 10,750 | train | 0.000872123 | 0.000880954 | 0.456807 | 0.00175308 |
416
- | 11,000 | train | 0.000892194 | 0.000714178 | 0.454127 | 0.00160637 |
417
- | 11,250 | train | 0.000900344 | 0.000727329 | 0.456301 | 0.00162767 |
418
- | 11,500 | train | 0.000902344 | 0.000759315 | 0.45815 | 0.00166166 |
419
- | 11,750 | train | 0.000908495 | 0.000824227 | 0.459405 | 0.00173272 |
420
- | 12,000 | train | 0.00091642 | 0.000833896 | 0.460866 | 0.00175032 |
421
- | 12,250 | train | 0.0009223 | 0.000845974 | 0.460934 | 0.00176827 |
422
- | 12,500 | train | 0.000924763 | 0.000815765 | 0.46163 | 0.00174053 |
423
- | 12,750 | train | 0.000923761 | 0.000778682 | 0.461164 | 0.00170244 |
424
- | 13,000 | train | 0.000920588 | 0.000750611 | 0.461097 | 0.0016712 |
425
- | 13,250 | train | 0.000917439 | 0.000692365 | 0.461269 | 0.0016098 |
426
- | 13,500 | train | 0.000913774 | 0.000652368 | 0.461055 | 0.00156614 |
427
- | 13,750 | train | 0.000910805 | 0.000610711 | 0.462036 | 0.00152152 |
428
- | 14,000 | train | 0.00090898 | 0.000583299 | 0.462182 | 0.00149228 |
429
- | 14,250 | train | 0.000907558 | 0.000552937 | 0.462141 | 0.0014605 |
430
- | 14,500 | train | 0.00090634 | 0.000519922 | 0.462532 | 0.00142626 |
431
- | 14,750 | train | 0.00090718 | 0.000488421 | 0.462017 | 0.0013956 |
432
- | 15,000 | train | 0.000909732 | 0.000724183 | 0.463154 | 0.00163391 |
433
- | 15,000 | validation | 0.000911385 | 0.000499765 | 0.436584 | 0.00141115 |
434
- | 15,250 | train | 0.000913244 | 0.000435868 | 0.46271 | 0.00134911 |
435
- | 15,500 | train | 0.000916544 | 0.000419084 | 0.462817 | 0.00133563 |
436
- | 15,625 | train | 0.000918795 | 0.000406611 | 0.462439 | 0.00132541 |
437
- | 15,625 | validation | 0.00091954 | 0.000456212 | 0.436919 | 0.00137575 |
438
-
439
- Tables use exact logged steps; missing values are — without carrying values forward. Repeated evaluations at one step use the latest logged value. Components average batch scalars weighted by examples; CE/PPL token metrics retain their valid-target weighting. Weight decay/CWD act in the optimizer, outside these forward losses.
440
-
441
- ---
442
-
443
- ## Limitations
444
-
445
- - **Token budget** — ~1.5 B tokens seen; below estimated optimum. Knowledge-intensive tasks
446
- will improve with more training.
447
- - **Gradient spike at step 40k** — Reorganized the attention pattern in layer 9 that
448
- previously captured long-range token correlations. A checkpoint from ~step 38k is expected
449
- to have better aggregate benchmark scores.
450
- - **PolyNorm exclusivity** — The quadratic branch has become partially redundant with the
451
- linear branch. Will be corrected in the next training run.
452
- - **Base model only** — Not instruction-tuned or aligned; purely a next-token-prediction
453
- base model.
454
-
455
- ---
456
-
457
- ## References
458
-
459
- All papers whose techniques are integrated into NeoLLM's architecture,
460
- training objective, or training stack:
461
-
462
- | Area | Technique | Paper title | Reference |
463
- |---|---|---|---|
464
- | Embeddings | Learnable Multipliers | Freeing the Scale of Language Model Matrix Layers | [arXiv:2601.04890](https://arxiv.org/abs/2601.04890) |
465
- | Embeddings | Leviathan | A Separable Architecture for Continuous Token Representation in Language Models | [arXiv:2601.22040](https://arxiv.org/abs/2601.22040) |
466
- | Embeddings / residual modulation | JTok / JTok-M | JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation | [arXiv:2602.00800](https://arxiv.org/abs/2602.00800) |
467
- | Embeddings | KHRONOS | KHRONOS: a Kernel-Based Neural Architecture for Rapid, Resource-Efficient Scientific Computation | [arXiv:2505.13315](https://arxiv.org/abs/2505.13315) |
468
- | Embeddings | Spelling Bee | Spelling Bee Embeddings for Language Modeling | [arXiv:2601.18030](https://arxiv.org/abs/2601.18030) |
469
- | Embeddings | Token embedding analysis | Token Embeddings Violate the Manifold Hypothesis | [arXiv:2504.01002](https://arxiv.org/abs/2504.01002) |
470
- | Attention / positions | FAN | Fourier Analysis Networks | [arXiv:2502.21309](https://arxiv.org/abs/2502.21309) |
471
- | Attention / positions | MEA | Explicit Multi-head Attention for Inter-head Interaction in Large Language Models | [arXiv:2601.19611](https://arxiv.org/abs/2601.19611) |
472
- | Attention / positions | LUCID | Attention with Preconditioned Representations | [arXiv:2602.10410](https://arxiv.org/abs/2602.10410) |
473
- | Attention / positions | Affine-Scaled Attention | Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention | [arXiv:2602.23057](https://arxiv.org/abs/2602.23057) |
474
- | Attention / positions | XSA | Exclusive Self Attention | [arXiv:2603.09078](https://arxiv.org/abs/2603.09078) |
475
- | Attention / positions | Directional Routing | Directional Routing in Transformers | [arXiv:2603.14923](https://arxiv.org/abs/2603.14923) |
476
- | Attention / positions | Gated Attention | Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free | [arXiv:2505.06708](https://arxiv.org/abs/2505.06708) |
477
- | Attention / positions | Momentum Attention | Momentum Attention | [arXiv:2411.03884](https://arxiv.org/abs/2411.03884) |
478
- | Attention / positions | IHA | Interleaved Head Attention | [arXiv:2602.21371](https://arxiv.org/abs/2602.21371) |
479
- | Attention / positions | REPO | Language Models with Context Re-Positioning | [arXiv:2512.14391](https://arxiv.org/abs/2512.14391) |
480
- | Attention / positions | GRAPE | Group Representational Position Encoding | [arXiv:2512.07805](https://arxiv.org/abs/2512.07805) |
481
- | Attention / positions | GOAT priors | You Need Better Attention Priors | [arXiv:2601.15380](https://arxiv.org/abs/2601.15380) |
482
- | Attention / positions | Hadamard o_proj | Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers | [arXiv:2603.08343](https://arxiv.org/abs/2603.08343) |
483
- | Residual / normalization | SeeDNorm | Self-Rescaled Dynamic Normalization | [arXiv:2510.22777](https://arxiv.org/abs/2510.22777) |
484
- | Residual / normalization | LNS | The Curse of Depth in LLMs | [arXiv:2502.05795](https://arxiv.org/abs/2502.05795) |
485
- | Residual / normalization | GPAS | Gradient-Preserving Activation Scaling | [arXiv:2506.22049](https://arxiv.org/abs/2506.22049) |
486
- | Residual / normalization | PolyNorm | PolyNorm / PolyCom | [arXiv:2602.04902](https://arxiv.org/abs/2602.04902) |
487
- | Residual / normalization | SimpleGPT | SimpleGPT | [arXiv:2602.01212](https://arxiv.org/abs/2602.01212) |
488
- | Residual / normalization | StackMemory / STACKTRANS | Recursive Transformer: Boosting Reasoning Ability with State Stack | [NeurIPS 2025](https://openreview.net/forum?id=2bbDg587uh) |
489
- | Residual / normalization | Attention Residuals | Attention Residuals | [arXiv:2603.15031](https://arxiv.org/abs/2603.15031) |
490
- | Residual / normalization | LAUREL | LAUREL: Learned Augmented Residual Layer | [arXiv:2411.07501](https://arxiv.org/abs/2411.07501) |
491
- | Objectives | TWEO | Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies | [arXiv:2511.23225](https://arxiv.org/abs/2511.23225) |
492
- | Objectives | NITP | Next Implicit Token Prediction for LLM Pre-training | [arXiv:2605.24956](https://arxiv.org/abs/2605.24956) |
493
- | Objectives | NextLat | Next-Latent Prediction Transformers Learn Compact World Models | [arXiv:2511.05963](https://arxiv.org/abs/2511.05963) |
494
- | Optimizer / training | AdEMAMix | The AdEMAMix Optimizer: Better, Faster, Older | [arXiv:2409.03137](https://arxiv.org/abs/2409.03137) |
495
- | Optimizer / training | DeltaMomentum | DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule | [arXiv:2608.19491](https://arxiv.org/abs/2608.19491) |
496
- | Optimizer / training | BF16 + SR | Stochastic Rounding for LLM Training: Theory and Practice | [arXiv:2502.20566](https://arxiv.org/abs/2502.20566) |
497
- | Optimizer / training | CWD | Cautious Weight Decay | [arXiv:2510.12402](https://arxiv.org/abs/2510.12402) |
498
- | Optimizer / training | WD correction | Correction of Decoupled Weight Decay | [arXiv:2512.08217](https://arxiv.org/abs/2512.08217) |
499
- | Optimizer / training | AdamHD | AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training | [arXiv:2511.14721](https://arxiv.org/abs/2511.14721) |
500
- | Optimizer / training | GradientStabilizer | GradientStabilizer | [arXiv:2502.17055](https://arxiv.org/abs/2502.17055) |
501
-
502
- ---
503
-
504
- ## Citation
505
-
506
- ```bibtex
507
- @misc{neollm2026,
508
- title = {NeoLLM: A Research Language Model Integrating Recent Attention and Normalization Techniques},
509
- author = {KitsuVp},
510
- year = {2026},
511
- url = {https://huggingface.co/KitsuVp/NeoLLM}
512
- }
513
- ```
514
-
515
- ---
516
-
517
- ## Author
518
-
519
- [@Kyokopom](https://x.com/Kyokopom) on X
520
-
521
- ---
522
-
523
- ## License
524
 
525
- Apache 2.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ library_name: transformers
 
3
  tags:
4
+ - generated_from_trainer
5
+ model-index:
6
+ - name: NeoLLM
7
+ results: []
 
 
 
 
8
  ---
9
 
10
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
11
+ should probably proofread and complete it, then remove this comment. -->
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
 
13
+ # NeoLLM
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
14
 
15
+ This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
16
+ It achieves the following results on the evaluation set:
17
+ - Loss: 4.7414
18
+ - Ce: 3.6766
19
+ - Valid Targets: 2083162.0
20
+ - Ppl: 39.5103
21
+ - Mile Loss: 4.3069
22
+ - Mile Ppl: 74.2125
23
+ - Top1 Accuracy: 0.3343
24
+ - Top3 Accuracy: 0.4945
25
+ - Topk Valid Targets: 2083162.0
26
+ - Ntp Loss: 4.3083
27
+ - Ntp Ce Unweighted: 3.6770
28
+ - Mile Reweighting Delta: 0.6303
29
+ - Mu Loss: 0.0011
30
+ - Tweo Loss: 0.0430
31
+ - Nitp Loss: 0.4327
32
+ - Total Model Loss: 4.7414
33
+ - Tweo Weighted Loss: 0.0004
34
+ - Nitp Weighted Loss: 0.4327
35
+ - Auxiliary Loss: 0.4331
36
+ - Regularization Loss: 0.0015
37
+
38
+ ## Model description
39
+
40
+ More information needed
41
+
42
+ ## Intended uses & limitations
43
+
44
+ More information needed
45
+
46
+ ## Training and evaluation data
47
+
48
+ More information needed
49
+
50
+ ## Training procedure
51
+
52
+ ### Training hyperparameters
53
+
54
+ The following hyperparameters were used during training:
55
+ - learning_rate: 0.001
56
+ - train_batch_size: 64
57
+ - eval_batch_size: 64
58
+ - seed: 42
59
+ - optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
60
+ - lr_scheduler_type: linear
61
+ - num_epochs: 1
62
+
63
+ ### Training results
64
+
65
+ | Training Loss | Epoch | Step | Validation Loss | Ce | Valid Targets | Ppl | Mile Loss | Mile Ppl | Top1 Accuracy | Top3 Accuracy | Topk Valid Targets | Ntp Loss | Ntp Ce Unweighted | Mile Reweighting Delta | Mu Loss | Tweo Loss | Nitp Loss | Total Model Loss | Tweo Weighted Loss | Nitp Weighted Loss | Auxiliary Loss | Regularization Loss |
66
+ |:-------------:|:-----:|:-----:|:---------------:|:------:|:-------------:|:-------:|:---------:|:--------:|:-------------:|:-------------:|:------------------:|:--------:|:-----------------:|:----------------------:|:-------:|:---------:|:---------:|:----------------:|:------------------:|:------------------:|:--------------:|:-------------------:|
67
+ | 5.6224 | 0.32 | 5000 | 5.2209 | 4.1900 | 2083162.0 | 66.0231 | 4.7535 | 115.9912 | 0.2844 | 0.4326 | 2083162.0 | 4.7541 | 4.1904 | 0.5634 | 0.0002 | 0.1631 | 0.4651 | 5.2209 | 0.0016 | 0.4651 | 0.4668 | 0.0019 |
68
+ | 5.3605 | 0.64 | 10000 | 4.9374 | 3.9089 | 2083162.0 | 49.8459 | 4.5085 | 90.7852 | 0.3108 | 0.4664 | 2083162.0 | 4.5096 | 3.9094 | 0.5995 | 0.0007 | 0.0910 | 0.4269 | 4.9374 | 0.0009 | 0.4269 | 0.4278 | 0.0016 |
69
+ | 5.2061 | 0.96 | 15000 | 4.7677 | 3.7050 | 2083162.0 | 40.6484 | 4.3282 | 75.8096 | 0.3315 | 0.4909 | 2083162.0 | 4.3296 | 3.7054 | 0.6232 | 0.0011 | 0.0476 | 0.4376 | 4.7677 | 0.0005 | 0.4376 | 0.4381 | 0.0015 |
70
+ | 5.1783 | 1.0 | 15625 | 4.7414 | 3.6766 | 2083162.0 | 39.5103 | 4.3069 | 74.2125 | 0.3343 | 0.4945 | 2083162.0 | 4.3083 | 3.6770 | 0.6303 | 0.0011 | 0.0430 | 0.4327 | 4.7414 | 0.0004 | 0.4327 | 0.4331 | 0.0015 |
71
+
72
+
73
+ ### Framework versions
74
+
75
+ - Transformers 5.20.2
76
+ - Pytorch 2.14.1+cu130
77
+ - Datasets 4.8.5
78
+ - Tokenizers 0.23.3
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:2772de785e901ea3aa12604f4b265f7c98d3a5cfe76cdf070a2546d1fdb78a1b
3
  size 169184776
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ae3ed666164ab86d92a4d286120fd118ff9b8f842371d403d032719fe70e36a7
3
  size 169184776
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:77c48203dcacdcff1397bca04d28eab1aa84c56896ed5a7c745395bca9156db6
3
  size 5329
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:915759cb11d297db4e45e8cc2a62f709eccdd9722429fc532af40cb05aa4b7d8
3
  size 5329