NeoLLM

NeoLLM is a 85.12 M parameter decoder-only language model trained from scratch on FineWeb-Edu with BF16 compute, completing training in approximately ~2h 49m on NVIDIA RTX PRO 6000 Blackwell Server Edition. It integrates a collection of recently published attention and normalization techniques into a single architecture, with the goal of studying how they interact during pretraining. The model is actively being developed and the current checkpoint represents an intermediate training state.

Author / contact: @Kyokopom on X Repository: KitsuVp/NeoLLM

---

Architecture

NeoLLM is a decoder-only transformer with the following configuration:

Parameter	Value
Hidden size	512
Layers	12
Attention heads	8
KV heads (GQA)	4
Head dim	64
Attention layers	full: 12
Intermediate size	1536
Vocabulary	LiquidAI/LFM2.5-1.2B-Thinking tokenizer (64,402 tokens)
Context length	512 tokens
Leviathan	enabled=True; d_seed=128, modes=8, knots=16
JTok / JTok-M	enabled=False / False; Torch continuous surfaces, experts=5, top_k=2
DAGFormer	enabled=False; {'predictor_dim': 256, 'predictor_layers': 2, 'predictor_heads': 4, 'local_dim': 128, 'input_source': 'tokens'}
Linear Depth Memory	enabled=False; groups=8, key dimension=8; shared branch history when combined with SiameseNorm
Virtual Head Attention	enabled=False; query/KV heads follow the configured geometry
Copy-Paste LM	enabled=True; causal vocabulary/pointer mixture

Parameter breakdown

Parameter bucket	Count
Total parameters	85.12M (85,119,928)
Embedding parameters (tied)	32.97M (32,973,824)
Non-embedding parameters	52.15M (52,146,104)
Effective trainable parameters	85.12M (85,119,928)

Weight tying is enabled: the input embedding matrix and the language-model head share the same parameters, so the effective trainable budget is total − embed = 52.15M.

Integrated techniques

NeoLLM combines architecture modules, optional auxiliary objectives, and training-time optimizer/stability components from the following papers.

Embedding and token representation

• Learnable Multipliers (arXiv:2601.04890) — Adds per-row and per-column learnable scalar parameters to selected matrix layers and, when enabled, embeddings.
• Leviathan (arXiv:2601.22040) — Optional continuous token embedding generator that can replace the discrete input lookup table.
• Leviathan-JTok / JTok-M (arXiv:2602.00800) — Optional continuous adaptation of JTok over the shared Leviathan coordinate. JTok applies one normalized surface as a multiplicative MLP gate; JTok-M routes a Sigmoid+TopK mixture of continuous surfaces and adds the paper's depth-scaled residual. Runtime jtok_kernel_backend selects the PyTorch reference or the CCE Triton adapter; the JTok surfaces remain outside MXFP8 conversion.
• KHRONOS (arXiv:2505.13315) — Kernel/basis reference used by the Leviathan continuous token generator implementation.
• Spelling Bee Embeddings (arXiv:2601.18030) — Augments token embeddings with character-level spelling information.
• Token Embedding Manifold analysis (arXiv:2504.01002) — Reference motivation for treating token embeddings as structured objects rather than unconstrained lookup rows.

Attention, positions, and output projection

• Virtual Head Attention (arXiv:2610.17959) — Optional group-specific query premix and low-rank head postmix. NeoLLM shares its gated query projection across corresponding head slots in KV groups and applies postmix after the gate; inference weights are not automatically folded into standard GQA projections.
• Copy-Paste LM (arXiv:2610.19247) — Optional causal pointer and sink mixed with the vocabulary distribution. CUDA training uses CCE's sparse Copy-Paste kernels; inference returns dense mixed logits.
• FAN (arXiv:2502.21309) — Fourier Analysis Networks. A portion of the projection channels are dedicated to periodic cosine/sine features.
• MEA (arXiv:2601.19611) — Explicit Multi-head Attention. Adds small learnable interaction matrices between attention heads for K and V.
• LUCID (arXiv:2602.10410) — Applies a learned lower-triangular preconditioner to V before attention, decorrelating value representations across positions.
• Affine-Scaled Attention (arXiv:2602.23057) — Adds two learnable per-head scalars (α and β) to the softmax weights: [α·softmax(QKᵀ) + β]·V.
• XSA (arXiv:2603.09078) — Exclusive Self Attention. After computing attention, removes the component of the output aligned with the token's own value vector.
• Directional Routing (arXiv:2603.14923) — Each head learns K=4 directions in the output space; a learned router suppresses the attention output along each direction per input.
• Gated Attention (arXiv:2505.06708) — A sigmoid gate is applied to the attention output before the output projection, introducing non-linearity and preventing attention sinks.
• Momentum Attention (arXiv:2411.03884) — Modifies Q and K by subtracting a fraction of the previous position's Q and K values (causal first-difference).
• Interleaved Head Attention / IHA (arXiv:2602.21371) — Builds pseudo-heads from learned cross-head mixtures to create multiple attention patterns per original head.
• REPO (arXiv:2512.14391) — Context re-positioning module that learns contextual position coordinates above a configurable start layer.
• GRAPE (arXiv:2512.07805) — Group representational position encoding used by the REPO-GRAPE positional path.
• GOAT priors (arXiv:2601.15380) — Optional factorized attention log-prior channels inspired by trainable attention priors.
• Hadamard output projection (arXiv:2603.08343) — Replaces dense attention output projection with a structured Hadamard transform plus lightweight scaling.

Normalization, residual flow, and MLP

• SeeDNorm (arXiv:2510.22777) — Applied to Q and K projections. Dynamically rescales normalization from the input's own statistics.
• LayerNorm Scaling / LNS (arXiv:2502.05795) — Each layer's output is scaled by 1/√ℓ where ℓ is the layer index.
• GPAS (arXiv:2506.22049) — Gradient-Preserving Activation Scaling for residual junctions.
• PolyNorm (arXiv:2602.04902) — Replaces the standard MLP activation with normalized linear, quadratic, and cubic branches.
• SimpleGPT (arXiv:2602.01212) — Second-order geometry-inspired normalization strategy applied inside MLP projections.
• StackMemory / STACKTRANS (NeurIPS 2025) — Optional differentiable hidden-state stack between decoder layers.
• Attention Residuals / AttnRes (arXiv:2603.15031) — Optional learned depth-wise aggregation over previous layer outputs or block summaries.
• LAUREL (arXiv:2411.07501) — Optional learned augmented residual layer with residual-weight and low-rank variants.

Training objectives and training-time regularizers

• Cut Cross Entropy (Apple repository) — Memory-efficient next-token loss that avoids materializing the full token-by-vocabulary logits tensor. NeoLLM remains compatible with the upstream package when the extensions below are disabled.
• MiLe Loss (arXiv:2310.19531) — Optional detached, mean-normalized predictive-entropy weighting of token losses, implemented inside the extended CCE path.
• Output Embedding Centering / mu-loss (arXiv:2601.02031) — Optional lambda * ||mean(output_embeddings)||^2 regularizer for output-logit stability.
• MEAP (arXiv:2502.07490) — Optional training-only input corruption that masks a fixed fraction of eligible tokens while preserving clean next-token labels, causal attention, and the inference path.
• TWEO (arXiv:2511.23225) — Optional Transformers Without Extreme Outliers activation regularizer for FP8/low-bit-friendly training.
• NITP (arXiv:2605.24956) — Optional Next Implicit Token Prediction auxiliary objective using shallow-layer implicit token targets and a cosine loss.
• NextLat (arXiv:2511.05963) — Optional next-latent prediction objective using latent dynamics, Smooth L1 supervision, and frozen-head KL.

Optional extended-CCE configuration

Feature	Enabled	Value
MiLe Loss	True	gamma=1.0
mu-loss	True	lambda=0.0001
MEAP	True	ratio=0.15

Copy-Paste uses CCE sparse mixture statistics for CUDA training and computes mu-loss in Torch. MEAP and patch training also use the extended cut-cross-entropy package. Kitsunp/ml-cross-entropy supplies the extended kernels. Copy-Paste exposes three compact scalars: unweighted NTP cross entropy, the MiLe reweighting delta, and the mu-loss penalty. Their sum reconstructs ntp_loss exactly. MEAP reports its eligible and selected counts from the masking kernel; the trainer logs the selected count and exact fraction. Copy-Paste diagnostics use sparse source-token candidates and CCE vocabulary reductions.

Optimizer and training stability

• Adam-family optimizer mode — Active variant: AdEMAMix. AdEMAMix (arXiv:2409.03137) contributes the optional beta3 slow-memory branch when ADEMAMIX=True. DeltaMomentum (arXiv:2608.19491) optionally replaces only the fast beta1 first-moment update on hidden Linear weights when DELTA=True; beta2 remains Adam's ordinary squared-gradient EMA in every mode.
• BF16 stochastic rounding (arXiv:2502.20566) — Optional master-free BF16 model update with stochastic rather than nearest rounding; the same write-back path is shared by AdamW, DeltaAdamW, AdEMAMix, and DeltaAdEMAMix.
• Cautious Weight Decay (arXiv:2510.12402) — Sign-selective weight decay variant used by the custom optimizer logic.
• Correction of Decoupled Weight Decay (arXiv:2512.08217) — Adapts decoupled weight decay during learning-rate decay.
• AdamHD (arXiv:2511.14721) — Decoupled Huber decay regularization reference used by the optimizer.
• Gradient conditioning — CAST active; history-adaptive spectral/norm policies by optimizer group; 46875 committed updates. References: CAST, GradientStabilizer.

---

Training

Setting	Value
Dataset	FineWeb-Edu (sample-10BT)
Tokens seen	~1.54B (46,875 steps × batch 64 × length 512)
Precision	BF16 model state (bfloat16: 85,117,752 elements, float32: 2,176 elements) + AdEMAMix FP32 master active; stochastic rounding disabled
Optimizer	AdEMAMix
Gradient transform	CAST active; history-adaptive spectral/norm policies by optimizer group; 46875 committed updates
Learning rate	1e-03 with 4,000-step linear warmup
Weight decay	coefficients [0.005, 0.01, 0.1]; WD 5/5 groups; CWD 2; Huber 0; Correction 0
Training time	~2h 49m
Hardware	NVIDIA RTX PRO 6000 Blackwell Server Edition

Training curve — total objective

Total includes MiLe when enabled, plus all active auxiliary objectives and regularizers. Without MiLe means model total minus its MiLe delta on the same forward; this is not a separate training run without MiLe.

Step	Train total	Val total	Train without MiLe	Val without MiLe
1	12.1748	—	12.1748	—
250	6.92165	—	6.81098	—
500	6.32294	—	6.11593	—
750	6.21923	—	5.97107	—
1,000	6.14066	—	5.86644	—
1,250	6.04576	—	5.75633	—
1,500	5.98662	—	5.68304	—
1,750	5.92494	—	5.60762	—
2,000	5.87028	—	5.54496	—
2,250	5.82733	—	5.49143	—
2,500	5.78775	—	5.44501	—
2,750	5.75972	—	5.41027	—
3,000	5.73812	—	5.37962	—
3,250	5.71257	—	5.34908	—
3,500	5.68944	—	5.31882	—
3,750	5.66448	—	5.28893	—
4,000	5.64254	—	5.26163	—
4,250	5.62076	—	5.23379	—
4,500	5.60988	—	5.21959	—
4,750	5.59333	—	5.19791	—
5,000	5.56235	5.18006	5.16405	4.67821
5,250	5.55727	—	5.15491	—
5,500	5.53039	—	5.12553	—
5,750	5.51757	—	5.108	—
6,000	5.50644	—	5.09359	—
6,250	5.49288	—	5.07417	—
6,500	5.48111	—	5.06071	—
6,750	5.46532	—	5.04157	—
7,000	5.44897	—	5.02103	—
7,250	5.43519	—	5.00655	—
7,500	5.42655	—	4.99677	—
7,750	5.41079	—	4.97685	—
8,000	5.40852	—	4.97309	—
8,250	5.38859	—	4.95186	—
8,500	5.38756	—	4.95128	—
8,750	5.37799	—	4.93914	—
9,000	5.37269	—	4.93084	—
9,250	5.35795	—	4.91615	—
9,500	5.34956	—	4.90236	—
9,750	5.34352	—	4.89517	—
10,000	5.3378	4.93715	4.88784	4.37472
10,250	5.32866	—	4.87895	—
10,500	5.32194	—	4.87199	—
10,750	5.31524	—	4.86301	—
11,000	5.29933	—	4.84461	—
11,250	5.30029	—	4.84369	—
11,500	5.29244	—	4.83437	—
11,750	5.28046	—	4.81989	—
12,000	5.27868	—	4.81762	—
12,250	5.27133	—	4.80888	—
12,500	5.27329	—	4.81183	—
12,750	5.25825	—	4.79397	—
13,000	5.25528	—	4.79151	—
13,250	5.24511	—	4.77776	—
13,500	5.24528	—	4.77692	—
13,750	5.23372	—	4.76672	—
14,000	5.22877	—	4.75996	—
14,250	5.22141	—	4.75205	—
14,500	5.21629	—	4.74683	—
14,750	5.21326	—	4.74084	—
15,000	5.20913	4.80225	4.73543	4.21431
15,250	5.2079	—	4.73471	—
15,500	5.19814	—	4.72473	—
15,750	5.18694	—	4.71318	—
16,000	5.18341	—	4.70899	—
16,250	5.18441	—	4.70854	—
16,500	5.17877	—	4.70221	—
16,750	5.17395	—	4.6971	—
17,000	5.17194	—	4.69328	—
17,250	5.16147	—	4.68416	—
17,500	5.15295	—	4.67441	—
17,750	5.15119	—	4.67035	—
18,000	5.15002	—	4.67267	—
18,250	5.14412	—	4.66156	—
18,500	5.14331	—	4.66264	—
18,750	5.13758	—	4.656	—
19,000	5.13016	—	4.6492	—
19,250	5.12905	—	4.64794	—
19,500	5.11831	—	4.63464	—
19,750	5.11731	—	4.63415	—
20,000	5.1124	4.69611	4.62951	4.108
20,250	5.10362	—	4.61909	—
20,500	5.10722	—	4.62153	—
20,750	5.09664	—	4.61203	—
21,000	5.09292	—	4.60665	—
21,250	5.09264	—	4.60657	—
21,500	5.09497	—	4.60913	—
21,750	5.07891	—	4.59159	—
22,000	5.07444	—	4.58506	—
22,250	5.06903	—	4.57962	—
22,500	5.06821	—	4.57875	—
22,750	5.06352	—	4.57497	—
23,000	5.06061	—	4.57036	—
23,250	5.05831	—	4.56748	—
23,500	5.05931	—	4.57006	—
23,750	5.04956	—	4.55906	—
24,000	5.04671	—	4.55624	—
24,250	5.04553	—	4.55364	—
24,500	5.03985	—	4.54844	—
24,750	5.03252	—	4.54105	—
25,000	5.03085	4.61978	4.53903	4.02042
25,250	5.02313	—	4.53086	—
25,500	5.02147	—	4.52864	—
25,750	5.01881	—	4.52752	—
26,000	5.0126	—	4.51905	—
26,250	5.01031	—	4.51787	—
26,500	5.00996	—	4.51547	—
26,750	4.99715	—	4.50377	—
27,000	4.99828	—	4.50468	—
27,250	4.99436	—	4.50039	—
27,500	4.99145	—	4.49675	—
27,750	4.98448	—	4.4886	—
28,000	4.97935	—	4.48331	—
28,250	4.97127	—	4.47389	—
28,500	4.97519	—	4.47814	—
28,750	4.96375	—	4.467	—
29,000	4.96242	—	4.46689	—
29,250	4.9587	—	4.46045	—
29,500	4.96057	—	4.46401	—
29,750	4.94622	—	4.4505	—
30,000	4.94963	4.54867	4.45152	3.9396
30,250	4.94143	—	4.44351	—
30,500	4.93341	—	4.4356	—
30,750	4.93536	—	4.43585	—
31,000	4.92798	—	4.4276	—
31,250	4.92189	—	4.42033	—
31,500	4.92311	—	4.42358	—
31,750	4.92186	—	4.4203	—
32,000	4.91127	—	4.41364	—
32,250	4.90887	—	4.40825	—
32,500	4.90549	—	4.40395	—
32,750	4.9039	—	4.40486	—
33,000	4.90276	—	4.40047	—
33,250	4.89339	—	4.39245	—
33,500	4.89649	—	4.39383	—
33,750	4.88808	—	4.38562	—
34,000	4.89087	—	4.38904	—
34,250	4.88617	—	4.38436	—
34,500	4.88154	—	4.38045	—
34,750	4.87624	—	4.37448	—
35,000	4.88484	4.4774	4.38245	3.85776
35,250	4.87747	—	4.37436	—
35,500	4.87091	—	4.3679	—
35,750	4.87288	—	4.37032	—
36,000	4.87139	—	4.36759	—
36,250	4.86037	—	4.35495	—
36,500	4.85734	—	4.35129	—
36,750	4.84907	—	4.34383	—
37,000	4.84381	—	4.3362	—
37,250	4.843	—	4.33716	—
37,500	4.83729	—	4.33009	—
37,750	4.83421	—	4.32631	—
38,000	4.82708	—	4.31862	—
38,250	4.82256	—	4.31407	—
38,500	4.82414	—	4.31338	—
38,750	4.81589	—	4.30858	—
39,000	4.81701	—	4.30461	—
39,250	4.81008	—	4.29886	—
39,500	4.80356	—	4.29269	—
39,750	4.79813	—	4.28588	—
40,000	4.78614	4.38607	4.27231	3.7617
40,250	4.78666	—	4.27274	—
40,500	4.78319	—	4.26749	—
40,750	4.77981	—	4.26525	—
41,000	4.77403	—	4.25862	—
41,250	4.77428	—	4.25976	—
41,500	4.76961	—	4.25327	—
41,750	4.75899	—	4.24413	—
42,000	4.7604	—	4.24324	—
42,250	4.75839	—	4.24059	—
42,500	4.74628	—	4.22826	—
42,750	4.74639	—	4.22882	—
43,000	4.7384	—	4.21734	—
43,250	4.73738	—	4.21956	—
43,500	4.73196	—	4.21293	—
43,750	4.72463	—	4.20567	—
44,000	4.717	—	4.19614	—
44,250	4.7177	—	4.19707	—
44,500	4.70877	—	4.18702	—
44,750	4.70304	—	4.18168	—
45,000	4.69766	4.29196	4.1748	3.65916
45,250	4.69312	—	4.17014	—
45,500	4.69122	—	4.16901	—
45,750	4.68483	—	4.16103	—
46,000	4.67806	—	4.15363	—
46,250	4.67461	—	4.14886	—
46,500	4.66757	—	4.1401	—
46,750	4.66395	—	4.13691	—
46,875	4.65736	4.25607	4.13202	3.61865

Cross-entropy with and without MiLe

CE without MiLe is unweighted CE; CE with MiLe adds the reweighting delta. Both exclude mu and auxiliary terms. ntp_loss includes mu when enabled.

Step	Train CE without MiLe	Val CE without MiLe	Train CE with MiLe	Val CE with MiLe
1	11.1209	—	11.1209	—
250	6.68166	—	6.79234	—
500	5.91843	—	6.12544	—
750	5.64135	—	5.8895	—
1,000	5.44158	—	5.7158	—
1,250	5.28659	—	5.57602	—
1,500	5.17681	—	5.48038	—
1,750	5.0816	—	5.39892	—
2,000	5.00609	—	5.33141	—
2,250	4.9421	—	5.278	—
2,500	4.89242	—	5.23516	—
2,750	4.85422	—	5.20367	—
3,000	4.81923	—	5.17773	—
3,250	4.78535	—	5.14885	—
3,500	4.75387	—	5.12448	—
3,750	4.72285	—	5.09841	—
4,000	4.69571	—	5.07662	—
4,250	4.66743	—	5.0544	—
4,500	4.65283	—	5.04311	—
4,750	4.62895	—	5.02437	—
5,000	4.59606	4.11421	4.99436	4.61606
5,250	4.58657	—	4.98893	—
5,500	4.55916	—	4.96401	—
5,750	4.5413	—	4.95087	—
6,000	4.52654	—	4.9394	—
6,250	4.50907	—	4.92778	—
6,500	4.49567	—	4.91608	—
6,750	4.47848	—	4.90223	—
7,000	4.45926	—	4.88719	—
7,250	4.44515	—	4.87379	—
7,500	4.43716	—	4.86694	—
7,750	4.41837	—	4.85231	—
8,000	4.41574	—	4.85117	—
8,250	4.39703	—	4.83376	—
8,500	4.39681	—	4.83308	—
8,750	4.3858	—	4.82465	—
9,000	4.37843	—	4.82029	—
9,250	4.36655	—	4.80835	—
9,500	4.35402	—	4.80123	—
9,750	4.34951	—	4.79786	—
10,000	4.34268	3.84337	4.79264	4.4058
10,250	4.33578	—	4.78549	—
10,500	4.33049	—	4.78045	—
10,750	4.32335	—	4.77559	—
11,000	4.30677	—	4.76149	—
11,250	4.30557	—	4.76218	—
11,500	4.29956	—	4.75764	—
11,750	4.28661	—	4.74718	—
12,000	4.2862	—	4.74727	—
12,250	4.27867	—	4.74112	—
12,500	4.28374	—	4.7452	—
12,750	4.26875	—	4.73303	—
13,000	4.26723	—	4.731	—
13,250	4.25558	—	4.72293	—
13,500	4.25732	—	4.72568	—
13,750	4.24932	—	4.71632	—
14,000	4.2448	—	4.71361	—
14,250	4.23818	—	4.70754	—
14,500	4.23542	—	4.70488	—
14,750	4.23267	—	4.7051	—
15,000	4.22814	3.72119	4.70183	4.30914
15,250	4.22966	—	4.70285	—
15,500	4.22158	—	4.69499	—
15,750	4.2129	—	4.68667	—
16,000	4.21032	—	4.68474	—
16,250	4.21184	—	4.68771	—
16,500	4.20815	—	4.68471	—
16,750	4.2049	—	4.68175	—
17,000	4.20241	—	4.68108	—
17,250	4.19556	—	4.67288	—
17,500	4.18764	—	4.66617	—
17,750	4.18599	—	4.66684	—
18,000	4.18907	—	4.66643	—
18,250	4.18084	—	4.6634	—
18,500	4.18375	—	4.66442	—
18,750	4.1786	—	4.66017	—
19,000	4.17479	—	4.65575	—
19,250	4.17573	—	4.65683	—
19,500	4.16364	—	4.64731	—
19,750	4.16665	—	4.64982	—
20,000	4.16271	3.64836	4.6456	4.23647
20,250	4.15374	—	4.63827	—
20,500	4.1594	—	4.6451	—
20,750	4.15193	—	4.63654	—
21,000	4.14849	—	4.63476	—
21,250	4.15075	—	4.63682	—
21,500	4.15494	—	4.64078	—
21,750	4.14018	—	4.6275	—
22,000	4.13595	—	4.62533	—
22,250	4.13187	—	4.62127	—
22,500	4.13241	—	4.62186	—
22,750	4.13233	—	4.62088	—
23,000	4.12988	—	4.62013	—
23,250	4.13011	—	4.62094	—
23,500	4.13553	—	4.62478	—
23,750	4.12705	—	4.61755	—
24,000	4.12571	—	4.61618	—
24,250	4.12673	—	4.61862	—
24,500	4.12049	—	4.6119	—
24,750	4.11613	—	4.6076	—
25,000	4.11793	3.60305	4.60974	4.20241
25,250	4.11258	—	4.60484	—
25,500	4.11351	—	4.60634	—
25,750	4.11371	—	4.60499	—
26,000	4.10946	—	4.60301	—
26,250	4.1122	—	4.60463	—
26,500	4.11159	—	4.60608	—
26,750	4.10172	—	4.5951	—
27,000	4.10486	—	4.59846	—
27,250	4.10295	—	4.59692	—
27,500	4.1012	—	4.59589	—
27,750	4.09568	—	4.59156	—
28,000	4.09635	—	4.59239	—
28,250	4.089	—	4.58638	—
28,500	4.09704	—	4.59408	—
28,750	4.08803	—	4.58479	—
29,000	4.08841	—	4.58394	—
29,250	4.08607	—	4.58432	—
29,500	4.09276	—	4.58932	—
29,750	4.08003	—	4.57574	—
30,000	4.08529	3.56742	4.58341	4.17649
30,250	4.08151	—	4.57944	—
30,500	4.07762	—	4.57543	—
30,750	4.0789	—	4.57841	—
31,000	4.07575	—	4.57612	—
31,250	4.0704	—	4.57197	—
31,500	4.07562	—	4.57515	—
31,750	4.07492	—	4.57648	—
32,000	4.0718	—	4.56943	—
32,250	4.06868	—	4.5693	—
32,500	4.06707	—	4.56861	—
32,750	4.07066	—	4.5697	—
33,000	4.06749	—	4.56978	—
33,250	4.06238	—	4.56332	—
33,500	4.06386	—	4.56653	—
33,750	4.05852	—	4.56097	—
34,000	4.06265	—	4.56448	—
34,250	4.05955	—	4.56136	—
34,500	4.05778	—	4.55888	—
34,750	4.05378	—	4.55553	—
35,000	4.06413	3.53975	4.56651	4.1594
35,250	4.05663	—	4.55974	—
35,500	4.05082	—	4.55383	—
35,750	4.05337	—	4.55594	—
36,000	4.05111	—	4.55491	—
36,250	4.04087	—	4.54629	—
36,500	4.03854	—	4.54458	—
36,750	4.03321	—	4.53845	—
37,000	4.0277	—	4.53531	—
37,250	4.03076	—	4.53661	—
37,500	4.02439	—	4.53159	—
37,750	4.02122	—	4.52912	—
38,000	4.01499	—	4.52344	—
38,250	4.01226	—	4.52076	—
38,500	4.01263	—	4.52339	—
38,750	4.00948	—	4.51679	—
39,000	4.00662	—	4.51902	—
39,250	4.00297	—	4.51419	—
39,500	3.99841	—	4.50928	—
39,750	3.99318	—	4.50543	—
40,000	3.9803	3.47171	4.49413	4.09608
40,250	3.98287	—	4.49679	—
40,500	3.98019	—	4.49589	—
40,750	3.97813	—	4.49268	—
41,000	3.97241	—	4.48782	—
41,250	3.97591	—	4.49042	—
41,500	3.97109	—	4.48743	—
41,750	3.9634	—	4.47826	—
42,000	3.9625	—	4.47965	—
42,250	3.96156	—	4.47936	—
42,500	3.95039	—	4.46841	—
42,750	3.95262	—	4.47019	—
43,000	3.9438	—	4.46485	—
43,250	3.94679	—	4.46461	—
43,500	3.94101	—	4.46005	—
43,750	3.93723	—	4.45619	—
44,000	3.92947	—	4.45033	—
44,250	3.93056	—	4.45119	—
44,500	3.92221	—	4.44396	—
44,750	3.9196	—	4.44097	—
45,000	3.91276	3.39729	4.43562	4.03009
45,250	3.90903	—	4.43202	—
45,500	3.90982	—	4.43203	—
45,750	3.9033	—	4.42711	—
46,000	3.89807	—	4.42249	—
46,250	3.89554	—	4.4213	—
46,500	3.88866	—	4.41612	—
46,750	3.88574	—	4.41278	—
46,875	3.88235	3.3691	4.4077	4.00652

Observed loss components — weighted contributions

Only contributions actually added to the objective are shown; raw values and subcomponents remain in training metrics. regularization_loss is the subtotal of mu, weighted TWeo, JTok-M and StackMemory entropy, excluding NITP/NextLat; do not add it again to those components. JTok-M auxiliary loss is training-only.

Step	Phase	mu	TWeo	NITP	Reg. subtotal
1	train	3.15217e-10	0.0784945	0.975458	0.0784945
250	train	4.89306e-06	0.0367801	0.092529	0.036785
500	train	1.1299e-05	0.0351412	0.162354	0.0351525
750	train	1.78538e-05	0.0413503	0.288359	0.0413682
1,000	train	2.56883e-05	0.0447554	0.380075	0.0447811
1,250	train	3.34338e-05	0.0464962	0.423209	0.0465297
1,500	train	4.05492e-05	0.0486952	0.457501	0.0487357
1,750	train	4.65059e-05	0.050363	0.475609	0.0504096
2,000	train	5.16706e-05	0.0512239	0.487588	0.0512755
2,250	train	5.62379e-05	0.052394	0.496878	0.0524503
2,500	train	5.99946e-05	0.0537986	0.498733	0.0538586
2,750	train	6.40936e-05	0.0536435	0.502342	0.0537076
3,000	train	6.8614e-05	0.0540551	0.506261	0.0541237
3,250	train	7.33329e-05	0.0549474	0.508702	0.0550208
3,500	train	7.86129e-05	0.0551793	0.509698	0.0552579
3,750	train	8.39059e-05	0.0559524	0.510037	0.0560363
4,000	train	8.94382e-05	0.0559785	0.509853	0.0560679
4,250	train	9.59615e-05	0.0560095	0.510252	0.0561055
4,500	train	0.00010285	0.0561268	0.510535	0.0562297
4,750	train	0.000109985	0.0563006	0.512547	0.0564106
5,000	train	0.00011709	0.0562102	0.511659	0.0563273
5,000	validation	0.000120763	0.0577314	0.506148	0.0578522
5,250	train	0.000124578	0.0564829	0.511733	0.0566075
5,500	train	0.000132405	0.0563175	0.509922	0.0564498
5,750	train	0.000139221	0.0565058	0.510052	0.056645
6,000	train	0.00014632	0.056213	0.510687	0.0563593
6,250	train	0.000154618	0.0560569	0.50889	0.0562115
6,500	train	0.000163404	0.0560464	0.508823	0.0562098
6,750	train	0.000171615	0.0559682	0.506956	0.0561398
7,000	train	0.000179744	0.0560061	0.505587	0.0561858
7,250	train	0.000187254	0.0562124	0.505	0.0563996
7,500	train	0.000194548	0.056055	0.503357	0.0562495
7,750	train	0.000200929	0.0559779	0.502299	0.0561789
8,000	train	0.000207682	0.05583	0.501309	0.0560377
8,250	train	0.000214204	0.0558754	0.498744	0.0560896
8,500	train	0.0002211	0.055783	0.498469	0.0560041
8,750	train	0.000227951	0.0556092	0.497504	0.0558372
9,000	train	0.000235428	0.0555747	0.496596	0.0558102
9,250	train	0.000242827	0.055523	0.493831	0.0557658
9,500	train	0.000251166	0.0555536	0.492529	0.0558047
9,750	train	0.000260123	0.0553886	0.49001	0.0556488
10,000	train	0.000268206	0.0554065	0.489488	0.0556747
10,000	validation	0.000272815	0.0560081	0.475072	0.0562809
10,250	train	0.000276362	0.055381	0.487516	0.0556574
10,500	train	0.000283774	0.0554759	0.485734	0.0557597
10,750	train	0.000290275	0.0552489	0.484115	0.0555392
11,000	train	0.000295432	0.055356	0.482186	0.0556514
11,250	train	0.000298808	0.0553168	0.4825	0.0556156
11,500	train	0.000302261	0.0553368	0.479166	0.055639
11,750	train	0.000305535	0.0552782	0.477699	0.0555837
12,000	train	0.000310376	0.0551929	0.475917	0.0555033
12,250	train	0.000317907	0.0552423	0.474643	0.0555602
12,500	train	0.000324583	0.0550624	0.472704	0.055387
12,750	train	0.000331254	0.0550377	0.469855	0.0553689
13,000	train	0.000338064	0.0549663	0.468974	0.0553043
13,250	train	0.000342587	0.0549168	0.466925	0.0552594
13,500	train	0.00034635	0.0548487	0.464406	0.055195
13,750	train	0.000350135	0.0547272	0.46232	0.0550774
14,000	train	0.00035395	0.0546607	0.460144	0.0550147
14,250	train	0.000356836	0.0546059	0.458901	0.0549627
14,500	train	0.000358567	0.0545838	0.456464	0.0549424
14,750	train	0.000363857	0.0545896	0.453209	0.0549535
15,000	train	0.000370105	0.0545117	0.452412	0.0548819
15,000	validation	0.000372586	0.0555699	0.437169	0.0559424
15,250	train	0.000375884	0.0543628	0.450308	0.0547386
15,500	train	0.000380394	0.0543172	0.448449	0.0546976
15,750	train	0.000384524	0.0544225	0.445469	0.054807
16,000	train	0.000386574	0.0541719	0.444104	0.0545585
16,250	train	0.000389381	0.0541986	0.442104	0.054588
16,500	train	0.000393275	0.0541057	0.439561	0.0544989
16,750	train	0.000395885	0.0540571	0.437748	0.054453
17,000	train	0.000399795	0.053881	0.436587	0.0542808
17,250	train	0.000403571	0.0540231	0.434165	0.0544266
17,500	train	0.000408138	0.0539277	0.432434	0.0543358
17,750	train	0.00041271	0.0538281	0.430115	0.0542408
18,000	train	0.000415497	0.0538348	0.429343	0.0542503
18,250	train	0.00041905	0.0538216	0.42648	0.0542407
18,500	train	0.00042564	0.0536493	0.424812	0.0540749
18,750	train	0.000433928	0.0535715	0.423396	0.0540054
19,000	train	0.000440317	0.0534228	0.420553	0.0538631
19,250	train	0.000444289	0.0534491	0.418319	0.0538934
19,500	train	0.000446276	0.0533493	0.417203	0.0537955
19,750	train	0.000447897	0.0532017	0.413841	0.0536496
20,000	train	0.000449893	0.0532471	0.41311	0.053697
20,000	validation	0.000453474	0.0542767	0.404917	0.0547302
20,250	train	0.000457152	0.0531759	0.411716	0.0536331
20,500	train	0.000463997	0.0531211	0.408538	0.0535851
20,750	train	0.000470717	0.052852	0.406777	0.0533227
21,000	train	0.000478824	0.0530122	0.404672	0.053491
21,250	train	0.000482851	0.0528679	0.40247	0.0533508
21,500	train	0.000485331	0.0528384	0.400871	0.0533238
21,750	train	0.00048955	0.0527953	0.398125	0.0532849
22,000	train	0.000493584	0.0527287	0.395885	0.0532223
22,250	train	0.000499277	0.0525797	0.394678	0.053079
22,500	train	0.000504938	0.0525759	0.393265	0.0530808
22,750	train	0.000510454	0.0523806	0.389743	0.052891
23,000	train	0.000512901	0.0523238	0.387646	0.0528367
23,250	train	0.000514408	0.0523405	0.384514	0.0528549
23,500	train	0.000517983	0.0522414	0.381771	0.0527593
23,750	train	0.000525708	0.0521769	0.379301	0.0527026
24,000	train	0.000535593	0.0520801	0.377914	0.0526157
24,250	train	0.000540242	0.0519962	0.374379	0.0525364
24,500	train	0.000545735	0.0520182	0.375394	0.0525639
24,750	train	0.000550947	0.0519893	0.372378	0.0525402
25,000	train	0.000557698	0.0518704	0.368674	0.0524281
25,000	validation	0.000561297	0.0526579	0.36415	0.0532192
25,250	train	0.000565064	0.0518177	0.3659	0.0523828
25,500	train	0.00057023	0.0517499	0.36281	0.0523202
25,750	train	0.000572807	0.0517255	0.361519	0.0522983
26,000	train	0.000573964	0.0515863	0.357434	0.0521603
26,250	train	0.000575875	0.0515236	0.353571	0.0520995
26,500	train	0.000579646	0.0514027	0.351897	0.0519823
26,750	train	0.000582985	0.0513482	0.350118	0.0519312
27,000	train	0.000582803	0.0513193	0.347919	0.0519022
27,250	train	0.000585465	0.0512601	0.345597	0.0518455
27,500	train	0.000591055	0.0512556	0.34371	0.0518467
27,750	train	0.000597539	0.0512002	0.341124	0.0517978
28,000	train	0.000603112	0.0510079	0.335354	0.0516109
28,250	train	0.000608173	0.0509093	0.333376	0.0515175
28,500	train	0.00061256	0.0508043	0.329693	0.0514169
28,750	train	0.000621452	0.0506423	0.327698	0.0512637
29,000	train	0.000627849	0.0505313	0.327321	0.0511592
29,250	train	0.000635529	0.0504001	0.323349	0.0510357
29,500	train	0.000640895	0.0501644	0.320441	0.0508053
29,750	train	0.000647324	0.0496523	0.320171	0.0502996
30,000	train	0.000654983	0.0482324	0.317337	0.0488874
30,000	validation	0.000661524	0.0480557	0.323467	0.0487173
30,250	train	0.000668144	0.046527	0.314802	0.0471951
30,500	train	0.000680071	0.0455264	0.311776	0.0462065
30,750	train	0.000694352	0.0451064	0.311143	0.0458007
31,000	train	0.000703548	0.0448821	0.306267	0.0455856
31,250	train	0.000707955	0.0446276	0.304585	0.0453356
31,500	train	0.000712285	0.044576	0.30267	0.0452882
31,750	train	0.00071916	0.0443457	0.300316	0.0450649
32,000	train	0.000725364	0.0440989	0.297012	0.0448242
32,250	train	0.000727025	0.0437106	0.29513	0.0444376
32,500	train	0.000730082	0.0426524	0.293496	0.0433824
32,750	train	0.000731875	0.0410812	0.292386	0.041813
33,000	train	0.00073376	0.0398083	0.292431	0.040542
33,250	train	0.000740471	0.0385905	0.29074	0.0393309
33,500	train	0.000749824	0.0373204	0.291891	0.0380702
33,750	train	0.0007613	0.0362162	0.290124	0.0369775
34,000	train	0.000770125	0.0355204	0.290103	0.0362905
34,250	train	0.000776597	0.0349369	0.289098	0.0357135
34,500	train	0.000782433	0.0343796	0.287505	0.035162
34,750	train	0.000784849	0.0337041	0.286214	0.0344889
35,000	train	0.000788423	0.0326584	0.284881	0.0334468
35,000	validation	0.000790073	0.0297227	0.287491	0.0305128
35,250	train	0.000792289	0.0318029	0.285134	0.0325952
35,500	train	0.000794765	0.0309306	0.285354	0.0317253
35,750	train	0.000800576	0.0303598	0.285782	0.0311604
36,000	train	0.000807598	0.0298437	0.285825	0.0306513
36,250	train	0.000814848	0.0292433	0.284023	0.0300582
36,500	train	0.000824677	0.0287538	0.283176	0.0295785
36,750	train	0.00083351	0.0281535	0.281635	0.028987
37,000	train	0.000841616	0.0274942	0.280167	0.0283358
37,250	train	0.000846027	0.0268056	0.278741	0.0276516
37,500	train	0.000850913	0.0261436	0.278702	0.0269945
37,750	train	0.00085335	0.025366	0.278864	0.0262193
38,000	train	0.000857264	0.0246922	0.278085	0.0255494
38,250	train	0.000864047	0.0241419	0.276799	0.0250059
38,500	train	0.000868205	0.0237032	0.276171	0.0245714
38,750	train	0.000869179	0.0232759	0.274954	0.0241451
39,000	train	0.000870413	0.0229331	0.274187	0.0238035
39,250	train	0.000872809	0.0225845	0.272432	0.0234573
39,500	train	0.000873916	0.0222541	0.271152	0.023128
39,750	train	0.000871009	0.0219353	0.269893	0.0228063
40,000	train	0.000871103	0.021678	0.269463	0.0225491
40,000	validation	0.000871672	0.0189539	0.270162	0.0198255
40,250	train	0.000873991	0.0213993	0.267596	0.0222733
40,500	train	0.00088176	0.0211078	0.265306	0.0219895
40,750	train	0.000888181	0.0207945	0.26544	0.0216826
41,000	train	0.000893933	0.0203912	0.264921	0.0212851
41,250	train	0.00089941	0.0198957	0.263059	0.0207951
41,500	train	0.00090407	0.0194754	0.261802	0.0203795
41,750	train	0.000908791	0.0190451	0.260774	0.0199538
42,000	train	0.000911179	0.0187175	0.261119	0.0196286
42,250	train	0.000911412	0.0183819	0.259737	0.0192933
42,500	train	0.000909966	0.0180941	0.258862	0.019004
42,750	train	0.000908324	0.0178047	0.257487	0.0187131
43,000	train	0.000907186	0.0175393	0.255095	0.0184465
43,250	train	0.000907195	0.0172605	0.254607	0.0181677
43,500	train	0.00090745	0.0170162	0.253995	0.0179237
43,750	train	0.000907476	0.0166959	0.250837	0.0176033
44,000	train	0.000907536	0.0164227	0.249345	0.0173303
44,250	train	0.000907688	0.0161352	0.249468	0.0170429
44,500	train	0.000907196	0.0158447	0.248062	0.0167518
44,750	train	0.000907071	0.015484	0.245686	0.0163911
45,000	train	0.000908256	0.0152576	0.245879	0.0161659
45,000	validation	0.000908406	0.012369	0.248596	0.0132774
45,250	train	0.000909574	0.0150328	0.245163	0.0159424
45,500	train	0.000910989	0.0148268	0.243449	0.0157378
45,750	train	0.000912062	0.0146301	0.242186	0.0155422
46,000	train	0.000912634	0.0144368	0.240214	0.0153494
46,250	train	0.000913242	0.0142638	0.238137	0.015177
46,500	train	0.000912901	0.0141332	0.236401	0.0150461
46,750	train	0.000912036	0.0139994	0.236255	0.0149114
46,875	train	0.000910712	0.0138564	0.2349	0.0147671
46,875	validation	0.000910327	0.0112369	0.23741	0.0121472

Tables use exact logged steps; missing values are — without carrying values forward. Repeated evaluations at one step use the latest logged value. Components average batch scalars weighted by examples; CE/PPL token metrics retain their valid-target weighting. Weight decay/CWD act in the optimizer, outside these forward losses.

---

Limitations

• Token budget — ~1.5 B tokens seen; below estimated optimum. Knowledge-intensive tasks will improve with more training.
• Gradient spike at step 40k — Reorganized the attention pattern in layer 9 that previously captured long-range token correlations. A checkpoint from ~step 38k is expected to have better aggregate benchmark scores.
• PolyNorm exclusivity — The quadratic branch has become partially redundant with the linear branch. Will be corrected in the next training run.
• Base model only — Not instruction-tuned or aligned; purely a next-token-prediction base model.

---

References

All papers whose techniques are integrated into NeoLLM's architecture, training objective, or training stack:

Area	Technique	Paper title	Reference
Embeddings	Learnable Multipliers	Freeing the Scale of Language Model Matrix Layers	arXiv:2601.04890
Embeddings	Leviathan	A Separable Architecture for Continuous Token Representation in Language Models	arXiv:2601.22040
Embeddings / residual modulation	JTok / JTok-M	JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation	arXiv:2602.00800
Embeddings	KHRONOS	KHRONOS: a Kernel-Based Neural Architecture for Rapid, Resource-Efficient Scientific Computation	arXiv:2505.13315
Embeddings	Spelling Bee	Spelling Bee Embeddings for Language Modeling	arXiv:2601.18030
Embeddings	Token embedding analysis	Token Embeddings Violate the Manifold Hypothesis	arXiv:2504.01002
Attention / positions	FAN	Fourier Analysis Networks	arXiv:2502.21309
Attention / positions	MEA	Explicit Multi-head Attention for Inter-head Interaction in Large Language Models	arXiv:2601.19611
Attention / positions	LUCID	Attention with Preconditioned Representations	arXiv:2602.10410
Attention / positions	Affine-Scaled Attention	Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention	arXiv:2602.23057
Attention / positions	XSA	Exclusive Self Attention	arXiv:2603.09078
Attention / positions	Directional Routing	Directional Routing in Transformers	arXiv:2603.14923
Attention / positions	Gated Attention	Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free	arXiv:2505.06708
Attention / positions	Momentum Attention	Momentum Attention	arXiv:2411.03884
Attention / positions	IHA	Interleaved Head Attention	arXiv:2602.21371
Attention / positions	REPO	Language Models with Context Re-Positioning	arXiv:2512.14391
Attention / positions	GRAPE	Group Representational Position Encoding	arXiv:2512.07805
Attention / positions	GOAT priors	You Need Better Attention Priors	arXiv:2601.15380
Attention / positions	Hadamard o_proj	Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers	arXiv:2603.08343
Residual / normalization	SeeDNorm	Self-Rescaled Dynamic Normalization	arXiv:2510.22777
Residual / normalization	LNS	The Curse of Depth in LLMs	arXiv:2502.05795
Residual / normalization	GPAS	Gradient-Preserving Activation Scaling	arXiv:2506.22049
Residual / normalization	PolyNorm	PolyNorm / PolyCom	arXiv:2602.04902
Residual / normalization	SimpleGPT	SimpleGPT	arXiv:2602.01212
Residual / normalization	StackMemory / STACKTRANS	Recursive Transformer: Boosting Reasoning Ability with State Stack	NeurIPS 2025
Residual / normalization	Attention Residuals	Attention Residuals	arXiv:2603.15031
Residual / normalization	Linear Depth Memory	Depth Memory	arXiv:2610.18809
Residual / normalization	LAUREL	LAUREL: Learned Augmented Residual Layer	arXiv:2411.07501
Objectives	TWEO	Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies	arXiv:2511.23225
Objectives	NITP	Next Implicit Token Prediction for LLM Pre-training	arXiv:2605.24956
Objectives	NextLat	Next-Latent Prediction Transformers Learn Compact World Models	arXiv:2511.05963
Optimizer / training	AdEMAMix	The AdEMAMix Optimizer: Better, Faster, Older	arXiv:2409.03137
Optimizer / training	DeltaMomentum	DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule	arXiv:2608.19491
Optimizer / training	BF16 + SR	Stochastic Rounding for LLM Training: Theory and Practice	arXiv:2502.20566
Optimizer / training	CWD	Cautious Weight Decay	arXiv:2510.12402
Optimizer / training	WD correction	Correction of Decoupled Weight Decay	arXiv:2512.08217
Optimizer / training	AdamHD	AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training	arXiv:2511.14721
Optimizer / training	GradientStabilizer	GradientStabilizer	arXiv:2502.17055

---

Citation

@misc{neollm2026,
  title  = {NeoLLM: A Research Language Model Integrating Recent Attention and Normalization Techniques},
  author = {KitsuVp},
  year   = {2026},
  url    = {https://huggingface.co/KitsuVp/NeoLLM}
}

---

Author

@Kyokopom on X

---

License

Apache 2.0