Instructions to use KitsuVp/NeoLLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use KitsuVp/NeoLLM with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="KitsuVp/NeoLLM", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("KitsuVp/NeoLLM", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use KitsuVp/NeoLLM with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "KitsuVp/NeoLLM"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KitsuVp/NeoLLM",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/KitsuVp/NeoLLM

SGLang

How to use KitsuVp/NeoLLM with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "KitsuVp/NeoLLM" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KitsuVp/NeoLLM",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "KitsuVp/NeoLLM" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KitsuVp/NeoLLM",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use KitsuVp/NeoLLM with Docker Model Runner:
```
docker model run hf.co/KitsuVp/NeoLLM
```

NeoLLM

NeoLLM is a 85.16 M parameter decoder-only language model trained from scratch on FineWeb-Edu with BF16 compute, completing training in approximately ~2h 03m on NVIDIA RTX PRO 6000 Blackwell Server Edition. It integrates a collection of recently published attention and normalization techniques into a single architecture, with the goal of studying how they interact during pretraining. The model is actively being developed and the current checkpoint represents an intermediate training state.

Author / contact: @Kyokopom on X Repository: KitsuVp/NeoLLM

Architecture

NeoLLM is a decoder-only transformer with the following configuration:

Parameter	Value
Hidden size	512
Layers	12
Attention heads	8
KV heads (GQA)	4
Head dim	64
Attention layers	full: 4, ckda: 8
GOAT position prior	enabled=False; replaces RoPE in full-attention layers
Intermediate size	1536
Vocabulary	LiquidAI/LFM2.5-1.2B-Thinking tokenizer (64,402 tokens)
Context length	512 tokens
Leviathan	enabled=True; d_seed=128, modes=8, knots=16
JTok / JTok-M	enabled=False / False; Torch continuous surfaces, experts=5, top_k=2
DAGFormer	enabled=False; {'predictor_dim': 256, 'predictor_layers': 2, 'predictor_heads': 4, 'local_dim': 128, 'input_source': 'tokens'}
Linear Depth Memory	enabled=False; groups=8, key dimension=8; shared branch history when combined with SiameseNorm
Virtual Head Attention	enabled=False; query/KV heads follow the configured geometry
Copy-Paste LM	enabled=True; causal vocabulary/pointer mixture

Parameter breakdown

Parameter bucket	Count
Total parameters	85.16M (85,156,584)
Embedding parameters (tied)	32.97M (32,973,824)
Non-embedding parameters	52.18M (52,182,760)
Effective trainable parameters	85.16M (85,156,584)

Weight tying is enabled: the input embedding matrix and the language-model head share the same parameters, so the effective trainable budget is total − embed = 52.18M.

Integrated techniques

NeoLLM combines architecture modules, optional auxiliary objectives, and training-time optimizer/stability components from the following papers.

Embedding and token representation

Learnable Multipliers (arXiv:2601.04890) — Adds per-row and per-column learnable scalar parameters to selected matrix layers and, when enabled, embeddings.
Leviathan (arXiv:2601.22040) — Optional continuous token embedding generator that can replace the discrete input lookup table.
Leviathan-JTok / JTok-M (arXiv:2602.00800) — Optional continuous adaptation of JTok over the shared Leviathan coordinate. JTok applies one normalized surface as a multiplicative MLP gate; JTok-M routes a Sigmoid+TopK mixture of continuous surfaces and adds the paper's depth-scaled residual. Runtime jtok_kernel_backend selects the PyTorch reference or the CCE Triton adapter; the JTok surfaces remain outside MXFP8 conversion.
KHRONOS (arXiv:2505.13315) — Kernel/basis reference used by the Leviathan continuous token generator implementation.
Spelling Bee Embeddings (arXiv:2601.18030) — Augments token embeddings with character-level spelling information.
Token Embedding Manifold analysis (arXiv:2504.01002) — Reference motivation for treating token embeddings as structured objects rather than unconstrained lookup rows.

Attention, positions, and output projection

Virtual Head Attention (arXiv:2610.17959) — Optional group-specific query premix and low-rank head postmix. NeoLLM shares its gated query projection across corresponding head slots in KV groups and applies postmix after the gate; inference weights are not automatically folded into standard GQA projections.
Copy-Paste LM (arXiv:2610.19247) — Optional causal pointer and sink mixed with the vocabulary distribution. CUDA training uses CCE's sparse Copy-Paste kernels; inference returns dense mixed logits.
FAN (arXiv:2502.21309) — Fourier Analysis Networks. A portion of the projection channels are dedicated to periodic cosine/sine features.
MEA (arXiv:2601.19611) — Explicit Multi-head Attention. Adds small learnable interaction matrices between attention heads for K and V.
LUCID (arXiv:2602.10410) — Applies a learned lower-triangular preconditioner to V before attention, decorrelating value representations across positions.
Affine-Scaled Attention (arXiv:2602.23057) — Adds two learnable per-head scalars (α and β) to the softmax weights: [α·softmax(QKᵀ) + β]·V.
XSA (arXiv:2603.09078) — Exclusive Self Attention. After computing attention, removes the component of the output aligned with the token's own value vector.
Directional Routing (arXiv:2603.14923) — Each head learns K=4 directions in the output space; a learned router suppresses the attention output along each direction per input.
Gated Attention (arXiv:2505.06708) — A sigmoid gate is applied to the attention output before the output projection, introducing non-linearity and preventing attention sinks.
Momentum Attention (arXiv:2411.03884) — Modifies Q and K by subtracting a fraction of the previous position's Q and K values (causal first-difference).
Interleaved Head Attention / IHA (arXiv:2602.21371) — Builds pseudo-heads from learned cross-head mixtures to create multiple attention patterns per original head.
REPO (arXiv:2512.14391) — Context re-positioning module that learns contextual position coordinates above a configurable start layer.
GRAPE (arXiv:2512.07805) — Group representational position encoding used by the REPO-GRAPE positional path.
GOAT priors (arXiv:2601.15380) — Optional factorized attention log-prior channels inspired by trainable attention priors.
Hadamard output projection (arXiv:2603.08343) — Replaces dense attention output projection with a structured Hadamard transform plus lightweight scaling.

Normalization, residual flow, and MLP

SeeDNorm (arXiv:2510.22777) — Applied to Q and K projections. Dynamically rescales normalization from the input's own statistics.
LayerNorm Scaling / LNS (arXiv:2502.05795) — Each layer's output is scaled by 1/√ℓ where ℓ is the layer index.
GPAS (arXiv:2506.22049) — Gradient-Preserving Activation Scaling for residual junctions.
PolyNorm (arXiv:2602.04902) — Replaces the standard MLP activation with normalized linear, quadratic, and cubic branches.
SimpleGPT (arXiv:2602.01212) — Second-order geometry-inspired normalization strategy applied inside MLP projections.
StackMemory / STACKTRANS (NeurIPS 2025) — Optional differentiable hidden-state stack between decoder layers.
Attention Residuals / AttnRes (arXiv:2603.15031) — Optional learned depth-wise aggregation over previous layer outputs or block summaries.
LAUREL (arXiv:2411.07501) — Optional learned augmented residual layer with residual-weight and low-rank variants.

Training objectives and training-time regularizers

Cut Cross Entropy (Apple repository) — Memory-efficient next-token loss that avoids materializing the full token-by-vocabulary logits tensor. NeoLLM remains compatible with the upstream package when the extensions below are disabled.
MiLe Loss (arXiv:2310.19531) — Optional detached, mean-normalized predictive-entropy weighting of token losses, implemented inside the extended CCE path.
Output Embedding Centering / mu-loss (arXiv:2601.02031) — Optional lambda * ||mean(output_embeddings)||^2 regularizer for output-logit stability.
MEAP (arXiv:2502.07490) — Optional training-only input corruption that masks a fixed fraction of eligible tokens while preserving clean next-token labels, causal attention, and the inference path.
TWEO (arXiv:2511.23225) — Optional Transformers Without Extreme Outliers activation regularizer for FP8/low-bit-friendly training.
NITP (arXiv:2605.24956) — Optional Next Implicit Token Prediction auxiliary objective using shallow-layer implicit token targets and a cosine loss.
NextLat (arXiv:2511.05963) — Optional next-latent prediction objective using latent dynamics, Smooth L1 supervision, and frozen-head KL.

Optional extended-CCE configuration

Feature	Enabled	Value
MiLe Loss	True	gamma=1.0
mu-loss	True	lambda=0.0001
MEAP	True	ratio=0.15

Copy-Paste uses CCE sparse mixture statistics for CUDA training and computes mu-loss in Torch. MEAP and patch training also use the extended cut-cross-entropy package. Kitsunp/ml-cross-entropy supplies the extended kernels. Copy-Paste exposes three compact scalars: unweighted NTP cross entropy, the MiLe reweighting delta, and the mu-loss penalty. Their sum reconstructs ntp_loss exactly. MEAP reports its eligible and selected counts from the masking kernel; the trainer logs the selected count and exact fraction. Copy-Paste diagnostics use sparse source-token candidates and CCE vocabulary reductions.

Optimizer and training stability

Adam-family optimizer mode — Active variant: AdamW. AdEMAMix (arXiv:2409.03137) contributes the optional beta3 slow-memory branch when ADEMAMIX=True. DeltaMomentum (arXiv:2608.19491) optionally replaces only the fast beta1 first-moment update on hidden Linear weights when DELTA=True; beta2 remains Adam's ordinary squared-gradient EMA in every mode.
BF16 stochastic rounding (arXiv:2502.20566) — Optional master-free BF16 model update with stochastic rather than nearest rounding; the same write-back path is shared by AdamW, DeltaAdamW, AdEMAMix, and DeltaAdEMAMix.
Cautious Weight Decay (arXiv:2510.12402) — Sign-selective weight decay variant used by the custom optimizer logic.
Correction of Decoupled Weight Decay (arXiv:2512.08217) — Adapts decoupled weight decay during learning-rate decay.
AdamHD (arXiv:2511.14721) — Decoupled Huber decay regularization reference used by the optimizer.
Gradient conditioning — CAST active; history-adaptive spectral/norm policies by optimizer group; 15625 committed updates. References: CAST, GradientStabilizer.

Training

Setting	Value
Dataset	FineWeb-Edu (sample-10BT)
Tokens seen	~1.54B (46,875 steps × batch 64 × length 512)
Precision	BF16 model state (bfloat16: 85,151,608 elements, float32: 4,976 elements) + AdamW mixed_bf16_sr_and_fp32_master
Optimizer	AdamW
Gradient transform	CAST active; history-adaptive spectral/norm policies by optimizer group; 15625 committed updates
Learning rate	1e-03 with 4,000-step linear warmup
Patch transition	Reset optimizer/conditioner at step 31,250; per-phase LR warmup; AdEMAMix disabled
Weight decay	coefficients [0.005, 0.01, 0.1]; WD 5/6 groups; CWD 2; Huber 0; Correction 0
Training time	~2h 03m
Hardware	NVIDIA RTX PRO 6000 Blackwell Server Edition

Training curve — total objective

Total includes MiLe when enabled, plus all active auxiliary objectives and regularizers. Without MiLe means model total minus its MiLe delta on the same forward; this is not a separate training run without MiLe.

Step	Train total	Val total	Train without MiLe	Val without MiLe
1	12.2071	—	12.2071	—
250	7.37598	—	7.36894	—
500	7.08916	—	7.07534	—
750	6.99827	—	6.98229	—
1,000	6.93662	—	6.91823	—
1,250	6.87968	—	6.85948	—
1,500	6.85047	—	6.82897	—
1,750	6.82404	—	6.80125	—
2,000	6.79811	—	6.77387	—
2,250	6.77948	—	6.75405	—
2,500	6.76317	—	6.73732	—
2,750	6.75339	—	6.72684	—
3,000	6.74587	—	6.71796	—
3,250	6.73324	—	6.70464	—
3,500	6.71965	—	6.68953	—
3,750	6.70662	—	6.67609	—
4,000	6.69384	—	6.6622	—
4,250	6.68378	—	6.65122	—
4,500	6.68088	—	6.64811	—
4,750	6.67116	—	6.63764	—
5,000	6.651	6.33686	6.61673	6.29133
5,250	6.64948	—	6.61472	—
5,500	6.63456	—	6.59919	—
5,750	6.6218	—	6.58565	—
6,000	6.61534	—	6.57882	—
6,250	6.60755	—	6.57008	—
6,500	6.59944	—	6.56122	—
6,750	6.58753	—	6.54933	—
7,000	6.57682	—	6.53778	—
7,250	6.56522	—	6.52585	—
7,500	6.56043	—	6.52059	—
7,750	6.54482	—	6.50456	—
8,000	6.54673	—	6.50613	—
8,250	6.53365	—	6.49314	—
8,500	6.53127	—	6.49075	—
8,750	6.52225	—	6.48118	—
9,000	6.52116	—	6.47965	—
9,250	6.50776	—	6.46609	—
9,500	6.50002	—	6.45732	—
9,750	6.49394	—	6.45075	—
10,000	6.49153	6.12292	6.44803	6.05622
10,250	6.48598	—	6.44305	—
10,500	6.48253	—	6.43927	—
10,750	6.48033	—	6.4363	—
11,000	6.46783	—	6.4234	—
11,250	6.46721	—	6.42275	—
11,500	6.4642	—	6.41949	—
11,750	6.45546	—	6.41033	—
12,000	6.45845	—	6.4128	—
12,250	6.45045	—	6.40461	—
12,500	6.45553	—	6.41023	—
12,750	6.4438	—	6.39783	—
13,000	6.44048	—	6.39424	—
13,250	6.43464	—	6.38773	—
13,500	6.43469	—	6.38738	—
13,750	6.42953	—	6.38232	—
14,000	6.43035	—	6.38328	—
14,250	6.42096	—	6.37315	—
14,500	6.42369	—	6.37611	—
14,750	6.42259	—	6.37402	—
15,000	6.42098	6.0723	6.3722	5.99667
15,250	6.42244	—	6.37447	—
15,500	6.42016	—	6.37173	—
15,750	6.41308	—	6.36404	—
16,000	6.41148	—	6.36238	—
16,250	6.4162	—	6.36681	—
16,500	6.41589	—	6.36659	—
16,750	6.41561	—	6.36566	—
17,000	6.41452	—	6.36465	—
17,250	6.4097	—	6.36028	—
17,500	6.40915	—	6.35898	—
17,750	6.40859	—	6.35836	—
18,000	6.41462	—	6.36545	—
18,250	6.41013	—	6.35961	—
18,500	6.41387	—	6.36278	—
18,750	6.41503	—	6.36461	—
19,000	6.4116	—	6.36016	—
19,250	6.42104	—	6.3706	—
19,500	6.40993	—	6.35903	—
19,750	6.41581	—	6.36492	—
20,000	6.41766	6.05771	6.36623	5.97291
20,250	6.41769	—	6.36601	—
20,500	6.42143	—	6.36955	—
20,750	6.42045	—	6.36886	—
21,000	6.42601	—	6.37341	—
21,250	6.43097	—	6.3787	—
21,500	6.43918	—	6.38748	—
21,750	6.4308	—	6.37838	—
22,000	6.43807	—	6.38492	—
22,250	6.44129	—	6.38811	—
22,500	6.45435	—	6.40085	—
22,750	6.45836	—	6.40501	—
23,000	6.46614	—	6.41277	—
23,250	6.47444	—	6.42093	—
23,500	6.49124	—	6.438	—
23,750	6.48626	—	6.43198	—
24,000	6.49129	—	6.43726	—
24,250	6.4955	—	6.4411	—
24,500	6.49493	—	6.44063	—
24,750	6.49545	—	6.44064	—
25,000	6.49829	6.13822	6.44368	6.04583
25,250	6.49334	—	6.4377	—
25,500	6.49485	—	6.4396	—
25,750	6.49492	—	6.43941	—
26,000	6.49395	—	6.43765	—
26,250	6.49951	—	6.44314	—
26,500	6.49555	—	6.43918	—
26,750	6.48714	—	6.43075	—
27,000	6.48551	—	6.42859	—
27,250	6.48516	—	6.42747	—
27,500	6.4833	—	6.42543	—
27,750	6.47752	—	6.41906	—
28,000	6.47448	—	6.4165	—
28,250	6.47302	—	6.41484	—
28,500	6.47569	—	6.417	—
28,750	6.46476	—	6.40574	—
29,000	6.46206	—	6.40322	—
29,250	6.4558	—	6.39616	—
29,500	6.46088	—	6.40092	—
29,750	6.44813	—	6.38878	—
30,000	6.44827	6.10304	6.38867	5.99909
30,250	6.44249	—	6.38228	—
30,500	6.43923	—	6.37926	—
30,750	6.43583	—	6.37485	—
31,000	6.42881	—	6.36726	—
31,250	6.421	—	6.35929	—
31,500	5.3043	—	4.97082	—
31,750	5.16695	—	4.78341	—
32,000	5.12219	—	4.72653	—
32,250	5.10113	—	4.69448	—
32,500	5.08835	—	4.67293	—
32,750	5.07676	—	4.6588	—
33,000	5.07125	—	4.6457	—
33,250	5.06328	—	4.63546	—
33,500	5.06395	—	4.63053	—
33,750	5.057	—	4.62161	—
34,000	5.06211	—	4.62438	—
34,250	5.05749	—	4.61758	—
34,500	5.05935	—	4.61795	—
34,750	5.06082	—	4.61662	—
35,000	5.0762	4.68222	4.62962	4.13819
35,250	5.07292	—	4.62418	—
35,500	5.07081	—	4.62016	—
35,750	5.07139	—	4.61963	—
36,000	5.07317	—	4.61856	—
36,250	5.06255	—	4.60507	—
36,500	5.0633	—	4.60396	—
36,750	5.05589	—	4.59619	—
37,000	5.05388	—	4.59042	—
37,250	5.05349	—	4.59123	—
37,500	5.05089	—	4.58645	—
37,750	5.0465	—	4.58071	—
38,000	5.04191	—	4.57527	—
38,250	5.04144	—	4.57399	—
38,500	5.04576	—	4.57612	—
38,750	5.04109	—	4.57359	—
39,000	5.0435	—	4.57074	—
39,250	5.0413	—	4.56876	—
39,500	5.03601	—	4.56386	—
39,750	5.03146	—	4.5574	—
40,000	5.0209	4.63135	4.54455	4.0504
40,250	5.02542	—	4.54959	—
40,500	5.02824	—	4.55001	—
40,750	5.02562	—	4.54841	—
41,000	5.0219	—	4.54302	—
41,250	5.02523	—	4.54739	—
41,500	5.02049	—	4.54063	—
41,750	5.01595	—	4.53672	—
42,000	5.01649	—	4.53607	—
42,250	5.01927	—	4.53746	—
42,500	5.0087	—	4.52705	—
42,750	5.0124	—	4.53123	—
43,000	5.00985	—	4.52501	—
43,250	5.01056	—	4.52837	—
43,500	4.9987	—	4.51446	—
43,750	4.98874	—	4.50358	—
44,000	4.97957	—	4.49141	—
44,250	4.9714	—	4.48315	—
44,500	4.95608	—	4.46543	—
44,750	4.94668	—	4.45473	—
45,000	4.93842	4.53103	4.44464	3.93374
45,250	4.92458	—	4.4306	—
45,500	4.91678	—	4.42322	—
45,750	4.90474	—	4.40819	—
46,000	4.89181	—	4.39414	—
46,250	4.88255	—	4.38307	—
46,500	4.87115	—	4.36981	—
46,750	4.86127	—	4.36009	—
46,875	4.85221	4.44013	4.35211	3.83573

Cross-entropy with and without MiLe

CE without MiLe is unweighted CE; CE with MiLe adds the reweighting delta. Both exclude mu and auxiliary terms. ntp_loss includes mu when enabled.

Step	Train CE without MiLe	Val CE without MiLe	Train CE with MiLe	Val CE with MiLe
1	11.1572	—	11.1572	—
250	7.31147	—	7.3185	—
500	7.02007	—	7.03388	—
750	6.92675	—	6.94273	—
1,000	6.85651	—	6.87491	—
1,250	6.79763	—	6.81783	—
1,500	6.76502	—	6.78651	—
1,750	6.73274	—	6.75553	—
2,000	6.70247	—	6.72671	—
2,250	6.67957	—	6.705	—
2,500	6.65739	—	6.68325	—
2,750	6.64303	—	6.66959	—
3,000	6.62973	—	6.65764	—
3,250	6.61194	—	6.64054	—
3,500	6.59554	—	6.62566	—
3,750	6.57779	—	6.60832	—
4,000	6.56094	—	6.59258	—
4,250	6.54738	—	6.57994	—
4,500	6.54229	—	6.57506	—
4,750	6.53008	—	6.5636	—
5,000	6.50843	6.1771	6.5427	6.22263
5,250	6.50534	—	6.54009	—
5,500	6.48891	—	6.52428	—
5,750	6.47643	—	6.51258	—
6,000	6.46916	—	6.50568	—
6,250	6.46069	—	6.49815	—
6,500	6.45363	—	6.49185	—
6,750	6.44286	—	6.48106	—
7,000	6.43176	—	6.4708	—
7,250	6.42016	—	6.45952	—
7,500	6.41755	—	6.45739	—
7,750	6.40196	—	6.44222	—
8,000	6.40619	—	6.44679	—
8,250	6.39346	—	6.43398	—
8,500	6.39333	—	6.43384	—
8,750	6.38396	—	6.42504	—
9,000	6.38417	—	6.42568	—
9,250	6.37194	—	6.41362	—
9,500	6.36379	—	6.4065	—
9,750	6.35853	—	6.40173	—
10,000	6.35626	5.96623	6.39977	6.03292
10,250	6.35166	—	6.39458	—
10,500	6.34732	—	6.39057	—
10,750	6.34482	—	6.38885	—
11,000	6.33253	—	6.37696	—
11,250	6.33297	—	6.37742	—
11,500	6.32902	—	6.37373	—
11,750	6.31969	—	6.36483	—
12,000	6.32263	—	6.36827	—
12,250	6.31474	—	6.36058	—
12,500	6.3207	—	6.36599	—
12,750	6.30823	—	6.35421	—
13,000	6.3055	—	6.35174	—
13,250	6.29849	—	6.34539	—
13,500	6.29935	—	6.34666	—
13,750	6.29299	—	6.3402	—
14,000	6.293	—	6.34007	—
14,250	6.28336	—	6.33117	—
14,500	6.28572	—	6.3333	—
14,750	6.28234	—	6.33091	—
15,000	6.28125	5.90545	6.33002	5.98108
15,250	6.28134	—	6.3293	—
15,500	6.278	—	6.32643	—
15,750	6.26935	—	6.31839	—
16,000	6.26653	—	6.31563	—
16,250	6.27105	—	6.32045	—
16,500	6.26816	—	6.31746	—
16,750	6.2652	—	6.31516	—
17,000	6.26239	—	6.31226	—
17,250	6.25775	—	6.30717	—
17,500	6.25333	—	6.30351	—
17,750	6.2519	—	6.30214	—
18,000	6.25638	—	6.30554	—
18,250	6.2484	—	6.29893	—
18,500	6.24939	—	6.30049	—
18,750	6.2479	—	6.29832	—
19,000	6.24259	—	6.29403	—
19,250	6.24847	—	6.2989	—
19,500	6.23562	—	6.28652	—
19,750	6.23809	—	6.28899	—
20,000	6.23558	5.84014	6.28702	5.92494
20,250	6.23152	—	6.2832	—
20,500	6.23307	—	6.28496	—
20,750	6.22799	—	6.27957	—
21,000	6.2262	—	6.2788	—
21,250	6.22686	—	6.27912	—
21,500	6.23008	—	6.28179	—
21,750	6.21589	—	6.26831	—
22,000	6.21612	—	6.26928	—
22,250	6.21252	—	6.2657	—
22,500	6.21509	—	6.2686	—
22,750	6.21315	—	6.26649	—
23,000	6.21186	—	6.26523	—
23,250	6.21143	—	6.26495	—
23,500	6.21665	—	6.26989	—
23,750	6.20603	—	6.26031	—
24,000	6.20275	—	6.25679	—
24,250	6.20287	—	6.25727	—
24,500	6.19596	—	6.25026	—
24,750	6.19049	—	6.2453	—
25,000	6.19005	5.74106	6.24466	5.83345
25,250	6.1815	—	6.23713	—
25,500	6.1796	—	6.23485	—
25,750	6.1752	—	6.23071	—
26,000	6.17245	—	6.22874	—
26,250	6.17269	—	6.22906	—
26,500	6.16825	—	6.22462	—
26,750	6.15753	—	6.21392	—
27,000	6.15678	—	6.21369	—
27,250	6.15273	—	6.21042	—
27,500	6.14849	—	6.20636	—
27,750	6.14237	—	6.20084	—
28,000	6.13868	—	6.19666	—
28,250	6.13396	—	6.19214	—
28,500	6.13682	—	6.19551	—
28,750	6.12617	—	6.1852	—
29,000	6.12317	—	6.18201	—
29,250	6.11698	—	6.17663	—
29,500	6.12089	—	6.18085	—
29,750	6.10887	—	6.16822	—
30,000	6.10953	5.64326	6.16913	5.74721
30,250	6.10279	—	6.163	—
30,500	6.09997	—	6.15994	—
30,750	6.09729	—	6.15828	—
31,000	6.09088	—	6.15243	—
31,250	6.08428	—	6.14599	—
31,500	4.76082	—	5.09431	—
31,750	4.56697	—	4.95051	—
32,000	4.49588	—	4.89154	—
32,250	4.45066	—	4.85731	—
32,500	4.41895	—	4.83438	—
32,750	4.39736	—	4.81532	—
33,000	4.37554	—	4.80109	—
33,250	4.35522	—	4.78304	—
33,500	4.34356	—	4.77698	—
33,750	4.32837	—	4.76376	—
34,000	4.32387	—	4.76161	—
34,250	4.31401	—	4.75392	—
34,500	4.30545	—	4.74686	—
34,750	4.29732	—	4.74152	—
35,000	4.3042	3.82901	4.75078	4.37305
35,250	4.2935	—	4.74224	—
35,500	4.2829	—	4.73355	—
35,750	4.28007	—	4.73182	—
36,000	4.27372	—	4.72833	—
36,250	4.25793	—	4.71542	—
36,500	4.25191	—	4.71126	—
36,750	4.24249	—	4.70219	—
37,000	4.23366	—	4.69712	—
37,250	4.2335	—	4.69577	—
37,500	4.22517	—	4.68961	—
37,750	4.21906	—	4.68485	—
38,000	4.21141	—	4.67806	—
38,250	4.20786	—	4.67531	—
38,500	4.20697	—	4.6766	—
38,750	4.20144	—	4.66894	—
39,000	4.19798	—	4.67074	—
39,250	4.19412	—	4.66666	—
39,500	4.18862	—	4.66077	—
39,750	4.18273	—	4.6568	—
40,000	4.16911	3.70109	4.64546	4.28203
40,250	4.17192	—	4.64775	—
40,500	4.16911	—	4.64734	—
40,750	4.16714	—	4.64436	—
41,000	4.16111	—	4.63999	—
41,250	4.16586	—	4.6437	—
41,500	4.15999	—	4.63984	—
41,750	4.15306	—	4.63229	—
42,000	4.15279	—	4.6332	—
42,250	4.15274	—	4.63455	—
42,500	4.1434	—	4.62504	—
42,750	4.14646	—	4.62762	—
43,000	4.13814	—	4.62298	—
43,250	4.13999	—	4.62218	—
43,500	4.12706	—	4.61131	—
43,750	4.1168	—	4.60196	—
44,000	4.10216	—	4.59032	—
44,250	4.09806	—	4.58632	—
44,500	4.08189	—	4.57255	—
44,750	4.07252	—	4.56447	—
45,000	4.05954	3.57298	4.55331	4.17027
45,250	4.04808	—	4.54206	—
45,500	4.04218	—	4.53574	—
45,750	4.02863	—	4.52518	—
46,000	4.01686	—	4.51452	—
46,250	4.00891	—	4.50839	—
46,500	3.99633	—	4.49767	—
46,750	3.988	—	4.48919	—
46,875	3.97999	3.49568	4.48009	4.10008

Observed loss components — weighted contributions

Only contributions actually added to the objective are shown; raw values and subcomponents remain in training metrics. regularization_loss is the subtotal of mu, weighted TWeo, JTok-M and StackMemory entropy, excluding NITP/NextLat; do not add it again to those components. JTok-M auxiliary loss is training-only.

Step	Phase	mu	TWeo	NITP	Reg. subtotal
1	train	3.29888e-10	0.0808869	0.969025	0.0808869
250	train	3.8521e-06	0.0303138	0.0271549	0.0303177
500	train	7.68142e-06	0.0238848	0.0313833	0.0238925
750	train	1.19229e-05	0.0230974	0.0324324	0.0231093
1,000	train	1.70043e-05	0.0240469	0.0376518	0.0240639
1,250	train	2.24079e-05	0.0237961	0.0380324	0.0238185
1,500	train	2.78352e-05	0.0237668	0.0401555	0.0237947
1,750	train	3.28625e-05	0.0241116	0.0443622	0.0241445
2,000	train	3.75912e-05	0.0242267	0.0471408	0.0242643
2,250	train	4.18728e-05	0.0242948	0.0501491	0.0243367
2,500	train	4.61807e-05	0.0246382	0.0552385	0.0246844
2,750	train	5.05082e-05	0.0248349	0.0589201	0.0248854
3,000	train	5.46911e-05	0.0251746	0.0630071	0.0252293
3,250	train	5.85548e-05	0.0254847	0.0671606	0.0255432
3,500	train	6.22264e-05	0.0253437	0.0685814	0.025406
3,750	train	6.59846e-05	0.0255351	0.0726999	0.0256011
4,000	train	6.96894e-05	0.0256563	0.0755377	0.025726
4,250	train	7.30426e-05	0.0256854	0.0780769	0.0257584
4,500	train	7.68797e-05	0.0255725	0.0801712	0.0256494
4,750	train	8.09081e-05	0.0255283	0.0819462	0.0256092
5,000	train	8.48196e-05	0.0255773	0.0826389	0.0256622
5,000	validation	8.68315e-05	0.0246664	0.0894746	0.0247532
5,250	train	8.901e-05	0.0255839	0.083706	0.0256729
5,500	train	9.3647e-05	0.0256139	0.0845738	0.0257075
5,750	train	9.74472e-05	0.0253804	0.0837354	0.0254778
6,000	train	0.000101403	0.0254294	0.0841311	0.0255308
6,250	train	0.000105929	0.0252662	0.0840195	0.0253721
6,500	train	0.000110291	0.0249628	0.0825147	0.0250731
6,750	train	0.000114825	0.024864	0.0814958	0.0249788
7,000	train	0.000118958	0.0247707	0.0811319	0.0248897
7,250	train	0.000123919	0.0246535	0.0809113	0.0247774
7,500	train	0.000129277	0.0244223	0.078488	0.0245515
7,750	train	0.000134021	0.0242397	0.078224	0.0243737
8,000	train	0.000139105	0.0239955	0.0758091	0.0241346
8,250	train	0.000143991	0.0238296	0.0757027	0.0239736
8,500	train	0.000148798	0.0234479	0.0738244	0.0235967
8,750	train	0.000153406	0.0232138	0.0738481	0.0233672
9,000	train	0.00015872	0.0230077	0.072311	0.0231665
9,250	train	0.000164027	0.0227205	0.071257	0.0228845
9,500	train	0.000169391	0.0222335	0.0711188	0.0224029
9,750	train	0.000174372	0.0214483	0.0705936	0.0216227
10,000	train	0.000179542	0.0209098	0.0706794	0.0210894
10,000	validation	0.000182401	0.0201629	0.0696504	0.0203453
10,250	train	0.000184779	0.0205981	0.0706104	0.0207829
10,500	train	0.00018953	0.02046	0.0713015	0.0206496
10,750	train	0.000194131	0.0202887	0.0710016	0.0204828
11,000	train	0.000198279	0.0200454	0.0706242	0.0202436
11,250	train	0.0002017	0.0198869	0.0696934	0.0200886
11,500	train	0.000205739	0.0197674	0.070496	0.0199731
11,750	train	0.000209338	0.0196162	0.0708146	0.0198255
12,000	train	0.000213316	0.0194485	0.070518	0.0196618
12,250	train	0.00021735	0.0193224	0.0703205	0.0195397
12,500	train	0.000220943	0.0191472	0.070165	0.0193682
12,750	train	0.000224731	0.0190224	0.0703456	0.0192471
13,000	train	0.000227983	0.0188226	0.0696899	0.0190506
13,250	train	0.000231379	0.0187499	0.070269	0.0189813
13,500	train	0.000234255	0.0184914	0.0693011	0.0187257
13,750	train	0.000237178	0.0184653	0.070629	0.0187024
14,000	train	0.000240336	0.018334	0.0717052	0.0185743
14,250	train	0.000242687	0.0181791	0.0713737	0.0184218
14,500	train	0.000244618	0.0180648	0.072079	0.0183094
14,750	train	0.000247427	0.01796	0.0734671	0.0182074
15,000	train	0.000249658	0.017792	0.0729089	0.0180417
15,000	validation	0.000250682	0.0174288	0.0735403	0.0176795
15,250	train	0.000251866	0.0177755	0.0751047	0.0180274
15,500	train	0.000253429	0.0176208	0.0758535	0.0178743
15,750	train	0.000255109	0.0175256	0.0769081	0.0177807
16,000	train	0.000256317	0.0173238	0.0782701	0.0175801
16,250	train	0.000257312	0.0171886	0.0783039	0.0174459
16,500	train	0.000258686	0.0170634	0.0811079	0.0173221
16,750	train	0.000260075	0.0168661	0.0833298	0.0171261
17,000	train	0.000261575	0.0166608	0.0853365	0.0169224
17,250	train	0.000262748	0.0164743	0.0857909	0.016737
17,500	train	0.000264425	0.0163145	0.0890699	0.0165789
17,750	train	0.000266239	0.0161565	0.0900327	0.0164228
18,000	train	0.00026749	0.0159942	0.0928199	0.0162616
18,250	train	0.000268807	0.0158714	0.0950671	0.0161402
18,500	train	0.000270501	0.015666	0.0974487	0.0159365
18,750	train	0.000272392	0.015554	0.100882	0.0158264
19,000	train	0.000273823	0.0153654	0.101935	0.0156392
19,250	train	0.00027554	0.0152988	0.106561	0.0155743
19,500	train	0.000277035	0.0151842	0.107952	0.0154612
19,750	train	0.000278132	0.0150072	0.111542	0.0152854
20,000	train	0.000279071	0.0147189	0.115647	0.014998
20,000	validation	0.000280374	0.0131139	0.11938	0.0133943
20,250	train	0.000281571	0.0145879	0.119619	0.0148695
20,500	train	0.00028306	0.0142504	0.121941	0.0145334
20,750	train	0.000283258	0.0139524	0.12664	0.0142357
21,000	train	0.000285022	0.0136085	0.133317	0.0138935
21,250	train	0.000286455	0.0130833	0.138481	0.0133698
21,500	train	0.000287885	0.0125558	0.144548	0.0128437
21,750	train	0.000289409	0.0120283	0.150173	0.0123178
22,000	train	0.000290493	0.0115729	0.156931	0.0118634
22,250	train	0.00029234	0.0111635	0.164135	0.0114558
22,500	train	0.000293594	0.010717	0.174741	0.0110106
22,750	train	0.000294663	0.0100905	0.181482	0.0103852
23,000	train	0.000295349	0.00953402	0.191075	0.00982938
23,250	train	0.000296546	0.0090999	0.200099	0.00939644
23,500	train	0.000297694	0.00878708	0.212264	0.00908477
23,750	train	0.000298638	0.00853235	0.217125	0.00883099
24,000	train	0.000300486	0.00831331	0.225897	0.0086138
24,250	train	0.000301828	0.00809252	0.229837	0.00839435
24,500	train	0.000302075	0.00787618	0.236492	0.00817826
24,750	train	0.000303344	0.0077097	0.242136	0.00801304
25,000	train	0.000304875	0.00756483	0.245754	0.00786971
25,000	validation	0.000305917	0.00676651	0.297695	0.00707243
25,250	train	0.000305665	0.00739388	0.24851	0.00769955
25,500	train	0.000306364	0.00724709	0.252451	0.00755346
25,750	train	0.00030753	0.00715907	0.256747	0.0074666
26,000	train	0.000308476	0.00703375	0.257863	0.00734222
26,250	train	0.000309133	0.00688335	0.263256	0.00719249
26,500	train	0.00030965	0.00678684	0.263829	0.00709649
26,750	train	0.000310672	0.00665888	0.266249	0.00696955
27,000	train	0.000311166	0.00657182	0.264932	0.00688299
27,250	train	0.000311056	0.0064815	0.267953	0.00679256
27,500	train	0.000310984	0.00636602	0.270267	0.006677
27,750	train	0.000311001	0.00622893	0.270144	0.00653993
28,000	train	0.000311403	0.00611439	0.271396	0.00642579
28,250	train	0.000311334	0.00599437	0.274572	0.00630571
28,500	train	0.000311527	0.00590852	0.273962	0.00622004
28,750	train	0.000311633	0.00578729	0.27347	0.00609892
29,000	train	0.000312034	0.00568883	0.27405	0.00600086
29,250	train	0.000312114	0.00558237	0.273284	0.00589448
29,500	train	0.000311875	0.00549642	0.274218	0.00580829
29,750	train	0.000311875	0.00538375	0.274222	0.00569562
30,000	train	0.000311747	0.00529821	0.273532	0.00560996
30,000	validation	0.000311705	0.00475377	0.350763	0.00506548
30,250	train	0.000311736	0.00520607	0.27397	0.0055178
30,500	train	0.000312118	0.00511878	0.273859	0.0054309
30,750	train	0.000312395	0.00505274	0.272187	0.00536514
31,000	train	0.000312702	0.00496985	0.271092	0.00528255
31,250	train	0.000313001	0.00488705	0.269813	0.00520005
31,500	train	0.000323058	0.00573666	0.203932	0.00605972
31,750	train	0.000323924	0.00558573	0.210537	0.00590966
32,000	train	0.000320403	0.0056485	0.224678	0.0059689
32,250	train	0.000316494	0.00570858	0.237793	0.00602508
32,500	train	0.000312253	0.00574866	0.247916	0.00606091
32,750	train	0.000307881	0.00576013	0.255371	0.00606801
33,000	train	0.000303929	0.00582456	0.264031	0.00612849
33,250	train	0.000300557	0.00581612	0.274126	0.00611667
33,500	train	0.000296944	0.00589449	0.280788	0.00619143
33,750	train	0.000293606	0.00592577	0.287024	0.00621937
34,000	train	0.00028993	0.00598094	0.294233	0.00627087
34,250	train	0.000286484	0.00603124	0.297248	0.00631773
34,500	train	0.000283078	0.00603973	0.306172	0.0063228
34,750	train	0.000279717	0.00610147	0.312917	0.00638118
35,000	train	0.000275896	0.00610148	0.319043	0.00637738
35,000	validation	0.000273866	0.00473305	0.304167	0.00500691
35,250	train	0.000271961	0.0061188	0.324297	0.00639076
35,500	train	0.000268012	0.00616773	0.330824	0.00643574
35,750	train	0.00026474	0.00614073	0.333158	0.00640547
36,000	train	0.000261146	0.00613509	0.338449	0.00639624
36,250	train	0.000257776	0.00610106	0.340778	0.00635884
36,500	train	0.000255107	0.00609307	0.345695	0.00634817
36,750	train	0.000252667	0.00608674	0.347362	0.0063394
37,000	train	0.0002506	0.00608031	0.350423	0.00633091
37,250	train	0.000247813	0.00608226	0.351397	0.00633007
37,500	train	0.000245688	0.00607994	0.354953	0.00632563
37,750	train	0.000243068	0.00603569	0.355367	0.00627876
38,000	train	0.000240267	0.0060762	0.357539	0.00631647
38,250	train	0.000238516	0.00608045	0.35981	0.00631896
38,500	train	0.000236898	0.00603063	0.362886	0.00626753
38,750	train	0.000234175	0.00601175	0.365896	0.00624593
39,000	train	0.000232296	0.00604342	0.366486	0.00627572
39,250	train	0.000230337	0.0060527	0.368359	0.00628304
39,500	train	0.000228235	0.00603682	0.368978	0.00626506
39,750	train	0.00022576	0.00597905	0.368457	0.00620481
40,000	train	0.000224625	0.00597015	0.369241	0.00619478
40,000	validation	0.000223949	0.00481836	0.344274	0.00504231
40,250	train	0.000223685	0.0059844	0.371457	0.00620808
40,500	train	0.000223104	0.00597895	0.374692	0.00620205
40,750	train	0.000221936	0.00600542	0.375041	0.00622736
41,000	train	0.000221015	0.00598002	0.375704	0.00620104
41,250	train	0.000220164	0.00599147	0.375318	0.00621163
41,500	train	0.000219283	0.00601003	0.374413	0.00622932
41,750	train	0.000218401	0.0059552	0.377488	0.0061736
42,000	train	0.0002174	0.00598068	0.377087	0.00619808
42,250	train	0.000215949	0.0059673	0.378535	0.00618325
42,500	train	0.000214798	0.00595994	0.37748	0.00617474
42,750	train	0.000213874	0.00594705	0.378618	0.00616093
43,000	train	0.000212817	0.00600602	0.380651	0.00621884
43,250	train	0.000211406	0.00598967	0.382181	0.00620108
43,500	train	0.000211098	0.00590486	0.381278	0.00611596
43,750	train	0.000210518	0.00584336	0.380727	0.00605388
44,000	train	0.000210143	0.00577437	0.383267	0.00598452
44,250	train	0.000209538	0.00570132	0.379179	0.00591086
44,500	train	0.000209154	0.00560471	0.377717	0.00581386
44,750	train	0.000208796	0.00550531	0.376491	0.0057141
45,000	train	0.000209053	0.00540991	0.379487	0.00561896
45,000	validation	0.000208989	0.00419734	0.356357	0.00440633
45,250	train	0.000209135	0.00531997	0.376989	0.00552911
45,500	train	0.000209275	0.00525706	0.375573	0.00546633
45,750	train	0.000208995	0.00515349	0.374198	0.00536249
46,000	train	0.000208498	0.00507658	0.372003	0.00528508
46,250	train	0.000208335	0.00500956	0.368943	0.00521789
46,500	train	0.000208313	0.00490162	0.368367	0.00510993
46,750	train	0.000208119	0.00482553	0.367054	0.00503365
46,875	train	0.000207968	0.00474899	0.367164	0.00495696
46,875	validation	0.000207941	0.00353233	0.336315	0.00374027

Tables use exact logged steps; missing values are — without carrying values forward. Repeated evaluations at one step use the latest logged value. Components average batch scalars weighted by examples; CE/PPL token metrics retain their valid-target weighting. Weight decay/CWD act in the optimizer, outside these forward losses.

Limitations

Token budget — ~1.5 B tokens seen; below estimated optimum. Knowledge-intensive tasks will improve with more training.
Gradient spike at step 40k — Reorganized the attention pattern in layer 9 that previously captured long-range token correlations. A checkpoint from ~step 38k is expected to have better aggregate benchmark scores.
PolyNorm exclusivity — The quadratic branch has become partially redundant with the linear branch. Will be corrected in the next training run.
Base model only — Not instruction-tuned or aligned; purely a next-token-prediction base model.

References

All papers whose techniques are integrated into NeoLLM's architecture, training objective, or training stack:

Area	Technique	Paper title	Reference
Embeddings	Learnable Multipliers	Freeing the Scale of Language Model Matrix Layers	arXiv:2601.04890
Embeddings	Leviathan	A Separable Architecture for Continuous Token Representation in Language Models	arXiv:2601.22040
Embeddings / residual modulation	JTok / JTok-M	JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation	arXiv:2602.00800
Embeddings	KHRONOS	KHRONOS: a Kernel-Based Neural Architecture for Rapid, Resource-Efficient Scientific Computation	arXiv:2505.13315
Embeddings	Spelling Bee	Spelling Bee Embeddings for Language Modeling	arXiv:2601.18030
Embeddings	Token embedding analysis	Token Embeddings Violate the Manifold Hypothesis	arXiv:2504.01002
Attention / positions	FAN	Fourier Analysis Networks	arXiv:2502.21309
Attention / positions	MEA	Explicit Multi-head Attention for Inter-head Interaction in Large Language Models	arXiv:2601.19611
Attention / positions	LUCID	Attention with Preconditioned Representations	arXiv:2602.10410
Attention / positions	Affine-Scaled Attention	Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention	arXiv:2602.23057
Attention / positions	XSA	Exclusive Self Attention	arXiv:2603.09078
Attention / positions	Directional Routing	Directional Routing in Transformers	arXiv:2603.14923
Attention / positions	Gated Attention	Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free	arXiv:2505.06708
Attention / positions	Momentum Attention	Momentum Attention	arXiv:2411.03884
Attention / positions	IHA	Interleaved Head Attention	arXiv:2602.21371
Attention / positions	REPO	Language Models with Context Re-Positioning	arXiv:2512.14391
Attention / positions	GRAPE	Group Representational Position Encoding	arXiv:2512.07805
Attention / positions	GOAT priors	You Need Better Attention Priors	arXiv:2601.15380
Attention / positions	Hadamard o_proj	Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers	arXiv:2603.08343
Residual / normalization	SeeDNorm	Self-Rescaled Dynamic Normalization	arXiv:2510.22777
Residual / normalization	LNS	The Curse of Depth in LLMs	arXiv:2502.05795
Residual / normalization	GPAS	Gradient-Preserving Activation Scaling	arXiv:2506.22049
Residual / normalization	PolyNorm	PolyNorm / PolyCom	arXiv:2602.04902
Residual / normalization	SimpleGPT	SimpleGPT	arXiv:2602.01212
Residual / normalization	StackMemory / STACKTRANS	Recursive Transformer: Boosting Reasoning Ability with State Stack	NeurIPS 2025
Residual / normalization	Attention Residuals	Attention Residuals	arXiv:2603.15031
Residual / normalization	Linear Depth Memory	Depth Memory	arXiv:2610.18809
Residual / normalization	LAUREL	LAUREL: Learned Augmented Residual Layer	arXiv:2411.07501
Objectives	TWEO	Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies	arXiv:2511.23225
Objectives	NITP	Next Implicit Token Prediction for LLM Pre-training	arXiv:2605.24956
Objectives	NextLat	Next-Latent Prediction Transformers Learn Compact World Models	arXiv:2511.05963
Optimizer / training	AdEMAMix	The AdEMAMix Optimizer: Better, Faster, Older	arXiv:2409.03137
Optimizer / training	DeltaMomentum	DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule	arXiv:2608.19491
Optimizer / training	BF16 + SR	Stochastic Rounding for LLM Training: Theory and Practice	arXiv:2502.20566
Optimizer / training	CWD	Cautious Weight Decay	arXiv:2510.12402
Optimizer / training	WD correction	Correction of Decoupled Weight Decay	arXiv:2512.08217
Optimizer / training	AdamHD	AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training	arXiv:2511.14721
Optimizer / training	GradientStabilizer	GradientStabilizer	arXiv:2502.17055

Citation

@misc{neollm2026,
  title  = {NeoLLM: A Research Language Model Integrating Recent Attention and Normalization Techniques},
  author = {KitsuVp},
  year   = {2026},
  url    = {https://huggingface.co/KitsuVp/NeoLLM}
}

Author

@Kyokopom on X

License

Apache 2.0

Downloads last month: 944

Safetensors

Model size

85.2M params

Tensor type

F32

BF16

Dataset used to train KitsuVp/NeoLLM

Papers for KitsuVp/NeoLLM