Upload README.md with huggingface_hub

README.md

@@ -1,90 +1,306 @@
 ---
-library_name: transformers
+language: en
+license: apache-2.0
 tags:
-- generated_from_trainer
-model-index:
-- name: NeoLLM
-  results: []
+- causal-lm
+- research
+- fp8
+- attention
+- normalization
+- neollm
+- pace
+datasets:
+- HuggingFaceFW/fineweb-edu
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
 # NeoLLM
 
-This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: 3.6218
-- Ntp Loss: 3.3629
-- Tweo Loss: 0.0027
-- Nitp Loss: 0.1235
-- Total Model Loss: 3.4865
-- Optimizer Step: 15625.0
-- Optimizer Metrics Due: 0.0
-- Pace Step: 0.0
-- Pace Update Due: 0.0
-- Pace Previous Iterate Active: 1.0
-- Conda Adaptive Scale Active: 1.0
-- Conda Scale Smoothed Metric Exact: 1.0
-- Conda Scale Beta: 0.0080
-- Conda Scale Energy Attenuation: 0.0053
-- Conda Scale Candidate Ratio Mean: 1.0006
-- Conda Scale Smoothed Ratio Mean: 0.9927
-- Conda Scale Applied Ratio Mean: 0.9927
-- Conda Scale Gain Mean: 0.1366
-- Conda Scale Gain Clip Fraction: 0.0
-- Conda Scale Ratio Min: 0.9214
-- Conda Scale Ratio Max: 1.0000
-- Conda Scale Low Limit Fraction: 0.0
-- Conda Scale High Limit Fraction: 0.0
-- Conda Scale Amplified Fraction: 0.0
-- Conda Scale Attenuated Fraction: 1.0
-- Conda Scale Energy Weighted Amplification: 0.0
-- Conda Scale Energy Weighted Attenuation Applied: 0.0073
-- Conda Scale Risk Ema Mean: 0.0111
-- Conda Scale Risk Drop Mean: 0.0058
-- Conda Scale Amplification Bonus Mean: 0.0058
-- Conda Scale Monitored Matrices: 146.0
-
-## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
-
-## Training procedure
-
-### Training hyperparameters
-
-The following hyperparameters were used during training:
-- learning_rate: 0.0006
-- train_batch_size: 64
-- eval_batch_size: 64
-- seed: 42
-- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: linear
-- lr_scheduler_warmup_steps: 0.1
-- num_epochs: 1
-
-### Training results
-
-| Training Loss | Epoch | Step | Validation Loss | Loss   | Model Loss | Metrics Due | Update Due | Previous Iterate Active | Adaptive Scale Active | Scale Smoothed Metric Exact | Scale Beta | Scale Energy Attenuation | Scale Candidate Ratio Mean | Scale Smoothed Ratio Mean | Scale Applied Ratio Mean | Scale Gain Mean | Scale Gain Clip Fraction | Scale Ratio Min | Scale Ratio Max | Scale Low Limit Fraction | Scale High Limit Fraction | Scale Amplified Fraction | Scale Attenuated Fraction | Scale Energy Weighted Amplification | Scale Energy Weighted Attenuation Applied | Scale Risk Ema Mean | Scale Risk Drop Mean | Scale Amplification Bonus Mean | Scale Monitored Matrices |
-|:-------------:|:-----:|:----:|:---------------:|:------:|:----------:|:-----------:|:----------:|:-----------------------:|:---------------------:|:---------------------------:|:----------:|:------------------------:|:--------------------------:|:-------------------------:|:------------------------:|:---------------:|:------------------------:|:---------------:|:---------------:|:------------------------:|:-------------------------:|:------------------------:|:-------------------------:|:-----------------------------------:|:-----------------------------------------:|:-------------------:|:--------------------:|:------------------------------:|:------------------------:|
-| 4.2787        | 0.32  | 0.0  | 4.1788          | 0.1946 | 4.0601     | 1.0         | 0.0        | 1.0                     | 1.0                   | 1.0                         | 0.0141     | 0.1788                   | 0.8214                     | 0.8242                    | 0.8242                   | 0.3658          | 0.2413                   | 0.0000          | 0.9996          | 0.0274                   | 0.0                       | 0.0                      | 1.0                       | 0.0                                 | 0.1758                                    | 0.1770              | 0.0002               | 0.0002                         | 146.0                    |
-| 3.8661        | 0.64  | 0.0  | 3.8389          | 0.1189 | 3.7143     | 1.0         | 0.0        | 1.0                     | 1.0                   | 1.0                         | 0.0100     | 0.0503                   | 0.9538                     | 0.9471                    | 0.9471                   | 0.2364          | 0.1863                   | 0.4435          | 0.9999          | 0.0                      | 0.0                       | 0.0                      | 1.0                       | 0.0                                 | 0.0529                                    | 0.0545              | 0.0042               | 0.0042                         | 146.0                    |
-| 3.6763        | 0.96  | 0.0  | 3.6376          | 0.1266 | 3.5009     | 1.0         | 0.0        | 1.0                     | 1.0                   | 1.0                         | 0.0082     | 0.0255                   | 0.9746                     | 0.9858                    | 0.9858                   | 0.1701          | 0.1385                   | 0.8801          | 1.0000          | 0.0                      | 0.0                       | 0.0                      | 1.0                       | 0.0                                 | 0.0142                                    | 0.0155              | 0.0001               | 0.0001                         | 146.0                    |
-| 3.6592        | 1.0   | 0.0  | 3.6218          | 0.1235 | 3.4865     | 0.0         | 0.0        | 1.0                     | 1.0                   | 1.0                         | 0.0080     | 0.0053                   | 1.0006                     | 0.9927                    | 0.9927                   | 0.1366          | 0.0                      | 0.9214          | 1.0000          | 0.0                      | 0.0                       | 0.0                      | 1.0                       | 0.0                                 | 0.0073                                    | 0.0111              | 0.0058               | 0.0058                         | 146.0                    |
-
-
-### Framework versions
-
-- Transformers 5.13.0
-- Pytorch 2.13.0+cu132
-- Datasets 5.0.0
-- Tokenizers 0.22.2
+NeoLLM is a **135 M parameter** decoder-only language model trained from scratch on
+[FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) in **FP8**
+precision, completing training in approximately **6 hours** on a single NVIDIA RTX 5090.
+It integrates a collection of recently published attention and normalization techniques
+into a single architecture, with the goal of studying how they interact during
+pretraining. The model is actively being developed and the current checkpoint represents
+an intermediate training state.
+
+> **Author / contact:** [@Kyokopom](https://x.com/Kyokopom) on X
+> **Repository:** [KitsuVp/NeoLLM](https://huggingface.co/KitsuVp/NeoLLM)
+
+---
+
+## Architecture
+
+NeoLLM is a decoder-only transformer with the following configuration:
+
+| Parameter | Value |
+|---|---|
+| Hidden size | 512 |
+| Layers | 12 |
+| Attention heads | 8 |
+| KV heads (GQA) | 4 |
+| Head dim | 64 |
+| Intermediate size | 1536 |
+| Vocabulary | Qwen3 tokenizer (64,402 tokens) |
+| Context length | 512 tokens |
+
+### Parameter breakdown
+
+| Parameter bucket | Count |
+|---|---|
+| **Total parameters** | 116.22M (116,216,184) |
+| **Embedding parameters** (tied) | 32.97M (32,973,824) |
+| **Non-embedding parameters** | 83.24M (83,242,360) |
+| **Effective trainable parameters** | 116.22M (116,216,184) |
+
+> Weight tying is **enabled**: the input embedding matrix and the language-model head
+> share the same parameters, so the effective trainable budget is
+> `total − embed = 83.24M`.
+
+### Integrated techniques
+
+NeoLLM combines architecture modules, optional auxiliary objectives, and
+training-time optimizer/stability components from the following papers.
+
+**Embedding and token representation**
+
+- **Learnable Multipliers** ([arXiv:2601.04890](https://arxiv.org/abs/2601.04890)) — Adds
+  per-row and per-column learnable scalar parameters to selected matrix layers and, when
+  enabled, embeddings.
+- **Leviathan** ([arXiv:2601.22040](https://arxiv.org/abs/2601.22040)) — Optional
+  continuous token embedding generator that can replace the discrete input lookup table.
+- **KHRONOS** ([arXiv:2505.13315](https://arxiv.org/abs/2505.13315)) — Kernel/basis
+  reference used by the Leviathan continuous token generator implementation.
+- **JTok / JTok-M** ([arXiv:2602.00800](https://arxiv.org/abs/2602.00800)) — Optional
+  token-indexed self-modulation surfaces over Leviathan coordinates.
+- **Spelling Bee Embeddings** ([arXiv:2601.18030](https://arxiv.org/abs/2601.18030)) —
+  Augments token embeddings with character-level spelling information.
+- **Token Embedding Manifold analysis** ([arXiv:2504.01002](https://arxiv.org/abs/2504.01002)) —
+  Reference motivation for treating token embeddings as structured objects rather than
+  unconstrained lookup rows.
+
+**Attention, positions, and output projection**
+
+- **FAN** ([arXiv:2502.21309](https://arxiv.org/abs/2502.21309)) — Fourier Analysis Networks.
+  A portion of the projection channels are dedicated to periodic cosine/sine features.
+- **MEA** ([arXiv:2601.19611](https://arxiv.org/abs/2601.19611)) — Explicit Multi-head
+  Attention. Adds small learnable interaction matrices between attention heads for K and V.
+- **LUCID** ([arXiv:2602.10410](https://arxiv.org/abs/2602.10410)) — Applies a learned
+  lower-triangular preconditioner to V before attention, decorrelating value representations
+  across positions.
+- **Affine-Scaled Attention** ([arXiv:2602.23057](https://arxiv.org/abs/2602.23057)) — Adds
+  two learnable per-head scalars (α and β) to the softmax weights:
+  `[α·softmax(QKᵀ) + β]·V`.
+- **XSA** ([arXiv:2603.09078](https://arxiv.org/abs/2603.09078)) — Exclusive Self Attention.
+  After computing attention, removes the component of the output aligned with the token's
+  own value vector.
+- **Directional Routing** ([arXiv:2603.14923](https://arxiv.org/abs/2603.14923)) — Each head
+  learns K=4 directions in the output space; a learned router suppresses the attention output
+  along each direction per input.
+- **Gated Attention** ([arXiv:2505.06708](https://arxiv.org/abs/2505.06708)) — A sigmoid gate
+  is applied to the attention output before the output projection, introducing non-linearity
+  and preventing attention sinks.
+- **Momentum Attention** ([arXiv:2411.03884](https://arxiv.org/abs/2411.03884)) — Modifies Q
+  and K by subtracting a fraction of the previous position's Q and K values (causal
+  first-difference).
+- **Interleaved Head Attention / IHA** ([arXiv:2602.21371](https://arxiv.org/abs/2602.21371)) —
+  Builds pseudo-heads from learned cross-head mixtures to create multiple attention patterns
+  per original head.
+- **REPO** ([arXiv:2512.14391](https://arxiv.org/abs/2512.14391)) — Context re-positioning
+  module that learns contextual position coordinates above a configurable start layer.
+- **GRAPE** ([arXiv:2512.07805](https://arxiv.org/abs/2512.07805)) — Group representational
+  position encoding used by the REPO-GRAPE positional path.
+- **GOAT priors** ([arXiv:2601.15380](https://arxiv.org/abs/2601.15380)) — Optional
+  factorized attention log-prior channels inspired by trainable attention priors.
+- **Hadamard output projection** ([arXiv:2603.08343](https://arxiv.org/abs/2603.08343)) —
+  Replaces dense attention output projection with a structured Hadamard transform plus
+  lightweight scaling.
+
+**Normalization, residual flow, and MLP**
+
+- **SeeDNorm** ([arXiv:2510.22777](https://arxiv.org/abs/2510.22777)) — Applied to Q and K
+  projections. Dynamically rescales normalization from the input's own statistics.
+- **LayerNorm Scaling / LNS** ([arXiv:2502.05795](https://arxiv.org/abs/2502.05795)) — Each
+  layer's output is scaled by 1/√ℓ where ℓ is the layer index.
+- **GPAS** ([arXiv:2506.22049](https://arxiv.org/abs/2506.22049)) — Gradient-Preserving
+  Activation Scaling for residual junctions.
+- **PolyNorm** ([arXiv:2602.04902](https://arxiv.org/abs/2602.04902)) — Replaces the standard
+  MLP activation with normalized linear, quadratic, and cubic branches.
+- **SimpleGPT** ([arXiv:2602.01212](https://arxiv.org/abs/2602.01212)) — Second-order
+  geometry-inspired normalization strategy applied inside MLP projections.
+- **StackMemory / STACKTRANS** ([NeurIPS 2025](https://openreview.net/forum?id=2bbDg587uh)) —
+  Optional differentiable hidden-state stack between decoder layers.
+- **Attention Residuals / AttnRes** ([arXiv:2603.15031](https://arxiv.org/abs/2603.15031)) —
+  Optional learned depth-wise aggregation over previous layer outputs or block summaries.
+- **LAUREL** ([arXiv:2411.07501](https://arxiv.org/abs/2411.07501)) — Optional learned
+  augmented residual layer with residual-weight and low-rank variants.
+
+**Training objectives and training-time regularizers**
+
+- **TWEO** ([arXiv:2511.23225](https://arxiv.org/abs/2511.23225)) — Optional
+  Transformers Without Extreme Outliers activation regularizer for FP8/low-bit-friendly
+  training.
+- **NITP** ([arXiv:2605.24956](https://arxiv.org/abs/2605.24956)) — Optional Next Implicit
+  Token Prediction auxiliary objective using shallow-layer implicit token targets and a
+  cosine loss.
+- **NextLat** ([arXiv:2511.05963](https://arxiv.org/abs/2511.05963)) — Optional next-latent
+  prediction objective using latent dynamics, Smooth L1 supervision, and frozen-head KL.
+
+**Optimizer and training stability**
+
+- **Conda** ([arXiv:2509.24218](https://arxiv.org/abs/2509.24218)) —
+  Column-Normalized Adam optimizer path used by the training script.
+- **Cautious Weight Decay** ([arXiv:2510.12402](https://arxiv.org/abs/2510.12402)) —
+  Sign-selective weight decay variant used by the custom optimizer logic.
+- **Correction of Decoupled Weight Decay** ([arXiv:2512.08217](https://arxiv.org/abs/2512.08217)) —
+  Adapts decoupled weight decay during learning-rate decay.
+- **AdamHD** ([arXiv:2511.14721](https://arxiv.org/abs/2511.14721)) —
+  Decoupled Huber decay regularization reference used by the optimizer.
+- **GradientStabilizer** ([arXiv:2502.17055](https://arxiv.org/abs/2502.17055)) —
+  Optional threshold-free gradient magnitude stabilizer.
+- **PACE** ([arXiv:2606.25086](https://arxiv.org/abs/2606.25086)) —
+  Optional iterate-average controller that trains for the EMA model returned at evaluation
+  and final serialization. The Conda-basis adaptation and its difference from AdamW are
+  documented below.
+
+---
+
+### PACE integration and AdamW-reference differences
+
+PACE follows Au and Block's returned-model objective: the live weights are pulled toward a
+power-law EMA with a clipped per-coordinate gain, and evaluation/final serialization use that
+EMA estimator.
+
+- **Reference AdamW rule:** the gain uses AdamW's original-coordinate diagonal
+  second moment, `eta * c * (1+t)^(-kappa) / (sqrt(v_hat) + eps)`.
+- **NeoLLM Conda rule (`mode=conda`):** for projected 2-D tensors, both the EMA
+  displacement and `v_hat` are represented in Conda's cached SVD basis. The control is
+  projected back after applying the diagonal gain. This is a deliberate change from AdamW
+  required to avoid mixing incompatible coordinate systems.
+- **Optional exact AdamW pullback geometry (`mode=adamw`):** an additional
+  original-coordinate second moment is maintained for projected matrices. The live optimizer
+  step remains Conda.
+- **Conda scale:** in `mode=conda`, Conda's matrix-update scale multiplies the unsaturated
+  gain automatically because it is part of the effective Conda preconditioner. AdamW has no
+  corresponding scale.
+- **Fixed algorithm internals:** the EMA is stored in FP32, the gain is clipped at `1`, and PACE
+  reuses each Conda group’s numerical epsilon. These are not exposed as independent switches.
+- **Minimal modes:** `use_pace=False` is plain Conda; `use_pace=True, c=0` is Conda+EMA;
+  `use_pace=True, c>0` is complete PACE.
+- **Ordering:** PACE runs only after Conda, CWD/CHD, and weight-decay correction have fully
+  updated the live weights.
+- **Disabled guarantee:** with `use_pace=False`, no PACE state is allocated and no existing
+  Conda arithmetic or parameter update is changed.
+- **Checkpoint policy:** resumable internal checkpoints retain live weights and complete optimizer
+  state, while evaluation and the final returned/Hub model always use the EMA when PACE is active.
+
+Current run: **disabled; no EMA state, auxiliary moment, or pullback is allocated**.
+
+---
+
+## Training
+
+| Setting | Value |
+|---|---|
+| Dataset | FineWeb-Edu (sample-10BT) |
+| Tokens seen | ~0.51B (15,625 steps × batch 64 × length 512) |
+| Precision | FP8 native (E4M3 weights/activations, E5M2 gradients) + BF16 fallback |
+| Optimizer | Conda (PACE disabled) |
+| PACE | disabled; no EMA state, auxiliary moment, or pullback is allocated |
+| Learning rate | 6e-04 with linear warmup (10 % of steps) |
+| Weight decay | 0.1 |
+| Training time | ~1h 24m |
+| Hardware | NVIDIA RTX 5090 (single GPU) |
+
+### Training curve
+
+| Step | Train Loss | Val Loss |
+|---|---|---|
+| 5,000 | 4.279 | 4.179 |
+| 10,000 | 3.866 | 3.839 |
+| 15,000 | 3.676 | 3.638 |
+| 15,625 | — | 3.622 |
+
+---
+
+## Limitations
+
+- **Token budget** — ~1.5 B tokens seen; below estimated optimum. Knowledge-intensive tasks
+  will improve with more training.
+- **Gradient spike at step 40k** — Reorganized the attention pattern in layer 9 that
+  previously captured long-range token correlations. A checkpoint from ~step 38k is expected
+  to have better aggregate benchmark scores.
+- **PolyNorm exclusivity** — The quadratic branch has become partially redundant with the
+  linear branch. Will be corrected in the next training run.
+- **Base model only** — Not instruction-tuned or aligned; purely a next-token-prediction
+  base model.
+
+---
+
+## References
+
+All papers whose techniques are integrated into NeoLLM's architecture,
+training objective, or training stack:
+
+| Area | Technique | Paper title | Reference |
+|---|---|---|---|
+| Embeddings | Learnable Multipliers | Freeing the Scale of Language Model Matrix Layers | [arXiv:2601.04890](https://arxiv.org/abs/2601.04890) |
+| Embeddings | Leviathan | A Separable Architecture for Continuous Token Representation in Language Models | [arXiv:2601.22040](https://arxiv.org/abs/2601.22040) |
+| Embeddings | KHRONOS | KHRONOS: a Kernel-Based Neural Architecture for Rapid, Resource-Efficient Scientific Computation | [arXiv:2505.13315](https://arxiv.org/abs/2505.13315) |
+| Embeddings | JTok / JTok-M | JTok: On Token Embedding as Another Axis of Scaling Law via Joint Token Self-Modulation | [arXiv:2602.00800](https://arxiv.org/abs/2602.00800) |
+| Embeddings | Spelling Bee | Spelling Bee Embeddings for Language Modeling | [arXiv:2601.18030](https://arxiv.org/abs/2601.18030) |
+| Embeddings | Token embedding analysis | Token Embeddings Violate the Manifold Hypothesis | [arXiv:2504.01002](https://arxiv.org/abs/2504.01002) |
+| Attention / positions | FAN | Fourier Analysis Networks | [arXiv:2502.21309](https://arxiv.org/abs/2502.21309) |
+| Attention / positions | MEA | Explicit Multi-head Attention for Inter-head Interaction in Large Language Models | [arXiv:2601.19611](https://arxiv.org/abs/2601.19611) |
+| Attention / positions | LUCID | Attention with Preconditioned Representations | [arXiv:2602.10410](https://arxiv.org/abs/2602.10410) |
+| Attention / positions | Affine-Scaled Attention | Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention | [arXiv:2602.23057](https://arxiv.org/abs/2602.23057) |
+| Attention / positions | XSA | Exclusive Self Attention | [arXiv:2603.09078](https://arxiv.org/abs/2603.09078) |
+| Attention / positions | Directional Routing | Directional Routing in Transformers | [arXiv:2603.14923](https://arxiv.org/abs/2603.14923) |
+| Attention / positions | Gated Attention | Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free | [arXiv:2505.06708](https://arxiv.org/abs/2505.06708) |
+| Attention / positions | Momentum Attention | Momentum Attention | [arXiv:2411.03884](https://arxiv.org/abs/2411.03884) |
+| Attention / positions | IHA | Interleaved Head Attention | [arXiv:2602.21371](https://arxiv.org/abs/2602.21371) |
+| Attention / positions | REPO | Language Models with Context Re-Positioning | [arXiv:2512.14391](https://arxiv.org/abs/2512.14391) |
+| Attention / positions | GRAPE | Group Representational Position Encoding | [arXiv:2512.07805](https://arxiv.org/abs/2512.07805) |
+| Attention / positions | GOAT priors | You Need Better Attention Priors | [arXiv:2601.15380](https://arxiv.org/abs/2601.15380) |
+| Attention / positions | Hadamard o_proj | Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers | [arXiv:2603.08343](https://arxiv.org/abs/2603.08343) |
+| Residual / normalization | SeeDNorm | Self-Rescaled Dynamic Normalization | [arXiv:2510.22777](https://arxiv.org/abs/2510.22777) |
+| Residual / normalization | LNS | The Curse of Depth in LLMs | [arXiv:2502.05795](https://arxiv.org/abs/2502.05795) |
+| Residual / normalization | GPAS | Gradient-Preserving Activation Scaling | [arXiv:2506.22049](https://arxiv.org/abs/2506.22049) |
+| Residual / normalization | PolyNorm | PolyNorm / PolyCom | [arXiv:2602.04902](https://arxiv.org/abs/2602.04902) |
+| Residual / normalization | SimpleGPT | SimpleGPT | [arXiv:2602.01212](https://arxiv.org/abs/2602.01212) |
+| Residual / normalization | StackMemory / STACKTRANS | Recursive Transformer: Boosting Reasoning Ability with State Stack | [NeurIPS 2025](https://openreview.net/forum?id=2bbDg587uh) |
+| Residual / normalization | Attention Residuals | Attention Residuals | [arXiv:2603.15031](https://arxiv.org/abs/2603.15031) |
+| Residual / normalization | LAUREL | LAUREL: Learned Augmented Residual Layer | [arXiv:2411.07501](https://arxiv.org/abs/2411.07501) |
+| Objectives | TWEO | Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies | [arXiv:2511.23225](https://arxiv.org/abs/2511.23225) |
+| Objectives | NITP | Next Implicit Token Prediction for LLM Pre-training | [arXiv:2605.24956](https://arxiv.org/abs/2605.24956) |
+| Objectives | NextLat | Next-Latent Prediction Transformers Learn Compact World Models | [arXiv:2511.05963](https://arxiv.org/abs/2511.05963) |
+| Optimizer / training | Conda | Column-Normalized Adam for Training Large Language Models Faster | [arXiv:2509.24218](https://arxiv.org/abs/2509.24218) |
+| Optimizer / training | CWD | Cautious Weight Decay | [arXiv:2510.12402](https://arxiv.org/abs/2510.12402) |
+| Optimizer / training | WD correction | Correction of Decoupled Weight Decay | [arXiv:2512.08217](https://arxiv.org/abs/2512.08217) |
+| Optimizer / training | AdamHD | AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training | [arXiv:2511.14721](https://arxiv.org/abs/2511.14721) |
+| Optimizer / training | GradientStabilizer | GradientStabilizer | [arXiv:2502.17055](https://arxiv.org/abs/2502.17055) |
+| Optimizer / training | PACE | Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models | [arXiv:2606.25086](https://arxiv.org/abs/2606.25086) |
+
+---
+
+## Citation
+
+```bibtex
+@misc{neollm2026,
+  title  = {NeoLLM: A Research Language Model Integrating Recent Attention and Normalization Techniques},
+  author = {KitsuVp},
+  year   = {2026},
+  url    = {https://huggingface.co/KitsuVp/NeoLLM}
+}
+```
+
+---
+
+## Author
+
+[@Kyokopom](https://x.com/Kyokopom) on X
+
+---
+
+## License
+
+Apache 2.0