LDT-10M

A 10.28M-parameter LLaMA-style language model, trained from scratch on FineWeb-Edu + DCLM.

Requested by DedeProGames on the model-requests board (#12).

Architecture

Parameter	Value
Params	10,284,480
Layers	5
d_model	320
Heads	5 (MHA, GQA not used at this scale)
FFN dim	896 (SwiGLU)
Vocab	12,288 (gollem BPE)
Context	512
Embeddings	Tied (lm_head → tok.weight)
Norm	RMSNorm (eps 1e-5)
Attention	RoPE + causal SDPA
Dtype	float32

Standard LLaMA block: RMSNorm → MHA (RoPE) → residual → RMSNorm → SwiGLU FFN → residual.

Training

	v1 (first ckpt)	v4	v7 (current)
Steps	4,000	16,000	30,000
Batch size	64	64	64
LR	3e-4 → 3e-5 (cosine)	1e-4 → 1e-5 (cosine, fresh optimizer)	1e-4 → 1e-5 (cosine, fresh optimizer)
Data	FineWeb-Edu + DCLM	+ FineWeb-Edu + DCLM	+ FineWeb-Edu + DCLM (continued)
Hardware	RTX 5090 (32 GB)	RTX 5090 (32 GB)	RTX 5090 (32 GB)
Final val loss	4.6020 (ppl 99.68)	3.8943 (ppl 49.12)	3.6892 (ppl 40.01)

v7 is a continuation of v4 (resumed from step 12,000 to step 30,000). The data file was cleaned up after training, so the exact cumulative token count is not re-derived here; the step count and val loss are the measured values.

⚠️ Honest caveat: undertrained and still incoherent

DedeProGames requested 2.6B tokens. This checkpoint is well short of that. At this token budget the model has learned the surface shape of English — real words, parseable sentences, occasional token loops — but the prose is still semantically incoherent (word salad): grammatically plausible sentences that don't mean what they say. The val loss (3.689) is well below the 7.38 unigram floor, so it genuinely uses context; the improvement 4.60 → 3.89 → 3.69 is real.

This is a continued checkpoint, not the final deliverable. More data is the fix; continued training toward the 2.6B budget is planned.

Eval (40 samples: 8 prompts × 5 seeds, temp 0.8, top-k 40, 160 new tokens)

Metric	v1	v4	v7
val loss	4.6020	3.8943	3.6892
perplexity	99.68	49.12	40.01
Below unigram floor (7.38)?	Yes	Yes	Yes
mean 4-gram loop fraction	n/a	~0.11	0.167
Degenerate (mean loop > 0.30)?	No	No	No
Semantically coherent?	No	No	No (improved, still word salad)

Sample outputs (v7, real generation from the weights)

"Once upon a time a human brain will experience a normal amount of memory. The body needs to be removed from the brain due to the fact that the brain is not working properly, and therefore the brain needs to get a lot of memory back to the very first and then there is a chance to think in the future."

"Once upon a time, you can easily make a decision about not being involved. - If you are not involved in a job, go to the office, and let your decision be made, then ask you to be a job as soon as possible."

These are real outputs from the v7 weights (not hand-picked for coherence). The text is grammatically structured — real words, parseable sentences, occasional token loops, no broken tokens — but semantically incoherent. That is the honest state of a 10M model at this token budget.

Usage

The model uses a custom LDT architecture (standard LLaMA block, no special tricks). The safetensors file contains 47 tensors with tied embeddings (lm_head is not stored separately; it shares tok.weight).

To load with a custom model class, you need a small LLaMA-style implementation matching the config above. The training script (train_ldt10m_fixed.py) contains the full architecture definition.

from model import LDT
from tokenizers import Tokenizer

model = LDT.from_pretrained("model.safetensors")  # re-binds the tied lm_head
tok = Tokenizer.from_file("tokenizer.json")

ids = tok.encode("The sun is", add_special_tokens=False).ids
out = model.generate(ids.unsqueeze(0), max_new_tokens=60, temperature=0.8, top_k=40, seed=0)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))

What this is NOT

• Not a 2.6B-token model (that's the target; this is a continued checkpoint well short of it)
• Not a coherent-text model (it produces grammatically-structured word salad at this token budget)
• Not a general-purpose assistant (it's a raw LM, no instruction tuning)
• Not a replacement for anything larger — it's a research checkpoint in a from-scratch training run