README.md

---
license: apache-2.0
pipeline_tag: text-generation
language: en
tags:
  - tiny
  - tiny-lm
  - tiny-model
  - slm
  - SLM
  - small-language-model
  - from-scratch
  - llama
datasets:
  - HuggingFaceFW/fineweb-edu
  - allenai/dclm-baseline
metrics:
  - perplexity
---

# LDT-10M

A 10.28M-parameter LLaMA-style language model, trained from scratch on FineWeb-Edu + DCLM.

**Requested by [DedeProGames](https://huggingface.co/DedeProGames) on the [model-requests board](https://huggingface.co/spaces/Compactbot/model-requests) (#12).**

## Architecture

| Parameter | Value |
|---|---|
| Params | **10,284,480** |
| Layers | 5 |
| d_model | 320 |
| Heads | 5 (MHA, GQA not used at this scale) |
| FFN dim | 896 (SwiGLU) |
| Vocab | 12,288 (gollem BPE) |
| Context | 512 |
| Embeddings | Tied (lm_head → tok.weight) |
| Norm | RMSNorm (eps 1e-5) |
| Attention | RoPE + causal SDPA |
| Dtype | float32 |

Standard LLaMA block: RMSNorm → MHA (RoPE) → residual → RMSNorm → SwiGLU FFN → residual.

## Training

| | v1 (first ckpt) | v4 | v7 | **v8 (current)** |
|---|---|---|---|---|
| Steps | 4,000 | 16,000 | 30,000 | **79,375** |
| Batch size | 64 | 64 | 64 | 64 |
| Seq length | 512 | 512 | 512 | 512 |
| Cumulative tokens | ~131M | ~308M | ~983M | **2.60B** |
| LR | 3e-4 → 3e-5 (cosine) | 1e-4 → 1e-5 (cosine, fresh optimizer) | 1e-4 → 1e-5 (cosine, fresh optimizer) | 1e-4 → 1e-5 (cosine, fresh optimizer) |
| Data | FineWeb-Edu + DCLM | + FineWeb-Edu + DCLM | + FineWeb-Edu + DCLM (continued) | + FineWeb-Edu + DCLM (continued) |
| Hardware | RTX 5090 (32 GB) | RTX 5090 (32 GB) | RTX 5090 (32 GB) | RTX 5090 (32 GB) |
| Final val loss | 4.6020 (ppl 99.68) | 3.8943 (ppl 49.12) | 3.6892 (ppl 40.01) | **3.63 (ppl 37.8)** |

v8 is the final checkpoint of the from-scratch run: 79,375 steps × 64 × 512 = **2,600,960,000 tokens (2.60B)**, which **hits DedeProGames' 2.6B-token target**. The val loss is read from the final checkpoint's recorded `val_loss` (3.63); the earlier columns' token counts are step-derived (steps × 64 × 512).

### ⚠️ Honest caveat: token target hit, but still incoherent and loop-prone

v8 reaches the requested 2.6B-token budget. The val loss (3.63) is well below the 7.38 unigram floor, so the model genuinely uses context; the improvement 4.60 → 3.89 → 3.69 → 3.63 is real but **marginal in the last stage** (3.69 → 3.63).

However, the 40-sample generation sweep below shows that **more tokens did not buy coherence — it bought more token loops.** The mean 4-gram loop fraction went **up** from v7 (0.167) to v8 (0.219), and the degenerate-sample rate (loop > 0.30) went from 0/40 to **14/40**. At this scale the model has learned the surface shape of English — real words, parseable sentence frames — but the prose is still **semantically incoherent** (word salad) and increasingly prone to hard repetition loops. This is the honest state of a 10M model at 2.6B tokens: the token budget is spent, and the quality ceiling of a 10M-param LM is what it is.

## Eval (40 samples: 8 prompts × 5 seeds, temp 0.8, top-k 40, 160 new tokens)

| Metric | v1 | v4 | v7 | **v8** |
|---|---|---|---|---|
| val loss | 4.6020 | 3.8943 | 3.6892 | **3.63** |
| perplexity | 99.68 | 49.12 | 40.01 | **37.8** |
| Below unigram floor (7.38)? | Yes | Yes | Yes | Yes |
| mean 4-gram loop fraction | n/a | ~0.11 | 0.167 | **0.219** |
| Degenerate samples (loop > 0.30) | n/a | 0/40 | 0/40 | **14/40** |
| Semantically coherent? | No | No | No | No (word salad, more loops) |

### Sample outputs (v8, real generation from the published weights — not hand-picked)

> "Once upon a time, the time of the Lord, the Lord, the Lord, the Lord, the Lord, the Lord, the Lord, the Lord, the Lord, the Lord, the Lord, the Lord, the Lord, the Lord, the Lord, the Lord, the Lord, the Lord, the Lord" (greedy)

> "She walked down the street. When she said she said she said she said she said she said she said she said she said she said she wa…" (temp 0.8, seed 0, loop frac 0.595)

> "A long time ago in a galaxy far far away from an galaxy that is almost no point from the distant distant galaxy called a…" (temp 0.8, seed 2, loop frac 0.587)

These are real outputs from the v8 weights. The text is grammatically structured — real words, parseable sentence frames — but semantically incoherent and loop-prone. That is the honest state of a 10M model at this scale.

## Usage

The model uses a custom `LDT` architecture (standard LLaMA block, no special tricks). The safetensors file contains 47 tensors with tied embeddings (lm_head is not stored separately; it shares `tok.weight`).

To load with a custom model class, you need a small LLaMA-style implementation matching the config above. The training script (`train_ldt10m_v8.py`) contains the full architecture definition.

```python
from model import LDT
from tokenizers import Tokenizer

model = LDT.from_pretrained("model.safetensors")  # re-binds the tied lm_head
tok = Tokenizer.from_file("tokenizer.json")

ids = tok.encode("The sun is", add_special_tokens=False).ids
out = model.generate(ids.unsqueeze(0), max_new_tokens=60, temperature=0.8, top_k=40, seed=0)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))
```

## What this is NOT

- Not a coherent-text model (it produces grammatically-structured word salad with frequent token loops at this scale)
- Not a general-purpose assistant (it's a raw LM, no instruction tuning)
- Not a replacement for anything larger — it is the final from-scratch checkpoint of a 10M-param run that hit its 2.6B-token budget