Mo_v0

Mo_v0 is a 73,461,120-parameter Llama-style language model, trained from random initialization on an English TinyStories sample. It is an educational experiment in scaling-law planning, architecture configuration, and language-model pretraining. Its intended use is short English story continuation.

The final model processed 20,480,000 input tokens over 5,000 successful optimizer updates. Its held-out test loss is 2.0470, with perplexity 7.7447, on the experiment's internal TinyStories test split.

Architecture

Setting	Value
Model type	Llama-style decoder-only causal Transformer
Total trainable parameters	73,461,120
Decoder layers	15
Hidden size	640
Query attention heads	10
KV heads (GQA)	5
Head dimension	64
Intermediate size	1728
Vocabulary size	8192
Training context length	256 tokens
Configured maximum context	512 tokens
Input/output embeddings	Tied
Saved parameter dtype	FP32

The model uses rotary position embeddings (RoPE), RMSNorm, SwiGLU, and grouped-query attention. Attention and MLP projection biases are disabled. Attention dropout is 0.0, and RMSNorm epsilon is 1e-5.

For tied embeddings, the parameter calculation used by the lab is:

head_dim = H / A = 640 / 10 = 64
per_layer = 2*H^2 + 2*H*K*head_dim + 3*H*I + 2*H
          = 4,547,840 parameters
N = V*H + L*per_layer + H
  = 73,461,120 parameters

Reasoning behind the configuration

• The classroom scaling calculation suggested a model near 75M parameters. With an 8,192-token vocabulary, a hidden size of 640, an intermediate size of 1,728, and 15 layers, the rounded architecture contains 73.46M parameters and meets the assignment's 60–100M requirement.
• The architecture allocates capacity to wider blocks and fewer layers than the lab's 24-layer, width-512 reference. A controlled comparison was not performed, so improved speed or quality over that reference is not claimed.
• Ten query heads share five KV heads. This reduces KV projection parameters and KV-cache storage relative to ten KV heads at the same depth and width.
• A compact vocabulary and shared input/output embeddings keep embedding parameters within the overall budget.
• Training uses 256-token blocks to keep the initial experiment manageable. The 512-token configuration limit does not establish performance on contexts longer than the 256 tokens used in training.

Dataset and preprocessing

• Source: roneneldan/TinyStories.
• Language: English.
• Dataset licence: CDLA-Sharing-1.0, as listed on the dataset card.
• Content: short synthetic stories generated with GPT-3.5 and GPT-4, using a restricted vocabulary.
• Sample: 100,000 nonempty, deduplicated stories sampled from the dataset's published training split. The published validation split was not used.

The notebook streams the training split, applies buffer shuffling with seed 42 and a 10,000-document buffer, normalizes whitespace, and removes exact duplicates using SHA-256 hashes. It then splits whole documents before training the tokenizer or packing tokens.

Internal split	Documents	Fraction
Training	90,000	90%
Validation	5,000	5%
Test	5,000	5%

A custom byte-level BPE tokenizer was fitted using training documents only. It has 8,192 entries, including PAD, BOS, EOS, and UNK. EOS is appended to each story during packing; BOS is not automatically inserted.

Stories are concatenated separately within each split and packed into non-overlapping 256-token blocks. The final incomplete block of each split is discarded. Exact normalized stories do not overlap across splits; near-duplicate stories are not detected by this method. Standard causal attention can cross EOS-separated document boundaries inside a block.

The downloaded sample's SHA-256 fingerprint is:

86725db033dd757ef19af2bc4156f4a85309ccd59d23542553ecb0380bf5d35c

The experiment record does not pin a dataset revision, so the fingerprint documents the downloaded sample but does not by itself guarantee a future download will reproduce it.

Scaling-law calculations and token budget

The classroom planning assumptions were D = 20N and C ≈ 6ND, where N is trainable parameters and D is processed training tokens. Substituting gives C ≈ 120N², so N ≈ sqrt(C / 120).

With the lab's illustrative compute budget of 6.75 × 10^17 FLOPs, the initial planning estimate is 75,000,000 parameters and 1,500,000,000 tokens.

Quantity	Value
Initial parameter target	75,000,000
Actual configured parameters	73,461,120
Updated token target at 20 tokens/parameter	1,469,222,400
Effective batch size	16 sequences
Input tokens per optimizer update	4,096
Practical plan	5,000 updates / 20,480,000 input tokens
Actual successful updates	5,000
Actual processed input tokens	20,480,000
Actual scored next-token targets	20,400,000
Coverage of the updated 20N target	1.3939%
Approximate compute for the actual run, 6ND	9.0269e+15 FLOPs

Each effective batch contains 4 microbatch sequences × 4 accumulation steps × 256 tokens = 4,096 input tokens. Causal loss scores 255 targets per sequence, giving 4,080 scored targets per update and 20,400,000 across the run.

The actual run is a smaller practical experiment and reaches about 1.39% of the 20N planning target. The 20-token ratio is a classroom rule of thumb, rather than a guarantee of optimal training for this model. The approximate FLOPs calculation omits sequence-attention costs, checkpoint recomputation, and implementation overhead. Processed-token counts can include repeated data; the record does not provide the number of unique packed training-token positions or the number of dataset passes.

Training setup

Setting	Value
Hardware / GPU model	Single NVIDIA Tesla T4 GPU
Training duration	1.8164 hours (1 hour, 48 minutes, 59 seconds)
Initialization	Random weights
Objective	Next-token cross-entropy
Optimizer	AdamW, following the supplied notebook
Peak learning rate	0.0003
Schedule	Cosine decay with 250 warmup updates
Adam betas	(0.9, 0.95)
Weight decay	0.1
Microbatch size	4 sequences
Gradient accumulation	4 microbatches/update
Effective batch size	16 sequences / 4,096 input tokens
Training block length	256 tokens
Successful optimizer updates	5,000
Random seed	42
Saved parameter dtype	FP32

The supplied training loop uses SDPA attention, gradient checkpointing, gradient clipping at norm 1.0, and device-dependent mixed precision. The precise autocast dtype was not captured in the uploaded experiment record. The GPU name and training duration were supplied from the completed training runtime; all other recorded results above come from experiment.json. The duration is training-loop wall time, including periodic evaluations and checkpoint saves, and excludes initial data/tokenizer preparation and the final evaluation after the loop.

Library	Recorded version
PyTorch	2.14.1+cu130
Transformers	4.46.3
Datasets	3.1.0
Tokenizers	0.20.3

Evaluation

Loss is mean next-token cross-entropy in nats. Perplexity is exp(loss). The model was evaluated on non-overlapping blocks of the internal held-out splits defined above.

Evaluation	Phase	Loss	Perplexity	Scored targets
Fixed validation subset	Before training	9.106078	9009.887047	20,400
Full internal validation split	After training	2.036329	7.662425	1,062,330
Full internal test split	After training	2.047011	7.744719	1,062,075

The initial subset and final full-validation rows cover different evaluation populations. For a matched before/after subset comparison, use the validation history in history.json. The final full validation and test results are close, but this does not establish performance beyond this corpus. Perplexities depend on the tokenizer, dataset, context length, and evaluation procedure.

Training and validation loss figure

The training curve records the last effective training batch at each evaluation boundary. The validation curve evaluates the same fixed subset of up to 20 batches. The table above reports the final full validation and test evaluations separately.

Example outputs and limitations

The notebook used the prompt Once upon a time, a little robot found a book., greedy decoding (do_sample=False), and max_new_tokens=60 for both samples. The recorded text includes the prompt.

Before training:

Once upon a time, a little robot found a book.oupoupoup Abi Abi Abi Abi Abi Abi Abi Abi Abi Abi Abi Abi noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise

After training:

Once upon a time, a little robot found a book. The book was very pretty and had a picture of a princess on it. The robot was very happy and played with the book all day long. One day, the robot saw a little girl who was sad. The robot wanted to help the little girl. The robot asked the little girl, "Can

The untrained sample repeats fragments such as Abi and noise. After training, the sample introduces a book, a princess picture, and a sad girl in a coherent short scene. Its wording remains simple and repeatedly returns to the robot and the book. The saved after-training sample ends mid-dialogue at the generation-length limit, so it does not demonstrate a completed story.

Additional limitations:

• The run processed far fewer tokens than the 20N planning estimate.
• Training data consists of English synthetic stories; Arabic and broader multilingual performance were not evaluated.
• The model was pretrained on story text without instruction fine-tuning or a conversational chat template.
• Only 256-token contexts were used in training. Longer-context quality has not been established.
• These held-out scores and one before/after prompt do not establish factual accuracy, general reasoning, long-story consistency, or performance outside the training domain.
• Whitespace-normalized exact deduplication does not remove all possible near duplicates.

Usage

The recorded training stack used Transformers 4.46.3 and Tokenizers 0.20.3. The following example loads the model on CPU and explicitly enables the inference cache. It keeps the prompt and requested completion within 256 tokens.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "vMo04/Mo_v0"
tokenizer = AutoTokenizer.from_pretrained(model_id, token=False)
model = AutoModelForCausalLM.from_pretrained(model_id, token=False).eval()

prompt = "Once upon a time, a little robot found a book."
inputs = tokenizer(
    prompt,
    return_tensors="pt",
    add_special_tokens=False,
    return_token_type_ids=False,
    truncation=True,
    max_length=196,
)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=60,
        do_sample=False,
        use_cache=True,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Experiment files

• experiment.json: recorded architecture, splits, token counts, hyperparameters, evaluation scores, examples, and versions.
• history.json: training and fixed-subset validation history.
• loss_curve.png: training and validation loss figure.
• model.safetensors, config.json, and tokenizer files: inference artifacts.

References

• Hoffmann et al., Training Compute-Optimal Large Language Models.
• TinyStories dataset and dataset licence.
• Eldan and Li, TinyStories: How Small Can Language Models Be and Still Speak Coherent English?.