workshop-pretraining-v2

A small (123.6M parameters) decoder-only Transformer language model, pretrained from scratch for only 100 optimizer steps as part of a hands-on LLM pretraining workshop.

This is an educational checkpoint, not a usable language model. It was trained on about 52 million tokens (roughly 0.5% of the 10B-token plan in the workshop notebook), so it produces mostly frequent words with no coherent meaning. See Limitations.

Model details

Item	Value
Architecture	Decoder-only Transformer (GPT-2 sized, with modern components)
Parameters	123,587,328
Layers / hidden size / heads	12 / 768 / 12 (head dim 64)
Context length	1024 tokens
Vocabulary	50,304 (GPT-2 BPE with 50,257 tokens, padded to a multiple of 64)
Feed-forward	4x expansion, ReLU² activation, no biases
Normalization	RMSNorm, pre-norm
Positional encoding	Rotary embeddings (RoPE, theta = 10000)
Output head	Tied with the input embedding, tanh logit soft-capping (cap = 30)
Dropout	0.1 (training only)
Tokenizer	tiktoken GPT-2 encoding (`<

This is a custom architecture, not GPT2LMHeadModel. Loading it requires trust_remote_code=True. The model code is in configuration_gpt2workshop.py and modeling_gpt2workshop.py in this repository.

Training

Item	Value
Data	HuggingFaceFW/fineweb-edu, sample-10BT subset, streamed and tokenized into uint16 shards
Training length	100 optimizer steps
Tokens per step	524,288 (micro-batch 4 x 1024 tokens x 128 gradient-accumulation steps)
Tokens seen	about 52.4M
Optimizer	AdamW, betas (0.9, 0.95), weight decay 0.1 (not applied to norms and 1-D parameters), fused
Learning rate	Peak 6e-4, linear warmup for 10 steps, then cosine decay to 6e-5
Gradient clipping	1.0
Precision	bfloat16 autocast
Hardware	1x NVIDIA GeForce RTX 3080 Laptop GPU (8 GB), about 93 minutes, about 9,000 tokens/s
Framework	PyTorch (no torch.compile)

Results

Metric	Value
Training loss, step 10	8.58
Training loss, step 50	7.28
Training loss, step 100	6.97
LAMBADA accuracy (steps 50 and 100)	0.0000

For reference, a uniform random guess over this vocabulary gives a loss of about 10.8.

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "kihyounghan/workshop-pretraining-v2"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).to(device)
model.eval()

prompt = "The best way to learn programming is"
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output_ids = model.generate(
        input_ids, max_new_tokens=50, do_sample=True, temperature=0.8, top_k=50
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

trust_remote_code=True runs the Python files stored in this repository. Review them first, or pin a specific commit with revision="<commit hash>".

Example output

Prompt: The theory of relativity states that

The theory of relativity states that you, they the research and the high of the following of ...

The model has learned which words are frequent in English, but not grammar or meaning yet.

Limitations

• Trained for 100 steps only, so outputs are repetitive, ungrammatical and unrelated to the prompt.
• Not evaluated for safety, factuality or bias. It is not suitable for any real application.
• Trained on English web text only. The training data may contain biases and errors.
• Intended for learning and for testing the pretraining, export and upload pipeline.

한국어 요약

LLM 사전학습(pretraining) 워크숍 실습용으로 처음부터 학습한 약 1.24억 파라미터 모델입니다. 100 스텝(약 5,200만 토큰)만 학습해서 실제로 쓸 수 있는 수준이 아니며, 학습 → 저장 → Hugging Face 업로드 과정을 확인하기 위한 예제 체크포인트입니다.