Transformer architectures: from BERT to diffusion
A 15-lesson course: each architecture written from scratch, checked against the real weights, then put to work. Free on Kaggle and Colab.
Fill-Mask • 0.1B • Updated • 32.8M • • 3.09kNote Lesson 1 · BERT from scratch with Google's weights in our own encoder, what its heads look at, and a personal-data detector for long documents. https://blog.korolevalab.com/blog/2026/transformer-architectures-01-bert/
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Paper • 1810.04805 • Published • 32Note The paper behind lesson 1.
answerdotai/ModernBERT-base
Fill-Mask • 0.1B • Updated • 2.77M • 1.16kNote Lesson 2 · From RoBERTa to ModernBERT: RoBERTa's recipe, DeBERTa, ELECTRA, ModernBERT's local and global attention, three generations on one task. https://blog.korolevalab.com/blog/2026/transformer-architectures-02-modernbert/
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
Paper • 2412.13663 • Published • 170Note The paper behind lesson 2.
sentence-transformers/all-MiniLM-L6-v2
Sentence Similarity • 22.7M • Updated • 166M • • 6.16kNote Lesson 3 · Embeddings and search: BM25 from scratch, in-batch negatives, a cross-encoder reranker, ColBERT's MaxSim, a hybrid search over 58,000 ML papers. https://blog.korolevalab.com/blog/2026/transformer-architectures-03-search/
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Paper • 1908.10084 • Published • 19Note The paper behind lesson 3.
openai-community/gpt2
Text Generation • 0.1B • Updated • 33.8M • 4.23kNote Lesson 4 · GPT-2 with OpenAI's weights in our code, the KV cache, the logit lens, and a chess GPT trained on Lichess games with the board found by a linear probe. https://blog.korolevalab.com/blog/2026/transformer-architectures-04-gpt/
google-t5/t5-small
Translation • 60.5M • Updated • 17.3M • • 702Note Lesson 5 · T5 and BART: T5 in our code with Google's weights, cross-attention alignments, span corruption, and a T5 that writes SQL for Spider, checked on SQLite. https://blog.korolevalab.com/blog/2026/transformer-architectures-05-t5/
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Paper • 1910.10683 • Published • 19Note The paper behind lesson 5.
Qwen/Qwen3-0.6B
Text Generation • 0.8B • Updated • 15.2M • 1.93kNote Lesson 6 · Modern LLM architecture: fourteen configs side by side, grouped-query attention, QK-norm, Qwen3 in PyTorch with the official weights, LoRA for voice commands to JSON. https://blog.korolevalab.com/blog/2026/transformer-architectures-06-modern/
Qwen3 Technical Report
Paper • 2505.09388 • Published • 348Note The report behind lesson 6.
allenai/OLMoE-1B-7B-0924
Text Generation • 7B • Updated • 41.8k • 160Note Lesson 7 · Mixture of experts: the router written four ways, OLMoE with its official weights, which experts English, Python, maths and Russian use, an expert cache. https://blog.korolevalab.com/blog/2026/transformer-architectures-07-moe/
OLMoE: Open Mixture-of-Experts Language Models
Paper • 2409.02060 • Published • 82Note The paper behind lesson 7.
openbmb/MiniCPM3-4B
Text Generation • Updated • 8.09k • 422Note Lesson 8 · The KV cache budget: multi-head latent attention in both forms, MiniCPM3-4B with its official weights, a latent cache on a T4, a quantized cache. https://blog.korolevalab.com/blog/2026/transformer-architectures-08-kv/
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
Paper • 2405.04434 • Published • 30Note DeepSeek-V2 introduced the latent attention of lesson 8.
TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
Text Generation • 1B • Updated • 42.3k • 194Note Lesson 9 · Long context: TinyLlama's cliff at token 2,049, position interpolation, NTK and YaRN from scratch, FlashAttention from the online softmax. https://blog.korolevalab.com/blog/2026/transformer-architectures-09-long/
YaRN: Efficient Context Window Extension of Large Language Models
Paper • 2309.00071 • Published • 84Note The paper behind lesson 9.
state-spaces/mamba-130m-hf
Text Generation • 0.1B • Updated • 50.2k • 76Note Lesson 10 · Mamba: a state space model from its equation, the selective scan with mamba-130m's weights, Mamba-2 as masked attention, hybrid layouts. https://blog.korolevalab.com/blog/2026/transformer-architectures-10-mamba/
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Paper • 2312.00752 • Published • 152Note The paper behind lesson 10.
google/vit-base-patch16-224
Image Classification • 86.6M • Updated • 15.2M • • 1.06kNote Lesson 11 · The Vision Transformer from scratch with Google's weights, where its heads look, DINO's heads finding objects, ViT against a CNN on satellite images. https://blog.korolevalab.com/blog/2026/transformer-architectures-11-vit/
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Paper • 2010.11929 • Published • 26Note The paper behind lesson 11.
openai/clip-vit-base-patch32
Zero-Shot Image Classification • Updated • 17M • 1.29kNote Lesson 12 · CLIP from scratch with OpenAI's weights, zero-shot geolocation over 211 countries, search over COCO, the modality gap, a word on a photo that changes the answer. https://blog.korolevalab.com/blog/2026/transformer-architectures-12-clip/
google/siglip-base-patch16-224
Zero-Shot Image Classification • 0.2B • Updated • 927k • 106Note Lesson 12 · SigLIP's sigmoid loss and attention pooling, and the two losses trained side by side. https://blog.korolevalab.com/blog/2026/transformer-architectures-12-clip/
Learning Transferable Visual Models From Natural Language Supervision
Paper • 2103.00020 • Published • 23Note The CLIP paper, behind lesson 12.
Sigmoid Loss for Language Image Pre-Training
Paper • 2303.15343 • Published • 12Note The SigLIP paper, behind lesson 12.