Skywalker1910 commited on
Commit
2cc9c22
·
verified ·
1 Parent(s): c6e0519

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +85 -0
README.md ADDED
@@ -0,0 +1,85 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ license: mit
5
+ tags:
6
+ - transformers
7
+ - pytorch
8
+ - language-model
9
+ - from-scratch
10
+ - lora
11
+ - educational
12
+ - gpt
13
+ - decoder-only
14
+ datasets:
15
+ - tiny_shakespeare
16
+ - databricks/databricks-dolly-15k
17
+ pipeline_tag: text-generation
18
+ ---
19
+
20
+ # BB8 — Transformer Language Model Built from Scratch
21
+
22
+ A GPT-style decoder-only Transformer implemented from scratch in PyTorch, plus LoRA fine-tuning experiments on pretrained Qwen models. This is a graduate-level learning project exploring LLM internals across 11 experiments.
23
+
24
+ ## What's in This Repo
25
+
26
+ This Hugging Face repository contains all trained checkpoints, configs, and experiment results from the BB8 project. The source code lives at [github.com/Skywalker1910/BB8](https://github.com/Skywalker1910/BB8).
27
+
28
+ ### Checkpoints
29
+
30
+ | Directory | Type | Parameters | Data | Val PPL | Description |
31
+ |---|---|---:|---|---:|---|
32
+ | `checkpoints/bb8-char-small-v001` | From scratch | 112K | Shakespeare | 5.32 | Historical baseline (had data leak) |
33
+ | `checkpoints/bb8-char-small-v002` | From scratch | 112K | Shakespeare | 5.40 | Corrected baseline |
34
+ | `checkpoints/bb8-char-medium-v006` | From scratch | 833K | Shakespeare | 9.57 | Scaling study (4 layers) |
35
+ | `checkpoints/bb8-char-medium-v006b` | From scratch | 1.2M | Shakespeare | 8.24 | Depth ablation (6 layers) |
36
+ | `checkpoints/bb8-bpe-shakespeare-v007a` | From scratch | 5.1M | Shakespeare | 83.83 | BPE vocab=1000 |
37
+ | `checkpoints/bb8-bpe-shakespeare-v007` | From scratch | 5.6M | Shakespeare | 355.01 | BPE vocab=3000 (best text) |
38
+ | `checkpoints/bb8-bpe-instruct-v003-dev` | From scratch | 5.1M | Dolly 15K | 19.10 | Instruction from scratch (failed) |
39
+ | `checkpoints/bb8-qwen-lora-v004-dev` | Qwen + LoRA | 8.8M trained | Dolly 15K | 7.97 | First pretrained adapter |
40
+ | `checkpoints/bb8-qwen-instruct-v008a` | Qwen-Instruct + LoRA | 4.4M trained | Dolly 15K | 7.81 | Quick test (rank=8) |
41
+ | `checkpoints/bb8-qwen-instruct-v008` | Qwen-Instruct + LoRA | 8.8M trained | Dolly 15K | 7.86 | Best chat model |
42
+ | `checkpoints/bb8-grounded-v005-pilot` | Qwen-Instruct + LoRA | 540K trained | Portfolio | 1.03 | Grounded retrieval pilot |
43
+
44
+ ### Outputs
45
+
46
+ Each `outputs/<run-name>/results.json` contains the full training history, evaluation metrics, config snapshot, and generated text samples.
47
+
48
+ ## Architecture (From Scratch)
49
+
50
+ The core BB8 model is a decoder-only Transformer with:
51
+ - Learned positional embeddings
52
+ - Pre-LayerNorm (normalize before attention)
53
+ - GELU activation in feed-forward layers
54
+ - Weight-tied LM head (shared with token embedding)
55
+ - Cosine LR schedule with linear warmup
56
+ - Custom character, word, and BPE tokenizers
57
+
58
+ All implemented from scratch in PyTorch — no HuggingFace model classes.
59
+
60
+ ## Key Results
61
+
62
+ - **v002** (112K params): Val PPL 5.40 on Shakespeare — learns character patterns
63
+ - **v006b** (1.2M params): 14% PPL improvement over v006 — proves depth > width
64
+ - **v003** (5M params, from scratch on Dolly): All smoke tests wrong — can't skip pretraining
65
+ - **v004** (Qwen + LoRA): Same data as v003, answers correctly — pretraining matters
66
+ - **v008** (Qwen-Instruct + LoRA): Best chat model, all smoke tests correct
67
+
68
+ ## Hardware
69
+
70
+ All experiments ran on a single NVIDIA RTX 4070 Laptop GPU (8 GB VRAM).
71
+
72
+ ## License
73
+
74
+ MIT — code and model weights are free to use, modify, and build upon.
75
+
76
+ ## Citation
77
+
78
+ ```
79
+ @misc{bb8-transformer,
80
+ author = {Aditya More},
81
+ title = {BB8: Transformer Language Model Built from Scratch},
82
+ year = {2026},
83
+ url = {https://github.com/Skywalker1910/BB8}
84
+ }
85
+ ```