File size: 6,875 Bytes
5557497 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 | ---
language:
- en
license: mit
tags:
- gpt2
- text-generation
- from-scratch
- educational
pipeline_tag: text-generation
---
<div align="center">
# ❄️ Winter‑Frost
**A small language model, built entirely from scratch — tokenizer, pretraining, and instruction‑tuning — on a single free GPU.**
</div>
---
Winter‑Frost is not built to compete with production models. It's a **from‑first‑principles build**: every stage of the pipeline — tokenization, pretraining, instruction tuning, deployment — was implemented and run from scratch on free‑tier hardware, as a hands‑on way to actually understand how LLMs work under the hood.
## 📋 Model summary
| | |
|---|---|
| 🧠 **Architecture** | GPT‑2 (decoder‑only transformer) |
| 🔢 **Parameters** | ~111M |
| ✂️ **Tokenizer** | Custom Byte‑Level BPE, vocab size 32,000 |
| 📏 **Context length** | 1,024 tokens |
| 📚 **Pretraining data** | ~4GB custom text corpus |
| 🎯 **Instruction tuning** | One epoch over the Alpaca dataset (~51,760 examples) |
| ⚙️ **Training hardware** | Single free‑tier NVIDIA T4 GPU (Google Colab) |
| 🎛️ **Precision** | fp16 mixed precision |
## 🛠️ How it was built
```
┌─────────────────┐ ┌──────────────────┐ ┌───────────────────┐ ┌─────────────┐
│ 1. Tokenizer │ ──▶ │ 2. Pretraining │ ──▶ │ 3. Instruction │ ──▶ │ 4. Deploy │
│ Byte‑Level BPE │ │ Raw next‑token │ │ tuning │ │ Push to │
│ trained from │ │ prediction on │ │ Alpaca dataset, │ │ Hugging │
│ scratch, 32k │ │ ~4GB corpus │ │ teaches Q→A │ │ Face Hub │
│ vocab │ │ (GPT‑2, 111M) │ │ format │ │ │
└─────────────────┘ └──────────────────┘ └───────────────────┘ └─────────────┘
```
**1. Tokenizer** — A Byte‑Level BPE tokenizer was trained from scratch on the target corpus, rather than reusing an existing model's vocabulary.
**2. Pretraining (Phase 1)** — The tokenizer streamed-encoded the ~4GB corpus into binary token files, feeding a `GPT2LMHeadModel` (768 hidden size, 12 layers, 12 heads) trained on next‑token prediction. This teaches raw language patterns — grammar, style, associations — but not instruction‑following.
**3. Instruction tuning (Phase 2)** — The pretrained checkpoint was fine‑tuned on the Alpaca instruction dataset via `trl`'s `SFTTrainer`, teaching the model to *respond* to a prompt instead of just continuing it.
**4. Deployment** — Final weights and tokenizer were pushed straight to the Hugging Face Hub from the training environment.
## 🚀 How to use it
### Quick start
```bash
pip install torch transformers
```
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_dir = "brucoder/winter-frost"
tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(model_dir)
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.eval()
prompt = """Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
What is a computer?
### Response:
"""
inputs = tokenizer(prompt, return_tensors="pt").to(device)
output = model.generate(
**inputs,
max_new_tokens=200,
do_sample=True,
temperature=0.7,
top_p=0.9,
repetition_penalty=1.15,
pad_token_id=tokenizer.pad_token_id,
)
response = tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response.strip())
```
### 📝 Prompt format
The model expects the Alpaca‑style instruction template it was fine‑tuned on:
```
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{your question or task here}
### Response:
```
Prompts that don't follow this format will still generate text, but responses tend to drift off‑task.
### 🎛️ Generation tips
| Setting | Effect |
|---|---|
| `repetition_penalty 1.2–1.4` | Reduces looping / repeated phrases |
| `temperature 0.5` (lower) | More focused, less rambling output |
| `do_sample=False` | Deterministic, greedy (most‑likely‑token) output |
## ⚠️ Limitations — please read before using
This model is small, and was trained on a small amount of data relative to production LLMs. Please calibrate expectations accordingly:
- 🧩 **World knowledge is limited and unreliable** — it only knows what appeared in its ~4GB training corpus plus Alpaca's instruction patterns. It will **confidently generate plausible‑sounding but incorrect information.**
- ➗ **Math and code are not reliable** — training data was not focused on arithmetic or programming.
- 🌀 **Topic drift** — on ambiguous or open‑ended prompts, the model may default to whatever topics were over‑represented in training, rather than staying on‑topic.
- 🎯 **Not instruction‑perfect** — it recognizes the instruction → response *format* but doesn't always answer the actual question asked.
- 🚫 **Do not use for factual, medical, legal, financial, or safety‑critical purposes.**
## 🎓 Intended use
Educational and experimental use — exploring from‑scratch LLM training, tokenizer behavior, small‑model generation quality, and the practical realities of training on constrained (free‑tier) hardware.
## 🔗 Related
A larger follow‑up model, **[`winter-frost-1-pro`](https://huggingface.co/brucoder/winter-frost-1-pro)** (~774M parameters, broader pretraining corpus via streaming), is trained under the same from‑scratch philosophy on the same free‑tier hardware. It's meaningfully bigger and broader, but shares the same category of limitations described above.
## 🙏 Acknowledgements
Built as a solo learning project using Hugging Face `transformers`, `datasets`, `trl`, `tokenizers`, and Google Colab's free GPU tier.
<div align="center">
*Made with curiosity, patience, and a lot of Colab reconnects.* ❄️
</div> |