| --- |
| language: |
| - en |
| license: mit |
| tags: |
| - gpt2 |
| - text-generation |
| - from-scratch |
| - educational |
| pipeline_tag: text-generation |
| --- |
| |
| <div align="center"> |
|
|
| # ❄️ Winter‑Frost |
|
|
| **A small language model, built entirely from scratch — tokenizer, pretraining, and instruction‑tuning — on a single free GPU.** |
|
|
|  |
|  |
|  |
|  |
|  |
|
|
| </div> |
|
|
| --- |
|
|
| Winter‑Frost is not built to compete with production models. It's a **from‑first‑principles build**: every stage of the pipeline — tokenization, pretraining, instruction tuning, deployment — was implemented and run from scratch on free‑tier hardware, as a hands‑on way to actually understand how LLMs work under the hood. |
|
|
| ## 📋 Model summary |
|
|
| | | | |
| |---|---| |
| | 🧠 **Architecture** | GPT‑2 (decoder‑only transformer) | |
| | 🔢 **Parameters** | ~111M | |
| | ✂️ **Tokenizer** | Custom Byte‑Level BPE, vocab size 32,000 | |
| | 📏 **Context length** | 1,024 tokens | |
| | 📚 **Pretraining data** | ~4GB custom text corpus | |
| | 🎯 **Instruction tuning** | One epoch over the Alpaca dataset (~51,760 examples) | |
| | ⚙️ **Training hardware** | Single free‑tier NVIDIA T4 GPU (Google Colab) | |
| | 🎛️ **Precision** | fp16 mixed precision | |
|
|
| ## 🛠️ How it was built |
|
|
| ``` |
| ┌─────────────────┐ ┌──────────────────┐ ┌───────────────────┐ ┌─────────────┐ |
| │ 1. Tokenizer │ ──▶ │ 2. Pretraining │ ──▶ │ 3. Instruction │ ──▶ │ 4. Deploy │ |
| │ Byte‑Level BPE │ │ Raw next‑token │ │ tuning │ │ Push to │ |
| │ trained from │ │ prediction on │ │ Alpaca dataset, │ │ Hugging │ |
| │ scratch, 32k │ │ ~4GB corpus │ │ teaches Q→A │ │ Face Hub │ |
| │ vocab │ │ (GPT‑2, 111M) │ │ format │ │ │ |
| └─────────────────┘ └──────────────────┘ └───────────────────┘ └─────────────┘ |
| ``` |
|
|
| **1. Tokenizer** — A Byte‑Level BPE tokenizer was trained from scratch on the target corpus, rather than reusing an existing model's vocabulary. |
|
|
| **2. Pretraining (Phase 1)** — The tokenizer streamed-encoded the ~4GB corpus into binary token files, feeding a `GPT2LMHeadModel` (768 hidden size, 12 layers, 12 heads) trained on next‑token prediction. This teaches raw language patterns — grammar, style, associations — but not instruction‑following. |
|
|
| **3. Instruction tuning (Phase 2)** — The pretrained checkpoint was fine‑tuned on the Alpaca instruction dataset via `trl`'s `SFTTrainer`, teaching the model to *respond* to a prompt instead of just continuing it. |
|
|
| **4. Deployment** — Final weights and tokenizer were pushed straight to the Hugging Face Hub from the training environment. |
|
|
| ## 🚀 How to use it |
|
|
| ### Quick start |
|
|
| ```bash |
| pip install torch transformers |
| ``` |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| import torch |
| |
| model_dir = "brucoder/winter-frost" |
| tokenizer = AutoTokenizer.from_pretrained(model_dir) |
| model = AutoModelForCausalLM.from_pretrained(model_dir) |
| |
| device = "cuda" if torch.cuda.is_available() else "cpu" |
| model.to(device) |
| model.eval() |
| |
| prompt = """Below is an instruction that describes a task. Write a response that appropriately completes the request. |
| |
| ### Instruction: |
| What is a computer? |
| |
| ### Response: |
| """ |
| |
| inputs = tokenizer(prompt, return_tensors="pt").to(device) |
| output = model.generate( |
| **inputs, |
| max_new_tokens=200, |
| do_sample=True, |
| temperature=0.7, |
| top_p=0.9, |
| repetition_penalty=1.15, |
| pad_token_id=tokenizer.pad_token_id, |
| ) |
| response = tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True) |
| print(response.strip()) |
| ``` |
|
|
| ### 📝 Prompt format |
|
|
| The model expects the Alpaca‑style instruction template it was fine‑tuned on: |
|
|
| ``` |
| Below is an instruction that describes a task. Write a response that appropriately completes the request. |
| |
| ### Instruction: |
| {your question or task here} |
| |
| ### Response: |
| ``` |
|
|
| Prompts that don't follow this format will still generate text, but responses tend to drift off‑task. |
|
|
| ### 🎛️ Generation tips |
|
|
| | Setting | Effect | |
| |---|---| |
| | `repetition_penalty 1.2–1.4` | Reduces looping / repeated phrases | |
| | `temperature 0.5` (lower) | More focused, less rambling output | |
| | `do_sample=False` | Deterministic, greedy (most‑likely‑token) output | |
|
|
| ## ⚠️ Limitations — please read before using |
|
|
| This model is small, and was trained on a small amount of data relative to production LLMs. Please calibrate expectations accordingly: |
|
|
| - 🧩 **World knowledge is limited and unreliable** — it only knows what appeared in its ~4GB training corpus plus Alpaca's instruction patterns. It will **confidently generate plausible‑sounding but incorrect information.** |
| - ➗ **Math and code are not reliable** — training data was not focused on arithmetic or programming. |
| - 🌀 **Topic drift** — on ambiguous or open‑ended prompts, the model may default to whatever topics were over‑represented in training, rather than staying on‑topic. |
| - 🎯 **Not instruction‑perfect** — it recognizes the instruction → response *format* but doesn't always answer the actual question asked. |
| - 🚫 **Do not use for factual, medical, legal, financial, or safety‑critical purposes.** |
|
|
| ## 🎓 Intended use |
|
|
| Educational and experimental use — exploring from‑scratch LLM training, tokenizer behavior, small‑model generation quality, and the practical realities of training on constrained (free‑tier) hardware. |
|
|
| ## 🔗 Related |
|
|
| A larger follow‑up model, **[`winter-frost-1-pro`](https://huggingface.co/brucoder/winter-frost-1-pro)** (~774M parameters, broader pretraining corpus via streaming), is trained under the same from‑scratch philosophy on the same free‑tier hardware. It's meaningfully bigger and broader, but shares the same category of limitations described above. |
|
|
| ## 🙏 Acknowledgements |
|
|
| Built as a solo learning project using Hugging Face `transformers`, `datasets`, `trl`, `tokenizers`, and Google Colab's free GPU tier. |
|
|
| <div align="center"> |
|
|
| *Made with curiosity, patience, and a lot of Colab reconnects.* ❄️ |
|
|
| </div> |
