|
|
|
|
| --- |
| language: en |
| license: apache-2.0 |
| tags: |
| - text-generation |
| - conversational |
| - gpt |
| - from-scratch |
| - lora |
| - pytorch |
| --- |
| |
| --- |
| language: en |
| license: mit |
| library_name: pytorch |
| tags: |
| - text-generation |
| - conversational |
| - transformer |
| - lora |
| - from-scratch |
| - pytorch |
| pipeline_tag: text-generation |
| --- |
|
|
| # TinyTalk-124M |
|
|
| **TinyTalk-124M** is a 124M-parameter conversational language model, built by fine-tuning a from-scratch pretrained transformer backbone (**ScratchLM-95M**) using LoRA (Low-Rank Adaptation). The entire pipeline — architecture, pretraining, and fine-tuning — was implemented independently in PyTorch, with no pretrained weights sourced from any external model at any stage. |
|
|
| --- |
|
|
| ## Model Details |
|
|
| | | | |
| |---|---| |
| | **Developed by** | Debarun Das | |
| | **Model type** | Decoder-only transformer, LoRA fine-tuned | |
| | **Language** | English | |
| | **License** | MIT | |
| | **Base model** | ScratchLM-95M (trained from scratch) | |
| | **Parameters** | ~124.2M | |
| | **Framework** | PyTorch | |
|
|
| --- |
|
|
| ## Architecture |
|
|
| TinyTalk-124M inherits its backbone from **ScratchLM-95M**, a 12-layer decoder-only transformer using: |
|
|
| - **RoPE (Rotary Positional Embeddings)** for relative position encoding |
| - **SwiGLU activation** in the feed-forward layers (as used in LLaMA, Mistral, and Gemma) |
| - **Causal self-attention** with Flash Attention (SDPA fallback) |
| - **Pre-Norm** transformer blocks with residual connections |
| - **GPT-2 BPE tokenizer** (50,257 vocabulary) |
|
|
| | Parameter | Value | |
| |---|---| |
| | Layers | 12 | |
| | Embedding Dimension | 512 | |
| | Attention Heads | 8 (head_dim = 64) | |
| | FFN Intermediate Dim | 2,730 | |
| | Context Length | 512 tokens | |
| | Base Perplexity (WikiText-103) | 24.40 | |
| |
| --- |
| |
| ## Fine-Tuning Approach |
| |
| TinyTalk-124M was produced using a parameter-efficient LoRA adaptation strategy: |
| |
| - The base model backbone remains **frozen** throughout fine-tuning |
| - Trainable low-rank matrices (**rank = 64, alpha = 128**) are injected into attention (`qkv_proj`, `out_proj`) and feed-forward (`w1`, `w2`, `w3`) projections |
| - Adapter weights are merged directly into the base after training: `W' = W₀ + B·A · (α/r)`, producing a standalone, dependency-free checkpoint |
| - Fine-tuned on **69,220 instruction/response conversational pairs** (ChatAlpaca-style format) |
| - Trained with AdamW, cosine learning rate decay, and mixed precision (AMP) |
| |
| | Metric | Value | |
| |---|---| |
| | Adaptation Method | LoRA (r=64, α=128) | |
| | Fine-Tuning Data | 69,220 instruction/response pairs | |
| | Training Epochs | 5 | |
| | Validation Perplexity | ~25.8 | |
| |
| --- |
| |
| ## Intended Use |
| |
| TinyTalk-124M is designed for: |
| |
| - Instruction-following and conversational text generation |
| - Research and educational demonstrations of from-scratch LLM pretraining and LoRA fine-tuning |
| - Lightweight experimentation on consumer-grade hardware |
| |
| As a 124M-parameter model, it is best suited to general conversational and instructional prompts rather than tasks requiring extensive world knowledge or long-context reasoning — its scale reflects a deliberate choice to make the full architecture-to-deployment pipeline reproducible on a single consumer GPU. |
| |
| --- |
| |
| ## How to Use |
| |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("Debarun12/tinytalk-124m") |
| model = AutoModelForCausalLM.from_pretrained("Debarun12/tinytalk-124m") |
| |
| prompt = "How can I improve my relationship with my family?" |
| inputs = tokenizer(prompt, return_tensors="pt") |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=80, |
| repetition_penalty=3.0, |
| do_sample=True, |
| temperature=0.7 |
| ) |
| |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
| |
| --- |
| |
| ## Training Data |
| |
| The fine-tuning dataset consists of 69,220 cleaned instruction/response conversational pairs in an Alpaca-style format, curated and deduplicated to maintain consistent formatting and answer quality. |
| |
| --- |
| |
| ## Training Infrastructure |
| |
| - **Hardware:** Single consumer GPU |
| - **Framework:** PyTorch (no external training libraries) |
| - **Precision:** Mixed precision (AMP) |
| - **Optimizer:** AdamW with cosine learning rate decay |
| |
| --- |
| |
| ## Citation |
| |
| If you use this model, please cite: |
| |
| ```bibtex |
| @misc{tinytalk124m2026, |
| author = {Debarun Das}, |
| title = {TinyTalk-124M: A Conversational Language Model Built From Scratch}, |
| year = {2026}, |
| publisher = {Hugging Face}, |
| howpublished = {\url{https://huggingface.co/Debarun12}} |
| } |
| ``` |
| |
| --- |
| |
| ## Author |
| |
| **Debarun Das** |
| [GitHub](https://github.com/debarun23) · [Hugging Face](https://huggingface.co/Debarun12) |
