---
language: en
license: apache-2.0
tags:
- text-generation
- conversational
- gpt
- from-scratch
- lora
- pytorch
---

 ---
language: en
license: mit
library_name: pytorch
tags:
  - text-generation
  - conversational
  - transformer
  - lora
  - from-scratch
  - pytorch
pipeline_tag: text-generation
---

# TinyTalk-124M

**TinyTalk-124M** is a 124M-parameter conversational language model, built by fine-tuning a from-scratch pretrained transformer backbone (**ScratchLM-95M**) using LoRA (Low-Rank Adaptation). The entire pipeline — architecture, pretraining, and fine-tuning — was implemented independently in PyTorch, with no pretrained weights sourced from any external model at any stage.

---

## Model Details

| | |
|---|---|
| **Developed by** | Debarun Das |
| **Model type** | Decoder-only transformer, LoRA fine-tuned |
| **Language** | English |
| **License** | MIT |
| **Base model** | ScratchLM-95M (trained from scratch) |
| **Parameters** | ~124.2M |
| **Framework** | PyTorch |

---

## Architecture

TinyTalk-124M inherits its backbone from **ScratchLM-95M**, a 12-layer decoder-only transformer using:

- **RoPE (Rotary Positional Embeddings)** for relative position encoding
- **SwiGLU activation** in the feed-forward layers (as used in LLaMA, Mistral, and Gemma)
- **Causal self-attention** with Flash Attention (SDPA fallback)
- **Pre-Norm** transformer blocks with residual connections
- **GPT-2 BPE tokenizer** (50,257 vocabulary)

| Parameter | Value |
|---|---|
| Layers | 12 |
| Embedding Dimension | 512 |
| Attention Heads | 8 (head_dim = 64) |
| FFN Intermediate Dim | 2,730 |
| Context Length | 512 tokens |
| Base Perplexity (WikiText-103) | 24.40 |

---

## Fine-Tuning Approach

TinyTalk-124M was produced using a parameter-efficient LoRA adaptation strategy:

- The base model backbone remains **frozen** throughout fine-tuning
- Trainable low-rank matrices (**rank = 64, alpha = 128**) are injected into attention (`qkv_proj`, `out_proj`) and feed-forward (`w1`, `w2`, `w3`) projections
- Adapter weights are merged directly into the base after training: `W' = W₀ + B·A · (α/r)`, producing a standalone, dependency-free checkpoint
- Fine-tuned on **69,220 instruction/response conversational pairs** (ChatAlpaca-style format)
- Trained with AdamW, cosine learning rate decay, and mixed precision (AMP)

| Metric | Value |
|---|---|
| Adaptation Method | LoRA (r=64, α=128) |
| Fine-Tuning Data | 69,220 instruction/response pairs |
| Training Epochs | 5 |
| Validation Perplexity | ~25.8 |

---

## Intended Use

TinyTalk-124M is designed for:

- Instruction-following and conversational text generation
- Research and educational demonstrations of from-scratch LLM pretraining and LoRA fine-tuning
- Lightweight experimentation on consumer-grade hardware

As a 124M-parameter model, it is best suited to general conversational and instructional prompts rather than tasks requiring extensive world knowledge or long-context reasoning — its scale reflects a deliberate choice to make the full architecture-to-deployment pipeline reproducible on a single consumer GPU.

---

## How to Use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Debarun12/tinytalk-124m")
model = AutoModelForCausalLM.from_pretrained("Debarun12/tinytalk-124m")

prompt = "How can I improve my relationship with my family?"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    repetition_penalty=3.0,
    do_sample=True,
    temperature=0.7
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

---

## Training Data

The fine-tuning dataset consists of 69,220 cleaned instruction/response conversational pairs in an Alpaca-style format, curated and deduplicated to maintain consistent formatting and answer quality.

---

## Training Infrastructure

- **Hardware:** Single consumer GPU
- **Framework:** PyTorch (no external training libraries)
- **Precision:** Mixed precision (AMP)
- **Optimizer:** AdamW with cosine learning rate decay

---

## Citation

If you use this model, please cite:

```bibtex
@misc{tinytalk124m2026,
  author = {Debarun Das},
  title = {TinyTalk-124M: A Conversational Language Model Built From Scratch},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Debarun12}}
}
```

---

## Author

**Debarun Das**
[GitHub](https://github.com/debarun23) · [Hugging Face](https://huggingface.co/Debarun12)