Upload folder using huggingface_hub

README.md

@@ -0,0 +1,102 @@
+---
+language:
+- en
+license: apache-2.0
+tags:
+- text-generation
+- conversational
+- pytorch
+- safetensors
+- causal-lm
+- slm
+- on-device
+pipeline_tag: text-generation
+widget:
+- text: "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n"
+---
+
+# PicoLM-80M-Instruct 🚀
+
+**PicoLM-80M-Instruct** is an ultra-compact, 80.24-million parameter causal language model designed for extreme efficiency, fast inference, and on-device deployment.
+
+Trained completely from scratch on Kaggle dual Tesla T4 GPUs with zero budget, PicoLM-80M proves what can be achieved through strict modern architecture optimizations (SwiGLU, Grouped-Query Attention, RMSNorm, QK-Norm, and Tied Embeddings) paired with dense educational synthetic data.
+
+---
+
+## 📌 Model Overview
+
+- **Developer:** Emre Polat
+- **Parameters:** 80,242,240 (~80.2M)
+- **Context Window:** 2,048 tokens
+- **Vocabulary:** 16,384 (Single-digit regex split, Byte-level BPE)
+- **Format:** Safetensors (FP16)
+- **Primary Language:** English + Python Code
+- **License:** Apache 2.0
+
+---
+
+## 📊 Empirical Benchmark Results (Verified)
+
+Unlike models with estimated metrics, all scores below were **empirically measured** directly on the model weights using standard log-likelihood evaluations:
+
+| Benchmark / Task | Random Baseline | SmolLM2-135M (HF) | Gemma 3 270M (Google) | PicoLM-80M-Instruct (Ours) |
+| :--- | :--- | :--- | :--- | :--- |
+| **HellaSwag (Commonsense)** | 25.00% | 42.10% | 37.70% | **31.20%** *(+6.2% above random)* |
+| **ARC-Easy (Science QA)** | 25.00% | 58.50% | 57.70% | **25.60%** *(Floor effect)* |
+| **Validation Perplexity** | ~16,384 | — | — | **14.65** |
+| **Factual QA ("Capital of France")** | Hallucination | Factual | Factual | **"The capital of France is Paris."** |
+| **Stop Token Discipline** | Loops | Strict | Strict | **100% strict `<|im_end|>` termination** |
+
+---
+
+## 🏗️ Architecture Specifications
+
+- **Layers:** 20 Transformer blocks (Deep & Thin)
+- **Hidden Dimension ($d_{model}$):** 576
+- **Attention Heads (Q):** 9 (Head dim = 64)
+- **KV Heads (GQA):** 3 (3:1 Grouped-Query Attention)
+- **Intermediate Dimension:** 1536 (SwiGLU)
+- **Normalization:** Pre-RMSNorm with **QK-Norm** (prevents attention logit blowup)
+- **Positional Encoding:** RoPE ($	heta = 10000$)
+- **Weights Tying:** Input embeddings and LM Head weights are shared (Zero parameter waste)
+
+---
+
+## 📚 Training Curriculum
+
+1. **Pre-Training (Stage 1 - ~307M Tokens):**
+   - 54% Cosmopedia v2 (Textbooks & reasoning)
+   - 23% FineWeb-Edu Dedup (Educational web articles)
+   - 12% Python-Edu (Algorithms and code syntax)
+   - 11% SmolTalk Dialogues
+   - Optimization: WSD (Warmup-Stable-Decay) across 2x Tesla T4 GPUs.
+2. **Chat Alignment (Stage 2 - SFT):**
+   - 15,000 curated dialogues from SmolTalk.
+   - Strict Prompt-Masking (`ignore_index = -100`) applied so only assistant responses receive loss updates.
+
+---
+
+## 💻 Quickstart Inference
+
+```python
+import torch
+from transformers import PreTrainedTokenizerFast
+
+# Load tokenizer
+tokenizer = PreTrainedTokenizerFast.from_pretrained("aethertp/PicoLM-80M-Instruct")
+
+# Format prompt using ChatML
+user_prompt = "What is the capital of France?"
+formatted = f"<|im_start|>user\n{user_prompt}<|im_end|>\n<|im_start|>assistant\n"
+input_ids = torch.tensor([tokenizer.encode(formatted, add_special_tokens=False)])
+
+# Generate response
+# (Model weights available in model.safetensors)
+```
+
+---
+
+## ⚠️ Limitations
+
+- **Factual Depth:** With 80M parameters, the model cannot serve as a reliable encyclopedia for obscure facts. Factual queries should be supported by RAG.
+- **Multi-step Math:** Elementary arithmetic works, but multi-variable algebra requires external verification.

chat_template.jinja

@@ -0,0 +1,4 @@
+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}

config.json

@@ -0,0 +1,17 @@
+{
+  "architectures": [
+    "PicoLMForCausalLM"
+  ],
+  "model_type": "picolm",
+  "vocab_size": 16384,
+  "hidden_size": 576,
+  "num_hidden_layers": 20,
+  "num_attention_heads": 9,
+  "num_key_value_heads": 3,
+  "intermediate_size": 1536,
+  "max_position_embeddings": 2048,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 10000.0,
+  "torch_dtype": "float16",
+  "transformers_version": "4.45.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5761e07f8e7fe770c67db71dc0189272d188a728a00596d78018334640372a7
+size 179380816

tokenizer_config.json

@@ -0,0 +1,8 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend"
+}