---
language:
- en
license: apache-2.0
tags:
- text-generation
- conversational
- pytorch
- safetensors
- causal-lm
- slm
- on-device
pipeline_tag: text-generation
widget:
- text: "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n"
---

> 🚀 **Major Update (September 2026):** **[PicoLM-V2-81M-Instruct](https://huggingface.co/aethertp/PicoLM-V2-81M-Instruct)** is officially released! 
> Featuring 36 layers of computational depth (MobileLLM-LS), a 24k vocabulary, and a massive **+16.4% gain on ARC-Easy (reaching 42.00%)**. We strongly recommend using V2!

# PicoLM-80M-Instruct 🚀

**PicoLM-80M-Instruct** is an ultra-compact, 80.24-million parameter causal language model designed for extreme efficiency, fast inference, and on-device deployment.

Trained completely from scratch on Kaggle dual Tesla T4 GPUs with zero budget, PicoLM-80M proves what can be achieved through strict modern architecture optimizations (SwiGLU, Grouped-Query Attention, RMSNorm, QK-Norm, and Tied Embeddings) paired with dense educational synthetic data.

---

## 📌 Model Overview

- **Developer:** Emre Polat
- **Parameters:** 80,242,240 (~80.2M)
- **Context Window:** 2,048 tokens
- **Vocabulary:** 16,384 (Single-digit regex split, Byte-level BPE)
- **Format:** Safetensors (FP16) & GGUF
- **Primary Language:** English + Python Code
- **License:** Apache 2.0

---

## 📊 Empirical Benchmark Results (Verified)

All scores below were **empirically measured** directly on the model weights using standard log-likelihood evaluations:

| Benchmark / Task | Random Baseline | SmolLM2-135M (HF) | Gemma 3 270M (Google) | PicoLM-80M-Instruct (Ours) |
| :--- | :--- | :--- | :--- | :--- |
| **HellaSwag (Commonsense)** | 25.00% | 42.10% | 37.70% | **31.20%** *(+6.2% above random)* |
| **ARC-Easy (Science QA)** | 25.00% | 58.50% | 57.70% | **25.60%** *(Floor effect)* |
| **Validation Perplexity** | ~16,384 | — | — | **14.65** |
| **Factual QA ("Capital of France")** | Hallucination | Factual | Factual | **"The capital of France is Paris."** |
| **Stop Token Discipline** | Loops | Strict | Strict | **100% strict `<|im_end|>` termination** |

---

## 💻 Quickstart (Transformers Native)

You can load and chat with PicoLM directly via Hugging Face `transformers`:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "aethertp/PicoLM-80M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).cuda()

messages = [{"role": "user", "content": "What is the capital of France?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

outputs = model.generate(**inputs, max_new_tokens=50, temperature=0.6, do_sample=True)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:]))
```

---

## ⚠️ Limitations

- **Factual Depth:** With 80M parameters, the model cannot serve as a comprehensive encyclopedia. Factual queries should be supported by RAG.
- **Multi-step Math:** Elementary arithmetic works, but complex multi-variable algebra requires external verification.