File size: 3,225 Bytes
54e18af 8ca632a 54e18af 22e993c 54e18af 8ca632a 54e18af 8ca632a 54e18af 8ca632a 54e18af 8ca632a 54e18af 8ca632a 54e18af 8ca632a 54e18af 8ca632a 54e18af 8ca632a 54e18af 8ca632a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 | ---
language:
- en
license: apache-2.0
tags:
- text-generation
- conversational
- pytorch
- safetensors
- causal-lm
- slm
- on-device
pipeline_tag: text-generation
widget:
- text: "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n"
---
> 🚀 **Major Update (September 2026):** **[PicoLM-V2-81M-Instruct](https://huggingface.co/aethertp/PicoLM-V2-81M-Instruct)** is officially released!
> Featuring 36 layers of computational depth (MobileLLM-LS), a 24k vocabulary, and a massive **+16.4% gain on ARC-Easy (reaching 42.00%)**. We strongly recommend using V2!
# PicoLM-80M-Instruct 🚀
**PicoLM-80M-Instruct** is an ultra-compact, 80.24-million parameter causal language model designed for extreme efficiency, fast inference, and on-device deployment.
Trained completely from scratch on Kaggle dual Tesla T4 GPUs with zero budget, PicoLM-80M proves what can be achieved through strict modern architecture optimizations (SwiGLU, Grouped-Query Attention, RMSNorm, QK-Norm, and Tied Embeddings) paired with dense educational synthetic data.
---
## 📌 Model Overview
- **Developer:** Emre Polat
- **Parameters:** 80,242,240 (~80.2M)
- **Context Window:** 2,048 tokens
- **Vocabulary:** 16,384 (Single-digit regex split, Byte-level BPE)
- **Format:** Safetensors (FP16) & GGUF
- **Primary Language:** English + Python Code
- **License:** Apache 2.0
---
## 📊 Empirical Benchmark Results (Verified)
All scores below were **empirically measured** directly on the model weights using standard log-likelihood evaluations:
| Benchmark / Task | Random Baseline | SmolLM2-135M (HF) | Gemma 3 270M (Google) | PicoLM-80M-Instruct (Ours) |
| :--- | :--- | :--- | :--- | :--- |
| **HellaSwag (Commonsense)** | 25.00% | 42.10% | 37.70% | **31.20%** *(+6.2% above random)* |
| **ARC-Easy (Science QA)** | 25.00% | 58.50% | 57.70% | **25.60%** *(Floor effect)* |
| **Validation Perplexity** | ~16,384 | — | — | **14.65** |
| **Factual QA ("Capital of France")** | Hallucination | Factual | Factual | **"The capital of France is Paris."** |
| **Stop Token Discipline** | Loops | Strict | Strict | **100% strict `<|im_end|>` termination** |
---
## 💻 Quickstart (Transformers Native)
You can load and chat with PicoLM directly via Hugging Face `transformers`:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "aethertp/PicoLM-80M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).cuda()
messages = [{"role": "user", "content": "What is the capital of France?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=50, temperature=0.6, do_sample=True)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:]))
```
---
## ⚠️ Limitations
- **Factual Depth:** With 80M parameters, the model cannot serve as a comprehensive encyclopedia. Factual queries should be supported by RAG.
- **Multi-step Math:** Elementary arithmetic works, but complex multi-variable algebra requires external verification.
|