| --- |
| language: |
| - en |
| license: apache-2.0 |
| tags: |
| - text-generation |
| - conversational |
| - pytorch |
| - safetensors |
| - causal-lm |
| - slm |
| - on-device |
| pipeline_tag: text-generation |
| widget: |
| - text: "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n" |
| --- |
| |
| > 🚀 **Major Update (September 2026):** **[PicoLM-V2-81M-Instruct](https://huggingface.co/aethertp/PicoLM-V2-81M-Instruct)** is officially released! |
| > Featuring 36 layers of computational depth (MobileLLM-LS), a 24k vocabulary, and a massive **+16.4% gain on ARC-Easy (reaching 42.00%)**. We strongly recommend using V2! |
|
|
| # PicoLM-80M-Instruct 🚀 |
|
|
| **PicoLM-80M-Instruct** is an ultra-compact, 80.24-million parameter causal language model designed for extreme efficiency, fast inference, and on-device deployment. |
|
|
| Trained completely from scratch on Kaggle dual Tesla T4 GPUs with zero budget, PicoLM-80M proves what can be achieved through strict modern architecture optimizations (SwiGLU, Grouped-Query Attention, RMSNorm, QK-Norm, and Tied Embeddings) paired with dense educational synthetic data. |
|
|
| --- |
|
|
| ## 📌 Model Overview |
|
|
| - **Developer:** Emre Polat |
| - **Parameters:** 80,242,240 (~80.2M) |
| - **Context Window:** 2,048 tokens |
| - **Vocabulary:** 16,384 (Single-digit regex split, Byte-level BPE) |
| - **Format:** Safetensors (FP16) & GGUF |
| - **Primary Language:** English + Python Code |
| - **License:** Apache 2.0 |
|
|
| --- |
|
|
| ## 📊 Empirical Benchmark Results (Verified) |
|
|
| All scores below were **empirically measured** directly on the model weights using standard log-likelihood evaluations: |
|
|
| | Benchmark / Task | Random Baseline | SmolLM2-135M (HF) | Gemma 3 270M (Google) | PicoLM-80M-Instruct (Ours) | |
| | :--- | :--- | :--- | :--- | :--- | |
| | **HellaSwag (Commonsense)** | 25.00% | 42.10% | 37.70% | **31.20%** *(+6.2% above random)* | |
| | **ARC-Easy (Science QA)** | 25.00% | 58.50% | 57.70% | **25.60%** *(Floor effect)* | |
| | **Validation Perplexity** | ~16,384 | — | — | **14.65** | |
| | **Factual QA ("Capital of France")** | Hallucination | Factual | Factual | **"The capital of France is Paris."** | |
| | **Stop Token Discipline** | Loops | Strict | Strict | **100% strict `<|im_end|>` termination** | |
| |
| --- |
| |
| ## 💻 Quickstart (Transformers Native) |
| |
| You can load and chat with PicoLM directly via Hugging Face `transformers`: |
| |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "aethertp/PicoLM-80M-Instruct" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).cuda() |
| |
| messages = [{"role": "user", "content": "What is the capital of France?"}] |
| prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = tokenizer(prompt, return_tensors="pt").to("cuda") |
| |
| outputs = model.generate(**inputs, max_new_tokens=50, temperature=0.6, do_sample=True) |
| print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:])) |
| ``` |
| |
| --- |
| |
| ## ⚠️ Limitations |
| |
| - **Factual Depth:** With 80M parameters, the model cannot serve as a comprehensive encyclopedia. Factual queries should be supported by RAG. |
| - **Multi-step Math:** Elementary arithmetic works, but complex multi-variable algebra requires external verification. |
|
|
