Text Generation
Transformers
Safetensors
GGUF
English
qwen3
causal-lm
chat
from-scratch
conversational
text-generation-inference
Instructions to use aethertp/PicoLM-V4-85M-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use aethertp/PicoLM-V4-85M-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="aethertp/PicoLM-V4-85M-Instruct") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("aethertp/PicoLM-V4-85M-Instruct") model = AutoModelForCausalLM.from_pretrained("aethertp/PicoLM-V4-85M-Instruct", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use aethertp/PicoLM-V4-85M-Instruct with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf aethertp/PicoLM-V4-85M-Instruct:Q4_K_M # Run inference directly in the terminal: llama cli -hf aethertp/PicoLM-V4-85M-Instruct:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf aethertp/PicoLM-V4-85M-Instruct:Q4_K_M # Run inference directly in the terminal: llama cli -hf aethertp/PicoLM-V4-85M-Instruct:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf aethertp/PicoLM-V4-85M-Instruct:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf aethertp/PicoLM-V4-85M-Instruct:Q4_K_M
Use Docker
docker model run hf.co/aethertp/PicoLM-V4-85M-Instruct:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use aethertp/PicoLM-V4-85M-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "aethertp/PicoLM-V4-85M-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "aethertp/PicoLM-V4-85M-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/aethertp/PicoLM-V4-85M-Instruct:Q4_K_M
- SGLang
How to use aethertp/PicoLM-V4-85M-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "aethertp/PicoLM-V4-85M-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "aethertp/PicoLM-V4-85M-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "aethertp/PicoLM-V4-85M-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "aethertp/PicoLM-V4-85M-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use aethertp/PicoLM-V4-85M-Instruct with Ollama:
ollama run hf.co/aethertp/PicoLM-V4-85M-Instruct:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use aethertp/PicoLM-V4-85M-Instruct with Docker Model Runner:
docker model run hf.co/aethertp/PicoLM-V4-85M-Instruct:Q4_K_M
- Lemonade
How to use aethertp/PicoLM-V4-85M-Instruct with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull aethertp/PicoLM-V4-85M-Instruct:Q4_K_M
Run and chat with the model
lemonade run user.PicoLM-V4-85M-Instruct-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Card review: measured wording for chat abilities, token counts, overlap-check limits, GGUF perplexity scope, benchmark SE wording
Browse files
README.md
CHANGED
|
@@ -17,7 +17,7 @@ datasets:
|
|
| 17 |
|
| 18 |
# PicoLM-V4-85M-Instruct
|
| 19 |
|
| 20 |
-
PicoLM-V4-85M-Instruct is a chat model with 84,960,832 parameters. It is [PicoLM-V4-85M](https://huggingface.co/aethertp/PicoLM-V4-85M), a base model trained from scratch
|
| 21 |
|
| 22 |
The weights are stored in the Qwen3 format of the `transformers` library (`Qwen3ForCausalLM`), so the model loads with `AutoModelForCausalLM` and `AutoTokenizer` without `trust_remote_code`. This was tested on transformers 4.57.1 and 5.22.1. GGUF files for llama.cpp are included (see Files). The tokenizer is the 24,576-token BPE vocabulary of PicoLM-V3; the architecture, pretraining and the base model's evaluation are described in the [base model card](https://huggingface.co/aethertp/PicoLM-V4-85M).
|
| 23 |
|
|
@@ -54,15 +54,15 @@ I'm PicoLM-V4, a small language model made by aethertp.
|
|
| 54 |
|
| 55 |
The prompt format is `<|im_start|>role\ncontent<|im_end|>\n` for each message, followed by `<|im_start|>assistant\n`. The model ends its turn with `<|im_end|>` (token id 2), which is the end-of-sequence token set in `config.json`; `generation_config.json` lists ids 2 and 0 (`<|endoftext|>`). Do not add a beginning-of-text token to the prompt: the model was fine-tuned without one.
|
| 56 |
|
| 57 |
-
**Generation settings.** The numbers in this card were measured with greedy decoding, with and without `repetition_penalty=1.1`. The recommended setting is greedy decoding with `repetition_penalty=1.1`. Sampling (temperature 0.7, top-p 0.9) looped less often in our test (3.6% of prompts against 5.6%) and passed 12 of our 24 panel checks against 11 (the one extra pass is the 12-word title check); in the answers we read, it produced more nonsense.
|
| 58 |
|
| 59 |
## Model details
|
| 60 |
|
| 61 |
-
The architecture is the one of the base model: 20 layers, hidden size 576, 9 query heads and 3 key/value heads with head dimension 64, SwiGLU feed-forward with intermediate size 1536, RMSNorm
|
| 62 |
|
| 63 |
## Fine-tuning data
|
| 64 |
|
| 65 |
-
All data comes from [`HuggingFaceTB/smol-smoltalk`](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk), the SmolTalk subset that HuggingFace built for small models, plus a small identity set written by us and a share of pretraining text. We used six of its sources. We did not use its `smol-magpie-ultra` source: it was generated with Llama-3.1-405B-Instruct
|
| 66 |
|
| 67 |
| Source | Real tokens in the training plan | Share |
|
| 68 |
|---|---|---|
|
|
@@ -86,7 +86,7 @@ All data comes from [`HuggingFaceTB/smol-smoltalk`](https://huggingface.co/datas
|
|
| 86 |
- `openhermes-50k` is a sample of [OpenHermes 2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5). Its card is tagged `Synthetic`, `GPT-4` and `Distillation`, describes the data as "primarily synthetically generated" and compiled from many sources, and names among them Airoboros 2.2, GPT-4-only subsets of ChatBot Arena and ShareGPT, GPT4-LLM, WizardLM Evol-Instruct, SlimOrca, Platypus, CamelAI and Glaive. The card text we read states no license for the compilation. It is about 31% of the fine-tuning tokens. Whether the terms of the services that produced such outputs restrict their use for training a model is a question we did not resolve and are not qualified to answer.
|
| 87 |
- `self-oss-instruct` corresponds to the Self-OSS-Starcoder2-Instruct dataset named in the SmolTalk card, and `explore-instruct-rewrite` to the Explore-Instruct-Rewriting dataset named there. We did not check the models or the licenses behind them.
|
| 88 |
- For `smol-contraints`, `smol-summarize-20k` and `smollm-rewrite-30k`, the SmolTalk card says the generation code is published but we did not verify which models generated the data.
|
| 89 |
-
- The fine-tuning data was not decontaminated against the evaluation sets of this card. The overlap was measured afterwards, and nothing was removed. Method: the 139,505 examples of the fine-tuning training file (42.6 million tokens, decoded back to text; it lacks 88 of the 522 identity conversations, which were added to the training mix later
|
| 90 |
|
| 91 |
| Benchmark | Items | 13 words: testable | 13 words: overlapping | 8 words: testable | 8 words: overlapping |
|
| 92 |
|---|---|---|---|---|---|
|
|
@@ -125,7 +125,7 @@ Two learning rates were tried with the final setup, 3e-4 and 5e-4. Before the ru
|
|
| 125 |
|
| 126 |
### Benchmarks
|
| 127 |
|
| 128 |
-
Zero-shot, same harness
|
| 129 |
|
| 130 |
| Model | HellaSwag | ARC-e | ARC-c | PIQA | Winogrande | OBQA | SciQ | BoolQ | LAMBADA acc | LAMBADA ppl | WikiText word ppl | WikiText bits/byte |
|
| 131 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -133,11 +133,11 @@ Zero-shot, same harness, settings and code as the base model's evaluation: lm-ev
|
|
| 133 |
| PicoLM-V4-85M (base) | 32.8 | 54.8 | 26.8 | 63.7 | 50.1 | 31.4 | 78.5 | 59.1 | 31.2 | 48.37 | 38.83 | 0.9872 |
|
| 134 |
| Instruct minus Base | +0.8 | +0.3 | -1.0 | -0.3 | -0.1 | +0.4 | +1.5 | -2.2 | -1.0 | +8.06 | +7.38 | +0.0469 |
|
| 135 |
|
| 136 |
-
Metrics are accuracy in percent unless a column says otherwise: `acc_norm` for HellaSwag, ARC-Challenge and OpenBookQA, `acc` for the others. Standard errors and the unnormalized and normalized pairs for every task are in `eval_results.json` in this repository. The standard errors of the harness are about 0.5 points for HellaSwag, 1.0 for ARC-Easy, 1.3 for ARC-Challenge, 1.1 for PIQA, 1.4 for Winogrande, 2.1 for OpenBookQA, 1.3 for SciQ, 0.9 for BoolQ and 0.
|
| 137 |
|
| 138 |
-
- **The multiple-choice scores
|
| 139 |
- **Raw-text perplexity got worse.** WikiText word perplexity rose from 38.83 to 46.21 (+19%) and LAMBADA perplexity from 48.37 to 56.43 (+17%). On held-out pretraining text, the average loss rose from 2.2826 to 2.4069 nats per token (32 windows of 1024 tokens from each of six sources of the base model's validation mix, measured at the end of fine-tuning). This is the cost of fine-tuning on chat data. If you need the model for plain text continuation, use the base model.
|
| 140 |
-
- BoolQ is
|
| 141 |
- MMLU, code benchmarks and instruction-following benchmarks such as IFEval were not run.
|
| 142 |
|
| 143 |
### Chat behavior
|
|
@@ -154,7 +154,7 @@ We generated answers with at most 256 new tokens for 360 single-turn prompts tak
|
|
| 154 |
|
| 155 |
We also ran 24 short prompts that we wrote ourselves, with a simple automatic check for each (a panel of identity, format, fact, arithmetic and memory questions). Greedy decoding passed 11 of 24 (10 if the check for "count from 1 to 5" is made strict, because the model kept counting past 5). With the checks as they were run:
|
| 156 |
|
| 157 |
-
- **Passed:** name and maker (`PicoLM-V4`, `aethertp`), "Are you ChatGPT?" answered no, remembering a name given earlier in a two-turn conversation (`Dana`), a three-bullet list, a JSON object, a simple Python function (`def add`), the capitals of France, Japan and Italy.
|
| 158 |
- **Failed:** all three arithmetic questions (12 + 15, 9 times 8, 100 minus 37), the capital of Germany, the winner of the 1998 World Cup, all five strict length or format constraints (one word, exactly two sentences, exactly three words, a title of at most 12 words, three colors separated by commas), "Reply with only YES or NO: is Paris in France?" (the model answered that Paris is not in France), a one-sentence summary under a system prompt, and writing a sentence in capital letters.
|
| 159 |
- On 12 prompts outside the training distribution, the answers to a Spanish question and to a translation into French were not meaningful, and requests for step-by-step reasoning reached wrong answers. The model was not trained on multi-turn data beyond the identity set, so longer conversations are untested.
|
| 160 |
|
|
@@ -170,7 +170,7 @@ The model is small, and fine-tuning did not change what it knows. It makes factu
|
|
| 170 |
|---|---|---|
|
| 171 |
| `model.safetensors` | 339.9 MB | weights in fp32 (`Qwen3ForCausalLM`) |
|
| 172 |
| `config.json`, `generation_config.json`, `tokenizer.json`, `tokenizer_config.json` | small | configuration, tokenizer, ChatML chat template (in `tokenizer_config.json`) |
|
| 173 |
-
| `eval_results.json` | small | raw output of the benchmark run above (Instruct and base, with standard errors) |
|
| 174 |
| `PicoLM-V4-85M-Instruct-F16.gguf` | 170.9 MB | GGUF, 16-bit floats |
|
| 175 |
| `PicoLM-V4-85M-Instruct-Q8_0.gguf` | 91.2 MB | GGUF, 8-bit quantization (recommended) |
|
| 176 |
| `PicoLM-V4-85M-Instruct-Q4_K_M.gguf` | 65.2 MB | GGUF, 4-bit quantization (visibly lossy, see below) |
|
|
@@ -182,12 +182,12 @@ The GGUF files were converted from the weights in this repository with llama.cpp
|
|
| 182 |
- The token ids of 40 test strings (English, code, digits, whitespace, accents, CJK, emoji, strings containing the special tokens `<|endoftext|>` and `<|im_end|>`) and of 4 chat prompts are identical to the ones from the `transformers` tokenizer.
|
| 183 |
- The template stored in the GGUF renders the same prompt as the one in `tokenizer_config.json`.
|
| 184 |
- Greedy decoding with the F16 file (no repetition penalty, 100 new tokens) gave exactly the same text as `transformers` for all 8 test chats.
|
| 185 |
-
- Agreement of the quantized files with F16 was measured on
|
| 186 |
|
| 187 |
| File | Perplexity ratio to F16 | 99th percentile KL divergence to F16 | Same top-1 token as F16 | Greedy text identical to F16 (8 chats) |
|
| 188 |
|---|---|---|---|---|
|
| 189 |
| Q8_0 | 1.0006 | 0.0056 | 98.0% | 7 of 8 (the longest answer diverged after 79 characters) |
|
| 190 |
-
| Q4_K_M | 1.0104 | 0.0713 | 94.1% | 5 of 8 (the other three
|
| 191 |
|
| 192 |
Q8_0 is close to F16 and is the recommended file. Q4_K_M saves 26 MB over Q8_0 and changes the output noticeably more, so we do not recommend it unless size matters. A command that was tested in this form:
|
| 193 |
|
|
|
|
| 17 |
|
| 18 |
# PicoLM-V4-85M-Instruct
|
| 19 |
|
| 20 |
+
PicoLM-V4-85M-Instruct is a chat model with 84,960,832 parameters. It is [PicoLM-V4-85M](https://huggingface.co/aethertp/PicoLM-V4-85M), a base model trained from scratch for 6.0 billion training tokens (part of them repeated, see the base model card), fine-tuned for two epochs on a mix of mostly synthetic instruction and chat data, a small identity set and a share of pretraining text (about 82 million tokens processed, 41 million per pass). In our probes the fine-tuned model follows some simple single-turn requests (its name and maker, short lists, JSON), ignores strict format and length constraints, failed all three arithmetic questions and two of five fact questions. Multi-turn conversation is largely untested. Read the Limitations section before using it.
|
| 21 |
|
| 22 |
The weights are stored in the Qwen3 format of the `transformers` library (`Qwen3ForCausalLM`), so the model loads with `AutoModelForCausalLM` and `AutoTokenizer` without `trust_remote_code`. This was tested on transformers 4.57.1 and 5.22.1. GGUF files for llama.cpp are included (see Files). The tokenizer is the 24,576-token BPE vocabulary of PicoLM-V3; the architecture, pretraining and the base model's evaluation are described in the [base model card](https://huggingface.co/aethertp/PicoLM-V4-85M).
|
| 23 |
|
|
|
|
| 54 |
|
| 55 |
The prompt format is `<|im_start|>role\ncontent<|im_end|>\n` for each message, followed by `<|im_start|>assistant\n`. The model ends its turn with `<|im_end|>` (token id 2), which is the end-of-sequence token set in `config.json`; `generation_config.json` lists ids 2 and 0 (`<|endoftext|>`). Do not add a beginning-of-text token to the prompt: the model was fine-tuned without one.
|
| 56 |
|
| 57 |
+
**Generation settings.** The numbers in this card were measured with greedy decoding, with and without `repetition_penalty=1.1`. The recommended setting is greedy decoding with `repetition_penalty=1.1`. The chat numbers below (not the benchmark scores, which are based on log-likelihoods) were measured with plain greedy decoding, with greedy decoding plus the repetition penalty, and with sampling. Sampling (temperature 0.7, top-p 0.9) looped less often in our test (3.6% of prompts against 5.6%; with 360 prompts this is a difference of about 1.3 standard errors, within noise) and passed 12 of our 24 panel checks against 11 for greedy decoding (the one extra pass is the 12-word title check, which we count as a chance pass); in the answers we read, it produced more nonsense. Do not rely on system messages: in our panel, a one-sentence summary under a system prompt failed.
|
| 58 |
|
| 59 |
## Model details
|
| 60 |
|
| 61 |
+
The architecture is the one of the base model: 20 layers, hidden size 576, 9 query heads and 3 key/value heads with head dimension 64, SwiGLU feed-forward with intermediate size 1536, RMSNorm, plus RMSNorm on queries and keys, RoPE (base 10,000), a vocabulary of 24,576 tokens with tied input and output embeddings, and a context length of 2048. No architecture or tokenizer changes were made for the fine-tune. The chat tokens `<|im_start|>` and `<|im_end|>` already existed in the tokenizer. Token id 2 (`<|im_end|>`) is not used in the pretraining data (see the base model card), so the fine-tune is where the model learns to use it.
|
| 62 |
|
| 63 |
## Fine-tuning data
|
| 64 |
|
| 65 |
+
All data comes from [`HuggingFaceTB/smol-smoltalk`](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk), the SmolTalk subset that HuggingFace built for small models, plus a small identity set written by us and a share of pretraining text. We used six of its sources. We did not use its `smol-magpie-ultra` source: it was generated with Llama-3.1-405B-Instruct. That decision was made on provenance grounds before any experiment. A later small experiment with arms of 5.7 to 6.8 million tokens of it gave no gain on our development loss (the arms ended 0.002 and 0.004 nats worse than their controls); the controls were not matched in tokens, so it is weak evidence.
|
| 66 |
|
| 67 |
| Source | Real tokens in the training plan | Share |
|
| 68 |
|---|---|---|
|
|
|
|
| 86 |
- `openhermes-50k` is a sample of [OpenHermes 2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5). Its card is tagged `Synthetic`, `GPT-4` and `Distillation`, describes the data as "primarily synthetically generated" and compiled from many sources, and names among them Airoboros 2.2, GPT-4-only subsets of ChatBot Arena and ShareGPT, GPT4-LLM, WizardLM Evol-Instruct, SlimOrca, Platypus, CamelAI and Glaive. The card text we read states no license for the compilation. It is about 31% of the fine-tuning tokens. Whether the terms of the services that produced such outputs restrict their use for training a model is a question we did not resolve and are not qualified to answer.
|
| 87 |
- `self-oss-instruct` corresponds to the Self-OSS-Starcoder2-Instruct dataset named in the SmolTalk card, and `explore-instruct-rewrite` to the Explore-Instruct-Rewriting dataset named there. We did not check the models or the licenses behind them.
|
| 88 |
- For `smol-contraints`, `smol-summarize-20k` and `smollm-rewrite-30k`, the SmolTalk card says the generation code is published but we did not verify which models generated the data.
|
| 89 |
+
- The fine-tuning data was not decontaminated against the evaluation sets of this card. The overlap was measured afterwards, and nothing was removed. Method: the 139,505 examples of the fine-tuning training file (42.6 million tokens, decoded back to text; it lacks 88 of the 522 identity conversations, which were added to the training mix later (they are short name-memory exchanges and were not checked); the replay text, which is pretraining data and about 8% of the fine-tuning tokens, is not part of this file and was not checked) and every string of every benchmark item (question, choices, context, passage) were lowercased and split into words. An item counts as overlapping if any run of 13 (or of 8) consecutive words of it also occurs in some training example. Items in which no string has that many words cannot be tested and are left out of the percentages. The result is below. A hit shows shared text. It does not show whether the question or the answer was also in the training data, and the checked strings include passages. Runs of 8 words also match ordinary phrases. Only exact word runs are detected, not paraphrases or near-duplicates. For the 13-word runs, only 125 of the 500 OpenBookQA items, 1,024 of 1,838 PIQA items, 1,576 of 2,376 ARC-Easy items and 877 of 1,172 ARC-Challenge items are long enough to be tested, so the zeros in those rows say little.
|
| 90 |
|
| 91 |
| Benchmark | Items | 13 words: testable | 13 words: overlapping | 8 words: testable | 8 words: overlapping |
|
| 92 |
|---|---|---|---|---|---|
|
|
|
|
| 125 |
|
| 126 |
### Benchmarks
|
| 127 |
|
| 128 |
+
Zero-shot, same harness version and settings as the base model's evaluation: lm-evaluation-harness 0.4.9.1 (transformers 4.57.1, datasets 3.6.0), full test or validation sets, maximum length 1024, fp32. The base model card's numbers were produced with torch 2.11.0+cu128 and the runs for this card with 2.11.0+cu130; the base row that was re-run in this session reproduces the base card's numbers. Raw text is scored without a chat template, exactly as for the base model. The base row was re-run in the same session as the Instruct row and gives the same numbers as in the base model card.
|
| 129 |
|
| 130 |
| Model | HellaSwag | ARC-e | ARC-c | PIQA | Winogrande | OBQA | SciQ | BoolQ | LAMBADA acc | LAMBADA ppl | WikiText word ppl | WikiText bits/byte |
|
| 131 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
|
|
| 133 |
| PicoLM-V4-85M (base) | 32.8 | 54.8 | 26.8 | 63.7 | 50.1 | 31.4 | 78.5 | 59.1 | 31.2 | 48.37 | 38.83 | 0.9872 |
|
| 134 |
| Instruct minus Base | +0.8 | +0.3 | -1.0 | -0.3 | -0.1 | +0.4 | +1.5 | -2.2 | -1.0 | +8.06 | +7.38 | +0.0469 |
|
| 135 |
|
| 136 |
+
Metrics are accuracy in percent unless a column says otherwise: `acc_norm` for HellaSwag, ARC-Challenge and OpenBookQA, `acc` for the others. Standard errors and the unnormalized and normalized pairs for every task are in `eval_results.json` in this repository. The standard errors of the harness are about 0.5 points for HellaSwag, 1.0 for ARC-Easy, 1.3 for ARC-Challenge, 1.1 for PIQA, 1.4 for Winogrande, 2.1 for OpenBookQA, 1.3 for SciQ, 0.9 for BoolQ and 0.65 for LAMBADA accuracy. The standard error of the difference between two models is about 1.4 times these figures.
|
| 137 |
|
| 138 |
+
- **The multiple-choice scores show no change beyond noise.** On these nine columns no difference between Instruct and base is larger than two standard errors of the difference (treating the two runs as independent). The largest are BoolQ (-2.2 points, about 1.7 standard errors) and SciQ (+1.5, about 0.8). Fine-tuning neither improved nor visibly damaged what these benchmarks measure. They do not measure chat quality.
|
| 139 |
- **Raw-text perplexity got worse.** WikiText word perplexity rose from 38.83 to 46.21 (+19%) and LAMBADA perplexity from 48.37 to 56.43 (+17%). On held-out pretraining text, the average loss rose from 2.2826 to 2.4069 nats per token (32 windows of 1024 tokens from each of six sources of the base model's validation mix, measured at the end of fine-tuning). This is the cost of fine-tuning on chat data. If you need the model for plain text continuation, use the base model.
|
| 140 |
+
- BoolQ is below the majority rate (62.2% "yes") for both models (56.9 and 59.1), so that column does not show reading comprehension for either.
|
| 141 |
- MMLU, code benchmarks and instruction-following benchmarks such as IFEval were not run.
|
| 142 |
|
| 143 |
### Chat behavior
|
|
|
|
| 154 |
|
| 155 |
We also ran 24 short prompts that we wrote ourselves, with a simple automatic check for each (a panel of identity, format, fact, arithmetic and memory questions). Greedy decoding passed 11 of 24 (10 if the check for "count from 1 to 5" is made strict, because the model kept counting past 5). With the checks as they were run:
|
| 156 |
|
| 157 |
+
- **Passed:** name and maker (`PicoLM-V4`, `aethertp`; two checks), "Are you ChatGPT?" answered no, remembering a name given earlier in a two-turn conversation (`Dana`), counting from 1 to 5 (the check was loose, see above), a three-bullet list, a JSON object, a simple Python function (`def add`), the capitals of France, Japan and Italy.
|
| 158 |
- **Failed:** all three arithmetic questions (12 + 15, 9 times 8, 100 minus 37), the capital of Germany, the winner of the 1998 World Cup, all five strict length or format constraints (one word, exactly two sentences, exactly three words, a title of at most 12 words, three colors separated by commas), "Reply with only YES or NO: is Paris in France?" (the model answered that Paris is not in France), a one-sentence summary under a system prompt, and writing a sentence in capital letters.
|
| 159 |
- On 12 prompts outside the training distribution, the answers to a Spanish question and to a translation into French were not meaningful, and requests for step-by-step reasoning reached wrong answers. The model was not trained on multi-turn data beyond the identity set, so longer conversations are untested.
|
| 160 |
|
|
|
|
| 170 |
|---|---|---|
|
| 171 |
| `model.safetensors` | 339.9 MB | weights in fp32 (`Qwen3ForCausalLM`) |
|
| 172 |
| `config.json`, `generation_config.json`, `tokenizer.json`, `tokenizer_config.json` | small | configuration, tokenizer, ChatML chat template (in `tokenizer_config.json`) |
|
| 173 |
+
| `eval_results.json` | small | raw output of the benchmark run above (Instruct and base, with standard errors); one label in it is Turkish (`yeniden koşu` means re-run) |
|
| 174 |
| `PicoLM-V4-85M-Instruct-F16.gguf` | 170.9 MB | GGUF, 16-bit floats |
|
| 175 |
| `PicoLM-V4-85M-Instruct-Q8_0.gguf` | 91.2 MB | GGUF, 8-bit quantization (recommended) |
|
| 176 |
| `PicoLM-V4-85M-Instruct-Q4_K_M.gguf` | 65.2 MB | GGUF, 4-bit quantization (visibly lossy, see below) |
|
|
|
|
| 182 |
- The token ids of 40 test strings (English, code, digits, whitespace, accents, CJK, emoji, strings containing the special tokens `<|endoftext|>` and `<|im_end|>`) and of 4 chat prompts are identical to the ones from the `transformers` tokenizer.
|
| 183 |
- The template stored in the GGUF renders the same prompt as the one in `tokenizer_config.json`.
|
| 184 |
- Greedy decoding with the F16 file (no repetition penalty, 100 new tokens) gave exactly the same text as `transformers` for all 8 test chats.
|
| 185 |
+
- Agreement of the quantized files with F16 was measured on held-out text from the base model's validation mix (a text file of 1.0 million characters, of which the first 30 windows of 1024 tokens, about 31 thousand tokens, were used, with the second half of each window scored; the statistics below therefore rest on about 15 thousand scored tokens). F16 perplexity on this text is 16.585 (`transformers` fp32: 16.585).
|
| 186 |
|
| 187 |
| File | Perplexity ratio to F16 | 99th percentile KL divergence to F16 | Same top-1 token as F16 | Greedy text identical to F16 (8 chats) |
|
| 188 |
|---|---|---|---|---|
|
| 189 |
| Q8_0 | 1.0006 | 0.0056 | 98.0% | 7 of 8 (the longest answer diverged after 79 characters) |
|
| 190 |
+
| Q4_K_M | 1.0104 | 0.0713 | 94.1% | 5 of 8 (the other three diverged within the first 54 characters) |
|
| 191 |
|
| 192 |
Q8_0 is close to F16 and is the recommended file. Q4_K_M saves 26 MB over Q8_0 and changes the output noticeably more, so we do not recommend it unless size matters. A command that was tested in this form:
|
| 193 |
|