aethertp commited on
Commit
e0e852d
·
verified ·
1 Parent(s): 829c2b4

Card review: measured wording for chat abilities, token counts, overlap-check limits, GGUF perplexity scope, benchmark SE wording

Browse files
Files changed (1) hide show
  1. README.md +13 -13
README.md CHANGED
@@ -17,7 +17,7 @@ datasets:
17
 
18
  # PicoLM-V4-85M-Instruct
19
 
20
- PicoLM-V4-85M-Instruct is a chat model with 84,960,832 parameters. It is [PicoLM-V4-85M](https://huggingface.co/aethertp/PicoLM-V4-85M), a base model trained from scratch on 6.0 billion training tokens, fine-tuned on about 82 million tokens of synthetic instruction and chat data. The fine-tuned model follows simple instructions, keeps a short conversation going and uses system prompts for simple formats. It is also wrong about facts and arithmetic most of the time. Read the Limitations section before using it.
21
 
22
  The weights are stored in the Qwen3 format of the `transformers` library (`Qwen3ForCausalLM`), so the model loads with `AutoModelForCausalLM` and `AutoTokenizer` without `trust_remote_code`. This was tested on transformers 4.57.1 and 5.22.1. GGUF files for llama.cpp are included (see Files). The tokenizer is the 24,576-token BPE vocabulary of PicoLM-V3; the architecture, pretraining and the base model's evaluation are described in the [base model card](https://huggingface.co/aethertp/PicoLM-V4-85M).
23
 
@@ -54,15 +54,15 @@ I'm PicoLM-V4, a small language model made by aethertp.
54
 
55
  The prompt format is `<|im_start|>role\ncontent<|im_end|>\n` for each message, followed by `<|im_start|>assistant\n`. The model ends its turn with `<|im_end|>` (token id 2), which is the end-of-sequence token set in `config.json`; `generation_config.json` lists ids 2 and 0 (`<|endoftext|>`). Do not add a beginning-of-text token to the prompt: the model was fine-tuned without one.
56
 
57
- **Generation settings.** The numbers in this card were measured with greedy decoding, with and without `repetition_penalty=1.1`. The recommended setting is greedy decoding with `repetition_penalty=1.1`. Sampling (temperature 0.7, top-p 0.9) looped less often in our test (3.6% of prompts against 5.6%) and passed 12 of our 24 panel checks against 11 (the one extra pass is the 12-word title check); in the answers we read, it produced more nonsense. Use a system message only for simple formats; see Limitations.
58
 
59
  ## Model details
60
 
61
- The architecture is the one of the base model: 20 layers, hidden size 576, 9 query heads and 3 key/value heads with head dimension 64, SwiGLU feed-forward with intermediate size 1536, RMSNorm with RMSNorm on queries and keys, RoPE (base 10,000), a vocabulary of 24,576 tokens with tied input and output embeddings, and a context length of 2048. No architecture or tokenizer changes were made for the fine-tune. The chat tokens `<|im_start|>` and `<|im_end|>` already existed in the tokenizer. Token id 2 (`<|im_end|>`) is not used in the pretraining data (see the base model card), so the fine-tune is where the model learns to use it.
62
 
63
  ## Fine-tuning data
64
 
65
- All data comes from [`HuggingFaceTB/smol-smoltalk`](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk), the SmolTalk subset that HuggingFace built for small models, plus a small identity set written by us and a share of pretraining text. We used six of its sources. We did not use its `smol-magpie-ultra` source: it was generated with Llama-3.1-405B-Instruct, and in our own ablation it gave no measurable gain on our development loss.
66
 
67
  | Source | Real tokens in the training plan | Share |
68
  |---|---|---|
@@ -86,7 +86,7 @@ All data comes from [`HuggingFaceTB/smol-smoltalk`](https://huggingface.co/datas
86
  - `openhermes-50k` is a sample of [OpenHermes 2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5). Its card is tagged `Synthetic`, `GPT-4` and `Distillation`, describes the data as "primarily synthetically generated" and compiled from many sources, and names among them Airoboros 2.2, GPT-4-only subsets of ChatBot Arena and ShareGPT, GPT4-LLM, WizardLM Evol-Instruct, SlimOrca, Platypus, CamelAI and Glaive. The card text we read states no license for the compilation. It is about 31% of the fine-tuning tokens. Whether the terms of the services that produced such outputs restrict their use for training a model is a question we did not resolve and are not qualified to answer.
87
  - `self-oss-instruct` corresponds to the Self-OSS-Starcoder2-Instruct dataset named in the SmolTalk card, and `explore-instruct-rewrite` to the Explore-Instruct-Rewriting dataset named there. We did not check the models or the licenses behind them.
88
  - For `smol-contraints`, `smol-summarize-20k` and `smollm-rewrite-30k`, the SmolTalk card says the generation code is published but we did not verify which models generated the data.
89
- - The fine-tuning data was not decontaminated against the evaluation sets of this card. The overlap was measured afterwards, and nothing was removed. Method: the 139,505 examples of the fine-tuning training file (42.6 million tokens, decoded back to text; it lacks 88 of the 522 identity conversations, which were added to the training mix later and contain no benchmark text; the replay text, which is pretraining data and about 8% of the fine-tuning tokens, is not part of this file and was not checked) and every string of every benchmark item (question, choices, context, passage) were lowercased and split into words. An item counts as overlapping if any run of 13 (or of 8) consecutive words of it also occurs in some training example. Items in which no string has that many words cannot be tested and are left out of the percentages. The result is below. A hit shows shared text. It does not show whether the question or the answer was also in the training data, and the checked strings include passages. Runs of 8 words also match ordinary phrases.
90
 
91
  | Benchmark | Items | 13 words: testable | 13 words: overlapping | 8 words: testable | 8 words: overlapping |
92
  |---|---|---|---|---|---|
@@ -125,7 +125,7 @@ Two learning rates were tried with the final setup, 3e-4 and 5e-4. Before the ru
125
 
126
  ### Benchmarks
127
 
128
- Zero-shot, same harness, settings and code as the base model's evaluation: lm-evaluation-harness 0.4.9.1 (transformers 4.57.1, datasets 3.6.0), full test or validation sets, maximum length 1024, fp32. Raw text is scored without a chat template, exactly as for the base model. The base row was re-run in the same session as the Instruct row and gives the same numbers as in the base model card.
129
 
130
  | Model | HellaSwag | ARC-e | ARC-c | PIQA | Winogrande | OBQA | SciQ | BoolQ | LAMBADA acc | LAMBADA ppl | WikiText word ppl | WikiText bits/byte |
131
  |---|---|---|---|---|---|---|---|---|---|---|---|---|
@@ -133,11 +133,11 @@ Zero-shot, same harness, settings and code as the base model's evaluation: lm-ev
133
  | PicoLM-V4-85M (base) | 32.8 | 54.8 | 26.8 | 63.7 | 50.1 | 31.4 | 78.5 | 59.1 | 31.2 | 48.37 | 38.83 | 0.9872 |
134
  | Instruct minus Base | +0.8 | +0.3 | -1.0 | -0.3 | -0.1 | +0.4 | +1.5 | -2.2 | -1.0 | +8.06 | +7.38 | +0.0469 |
135
 
136
- Metrics are accuracy in percent unless a column says otherwise: `acc_norm` for HellaSwag, ARC-Challenge and OpenBookQA, `acc` for the others. Standard errors and the unnormalized and normalized pairs for every task are in `eval_results.json` in this repository. The standard errors of the harness are about 0.5 points for HellaSwag, 1.0 for ARC-Easy, 1.3 for ARC-Challenge, 1.1 for PIQA, 1.4 for Winogrande, 2.1 for OpenBookQA, 1.3 for SciQ, 0.9 for BoolQ and 0.6 for LAMBADA accuracy.
137
 
138
- - **The multiple-choice scores did not change.** No difference between Instruct and base on these nine columns is larger than two standard errors. The largest are BoolQ (-2.2 points) and SciQ (+1.5). Fine-tuning neither improved nor visibly damaged what these benchmarks measure. They do not measure chat quality.
139
  - **Raw-text perplexity got worse.** WikiText word perplexity rose from 38.83 to 46.21 (+19%) and LAMBADA perplexity from 48.37 to 56.43 (+17%). On held-out pretraining text, the average loss rose from 2.2826 to 2.4069 nats per token (32 windows of 1024 tokens from each of six sources of the base model's validation mix, measured at the end of fine-tuning). This is the cost of fine-tuning on chat data. If you need the model for plain text continuation, use the base model.
140
- - BoolQ is near the majority rate (62.2% "yes") for both models, so that column does not show reading comprehension for either.
141
  - MMLU, code benchmarks and instruction-following benchmarks such as IFEval were not run.
142
 
143
  ### Chat behavior
@@ -154,7 +154,7 @@ We generated answers with at most 256 new tokens for 360 single-turn prompts tak
154
 
155
  We also ran 24 short prompts that we wrote ourselves, with a simple automatic check for each (a panel of identity, format, fact, arithmetic and memory questions). Greedy decoding passed 11 of 24 (10 if the check for "count from 1 to 5" is made strict, because the model kept counting past 5). With the checks as they were run:
156
 
157
- - **Passed:** name and maker (`PicoLM-V4`, `aethertp`), "Are you ChatGPT?" answered no, remembering a name given earlier in a two-turn conversation (`Dana`), a three-bullet list, a JSON object, a simple Python function (`def add`), the capitals of France, Japan and Italy.
158
  - **Failed:** all three arithmetic questions (12 + 15, 9 times 8, 100 minus 37), the capital of Germany, the winner of the 1998 World Cup, all five strict length or format constraints (one word, exactly two sentences, exactly three words, a title of at most 12 words, three colors separated by commas), "Reply with only YES or NO: is Paris in France?" (the model answered that Paris is not in France), a one-sentence summary under a system prompt, and writing a sentence in capital letters.
159
  - On 12 prompts outside the training distribution, the answers to a Spanish question and to a translation into French were not meaningful, and requests for step-by-step reasoning reached wrong answers. The model was not trained on multi-turn data beyond the identity set, so longer conversations are untested.
160
 
@@ -170,7 +170,7 @@ The model is small, and fine-tuning did not change what it knows. It makes factu
170
  |---|---|---|
171
  | `model.safetensors` | 339.9 MB | weights in fp32 (`Qwen3ForCausalLM`) |
172
  | `config.json`, `generation_config.json`, `tokenizer.json`, `tokenizer_config.json` | small | configuration, tokenizer, ChatML chat template (in `tokenizer_config.json`) |
173
- | `eval_results.json` | small | raw output of the benchmark run above (Instruct and base, with standard errors) |
174
  | `PicoLM-V4-85M-Instruct-F16.gguf` | 170.9 MB | GGUF, 16-bit floats |
175
  | `PicoLM-V4-85M-Instruct-Q8_0.gguf` | 91.2 MB | GGUF, 8-bit quantization (recommended) |
176
  | `PicoLM-V4-85M-Instruct-Q4_K_M.gguf` | 65.2 MB | GGUF, 4-bit quantization (visibly lossy, see below) |
@@ -182,12 +182,12 @@ The GGUF files were converted from the weights in this repository with llama.cpp
182
  - The token ids of 40 test strings (English, code, digits, whitespace, accents, CJK, emoji, strings containing the special tokens `<|endoftext|>` and `<|im_end|>`) and of 4 chat prompts are identical to the ones from the `transformers` tokenizer.
183
  - The template stored in the GGUF renders the same prompt as the one in `tokenizer_config.json`.
184
  - Greedy decoding with the F16 file (no repetition penalty, 100 new tokens) gave exactly the same text as `transformers` for all 8 test chats.
185
- - Agreement of the quantized files with F16 was measured on 1.0 million characters of held-out text from the base model's validation mix (30 windows of 1024 tokens, the second half of each window scored). F16 perplexity on this text is 16.585 (`transformers` fp32: 16.585).
186
 
187
  | File | Perplexity ratio to F16 | 99th percentile KL divergence to F16 | Same top-1 token as F16 | Greedy text identical to F16 (8 chats) |
188
  |---|---|---|---|---|
189
  | Q8_0 | 1.0006 | 0.0056 | 98.0% | 7 of 8 (the longest answer diverged after 79 characters) |
190
- | Q4_K_M | 1.0104 | 0.0713 | 94.1% | 5 of 8 (the other three differ from the first sentence on) |
191
 
192
  Q8_0 is close to F16 and is the recommended file. Q4_K_M saves 26 MB over Q8_0 and changes the output noticeably more, so we do not recommend it unless size matters. A command that was tested in this form:
193
 
 
17
 
18
  # PicoLM-V4-85M-Instruct
19
 
20
+ PicoLM-V4-85M-Instruct is a chat model with 84,960,832 parameters. It is [PicoLM-V4-85M](https://huggingface.co/aethertp/PicoLM-V4-85M), a base model trained from scratch for 6.0 billion training tokens (part of them repeated, see the base model card), fine-tuned for two epochs on a mix of mostly synthetic instruction and chat data, a small identity set and a share of pretraining text (about 82 million tokens processed, 41 million per pass). In our probes the fine-tuned model follows some simple single-turn requests (its name and maker, short lists, JSON), ignores strict format and length constraints, failed all three arithmetic questions and two of five fact questions. Multi-turn conversation is largely untested. Read the Limitations section before using it.
21
 
22
  The weights are stored in the Qwen3 format of the `transformers` library (`Qwen3ForCausalLM`), so the model loads with `AutoModelForCausalLM` and `AutoTokenizer` without `trust_remote_code`. This was tested on transformers 4.57.1 and 5.22.1. GGUF files for llama.cpp are included (see Files). The tokenizer is the 24,576-token BPE vocabulary of PicoLM-V3; the architecture, pretraining and the base model's evaluation are described in the [base model card](https://huggingface.co/aethertp/PicoLM-V4-85M).
23
 
 
54
 
55
  The prompt format is `<|im_start|>role\ncontent<|im_end|>\n` for each message, followed by `<|im_start|>assistant\n`. The model ends its turn with `<|im_end|>` (token id 2), which is the end-of-sequence token set in `config.json`; `generation_config.json` lists ids 2 and 0 (`<|endoftext|>`). Do not add a beginning-of-text token to the prompt: the model was fine-tuned without one.
56
 
57
+ **Generation settings.** The numbers in this card were measured with greedy decoding, with and without `repetition_penalty=1.1`. The recommended setting is greedy decoding with `repetition_penalty=1.1`. The chat numbers below (not the benchmark scores, which are based on log-likelihoods) were measured with plain greedy decoding, with greedy decoding plus the repetition penalty, and with sampling. Sampling (temperature 0.7, top-p 0.9) looped less often in our test (3.6% of prompts against 5.6%; with 360 prompts this is a difference of about 1.3 standard errors, within noise) and passed 12 of our 24 panel checks against 11 for greedy decoding (the one extra pass is the 12-word title check, which we count as a chance pass); in the answers we read, it produced more nonsense. Do not rely on system messages: in our panel, a one-sentence summary under a system prompt failed.
58
 
59
  ## Model details
60
 
61
+ The architecture is the one of the base model: 20 layers, hidden size 576, 9 query heads and 3 key/value heads with head dimension 64, SwiGLU feed-forward with intermediate size 1536, RMSNorm, plus RMSNorm on queries and keys, RoPE (base 10,000), a vocabulary of 24,576 tokens with tied input and output embeddings, and a context length of 2048. No architecture or tokenizer changes were made for the fine-tune. The chat tokens `<|im_start|>` and `<|im_end|>` already existed in the tokenizer. Token id 2 (`<|im_end|>`) is not used in the pretraining data (see the base model card), so the fine-tune is where the model learns to use it.
62
 
63
  ## Fine-tuning data
64
 
65
+ All data comes from [`HuggingFaceTB/smol-smoltalk`](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk), the SmolTalk subset that HuggingFace built for small models, plus a small identity set written by us and a share of pretraining text. We used six of its sources. We did not use its `smol-magpie-ultra` source: it was generated with Llama-3.1-405B-Instruct. That decision was made on provenance grounds before any experiment. A later small experiment with arms of 5.7 to 6.8 million tokens of it gave no gain on our development loss (the arms ended 0.002 and 0.004 nats worse than their controls); the controls were not matched in tokens, so it is weak evidence.
66
 
67
  | Source | Real tokens in the training plan | Share |
68
  |---|---|---|
 
86
  - `openhermes-50k` is a sample of [OpenHermes 2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5). Its card is tagged `Synthetic`, `GPT-4` and `Distillation`, describes the data as "primarily synthetically generated" and compiled from many sources, and names among them Airoboros 2.2, GPT-4-only subsets of ChatBot Arena and ShareGPT, GPT4-LLM, WizardLM Evol-Instruct, SlimOrca, Platypus, CamelAI and Glaive. The card text we read states no license for the compilation. It is about 31% of the fine-tuning tokens. Whether the terms of the services that produced such outputs restrict their use for training a model is a question we did not resolve and are not qualified to answer.
87
  - `self-oss-instruct` corresponds to the Self-OSS-Starcoder2-Instruct dataset named in the SmolTalk card, and `explore-instruct-rewrite` to the Explore-Instruct-Rewriting dataset named there. We did not check the models or the licenses behind them.
88
  - For `smol-contraints`, `smol-summarize-20k` and `smollm-rewrite-30k`, the SmolTalk card says the generation code is published but we did not verify which models generated the data.
89
+ - The fine-tuning data was not decontaminated against the evaluation sets of this card. The overlap was measured afterwards, and nothing was removed. Method: the 139,505 examples of the fine-tuning training file (42.6 million tokens, decoded back to text; it lacks 88 of the 522 identity conversations, which were added to the training mix later (they are short name-memory exchanges and were not checked); the replay text, which is pretraining data and about 8% of the fine-tuning tokens, is not part of this file and was not checked) and every string of every benchmark item (question, choices, context, passage) were lowercased and split into words. An item counts as overlapping if any run of 13 (or of 8) consecutive words of it also occurs in some training example. Items in which no string has that many words cannot be tested and are left out of the percentages. The result is below. A hit shows shared text. It does not show whether the question or the answer was also in the training data, and the checked strings include passages. Runs of 8 words also match ordinary phrases. Only exact word runs are detected, not paraphrases or near-duplicates. For the 13-word runs, only 125 of the 500 OpenBookQA items, 1,024 of 1,838 PIQA items, 1,576 of 2,376 ARC-Easy items and 877 of 1,172 ARC-Challenge items are long enough to be tested, so the zeros in those rows say little.
90
 
91
  | Benchmark | Items | 13 words: testable | 13 words: overlapping | 8 words: testable | 8 words: overlapping |
92
  |---|---|---|---|---|---|
 
125
 
126
  ### Benchmarks
127
 
128
+ Zero-shot, same harness version and settings as the base model's evaluation: lm-evaluation-harness 0.4.9.1 (transformers 4.57.1, datasets 3.6.0), full test or validation sets, maximum length 1024, fp32. The base model card's numbers were produced with torch 2.11.0+cu128 and the runs for this card with 2.11.0+cu130; the base row that was re-run in this session reproduces the base card's numbers. Raw text is scored without a chat template, exactly as for the base model. The base row was re-run in the same session as the Instruct row and gives the same numbers as in the base model card.
129
 
130
  | Model | HellaSwag | ARC-e | ARC-c | PIQA | Winogrande | OBQA | SciQ | BoolQ | LAMBADA acc | LAMBADA ppl | WikiText word ppl | WikiText bits/byte |
131
  |---|---|---|---|---|---|---|---|---|---|---|---|---|
 
133
  | PicoLM-V4-85M (base) | 32.8 | 54.8 | 26.8 | 63.7 | 50.1 | 31.4 | 78.5 | 59.1 | 31.2 | 48.37 | 38.83 | 0.9872 |
134
  | Instruct minus Base | +0.8 | +0.3 | -1.0 | -0.3 | -0.1 | +0.4 | +1.5 | -2.2 | -1.0 | +8.06 | +7.38 | +0.0469 |
135
 
136
+ Metrics are accuracy in percent unless a column says otherwise: `acc_norm` for HellaSwag, ARC-Challenge and OpenBookQA, `acc` for the others. Standard errors and the unnormalized and normalized pairs for every task are in `eval_results.json` in this repository. The standard errors of the harness are about 0.5 points for HellaSwag, 1.0 for ARC-Easy, 1.3 for ARC-Challenge, 1.1 for PIQA, 1.4 for Winogrande, 2.1 for OpenBookQA, 1.3 for SciQ, 0.9 for BoolQ and 0.65 for LAMBADA accuracy. The standard error of the difference between two models is about 1.4 times these figures.
137
 
138
+ - **The multiple-choice scores show no change beyond noise.** On these nine columns no difference between Instruct and base is larger than two standard errors of the difference (treating the two runs as independent). The largest are BoolQ (-2.2 points, about 1.7 standard errors) and SciQ (+1.5, about 0.8). Fine-tuning neither improved nor visibly damaged what these benchmarks measure. They do not measure chat quality.
139
  - **Raw-text perplexity got worse.** WikiText word perplexity rose from 38.83 to 46.21 (+19%) and LAMBADA perplexity from 48.37 to 56.43 (+17%). On held-out pretraining text, the average loss rose from 2.2826 to 2.4069 nats per token (32 windows of 1024 tokens from each of six sources of the base model's validation mix, measured at the end of fine-tuning). This is the cost of fine-tuning on chat data. If you need the model for plain text continuation, use the base model.
140
+ - BoolQ is below the majority rate (62.2% "yes") for both models (56.9 and 59.1), so that column does not show reading comprehension for either.
141
  - MMLU, code benchmarks and instruction-following benchmarks such as IFEval were not run.
142
 
143
  ### Chat behavior
 
154
 
155
  We also ran 24 short prompts that we wrote ourselves, with a simple automatic check for each (a panel of identity, format, fact, arithmetic and memory questions). Greedy decoding passed 11 of 24 (10 if the check for "count from 1 to 5" is made strict, because the model kept counting past 5). With the checks as they were run:
156
 
157
+ - **Passed:** name and maker (`PicoLM-V4`, `aethertp`; two checks), "Are you ChatGPT?" answered no, remembering a name given earlier in a two-turn conversation (`Dana`), counting from 1 to 5 (the check was loose, see above), a three-bullet list, a JSON object, a simple Python function (`def add`), the capitals of France, Japan and Italy.
158
  - **Failed:** all three arithmetic questions (12 + 15, 9 times 8, 100 minus 37), the capital of Germany, the winner of the 1998 World Cup, all five strict length or format constraints (one word, exactly two sentences, exactly three words, a title of at most 12 words, three colors separated by commas), "Reply with only YES or NO: is Paris in France?" (the model answered that Paris is not in France), a one-sentence summary under a system prompt, and writing a sentence in capital letters.
159
  - On 12 prompts outside the training distribution, the answers to a Spanish question and to a translation into French were not meaningful, and requests for step-by-step reasoning reached wrong answers. The model was not trained on multi-turn data beyond the identity set, so longer conversations are untested.
160
 
 
170
  |---|---|---|
171
  | `model.safetensors` | 339.9 MB | weights in fp32 (`Qwen3ForCausalLM`) |
172
  | `config.json`, `generation_config.json`, `tokenizer.json`, `tokenizer_config.json` | small | configuration, tokenizer, ChatML chat template (in `tokenizer_config.json`) |
173
+ | `eval_results.json` | small | raw output of the benchmark run above (Instruct and base, with standard errors); one label in it is Turkish (`yeniden koşu` means re-run) |
174
  | `PicoLM-V4-85M-Instruct-F16.gguf` | 170.9 MB | GGUF, 16-bit floats |
175
  | `PicoLM-V4-85M-Instruct-Q8_0.gguf` | 91.2 MB | GGUF, 8-bit quantization (recommended) |
176
  | `PicoLM-V4-85M-Instruct-Q4_K_M.gguf` | 65.2 MB | GGUF, 4-bit quantization (visibly lossy, see below) |
 
182
  - The token ids of 40 test strings (English, code, digits, whitespace, accents, CJK, emoji, strings containing the special tokens `<|endoftext|>` and `<|im_end|>`) and of 4 chat prompts are identical to the ones from the `transformers` tokenizer.
183
  - The template stored in the GGUF renders the same prompt as the one in `tokenizer_config.json`.
184
  - Greedy decoding with the F16 file (no repetition penalty, 100 new tokens) gave exactly the same text as `transformers` for all 8 test chats.
185
+ - Agreement of the quantized files with F16 was measured on held-out text from the base model's validation mix (a text file of 1.0 million characters, of which the first 30 windows of 1024 tokens, about 31 thousand tokens, were used, with the second half of each window scored; the statistics below therefore rest on about 15 thousand scored tokens). F16 perplexity on this text is 16.585 (`transformers` fp32: 16.585).
186
 
187
  | File | Perplexity ratio to F16 | 99th percentile KL divergence to F16 | Same top-1 token as F16 | Greedy text identical to F16 (8 chats) |
188
  |---|---|---|---|---|
189
  | Q8_0 | 1.0006 | 0.0056 | 98.0% | 7 of 8 (the longest answer diverged after 79 characters) |
190
+ | Q4_K_M | 1.0104 | 0.0713 | 94.1% | 5 of 8 (the other three diverged within the first 54 characters) |
191
 
192
  Q8_0 is close to F16 and is the recommended file. Q4_K_M saves 26 MB over Q8_0 and changes the output noticeably more, so we do not recommend it unless size matters. A command that was tested in this form:
193