MP-01 100M base: cooldown checkpoint (step 4000), new model card with compute disclosure, comparison and logo

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+manprot-logo.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -6,91 +6,113 @@ language:
 library_name: vbq
 pipeline_tag: text-generation
 tags:
-- base-model
 - pretraining
 - decoder-only
 - small-language-model
 ---
 
-# MP-01 (9500) 100M base
+<p align="center"><img src="manprot-logo.png" alt="Manprot" width="360"></p>
 
-MP-01 (9500) 100M base is a 100M-parameter decoder-only language model from Manprot, grown from a 75M base and pretrained further on new English data.
-This is the **base model** (no chat or instruction tuning). It continues text; it does not follow instructions.
+# MP-01 100M base
 
-- **Author:** Surelynotvain (organisation: Manprot)
-- **Checkpoint:** training step 9500 of 34341 (expansion run `vbq-100m-expand`)
-- **Parameters:** 100.6M (tied input/output embeddings)
+MP-01 100M base is a 100.6M-parameter decoder-only language model made by [Manprot](https://huggingface.co/Manprot).
+It is a **base model that is not pure pretraining**: it continues text, but it started from chat fine-tuned weights (see *Lineage*), so it may carry chat-style behaviour. It does not follow instructions reliably.
+
+## What happened to this model
+
+1. A 75M-parameter model was pretrained from scratch on 4.0B English tokens.
+2. It was grown to 18 layers (100M parameters) and pretrained for 0.61B more tokens.
+3. It was fine-tuned for chat (0.18B tokens). The chat weights were then used as the starting point for the expansion.
+4. **Expansion:** 12,700 steps of continued pretraining on about 3.3B new English tokens from the same datasets, using a 262k-token batch. This ran on a cloud A100 and was resumed in chunks.
+5. **Cooldown:** the learning rate was decayed to its minimum over 4,000 steps (1.05B tokens) on a second, unseen corpus, starting from step 15,000. This final checkpoint is the one released here.
+
+A separate local cooldown (170 steps from step 13,000) was run as an experiment. It is **not** in this release's lineage.
 
 ## Architecture
 
 | | |
 |---|---|
 | Layers | 18 |
-| Model width (d_model) | 640 |
+| Width (d_model) | 640 |
 | Attention heads / KV heads | 10 / 5 (grouped-query attention, QK-norm) |
 | MLP hidden | 1664 (SwiGLU) |
-| Vocabulary | 32768 (byte-level BPE, `tokenizer.json`) |
-| Context length | 2048 tokens (RoPE) |
-| Normalisation | RMSNorm |
+| Vocabulary | 32,768 (byte-level BPE, `tokenizer.json`) |
+| Context length | 1024 tokens during this stage (trained up to 2048 later in the planned run; not reached here) |
+| Parameters | 100.6M, tied input/output embeddings |
+
+This is a custom architecture, not a `transformers` model. Load it with the `vbq/` code in this repository (see *Loading*).
+
+## Training data
 
-This is a custom architecture, not a `transformers` model. Load it with the `vbq/` code in this repository.
+Mixture of the expansion stages, by share of tokens: FineWeb-Edu 34%, Wikipedia 24%, Cosmopedia 18%, DCLM 12%, FineMath 5%, Stack Exchange 3%, Python code 3%. The cooldown corpus is a different packed corpus built from the same sources. Training data is not released.
 
-## Training
+## Compute used for this release
 
-- Started from a 75M-parameter base, grown to 18 layers, then continued on new English data.
-- Optimizer: Muon. Peak learning rate 1.2e-3, batch 262,144 tokens, weight decay 0.1, z-loss 1e-4.
-- Sequence length 1,024 tokens for the first part, 2,048 from step 24,038 (not reached yet).
-- Data mixture of this stage (share of tokens): FineWeb-Edu 34%, Wikipedia 24%, Cosmopedia 18%,
-  DCLM 12%, FineMath 5%, Stack Exchange 3%, Python code 3%.
-- Expansion steps completed when this checkpoint was saved: 9500 of 34341
-  (about 1.88B tokens in this run).
+Hours are wall-clock GPU time from the run logs. Costs in credits are not listed.
+
+| Stage | Hardware | GPU hours | Tokens |
+|---|---|---:|---:|
+| Pretraining, 75M base | local AMD Radeon RX 7600 (8 GB) | not totalled from logs | 4.0B |
+| Growth to 100M, first stage | local AMD Radeon RX 7600 | 3.94 (one segment) | 0.61B |
+| Chat fine-tune | local AMD Radeon RX 7600 | 2.78 | 0.18B |
+| Expansion, chunk 1 (steps 2,314 to 14,230) | cloud A100 40 GB | 4.00 | 3.1B new |
+| Expansion, chunk 2 (steps 14,231 to 15,000) | cloud A100 40 GB | ~0.35 | 0.2B new |
+| Cooldown, 4,000 steps | cloud H200 141 GB | 0.68 | 1.05B |
+| Benchmarks of released checkpoints | local GPU | a few minutes each | |
+| Comparison benchmarks (7 models) | cloud H200 | a few minutes each, run in parallel | |
+
+Note: the local GPU's elapsed times are per recorded segment; long runs were restarted, so the 75M pretraining total is not reliable from the logs. The expansion continued beyond step 15,000 on the cloud and that part is not included in this checkpoint.
 
 ## Evaluation
 
-Scored on this repository's own harness (`scripts/eval_checkpoint.py`), zero-shot.
-
-Benchmark tasks use 500 questions each (the same sample every time). With 500 questions per task, a
-difference under about 2 points is within noise.
-
-| Task | Score |
-|---|---:|
-| arc_challenge | 27.6 |
-| arc_easy | 52.8 |
-| hellaswag | 29.8 |
-| openbookqa | 27.6 |
-| piqa | 62.4 |
-| sciq | 71.0 |
-| winogrande | 51.6 |
-| **Average (7 tasks)** | **46.11** |
-| **BLiMP (67 paradigms, grammatical vs ungrammatical minimal pairs)** | **76.88** |
-
-Validation loss (held-out text, nats per token): **2.7780** overall.
-
-| Source | Val loss |
-|---|---:|
-| fineweb_edu | 2.941 |
-| wikipedia | 2.714 |
-| cosmopedia | 2.183 |
-| dclm | 3.387 |
-| stackexchange | 2.566 |
-| finemath | 3.402 |
-| code_python | 1.823 |
+Scored by this repository's harness (`scripts/eval_checkpoint.py`, `scripts/eval_hf.py`): zero-shot, 500 questions per task (the same sample each time), accuracy or length-normalised accuracy as in the task definition. BLiMP is zero-shot minimal pairs (67 paradigms, about 1,000 pairs each). With 500 questions a task, differences under about 2 points are within noise.
+
+### Against other models
+
+Same harness and the same 500 questions for every model. Models are measured with their own tokenizers. Validation loss is not comparable across tokenizers, so it is not shown here.
+
+| Model | Params | arc_challenge | arc_easy | hellaswag | openbookqa | piqa | sciq | winogrande | Avg (7) | BLiMP |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| **MP-01 100M base (this model)** | 100.6M | 29.2 | 54.0 | 29.4 | 29.4 | 62.8 | 72.2 | 54.0 | **47.29** | 78.70 |
+| gpt-neo-125m | 125.2M | 25.2 | 41.8 | 27.8 | 25.6 | 63.4 | 73.4 | 50.4 | **43.94** | 79.26 |
+| opt-125m | 125.2M | 25.4 | 39.8 | 30.6 | 27.8 | 61.0 | 70.0 | 50.8 | **43.63** | 81.41 |
+| GPT-2 | 124.4M | 24.8 | 39.6 | 29.6 | 25.8 | 64.2 | 67.6 | 50.2 | **43.11** | 82.51 |
+| pythia-160m | 162.3M | 25.4 | 41.8 | 26.0 | 26.4 | 62.8 | 69.0 | 47.8 | **42.74** | 78.53 |
+| distilGPT-2 | 81.9M | 25.2 | 36.0 | 25.4 | 26.0 | 59.0 | 64.8 | 50.2 | **40.94** | 79.53 |
+| GPT-1 | 116.5M | 25.8 | 34.4 | 22.8 | 26.6 | 55.8 | 59.0 | 53.6 | **39.71** | 77.59 |
+
+Where we stand:
+- **7-task average:** this model is first of the models listed, by about 3.4 points over the best comparison model.
+- **BLiMP:** this model trails GPT-2 (82.5), OPT-125M (81.4), DistilGPT2 (79.5) and GPT-neo-125M (79.3) on minimal-pair grammar. It is ahead of Pythia-160M (78.5) and GPT-1 (77.6), by 0.2 and 1.1 points.
+- The comparison models are older and smaller-data models; the comparison is not a like-for-like training-budget comparison.
+- Cerebras-GPT-111M could not be loaded from the Hub and is not included.
+
+### Progress across checkpoints of this model
+
+| Checkpoint | Val loss | arc_challenge | arc_easy | hellaswag | openbookqa | piqa | sciq | winogrande | Avg (7) | BLiMP |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| Expansion step 9500 | 2.7780 | 27.6 | 52.8 | 29.8 | 27.6 | 62.4 | 71.0 | 51.6 | 46.11 | 76.88 |
+| Expansion step 13000 | 2.7722 | 28.8 | 49.8 | 28.4 | 29.0 | 61.6 | 70.4 | 51.8 | 45.69 | 78.03 |
+| Expansion step 15000 (start of cooldown) | 2.7730 | 26.8 | 52.6 | 29.0 | 30.0 | 62.0 | 69.8 | 53.0 | 46.17 | 77.66 |
+| Cooldown, step 4000 (this release) | 2.6531 | 29.2 | 54.0 | 29.4 | 29.4 | 62.8 | 72.2 | 54.0 | 47.29 | 78.70 |
+
+Validation loss is measured on the held-out split of the first expansion corpus, which the model trained on, so it is not a clean test and the cooldown's lower loss should not be read as general improvement on its own. The benchmark average rose after the cooldown, and that is the more useful signal.
 
 ## Limitations
 
-- Base model: it has no instruction or chat tuning and can produce factually wrong or repetitive text.
-- Small size (100M parameters) and small benchmarks; these numbers are indicative only.
-- Trained on web and educational text that may contain bias and errors.
-- The training data is not released with this model.
+- Not an instruction model. It can produce factually wrong, repetitive or unfocused text.
+- Chat-tuned starting weights: expect some chat-style behaviour, and it has not been checked for safety.
+- Small size and small benchmarks. These numbers are indicative, and they are not measured on the full test sets.
+- Trained on web and educational text that can contain bias and errors.
+- Training data is not released.
 
 ## Loading
 
 ```python
-import torch
+import json
 from tokenizers import Tokenizer
-from vbq.model import VBQ, VBQConfig
 from safetensors.torch import load_file
-import json
+from vbq.model import VBQ, VBQConfig
 
 cfg = VBQConfig(**json.load(open("config.json"))["model_cfg"])
 model = VBQ(cfg)
@@ -102,7 +124,8 @@ tok = Tokenizer.from_file("tokenizer.json")
 ## Repository contents
 
 - `model.safetensors`: weights (bf16)
-- `config.json`: architecture and training configuration
+- `config.json`: architecture and lineage
 - `tokenizer.json`: tokenizer
 - `vbq/`: model code
 - `eval_results.json`: full evaluation output for this checkpoint
+- `manprot-logo.png`: logo

config.json

@@ -18,7 +18,15 @@
     "n_experts": 0,
     "capacity_factor": 1.25
   },
-  "train_step": 9500,
-  "total_steps": 34341,
-  "display_name": "MP-01 (9500) 100M base"
+  "train_step": 4000,
+  "total_steps": null,
+  "display_name": "MP-01 (9500) 100M base",
+  "stage": "cooldown from expansion step 15000; 4,000 steps of 262,144 tokens on a second corpus",
+  "lineage": [
+    "EB-75M base (4.0B tokens)",
+    "grown to 100M (0.61B)",
+    "chat fine-tune (0.18B)",
+    "expansion (to step 15000)",
+    "cooldown (4000 steps)"
+  ]
 }

eval_results.json

@@ -1,99 +1,182 @@
 {
-  "ckpt": "step_009500.pt",
-  "step": 9500,
-  "val": {
-    "fineweb_edu": 2.9408107101917267,
-    "wikipedia": 2.713771253824234,
-    "cosmopedia": 2.1831925809383392,
-    "dclm": 3.386944741010666,
-    "stackexchange": 2.5659963488578796,
-    "finemath": 3.4021530747413635,
-    "code_python": 1.8225234597921371,
-    "overall": 2.7780119873583318
-  },
-  "bench": {
-    "arc_challenge": 27.6,
-    "arc_easy": 52.8,
-    "hellaswag": 29.8,
-    "openbookqa": 27.6,
-    "piqa": 62.4,
-    "sciq": 71.0,
-    "winogrande": 51.6
-  },
-  "bench_avg": 46.114285714285714,
-  "blimp": {
-    "per_paradigm": {
-      "adjunct_island": 82.9,
-      "anaphor_gender_agreement": 92.3,
-      "anaphor_number_agreement": 97.3,
-      "animate_subject_passive": 75.9,
-      "animate_subject_trans": 83.8,
-      "causative": 75.4,
-      "complex_NP_island": 50.3,
-      "coordinate_structure_constraint_complex_left_branch": 53.5,
-      "coordinate_structure_constraint_object_extraction": 75.9,
-      "determiner_noun_agreement_1": 98.2,
-      "determiner_noun_agreement_2": 97.8,
-      "determiner_noun_agreement_irregular_1": 90.6,
-      "determiner_noun_agreement_irregular_2": 95.9,
-      "determiner_noun_agreement_with_adj_2": 93.3,
-      "determiner_noun_agreement_with_adj_irregular_1": 90.1,
-      "determiner_noun_agreement_with_adj_irregular_2": 89.2,
-      "determiner_noun_agreement_with_adjective_1": 93.8,
-      "distractor_agreement_relational_noun": 84.0,
-      "distractor_agreement_relative_clause": 77.5,
-      "drop_argument": 76.5,
-      "ellipsis_n_bar_1": 83.8,
-      "ellipsis_n_bar_2": 87.6,
-      "existential_there_object_raising": 79.5,
-      "existential_there_quantifiers_1": 97.3,
-      "existential_there_quantifiers_2": 23.3,
-      "existential_there_subject_raising": 85.5,
-      "expletive_it_object_raising": 78.4,
-      "inchoative": 63.4,
-      "intransitive": 73.4,
-      "irregular_past_participle_adjectives": 90.8,
-      "irregular_past_participle_verbs": 92.5,
-      "irregular_plural_subject_verb_agreement_1": 88.8,
-      "irregular_plural_subject_verb_agreement_2": 91.9,
-      "left_branch_island_echo_question": 45.0,
-      "left_branch_island_simple_question": 54.8,
-      "matrix_question_npi_licensor_present": 44.7,
-      "npi_present_1": 55.7,
-      "npi_present_2": 60.9,
-      "only_npi_licensor_present": 62.0,
-      "only_npi_scope": 47.4,
-      "passive_1": 88.4,
-      "passive_2": 86.0,
-      "principle_A_c_command": 63.5,
-      "principle_A_case_1": 100.0,
-      "principle_A_case_2": 95.5,
-      "principle_A_domain_1": 96.2,
-      "principle_A_domain_2": 74.7,
-      "principle_A_domain_3": 57.6,
-      "principle_A_reconstruction": 36.3,
-      "regular_plural_subject_verb_agreement_1": 92.4,
-      "regular_plural_subject_verb_agreement_2": 91.7,
-      "sentential_negation_npi_licensor_present": 96.8,
-      "sentential_negation_npi_scope": 68.9,
-      "sentential_subject_island": 30.2,
-      "superlative_quantifiers_1": 91.8,
-      "superlative_quantifiers_2": 80.3,
-      "tough_vs_raising_1": 64.3,
-      "tough_vs_raising_2": 86.2,
-      "transitive": 86.9,
-      "wh_island": 70.5,
-      "wh_questions_object_gap": 75.6,
-      "wh_questions_subject_gap": 90.7,
-      "wh_questions_subject_gap_long_distance": 81.2,
-      "wh_vs_that_no_gap": 97.1,
-      "wh_vs_that_no_gap_long_distance": 96.4,
-      "wh_vs_that_with_gap": 46.1,
-      "wh_vs_that_with_gap_long_distance": 24.8
+  "checkpoint": "cooldown step 4000",
+  "result": {
+    "ckpt": "step_004000.pt",
+    "step": 4000,
+    "val": {
+      "fineweb_edu": 2.8250027298927307,
+      "wikipedia": 2.5901024639606476,
+      "cosmopedia": 2.018482878804207,
+      "dclm": 3.28472700715065,
+      "stackexchange": 2.518112599849701,
+      "finemath": 3.240337073802948,
+      "code_python": 1.6927177011966705,
+      "overall": 2.6531425672769546
+    },
+    "bench": {
+      "arc_challenge": 29.2,
+      "arc_easy": 54.0,
+      "hellaswag": 29.4,
+      "openbookqa": 29.4,
+      "piqa": 62.8,
+      "sciq": 72.2,
+      "winogrande": 54.0
     },
-    "paradigm_avg": 76.88059701492537,
-    "overall": 76.88059701492537,
-    "pairs": 67000
+    "bench_avg": 47.285714285714285,
+    "blimp": {
+      "per_paradigm": {
+        "adjunct_island": 85.7,
+        "anaphor_gender_agreement": 97.9,
+        "anaphor_number_agreement": 98.9,
+        "animate_subject_passive": 77.2,
+        "animate_subject_trans": 86.5,
+        "causative": 75.7,
+        "complex_NP_island": 50.8,
+        "coordinate_structure_constraint_complex_left_branch": 66.2,
+        "coordinate_structure_constraint_object_extraction": 78.0,
+        "determiner_noun_agreement_1": 97.8,
+        "determiner_noun_agreement_2": 97.2,
+        "determiner_noun_agreement_irregular_1": 91.8,
+        "determiner_noun_agreement_irregular_2": 96.4,
+        "determiner_noun_agreement_with_adj_2": 93.9,
+        "determiner_noun_agreement_with_adj_irregular_1": 87.9,
+        "determiner_noun_agreement_with_adj_irregular_2": 90.1,
+        "determiner_noun_agreement_with_adjective_1": 95.4,
+        "distractor_agreement_relational_noun": 85.8,
+        "distractor_agreement_relative_clause": 73.1,
+        "drop_argument": 76.1,
+        "ellipsis_n_bar_1": 78.1,
+        "ellipsis_n_bar_2": 89.6,
+        "existential_there_object_raising": 80.4,
+        "existential_there_quantifiers_1": 97.5,
+        "existential_there_quantifiers_2": 47.1,
+        "existential_there_subject_raising": 85.9,
+        "expletive_it_object_raising": 76.0,
+        "inchoative": 65.6,
+        "intransitive": 76.5,
+        "irregular_past_participle_adjectives": 89.0,
+        "irregular_past_participle_verbs": 87.0,
+        "irregular_plural_subject_verb_agreement_1": 89.9,
+        "irregular_plural_subject_verb_agreement_2": 93.1,
+        "left_branch_island_echo_question": 35.2,
+        "left_branch_island_simple_question": 65.1,
+        "matrix_question_npi_licensor_present": 51.0,
+        "npi_present_1": 59.7,
+        "npi_present_2": 62.4,
+        "only_npi_licensor_present": 91.1,
+        "only_npi_scope": 73.7,
+        "passive_1": 89.7,
+        "passive_2": 89.0,
+        "principle_A_c_command": 55.3,
+        "principle_A_case_1": 100.0,
+        "principle_A_case_2": 96.9,
+        "principle_A_domain_1": 97.0,
+        "principle_A_domain_2": 73.7,
+        "principle_A_domain_3": 61.2,
+        "principle_A_reconstruction": 29.9,
+        "regular_plural_subject_verb_agreement_1": 95.4,
+        "regular_plural_subject_verb_agreement_2": 90.8,
+        "sentential_negation_npi_licensor_present": 93.9,
+        "sentential_negation_npi_scope": 59.8,
+        "sentential_subject_island": 31.9,
+        "superlative_quantifiers_1": 88.6,
+        "superlative_quantifiers_2": 81.1,
+        "tough_vs_raising_1": 64.8,
+        "tough_vs_raising_2": 91.7,
+        "transitive": 89.2,
+        "wh_island": 73.2,
+        "wh_questions_object_gap": 77.5,
+        "wh_questions_subject_gap": 89.9,
+        "wh_questions_subject_gap_long_distance": 84.5,
+        "wh_vs_that_no_gap": 96.5,
+        "wh_vs_that_no_gap_long_distance": 95.8,
+        "wh_vs_that_with_gap": 47.0,
+        "wh_vs_that_with_gap_long_distance": 32.1
+      },
+      "paradigm_avg": 78.69701492537313,
+      "overall": 78.69701492537314,
+      "pairs": 67000
+    },
+    "seconds": 125
   },
-  "seconds": 124
+  "comparison": {
+    "EleutherAI/gpt-neo-125m": {
+      "bench_avg": 43.94285714285714,
+      "blimp": 79.25820895522388,
+      "bench": {
+        "arc_challenge": 25.2,
+        "arc_easy": 41.8,
+        "hellaswag": 27.8,
+        "openbookqa": 25.6,
+        "piqa": 63.4,
+        "sciq": 73.4,
+        "winogrande": 50.4
+      }
+    },
+    "facebook/opt-125m": {
+      "bench_avg": 43.628571428571426,
+      "blimp": 81.40597014925373,
+      "bench": {
+        "arc_challenge": 25.4,
+        "arc_easy": 39.8,
+        "hellaswag": 30.6,
+        "openbookqa": 27.8,
+        "piqa": 61.0,
+        "sciq": 70.0,
+        "winogrande": 50.8
+      }
+    },
+    "openai-community/gpt2": {
+      "bench_avg": 43.114285714285714,
+      "blimp": 82.51194029850747,
+      "bench": {
+        "arc_challenge": 24.8,
+        "arc_easy": 39.6,
+        "hellaswag": 29.6,
+        "openbookqa": 25.8,
+        "piqa": 64.2,
+        "sciq": 67.6,
+        "winogrande": 50.2
+      }
+    },
+    "EleutherAI/pythia-160m": {
+      "bench_avg": 42.74285714285714,
+      "blimp": 78.52835820895523,
+      "bench": {
+        "arc_challenge": 25.4,
+        "arc_easy": 41.8,
+        "hellaswag": 26.0,
+        "openbookqa": 26.4,
+        "piqa": 62.8,
+        "sciq": 69.0,
+        "winogrande": 47.8
+      }
+    },
+    "distilbert/distilgpt2": {
+      "bench_avg": 40.94285714285714,
+      "blimp": 79.53283582089553,
+      "bench": {
+        "arc_challenge": 25.2,
+        "arc_easy": 36.0,
+        "hellaswag": 25.4,
+        "openbookqa": 26.0,
+        "piqa": 59.0,
+        "sciq": 64.8,
+        "winogrande": 50.2
+      }
+    },
+    "openai-community/openai-gpt": {
+      "bench_avg": 39.714285714285715,
+      "blimp": 77.5910447761194,
+      "bench": {
+        "arc_challenge": 25.8,
+        "arc_easy": 34.4,
+        "hellaswag": 22.8,
+        "openbookqa": 26.6,
+        "piqa": 55.8,
+        "sciq": 59.0,
+        "winogrande": 53.6
+      }
+    }
+  }
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:621d2129dd34133fa4fcb623db7ff05d94755e0271eba4473e7625cdf8e84030
+oid sha256:9765866c8750b456322cd262a1e6160dc2d702619d2ba8385f2eaf73b812b7c0
 size 201265144