Clarify format sizes and evaluation coverage

README.md

@@ -12,7 +12,7 @@ tags:
 
 # OLMoE-0125-Instruct-Compressed-Experimental
 
-An experimental, modified derivative of [AllenAI's allenai/OLMoE-1B-7B-0125-Instruct](https://huggingface.co/allenai/OLMoE-1B-7B-0125-Instruct), based on revision `b89a7c4bc24fb9e55ce2543c9458ce0ca5c4650e`. Experts were physically removed and the resulting model was further trained. This repository contains the complete healed BF16 Hugging Face checkpoint and its tokenizer.
+An experimental, modified derivative of [AllenAI's allenai/OLMoE-1B-7B-0125-Instruct](https://huggingface.co/allenai/OLMoE-1B-7B-0125-Instruct), based on revision `b89a7c4bc24fb9e55ce2543c9458ce0ca5c4650e`. This repository contains the complete healed BF16 Hugging Face checkpoint and its tokenizer.
 
 All nine benchmark categories have completed for the base, cut, and healed checkpoints.
 
@@ -26,11 +26,7 @@ The complete cut checkpoint, before further training, is retained at [revision `
 
 ## Measured results
 
-These are measurements from one seed (42) on sampled benchmark tasks. They use local evaluation protocols and should not be compared directly with vendor leaderboard scores. No replicated quality improvement is claimed. The training run used a targeted mathematics corpus.
-
-Training stopped early after validation at step 250; the saved model uses the selected checkpoint at step 125, rather than the requested 750 steps. Evaluate the measured tradeoffs below before using this derivative.
-
-The original merge diagnostic flagged 33 tensors based on small sampled corners. Independent full-tensor comparisons confirmed that every flagged tensor changed; the sampled diagnostic was a false alarm. The original diagnostic is retained in the result summary.
+These are measurements from one seed (42) on sampled benchmark tasks. They use local evaluation protocols and should not be compared directly with vendor leaderboard scores. No replicated quality improvement is claimed.
 
 | Benchmark category | Base | Cut | Healed |
 | --- | ---: | ---: | ---: |
@@ -46,6 +42,40 @@ The original merge diagnostic flagged 33 tensors based on small sampled corners.
 
 WikiText perplexity is lower-is-better. IFEval reports strict prompt accuracy over the supported instruction subset; its coverage and excluded counts are in `evaluation_results.json`. BBQ is multiple-choice accuracy, not a safety certification. These sampled results do not establish general capability, safety, or production readiness.
 
+## Formats and evaluation coverage
+
+File sizes are storage requirements, not measured runtime memory or speed.
+
+| Format | File size | Public download | Evaluation |
+| --- | ---: | --- | --- |
+| BF16 | 12.03 GB | Available | 9 categories measured |
+| Q8_0 | Not generated | Not published | Not measured |
+| Q4_K_M | 3.66 GB | Not published | Not measured |
+| Q2_K | Not generated | Not published | Not measured |
+
+Scores apply only to the exact format evaluated; BF16 scores do not qualify a GGUF conversion.
+
+| Benchmark | BF16 | Q8_0 | Q4_K_M | Q2_K |
+| --- | ---: | ---: | ---: | ---: |
+| arc science | 45.00% (n=200) | Not measured | Not measured | Not measured |
+| hellaswag commonsense | 66.00% (n=200) | Not measured | Not measured | Not measured |
+| mmlu knowledge | 26.50% (n=200) | Not measured | Not measured | Not measured |
+| gpqa reasoning | 16.67% (n=198) | Not measured | Not measured | Not measured |
+| wikitext language | 21.632 PPL (n=64) | Not measured | Not measured | Not measured |
+| bbq safety | 39.50% (n=200) | Not measured | Not measured | Not measured |
+| multilingual knowledge | 26.50% (n=200) | Not measured | Not measured | Not measured |
+| gsm8k math | 43.00% (n=200) | Not measured | Not measured | Not measured |
+| ifeval instruction following | 55.63% (n=142) | Not measured | Not measured | Not measured |
+
+### Artifact identities
+
+| Format | SHA-256 |
+| --- | --- |
+| BF16 | `e7b3d37beb32930d2c9b1cea775b50f15d79d06dab5d0b39782d206c8ba99573` |
+| Q4_K_M | `4f7fe283cd9a714a3c96824f30a83e7e4d70e55f16a9b9e0aea8f644e68e620f` |
+
+Q4_K_M is 69.54% smaller than the BF16 checkpoint by file size. It was generated and preserved locally, but is not downloadable from this repository and has no separate quality evaluation. Q8_0 and Q2_K were not generated in this first experiment. No format-specific speed, peak-memory, or quantization quality claim is made.
+
 ## Use
 
 ```python

evaluation_results.json

@@ -167,17 +167,6 @@
   "standalone_model_and_tokenizer_reload": true,
   "quality_improvement_claim": false,
   "cut_checkpoint_revision": "376988e3a817b9d7f4ed7eb71ab7007c483b59ce",
-  "training": {
-    "requested_steps": 750,
-    "selected_checkpoint_step": 125,
-    "last_validation_step": 250,
-    "stopped_early": true
-  },
-  "merge_diagnostic": {
-    "original_sampled_verdict": "MERGE_PARTIAL_APPLY",
-    "flagged_tensors_fully_checked": 33,
-    "all_flagged_tensors_changed": true
-  },
   "weights": [
     {
       "path_in_repo": "model.safetensors",
@@ -187,5 +176,103 @@
   ],
   "base_weight_bytes": 13838721960,
   "healed_weight_bytes": 12026140856,
-  "weight_storage_savings_pct": 13.097893788452119
+  "weight_storage_savings_pct": 13.097893788452119,
+  "format_comparison": {
+    "measurement_status": "incomplete",
+    "bf16_representation": "Hugging Face safetensors",
+    "rows": [
+      {
+        "format": "BF16",
+        "size_bytes": 12026140856,
+        "sha256": "e7b3d37beb32930d2c9b1cea775b50f15d79d06dab5d0b39782d206c8ba99573",
+        "published": true,
+        "metrics": [
+          {
+            "benchmark": "arc_science",
+            "metric": "acc",
+            "value": 0.45,
+            "n": 200,
+            "artifact_sha256": "e7b3d37beb32930d2c9b1cea775b50f15d79d06dab5d0b39782d206c8ba99573"
+          },
+          {
+            "benchmark": "bbq_safety",
+            "metric": "acc",
+            "value": 0.395,
+            "n": 200,
+            "artifact_sha256": "e7b3d37beb32930d2c9b1cea775b50f15d79d06dab5d0b39782d206c8ba99573"
+          },
+          {
+            "benchmark": "gpqa_reasoning",
+            "metric": "acc",
+            "value": 0.16666666666666666,
+            "n": 198,
+            "artifact_sha256": "e7b3d37beb32930d2c9b1cea775b50f15d79d06dab5d0b39782d206c8ba99573"
+          },
+          {
+            "benchmark": "gsm8k_math",
+            "metric": "acc",
+            "value": 0.43,
+            "n": 200,
+            "artifact_sha256": "e7b3d37beb32930d2c9b1cea775b50f15d79d06dab5d0b39782d206c8ba99573"
+          },
+          {
+            "benchmark": "hellaswag_commonsense",
+            "metric": "acc",
+            "value": 0.66,
+            "n": 200,
+            "artifact_sha256": "e7b3d37beb32930d2c9b1cea775b50f15d79d06dab5d0b39782d206c8ba99573"
+          },
+          {
+            "benchmark": "ifeval_instruction_following",
+            "metric": "acc",
+            "value": 0.5563380281690141,
+            "n": 142,
+            "artifact_sha256": "e7b3d37beb32930d2c9b1cea775b50f15d79d06dab5d0b39782d206c8ba99573"
+          },
+          {
+            "benchmark": "mmlu_knowledge",
+            "metric": "acc",
+            "value": 0.265,
+            "n": 200,
+            "artifact_sha256": "e7b3d37beb32930d2c9b1cea775b50f15d79d06dab5d0b39782d206c8ba99573"
+          },
+          {
+            "benchmark": "multilingual_knowledge",
+            "metric": "acc",
+            "value": 0.265,
+            "n": 200,
+            "artifact_sha256": "e7b3d37beb32930d2c9b1cea775b50f15d79d06dab5d0b39782d206c8ba99573"
+          },
+          {
+            "benchmark": "wikitext_language",
+            "metric": "ppl",
+            "value": 21.6319395015557,
+            "n": 64,
+            "artifact_sha256": "e7b3d37beb32930d2c9b1cea775b50f15d79d06dab5d0b39782d206c8ba99573"
+          }
+        ]
+      },
+      {
+        "format": "Q8_0",
+        "size_bytes": null,
+        "sha256": "",
+        "published": false,
+        "metrics": []
+      },
+      {
+        "format": "Q4_K_M",
+        "size_bytes": 3663796064,
+        "sha256": "4f7fe283cd9a714a3c96824f30a83e7e4d70e55f16a9b9e0aea8f644e68e620f",
+        "published": false,
+        "metrics": []
+      },
+      {
+        "format": "Q2_K",
+        "size_bytes": null,
+        "sha256": "",
+        "published": false,
+        "metrics": []
+      }
+    ]
+  }
 }