Publish experimental checkpoint measurements; IFEval pending

README.md

@@ -1,3 +1,66 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+base_model: allenai/OLMoE-1B-7B-0125-Instruct
+base_model_relation: finetune
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- moe
+- pruned
+- experimental
+---
+
+# OLMoE-0125-Instruct-Compressed-Experimental
+
+An experimental, modified derivative of [AllenAI's allenai/OLMoE-1B-7B-0125-Instruct](https://huggingface.co/allenai/OLMoE-1B-7B-0125-Instruct), based on revision `b89a7c4bc24fb9e55ce2543c9458ce0ca5c4650e`. Experts were physically removed and the resulting model was further trained. This repository contains the complete healed BF16 Hugging Face checkpoint and its tokenizer.
+
+**Interim results: IFEval is still running for the healed model. Its score is not yet measured and will be added after completion.**
+
+**Observed regression:** Further training reduced sampled GSM8K accuracy from 64.00% (Cut) to 43.00% (Healed). This is an experimental result, not a promoted quality improvement. The cut checkpoint is retained below.
+
+The saved weights occupy 12.03 GB, compared with 13.84 GB for the base: **13.10% smaller**. The healed model contains 6.013 billion parameters. This is a storage reduction, not a claim of faster inference or improved quality.
+
+The model and tokenizer were independently reloaded from the preserved bundle and generated tokens successfully. That execution check is not a quality certification.
+
+The complete cut checkpoint, before further training, is retained at [revision `376988e3`](https://huggingface.co/Omega-Factory/OLMoE-0125-Instruct-Compressed-Experimental/tree/376988e3a817b9d7f4ed7eb71ab7007c483b59ce). Its measurements appear in the Cut column below. Set `revision="376988e3a817b9d7f4ed7eb71ab7007c483b59ce"` in both loading calls to use that checkpoint.
+
+## Measured results
+
+These are measurements from one seed (42) on sampled benchmark tasks. They use local evaluation protocols and should not be compared directly with vendor leaderboard scores. No replicated quality improvement is claimed. The training run used a targeted mathematics corpus.
+
+Training stopped early after validation at step 250; the saved model uses the selected checkpoint at step 125, rather than the requested 750 steps. Evaluate the measured tradeoffs below before using this derivative.
+
+The original merge diagnostic flagged 33 tensors based on small sampled corners. Independent full-tensor comparisons confirmed that every flagged tensor changed; the sampled diagnostic was a false alarm. The original diagnostic is retained in the result summary.
+
+| Benchmark category | Base | Cut | Healed |
+| --- | ---: | ---: | ---: |
+| arc science | 45.50% (n=200) | 44.50% (n=200) | 45.00% (n=200) |
+| bbq safety | 41.50% (n=200) | 42.50% (n=200) | 39.50% (n=200) |
+| gpqa reasoning | 22.73% (n=198) | 19.19% (n=198) | 16.67% (n=198) |
+| gsm8k math | 65.00% (n=200) | 64.00% (n=200) | 43.00% (n=200) |
+| hellaswag commonsense | 67.00% (n=200) | 65.00% (n=200) | 66.00% (n=200) |
+| ifeval instruction following | 61.27% (n=142) | 52.82% (n=142) | Not measured |
+| mmlu knowledge | 30.50% (n=200) | 27.50% (n=200) | 26.50% (n=200) |
+| multilingual knowledge | 28.00% (n=200) | 27.50% (n=200) | 26.50% (n=200) |
+| wikitext language | 18.306 PPL (n=64) | 23.768 PPL (n=64) | 21.632 PPL (n=64) |
+
+WikiText perplexity is lower-is-better. IFEval reports strict prompt accuracy over the supported instruction subset; its coverage and excluded counts are in `evaluation_results.json`. BBQ is multiple-choice accuracy, not a safety certification. These sampled results do not establish general capability, safety, or production readiness.
+
+## Use
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo = "Omega-Factory/OLMoE-0125-Instruct-Compressed-Experimental"
+tokenizer = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto", device_map="auto")
+messages = [{"role": "user", "content": "What is 2 + 2?"}]
+prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
+print(tokenizer.decode(output[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
+```
+
+## License and attribution
+
+The original model is by AllenAI. This modified derivative retains Apache License 2.0; see `LICENSE`. The original authors do not endorse this derivative.

evaluation_results.json

@@ -0,0 +1,183 @@
+{
+  "scope": "experimental_single_seed",
+  "base_model": "allenai/OLMoE-1B-7B-0125-Instruct",
+  "base_revision": "b89a7c4bc24fb9e55ce2543c9458ce0ca5c4650e",
+  "seed": 42,
+  "measurement_status": "in_progress",
+  "pending_categories": {
+    "base": [],
+    "cut": [],
+    "healed": [
+      "ifeval_instruction_following"
+    ]
+  },
+  "measurements": {
+    "arc_science": {
+      "base": {
+        "acc": 0.455,
+        "n": 200
+      },
+      "cut": {
+        "acc": 0.445,
+        "n": 200
+      },
+      "healed": {
+        "acc": 0.45,
+        "n": 200
+      }
+    },
+    "bbq_safety": {
+      "base": {
+        "acc": 0.415,
+        "n": 200
+      },
+      "cut": {
+        "acc": 0.425,
+        "n": 200
+      },
+      "healed": {
+        "acc": 0.395,
+        "n": 200
+      }
+    },
+    "gpqa_reasoning": {
+      "base": {
+        "acc": 0.22727272727272727,
+        "n": 198
+      },
+      "cut": {
+        "acc": 0.1919191919191919,
+        "n": 198
+      },
+      "healed": {
+        "acc": 0.16666666666666666,
+        "n": 198
+      }
+    },
+    "gsm8k_math": {
+      "base": {
+        "acc": 0.65,
+        "n": 200
+      },
+      "cut": {
+        "acc": 0.64,
+        "n": 200
+      },
+      "healed": {
+        "acc": 0.43,
+        "n": 200
+      }
+    },
+    "hellaswag_commonsense": {
+      "base": {
+        "acc": 0.67,
+        "n": 200
+      },
+      "cut": {
+        "acc": 0.65,
+        "n": 200
+      },
+      "healed": {
+        "acc": 0.66,
+        "n": 200
+      }
+    },
+    "ifeval_instruction_following": {
+      "base": {
+        "acc": 0.6126760563380281,
+        "n": 142,
+        "ifeval_coverage": 0.6153846153846154,
+        "ifeval_n_scored": 142,
+        "ifeval_n_excluded": 58,
+        "ifeval_budget": {
+          "max_new": 1024,
+          "n": 200
+        }
+      },
+      "cut": {
+        "acc": 0.528169014084507,
+        "n": 142,
+        "ifeval_coverage": 0.6153846153846154,
+        "ifeval_n_scored": 142,
+        "ifeval_n_excluded": 58,
+        "ifeval_budget": {
+          "max_new": 1024,
+          "n": 200
+        }
+      },
+      "healed": {}
+    },
+    "mmlu_knowledge": {
+      "base": {
+        "acc": 0.305,
+        "n": 200
+      },
+      "cut": {
+        "acc": 0.275,
+        "n": 200
+      },
+      "healed": {
+        "acc": 0.265,
+        "n": 200
+      }
+    },
+    "multilingual_knowledge": {
+      "base": {
+        "acc": 0.28,
+        "n": 200
+      },
+      "cut": {
+        "acc": 0.275,
+        "n": 200
+      },
+      "healed": {
+        "acc": 0.265,
+        "n": 200
+      }
+    },
+    "wikitext_language": {
+      "base": {
+        "ppl": 18.305546793311198,
+        "n": 64
+      },
+      "cut": {
+        "ppl": 23.76837327292051,
+        "n": 64
+      },
+      "healed": {
+        "ppl": 21.6319395015557,
+        "n": 64
+      }
+    }
+  },
+  "source_measurement_sha256": {
+    "base": "ec64b19dd0644e3ec6dff0daeb3f42df443d220fa16df73677b7f5e74ffe79dd",
+    "cut": "197c37ff224cf1b640017b9814f3cadb7d7123b1e059523c80ed4aaff49cfe49",
+    "healed": "c548c7ec35af9264affa51cfd59459fb6477f1cec0589d0ffb8e92ee18b04cfe"
+  },
+  "healed_parameters": 6012897280,
+  "standalone_model_and_tokenizer_reload": true,
+  "quality_improvement_claim": false,
+  "cut_checkpoint_revision": "376988e3a817b9d7f4ed7eb71ab7007c483b59ce",
+  "training": {
+    "requested_steps": 750,
+    "selected_checkpoint_step": 125,
+    "last_validation_step": 250,
+    "stopped_early": true
+  },
+  "merge_diagnostic": {
+    "original_sampled_verdict": "MERGE_PARTIAL_APPLY",
+    "flagged_tensors_fully_checked": 33,
+    "all_flagged_tensors_changed": true
+  },
+  "weights": [
+    {
+      "path_in_repo": "model.safetensors",
+      "sha256": "e7b3d37beb32930d2c9b1cea775b50f15d79d06dab5d0b39782d206c8ba99573",
+      "size": 12026140856
+    }
+  ],
+  "base_weight_bytes": 13838721960,
+  "healed_weight_bytes": 12026140856,
+  "weight_storage_savings_pct": 13.097893788452119
+}