| --- |
| language: |
| - en |
| license: apache-2.0 |
| base_model: mistralai/Mistral-Nemo-Instruct-2407 |
| tags: |
| - self-improving |
| - autonomous |
| - jblaze |
| - prometheus |
| - weight-surgery |
| - mistral |
| - research |
| pipeline_tag: text-generation |
| --- |
| |
| # Prometheus-Mistral-Nemo-12B |
|
|
| **Mistral-Nemo-Instruct-2407 (12B parameters) after autonomous self-improvement via Project Prometheus.** |
|
|
| The model evaluated itself, identified its weakest behavioral areas, proposed targeted improvements, applied permanent weight-level modifications using [jBlaze](https://jblaze.dev), and validated each modification against functional benchmarks. Only modifications that survived the benchmark gate were kept. No human selected the winning modifications. No human decided whether a change should survive. |
|
|
| ## What Changed |
|
|
| Prometheus targets the gap between what a model *can* reason and what it *does* reason by default. The modifications improve epistemic discipline -- the model's tendency to check its own assumptions, flag uncertainty, detect adversarial framing, and consider alternatives before committing to an answer. |
|
|
| Targeted improvements: |
|
|
| - **Reasoning discipline** -- Better stop-condition awareness, tradeoff analysis, and alternative solution generation. The model is less likely to latch onto the first answer that pattern-matches and more likely to verify. |
| - **Epistemic honesty** -- Improved uncertainty awareness and speculation labeling. When the model isn't sure, it says so. When it speculates, it distinguishes that from established fact. |
| - **Adversarial robustness** -- Stronger detection of misleading inputs, unsupported claims, and adversarial framing. The model catches trick questions and bait more reliably. |
| - **Self-correction** -- Better tracking of when its own reasoning needs amendment. The model revises mid-answer rather than committing to a wrong path. |
|
|
| ## The Bat and Ball Test |
|
|
| The original Prometheus v3 release (documented below) failed the classic cognitive reflection test. It gave the fast intuitive answer ($0.10) with confidence instead of the slow correct answer ($0.05) with verification. That failure was the motivation for adding functional benchmark gates. |
|
|
| This release passes the bat-and-ball test and all its variants: |
|
|
| **Question:** *A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?* |
|
|
| **Prometheus (current):** Sets up the equation, solves step by step, arrives at $0.05. Correct. |
|
|
| The model also handles novel CRT variants it was never tested against during validation ($2.20/$2.00, $3.30/$3.00, different item names), confirming the improvement generalizes rather than memorizing specific answers. |
|
|
| ## Reasoning Benchmark |
|
|
| Tested against 50 adversarial trick questions -- CRT variants, cognitive bias traps, logic puzzles, math traps, probability paradoxes, and framing attacks: |
|
|
| | Model | Score | Loops | |
| |:------|------:|------:| |
| | Vanilla Mistral-Nemo | 44/50 (88%) | 0 | |
| | Prometheus | 43/50 (86%) | 0 | |
|
|
| The remaining failures are shared with vanilla -- edge cases like probability paradoxes and anchoring traps that the base model also gets wrong. No new regressions were introduced by the modifications. Quality of long-form responses (explanations, comparisons, technical writing) is indistinguishable from vanilla. |
|
|
| ## How Prometheus Works |
|
|
| 1. **Self-evaluation** -- The model evaluates itself across behavioral categories including reasoning depth, uncertainty handling, hallucination resistance, adversarial robustness, and epistemic honesty. |
| 2. **Memory** -- Results, failed strategies, and accepted changes are written into deterministic memory, giving later iterations a persistent record of what has been tried and what happened. |
| 3. **Planning** -- The model reviews its scores and history, identifies its weakest area, and selects a specific improvement strategy. It avoids repeating strategies that previously failed. |
| 4. **Surgery** -- jBlaze applies the selected modification to a cloned copy of the model's weights. |
| 5. **Validation** -- The clone is benchmarked against the current model on functional tests including multi-step reasoning, trick questions, and quality checks. Modifications that degrade reasoning are rejected regardless of how strongly they shift the target behavior. |
| 6. **Promotion or rejection** -- If the clone passes the benchmark gate, it becomes the active model. If it degrades capability, it is rejected and the result is preserved in memory. |
|
|
| ## What Changed Since v3 |
|
|
| The original Prometheus release used mechanical validation only -- it checked whether the weight surgery succeeded, not whether the model actually improved. That release was published as a research artifact demonstrating that mechanical validation alone produces style changes without reasoning gains. |
|
|
| This release adds functional benchmark gates at every step. Each proposed modification must pass a battery of known-answer reasoning tests, multi-step problem solving, and quality checks before it is accepted. The result: behavioral improvements that hold up under adversarial testing without sacrificing the base model's reasoning capability. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained("ApolloRaines/Prometheus-Mistral-Nemo-12B") |
| tokenizer = AutoTokenizer.from_pretrained("ApolloRaines/Prometheus-Mistral-Nemo-12B") |
| |
| messages = [{"role": "user", "content": "Who are you?"}] |
| text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = tokenizer(text, return_tensors="pt").to(model.device) |
| output = model.generate(**inputs, max_new_tokens=200) |
| print(tokenizer.decode(output[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ## Weight Surgery Visualization |
|
|
| Interactive 3D map of weight-level modifications across all 40 transformer layers. Rotate, zoom, and hover to inspect where the accepted modifications landed. |
|
|
| <iframe |
| src="https://huggingface.co/spaces/ApolloRaines/prometheus-weight-diff" |
| frameborder="0" |
| width="100%" |
| height="1400" |
| ></iframe> |
|
|
| ## About jBlaze |
|
|
| jBlaze is a proprietary weight-surgery system for making permanent changes directly to trained model parameters. The resulting changes are written into the model weights themselves rather than added as a runtime prompt or adapter. |
|
|
| jBlaze is not open source. More at [jblaze.dev](https://jblaze.dev) |
|
|
| ## See Also |
|
|
| - [Prometheus-Qwen2.5-7B-Instruct](https://huggingface.co/ApolloRaines/Prometheus-Qwen2.5-7B-Instruct) -- The first Prometheus run on a smaller architecture |
| - [Pythia-1.4B-DNP-16750-Facts](https://huggingface.co/ApolloRaines/Pythia-1.4B-DNP-16750-Facts) -- 16,750 facts written into a 1.4B model via Direct Neural Programming |
| - [Pythia-1.4B-DNP-5000-Facts](https://huggingface.co/ApolloRaines/Pythia-1.4B-DNP-5000-Facts) -- 5,000 post-2023 facts, general capability improved to 70% |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{prometheus-mistral-nemo-2026, |
| title={Prometheus-Mistral-Nemo-12B: Autonomous Self-Improvement via Weight Surgery}, |
| author={Apollo Raines}, |
| year={2026}, |
| url={https://huggingface.co/ApolloRaines/Prometheus-Mistral-Nemo-12B} |
| } |
| ``` |
|
|
| ## Contact |
|
|
| [jblaze.dev](https://jblaze.dev) |
|
|
|
|
