README.md

---
license: apache-2.0
language:
  - code
tags:
  - code
  - fill-in-the-middle
  - multi-token-prediction
  - speculative-decoding
  - mlx
  - apple-silicon
library_name: transformers
pipeline_tag: text-generation
---

<div id="wisp-experience" style="color:#354b3b;background:#f4f1e8;padding:clamp(18px,3vw,38px);margin:8px 0 0;position:relative;background-image:radial-gradient(#d7dbc9 0.55px,transparent 0.55px);background-size:8px 8px;font-size:15px;line-height:1.8;overflow-wrap:anywhere;--wisp-gutter:calc(clamp(0px, calc(100vw - 768px), 16px) + clamp(0px, calc(100vw - 1024px), 56px) + clamp(0px, calc(100vw - 1280px), 16px) + clamp(0px, calc(100vw - 1536px), 48px));width:fit-content;min-width:100%;max-width:calc(100% + var(--wisp-gutter));box-sizing:border-box;"><!-- The Wisp Fieldbook: a native Hugging Face model card. -->
<div class="group" id="wisp-cover" style="color:#263f32;padding:0;margin:0;background:transparent;">
<div style="display:flex;justify-content:space-between;align-items:center;flex-wrap:wrap;gap:12px;padding-bottom:18px;margin-bottom:36px;background:transparent;border-bottom:1px solid #bfccb1;">
<h1 id="wisp-coder-110m" style="font-family:Arial,sans-serif;font-size:13px;font-weight:600;letter-spacing:1.4px;text-transform:uppercase;color:#263f32;margin:0;line-height:1.6;">Wisp Coder 110M</h1>
<span style="font-family:Georgia,serif;font-size:13px;font-style:italic;color:#8e3049;">The Fieldbook / No. 001</span>
</div>
<div style="display:flex;align-items:center;flex-wrap:wrap;gap:20px;background:transparent;">
<div style="flex:1 1 250px;min-width:0;background:transparent;">
<p style="font-size:10px;letter-spacing:1.5px;color:#5d704f;margin:0 0 18px;">CODE · CONTEXT · CURIOSITY</p>
<p style="font-family:Georgia,serif;font-size:clamp(62px,7vw,84px);font-weight:400;letter-spacing:-4px;line-height:1;color:#263f32;margin:0 0 24px;">Mind<br/><em style="color:#ad3f5a;">the gap.</em></p>
<p style="font-family:Georgia,serif;font-size:20px;line-height:1.5;color:#3b5041;margin:0 0 24px;">A small code model.<br/>A little room for possibility.</p>
<a class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:scale-[1.01] hover:brightness-[102%] focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" href="https://huggingface.co/spaces/philipjohnbasile/wisp-coder-demo" style="display:inline-block;color:#fffdf5;padding:13px 19px;text-decoration:none;font-size:14px;font-weight:600;text-decoration-thickness:1px;text-underline-offset:3px;background:#263f32;border-bottom:3px solid #a1b68c;">Enter the playground <span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:translate-x-0.5" style="display:inline-block;">↗</span></a>
<p style="font-size:11px;line-height:1.7;color:#50614e;margin:22px 0 0;">An open research experiment<br/>by Philip John Basile</p>
</div>
<div style="flex:1.15 1 270px;min-width:0;max-width:100%;padding:18px 14px;background:transparent;">
<img alt="Original painted wildflowers in pink, coral, cobalt and green. They gently turn when you hover over the cover." class="wisp-botanical motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:scale-105 group-hover:rotate-6" src="./assets/wisp-wildflowers.png" style="width:100%;max-width:100%;height:auto;object-fit:contain;margin:0;border:0;"/>
<p style="font-family:Georgia,serif;font-style:italic;color:#5d704f;font-size:14px;text-align:right;margin:20px 5px 0;">Small model. Open questions.</p>
</div>
</div>
<div style="display:flex;flex-wrap:wrap;gap:14px 25px;margin-top:34px;padding-top:22px;background:transparent;border-top:1px solid #bfccb1;">
<div style=";background:transparent;"><span style="font-family:Georgia,serif;font-size:26px;color:#263f32;">100.7M</span><br/><span style="font-size:10px;letter-spacing:1px;color:#50614e;">TRUNK PARAMETERS</span></div>
<div style=";background:transparent;"><span style="font-family:Georgia,serif;font-size:26px;color:#ad3f5a;">108.2M</span><br/><span style="font-size:10px;letter-spacing:1px;color:#50614e;">INCLUDING MTP</span></div>
<div style=";background:transparent;"><span style="font-family:Georgia,serif;font-size:26px;color:#263f32;">2,048</span><br/><span style="font-size:10px;letter-spacing:1px;color:#50614e;">TOKEN CONTEXT</span></div>
</div>
</div>
<div id="wisp-index" style="display:flex;flex-wrap:wrap;gap:10px 18px;margin:0;background:#f4f1e8;padding:16px 0;border-bottom:1px solid #bfccb1;">
<a class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" href="#start-here" style="display:inline-block;padding:0;color:#354b3b;font-size:12px;text-decoration:none;text-decoration-thickness:1px;text-underline-offset:3px;border:0;"><span style="color:#ad3f5a;">01</span>   Start here</a>
<a class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" href="#architecture" style="display:inline-block;padding:0;color:#354b3b;font-size:12px;text-decoration:none;text-decoration-thickness:1px;text-underline-offset:3px;border:0;"><span style="color:#ad3f5a;">02</span>   Architecture</a>
<a class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" href="#usage" style="display:inline-block;padding:0;color:#354b3b;font-size:12px;text-decoration:none;text-decoration-thickness:1px;text-underline-offset:3px;border:0;"><span style="color:#ad3f5a;">03</span>   Usage</a>
<a class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" href="#evaluation" style="display:inline-block;padding:0;color:#354b3b;font-size:12px;text-decoration:none;text-decoration-thickness:1px;text-underline-offset:3px;border:0;"><span style="color:#ad3f5a;">04</span>   Evidence</a>
<a class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" href="#limitations" style="display:inline-block;padding:0;color:#354b3b;font-size:12px;text-decoration:none;text-decoration-thickness:1px;text-underline-offset:3px;border:0;"><span style="color:#ad3f5a;">05</span>   Limits</a>
</div><div id="wisp-path" style="margin:0 0 0 6px;padding:0 0 0 clamp(14px,2vw,23px);border-left:1px solid #becdad;">

<div style="margin:0 0 24px;padding:44px 0 0;background:transparent;position:relative;"><span aria-hidden="true" style="position:absolute;left:calc(-1 * clamp(14px,2vw,23px) - 7px);top:43px;color:#ad3f5a;background:#f4f1e8;font-size:14px;line-height:1;">❋</span>
<p style="margin:0 0 12px;color:#9f3d58;font-size:10px;letter-spacing:1.6px;text-transform:uppercase;">01 / START HERE</p>
<h2 id="start-here" style="font-family:Georgia,serif;font-size:clamp(32px,4.3vw,46px);line-height:1.08;font-weight:400;letter-spacing:-1.6px;color:#263f32;margin:0;padding:0;border:0;scroll-margin-top:24px;">At the cursor.</h2>
</div>

<div id="wisp-cursor" style="color:#263f32;padding:0;margin:0 0 30px;background:transparent;">
<p style="font-size:9px;letter-spacing:1.1px;color:#64775a;margin:0 0 9px;">THE CODE BEFORE. THE POSSIBILITY WITHIN. THE CODE AFTER.</p>


<div style="display:flex;align-items:flex-start;flex-wrap:wrap;gap:10px;background:transparent;">
<details class="group" id="wisp-prefix" open="" style="flex:1 1 150px;min-width:0;color:#263f32;padding:14px 10px 0 0;border-top:2px solid #b7c8a6;">
<summary class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" style="cursor:pointer;list-style:none;padding:0;color:#263f32;">
<span style="display:block;font-size:10px;letter-spacing:1px;color:#354b3b;margin:0 0 10px;">YOU PROVIDE</span>
<span style="display:flex;align-items:center;justify-content:space-between;font-family:Georgia,serif;font-size:24px;color:#354b3b;margin:0 0 6px;">Prefix<span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-open:rotate-180" style="display:inline-block;font-family:Arial,sans-serif;font-size:18px;color:#354b3b;">⌄</span></span>
<span style="display:block;font-size:13px;line-height:1.6;color:#354b3b;">Code before the gap</span>
</summary>
<div style="padding:16px 0 0;background:transparent;"><p style="font-size:13px;line-height:1.7;color:#354b3b;margin:0 0 12px;">The source context before the missing span.</p><code style="display:block;overflow-wrap:anywhere;white-space:normal;background:#e8eddf;color:#263f32;font-size:11px;padding:2px 5px;font-weight:500;">&lt;|fim_prefix|&gt;</code></div>
</details>
<details class="group" id="wisp-middle" open="" style="flex:1 1 150px;min-width:0;color:#263f32;padding:14px 10px 0 0;border-top:2px solid #ad3f5a;">
<summary class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" style="cursor:pointer;list-style:none;padding:0;color:#263f32;">
<span style="display:block;font-size:10px;letter-spacing:1px;color:#963c54;margin:0 0 10px;">WISP SUGGESTS</span>
<span style="display:flex;align-items:center;justify-content:space-between;font-family:Georgia,serif;font-size:24px;color:#963c54;margin:0 0 6px;">Middle<span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-open:rotate-180" style="display:inline-block;font-family:Arial,sans-serif;font-size:18px;color:#963c54;">⌄</span></span>
<span style="display:block;font-size:13px;line-height:1.6;color:#963c54;">The missing span</span>
</summary>
<div style="padding:16px 0 0;background:transparent;"><p style="font-size:13px;line-height:1.7;color:#354b3b;margin:0 0 12px;">The model predicts this span using both sides of the gap. Review its suggestion before use.</p><code style="display:block;overflow-wrap:anywhere;white-space:normal;background:#e8eddf;color:#263f32;font-size:11px;padding:2px 5px;font-weight:500;">&lt;|fim_middle|&gt;</code></div>
</details>
<details class="group" id="wisp-suffix" open="" style="flex:1 1 150px;min-width:0;color:#263f32;padding:14px 10px 0 0;border-top:2px solid #b7c8a6;">
<summary class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" style="cursor:pointer;list-style:none;padding:0;color:#263f32;">
<span style="display:block;font-size:10px;letter-spacing:1px;color:#354b3b;margin:0 0 10px;">YOU PROVIDE</span>
<span style="display:flex;align-items:center;justify-content:space-between;font-family:Georgia,serif;font-size:24px;color:#354b3b;margin:0 0 6px;">Suffix<span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-open:rotate-180" style="display:inline-block;font-family:Arial,sans-serif;font-size:18px;color:#354b3b;">⌄</span></span>
<span style="display:block;font-size:13px;line-height:1.6;color:#354b3b;">Code after the gap</span>
</summary>
<div style="padding:16px 0 0;background:transparent;"><p style="font-size:13px;line-height:1.7;color:#354b3b;margin:0 0 12px;">The source context after the missing span. In PSM prompt order, this comes before the generated middle.</p><code style="display:block;overflow-wrap:anywhere;white-space:normal;background:#e8eddf;color:#263f32;font-size:11px;padding:2px 5px;font-weight:500;">&lt;|fim_suffix|&gt;</code></div>
</details>
</div>
<p style="font-size:13px;line-height:1.7;color:#354b3b;margin:18px 0 0;">Provide the code on both sides of a gap. Wisp predicts the middle. <a href="https://huggingface.co/spaces/philipjohnbasile/wisp-coder-demo" style="color:#963c54;text-decoration:underline;text-decoration-thickness:1px;text-underline-offset:3px;">Explore it in the playground ↗</a></p>
</div>
<p style="font-size:10px;letter-spacing:1px;margin:30px 0 0;color:#64775a;">A FEW THINGS TO KNOW ALONG THE WAY</p>
<div style="display:block;grid-template-columns:repeat(auto-fit,minmax(min(100%,250px),1fr));align-items:start;gap:14px;margin:0 0 30px;background:transparent;">
<details class="group" id="wisp-note-1" open="" style="color:#263f32;padding:18px 0 0;margin:0;border-bottom:1px solid #d4ddc8;">
<summary class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" style="cursor:pointer;list-style:none;padding:0 0 10px;color:#263f32;">
<span style="display:flex;justify-content:space-between;align-items:center;font-size:14px;letter-spacing:1.6px;color:#8e3049;margin:0;float:right;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-open:rotate-180" style="font-size:17px;display:inline-block;">⌄</span></span>
<span style="display:block;font-family:Georgia,serif;font-size:22px;font-weight:400;line-height:1.3;color:#263f32;">What is it for?</span>
</summary>
<div style="padding:0 0 18px;background:transparent;"><p style="font-size:14px;line-height:1.75;color:#354b3b;margin:0;border-top:1px dashed #aebaa0;padding-top:15px;padding:0;border:0;">Research on code completion, fill-in-the-middle (FIM), and native multi-token prediction (MTP).</p></div>
</details>
<details class="group" id="wisp-note-2" open="" style="color:#263f32;padding:18px 0 0;margin:0;border-bottom:1px solid #d4ddc8;">
<summary class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" style="cursor:pointer;list-style:none;padding:0 0 10px;color:#263f32;">
<span style="display:flex;justify-content:space-between;align-items:center;font-size:14px;letter-spacing:1.6px;color:#8e3049;margin:0;float:right;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-open:rotate-180" style="font-size:17px;display:inline-block;">⌄</span></span>
<span style="display:block;font-family:Georgia,serif;font-size:22px;font-weight:400;line-height:1.3;color:#263f32;">What did I build?</span>
</summary>
<div style="padding:0 0 18px;background:transparent;"><p style="font-size:14px;line-height:1.75;color:#354b3b;margin:0;border-top:1px dashed #aebaa0;padding-top:15px;padding:0;border:0;">Philip John Basile's model trained from step zero: 100.7M trunk parameters, 108.2M including the MTP module.</p></div>
</details>
<details class="group" id="wisp-note-3" open="" style="color:#263f32;padding:18px 0 0;margin:0;border-bottom:1px solid #d4ddc8;">
<summary class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" style="cursor:pointer;list-style:none;padding:0 0 10px;color:#263f32;">
<span style="display:flex;justify-content:space-between;align-items:center;font-size:14px;letter-spacing:1.6px;color:#8e3049;margin:0;float:right;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-open:rotate-180" style="font-size:17px;display:inline-block;">⌄</span></span>
<span style="display:block;font-family:Georgia,serif;font-size:22px;font-weight:400;line-height:1.3;color:#263f32;">What runs it?</span>
</summary>
<div style="padding:0 0 18px;background:transparent;"><p style="font-size:14px;line-height:1.75;color:#354b3b;margin:0;border-top:1px dashed #aebaa0;padding-top:15px;padding:0;border:0;">The trunk uses standard Transformers or MLX-LM. MTP requires the explicit packaged MLX reference runtime; generic loaders do not activate the sidecar.</p></div>
</details>
<details class="group" id="wisp-note-4" open="" style="color:#263f32;padding:18px 0 0;margin:0;border-bottom:1px solid #d4ddc8;">
<summary class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" style="cursor:pointer;list-style:none;padding:0 0 10px;color:#263f32;">
<span style="display:flex;justify-content:space-between;align-items:center;font-size:14px;letter-spacing:1.6px;color:#8e3049;margin:0;float:right;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-open:rotate-180" style="font-size:17px;display:inline-block;">⌄</span></span>
<span style="display:block;font-family:Georgia,serif;font-size:22px;font-weight:400;line-height:1.3;color:#263f32;">What was checked recently?</span>
</summary>
<div style="padding:0 0 18px;background:transparent;"><p style="font-size:14px;line-height:1.75;color:#354b3b;margin:0;border-top:1px dashed #aebaa0;padding-top:15px;padding:0;border:0;">CPU float32 completions with PyTorch 2.13.0 and Transformers 5.14.1 on 2026-09-10. These are functional smokes, not a coding-quality or latency benchmark.</p></div>
</details>
<details class="group" id="wisp-note-5" open="" style="color:#263f32;padding:18px 0 0;margin:0;border-bottom:1px solid #d4ddc8;">
<summary class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" style="cursor:pointer;list-style:none;padding:0 0 10px;color:#263f32;">
<span style="display:flex;justify-content:space-between;align-items:center;font-size:14px;letter-spacing:1.6px;color:#8e3049;margin:0;float:right;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-open:rotate-180" style="font-size:17px;display:inline-block;">⌄</span></span>
<span style="display:block;font-family:Georgia,serif;font-size:22px;font-weight:400;line-height:1.3;color:#263f32;">How much memory?</span>
</summary>
<div style="padding:0 0 18px;background:transparent;"><p style="font-size:14px;line-height:1.75;color:#354b3b;margin:0;border-top:1px dashed #aebaa0;padding-top:15px;padding:0;border:0;">Trunk weight file: about 201 MB in BF16, or about 403 MB as FP32 tensors. Runtime overhead and the KV cache require additional RAM; these are weight sizes, not total-memory requirements.</p></div>
</details>
<details class="group" id="wisp-note-6" open="" style="color:#263f32;padding:18px 0 0;margin:0;border-bottom:1px solid #d4ddc8;">
<summary class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" style="cursor:pointer;list-style:none;padding:0 0 10px;color:#263f32;">
<span style="display:flex;justify-content:space-between;align-items:center;font-size:14px;letter-spacing:1.6px;color:#8e3049;margin:0;float:right;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-open:rotate-180" style="font-size:17px;display:inline-block;">⌄</span></span>
<span style="display:block;font-family:Georgia,serif;font-size:22px;font-weight:400;line-height:1.3;color:#263f32;">What are the limits?</span>
</summary>
<div style="padding:0 0 18px;background:transparent;"><p style="font-size:14px;line-height:1.75;color:#354b3b;margin:0;border-top:1px dashed #aebaa0;padding-top:15px;padding:0;border:0;">2,048-token context; limited completion quality; no chat tuning. The registered short-hole comparison established no superiority, and no production MTP speedup is claimed.</p></div>
</details>
</div>
<h3 id="quick-start--ordinary-completion-on-cpu" style="color:#263f32;font-family:Georgia,serif;font-size:25px;font-weight:400;line-height:1.3;margin:30px 0 16px;">Quick start — ordinary completion on CPU</h3>
<details class="group" open="" style="margin:24px 0;padding:0;color:#263f32;border-top:1px solid #bdcdb0;">
<summary class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" style="cursor:pointer;padding:14px 0;color:#536d43;font-family:Georgia,serif;font-size:19px;background:transparent;">Run an ordinary completion on CPU</summary>
<div style="padding:0;background:transparent;">


```bash
pip install "torch==2.13.0" "transformers==5.14.1"
```




```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "philipjohnbasile/wisp-coder-110m"
revision = "186479fd8c1eb767e0d886014fb78b3426dc509a"  # pinned weights/tokenizer
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision)
model = AutoModelForCausalLM.from_pretrained(
    repo, revision=revision, dtype=torch.float32, trust_remote_code=False,
).eval()
inputs = tokenizer("def is_even(n):\n    return ", return_tensors="pt", add_special_tokens=False)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False,
                            pad_token_id=tokenizer.pad_token_id)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```


</div>
</details>
<p style="color:#354b3b;">Review generated code before using it. The demo also provides FIM prompts and displays raw suggestions without executing them. The full training, data-provenance, and evaluation disclosures follow.</p>
<h3 id="the-idea" style="color:#263f32;font-family:Georgia,serif;font-size:25px;font-weight:400;line-height:1.3;margin:30px 0 16px;">The idea</h3>
<p style="color:#354b3b;">A small code model built for one job: completing code at the cursor. It targets
editor inline suggestions, but no production-runtime latency claim is made.</p>
<p style="color:#354b3b;">Two things make it unusual, and they are the reason it exists rather than a
feature list.</p>
<p style="color:#354b3b;"><strong style="color:#263f32;">It does fill in the middle natively.</strong> Each tokenized source document was split
into chunks of at most 1024 tokens, and 70 percent of those chunks were
independently transformed so the model predicts a middle span given both the
prefix and the suffix. This is not a per-document or per-window rate. Cursor
completion is the primary format, not an inference-time prompt trick added to an
ordinary left-to-right checkpoint.</p>
<p style="color:#354b3b;"><strong style="color:#263f32;">It has multi-token-prediction heads trained in from step zero.</strong> A shared MTP
module, applied recursively in the Qwen3-Next style, predicts two or more tokens
ahead from the trunk's own hidden state. The LM head is shared between trunk and
MTP module, which ties the two output distributions to a common projection. That
is a useful architectural choice, not a guarantee that draft and target
distributions agree; how close they actually are is what acceptance measures.</p>
<p style="color:#354b3b;"><strong style="color:#263f32;">What is and is not claimed here.</strong> Small models with MTP drafters exist: Google
publishes <code style="color:#263f32;background:#e8eddf;overflow-wrap:anywhere;white-space:normal;font-weight:500;padding:2px 5px;">gemma-4-E2B-it-assistant</code> and related MTP assistant checkpoints. The
artifact contribution is the <em>combination</em> at this scale, a small code model
trained from step zero with both fill-in-the-middle on most tokenized chunks and a
recursively shared MTP module, plus a paired measurement of how suffix information
changes draft acceptance. A targeted search found no exact prior measurement, but
that is not proof of absence. No exclusivity or first-of-its-kind claim is made.</p>
<div style="margin:0 0 24px;padding:44px 0 0;background:transparent;position:relative;"><span aria-hidden="true" style="position:absolute;left:calc(-1 * clamp(14px,2vw,23px) - 7px);top:43px;color:#ad3f5a;background:#f4f1e8;font-size:14px;line-height:1;">❋</span>
<p style="margin:0 0 12px;color:#9f3d58;font-size:10px;letter-spacing:1.6px;text-transform:uppercase;">02 / WHAT IT IS NOT</p>
<h2 id="what-it-is-not" style="font-family:Georgia,serif;font-size:clamp(32px,4.3vw,46px);line-height:1.08;font-weight:400;letter-spacing:-1.6px;color:#263f32;margin:0;padding:0;border:0;scroll-margin-top:24px;">Keep the expectations honest.</h2>
</div>
<p style="color:#354b3b;">It is not a chat model, an instruction-following model, or an agent. It completes
text. Asking it to refactor a module produces nonsense.</p>
<p style="color:#354b3b;">It is also not competitive with Qwen2.5-Coder on raw completion quality, and the
arithmetic says it cannot be: Qwen2.5-Coder-0.5B saw roughly 5.5 trillion tokens
and Wisp saw 5 billion, three orders of magnitude fewer. If you want the best
completions available at small scale, use Qwen. Use Wisp to study FIM plus native
speculative drafting on Apple Silicon, or how those two interact.</p>
<div class="group" style="margin:0 0 24px;padding:44px 0 0;background:transparent;position:relative;display:flow-root;"><span aria-hidden="true" style="position:absolute;left:calc(-1 * clamp(14px,2vw,23px) - 7px);top:43px;color:#ad3f5a;background:#f4f1e8;font-size:14px;line-height:1;">❋</span><img alt="" class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:scale-105 group-hover:-rotate-6" src="./assets/wisp-wildflowers.png" style="width:180px;max-width:45%;height:auto;object-fit:contain;float:right;margin:-5px 0 12px 15px;border:0;"/>
<p style="margin:0 0 12px;color:#9f3d58;font-size:10px;letter-spacing:1.6px;text-transform:uppercase;">03 / ARCHITECTURE</p>
<h2 id="architecture" style="font-family:Georgia,serif;font-size:clamp(32px,4.3vw,46px);line-height:1.08;font-weight:400;letter-spacing:-1.6px;color:#263f32;margin:0;padding:0;border:0;scroll-margin-top:24px;">Under the petals.</h2>
</div>
<div id="wisp-anatomy" style="margin:8px 0 30px;padding:0 0 24px;border-bottom:1px solid #bdcdb0;">
<p style="font-family:Georgia,serif;font-size:21px;color:#354b3b;margin:0 0 22px;">Twelve layers. One shared module. A little further ahead.</p>
<div style="display:flex;flex-wrap:wrap;align-items:flex-start;gap:28px;">
<div class="group" style="flex:1.7 1 280px;min-width:0;">
<div aria-hidden="true" style="display:grid;grid-template-columns:repeat(12,minmax(0,1fr));gap:5px;"><div style="min-width:0;text-align:center;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:-translate-y-px" style="display:block;height:72px;border:1px solid #9db18f;border-radius:22px 22px 3px 3px;background:#dfe6d5;"></span><span style="display:block;color:#536747;font-size:9px;margin-top:7px;">01</span></div><div style="min-width:0;text-align:center;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:-translate-y-px" style="display:block;height:72px;border:1px solid #9db18f;border-radius:22px 22px 3px 3px;background:#dfe6d5;"></span><span style="display:block;color:#536747;font-size:9px;margin-top:7px;">02</span></div><div style="min-width:0;text-align:center;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:-translate-y-px" style="display:block;height:72px;border:1px solid #9db18f;border-radius:22px 22px 3px 3px;background:#bccdab;"></span><span style="display:block;color:#536747;font-size:9px;margin-top:7px;">03</span></div><div style="min-width:0;text-align:center;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:-translate-y-px" style="display:block;height:72px;border:1px solid #9db18f;border-radius:22px 22px 3px 3px;background:#dfe6d5;"></span><span style="display:block;color:#536747;font-size:9px;margin-top:7px;">04</span></div><div style="min-width:0;text-align:center;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:-translate-y-px" style="display:block;height:72px;border:1px solid #9db18f;border-radius:22px 22px 3px 3px;background:#dfe6d5;"></span><span style="display:block;color:#536747;font-size:9px;margin-top:7px;">05</span></div><div style="min-width:0;text-align:center;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:-translate-y-px" style="display:block;height:72px;border:1px solid #9db18f;border-radius:22px 22px 3px 3px;background:#bccdab;"></span><span style="display:block;color:#536747;font-size:9px;margin-top:7px;">06</span></div><div style="min-width:0;text-align:center;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:-translate-y-px" style="display:block;height:72px;border:1px solid #9db18f;border-radius:22px 22px 3px 3px;background:#dfe6d5;"></span><span style="display:block;color:#536747;font-size:9px;margin-top:7px;">07</span></div><div style="min-width:0;text-align:center;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:-translate-y-px" style="display:block;height:72px;border:1px solid #9db18f;border-radius:22px 22px 3px 3px;background:#dfe6d5;"></span><span style="display:block;color:#536747;font-size:9px;margin-top:7px;">08</span></div><div style="min-width:0;text-align:center;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:-translate-y-px" style="display:block;height:72px;border:1px solid #9db18f;border-radius:22px 22px 3px 3px;background:#bccdab;"></span><span style="display:block;color:#536747;font-size:9px;margin-top:7px;">09</span></div><div style="min-width:0;text-align:center;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:-translate-y-px" style="display:block;height:72px;border:1px solid #9db18f;border-radius:22px 22px 3px 3px;background:#dfe6d5;"></span><span style="display:block;color:#536747;font-size:9px;margin-top:7px;">10</span></div><div style="min-width:0;text-align:center;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:-translate-y-px" style="display:block;height:72px;border:1px solid #9db18f;border-radius:22px 22px 3px 3px;background:#dfe6d5;"></span><span style="display:block;color:#536747;font-size:9px;margin-top:7px;">11</span></div><div style="min-width:0;text-align:center;"><span class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:-translate-y-px" style="display:block;height:72px;border:1px solid #9db18f;border-radius:22px 22px 3px 3px;background:#bccdab;"></span><span style="display:block;color:#536747;font-size:9px;margin-top:7px;">12</span></div></div>
<p style="color:#263f32;font-size:12px;margin:15px 0 3px;letter-spacing:1px;">12 TRUNK LAYERS</p>
<p style="color:#50614e;font-size:12px;line-height:1.7;margin:0;">768 hidden dimensions · 12 attention heads<br/>4 KV heads · SwiGLU 2,048</p>
</div>
<div class="group" style="flex:1 1 180px;min-width:0;">
<div style="display:flex;align-items:center;gap:15px;min-height:92px;">
<span aria-hidden="true" class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:rotate-6" style="display:flex;align-items:center;justify-content:center;flex:0 0 72px;height:72px;border:1px solid #b3677a;border-radius:50%;font-family:Georgia,serif;font-size:44px;line-height:1;color:#ad3f5a;">1</span>
<span style="font-family:Georgia,serif;font-size:23px;line-height:1.2;color:#963c54;">Shared.<br/><em>Reused.</em></span>
</div>
<p style="color:#963c54;font-size:12px;margin:15px 0 3px;letter-spacing:1px;">1 MTP TRANSFORMER BLOCK</p>
<p style="color:#50614e;font-size:12px;line-height:1.7;margin:0;">Trained to depth 2.<br/>Applied recursively at inference.</p>
</div>
</div>
<p style="color:#50614e;font-size:12px;line-height:1.7;margin:22px 0 0;">The trunk and MTP module share the LM head. The sidecar requires the explicit MTP runtime below.</p>
</div>
<div style="display:grid;grid-template-columns:repeat(auto-fit,minmax(min(100%,250px),1fr));gap:14px;margin:24px 0;background:transparent;">
<a class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" href="https://huggingface.co/philipjohnbasile/wisp-coder-110m/blob/main/config.json" style="display:block;padding:10px 0 20px;color:#263f32;text-decoration:none;text-decoration-thickness:1px;text-underline-offset:3px;margin:0;border-bottom:1px solid #bdcdb0;">
<span style="display:block;font-size:10px;letter-spacing:1.4px;color:#50614e;margin-bottom:12px;">ROUTE A / THE TRUNK</span><span style="display:block;font-family:Georgia,serif;font-size:27px;line-height:1.2;color:#263f32;margin-bottom:12px;">An ordinary Llama decoder.</span><span style="display:block;font-size:13px;line-height:1.7;color:#354b3b;">100.7M parameters. Standard Transformers and MLX-LM loading.</span><span style="display:block;font-size:12px;color:#8e3049;margin-top:16px;">Inspect the configuration ↗</span>
</a>
<a class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" href="https://huggingface.co/philipjohnbasile/wisp-coder-110m/blob/main/wisp_mtp_reference.py" style="display:block;padding:10px 0 20px;color:#263f32;text-decoration:none;text-decoration-thickness:1px;text-underline-offset:3px;margin:0;border-bottom:1px solid #bdcdb0;">
<span style="display:block;font-size:10px;letter-spacing:1.4px;color:#8e3049;margin-bottom:12px;">ROUTE B / NATIVE MTP</span><span style="display:block;font-family:Georgia,serif;font-size:27px;line-height:1.2;color:#263f32;margin-bottom:12px;">A shared drafting module.</span><span style="display:block;font-size:13px;line-height:1.7;color:#354b3b;">108.2M parameters with the sidecar. Use the explicit MLX correctness runtime.</span><span style="display:block;font-size:12px;color:#8e3049;margin-top:16px;">Read the reference runtime ↗</span>
</a>
</div>
<p style="font-size:13px;color:#354b3b;"><a href="https://huggingface.co/philipjohnbasile/wisp-coder-110m/blob/main/export_manifest.json" style="color:#963c54;text-decoration-thickness:1px;text-underline-offset:3px;">Export manifest ↗</a>   ·   <a href="https://huggingface.co/philipjohnbasile/wisp-coder-110m/blob/main/mtp_config.json" style="color:#963c54;text-decoration-thickness:1px;text-underline-offset:3px;">MTP configuration ↗</a></p>
<table style="color:#354b3b;width:100%;max-width:100%;display:block;overflow-x:auto;border:0;border-radius:0;border-collapse:collapse;font-size:13px;margin:24px 0;">
<thead>
<tr>
<th style="color:#963c54;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;font-weight:600;"></th>
<th style="color:#963c54;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;font-weight:600;"></th>
</tr>
</thead>
<tbody>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Parameters</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">108.2M total, 100.7M trunk</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Layers</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">12, d_model 768, 12 heads / 4 KV heads, SwiGLU 2048</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Context</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">2048, RoPE theta 100k</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Vocab</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">32,768 byte-level BPE, digits split, FIM sentinels</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">MTP</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">1 shared module (1 transformer block), trained to depth 2, recursive at inference</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Precision</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">bfloat16 weights, trained with float32 master weights</td>
</tr>
</tbody>
</table>
<p style="color:#354b3b;">The trunk is an ordinary Llama decoder. It loads in <code style="color:#263f32;background:#e8eddf;overflow-wrap:anywhere;white-space:normal;font-weight:500;padding:2px 5px;">transformers</code> and <code style="color:#263f32;background:#e8eddf;overflow-wrap:anywhere;white-space:normal;font-weight:500;padding:2px 5px;">mlx_lm</code>
with no custom modelling code and no <code style="color:#263f32;background:#e8eddf;overflow-wrap:anywhere;white-space:normal;font-weight:500;padding:2px 5px;">trust_remote_code</code>. The MTP module ships
alongside as <code style="color:#263f32;background:#e8eddf;overflow-wrap:anywhere;white-space:normal;font-weight:500;padding:2px 5px;">mtp.safetensors</code> and is ignorable by runtimes that cannot use it.
Generic Llama runtimes do not consume the sidecar automatically. The package
therefore also ships <code style="color:#263f32;background:#e8eddf;overflow-wrap:anywhere;white-space:normal;font-weight:500;padding:2px 5px;">wisp_mtp_model.py</code> and <code style="color:#263f32;background:#e8eddf;overflow-wrap:anywhere;white-space:normal;font-weight:500;padding:2px 5px;">wisp_mtp_reference.py</code>, a
manifest-bound MLX correctness decoder that requires the exact sidecar schema,
verifies every declared package hash, exposes the resolved runtime contract and
executed route, and fails unless its MTP path reproduces target greedy tokens.
It recomputes full prefixes and is not a production speed path.</p>
<p style="color:#354b3b;">The export was verified against <code style="color:#263f32;background:#e8eddf;overflow-wrap:anywhere;white-space:normal;font-weight:500;padding:2px 5px;">transformers</code> numerically, not assumed: relative
logit delta 2.16e-3 and argmax agreement 1.0000 against the MLX original on the
same tokens, which pins the RoPE convention, the RMSNorm epsilon, and the grouped
query head order.</p>
<div style="margin:0 0 24px;padding:44px 0 0;background:transparent;position:relative;"><span aria-hidden="true" style="position:absolute;left:calc(-1 * clamp(14px,2vw,23px) - 7px);top:43px;color:#ad3f5a;background:#f4f1e8;font-size:14px;line-height:1;">❋</span>
<p style="margin:0 0 12px;color:#9f3d58;font-size:10px;letter-spacing:1.6px;text-transform:uppercase;">04 / TRAINING DATA</p>
<h2 id="training-data" style="font-family:Georgia,serif;font-size:clamp(32px,4.3vw,46px);line-height:1.08;font-weight:400;letter-spacing:-1.6px;color:#263f32;margin:0;padding:0;border:0;scroll-margin-top:24px;">Where it grew.</h2>
</div>
<p style="color:#354b3b;">The configured target mixture was 92 percent code from
<code style="color:#263f32;background:#e8eddf;overflow-wrap:anywhere;white-space:normal;font-weight:500;padding:2px 5px;">bigcode/starcoderdata</code> and 8 percent <code style="color:#263f32;background:#e8eddf;overflow-wrap:anywhere;white-space:normal;font-weight:500;padding:2px 5px;">HuggingFaceFW/fineweb-edu</code>. This is a
source percentage, not a permissive-license percentage. StarCoderData declares
<code style="color:#263f32;background:#e8eddf;overflow-wrap:anywhere;white-space:normal;font-weight:500;padding:2px 5px;">license: other</code>, and its original repository terms and relevant attribution
clauses remain applicable. FineWeb-Edu is ODC-By 1.0 and remains subject to
Common Crawl terms. Run 1 retained no row-level source manifest, so no per-file
licensing or attribution guarantee is made.</p>
<details class="group" open="" style="margin:24px 0;padding:0;color:#263f32;border-top:1px solid #bdcdb0;">
<summary class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" style="cursor:pointer;padding:14px 0;color:#536d43;font-family:Georgia,serif;font-size:19px;background:transparent;">Training-data provenance notes</summary>
<div style="padding:0;background:transparent;">
<p style="color:#354b3b;">Every run 1 document received structural size, line-length, and character
distribution filters. Because the loader expected <code style="color:#263f32;background:#e8eddf;overflow-wrap:anywhere;white-space:normal;font-weight:500;padding:2px 5px;">path</code> instead of
StarCoderData's <code style="color:#263f32;background:#e8eddf;overflow-wrap:anywhere;white-space:normal;font-weight:500;padding:2px 5px;">max_stars_repo_path</code>, Python AST and JSON extension filtering
did not activate for run 1. The schema-aware repair applies only to future
source streaming, not to these weights. Corpus content was never executed.
Read the published <a href="https://huggingface.co/philipjohnbasile/wisp-coder-110m/blob/main/evidence/documents/DATA.md" style="color:#963c54;text-decoration-thickness:1px;text-underline-offset:3px;">training-data disclosure</a>
and <a href="https://huggingface.co/philipjohnbasile/wisp-coder-110m/blob/main/evidence/receipts/training_data_receipt.json" style="color:#963c54;text-decoration-thickness:1px;text-underline-offset:3px;">machine-checked training-data receipt</a>.</p>
<p style="color:#354b3b;">The 32K tokenizer was trained on 400,000 documents sampled round-robin across
the eleven source entries, not according to the later token-budget weights.
Current shard bytes are fully attested after the build, but the receipt is not a
raw-row manifest and cannot prove exact original example boundaries.</p>
</div>
</details>
<p style="color:#354b3b;">The Apache 2.0 metadata describes the released Wisp artifact. It does not
override training-source terms or licenses applicable to generated code.</p>
<div style="margin:0 0 24px;padding:44px 0 0;background:transparent;position:relative;"><span aria-hidden="true" style="position:absolute;left:calc(-1 * clamp(14px,2vw,23px) - 7px);top:43px;color:#ad3f5a;background:#f4f1e8;font-size:14px;line-height:1;">❋</span>
<p style="margin:0 0 12px;color:#9f3d58;font-size:10px;letter-spacing:1.6px;text-transform:uppercase;">05 / USAGE</p>
<h2 id="usage" style="font-family:Georgia,serif;font-size:clamp(32px,4.3vw,46px);line-height:1.08;font-weight:400;letter-spacing:-1.6px;color:#263f32;margin:0;padding:0;border:0;scroll-margin-top:24px;">Give it a few lines.</h2>
</div>
<p style="color:#354b3b;">Plain completion:</p>


```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("philipjohnbasile/wisp-coder-110m")
tok = AutoTokenizer.from_pretrained("philipjohnbasile/wisp-coder-110m")
print(tok.decode(model.generate(**tok("def quicksort(arr):", return_tensors="pt"),
                                max_new_tokens=64)[0]))
```


<p style="color:#354b3b;">Fill in the middle. The model was trained with both orderings, evenly split:</p>


```python
# PSM: prefix, suffix, then generate the middle
prompt = f"<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>"
```


<p style="color:#354b3b;">Exercise the native sidecar explicitly. This is a correctness check, not a
latency benchmark:</p>


```python
import os
import sys
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer

package = snapshot_download(
    "philipjohnbasile/wisp-coder-110m",
    local_dir=os.path.abspath("wisp-coder-110m-package"),
)
sys.path.insert(0, package)
from wisp_mtp_reference import WispMTPReferenceRuntime

tok = Tokenizer.from_file(os.path.join(package, "tokenizer.json"))
ids = tok.encode(prompt, add_special_tokens=False).ids
runtime = WispMTPReferenceRuntime.load(package)
result = runtime.verify_greedy_parity(ids, max_new_tokens=16, depth=2)
print(result["mtp_route"])
```


<p style="color:#354b3b;">For evaluation, the trunk can be configured as Continue.dev's autocomplete model
while leaving a larger model as the chat and agent backend. Target-runtime batch-1
latency and editor user experience have not yet been measured, so this is not a
sub-100 ms claim.</p>
<div style="margin:0 0 24px;padding:44px 0 0;background:transparent;position:relative;"><span aria-hidden="true" style="position:absolute;left:calc(-1 * clamp(14px,2vw,23px) - 7px);top:43px;color:#ad3f5a;background:#f4f1e8;font-size:14px;line-height:1;">❋</span>
<p style="margin:0 0 12px;color:#9f3d58;font-size:10px;letter-spacing:1.6px;text-transform:uppercase;">06 / EVALUATION</p>
<h2 id="evaluation" style="font-family:Georgia,serif;font-size:clamp(32px,4.3vw,46px);line-height:1.08;font-weight:400;letter-spacing:-1.6px;color:#263f32;margin:0;padding:0;border:0;scroll-margin-top:24px;">Follow the evidence.</h2>
</div>
<div style="display:grid;grid-template-columns:repeat(auto-fit,minmax(min(100%,190px),1fr));gap:12px;margin:0 0 24px;background:transparent;">
<div style="padding:12px 0 18px;color:#263f32;background:transparent;border-top:1px solid #bdcdb0;"><p style="font-size:10px;letter-spacing:1px;color:#4a603d;margin:0 0 9px;">FUNCTIONAL CHECKS</p><p style="font-family:Georgia,serif;font-size:23px;line-height:1.2;color:#263f32;margin:0 0 10px;">CPU completions.</p><p style="font-size:13px;line-height:1.7;color:#354b3b;margin:0;">CPU completion smokes are documented above. They do not measure coding quality or latency.</p></div>
<div style="padding:12px 0 18px;color:#263f32;background:transparent;border-top:1px solid #bdcdb0;"><p style="font-size:10px;letter-spacing:1px;color:#8e3049;margin:0 0 9px;">SHORT-HOLE COMPARISON</p><p style="font-family:Georgia,serif;font-size:23px;line-height:1.2;color:#263f32;margin:0 0 10px;">No superiority.</p><p style="font-size:13px;line-height:1.7;color:#354b3b;margin:0;">No superiority was established. <a href="https://huggingface.co/philipjohnbasile/wisp-coder-110m/blob/main/fim-competitive-v1/FIM_RESULTS.md" style="color:#963c54;text-decoration-thickness:1px;text-underline-offset:3px;">Read the comparison ↗</a></p></div>
<div style="padding:12px 0 18px;color:#263f32;background:transparent;border-top:1px solid #bdcdb0;"><p style="font-size:10px;letter-spacing:1px;color:#8e3049;margin:0 0 9px;">MTP RUNTIME</p><p style="font-family:Georgia,serif;font-size:23px;line-height:1.2;color:#263f32;margin:0 0 10px;">Speed is open.</p><p style="font-size:13px;line-height:1.7;color:#354b3b;margin:0;">No production MTP speedup is claimed. The packaged MLX runtime checks correctness.</p></div>
</div>
<h3 id="registered-final-results" style="color:#263f32;font-family:Georgia,serif;font-size:25px;font-weight:400;line-height:1.3;margin:30px 0 16px;">Registered final results</h3>
<details class="group" open="" style="margin:24px 0;padding:0;color:#263f32;border-top:1px solid #bdcdb0;">
<summary class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" style="cursor:pointer;padding:14px 0;color:#536d43;font-family:Georgia,serif;font-size:19px;background:transparent;">Complete registered results</summary>
<div style="padding:0;background:transparent;">
<table style="color:#354b3b;width:100%;max-width:100%;display:block;overflow-x:auto;border:0;border-radius:0;border-collapse:collapse;font-size:13px;margin:24px 0;">
<thead>
<tr>
<th style="color:#963c54;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;font-weight:600;">Measurement</th>
<th style="color:#963c54;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;font-weight:600;">Result</th>
</tr>
</thead>
<tbody>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Final validation main NLL</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">1.3873 [1.3428, 1.4335], perplexity 4.00</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Validation MTP depth 1 NLL</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">1.5727 [1.5236, 1.6234]</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Validation MTP depth 2 NLL</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">1.6513 [1.6013, 1.7028]</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">FIM / shuffled-suffix acceptance, depth 2</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">1.0268 [1.0209, 1.0331], POSITIVE, 200 documents</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Trained minus initialized acceptance-ratio lift</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">+0.0268 [+0.0210, +0.0331], CLEARS_CONTROL</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Acceptance interpretation</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">POSITIVE_AND_CLEARS_CONTROL</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">FIM-training effect on shuffled-FIM minus L2R acceptance</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">+0.0814 [+0.0747, +0.0883], POSITIVE, 200 documents</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">FIM-training effect on true-suffix minus shuffled-suffix acceptance</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">+0.0049 [-0.0026, +0.0122], NULL: the interval includes zero</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Adaptive minus fixed accepted drafts per verification</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">+0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Adaptive minus fixed output tokens per target forward</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">+0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Adaptive minus fixed drafts issued per output token</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">+0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents (issuance proxy)</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Adaptive minus fixed draft recursions per output token</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">+0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Rollout policies selected on calibration</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">adaptive_h0.7 versus fixed_d4</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Branch-local greedy replay</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Exact argmax for all 38400 emitted tokens across 600 scored policy-document rollouts</td>
</tr>
<tr>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Cross-policy output identity</td>
<td style="color:#263f32;border:0;border-bottom:1px solid #cbd4bf;padding:10px 8px;vertical-align:top;text-align:left;line-height:1.65;">Identical realized output branches for all 100 calibration/test documents across compared policies</td>
</tr>
</tbody>
</table>
</div>
</details>
<p style="color:#354b3b;">Validation intervals measure Monte Carlo uncertainty from the frozen random-window sampler. Acceptance and rollout intervals resample paired target documents. They do not measure training-run or model uncertainty. Null and negative outcomes are retained rather than filtered from the release.</p>
<details class="group" open="" style="margin:24px 0;padding:0;color:#263f32;border-top:1px solid #bdcdb0;">
<summary class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] hover:translate-x-px focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-blue-500" style="cursor:pointer;padding:14px 0;color:#536d43;font-family:Georgia,serif;font-size:19px;background:transparent;">Read the sampler, replay and provenance notes</summary>
<div style="padding:0;background:transparent;">
<p style="color:#354b3b;">Rollout endpoint scope: The primary endpoint measures accepted drafts per verification. It does not establish verification-width cost or deployment latency. Draft recursions per output token is the registered drafter-work companion; issued drafts per output token is retained only as an issuance proxy. Target forwards exclude the added post-hoc branch-replay forward and independent verification pass.</p>
<p style="color:#354b3b;">Independent replay provenance scope: Unsigned local attestation bound to a pushed pre-execution receipt commit and the registered source, inputs, checkpoint, and argv; it is not a signed external or trusted-execution witness.</p>
<p style="color:#354b3b;">Format-ablation provenance limitation: This is repository-revision evidence, not a per-document raw-corpus manifest for run 1. Source drift between the run 1 build and the captured cache refs cannot be ruled out.</p>
<p style="color:#354b3b;">Format-ablation runtime limitation: Run 1 did not record source-file hashes or sampler state in its pre-fix checkpoints. A post-build receipt now hashes all 52 current shard files and binds deterministic visible-grammar normalization, but it does not prove that the bytes were unchanged since training began or recover exact original units. The step-300 process recovery reset the legacy sampler, causing 78,643,200 scheduled token positions, 1.57 percent of the training budget, to replay earlier random windows. sampler_reset_steps [300] matches that reset timing in both arms. Later checkpoints preserve exact sampler RNG state. Exact sampled-example equivalence still cannot be proven.</p>
</div>
</details>
<p style="color:#354b3b;">One control worth repeating here, because acceptance rate is easy to report
dishonestly: <strong style="color:#263f32;">an initialized model scored higher acceptance than a trained one
in the E0 instrument pilot</strong> (0.482 against 0.331 for a model deliberately
overfit on 100k tokens), because two near-uniform distributions agree trivially.
Those are pilot diagnostics, not run 1 results. Any final acceptance number
without its exact initialized floor beside it is uninterpretable.</p>
<div style="margin:0 0 24px;padding:44px 0 0;background:transparent;position:relative;"><span aria-hidden="true" style="position:absolute;left:calc(-1 * clamp(14px,2vw,23px) - 7px);top:43px;color:#ad3f5a;background:#f4f1e8;font-size:14px;line-height:1;">❋</span>
<p style="margin:0 0 12px;color:#9f3d58;font-size:10px;letter-spacing:1.6px;text-transform:uppercase;">07 / LIMITATIONS</p>
<h2 id="limitations" style="font-family:Georgia,serif;font-size:clamp(32px,4.3vw,46px);line-height:1.08;font-weight:400;letter-spacing:-1.6px;color:#263f32;margin:0;padding:0;border:0;scroll-margin-top:24px;">Room to grow.</h2>
</div>
<ul>
<li>5B training tokens. Small, and it shows on unfamiliar APIs, which it will
hallucinate confidently.</li>
<li>2048 token context.</li>
<li>Ten languages, weighted toward Python, JavaScript, and TypeScript. Weaker
everywhere else, and untested outside that set.</li>
<li>No safety tuning of any kind. It is a base completion model trained on public
GitHub code, and it will reproduce patterns from that data, including insecure
ones. Review what it writes.</li>
<li>No row-level training-source manifest, license mapping, or attribution index.
Exact aggregate train-token totals survive in the recovered final build log,
but accepted rows, rejection counts, and per-source validation overshoot
cannot be reconstructed from the shards.</li>
<li>The tokenizer document sampler was round-robin by source entry rather than
weighted like the training-token mixture.</li>
<li>Training windows are sampled with replacement. The registered schedule's
uniform-interval approximation covers about 62.57 percent of corpus positions
at least once, rather than exposing every position once.</li>
<li>The source-stratified validation shard is not repository- or document-disjoint.
A chunked source document can cross the validation-to-training boundary. The
separate publication holdout is repository-disjoint.</li>
<li>The MTP sidecar is not consumed automatically by generic Llama runtimes, and
no production-runtime MTP latency or speedup is claimed.</li>
<li>Trained and evaluated on one machine, one seed, one run.</li>
</ul>
<div style="margin:0 0 24px;padding:44px 0 0;background:transparent;position:relative;"><span aria-hidden="true" style="position:absolute;left:calc(-1 * clamp(14px,2vw,23px) - 7px);top:43px;color:#ad3f5a;background:#f4f1e8;font-size:14px;line-height:1;">❋</span>
<p style="margin:0 0 12px;color:#9f3d58;font-size:10px;letter-spacing:1.6px;text-transform:uppercase;">08 / CITATION</p>
<h2 id="citation" style="font-family:Georgia,serif;font-size:clamp(32px,4.3vw,46px);line-height:1.08;font-weight:400;letter-spacing:-1.6px;color:#263f32;margin:0;padding:0;border:0;scroll-margin-top:24px;">Leave a trail.</h2>
</div>


```bibtex
@software{wisp_coder,
  title  = {Wisp: fill-in-the-middle and native multi-token prediction in a small code model},
  year   = {2026},
  url    = {https://huggingface.co/philipjohnbasile/wisp-coder-110m}
}
```


</div><p style="color:#354b3b;"><a href="https://huggingface.co/spaces/philipjohnbasile/local-ai-guide" style="color:#963c54;text-decoration-thickness:1px;text-underline-offset:3px;">Explore the model guide</a> · <a href="https://huggingface.co/philipjohnbasile" style="color:#963c54;text-decoration-thickness:1px;text-underline-offset:3px;">All public work</a></p><p style="color:#354b3b;"><strong style="color:#263f32;"><a href="https://huggingface.co/spaces/philipjohnbasile/wisp-coder-demo" style="color:#963c54;text-decoration-thickness:1px;text-underline-offset:3px;">Try the interactive CPU demo</a></strong> ·
<a href="https://huggingface.co/philipjohnbasile/wisp-coder-110m/blob/main/fim-competitive-v1/FIM_RESULTS.md" style="color:#963c54;text-decoration-thickness:1px;text-underline-offset:3px;">Short-hole evaluation</a> ·
<a href="https://huggingface.co/collections/philipjohnbasile/start-here-selected-work-6aa309b179b2a196db52a12b" style="color:#963c54;text-decoration-thickness:1px;text-underline-offset:3px;">Selected work</a></p><div class="group" style="margin-top:36px;padding:36px 0 10px;display:flex;align-items:center;flex-wrap:wrap;gap:18px;color:#263f32;background:transparent;margin:30px 0 0;border-top:1px solid #bdcdb0;">
<img alt="" class="motion-safe:transition-transform motion-safe:duration-[400ms] motion-safe:[transition-timing-function:ease] group-hover:scale-105 group-hover:-rotate-6" src="./assets/wisp-wildflowers.png" style="flex:0 1 180px;width:180px;max-width:100%;height:auto;object-fit:contain;margin:0;border:0;"/>
<div style="flex:1 1 210px;min-width:0;background:transparent;">
<p style="font-size:11px;letter-spacing:1.5px;color:#50614e;margin:0 0 8px;">SMALL MODEL. OPEN QUESTIONS.</p>
<p style="font-family:Georgia,serif;font-size:42px;letter-spacing:-2px;line-height:1.15;color:#263f32;margin:0 0 12px;">Stay <em style="color:#ad3f5a;">curious.</em></p>
<p style="font-size:14px;line-height:1.8;margin:0;color:#354b3b;"><a href="https://huggingface.co/spaces/philipjohnbasile/wisp-coder-demo" style="color:#963c54;text-decoration-thickness:1px;text-underline-offset:3px;">Return to the garden ↗</a>   ·   <a href="https://huggingface.co/philipjohnbasile" style="color:#963c54;text-decoration-thickness:1px;text-underline-offset:3px;">More from Philip</a></p>
</div>
</div>
<p style="color:#354b3b;"><sub>The Wisp Fieldbook. Original AI-generated floral artwork shared with the Wisp demo. <a href="https://huggingface.co/philipjohnbasile/wisp-coder-110m/blob/main/assets/ARTWORK.md" style="color:#963c54;text-decoration-thickness:1px;text-underline-offset:3px;">Artwork notes</a>.</sub></p>
</div>