philipjohnbasile commited on
Commit
a9bb1bd
·
verified ·
1 Parent(s): 76becf5

Clear prior rendered model card before superseding publication

Browse files
Files changed (1) hide show
  1. README.md +0 -230
README.md DELETED
@@ -1,230 +0,0 @@
1
- ---
2
- license: apache-2.0
3
- language:
4
- - code
5
- tags:
6
- - code
7
- - fill-in-the-middle
8
- - multi-token-prediction
9
- - speculative-decoding
10
- - mlx
11
- - apple-silicon
12
- library_name: transformers
13
- pipeline_tag: text-generation
14
- ---
15
-
16
- # Wisp Coder 110M
17
-
18
- A small code model built for one job: completing code at the cursor. It targets
19
- editor inline suggestions, but no production-runtime latency claim is made.
20
-
21
- Two things make it unusual, and they are the reason it exists rather than a
22
- feature list.
23
-
24
- **It does fill in the middle natively.** Each tokenized source document was split
25
- into chunks of at most 1024 tokens, and 70 percent of those chunks were
26
- independently transformed so the model predicts a middle span given both the
27
- prefix and the suffix. This is not a per-document or per-window rate. Cursor
28
- completion is the primary format, not an inference-time prompt trick added to an
29
- ordinary left-to-right checkpoint.
30
-
31
- **It has multi-token-prediction heads trained in from step zero.** A shared MTP
32
- module, applied recursively in the Qwen3-Next style, predicts two or more tokens
33
- ahead from the trunk's own hidden state. The LM head is shared between trunk and
34
- MTP module, which ties the two output distributions to a common projection. That
35
- is a useful architectural choice, not a guarantee that draft and target
36
- distributions agree; how close they actually are is what acceptance measures.
37
-
38
- **What is and is not claimed here.** Small models with MTP drafters exist: Google
39
- publishes `gemma-4-E2B-it-assistant` and related MTP assistant checkpoints. The
40
- artifact contribution is the *combination* at this scale, a small code model
41
- trained from step zero with both fill-in-the-middle on most tokenized chunks and a
42
- recursively shared MTP module, plus a paired measurement of how suffix information
43
- changes draft acceptance. A targeted search found no exact prior measurement, but
44
- that is not proof of absence. No exclusivity or first-of-its-kind claim is made.
45
-
46
- ## What it is not
47
-
48
- It is not a chat model, an instruction-following model, or an agent. It completes
49
- text. Asking it to refactor a module produces nonsense.
50
-
51
- It is also not competitive with Qwen2.5-Coder on raw completion quality, and the
52
- arithmetic says it cannot be: Qwen2.5-Coder-0.5B saw roughly 5.5 trillion tokens
53
- and Wisp saw 5 billion, three orders of magnitude fewer. If you want the best
54
- completions available at small scale, use Qwen. Use Wisp to study FIM plus native
55
- speculative drafting on Apple Silicon, or how those two interact.
56
-
57
- ## Architecture
58
-
59
- | | |
60
- |---|---|
61
- | Parameters | 108.2M total, 100.7M trunk |
62
- | Layers | 12, d_model 768, 12 heads / 4 KV heads, SwiGLU 2048 |
63
- | Context | 2048, RoPE theta 100k |
64
- | Vocab | 32,768 byte-level BPE, digits split, FIM sentinels |
65
- | MTP | 1 shared module (1 transformer block), trained to depth 2, recursive at inference |
66
- | Precision | bfloat16 weights, trained with float32 master weights |
67
-
68
- The trunk is an ordinary Llama decoder. It loads in `transformers` and `mlx_lm`
69
- with no custom modelling code and no `trust_remote_code`. The MTP module ships
70
- alongside as `mtp.safetensors` and is ignorable by runtimes that cannot use it.
71
- Generic Llama runtimes do not consume the sidecar automatically. The package
72
- therefore also ships `wisp_mtp_model.py` and `wisp_mtp_reference.py`, a
73
- manifest-bound MLX correctness decoder that requires the exact sidecar schema,
74
- verifies every declared package hash, exposes the resolved runtime contract and
75
- executed route, and fails unless its MTP path reproduces target greedy tokens.
76
- It recomputes full prefixes and is not a production speed path.
77
-
78
- The export was verified against `transformers` numerically, not assumed: relative
79
- logit delta 2.16e-3 and argmax agreement 1.0000 against the MLX original on the
80
- same tokens, which pins the RoPE convention, the RMSNorm epsilon, and the grouped
81
- query head order.
82
-
83
- ## Training data
84
-
85
- The configured target mixture was 92 percent code from
86
- `bigcode/starcoderdata` and 8 percent `HuggingFaceFW/fineweb-edu`. This is a
87
- source percentage, not a permissive-license percentage. StarCoderData declares
88
- `license: other`, and its original repository terms and relevant attribution
89
- clauses remain applicable. FineWeb-Edu is ODC-By 1.0 and remains subject to
90
- Common Crawl terms. Run 1 retained no row-level source manifest, so no per-file
91
- licensing or attribution guarantee is made.
92
-
93
- Every run 1 document received structural size, line-length, and character
94
- distribution filters. Because the loader expected `path` instead of
95
- StarCoderData's `max_stars_repo_path`, Python AST and JSON extension filtering
96
- did not activate for run 1. The schema-aware repair applies only to future
97
- source streaming, not to these weights. Corpus content was never executed.
98
- Full details and the machine-checked disclosure are in `DATA.md` and
99
- `config/training_data_receipt.json`.
100
-
101
- The 32K tokenizer was trained on 400,000 documents sampled round-robin across
102
- the eleven source entries, not according to the later token-budget weights.
103
- Current shard bytes are fully attested after the build, but the receipt is not a
104
- raw-row manifest and cannot prove exact original example boundaries.
105
-
106
- The Apache 2.0 metadata describes the released Wisp artifact. It does not
107
- override training-source terms or licenses applicable to generated code.
108
-
109
- ## Usage
110
-
111
- Plain completion:
112
-
113
- ```python
114
- from transformers import AutoModelForCausalLM, AutoTokenizer
115
-
116
- model = AutoModelForCausalLM.from_pretrained("philipjohnbasile/wisp-coder-110m")
117
- tok = AutoTokenizer.from_pretrained("philipjohnbasile/wisp-coder-110m")
118
- print(tok.decode(model.generate(**tok("def quicksort(arr):", return_tensors="pt"),
119
- max_new_tokens=64)[0]))
120
- ```
121
-
122
- Fill in the middle. The model was trained with both orderings, evenly split:
123
-
124
- ```python
125
- # PSM: prefix, suffix, then generate the middle
126
- prompt = f"<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>"
127
- ```
128
-
129
- Exercise the native sidecar explicitly. This is a correctness check, not a
130
- latency benchmark:
131
-
132
- ```python
133
- import os
134
- import sys
135
- from huggingface_hub import snapshot_download
136
- from tokenizers import Tokenizer
137
-
138
- package = snapshot_download(
139
- "philipjohnbasile/wisp-coder-110m",
140
- local_dir=os.path.abspath("wisp-coder-110m-package"),
141
- )
142
- sys.path.insert(0, package)
143
- from wisp_mtp_reference import WispMTPReferenceRuntime
144
-
145
- tok = Tokenizer.from_file(os.path.join(package, "tokenizer.json"))
146
- ids = tok.encode(prompt, add_special_tokens=False).ids
147
- runtime = WispMTPReferenceRuntime.load(package)
148
- result = runtime.verify_greedy_parity(ids, max_new_tokens=16, depth=2)
149
- print(result["mtp_route"])
150
- ```
151
-
152
- For evaluation, the trunk can be configured as Continue.dev's autocomplete model
153
- while leaving a larger model as the chat and agent backend. Target-runtime batch-1
154
- latency and editor user experience have not yet been measured, so this is not a
155
- sub-100 ms claim.
156
-
157
- ## Evaluation
158
-
159
- ### Registered final results
160
-
161
- | Measurement | Result |
162
- |---|---|
163
- | Final validation main NLL | 1.3873 [1.3428, 1.4335], perplexity 4.00 |
164
- | Validation MTP depth 1 NLL | 1.5727 [1.5236, 1.6234] |
165
- | Validation MTP depth 2 NLL | 1.6513 [1.6013, 1.7028] |
166
- | FIM / shuffled-suffix acceptance, depth 2 | 1.0268 [1.0209, 1.0331], POSITIVE, 200 documents |
167
- | Trained minus initialized acceptance-ratio lift | +0.0268 [+0.0210, +0.0331], CLEARS_CONTROL |
168
- | Acceptance interpretation | POSITIVE_AND_CLEARS_CONTROL |
169
- | FIM-training effect on shuffled-FIM minus L2R acceptance | +0.0814 [+0.0747, +0.0883], POSITIVE, 200 documents |
170
- | FIM-training effect on true-suffix minus shuffled-suffix acceptance | +0.0049 [-0.0026, +0.0122], NULL: the interval includes zero |
171
- | Adaptive minus fixed accepted drafts per verification | +0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents |
172
- | Adaptive minus fixed output tokens per target forward | +0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents |
173
- | Adaptive minus fixed drafts issued per output token | +0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents (issuance proxy) |
174
- | Adaptive minus fixed draft recursions per output token | +0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents |
175
- | Rollout policies selected on calibration | adaptive_h0.7 versus fixed_d4 |
176
- | Branch-local greedy replay | Exact argmax for all 38400 emitted tokens across 600 scored policy-document rollouts |
177
- | Cross-policy output identity | Identical realized output branches for all 100 calibration/test documents across compared policies |
178
-
179
- Validation intervals measure Monte Carlo uncertainty from the frozen random-window sampler. Acceptance and rollout intervals resample paired target documents. They do not measure training-run or model uncertainty. Null and negative outcomes are retained rather than filtered from the release.
180
-
181
- Rollout endpoint scope: The primary endpoint measures accepted drafts per verification. It does not establish verification-width cost or deployment latency. Draft recursions per output token is the registered drafter-work companion; issued drafts per output token is retained only as an issuance proxy. Target forwards exclude the added post-hoc branch-replay forward and independent verification pass.
182
-
183
- Independent replay provenance scope: Unsigned local attestation bound to a pushed pre-execution receipt commit and the registered source, inputs, checkpoint, and argv; it is not a signed external or trusted-execution witness.
184
-
185
- Format-ablation provenance limitation: This is repository-revision evidence, not a per-document raw-corpus manifest for run 1. Source drift between the run 1 build and the captured cache refs cannot be ruled out.
186
-
187
- Format-ablation runtime limitation: Run 1 did not record source-file hashes or sampler state in its pre-fix checkpoints. A post-build receipt now hashes all 52 current shard files and binds deterministic visible-grammar normalization, but it does not prove that the bytes were unchanged since training began or recover exact original units. The step-300 process recovery reset the legacy sampler, causing 78,643,200 scheduled token positions, 1.57 percent of the training budget, to replay earlier random windows. sampler_reset_steps [300] matches that reset timing in both arms. Later checkpoints preserve exact sampler RNG state. Exact sampled-example equivalence still cannot be proven.
188
-
189
- One control worth repeating here, because acceptance rate is easy to report
190
- dishonestly: **an initialized model scored higher acceptance than a trained one
191
- in the E0 instrument pilot** (0.482 against 0.331 for a model deliberately
192
- overfit on 100k tokens), because two near-uniform distributions agree trivially.
193
- Those are pilot diagnostics, not run 1 results. Any final acceptance number
194
- without its exact initialized floor beside it is uninterpretable.
195
-
196
- ## Limitations
197
-
198
- - 5B training tokens. Small, and it shows on unfamiliar APIs, which it will
199
- hallucinate confidently.
200
- - 2048 token context.
201
- - Ten languages, weighted toward Python, JavaScript, and TypeScript. Weaker
202
- everywhere else, and untested outside that set.
203
- - No safety tuning of any kind. It is a base completion model trained on public
204
- GitHub code, and it will reproduce patterns from that data, including insecure
205
- ones. Review what it writes.
206
- - No row-level training-source manifest, license mapping, or attribution index.
207
- Exact aggregate train-token totals survive in the recovered final build log,
208
- but accepted rows, rejection counts, and per-source validation overshoot
209
- cannot be reconstructed from the shards.
210
- - The tokenizer document sampler was round-robin by source entry rather than
211
- weighted like the training-token mixture.
212
- - Training windows are sampled with replacement. The registered schedule's
213
- uniform-interval approximation covers about 62.57 percent of corpus positions
214
- at least once, rather than exposing every position once.
215
- - The source-stratified validation shard is not repository- or document-disjoint.
216
- A chunked source document can cross the validation-to-training boundary. The
217
- separate publication holdout is repository-disjoint.
218
- - The MTP sidecar is not consumed automatically by generic Llama runtimes, and
219
- no production-runtime MTP latency or speedup is claimed.
220
- - Trained and evaluated on one machine, one seed, one run.
221
-
222
- ## Citation
223
-
224
- ```bibtex
225
- @software{wisp_coder,
226
- title = {Wisp: fill-in-the-middle and native multi-token prediction in a small code model},
227
- year = {2026},
228
- url = {https://huggingface.co/philipjohnbasile/wisp-coder-110m}
229
- }
230
- ```