Update README.md

README.md

@@ -1,4 +1,218 @@
 ---
 license: apache-2.0
+base_model: Qwen/Qwen3.8-27B
+library_name: transformers
+pipeline_tag: text-generation
+language:
+- en
+tags:
+- qwen3_5
+- pruning
+- depth-pruning
+- distillation
+- reasoning
 ---
-Marlowe is an experimental reasoning model built on consumer hardware, to run on consumer hardware. Inheriting training data from Qwen3.8:27b, with architectural changes of my own to preserve intelligence with a higher throughput with a lower footprint.
+
+# Marlowe-22B (R1.5)
+
+**A 27B-class reasoning model compressed to 22.3B so it runs entirely on one 16 GB consumer GPU —
+and measured, domain by domain, against the model it came from.**
+
+Marlowe-22B removes 12 layers from **Qwen3.8-27B** and heals the loss by distilling from the 27B
+itself. R1.5 keeps **94.8% of the facts its parent demonstrably knows**, at **81% of the
+parameters**, with the whole model resident in 16 GB at Q4_K_M — no layer offload, no paging.
+
+This is round three of a planned seven-round program. R1.7, R1.8 and R2 are scheduled; the
+roadmap is at the bottom, along with what each round is built to move.
+
+| | |
+|---|---|
+| parameters | 22.30 B (parent: 27 B) |
+| layers | 52 — 36 linear-attention (DeltaNet) + 16 full-attention (parent: 64 = 48 + 16) |
+| hidden / intermediate | 5120 / 17408 |
+| KV heads × head dim | 4 × 256, on the 16 full-attention layers only |
+| vocabulary | 248,320 |
+| max position embeddings | 262,144 |
+| modality | text only (the parent's vision tower is not included) |
+| thinking | on by default; `reasoning_effort` xhigh / medium / low |
+| release build | GGUF **Q4_K_M, 13.74 GB** |
+
+## Why compress this model in particular
+
+The parent is a hybrid: most layers are linear-attention (DeltaNet), and only every fourth is full
+attention. Its KV cache is therefore tiny — **64 KB per token at f16, ~34 KB at q8_0** — so long
+contexts are cheap to hold. What it cannot do is fit in 16 GB at a bit width that leaves its
+answers intact. Marlowe-22B is the same architecture, twelve layers shorter, at the size where a
+4-bit build and a long context fit on one card together.
+
+## Method
+
+1. **Layer profiling.** Each layer scored by the KL divergence its removal induces on held-out
+   text — measured, not assumed from depth heuristics.
+2. **Depth pruning.** The twelve cheapest layers by that measure: indices
+   `4, 5, 8, 9, 13, 14, 16, 17, 37, 38, 40, 41` — all linear-attention. Every full-attention layer
+   survived, as did the first four and the last ten layers.
+3. **Heal (R1).** LoRA (r=32, α=64) trained against the parent's **top-64 log-probs** — forward KL
+   against a real distribution, not hard labels — over **27.5 M tokens** of a 35 M-token teacher
+   cache (arXiv, open-web-math, AlgebraicStack, FineWeb-Edu, PG-19, plus 1,314 reasoning traces
+   from the parent). Stopped deliberately when held-out KL flattened, rather than at budget
+   exhaustion.
+4. **Targeted repair (R1.5).** Probing located specific facts the pruned model had lost — RMSNorm's
+   mean-of-squares, RoPE's exponent, the GELU cubic constant, Adam's bias correction. A 1.5 M-token
+   pass on documents carrying exactly those facts closed them.
+
+The adapter is merged; these are plain weights.
+
+## Results
+
+### Knowledge retention against the parent
+
+Facts are mined from documents **hash-dropped against the heal corpus**, and **the parent is scored
+first: every item the parent itself fails is discarded** — so only knowledge the 27B demonstrably
+has is counted, and the student is never charged for its teacher's gaps. Denominator: **3,859
+parent-verified facts** (6,002 mined items before gating). Each cell is the share of those facts
+the model also knows.
+
+| domain | pruned, unhealed | **R1.5** | recovered by healing |
+|---|---:|---:|---:|
+| chemistry / biology | 90.5% | **96.0%** | +5.5 |
+| physics | 79.8% | **85.3%** | +5.5 |
+| engineering | 88.9% | **91.6%** | +2.7 |
+| statistics | 87.4% | **89.2%** | +1.8 |
+| mathematics | 93.4% | **94.8%** | +1.4 |
+| general | 96.3% | **97.4%** | +1.1 |
+| machine learning | 92.4% | **93.2%** | +0.8 |
+| computer science | 91.2% | **89.9%** | −1.3 |
+| **overall** | **93.3%** | **94.8%** | **+1.5** |
+
+Five domains sit above 91%; the program's standard is **90% in every domain, not on average**, and
+physics, statistics and computer science are the remaining targets — see the roadmap.
+
+### Derivation, not recall
+
+120 generated engineering problems, parameters sampled outside tutorial values and answers computed
+from the defining formula, so a memorised answer scores zero: RoPE frequencies, GELU, Adam updates,
+softmax, cosine schedules, KV-cache sizing, LoRA parameter counts. **R1.5 answers 92 of 120.**
+
+The sharper number is the hardest 60 — the families the pruned model failed outright:
+
+| | correct |
+|---|---:|
+| pruned, unhealed | 10 / 60 |
+| **R1.5** | **32 / 60** |
+| parent (27B) | 41 / 60 |
+
+Healing recovered **71% of the gap** pruning opened on the material it damaged most.
+
+### Arithmetic fidelity
+
+Teacher-forced on held-out worked computations, the parent predicts the next computed digit with
+**67.9%** top-1 accuracy; R1.5 reaches **64.1%** — a 0.115-nat gap in log-probability. KV-cache and
+attention-scaling arithmetic is already at parent level; rotation- and trigonometry-heavy work
+(RoPE, cosine schedules) is where the remaining distance lives, and it is R1.7's first target.
+
+## How it is measured
+
+The instruments matter as much as the weights here, and every one of them is built to make a
+flattering result hard to get:
+
+- **Parent-gated retention bank** — the teacher is scored first and its own failures are dropped,
+  so the headline cannot be inflated by items nobody knows. Sources are hash-dropped against the
+  training corpus, and mined from documents rather than hand-written, so the bank cannot grade its
+  author's homework.
+- **Generated derivation problems** — computed answers, parameters outside tutorial ranges, plus a
+  recorded *recall trap* per family: the exact value a model lands on when it quotes a remembered
+  formula with the wrong exponent. Wrong answers say *which* shortcut was taken.
+- **Teacher-forced digit scoring** — separates "can it do the arithmetic" from "can it run its own
+  derivation", in five minutes and with no grading ambiguity.
+- **Position-resolved KL** (in progress) — KL against the parent at every position of a full
+  reasoning trace, up to 24K tokens, on the parent's traces and the student's own. Every heal so
+  far optimised KL on 768-token windows; this measures whether that proxy holds at long horizons.
+- **Pre-registered decision rules.** Each experiment's confirm/refute criteria are written down
+  before the data exists, and results are reported against them either way. Several promising
+  hypotheses have been refuted by their own tests and recorded as such.
+
+## Running it
+
+**llama.cpp** — all 52 layers on the GPU, quantised KV:
+
+```bash
+llama-server -m marlowe-22b-q4_K_M.gguf -ngl 99 \
+  -c 32768 -fa on -ctk q8_0 -ctv q8_0 \
+  --temp 1.0 --top-p 0.95 --top-k 20 \
+  --reasoning-format deepseek
+```
+
+**transformers:**
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained("Marlowe-22B-R1.5")
+model = AutoModelForCausalLM.from_pretrained("Marlowe-22B-R1.5", dtype="bfloat16",
+                                             device_map="auto")
+msgs = [{"role": "user", "content": "Derive the RoPE inverse frequency for head_dim 144, "
+                                    "base 21000, pair index 7."}]
+ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt",
+                              reasoning_effort="xhigh").to(model.device)
+out = model.generate(ids, max_new_tokens=8192, do_sample=True, temperature=1.0,
+                     top_p=0.95, top_k=20)
+print(tok.decode(out[0][ids.shape[-1]:]))
+```
+
+**Sampling — use temperature 1.0, top-p 0.95, top-k 20** (the parent's thinking preset), and do not
+decode greedily: at temperature 0, 24 of 59 traces that used their full thinking budget contained
+verbatim repetition loops. This is a property of the parent's thinking mode, inherited rather than
+introduced by pruning, and sampling is the fix for both models.
+
+The chat template defaults to `reasoning_effort="xhigh"`; `medium` and `low` are supported, as is
+`enable_thinking=False`.
+
+**Speed and memory.** On a 16 GB RTX 4080 SUPER at Q4_K_M with every layer resident and q8_0 KV:
+**~36 tokens/s per stream with two concurrent streams.** Weights are 13.74 GB; a 32K context adds
+roughly 1.1 GB.
+
+## Roadmap — three training rounds scheduled, plus two acceleration variants
+
+| round | goal | status |
+|---|---|---|
+| R1 | heal the pruning loss broadly | done — 27.5 M tokens |
+| R1.5 | close specific probed fact losses | **this release** |
+| **R1.7** | match the 27B on long-context, reasoning-heavy work; close the rotation/trig arithmetic gap | measurement built and running |
+| **R1.8** | professional depth across every major engineering field | scheduled |
+| **R2** | tool use and agentic execution on the strengthened base | scheduled |
+
+Each round is gated on the full instrument set — retention, derivation, arithmetic fidelity and
+long-horizon KL — with checkpoints scored during training rather than only at the end, so a round
+that trades one capability for another is caught while it runs.
+
+Two acceleration variants are planned on the same weights: **-super** (a multi-token-prediction
+draft head, 1.3–1.7× on code) and **-turbo** (EAGLE-3, 3–4× on supporting runtimes). This release
+is the dense build: `mtp_num_hidden_layers` is 0.
+
+The same pipeline then ladders downward — an 18B distilled from this model's successor, and a 9B
+distilled into a native base — reusing the teacher caches and evaluation banks built here.
+
+## Scope of this release
+
+- **Text only.** Image and video inputs are not supported; the vision tower was dropped to spend
+  the budget on reasoning.
+- **No public benchmark scores are claimed.** The held-out benchmark subsets here are too small to
+  publish, and none were used in training. Results above are from the project's own instruments,
+  all of which are described in full so they can be argued with.
+- **Long-horizon parity against the parent is being measured now** and will be reported with R1.7.
+- It inherits the parent's biases, refusals and knowledge cutoff; no alignment training was added.
+- It thinks at length. On multi-stage engineering calculations it will carry more intermediate
+  precision than the answer needs — ask explicitly for the precision you want, and give it room.
+
+## Training data and attribution
+
+Public data, subsampled: **arXiv, open-web-math, AlgebraicStack** (permissive-licensed subset of
+The Stack) and **FineWeb-Edu** — all **ODC-By**, whose attribution travels with this model — plus
+**PG-19** (public domain). Reasoning traces were generated by the parent model itself. No benchmark
+test sets were included; the corpus was hash-checked against the evaluation banks.
+
+## License
+
+**Apache 2.0**, inherited from Qwen3.8-27B. This is a derivative work of that model; the parent's
+terms apply to it and to anything derived from it.