Clef-flash at 24 of 32 layers

Cloudflare's Clef-flash is a decision model: it reads a piece of text and a question and returns a probability for every allowed answer, without generating text. This repository is the same model with layers 24–31, the text head, the vision encoder and Cloudflare's 122M-parameter decision head removed, and the answer read from layer 24 with a small linear head instead.

On tasks where the answer is in the text (reading comprehension, entailment, picking an intent label) it matches the full model or beats it. On tasks that need specialised knowledge (MMLU-Pro, some legal-doctrine questions) it is clearly worse. It needs about a quarter less compute, and about 4 GB of memory at 4-bit on a Mac.

Credit where it is due: the underlying functionality is Cloudflare's. The same recipe on the base model Clef-flash was trained from (Qwen3.5-9B) loses 13–53 points (see below). What this repository shows is that, after their training, the decision is already made by layer 24.

Results

Against the full model, same questions

Decision Index public 0.3 suite (kit @ f5a6e4f), fixed random samples of 500 questions per benchmark (seed 2026), the full model run with Cloudflare's own released code and head. The numbers are accuracy.

Benchmark Full Clef-flash This cut This cut, 4-bit
WinoGrande 97.2% 96.6% 96.0%
BBH (fixed-option) 68.8% 68.8% 67.6%
HellaSwag 98.2% 98.4% 97.4%
MuSR 87.0% 85.6% 84.2%
MMLU-Pro 64.4% 57.4% 56.0%
BANKING77 90.4% 93.4% 93.2%
CLINC150 + out-of-scope 67.6% 91.8% 90.8%

On WinoGrande the two gave the same answer on 491 of 500 questions. On CLINC150 the full model, with Cloudflare's own head, answers "out of scope" for 231 of 500 questions when only 95 are genuinely out of scope; at layer-24 it does not have that bias.

Where the MMLU-Pro gap comes from. The same head read from the full model at layer 32 scores 60.4%, against 57.4% at layer 24 and 64.4% for Cloudflare's head. So about 3 of the 7 points live in layers 25–32, and about 4 are Cloudflare's head beating a single linear vector at the same depth.

Reading vs recall. Across 29 tasks (these seven, our nine yes/no families, fourteen LegalBench tasks), labelled "reading" or "recall" in advance by an independent annotator who never saw the results: on reading tasks the cut never loses more than 2 points (mean +2.8); on recall tasks it averages βˆ’4.5, driven by MMLU-Pro and two legal-doctrine tasks, while plain factual recall (which country a city is in) is unaffected. The split is significant on the pre-registered permutation test (p = 0.015) but not on a rank test (p = 0.054).

Yes/no questions, our own held-out tasks

2,700 questions across nine task families none of which were used for training (true/false statements about cities, companies, translations, numbers, common claims and counterfactuals; reading comprehension; inference; sentiment).

Accuracy AUROC Log loss Calibration error (ECE)
Full Clef-flash with its own head 91.4% 0.960 0.221 0.062
This cut, full precision 91.3% 0.959 0.220 0.058
This cut, 4-bit (MLX) 91.1% 0.955 0.226 0.059

Refitting the head on different samples of its training data moves log loss by about Β±0.008, so the three rows are within noise of each other.

The base-model control

The same heads trained the same way on base Qwen3.5-9B (layers 0–23), on the same questions:

Benchmark Clef-flash cut Base Qwen3.5-9B cut
WinoGrande 96.8% 68.5%
MuSR 84.5% 52.0%
HellaSwag (2,000-question sample) 98.6% 78.7%
BANKING77 (1,000-question sample) 93.6% 40.5%
BBH 67.3% 56.1%
MMLU-Pro 58.2% 43.9%

Cloudflare's training data is not public and may include the training splits of some of these benchmarks.

Size and speed

Memory (short prompts) Download Speed per question (Apple M5 Max)
Full Clef-flash backbone 18.5 GB 19.1 GB 62 ms
This cut, PyTorch bf16 (NVIDIA L4) 12.8 GB 12.4 GB 68 ms (L4)
This cut, PyTorch 4-bit NF4 (NVIDIA L4) 5.0 GB weights 12.4 GB, quantised on load 63 ms (L4)
This cut, MLX full precision 12.8 GB 12.4 GB 40 ms
This cut, MLX 4-bit 4.2 GB 3.5 GB 37 ms

Lower than 4-bit with plain affine quantization costs real accuracy (3-bit: 88.5% on the yes/no tasks). Several community builds of the full model reach a similar size with careful mixed-precision quantization and keep every feature; this cut is a different trade, not a smaller one.

Use

There are two versions of the model, PyTorch and Apple's MLX. Scoring the same on our held-out tests.

PyTorch (NVIDIA, CPU, Mac): pytorch/

from reader_torch import Reader
r = Reader.load("pytorch", device="cuda")      # or "cpu" / "mps"; quantize="nf4" for 4-bit on NVIDIA
r.predict("The city of Kagoshima is in Japan.", "Is this statement true?")             # P(yes) β‰ˆ 0.98
r.choose("Which planet is largest?", "Which option answers the question?",
         {"A": "Mars", "B": "Jupiter", "C": "Venus"})                                 # {"B": 0.9997, ...}

MLX (Apple silicon, 4-bit): mlx/, same predict and choose calls.

The two calls cover yes/no and multiple-choice questions. Clef's "score" (ordered) questions are not supported.

How it was built and measured

  • The cut: layers 0–23 are copied from Cloudflare/clef-flash, and none of them were retrained.
  • Yes/no head: logistic regression on the layer-24 activations at three positions (end of the text, end of the question, last prompt token): 12,288 weights. Trained on 7,623 items from SNLI, WANLI, FEVER, StrategyQA, ARC, CommonsenseQA, HellaSwag, PAWS and DBpedia-14.
  • Multiple-choice head: each option's tokens are averaged at layer 24 and scored by one shared 4,096-number vector, softmax across options. Trained on ARC and CommonsenseQA training splits.
  • No Decision Index data was used for training. Every benchmark was asked with the same prompt wording, and no input was cut short to fit.
  • Checks: shuffling the order of the options leaves the scores unchanged (e.g. WinoGrande 96.8% β†’ 97.4%).
  • Caveats: samples rather than the full suite; one training run per head; one machine. The Decision Index board requires the full suite, so these numbers are not a board submission. HellaSwag's training split was used for the yes/no head (not the multiple-choice head that produced the HellaSwag score above).

Limits

  • It answers yes/no and multiple-choice questions only. It cannot generate free text, answer ordered "score" questions, or read images.
  • Knowledge-heavy multiple choice (MMLU-Pro) is clearly below the full model.
  • Date arithmetic is at chance (54.5% on deadline-window questions); use a solver.
  • Negated questions ("Is this statement false?") are read correctly about 80% of the time.
  • It was only tested in English. Memory use on long documents has not been measured for the 4-bit version.

Prior art

Reading a classifier off a middle layer and dropping the layers above is not new: see Chip-Tuning (Zhu et al., 2024), "Why Lift so Heavy?" (Yuan et al., 2024), Layer Enhanced Classification (Sawtell et al., 2024), and work on redundant deep layers (Gromov et al., 2024; ShortGPT, 2024). This repository applies it to a model post-trained for decisions and compares it against that model's own purpose-built head.

Licence

Apache-2.0, as the models it is built from: Cloudflare/clef-flash, post-trained from Qwen/Qwen3.5-9B. Head training data: permissive licences only (non-commercial and undeclared-licence datasets excluded). Decision Index benchmark data was built locally for evaluation and is not redistributed here.

Downloads last month

-

Downloads are not tracked for this model. How to track
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for lanestp/clef-flash-cut-L24-4bit

Finetuned
Qwen/Qwen3.5-9B
Quantized
(64)
this model