| --- |
| license: apache-2.0 |
| library_name: safetensors |
| tags: |
| - adaptive-inference |
| - routing |
| - decision-model |
| base_model: |
| - Qwen/Qwen3.5-4B |
| - Qwen/Qwen3.8-27B |
| --- |
| |
| # HDM self-routing blocks, fitted on MMLU-Pro |
|
|
| Two small blocks from [**The Value of Thinking: Learning When to Reason in Hybrid |
| Decision Models**](https://arxiv.org/abs/2611.16961). Each one decides, before either mode runs, whether a question |
| is worth thinking about. |
|
|
| A block reads the hidden state at the readout position of the fast pass together |
| with eight statistics of the fast answer, and returns a distribution over the four |
| joint outcomes of the two modes. The fast mode's confidence and the value of |
| thinking are the two marginals of that distribution. The decision costs no forward |
| pass of its own, because it reads a state the fast pass has already produced. |
|
|
| Paper: <https://arxiv.org/abs/2611.16961> |
| Code: <https://github.com/zhengxuyu/HDM-code> |
|
|
| | | `head-4b-mmlu-pro` | `head-27b-mmlu-pro` | |
| |---|---|---| |
| | backbone | `Qwen/Qwen3.5-4B` | `Qwen/Qwen3.8-27B` | |
| | readout | `coded` | `coded` | |
| | layer | `-1`, forced | `-1`, forced | |
| | projection | none, raw hidden state | none, raw hidden state | |
| | input width | 2560 + 8 | 5120 + 8 | |
| | size | 2.6 MB | 5.3 MB | |
|
|
| ## What they were fitted on |
|
|
| MMLU-Pro, 1,154 questions over eleven subject categories, with chemistry, economics |
| and psychology held out: 346 questions the block never saw, split at the level of the |
| category rather than the question. Supervised objective, seed 0. Fitted once and used |
| unchanged on every other benchmark in the paper. |
|
|
| | on the held-out categories | 4B | 27B | |
| |---|---|---| |
| | fast mode accuracy | 0.543 | 0.650 | |
| | slow mode accuracy | 0.853 | 0.887 | |
| | thinking helps | 0.335 | 0.272 | |
| | thinking hurts | 0.026 | 0.035 | |
| | AUROC, value of thinking against where thinking helps | 0.804 | 0.879 | |
| | AUROC, fast answer correct | 0.797 | 0.868 | |
|
|
| Read the last two rows beside the baselines in the report files here. On the 4B a |
| probe over the eight statistics alone scores 0.815 on the first, slightly above the |
| block, and the fast mode's own top probability scores 0.841 on the second. The raw |
| last layer is read anyway, for a reason that is not accuracy: every transformer has |
| a last layer, and a projection fitted on one backbone's features does not transfer |
| to another. |
|
|
| ## Using one |
|
|
| `hde` is the engine from the code repository above; `pip install -e .` there puts |
| it on the path. |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| from hde.core.block import DecisionHead |
| |
| path = hf_hub_download("fredreick/HDM-blocks", "head-4b-mmlu-pro.safetensors") |
| block = DecisionHead.load(path) |
| ``` |
|
|
| Serving refuses a pairing the block was not fitted for: the model id, the revision, |
| the readout and the hidden size are all compared before the block is attached, so a |
| block served against the wrong backbone raises rather than reading noise. |
|
|
| ## A block is fitted to its features, not to a task |
|
|
| Changing the backbone, the layer, the readout or the numeric precision changes the |
| distribution these were fitted on, and the calibration goes with it. A block is a few |
| hundred kilobytes and fits in minutes, so refitting is the ordinary move. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{yu2026valueofthinking, |
| title = {The Value of Thinking: Learning When to Reason in Hybrid Decision Models}, |
| author = {Yu, Zhengxu and Xu, Ke and Fang, Meng and Ammar, Haitham and |
| Luo, Weilin and Wang, Jun}, |
| journal = {arXiv preprint arXiv:2611.16961}, |
| year = {2026}, |
| eprint = {2611.16961}, |
| archivePrefix = {arXiv}, |
| primaryClass = {cs.AI} |
| } |
| ``` |
|
|
