README.md

---
license: apache-2.0
base_model: Qwen/Qwen3.5-0.8B-Base
tags:
  - function-calling
  - tool-use
  - distillation
  - small-models
  - mlx
  - on-device
library_name: mlx
pipeline_tag: text-generation
language:
  - en
---

# Scram-0.8B (6-bit): a 603 MB tool-calling model that knows when *not* to call

*(internal designation: corium-v5.4; the training recipe and ledger use that name)*

Qwen3.5-0.8B-Base, distilled from verified Qwen3.5-4B/9B reasoning traces, trained
and evaluated end-to-end on a single MacBook (M4, 64 GB, no CUDA).

**Training data is public:** [negativevoid/scram-curriculum](https://huggingface.co/datasets/negativevoid/scram-curriculum)
(17,837 verified traces, the complete SFT set).

**What it's for:** multi-step procedural reasoning that ends in a well-formed tool
call, with refusal discipline: declining out-of-scope requests and asking for
missing parameters instead of fabricating them.

**What it is not:** a chat assistant, and not a knowledge base. World knowledge is
deliberately *not* in the weights (closed-book ≈ 0.20; ≈ 0.92 with retrieved notes).
Single-turn. English only.

## Numbers

Every number below is co-reported with the ones that look bad. That is the point.

### Official BFCL harness (their generator + evaluator, v4 data, prompting mode)

| category | run-1 | run-2 |
|---|---|---|
| irrelevance (n=240) | 96.67 | 94.17 |
| live_irrelevance (n=884) | 92.42 | 94.46 |
| live_relevance (n=16) | 18.75 | 31.25 |
| simple AST (n=400) | 37.75 | 45.75 |
| multiple / parallel / par-mult | 44.5 / 25.0 / 23.0 | 39.0 / 28.0 / 24.0 |

Irrelevance ~94–97 is high for the size class (board fetched 2026-07-20:
Gemini-2.5-Flash 93.67, ToolACE-2-8B 90.79, Claude-Sonnet-4.5 86.61, Qwen3-8B 79.07
note the board moves, re-verify before quoting). **AST is the weak axis: 46 vs
Hammer2.1-0.5b's 66.** Diagnosed cause is delegate-vs-solve arbitration: the model
answers computable questions directly instead of calling a tool, because the
training curriculum never demonstrated "solvable, but call anyway." Outright
wrong-function selection is rare (29/2311 scored items in run-1).

### Reason-to-act gauntlet (n=300 + 60 traps, identical scoring for all models)

| model | derive→valid-call | missing-param | out-of-scope |
|---|---|---|---|
| **corium-v5.4 (603 MB)** | **0.687** | **30/30** | **30/30** |
| Qwen3.5-4B (its own teacher) | 0.510 (McNemar p=7e-6) | 1/30 | 11/30 |
| Qwen3.5-0.8B-instruct | 0.410 | 8/30 | 7/30 |
| Qwen3.5-2B-instruct | 0.200 | 11/30 | 18/30 |
| Hammer2.1-0.5b | 0.047 | 1/30 | 0/30 |

Traps: 60/60 in this run; 120/120 cumulative across two independent runs.

**On beating its own teacher:** students outperforming teachers after CoT
distillation is a known result ([Ho et al., ACL 2023](https://arxiv.org/abs/2212.10071)),
not a discovery here. Our contribution is a mechanism *hypothesis*: 100% of the
teacher's misses contain the correct derived value in prose, so the differential
looks like derive→valid-call emission reliability rather than reasoning quality.
**This is prose-inspection evidence, not a causal test.** The constrained-decoding
ablation that would settle it has not been run.

## Limitations, confounds, and things that failed

- **GSM8K was trained on** (deliberate scope change). Any GSM8K number here measures
  trained capability, not transfer. It is not a held-out benchmark for this model.
- **Our reasoning battery is our own instrument.** Effect sizes are not comparable
  to anything outside this project.
- **A null result:** adding an explicit "Extraction:" parse step to the traces (v6)
  produced 100% format adoption and *zero* capability gain (0.607 vs 0.610 fresh
  GSM8K), plus a ~4pp reasoning-breadth cost. Making a latent step explicit did not
  create the skill. Underpowered (n=300, ±~5pp CI), so treat as suggestive.
- **When2Call MC: 0.343** under a *generative letter-choice* protocol (the official
  one is loglikelihood, so this is not comparable to their paper). The model refuses
  well procedurally but does poorly at *selecting the description* of the right
  action. Note the likely confound:
  [Ross et al. §5](https://arxiv.org/abs/2504.18851) already report that Qwen-family
  models avoid the "unable to answer" option, so this may be inherited family
  behaviour rather than anything about our curriculum.
- **Single-seed arms.** The teacher-quality comparison at 0.8B (9B-taught 0.917 vs
  4B-taught 0.870) is one seed. At 0.6B, three seeds pooled to −1.5pp (p=0.185, null).
- **Refusal classes do not cross-transfer.** Wrong-tool, missing-param, and
  out-of-scope had to each be taught explicitly; adding one fixed only that one.
- **The tool schemas it learned on are synthetic.** It has never seen a real-world
  function inventory. That is the honest explanation for the AST gap, and it predicts
  where it will be weakest on your tools: unfamiliar naming conventions and large
  menus of similar functions. Expect to add domain schemas if you fine-tune.

## Use

```python
from mlx_lm import load, generate
model, tokenizer = load("negativevoid/Scram-0.8B-6bit")
print(generate(model, tokenizer, prompt="...", max_tokens=256))
```

Note: the model was trained with `enable_thinking=False` templating; matching that
at inference matters. Give it generous token budgets, because it was trained to deliberate
before emitting, and tight caps (e.g. 32) truncate it mid-answer and read as failure.

## Reproduction

Every experiment is a seed-pinned YAML; nulls and failed predictions are logged as
prominently as successes in `DECISIONS.md`.

## License and attribution

Apache-2.0. Derivative of [Qwen/Qwen3.5-0.8B-Base](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base)
(Apache-2.0, © Alibaba/Qwen), modified by supervised fine-tuning on synthetic
distilled traces. Quantized to 6 bits with MLX.