SmolLM2-360M-SmartHome-WeightEdit

A smart-home tool-calling model made from HuggingFaceTB/SmolLM2-360M-Instruct by direct, closed-form weight editing β€” no gradient training, no training loop. 1,966,080 weights (0.54% of the model) in the MLP output projections of layers 21, 23, 25 and 27 were changed.

It turns smart-home requests (multi-turn) into calls to six tools: toggle_lights, set_thermostat, lock_door, get_device_status, set_scene and intent_unclear. The task definition and tool schema come from distil-labs/distil-home-assistant-qwen3 (Apache-2.0); see that model card for the full schema and system prompt, which this model uses unchanged.

Note. This release is mainly a demonstration of how far weight editing alone can move a weak base model (official test 24% β†’ 66%). Its absolute accuracy is still modest β€” roughly one request in three is mishandled β€” so it is not suitable for controlling real devices.

Usage

SmolLM2's chat template ignores the tools= argument, so the tool list goes into the system prompt. This is exactly the prompt format used in the evaluation:

import json
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "aitos-io/SmolLM2-360M-SmartHome-WeightEdit"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

SYSTEM = "..."  # the system prompt from the distil-labs/distil-home-assistant-qwen3 model card, unchanged
TOOLS = [...]   # the six tool definitions from the same model card, unchanged
system = (SYSTEM + "# Tools\n\n<tools>\n" + "\n".join(json.dumps(t) for t in TOOLS) + "\n</tools>\n\n"
          "For the function call, return a json object with function name and arguments within "
          "<tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>")
# earlier assistant tool calls in the history are written as plain JSON text: {"name": ..., "arguments": {...}}
messages = [{"role": "system", "content": system}, {"role": "user", "content": "Is the front door locked?"}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(text, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

The model answers either with bare JSON or wrapped in <tool_call> tags; accept both.

Results

All numbers below were measured on the files in this repository, after merging and saving in the released precision.

Tool-call accuracy (95% CI) This model HuggingFaceTB/SmolLM2-360M-Instruct (unedited) distil-labs/distil-home-assistant-qwen3 (fine-tuned, reference)
Official test set (distil-smart-home test.jsonl, n=50) 66.0% (52.2%–77.6%) 24.0% (14.3%–37.4%) 92.0% (81.2%–96.8%)
Official seed set (distil-smart-home train.jsonl, n=50) 66.0% (52.2%–77.6%) 32.0% (20.8%–45.8%) 98.0% (89.5%–99.6%)
Independent held-out set C (n=120) 58.3% (49.4%–66.8%) 20.0% (13.8%–28.0%) 91.7% (85.3%–95.4%)
Practice set B (n=120) 75.8% (67.4%–82.6%) 25.8% (18.8%–34.3%) 98.3% (94.1%–99.5%)
General ability This model HuggingFaceTB/SmolLM2-360M-Instruct (unedited) distil-labs/distil-home-assistant-qwen3 (fine-tuned, reference)
ARC-Easy (200 test questions, letter answer) 22.5% 23.5% 36.5%
WikiText-2 perplexity (40 paragraphs) 24.29 23.76 48.25

The official test set was not used for any decision during editing; it was evaluated once, on the final model. Set C was written by an independent agent that could not see the editing code or the practice data. Intervals are Wilson 95% intervals; the official sets are small (50 items), so differences of a few points are within noise.

Honest comparison. The fine-tuned reference was trained (with gradients) on synthetic data for this exact task and is still more accurate. This model shows how far editing alone can take a general model, without any training.

How it was made

  1. Diagnosis. The unedited model fails mainly at a few decision tokens: the function name (e.g. answering "is the front door locked?" with lock_door instead of get_device_status), refusals (intent_unclear) and output format. Linear probes show the user's intent is 100% decodable from middle layers while the output is right only 24% of the time on practice data β€” an obstacle, not a missing capability.
  2. Closed-form protected edit. For each of layers 21, 23, 25 and 27: teacher-force the gold answers on practice data, collect the MLP input activations at positions the model gets wrong (keys K) and the hidden-state shift that would make the gold token beat its strongest competitor (values R), then solve Ξ”W = Rα΅€ K (Kα΅€K + λ·nΒ·C)⁻¹, where C is the activation covariance of positions that must not change: already-correct answers (weighted 4Γ—), prompt tokens, ordinary text (WikiText-2) and general question answering (ARC-Easy train, weighted 4Γ—). Each layer's edit is applied at dose 0.85 before the next layer is solved.
  3. Gates. An edit is kept only if practice accuracy rises by β‰₯ 2 points, already-good categories do not drop, mean KL on ordinary text stays ≀ 0.03 and ARC-Easy drops ≀ 2 points.
  • Released in float32. Saved in bfloat16, the same edit lost 3 points on ARC-Easy (26.0% β†’ 23.0%), failing the ≀ 2-point gate; in float32 it is 23.5% β†’ 22.5%. Note that both are close to chance (25% for four options): ARC-Easy says little about this model's general ability, and WikiText-2 perplexity (23.76 β†’ 24.29) is the more informative check here.
  • Protection strength Ξ» = 2 (1 for the other releases): with Ξ» = 1 the edit reached 70.8% on held-out set C but changed ordinary-text predictions too much (KL 0.066 > 0.03).
  • During teacher forcing the gold answer follows the model's own format choice per example (bare JSON or <tool_call>-wrapped), so the edit corrects decisions, not formatting habits.
  • The fine-tuned reference is built on a different base (Qwen3-0.6B); it is listed only as an indication of what task-specific training reaches on the same data.
  • Practice data: 240 template cases plus 300 diverse cases written by an independent agent; dose and gates were checked on a separate 120-case practice set.

Limitations

  • Covers only the six tools and the room / door / scene values of the distil-smart-home schema.
  • Practice and held-out sets were generated by AI agents (templates and free writing), not collected from real users.
  • Evaluation sets are small; treat point estimates with care.
  • The edit was tuned for this task only; behaviour outside it was checked only via WikiText-2 perplexity and ARC-Easy.

License

Apache-2.0, same as the base model. Base model: HuggingFaceTB/SmolLM2-360M-Instruct.

Downloads last month
49
Safetensors
Model size
0.4B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for aitos-io/SmolLM2-360M-SmartHome-WeightEdit

Finetuned
(209)
this model