lukeslp/drummer2-tools

Drummer-540M Tools 0.10.0

This repository contains Tools 0.10.0, a narrow tool-proposal fine-tune of the 542M-parameter Drummer base at 10.85B pretraining tokens. It is trained for a fixed catalog of tools. It is a research model, not a general agent and not a product.

Training

• Parent: the completed 542M base checkpoint at 10.85B tokens (drummer2-540m-100pct-step-82750), not the later continued Foundation 0.12.0 checkpoint.
• Corpus: tools-declared-v6, 11,054 records; profile declared-v2.
• Three passes, 204 updates, 1,656,936 supervised target tokens; learning rate 1e-4.
• The corpus is execution-verified against the tool executor. It teaches six local tools and four live web tools plus refusal/clarification behavior. Tool results are not model-written facts.

Results

Measure	Tools 0.10.0	Comparison / note
Sealed execution test	116/120 (96.7%)	Tools 0.9.0: 107/120
Live web tools	56/60 (93%)	Tools 0.9.0: 55/60
Prohibited-tool calls	0/100	no false calls in this set
Transfer-v2 total	70/96	prior model: 43/96
Held-out tools declared in the prompt but absent from training	0/24	prior model: 0/24
Last-N-character probe	16/16	prior model: 0/16
Clarification	10/12	prior model: 6/12
Precision at coverage, transfer test	1.000 at 0.647 coverage	prior model: 0.677 at 0.456
p95 latency, Apple-GPU (MLX) serving path	1.2 s	recorded qualification run

The held-out result is the central limitation. Fine-tuned on the same rows, Qwen2.5-0.5B-Instruct made a valid call naming the unseen tool in 11/24 and SmolLM2-360M-Instruct in 10/24; Drummer made no valid selection (0/24). Exact-and-executed scores were 7/24, 8/24 and 0/24 respectively. Qwen2.5-0.5B base, with no instruction tuning, made a valid call naming the unseen tool in 8/24 and executed 7/24. OPT-350M, Pythia-410M and BLOOM-560M controls, pretrained on 180B to 350B tokens, also made no valid selection (0/24); counting the name in raw output they emitted it 2, 8 and 10 times, every one invalid.

A preregistered function-masking experiment replaced every training example's tool and argument identifiers with fresh opaque aliases. The primary result was 0/24 held-out tools: masking did not recover declaration reading. It also decisively harmed trained-tool execution, which fell from 116/120 to 88/120 (31 paired losses, 3 gains; p = 7.7e-07); invalid responses rose from 2 to 21, precision fell from 0.971 to 0.784 and coverage from 0.972 to 0.708. A post-hoc 4-to-9 shift in correct-name emissions (invalid calls that name the right tool) was unregistered, one-seed evidence from five discordant pairs (one-sided p = 0.0312, two-sided 0.0625); the same signature appears without masking in the sub-350B reference arms, so it reads as more copying, not recovered capability. Across the tested model families, the observed capability boundary lies within the 350B-to-roughly-4T pretraining interval; this does not prove a universal scaling threshold or decoder-architecture limit.

Use

Use this checkpoint only for research on fixed-catalog tool calling, execution-verified training and the difference between memorizing a tool catalog and reading a novel declaration. The package uses Drummer's inference_v2 format and includes its tokenizer and manifest.

Limitations and safety

• No safety tuning. Do not grant it unsupervised access to external systems or consequential tools.
• It is good at the exact trained catalog and fails to use unseen tools from prompt declarations: 0/24 on the sealed test.
• It can propose wrong tools or arguments. Validate every proposal against a strict allowlist and schema, execute at most once, and keep side effects behind human confirmation.
• It invents facts and should not answer factual questions from its weights. Use deterministic tools and quote tool outputs.
• The 10.85B-token Foundation 0.11.0 parent scored 0.257 on MMLU, chance level. General knowledge and reasoning are weak.
• Context length is 2,048 tokens. The model is primarily English.
• This repository is not the continued 15.85B-token foundation. It is fine-tuned from the earlier 10.85B base.

License and author

Weights and code: Apache-2.0. Card and project-authored documentation: CC BY 4.0. Training-source terms remain as documented for the Drummer base (https://huggingface.co/lukeslp/drummer2-540m). About 8% of that base's tokens were smol-smoltalk, text generated by Llama-3.1-405B; that model's license includes a naming clause, and this fine-tune inherits the disclosure. The function-masking design follows Lin et al., 2024 (Hammer, arXiv:2410.04587).

Luke Steuber - lukesteuber.com - GitHub

Full evidence and live demo: https://drummer-demo.dr.eamer.dev