devstral-small-2-24b-instruct-2512

Derived from mistralai/Devstral-Small-2-24B-Instruct-2512. Weights revision 55c5b41e. This is that model's implementation on Tenstorrent hardware; see the original model card for training and evaluation details.

Devstral Small 2 is Mistral's 24B agentic coding model, tuned for software engineering work: exploring a repository, editing several files, and driving tools. This package serves its text decoder on Tenstorrent Blackhole through vLLM with tool calling enabled. It is an experimental community bring-up: text only, two devices minimum, and validated on a TT-QuietBox 2. Profiles p300, p150x4, p300x2.

Runs on p300 or p150x4 or p300x2 — see the serve profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Architecture	24B dense decoder-only transformer, FP8 E4M3 weights dequantised to bfloat16 at load; text decoder only
Hardware	p150x4, p300, p300x2
Context	65,536 tokens
License	Apache 2.0
Status	Experimental community bring-up

Intended use

Direct use: Agentic coding over an OpenAI-compatible API — repository exploration, multi-file edits, and tool-driven workflows — for one user or a small batch on a single TT-QuietBox 2.

Out-of-scope use: Image or any non-text input: the Pixtral vision tower is not ported, so this serves the language model only. Also out of scope: a single-device deployment, which does not boot, and context beyond each profile's configured max_model_len, which was not validated.

Quickstart

uv tool install tenstorrent   # once — the Tenstorrent CLI, `tt`
tt model pull anirud/devstral-small-2-24b-instruct-2512
tt serve anirud/devstral-small-2-24b-instruct-2512

tt model pull (or tt-model pull --with-weights) downloads the Docker image and the mistralai/Devstral-Small-2-24B-Instruct-2512 weights at 55c5b41e98c2dbd21b0c8afffc540dcfc9eb5128 (into your HF cache; they are not in the image). tt serve (or tt-model serve) starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.

Without tt-cli — tt-model alone does the whole job:

tt-model pull  anirud/devstral-small-2-24b-instruct-2512 --with-weights
tt-model serve anirud/devstral-small-2-24b-instruct-2512

Validated on a TT-QuietBox 2 (2 x Blackhole p300c, 4 devices), with 2, 4 and 4 devices opened for the p300, p150x4 and p300x2 profiles. Every profile was measured on this image; the figures are under Expected performance.

Serve profiles

One image serves every profile below; pick one with --profile.

profile	hardware	mesh	max_num_seqs	max_model_len
p300	p300	P300	4	32768
p150x4 (default)	p150x4	P150x4	8	65536
p300x2	p300x2	P300x2	32	65536

Using it

The server speaks the OpenAI API at http://127.0.0.1:20000/v1 (or whichever port your serve command reported; chat completions, completions, and /v1/models). Pass "model": "mistralai/Devstral-Small-2-24B-Instruct-2512" — the weights id, not this package's name.

Tool calling is enabled (--tool-call-parser mistral): a request that passes tools comes back with tool_calls and finish_reason: tool_calls.

A complete request, with the sampling recipe this checkpoint expects:

curl http://127.0.0.1:20000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mistralai/Devstral-Small-2-24B-Instruct-2512",
    "messages": [{"role": "user", "content": "Write a Python function that merges two sorted lists."}],
    "temperature": 0.15
  }'

Tool calling is enabled with the Mistral parser. Send an OpenAI tools payload and the reply carries finish_reason: tool_calls with the arguments as JSON; return the result as a role: tool message to continue the exchange. This is not a reasoning model, so no reasoning parser is configured and no thinking block is emitted.

Expected performance

Evaluations

Run with lm_eval 0.4.10.dev0 (the Tenstorrent tstescoTT/lm-evaluation-harness evals-common fork) against this package's own serving stack on the p150x4 profile, sampling at the checkpoint's own recipe (temperature 0.15). No reference run was available, so these are absolute Tenstorrent numbers with no delta: the upstream card publishes only agentic results (SWE-Bench Verified 68.0%, SWE-Bench Multilingual 55.7%, Terminal Bench 2 22.5%), and lm_eval cannot run those harnesses.

task	subset	recipe	result
HumanEval, instruct format	164 problems, all	temperature 0.15, pass@1,create_test	88.4% (145/164); 164/164 answered
HumanEval, raw completion	164 problems, all	same, completions endpoint	39.0% (64/164); 164/164 answered
LiveCodeBench v6	175 of the v6 split	temperature 0.15, 2,048 new tokens, acc,none	32.0%; 174/175 answered

Known quality finding

The two HumanEval rows are the same benchmark on the same build, 49 points apart, and the difference is protocol alone. Posed as a chat turn the model scores 88.4%. Asked to continue a bare function signature it scores 39.0% and sometimes answers in the wrong register — one completion returned # TODO: Implement this function. The instruct row is the fair number for an instruction-tuned model; the raw-completion row is the protocol HumanEval defines. Both are published because choosing one would be a presentation decision, not a measurement.

Against a CPU reference of this same checkpoint, short closed-form answers match exactly, while free-form code diverges token-for-token and stays functionally correct — a bfloat8_b device will not reproduce an fp32 host run verbatim in open-ended generation.

One defect was found and fixed during bring-up and is closed in this image: requests above 1,024 input tokens killed the engine, because empty multimodal keyword arguments reached a text-only prefill signature. Inputs from 128 to 8,192 tokens now pass a standing regression, and a live client session has driven it at 9,900-token prompts.

Sampling note for evaluators

The checkpoint's generation_config.json sets temperature 0.15 and the upstream card uses that value in every example; it is the recipe above. This package serves with sample_on_device_mode: all, so sampling runs on the device rather than the host.

Performance methodology

Closed-loop latency sweep of every profile, measured 2026-10-07 on a TT-QuietBox 2 (2 x Blackhole p300c, 4 devices) with 2, 4 and 4 devices opened for p300, p150x4 and p300x2, served from this image with tt-model serve and driven by vllm bench serve 0.26.0 inside the container. Prompts are random token ids of exactly ISL tokens; output is pinned to OSL tokens with ignore_eos, so every row holds the batch full for its whole length — these are sustained-rate numbers, not natural-conversation ones. One unrecorded warm-up request precedes each input length. N = 4 x users up to 4k input, 2 x users up to 16k, and users beyond. TTFT is the time to the first streamed token; TPOT = (E2EL - TTFT) / (OSL - 1). Values are medians over the row. A 300 s median-E2EL fence was configured to stop the ladder; no row reached it — the slowest measured median E2EL was 277.3 s, at 32,768 input and 32 users on p300x2. Characterization only, no configured targets. First boot compiles kernels and takes about 360 s; that is startup, not inference, and is excluded from every row below.

Column key: ISL/OSL = input/output tokens; Users = concurrent streams; N = prompts in the row; TTFT = time to first token; TPOT = time per output token; Decode = output tokens per second per user; Output = output tokens per second across all users; E2EL = end-to-end latency.

p150x4 — (1, 4) mesh, 65,536-token context; 20 of 20 planned rows measured

ISL	OSL	Users	N	TTFT (ms)	TPOT (ms)	Decode (tok/s/user)	Output (tok/s)	E2EL (s)	Status
128	128	1	4	132	35.5	28.2	28	4.62	ok
128	128	2	8	178	37.0	27.0	52	4.89	ok
128	128	4	16	446	42.3	23.7	88	5.78	ok
128	128	8	32	729	49.5	20.2	146	7.08	ok
1,024	256	1	4	139	36.0	27.8	28	9.32	ok
1,024	256	2	8	194	38.1	26.2	52	9.86	ok
1,024	256	4	16	368	42.5	23.5	92	11.13	ok
1,024	256	8	32	533	49.8	20.1	151	13.54	ok
4,096	256	1	4	497	37.1	26.9	26	9.86	ok
4,096	256	2	8	631	39.4	25.4	48	10.68	ok
4,096	256	4	16	704	43.8	22.8	81	12.39	ok
4,096	256	8	32	859	52.1	19.2	124	15.85	ok
16,384	256	1	2	1,444	41.9	23.9	21	12.12	ok
16,384	256	2	4	1,855	44.1	22.7	34	14.87	ok
16,384	256	4	8	2,009	48.1	20.8	51	20.22	ok
16,384	256	8	16	10,478	57.2	17.5	48	33.55	ok
32,768	256	1	1	318	47.2	21.2	21	12.35	ok
32,768	256	2	2	3,935	63.4	15.8	25	20.10	ok
32,768	256	4	4	8,303	93.5	10.7	25	32.15	ok
32,768	256	8	8	6,452	259.8	3.8	22	72.71	ok

p300x2 — (1, 4) mesh, 65,536-token context; 30 of 30 planned rows measured

ISL	OSL	Users	N	TTFT (ms)	TPOT (ms)	Decode (tok/s/user)	Output (tok/s)	E2EL (s)	Status
128	128	1	4	131	37.7	26.5	26	4.90	ok
128	128	2	8	178	39.7	25.2	49	5.23	ok
128	128	4	16	446	44.6	22.4	83	6.13	ok
128	128	8	32	730	51.6	19.4	140	7.29	ok
128	128	16	64	1,363	73.5	13.6	191	10.70	ok
128	128	32	128	1,771	117.5	8.5	234	17.50	ok
1,024	256	1	4	138	39.1	25.5	25	10.12	ok
1,024	256	2	8	191	41.2	24.2	48	10.70	ok
1,024	256	4	16	373	45.5	22.0	85	11.98	ok
1,024	256	8	32	486	53.5	18.7	142	14.49	ok
1,024	256	16	64	914	74.3	13.5	198	20.69	ok
1,024	256	32	128	1,791	118.9	8.4	239	34.19	ok
4,096	256	1	4	526	43.6	22.9	22	11.62	ok
4,096	256	2	8	698	45.7	21.9	41	12.36	ok
4,096	256	4	16	778	50.2	19.9	72	14.34	ok
4,096	256	8	32	933	58.0	17.3	114	18.01	ok
4,096	256	16	64	5,665	92.3	10.8	138	30.30	ok
4,096	256	32	128	12,445	164.7	6.1	138	54.23	ok
16,384	256	1	2	1,443	61.5	16.2	15	17.14	ok
16,384	256	2	4	1,853	63.4	15.8	26	19.79	ok
16,384	256	4	8	2,005	67.5	14.8	41	25.10	ok
16,384	256	8	16	2,045	80.5	12.4	55	35.64	ok
16,384	256	16	32	53,193	146.9	6.8	35	96.81	ok
16,384	256	32	64	126,224	136.4	7.3	36	157.21	ok
32,768	256	1	1	317	85.0	11.8	12	21.99	ok
32,768	256	2	2	2,826	95.1	10.5	19	27.08	ok
32,768	256	4	4	17,966	92.1	10.9	25	41.46	ok
32,768	256	8	8	16,683	91.6	10.9	24	55.20	ok
32,768	256	16	16	131,445	150.1	6.7	16	169.96	ok
32,768	256	32	32	238,627	150.0	6.7	17	277.29	ok

p300 — (1, 2) mesh, 32,768-token context; 12 of 12 planned rows measured

ISL	OSL	Users	N	TTFT (ms)	TPOT (ms)	Decode (tok/s/user)	Output (tok/s)	E2EL (s)	Status
128	128	1	4	215	47.2	21.2	21	6.21	ok
128	128	2	8	281	48.5	20.6	40	6.46	ok
128	128	4	16	727	52.1	19.2	69	7.38	ok
1,024	256	1	4	224	47.3	21.2	21	12.28	ok
1,024	256	2	8	308	48.9	20.4	40	12.73	ok
1,024	256	4	16	605	52.9	18.9	72	14.08	ok
4,096	256	1	4	802	48.5	20.6	20	12.95	ok
4,096	256	2	8	1,029	50.2	19.9	36	13.99	ok
4,096	256	4	16	1,129	55.2	18.1	63	16.12	ok
16,384	256	1	2	1,691	52.0	19.2	17	14.96	ok
16,384	256	2	4	2,977	53.9	18.5	24	21.04	ok
16,384	256	4	8	3,798	60.8	16.5	32	32.12	ok

Limitations

The Pixtral vision tower is not ported. This package serves the text decoder only; image inputs are not supported and no multimodal endpoint is exposed.

There is no single-device profile, and two devices is the minimum. The weights and KV cache fit in one 32 GB device, but a prefill matmul overruns L1 by about 56 KB: tt_transformers picks the same MLP prefill grids on one device as on four, so one device's cores carry the whole MLP instead of a quarter of it. The configuration was attempted, did not boot, and is deliberately not declared.

max_model_len is a configured serving limit, not the model's. The upstream checkpoint advertises a 256k context; this package configures 32,768 tokens on p300 and 65,536 on p150x4 and p300x2. Longer contexts were not validated here.

All three declared profiles were measured on this image, and every row in the performance tables above completed. Accuracy evaluations were run on p150x4 only.

The image cannot be rebuilt from public source. The tt-metal branch it was built from lives on the build host and has not been pushed, so provenance below reports the commit as not published rather than linking it. The contents are shipped in code/ and are byte-identical to the model code inside the image.

tt-model info cannot resolve a local bundle directory — it accepts only a Hugging Face repo id — so inspecting a staged package before publishing is not possible with that command.

Risks and safety considerations

Two numerical paths in this port differ from a reference implementation and could change results without raising an error. The FP8 E4M3 checkpoint is dequantised to bfloat16 at load using its per-tensor scales, and the Llama-4 style query scaling is computed on the device. Both were checked against a host reference during bring-up — the query scale matched at every position tested, and the dequantisation was asserted against an independent implementation — but accumulated bfloat16 drift over long generations is not something those checks bound.

Free-form generation is not reproducible token-for-token against a host run of the same checkpoint, as described under Known quality finding. Treat exact-output comparisons against a CPU or GPU reference as expected to differ; compare behaviour, not tokens.

Licensing

The weights are Apache-2.0, as declared by the upstream repository; the upstream repository is not gated and ships no separate LICENSE file. The tt-metal model code in code/ is Apache-2.0. No non-commercial or redistribution restriction applies to either.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/anirud/devstral-small-2-24b-instruct-2512/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli).

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	a local checkout — commit not published
vLLM	v0.26.0
vllm-tt-plugin	a local checkout — commit not published
code/ digest	71722ddade285f7f (sha256, first 16 hex digits)
built	2026-10-01T15:37:57+00:00 by tt-model 0.1.0