Instructions to use Qwen/Qwen2.5-Coder-7B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Qwen/Qwen2.5-Coder-7B-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Qwen/Qwen2.5-Coder-7B-Instruct") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct") model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Inference
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Qwen/Qwen2.5-Coder-7B-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Qwen/Qwen2.5-Coder-7B-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Qwen/Qwen2.5-Coder-7B-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Qwen/Qwen2.5-Coder-7B-Instruct
- SGLang
How to use Qwen/Qwen2.5-Coder-7B-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Qwen/Qwen2.5-Coder-7B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Qwen/Qwen2.5-Coder-7B-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Qwen/Qwen2.5-Coder-7B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Qwen/Qwen2.5-Coder-7B-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use Qwen/Qwen2.5-Coder-7B-Instruct with Docker Model Runner:
docker model run hf.co/Qwen/Qwen2.5-Coder-7B-Instruct
<tool_call> / </tool_call> embeddings are untrained: LoRA fine-tunes cannot learn to emit tool calls
Summary
In this checkpoint the input-embedding rows of <tool_call> (151657) and </tool_call> (151658) are all zero, and theirlm_head rows are identical (up to bf16 rounding) to those of a cluster of about 2,357 unused tokens (e.g. 151646-151656,
151665-152063, and some low ids such as 124). The chat template still emits these tags for tool calls. Consequences:
- the released model rarely produces hermes-parsable tool calls (in our agent harness: 0 parsed calls in 6,219 turns);
- LoRA fine-tuning on tool-call data cannot fix it, because LoRA leaves
embed_tokensandlm_headfrozen and tokens with
identical lm_head rows always get identical logits:log p(<tool_call>) <= -ln 2357 = -7.77. We measured -7.90 after
training, and 0% of generated turns parsed as tool calls.
Qwen/Qwen2.5-7B-Instruct does not have this: its tag rows are trained and distinct.
Expected
Either trained tag rows (as in Qwen2.5-7B-Instruct) or a note on the model card that the tool-call tags are untrained in
this checkpoint, so fine-tuners know to train them.
Actual (read from the public safetensors, no download of the full model)
| Qwen2.5-Coder-7B-Instruct | Qwen2.5-7B-Instruct | |
|---|---|---|
embed_tokens[151657 / 151658] |
max abs 1.2e-37, L2 0.0 | L2 0.0172 / 0.0152 |
lm_head[151657] vs ids 124, 151646, 151656, 151665, 152063, 151658 |
max abs diff <= 3.1e-5 | 5.6e-3 to the cluster, 6.4e-3 to 151658 |
Reproduce with range_read_rows.py (inline below, stdlib only, HTTP range requests, under 1 MB per model):
python3 range_read_rows.py Qwen/Qwen2.5-Coder-7B-Instruct Qwen/Qwen2.5-7B-Instruct
Or, if the model is already downloaded (this snippet was not run for the draft; the range-read script was):
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct", torch_dtype="bfloat16")
e, h = m.get_input_embeddings().weight, m.get_output_embeddings().weight
print(e[[151657, 151658]].abs().max()) # 0 (up to 1e-37)
print((h[151657] - h[151646]).abs().max(), (h[151657] - h[151658]).abs().max()) # bf16 rounding only
A CPU-only simulation of the fine-tuning failure and the fix (tiny random Qwen2 with the real tokenizer, about 1 minute):repro_tiny.py (inline below). Output: LoRA only peaks at log p -7.80 (cap -ln N = -7.79 for its N = 2,412) and greedy output
is a random token followed by the JSON body; with the fix, -0.04 / -0.05 and a parsable call.
What worked for fine-tuners (workaround)
- Set each zero tag row, in both
embed_tokensandlm_head, to the mean of the rows of its BPE pieces
(<,tool,_call,>= 27, 14172, 13429, 29;</= 522 for the closing tag). - Train only these rows next to LoRA:
LoraConfig(..., trainable_token_indices={"embed_tokens": [151657, 151658], "lm_head": [151657, 151658]})(PEFT >= 0.15; 14,336 extra parameters). - Merge before serving with vLLM (its LoRA loader does not apply trainable-token rows).
With this, the tag log-probabilities reached -0.014 / -0.0002 after 24 steps and about -0.002 / 0.0 at serving time, and
96.2% of evaluation turns were valid tool calls. Re-initialising the rows but keeping them frozen is not enough (the tag
can never become the greedy choice).
Suggested fix
- Model card: a short note that
<tool_call>/</tool_call>(and the other added tokens with zero rows) are untrained in
this checkpoint, with the workaround above; or - if feasible, a checkpoint revision with trained tag rows, as in Qwen2.5-7B-Instruct.
Other sizes of Qwen2.5-Coder were not checked.
range_read_rows.py (stdlib only)
"""Read single embed_tokens / lm_head rows of a public HF model by HTTP range requests (stdlib only, no token).
python3 -I range_read_rows.py Qwen/Qwen2.5-Coder-7B-Instruct [Qwen/Qwen2.5-7B-Instruct ...]
python3 -I range_read_rows.py --selftest
Downloads the safetensors index, each needed shard's header, and one 7 KB row per (tensor, id): well under 1 MB per model.
Prints, per model: max|x| and L2 norm of the input rows of <tool_call>/</tool_call> (151657/151658), and the max abs
difference of lm_head[151657] to a few rows of the untrained-token range and to lm_head[151658].
"""
import json, math, struct, sys, urllib.request
HF = "https://huggingface.co/{repo}/resolve/main/{f}"
TAGS = [151657, 151658]
CLUSTER = [124, 151646, 151656, 151665, 152063] # ids reported in the untrained cluster (pilot LOG.md, PILOT_RUNBOOK_v2.md)
def get(url, rng=None):
req = urllib.request.Request(url, headers={"Range": f"bytes={rng[0]}-{rng[1]}"} if rng else {})
with urllib.request.urlopen(req, timeout=60) as r:
return r.read()
def bf16(b):
return [struct.unpack("<f", b"\x00\x00" + b[i:i + 2])[0] for i in range(0, len(b), 2)]
def rows(repo, tensor, ids, wmap, cache):
f = wmap[tensor]
url = HF.format(repo=repo, f=f)
if f not in cache:
n = struct.unpack("<Q", get(url, (0, 7)))[0]
cache[f] = (8 + n, json.loads(get(url, (8, 8 + n - 1))))
base, hdr = cache[f]
t = hdr[tensor]
assert t["dtype"] == "BF16", t["dtype"]
width = t["shape"][1] * 2
start = base + t["data_offsets"][0]
return {i: bf16(get(url, (start + i * width, start + (i + 1) * width - 1))) for i in ids}
def l2(v):
return math.sqrt(sum(x * x for x in v))
def maxdiff(a, b):
return max(abs(x - y) for x, y in zip(a, b))
def check(repo):
wmap = json.loads(get(HF.format(repo=repo, f="model.safetensors.index.json")))["weight_map"]
cache = {}
emb = rows(repo, "model.embed_tokens.weight", TAGS, wmap, cache)
lm = rows(repo, "lm_head.weight", TAGS + CLUSTER, wmap, cache)
return {"repo": repo,
"embed_max_abs": {i: max(map(abs, emb[i])) for i in TAGS},
"embed_l2": {i: round(l2(emb[i]), 4) for i in TAGS},
"lm_head_151657_maxdiff_to": {i: maxdiff(lm[151657], lm[i]) for i in CLUSTER + [151658]},
"lm_head_l2": {i: round(l2(lm[i]), 4) for i in TAGS}}
def selftest():
assert bf16(struct.pack("<f", 1.0)[2:]) == [1.0]
assert bf16(b"\x00\x00\x80\xbf") == [0.0, -1.0]
assert maxdiff([1, 2], [1, 2.5]) == 0.5 and l2([3, 4]) == 5
print("selftest ok")
if __name__ == "__main__":
if sys.argv[1:] == ["--selftest"]:
selftest(); sys.exit(0)
for r in sys.argv[1:]:
print(json.dumps(check(r)))
repro_tiny.py (CPU simulation of the failure and the fix; also re-run on peft 0.21.2 / transformers 5.22.1 with identical output)
"""CPU reproduction of the tag-row bug on a tiny random Qwen2 (about 1 min, no GPU, no 7B download).
uv run --no-project --python 3.12 --with torch==2.6.0 --with transformers==4.51.3 --with peft==0.15.2 \
--with accelerate==1.6.0 python repro_tiny.py
Needs the 11 MB tokenizer of trl-internal-testing/tiny-Qwen2ForCausalLM-2.5 (real Qwen2.5 vocab, <tool_call> = 151657).
The tiny model is made degenerate the way Qwen2.5-Coder-7B-Instruct is: input rows of 151657/151658 are zero and their
lm_head rows equal those of ~2.4k other rows. Then:
A LoRA only (embed_tokens, lm_head frozen) -> log p(<tool_call>) never exceeds -ln N; greedy output has no parsable call
B rows set to mean of BPE pieces + trainable_token_indices on embed_tokens and lm_head -> log p > -1, parsable call
Also shows the proposed check: target special tokens whose input embedding is all zero.
"""
import json, math, re
import torch
from peft import LoraConfig, get_peft_model
from transformers import AutoTokenizer, Qwen2Config, Qwen2ForCausalLM
OPEN, CLOSE, IM_START, IM_END = 151657, 151658, 151644, 151645
CALL = re.compile(r"<tool_call>(.*?)</tool_call>", re.S) # vLLM hermes parser pattern
STEPS, LR = 60, 5e-3
def degenerate_model():
torch.manual_seed(0)
cfg = Qwen2Config(vocab_size=152064, hidden_size=64, intermediate_size=128, num_hidden_layers=2, num_attention_heads=4,
num_key_value_heads=2, tie_word_embeddings=False, max_position_embeddings=256)
m = Qwen2ForCausalLM(cfg)
cluster = [OPEN, CLOSE] + list(range(151646, 151657)) + list(range(151665, 152064)) + list(range(140000, 142000))
with torch.no_grad():
m.lm_head.weight.mul_(10) # vocab rows get a norm comparable to a trained tag row (see note in the paper)
m.model.embed_tokens.weight[[OPEN, CLOSE]] = 0
m.lm_head.weight[cluster] = m.lm_head.weight[151646].clone()
return m, len(set(cluster))
def zero_embedding_tokens(model, tok, texts=("<tool_call>", "</tool_call>")):
"""The proposed warning: special tokens the data will train on whose input embedding is exactly zero."""
emb = model.get_input_embeddings().weight
return [(t, i) for t in texts for i in [tok.convert_tokens_to_ids(t)] if emb[i].abs().max().item() == 0.0]
def data(tok, n=8):
g = torch.Generator().manual_seed(1)
body = tok('\n{"name": "bash", "arguments": {"cmd": "ls"}}\n', add_special_tokens=False)["input_ids"]
rows = []
for _ in range(n):
pre = [IM_START, 77091, 198] + torch.randint(1000, 30000, (6,), generator=g).tolist() # '<|im_start|>assistant\n' + noise
call = [OPEN] + body + [CLOSE, IM_END]
rows.append((pre, pre + call, [-100] * len(pre) + call))
return rows
def tag_logprob(m, rows):
x = torch.tensor([r[1] for r in rows]); y = torch.tensor([r[2] for r in rows])
with torch.no_grad():
lp = m(input_ids=x).logits[:, :-1].float().log_softmax(-1)
lab = y[:, 1:]
return {t: lp[..., t][lab == t].mean().item() for t in (OPEN, CLOSE)}
def train(m, rows):
x = torch.tensor([r[1] for r in rows]); y = torch.tensor([r[2] for r in rows])
opt = torch.optim.AdamW([p for p in m.parameters() if p.requires_grad], lr=LR, weight_decay=0.0)
best = -1e9
for _ in range(STEPS):
loss = m(input_ids=x, labels=y).loss
opt.zero_grad(); loss.backward(); opt.step()
best = max(best, min(tag_logprob(m, rows).values()))
return best
def greedy_calls(m, tok, rows):
x = torch.tensor([rows[0][0]])
with torch.no_grad():
y = m.generate(input_ids=x, attention_mask=torch.ones_like(x), max_new_tokens=256, do_sample=False,
eos_token_id=[CLOSE, IM_END], pad_token_id=IM_END)
text = tok.decode(y[0, x.shape[1]:].tolist(), skip_special_tokens=False)
calls = []
for body in CALL.findall(text):
try:
c = json.loads(body.strip())
except ValueError:
continue
if isinstance(c, dict) and "name" in c and "arguments" in c:
calls.append(c)
return text, calls
def main():
tok = AutoTokenizer.from_pretrained("trl-internal-testing/tiny-Qwen2ForCausalLM-2.5")
assert tok.convert_tokens_to_ids(["<tool_call>", "</tool_call>"]) == [OPEN, CLOSE]
rows = data(tok)
m, n = degenerate_model()
print("zero-embedding target tokens:", zero_embedding_tokens(m, tok))
a = get_peft_model(m, LoraConfig(r=8, lora_alpha=16, target_modules="all-linear", task_type="CAUSAL_LM"))
best_a = train(a, rows)
text_a, calls_a = greedy_calls(a, tok, rows)
print(f"A LoRA only: cluster N={n}, best tag log-prob {best_a:.3f} (cap -ln N = {-math.log(n):.3f}), "
f"greedy {text_a[:60]!r}, parsed calls {len(calls_a)}")
assert best_a <= -math.log(n) + 1e-3 and not calls_a
m, _ = degenerate_model()
with torch.no_grad(): # init each zero row from the mean of its BPE pieces ('<','tool','_call','>' / '</',...)
bt = tok.backend_tokenizer
for t, i in (("<tool_call>", OPEN), ("</tool_call>", CLOSE)):
p = [x.id for s, _ in bt.pre_tokenizer.pre_tokenize_str(t) for x in bt.model.tokenize(s)] # skip added-token match
m.model.embed_tokens.weight[i] = m.model.embed_tokens.weight[p].mean(0)
m.lm_head.weight[i] = m.lm_head.weight[p].mean(0)
assert zero_embedding_tokens(m, tok) == []
b = get_peft_model(m, LoraConfig(r=8, lora_alpha=16, target_modules="all-linear", task_type="CAUSAL_LM",
trainable_token_indices={"embed_tokens": [OPEN, CLOSE], "lm_head": [OPEN, CLOSE]}))
train(b, rows)
lp_b = tag_logprob(b, rows)
text_b, calls_b = greedy_calls(b, tok, rows)
print(f"B init + trainable rows: tag log-probs {lp_b[OPEN]:.3f} / {lp_b[CLOSE]:.3f}, greedy {text_b[:60]!r}, parsed calls {len(calls_b)}")
assert min(lp_b.values()) > -1.0 and calls_b
print("PASS (bug reproduced, fix works)")
if __name__ == "__main__":
main()