<tool_call> / </tool_call> embeddings are untrained: LoRA fine-tunes cannot learn to emit tool calls

#33
by euisuh - opened

Summary

In this checkpoint the input-embedding rows of <tool_call> (151657) and </tool_call> (151658) are all zero, and their
lm_head rows are identical (up to bf16 rounding) to those of a cluster of about 2,357 unused tokens (e.g. 151646-151656,
151665-152063, and some low ids such as 124). The chat template still emits these tags for tool calls. Consequences:

  • the released model rarely produces hermes-parsable tool calls (in our agent harness: 0 parsed calls in 6,219 turns);
  • LoRA fine-tuning on tool-call data cannot fix it, because LoRA leaves embed_tokens and lm_head frozen and tokens with
    identical lm_head rows always get identical logits: log p(<tool_call>) <= -ln 2357 = -7.77. We measured -7.90 after
    training, and 0% of generated turns parsed as tool calls.

Qwen/Qwen2.5-7B-Instruct does not have this: its tag rows are trained and distinct.

Expected

Either trained tag rows (as in Qwen2.5-7B-Instruct) or a note on the model card that the tool-call tags are untrained in
this checkpoint, so fine-tuners know to train them.

Actual (read from the public safetensors, no download of the full model)

Qwen2.5-Coder-7B-Instruct Qwen2.5-7B-Instruct
embed_tokens[151657 / 151658] max abs 1.2e-37, L2 0.0 L2 0.0172 / 0.0152
lm_head[151657] vs ids 124, 151646, 151656, 151665, 152063, 151658 max abs diff <= 3.1e-5 5.6e-3 to the cluster, 6.4e-3 to 151658

Reproduce with range_read_rows.py (inline below, stdlib only, HTTP range requests, under 1 MB per model):

python3 range_read_rows.py Qwen/Qwen2.5-Coder-7B-Instruct Qwen/Qwen2.5-7B-Instruct

Or, if the model is already downloaded (this snippet was not run for the draft; the range-read script was):

from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct", torch_dtype="bfloat16")
e, h = m.get_input_embeddings().weight, m.get_output_embeddings().weight
print(e[[151657, 151658]].abs().max())                      # 0 (up to 1e-37)
print((h[151657] - h[151646]).abs().max(), (h[151657] - h[151658]).abs().max())   # bf16 rounding only

A CPU-only simulation of the fine-tuning failure and the fix (tiny random Qwen2 with the real tokenizer, about 1 minute):
repro_tiny.py (inline below). Output: LoRA only peaks at log p -7.80 (cap -ln N = -7.79 for its N = 2,412) and greedy output
is a random token followed by the JSON body; with the fix, -0.04 / -0.05 and a parsable call.

What worked for fine-tuners (workaround)

  1. Set each zero tag row, in both embed_tokens and lm_head, to the mean of the rows of its BPE pieces
    (<, tool, _call, > = 27, 14172, 13429, 29; </ = 522 for the closing tag).
  2. Train only these rows next to LoRA: LoraConfig(..., trainable_token_indices={"embed_tokens": [151657, 151658], "lm_head": [151657, 151658]}) (PEFT >= 0.15; 14,336 extra parameters).
  3. Merge before serving with vLLM (its LoRA loader does not apply trainable-token rows).

With this, the tag log-probabilities reached -0.014 / -0.0002 after 24 steps and about -0.002 / 0.0 at serving time, and
96.2% of evaluation turns were valid tool calls. Re-initialising the rows but keeping them frozen is not enough (the tag
can never become the greedy choice).

Suggested fix

  • Model card: a short note that <tool_call> / </tool_call> (and the other added tokens with zero rows) are untrained in
    this checkpoint, with the workaround above; or
  • if feasible, a checkpoint revision with trained tag rows, as in Qwen2.5-7B-Instruct.

Other sizes of Qwen2.5-Coder were not checked.

range_read_rows.py (stdlib only)
"""Read single embed_tokens / lm_head rows of a public HF model by HTTP range requests (stdlib only, no token).
  python3 -I range_read_rows.py Qwen/Qwen2.5-Coder-7B-Instruct [Qwen/Qwen2.5-7B-Instruct ...]
  python3 -I range_read_rows.py --selftest
Downloads the safetensors index, each needed shard's header, and one 7 KB row per (tensor, id): well under 1 MB per model.
Prints, per model: max|x| and L2 norm of the input rows of <tool_call>/</tool_call> (151657/151658), and the max abs
difference of lm_head[151657] to a few rows of the untrained-token range and to lm_head[151658].
"""
import json, math, struct, sys, urllib.request

HF = "https://huggingface.co/{repo}/resolve/main/{f}"
TAGS = [151657, 151658]
CLUSTER = [124, 151646, 151656, 151665, 152063]  # ids reported in the untrained cluster (pilot LOG.md, PILOT_RUNBOOK_v2.md)


def get(url, rng=None):
    req = urllib.request.Request(url, headers={"Range": f"bytes={rng[0]}-{rng[1]}"} if rng else {})
    with urllib.request.urlopen(req, timeout=60) as r:
        return r.read()


def bf16(b):
    return [struct.unpack("<f", b"\x00\x00" + b[i:i + 2])[0] for i in range(0, len(b), 2)]


def rows(repo, tensor, ids, wmap, cache):
    f = wmap[tensor]
    url = HF.format(repo=repo, f=f)
    if f not in cache:
        n = struct.unpack("<Q", get(url, (0, 7)))[0]
        cache[f] = (8 + n, json.loads(get(url, (8, 8 + n - 1))))
    base, hdr = cache[f]
    t = hdr[tensor]
    assert t["dtype"] == "BF16", t["dtype"]
    width = t["shape"][1] * 2
    start = base + t["data_offsets"][0]
    return {i: bf16(get(url, (start + i * width, start + (i + 1) * width - 1))) for i in ids}


def l2(v):
    return math.sqrt(sum(x * x for x in v))


def maxdiff(a, b):
    return max(abs(x - y) for x, y in zip(a, b))


def check(repo):
    wmap = json.loads(get(HF.format(repo=repo, f="model.safetensors.index.json")))["weight_map"]
    cache = {}
    emb = rows(repo, "model.embed_tokens.weight", TAGS, wmap, cache)
    lm = rows(repo, "lm_head.weight", TAGS + CLUSTER, wmap, cache)
    return {"repo": repo,
            "embed_max_abs": {i: max(map(abs, emb[i])) for i in TAGS},
            "embed_l2": {i: round(l2(emb[i]), 4) for i in TAGS},
            "lm_head_151657_maxdiff_to": {i: maxdiff(lm[151657], lm[i]) for i in CLUSTER + [151658]},
            "lm_head_l2": {i: round(l2(lm[i]), 4) for i in TAGS}}


def selftest():
    assert bf16(struct.pack("<f", 1.0)[2:]) == [1.0]
    assert bf16(b"\x00\x00\x80\xbf") == [0.0, -1.0]
    assert maxdiff([1, 2], [1, 2.5]) == 0.5 and l2([3, 4]) == 5
    print("selftest ok")


if __name__ == "__main__":
    if sys.argv[1:] == ["--selftest"]:
        selftest(); sys.exit(0)
    for r in sys.argv[1:]:
        print(json.dumps(check(r)))
repro_tiny.py (CPU simulation of the failure and the fix; also re-run on peft 0.21.2 / transformers 5.22.1 with identical output)
"""CPU reproduction of the tag-row bug on a tiny random Qwen2 (about 1 min, no GPU, no 7B download).
  uv run --no-project --python 3.12 --with torch==2.6.0 --with transformers==4.51.3 --with peft==0.15.2 \
      --with accelerate==1.6.0 python repro_tiny.py
Needs the 11 MB tokenizer of trl-internal-testing/tiny-Qwen2ForCausalLM-2.5 (real Qwen2.5 vocab, <tool_call> = 151657).
The tiny model is made degenerate the way Qwen2.5-Coder-7B-Instruct is: input rows of 151657/151658 are zero and their
lm_head rows equal those of ~2.4k other rows. Then:
  A  LoRA only (embed_tokens, lm_head frozen)  -> log p(<tool_call>) never exceeds -ln N; greedy output has no parsable call
  B  rows set to mean of BPE pieces + trainable_token_indices on embed_tokens and lm_head -> log p > -1, parsable call
Also shows the proposed check: target special tokens whose input embedding is all zero.
"""
import json, math, re
import torch
from peft import LoraConfig, get_peft_model
from transformers import AutoTokenizer, Qwen2Config, Qwen2ForCausalLM

OPEN, CLOSE, IM_START, IM_END = 151657, 151658, 151644, 151645
CALL = re.compile(r"<tool_call>(.*?)</tool_call>", re.S)  # vLLM hermes parser pattern
STEPS, LR = 60, 5e-3


def degenerate_model():
    torch.manual_seed(0)
    cfg = Qwen2Config(vocab_size=152064, hidden_size=64, intermediate_size=128, num_hidden_layers=2, num_attention_heads=4,
                      num_key_value_heads=2, tie_word_embeddings=False, max_position_embeddings=256)
    m = Qwen2ForCausalLM(cfg)
    cluster = [OPEN, CLOSE] + list(range(151646, 151657)) + list(range(151665, 152064)) + list(range(140000, 142000))
    with torch.no_grad():
        m.lm_head.weight.mul_(10)  # vocab rows get a norm comparable to a trained tag row (see note in the paper)
        m.model.embed_tokens.weight[[OPEN, CLOSE]] = 0
        m.lm_head.weight[cluster] = m.lm_head.weight[151646].clone()
    return m, len(set(cluster))


def zero_embedding_tokens(model, tok, texts=("<tool_call>", "</tool_call>")):
    """The proposed warning: special tokens the data will train on whose input embedding is exactly zero."""
    emb = model.get_input_embeddings().weight
    return [(t, i) for t in texts for i in [tok.convert_tokens_to_ids(t)] if emb[i].abs().max().item() == 0.0]


def data(tok, n=8):
    g = torch.Generator().manual_seed(1)
    body = tok('\n{"name": "bash", "arguments": {"cmd": "ls"}}\n', add_special_tokens=False)["input_ids"]
    rows = []
    for _ in range(n):
        pre = [IM_START, 77091, 198] + torch.randint(1000, 30000, (6,), generator=g).tolist()  # '<|im_start|>assistant\n' + noise
        call = [OPEN] + body + [CLOSE, IM_END]
        rows.append((pre, pre + call, [-100] * len(pre) + call))
    return rows


def tag_logprob(m, rows):
    x = torch.tensor([r[1] for r in rows]); y = torch.tensor([r[2] for r in rows])
    with torch.no_grad():
        lp = m(input_ids=x).logits[:, :-1].float().log_softmax(-1)
    lab = y[:, 1:]
    return {t: lp[..., t][lab == t].mean().item() for t in (OPEN, CLOSE)}


def train(m, rows):
    x = torch.tensor([r[1] for r in rows]); y = torch.tensor([r[2] for r in rows])
    opt = torch.optim.AdamW([p for p in m.parameters() if p.requires_grad], lr=LR, weight_decay=0.0)
    best = -1e9
    for _ in range(STEPS):
        loss = m(input_ids=x, labels=y).loss
        opt.zero_grad(); loss.backward(); opt.step()
        best = max(best, min(tag_logprob(m, rows).values()))
    return best


def greedy_calls(m, tok, rows):
    x = torch.tensor([rows[0][0]])
    with torch.no_grad():
        y = m.generate(input_ids=x, attention_mask=torch.ones_like(x), max_new_tokens=256, do_sample=False,
                       eos_token_id=[CLOSE, IM_END], pad_token_id=IM_END)
    text = tok.decode(y[0, x.shape[1]:].tolist(), skip_special_tokens=False)
    calls = []
    for body in CALL.findall(text):
        try:
            c = json.loads(body.strip())
        except ValueError:
            continue
        if isinstance(c, dict) and "name" in c and "arguments" in c:
            calls.append(c)
    return text, calls


def main():
    tok = AutoTokenizer.from_pretrained("trl-internal-testing/tiny-Qwen2ForCausalLM-2.5")
    assert tok.convert_tokens_to_ids(["<tool_call>", "</tool_call>"]) == [OPEN, CLOSE]
    rows = data(tok)

    m, n = degenerate_model()
    print("zero-embedding target tokens:", zero_embedding_tokens(m, tok))
    a = get_peft_model(m, LoraConfig(r=8, lora_alpha=16, target_modules="all-linear", task_type="CAUSAL_LM"))
    best_a = train(a, rows)
    text_a, calls_a = greedy_calls(a, tok, rows)
    print(f"A LoRA only: cluster N={n}, best tag log-prob {best_a:.3f} (cap -ln N = {-math.log(n):.3f}), "
          f"greedy {text_a[:60]!r}, parsed calls {len(calls_a)}")
    assert best_a <= -math.log(n) + 1e-3 and not calls_a

    m, _ = degenerate_model()
    with torch.no_grad():  # init each zero row from the mean of its BPE pieces ('<','tool','_call','>' / '</',...)
        bt = tok.backend_tokenizer
        for t, i in (("<tool_call>", OPEN), ("</tool_call>", CLOSE)):
            p = [x.id for s, _ in bt.pre_tokenizer.pre_tokenize_str(t) for x in bt.model.tokenize(s)]  # skip added-token match
            m.model.embed_tokens.weight[i] = m.model.embed_tokens.weight[p].mean(0)
            m.lm_head.weight[i] = m.lm_head.weight[p].mean(0)
    assert zero_embedding_tokens(m, tok) == []
    b = get_peft_model(m, LoraConfig(r=8, lora_alpha=16, target_modules="all-linear", task_type="CAUSAL_LM",
                                     trainable_token_indices={"embed_tokens": [OPEN, CLOSE], "lm_head": [OPEN, CLOSE]}))
    train(b, rows)
    lp_b = tag_logprob(b, rows)
    text_b, calls_b = greedy_calls(b, tok, rows)
    print(f"B init + trainable rows: tag log-probs {lp_b[OPEN]:.3f} / {lp_b[CLOSE]:.3f}, greedy {text_b[:60]!r}, parsed calls {len(calls_b)}")
    assert min(lp_b.values()) > -1.0 and calls_b
    print("PASS (bug reproduced, fix works)")


if __name__ == "__main__":
    main()

Sign up or log in to comment