config.json's attention_factor: 32 silently lobotomizes this model under vanilla transformers (4.44 → 5.x)
This llamafied mirror produces a severely degraded model when loaded with vanilla transformers — every version from 4.44.1 (the version stamped in the config) through 5.14.1. The weights are fine; the bug is one field in config.json.
Root cause
rope_scaling contains:
"attention_factor": 32,
"factor": 32,
"rope_type": "longrope"
In transformers' modeling_rope_utils.py, _compute_longrope_parameters honors a config-specified attention_factor verbatim and only computes the correct longrope value when the field is absent:
attention_factor = config.rope_scaling.get("attention_factor")
if attention_factor is None:
attention_factor = math.sqrt(1 + math.log(factor) / math.log(original_max_position_embeddings))
For this model (factor = 131072/4096 = 32, original_max_position_embeddings = 4096) the correct value is sqrt(1 + ln(32)/ln(4096)) ≈ 1.19. The config's literal 32 scales the rotary cos/sin tables ~27× too large, saturating attention softmax. The model stays grammatically fluent (weights intact) but loses factual retrieval almost entirely.
microsoft/Phi-3.5-mini-instruct's config has no attention_factor/factor fields and routes through the dedicated phi3 code path, so it is unaffected. This semantics has been unchanged since at least v4.44.1, so the mirror has behaved this way since its upload (2024-08-20).
Minimal repro
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
for name in ["unsloth/Phi-3.5-mini-instruct", "microsoft/Phi-3.5-mini-instruct"]:
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.bfloat16, device_map="cuda").eval()
text = "The capital of France is Paris. Water boils at 100 degrees Celsius at sea level."
enc = tok(text, return_tensors="pt").to("cuda")
with torch.no_grad():
loss = model(**enc, labels=enc.input_ids).loss
prompt = tok("The capital of France is", return_tensors="pt").to("cuda")
out = model.generate(**prompt, max_new_tokens=8, do_sample=False)
print(name, "| loss:", round(loss.item(), 3), "|", tok.decode(out[0], skip_special_tokens=True))
Results (NVIDIA DGX Spark GB10, torch 2.10.0a0+nv25.11, bf16, SDPA — but reproducible anywhere):
| repo | loss on trivial text | greedy completion of "The capital of France is" |
|---|---|---|
| unsloth mirror, transformers 5.14.1 | 5.354 | "…the most important thing in the world of" |
| unsloth mirror, transformers 4.56.2 | 5.354 | (same degraded behavior) |
| unsloth mirror, transformers 4.44.1 | 4.75 | "…a matter of choice." |
| microsoft original, transformers 4.56.2 | 1.328 | "…Paris. Paris is the" |
Local shard sha256s verified against the hub — this is the published artifact, not a corrupted download. The fused→split qkv weight conversion itself is lossless; only the config is wrong.
Downstream impact
NVIDIA's DGX Spark playbook (nvidia.com/dgx-spark/playbooks → unsloth fine-tuning) hardcodes this repo in its validation script; QLoRA training on it shows flat loss ~7 and grad_norm up to 1e17, failing the playbook's own "loss decreasing" criterion. Swapping to microsoft/Phi-3.5-mini-instruct (+ use_exact_model_name=True so unsloth doesn't remap back to this mirror) trains healthily (step-1 loss 2.30, grad_norm 0.2–0.8).
Suggested fix
Delete attention_factor and factor from rope_scaling (letting transformers compute ≈1.19), or set attention_factor to the correct computed value. Alternatively deprecate the llamafied mirror in favor of the original repo, which current unsloth handles natively via its Phi3 path.