Incompatible with current transformers (5.x) and vLLM: ROPE "default", tied-weights format, mask signature, cache_position, and non-standard head_dim=128 — model loads but generates garbage

#6
by TreeLoys - opened

Model: XingChen-AGI/TeleOCR (revision a32e21d4f766317a6f42f341be57d374362bf843) Use case: full-book OCR of a 335-page Russian chemistry monograph (1956), document parsing → Markdown/LaTeX.

Environment: Python 3.12.16, Windows 11, RTX 5070 12GB, torch 2.14.1+cu132, transformers 5.17.0 (also tested 5.10.0 and 4.51.3), vLLM official Docker image (Nov 2026).

The model card says only pip install -U transformers — no version pin, so I assumed latest 5.x is supported. It is not, currently.

Summary. The custom modeling_naviocr.py targets an older transformers API. With transformers 5.17 the model cannot be loaded without 6 separate patches, and even after patching it loads but produces garbage output. Native vLLM loading also fails due to a non-standard attention shape (head_dim=128 with hidden_size=1024), which neither the stock Qwen2_5_VLForConditionalGeneration nor vLLM can build.

Issue 1 — AutoModelForMultimodalLM is missing from auto_map → falls back to stock class → weight mismatch. auto_map only contains AutoModel, AutoModelForCausalLM, AutoModelForImageTextToText, AutoModelForVision2Seq, so the card-recommended AutoModelForMultimodalLM resolves to the stock class, which builds the text model with head_dim = hidden_size // num_heads = 64 and crashes:

text
model.language_model.layers.{0...27}.self_attn.q_proj.weight | MISMATCH - ckpt: [2048, 1024] vs model: [1024, 1024]
model.language_model.layers.{0...27}.self_attn.k_proj.weight | MISMATCH - ckpt: [1024, 1024] vs model: [512, 1024]
RuntimeError: You set ignore_mismatched_sizes to False ...
The checkpoint was clearly trained with head_dim=128 (16 heads × 128 = 2048, GQA 8×128 = 1024), which config.json does declare — but this loader path ignores it. Workaround: load via AutoModelForImageTextToText (it is mapped to the custom class). Request: add "AutoModelForMultimodalLM" to auto_map or fix the card.

Issue 2 — ROPE: KeyError: 'default'. rope_scaling = {"mrope_section": [16, 24, 24], "rope_type": "default", "type": "default"} → custom code does ROPE_INIT_FUNCTIONS[self.rope_type] (modeling_naviocr.py:508), but transformers 5.x removed "default" from that registry. Additionally, 5.x weight-init calls module.compute_default_rope_parameters(...) on any *RotaryEmbedding with rope_type "default" — the class doesn't implement it → 'NoneType' object is not callable. Note: "default" + mrope_section is itself unusual; stock Qwen2.5-VL does not declare rope this way — I suspect this is why generation stays broken (Issue 6).

Issue 3 — _tied_weights_keys must be a dict in 5.x. _tied_weights_keys = ["lm_head.weight"] → AttributeError: 'list' object has no attribute 'keys' in get_expanded_tied_weights_keys. Embed path here is model.language_model.embed_tokens, not model.embed_tokens.

Issue 4 — cache_position arrives None in prepare_inputs_for_generation (modeling_naviocr.py:1626 → TypeError: 'NoneType' object is not subscriptable). 5.x API change; also past_key_values.get_seq_length() was renamed. Getting this wrong silently re-injects pixel_values at every decode step.

Issue 5 — mask utils signature: create_causal_mask() got an unexpected keyword argument 'input_embeds' (5.17 expects inputs_embeds, no cache_position).

Issue 6 — after all patches: loads, but generation is garbage (fp16 and bf16, greedy, real document pages): iiiiiiii…, tabletabletable…, >{>{>{…. My hypothesis is the mrope handling (Issue 2): the model needs Qwen2.5-VL-style 3D rope with mrope_section [16,24,24] over head_dim=128, but the patched "default" path applies plain 1D rope — vision-token positions end up wrong.

Issue 7 — vLLM cannot load the model natively. Since model_type=qwen2_5_vl, vLLM uses its native implementation and never looks at auto_map:

text
assert sum(self.mrope_section) == rotary_dim // 2 # mrope.py:296 → 64 != 32 (AssertionError)
vLLM derives head_dim = hidden_size // num_heads = 64 and ignores config.head_dim = 128. Patching that line then crashes the vision tower (shared line in qwen2_5_vl.py:671), so a proper fix must be scoped to the text stack.

Requests:

Pin exact supported versions in the model card (transformers/torch/vllm), instead of pip install -U transformers.
Add AutoModelForMultimodalLM to auto_map, or correct the card.
Publish a transformers-5.x-compatible modeling_naviocr.py (or state clearly that transformers <5.0 is required — currently neither 4.51 nor 5.x works out of the box).
Clarify the intended rope config: is rope_type: "default" + mrope_section really what was trained with? A corrected config.json might make the model loadable by stock transformers and vLLM without custom code.
Ideally a vLLM-compatible revision or a GGUF (the head_dim=128 "wide attention" is invisible to both stock transformers and vLLM config parsers).
Patches I applied locally (for reference): (1) rope fallback + compute_default_rope_parameters(); (2) _tied_weights_keys = {"lm_head.weight": "model.language_model.embed_tokens.weight"}; (3) cache_position fallback; (4) mask_kwargs "input_embeds"→"inputs_embeds", dropped cache_position; (5) load via AutoModelForImageTextToText, popped unsupported mm_token_type_ids. Happy to share the full diff or test any fixed revision — the model is #1 on OmniDocBench and I'd really like to use it.

Sign up or log in to comment