Fix HF v5 inference
Tracking PR in GPTQModel: https://github.com/ModelCloud/GPTQModel/pull/3535
This upstream fix is the same code that ships in GPTQModel's
ricursivemodel
definition. That PR adds Ripple support plus the load-time compatibility patch and
carries the full test suite (tests/test_ricursive_support.py) and benchmark numbers.
Fix generate() and attention masking under Transformers 5.x
Symptom
Under Transformers 5.x, generate() fails on the very first prefill:
RuntimeError: not enough values to unpack (expected 3, got 2)
prepare_inputs_for_generation does input_ids[:, cache_position] wheneverinput_ids.shape[1] != 1. Transformers 5 pre-creates an empty DynamicCache and passescache_position=None at prefill, so that evaluates input_ids[:, None] and adds a
spurious axis: (batch, seq) becomes (batch, 1, seq). The embedding lookup then yields
4-D hidden states and the first decoder layer dies onbsz, q_len, _ = hidden_states.shape.
That line is only the first of six defects. Padded batches decode to garbage and
generations longer than sliding_window crash.
Bug → Fix
| # | Location | Bug | Fix |
|---|---|---|---|
| 1 | prepare_inputs_for_generation |
input_ids[:, None] on prefill → 4-D hidden states → crash |
rebuild cache_position; slice only when warm |
| 2 | _update_causal_mask |
decode row left fully -inf → constant token; 1.0 for visible keys; removed get_max_cache_shape() |
key axis from cache_position; 0.0 / finfo.min |
| 3 | prepare_inputs_for_generation |
position_ids overwritten per row → wrong RoPE under left padding |
derive from attention_mask.cumsum(-1) - 1 |
| 4 | _apply_sliding_window_mask |
arange(kv_len) assumes keys at 0..kv_len-1; after eviction valid keys masked (window 16 @ pos 32 → 0/16 visible) |
anchor at cp - kv_len + 1; skip when non-binding |
| 5 | _apply_sliding_window_mask |
hybrid cache: layer holds 1024 keys, mask has 1025 → expanded size (1024) must match existing (1025) |
trim mask to [..., -kv_len:] |
| 6 | RicursiveForCausalLM |
transformers 5 drops GenerationMixin for remote code → no generate |
inherit GenerationMixin explicitly |
Notes on the two that are easy to miss:
- #2 only shows up with an explicit padding mask — an all-ones mask returns early. So
single-sequence generation looked healthy while batched evaluation silently scored 0%.
The old code also wrote1.0for visible keys, adding a constant bias to every score. - #4 is a silent correctness bug, not a crash. The window mask progressively excluded
keys that were still valid, degrading long-context output without any error.
Also fixed: position_ids derived by transformers from caller tensors arrive on CPU while
the model is on GPU, so RoPE fails with mat2 is on cpu. input_ids, attention_mask,inputs_embeds and position_ids are now all moved to the model device.
A module-level RICURSIVE_TF5_COMPAT marker is set on the class so downstream loaders that
monkeypatch older checkpoints stand down instead of stacking patches.
Validation
Transformers 5.22.1. "Bit-identical" means exact tensor equality of generated token
ids / logits — not a tolerance comparison.
| Check | Result |
|---|---|
generate() on prefill |
works (previously raised on the first forward) |
| cached vs uncached, 40 tokens | bit-identical, incl. past sliding_window=32 (eviction path) |
| left-padded batch row 0 vs same row unpadded | bit-identical |
| padded vs unpadded prefill logits | bit-identical |
| native file vs independent reference patch — prefill logits | bit-identical |
| native file vs reference patch — decode, padded, uncached | bit-identical |
output.dtype preserved |
yes |
| real 65B GPTQ checkpoint | loads, marker detected, patch stood down, generates correctly |
The native implementation does not merely approximate the reference patch — it produces
exactly the same tokens on every path exercised.
Sync
The identical modeling_ricursive.py (sha256 b2ca81fa07dcdeab…, 30790 bytes) is
published in all three places:
| Location | Revision | Status |
|---|---|---|
| this PR | refs/pr/1 |
source of truth |
ModelCloud/Ripple-65B-Preview-GPTQ-W4A16-v1 |
main |
byte-identical |
ModelCloud/Ripple-65B-Preview-tf5-patch |
main |
byte-identical (mirror + PATCH.diff) |
The GPTQModel PR (#3535) carries the
equivalent fix as a load-time runtime patch (gptqmodel/utils/ricursive_patch.py) for
checkpoints that still hold the original file. The two implementations were compared
directly on the same checkpoint:
| Path | HF PR file vs GPTQModel runtime patch |
|---|---|
| prefill logits | bit-identical |
| prefill logits, padded | bit-identical |
| decode, 40 tokens, cached | bit-identical |
| decode, 40 tokens, uncached | bit-identical |
| decode, padded batch | bit-identical |
Compatibility
Backward compatible with Transformers 4.x: GenerationMixin is importable fromtransformers.generation.utils, and listing it after PreTrainedModel is valid whenPreTrainedModel already inherits it.
Files
modeling_ricursive.py patched drop-in replacement (only file changed)
Mirror of the patched file and a unified diff:
https://huggingface.co/ModelCloud/Ripple-65B-Preview-tf5-patch