File size: 4,880 Bytes
ede5883 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 | ---
license: apache-2.0
library_name: transformers
tags:
- ricursive
- ripple
- transformers
- patch
---
# Ripple-65B-Preview — Transformers 5.x compatibility patch
Drop-in replacement for `modeling_ricursive.py` from
[`ricursiveAI/Ripple-65B-Preview`](https://huggingface.co/ricursiveAI/Ripple-65B-Preview).
The upstream custom code was written against the Transformers 4.x cache API.
Under Transformers 5.x, `generate()` fails on the very first prefill, and even
after that single line is fixed, padded batches decode incorrectly and
generations longer than `sliding_window` crash.
This repository distributes the fix only. It contains **no weights**; use it
with the original checkpoint.
## Apply
Copy the patched file over the upstream one:
```bash
cp modeling_ricursive.py /path/to/Ripple-65B-Preview/modeling_ricursive.py
```
Or apply the unified diff from the root of a checkout of the model repository:
```bash
git apply PATCH.diff
```
Then load as usual:
```python
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"ricursiveAI/Ripple-65B-Preview",
trust_remote_code=True,
)
```
## What is broken upstream
1. **Spurious axis on prefill.** `prepare_inputs_for_generation` does
`input_ids[:, cache_position]` whenever `input_ids.shape[1] != 1`.
Transformers 5 pre-creates an empty `DynamicCache` and passes
`cache_position=None` for the prefill call, so this becomes
`input_ids[:, None]`: `(batch, seq)` turns into `(batch, 1, seq)`. The
embedding lookup returns 4-D hidden states and the first decoder layer dies
on `bsz, q_len, _ = hidden_states.shape`.
2. **Every decode step fully masked.** `_update_causal_mask` seeds the mask with
`finfo.min` and only writes the visible diagonal via `triu` when
`sequence_length != 1`. At decode `sequence_length == 1`, so the whole row
stays masked and the model emits a constant token. The function also writes
`1.0` (not `0.0`) for visible keys, which biases every score, and calls
`past_key_values.get_max_cache_shape()`, which Transformers 5 removed.
3. **`position_ids` ignore padding.** They are overwritten with
`cache_position` for every row, which is only correct when nothing is padded.
With left padding, a short row's real tokens sit at higher indices, so a
shared position vector shifts its RoPE phases and the row decodes to garbage.
4. **Sliding window uses the wrong absolute positions.**
`_apply_sliding_window_mask` builds `arange(kv_len)` and assumes the keys sit
at positions `0..kv_len-1`. Once the hybrid cache evicts from the left they
sit at `[cp-kv_len+1 .. cp]`, so the mask excludes progressively more valid
keys (e.g. window 16 at position 32 leaves 0 of 16 keys visible).
5. **Mask shape crash past the window.** Transformers 5 uses a hybrid cache:
sliding layers hold at most `sliding_window` keys while full-attention layers
keep growing. The model builds one mask per forward, so the shape no longer
matches the sliding layer's key tensor
(`RuntimeError: expanded size (1024) must match existing size (1025)`).
6. **`position_ids` left on CPU.** Transformers derives them from the caller's
tensors before the generation hook runs, so a CPU prompt yields CPU positions
even after the inputs are moved, and RoPE fails with `mat2 is on cpu`.
7. **Missing `GenerationMixin`.** Transformers 5 no longer injects it into
remote-code classes, so `generate` can be missing entirely and
`generation_config` is never built.
## What the patch changes
* `prepare_inputs_for_generation` rebuilds a missing `cache_position`, slices
only when the cache is warm, derives per-row `position_ids` from the padding
mask, and moves `input_ids` / `attention_mask` / `inputs_embeds` /
`position_ids` onto the model device.
* `_update_causal_mask` sizes the key axis from `cache_position`, emits `0.0`
for visible keys and `finfo.min` elsewhere, and never returns an all-masked
single-token decode row.
* `_apply_sliding_window_mask` trims the shared mask to the keys the layer
holds, anchors key positions at the first key the cache still holds, and is
skipped when the window cannot bind.
* `RicursiveForCausalLM` inherits `GenerationMixin` explicitly.
The patch is backward compatible with Transformers 4.x.
## Validation
* `generate()` works under Transformers 5.22.1.
* Cached and uncached generation are bit-identical, including past
`sliding_window` (the eviction path).
* Left-padded batches decode bit-identically to the same rows generated
unpadded.
* The patched implementation is bit-identical to an independently written
reference patch across prefill logits, decode, padded and uncached paths.
## Files
```
modeling_ricursive.py patched drop-in replacement
PATCH.diff unified diff against the upstream file
```
|