Transformers 5.x generate() and attention-mask fixes for Ripple

ede5883 verified about 7 hours ago

4.88 kB

---
license: apache-2.0
library_name: transformers
tags:
  - ricursive
  - ripple
  - transformers
  - patch
---
# Ripple-65B-Preview — Transformers 5.x compatibility patch

Drop-in replacement for `modeling_ricursive.py` from
[`ricursiveAI/Ripple-65B-Preview`](https://huggingface.co/ricursiveAI/Ripple-65B-Preview).

The upstream custom code was written against the Transformers 4.x cache API.
Under Transformers 5.x, `generate()` fails on the very first prefill, and even
after that single line is fixed, padded batches decode incorrectly and
generations longer than `sliding_window` crash.

This repository distributes the fix only. It contains **no weights**; use it
with the original checkpoint.

## Apply

Copy the patched file over the upstream one:

```bash
cp modeling_ricursive.py /path/to/Ripple-65B-Preview/modeling_ricursive.py
```

Or apply the unified diff from the root of a checkout of the model repository:

```bash
git apply PATCH.diff
```

Then load as usual:

```python
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    "ricursiveAI/Ripple-65B-Preview",
    trust_remote_code=True,
)
```

## What is broken upstream

1. **Spurious axis on prefill.** `prepare_inputs_for_generation` does
   `input_ids[:, cache_position]` whenever `input_ids.shape[1] != 1`.
   Transformers 5 pre-creates an empty `DynamicCache` and passes
   `cache_position=None` for the prefill call, so this becomes
   `input_ids[:, None]`: `(batch, seq)` turns into `(batch, 1, seq)`. The
   embedding lookup returns 4-D hidden states and the first decoder layer dies
   on `bsz, q_len, _ = hidden_states.shape`.

2. **Every decode step fully masked.** `_update_causal_mask` seeds the mask with
   `finfo.min` and only writes the visible diagonal via `triu` when
   `sequence_length != 1`. At decode `sequence_length == 1`, so the whole row
   stays masked and the model emits a constant token. The function also writes
   `1.0` (not `0.0`) for visible keys, which biases every score, and calls
   `past_key_values.get_max_cache_shape()`, which Transformers 5 removed.

3. **`position_ids` ignore padding.** They are overwritten with
   `cache_position` for every row, which is only correct when nothing is padded.
   With left padding, a short row's real tokens sit at higher indices, so a
   shared position vector shifts its RoPE phases and the row decodes to garbage.
4. **Sliding window uses the wrong absolute positions.**
   `_apply_sliding_window_mask` builds `arange(kv_len)` and assumes the keys sit
   at positions `0..kv_len-1`. Once the hybrid cache evicts from the left they
   sit at `[cp-kv_len+1 .. cp]`, so the mask excludes progressively more valid
   keys (e.g. window 16 at position 32 leaves 0 of 16 keys visible).

5. **Mask shape crash past the window.** Transformers 5 uses a hybrid cache:
   sliding layers hold at most `sliding_window` keys while full-attention layers
   keep growing. The model builds one mask per forward, so the shape no longer
   matches the sliding layer's key tensor
   (`RuntimeError: expanded size (1024) must match existing size (1025)`).

6. **`position_ids` left on CPU.** Transformers derives them from the caller's
   tensors before the generation hook runs, so a CPU prompt yields CPU positions
   even after the inputs are moved, and RoPE fails with `mat2 is on cpu`.
7. **Missing `GenerationMixin`.** Transformers 5 no longer injects it into
   remote-code classes, so `generate` can be missing entirely and
   `generation_config` is never built.
## What the patch changes
* `prepare_inputs_for_generation` rebuilds a missing `cache_position`, slices
  only when the cache is warm, derives per-row `position_ids` from the padding
  mask, and moves `input_ids` / `attention_mask` / `inputs_embeds` /
  `position_ids` onto the model device.
* `_update_causal_mask` sizes the key axis from `cache_position`, emits `0.0`
  for visible keys and `finfo.min` elsewhere, and never returns an all-masked
  single-token decode row.
* `_apply_sliding_window_mask` trims the shared mask to the keys the layer
  holds, anchors key positions at the first key the cache still holds, and is
  skipped when the window cannot bind.
* `RicursiveForCausalLM` inherits `GenerationMixin` explicitly.
The patch is backward compatible with Transformers 4.x.
## Validation
* `generate()` works under Transformers 5.22.1.
* Cached and uncached generation are bit-identical, including past
  `sliding_window` (the eviction path).
* Left-padded batches decode bit-identically to the same rows generated
  unpadded.
* The patched implementation is bit-identical to an independently written
  reference patch across prefill logits, decode, padded and uncached paths.
## Files
```
modeling_ricursive.py   patched drop-in replacement
PATCH.diff              unified diff against the upstream file
```

	---
	license: apache-2.0
	library_name: transformers
	tags:
	- ricursive
	- ripple
	- transformers
	- patch
	---

	# Ripple-65B-Preview — Transformers 5.x compatibility patch

	Drop-in replacement for `modeling_ricursive.py` from
	[`ricursiveAI/Ripple-65B-Preview`](https://huggingface.co/ricursiveAI/Ripple-65B-Preview).

	The upstream custom code was written against the Transformers 4.x cache API.
	Under Transformers 5.x, `generate()` fails on the very first prefill, and even
	after that single line is fixed, padded batches decode incorrectly and
	generations longer than `sliding_window` crash.

	This repository distributes the fix only. It contains no weights; use it
	with the original checkpoint.

	## Apply

	Copy the patched file over the upstream one:

	```bash
	cp modeling_ricursive.py /path/to/Ripple-65B-Preview/modeling_ricursive.py
	```

	Or apply the unified diff from the root of a checkout of the model repository:

	```bash
	git apply PATCH.diff
	```

	Then load as usual:

	```python
	from transformers import AutoModelForCausalLM

	model = AutoModelForCausalLM.from_pretrained(
	"ricursiveAI/Ripple-65B-Preview",
	trust_remote_code=True,
	)
	```

	## What is broken upstream

	1. Spurious axis on prefill. `prepare_inputs_for_generation` does
	`input_ids[:, cache_position]` whenever `input_ids.shape[1] != 1`.
	Transformers 5 pre-creates an empty `DynamicCache` and passes
	`cache_position=None` for the prefill call, so this becomes
	`input_ids[:, None]`: `(batch, seq)` turns into `(batch, 1, seq)`. The
	embedding lookup returns 4-D hidden states and the first decoder layer dies
	on `bsz, q_len, _ = hidden_states.shape`.

	2. Every decode step fully masked. `_update_causal_mask` seeds the mask with
	`finfo.min` and only writes the visible diagonal via `triu` when
	`sequence_length != 1`. At decode `sequence_length == 1`, so the whole row
	stays masked and the model emits a constant token. The function also writes
	`1.0` (not `0.0`) for visible keys, which biases every score, and calls
	`past_key_values.get_max_cache_shape()`, which Transformers 5 removed.

	3. `position_ids` ignore padding. They are overwritten with
	`cache_position` for every row, which is only correct when nothing is padded.
	With left padding, a short row's real tokens sit at higher indices, so a
	shared position vector shifts its RoPE phases and the row decodes to garbage.

	4. Sliding window uses the wrong absolute positions.
	`_apply_sliding_window_mask` builds `arange(kv_len)` and assumes the keys sit
	at positions `0..kv_len-1`. Once the hybrid cache evicts from the left they
	sit at `[cp-kv_len+1 .. cp]`, so the mask excludes progressively more valid
	keys (e.g. window 16 at position 32 leaves 0 of 16 keys visible).

	5. Mask shape crash past the window. Transformers 5 uses a hybrid cache:
	sliding layers hold at most `sliding_window` keys while full-attention layers
	keep growing. The model builds one mask per forward, so the shape no longer
	matches the sliding layer's key tensor
	(`RuntimeError: expanded size (1024) must match existing size (1025)`).

	6. `position_ids` left on CPU. Transformers derives them from the caller's
	tensors before the generation hook runs, so a CPU prompt yields CPU positions
	even after the inputs are moved, and RoPE fails with `mat2 is on cpu`.

	7. Missing `GenerationMixin`. Transformers 5 no longer injects it into
	remote-code classes, so `generate` can be missing entirely and
	`generation_config` is never built.

	## What the patch changes

	* `prepare_inputs_for_generation` rebuilds a missing `cache_position`, slices
	only when the cache is warm, derives per-row `position_ids` from the padding
	mask, and moves `input_ids` / `attention_mask` / `inputs_embeds` /
	`position_ids` onto the model device.
	* `_update_causal_mask` sizes the key axis from `cache_position`, emits `0.0`
	for visible keys and `finfo.min` elsewhere, and never returns an all-masked
	single-token decode row.
	* `_apply_sliding_window_mask` trims the shared mask to the keys the layer
	holds, anchors key positions at the first key the cache still holds, and is
	skipped when the window cannot bind.
	* `RicursiveForCausalLM` inherits `GenerationMixin` explicitly.

	The patch is backward compatible with Transformers 4.x.

	## Validation

	* `generate()` works under Transformers 5.22.1.
	* Cached and uncached generation are bit-identical, including past
	`sliding_window` (the eviction path).
	* Left-padded batches decode bit-identically to the same rows generated
	unpadded.
	* The patched implementation is bit-identical to an independently written
	reference patch across prefill logits, decode, padded and uncached paths.

	## Files

	```
	modeling_ricursive.py patched drop-in replacement
	PATCH.diff unified diff against the upstream file
	```