README.md

metadata

license: apache-2.0
library_name: transformers
tags:
  - ricursive
  - ripple
  - transformers
  - patch

Ripple-65B-Preview — Transformers 5.x compatibility patch

Drop-in replacement for modeling_ricursive.py from ricursiveAI/Ripple-65B-Preview.

The upstream custom code was written against the Transformers 4.x cache API. Under Transformers 5.x, generate() fails on the very first prefill, and even after that single line is fixed, padded batches decode incorrectly and generations longer than sliding_window crash.

This repository distributes the fix only. It contains no weights; use it with the original checkpoint.

Apply

Copy the patched file over the upstream one:

cp modeling_ricursive.py /path/to/Ripple-65B-Preview/modeling_ricursive.py

Or apply the unified diff from the root of a checkout of the model repository:

git apply PATCH.diff

Then load as usual:

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "ricursiveAI/Ripple-65B-Preview",
    trust_remote_code=True,
)

What is broken upstream

1. Spurious axis on prefill. prepare_inputs_for_generation does input_ids[:, cache_position] whenever input_ids.shape[1] != 1. Transformers 5 pre-creates an empty DynamicCache and passes cache_position=None for the prefill call, so this becomes input_ids[:, None]: (batch, seq) turns into (batch, 1, seq). The embedding lookup returns 4-D hidden states and the first decoder layer dies on bsz, q_len, _ = hidden_states.shape.

2. Every decode step fully masked. _update_causal_mask seeds the mask with finfo.min and only writes the visible diagonal via triu when sequence_length != 1. At decode sequence_length == 1, so the whole row stays masked and the model emits a constant token. The function also writes 1.0 (not 0.0) for visible keys, which biases every score, and calls past_key_values.get_max_cache_shape(), which Transformers 5 removed.

3. position_ids ignore padding. They are overwritten with cache_position for every row, which is only correct when nothing is padded. With left padding, a short row's real tokens sit at higher indices, so a shared position vector shifts its RoPE phases and the row decodes to garbage.

4. Sliding window uses the wrong absolute positions. _apply_sliding_window_mask builds arange(kv_len) and assumes the keys sit at positions 0..kv_len-1. Once the hybrid cache evicts from the left they sit at [cp-kv_len+1 .. cp], so the mask excludes progressively more valid keys (e.g. window 16 at position 32 leaves 0 of 16 keys visible).

5. Mask shape crash past the window. Transformers 5 uses a hybrid cache: sliding layers hold at most sliding_window keys while full-attention layers keep growing. The model builds one mask per forward, so the shape no longer matches the sliding layer's key tensor (RuntimeError: expanded size (1024) must match existing size (1025)).

6. position_ids left on CPU. Transformers derives them from the caller's tensors before the generation hook runs, so a CPU prompt yields CPU positions even after the inputs are moved, and RoPE fails with mat2 is on cpu.

7. Missing GenerationMixin. Transformers 5 no longer injects it into remote-code classes, so generate can be missing entirely and generation_config is never built.

What the patch changes

• prepare_inputs_for_generation rebuilds a missing cache_position, slices only when the cache is warm, derives per-row position_ids from the padding mask, and moves input_ids / attention_mask / inputs_embeds / position_ids onto the model device.
• _update_causal_mask sizes the key axis from cache_position, emits 0.0 for visible keys and finfo.min elsewhere, and never returns an all-masked single-token decode row.
• _apply_sliding_window_mask trims the shared mask to the keys the layer holds, anchors key positions at the first key the cache still holds, and is skipped when the window cannot bind.
• RicursiveForCausalLM inherits GenerationMixin explicitly.

The patch is backward compatible with Transformers 4.x.

Validation

• generate() works under Transformers 5.22.1.
• Cached and uncached generation are bit-identical, including past sliding_window (the eviction path).
• Left-padded batches decode bit-identically to the same rows generated unpadded.
• The patched implementation is bit-identical to an independently written reference patch across prefill logits, decode, padded and uncached paths.

Files

modeling_ricursive.py   patched drop-in replacement
PATCH.diff              unified diff against the upstream file