---
license: apache-2.0
library_name: transformers
tags:
  - ricursive
  - ripple
  - transformers
  - patch
---

# Ripple-65B-Preview — Transformers 5.x compatibility patch

Drop-in replacement for `modeling_ricursive.py` from
[`ricursiveAI/Ripple-65B-Preview`](https://huggingface.co/ricursiveAI/Ripple-65B-Preview).

The upstream custom code was written against the Transformers 4.x cache API.
Under Transformers 5.x, `generate()` fails on the very first prefill, and even
after that single line is fixed, padded batches decode incorrectly and
generations longer than `sliding_window` crash.

This repository distributes the fix only. It contains **no weights**; use it
with the original checkpoint.

## Apply

Copy the patched file over the upstream one:

```bash
cp modeling_ricursive.py /path/to/Ripple-65B-Preview/modeling_ricursive.py
```

Or apply the unified diff from the root of a checkout of the model repository:

```bash
git apply PATCH.diff
```

Then load as usual:

```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "ricursiveAI/Ripple-65B-Preview",
    trust_remote_code=True,
)
```

## What is broken upstream

1. **Spurious axis on prefill.** `prepare_inputs_for_generation` does
   `input_ids[:, cache_position]` whenever `input_ids.shape[1] != 1`.
   Transformers 5 pre-creates an empty `DynamicCache` and passes
   `cache_position=None` for the prefill call, so this becomes
   `input_ids[:, None]`: `(batch, seq)` turns into `(batch, 1, seq)`. The
   embedding lookup returns 4-D hidden states and the first decoder layer dies
   on `bsz, q_len, _ = hidden_states.shape`.

2. **Every decode step fully masked.** `_update_causal_mask` seeds the mask with
   `finfo.min` and only writes the visible diagonal via `triu` when
   `sequence_length != 1`. At decode `sequence_length == 1`, so the whole row
   stays masked and the model emits a constant token. The function also writes
   `1.0` (not `0.0`) for visible keys, which biases every score, and calls
   `past_key_values.get_max_cache_shape()`, which Transformers 5 removed.

3. **`position_ids` ignore padding.** They are overwritten with
   `cache_position` for every row, which is only correct when nothing is padded.
   With left padding, a short row's real tokens sit at higher indices, so a
   shared position vector shifts its RoPE phases and the row decodes to garbage.

4. **Sliding window uses the wrong absolute positions.**
   `_apply_sliding_window_mask` builds `arange(kv_len)` and assumes the keys sit
   at positions `0..kv_len-1`. Once the hybrid cache evicts from the left they
   sit at `[cp-kv_len+1 .. cp]`, so the mask excludes progressively more valid
   keys (e.g. window 16 at position 32 leaves 0 of 16 keys visible).

5. **Mask shape crash past the window.** Transformers 5 uses a hybrid cache:
   sliding layers hold at most `sliding_window` keys while full-attention layers
   keep growing. The model builds one mask per forward, so the shape no longer
   matches the sliding layer's key tensor
   (`RuntimeError: expanded size (1024) must match existing size (1025)`).

6. **`position_ids` left on CPU.** Transformers derives them from the caller's
   tensors before the generation hook runs, so a CPU prompt yields CPU positions
   even after the inputs are moved, and RoPE fails with `mat2 is on cpu`.

7. **Missing `GenerationMixin`.** Transformers 5 no longer injects it into
   remote-code classes, so `generate` can be missing entirely and
   `generation_config` is never built.

## What the patch changes

* `prepare_inputs_for_generation` rebuilds a missing `cache_position`, slices
  only when the cache is warm, derives per-row `position_ids` from the padding
  mask, and moves `input_ids` / `attention_mask` / `inputs_embeds` /
  `position_ids` onto the model device.
* `_update_causal_mask` sizes the key axis from `cache_position`, emits `0.0`
  for visible keys and `finfo.min` elsewhere, and never returns an all-masked
  single-token decode row.
* `_apply_sliding_window_mask` trims the shared mask to the keys the layer
  holds, anchors key positions at the first key the cache still holds, and is
  skipped when the window cannot bind.
* `RicursiveForCausalLM` inherits `GenerationMixin` explicitly.

The patch is backward compatible with Transformers 4.x.

## Validation

* `generate()` works under Transformers 5.22.1.
* Cached and uncached generation are bit-identical, including past
  `sliding_window` (the eviction path).
* Left-padded batches decode bit-identically to the same rows generated
  unpadded.
* The patched implementation is bit-identical to an independently written
  reference patch across prefill logits, decode, padded and uncached paths.

## Files

```
modeling_ricursive.py   patched drop-in replacement
PATCH.diff              unified diff against the upstream file
```