Transformers
mamba2
vertical-chunking

The fused chunked path exposed by this repo is deprecated in favor of the upstream split chunked path introduced in transformers#46084.

chunkable-mamba2

Custom Mamba2 model and configuration classes for πŸ€— Transformers that add support for vertically chunked inference, which processes input sequences in fixed-size vertical chunks through all model layers with constant memory usage, regardless of sequence length.

What this repository provides

  • ChunkableMamba2Config: extends Mamba2Config with a use_mem_eff_path option for the memory-efficient CUDA kernel path.
  • ChunkableMamba2Model: extends Mamba2Model with a chunkable mixer and cache that correctly propagate the recurrent states across vertical chunks.
  • chunkable_mamba_split_conv1d_scan_combined: modified mamba_split_conv1d_scan_combined kernel wrapper that passes cache parameters through the SSD scan so that conv and SSM states are properly initialized and exported during chunked inference.

Usage

This repository is designed to be referenced directly from Hugging Face model configs via auto_map, so that models can be loaded with trust_remote_code=True without any local installation:

"auto_map": {
    "AutoConfig": "dynatrace-oss/chunkable-mamba2--configuration_chunkable_mamba2.ChunkableMamba2Config",
    "AutoModel": "dynatrace-oss/chunkable-mamba2--modeling_chunkable_mamba2.ChunkableMamba2Model"
}

Models

This code was created for the following embedding models:

Requirements

pip install transformers==5.21.0 kernels==0.17.2 einops

License

Apache-2.0

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Collection including dynatrace-oss/chunkable-mamba2

Paper for dynatrace-oss/chunkable-mamba2