File size: 725 Bytes
0ba3a54 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | <!-- SPDX-License-Identifier: Apache-2.0 -->
# oblivion_vllm
vLLM plugin that registers the `OblivionForCausalLM` architecture so vLLM can
**serve** the [Oblivion](https://huggingface.co/IcyApril/Oblivion) checkpoint.
This is runtime code only — install it and serve.
```bash
pip install .
VLLM_PLUGINS=oblivion_register vllm serve <oblivion-checkpoint> \
--mamba-cache-mode=align --trust-remote-code
# add --quantization fp8 to serve in FP8 (dynamic, fits one GPU)
```
The model is hybrid (its Gated-DeltaNet donor is a linear-attention layer), so
`--mamba-cache-mode=align` is required. Needs a vLLM build with MuseGlimmer
support; the plugin declares no dependencies of its own so it won't shadow it.
Apache-2.0.
|