| # OXR vLLM plugin |
|
|
| Registers the OXR architecture through `vllm.general_plugins`. Requires |
| Python 3.12+, vLLM 0.25.1 and Transformers 5.13.1. It does not modify the |
| vLLM installation. The supported Docker runtime uses PyTorch 2.11.0+cu130. |
|
|
| Install from the model repository root in a Linux CUDA environment: |
|
|
| ```bash |
| python -m pip install ./plugins/vllm |
| ``` |
|
|
| Start with the model files and weights in the repository root: |
|
|
| ```bash |
| VLLM_PLUGINS=oxr vllm serve . \ |
| --trust-remote-code \ |
| --served-model-name oxr \ |
| --chat-template ./chat_template.jinja \ |
| --dtype bfloat16 \ |
| --attention-config '{"backend":"TRITON_ATTN"}' \ |
| --max-model-len 16384 |
| ``` |
|
|
| The LA model in `layout_analysis/` is served separately by the OXR layout |
| analysis service, not by this plugin. |
|
|
| To roll back to vLLM 0.19.1, use the previous model plugin together with the |
| matching previous OXR launcher and Docker image. Do not install this plugin |
| into the old runtime. |
|
|
