| --- |
| license: other |
| license_name: mrl |
| license_link: https://mistral.ai/licenses/MRL-0.1.md |
| base_model: TheDrummer/Behemoth-R1-123B-v2 |
| base_model_relation: quantized |
| tags: |
| - qtip |
| - 2-bit |
| - quantized |
| - rocm |
| - mistral |
| pipeline_tag: text-generation |
| --- |
| |
| # Behemoth-R1-123B-v2 QTIP 2-bit |
|
|
| A 2-bit [QTIP](https://arxiv.org/abs/2406.11235) quantization of |
| [TheDrummer/Behemoth-R1-123B-v2](https://huggingface.co/TheDrummer/Behemoth-R1-123B-v2), which is a fine-tune of |
| Mistral-Large-Instruct-2411 (88 layers, hidden 12288, GQA 96/8). It is about 32 GB on disk and runs a 123B model on two |
| 32 GB AMD Radeon AI PRO R9700 cards. |
|
|
| The quantizer and the inference runtime are in [c000507/Behemoth-qtip](https://github.com/c000507/Behemoth-qtip): a |
| C++/HIP reimplementation of the QTIP pipeline for ROCm (gfx1201), plus `behemoth_server`, an OpenAI-compatible server |
| with QTIP trellis-decode kernels and two-GPU tensor parallelism. |
|
|
| ## Format |
|
|
| | | | |
| |---|---| |
| | q/o, gate/up/down projections | QTIP 2-bit HYB code: bitshift trellis L=16, K=2, V=2, `quantlut_sym` with a 2^9-entry LUT, 16x16 tiles | |
| | k/v projections | FP8 e4m3 with per-channel scale, kept from the source FP8 model (`kv_format: fp8`) | |
| | Embeddings, LM head, norms | bf16 | |
| | Layout | QTIP-hfized llama layout (`trellis`, `SU`, `SV`, `tlut`, `rcp`, `tp_rank` per linear); `quip_params` in `config.json` | |
| | Incoherence processing | random sign vectors + Hadamard transforms on both sides | |
| | Rounding | block LDLQ (16-blocks) with tail-biting Viterbi, sigma_reg 1e-2, scale override 0.9 | |
| |
| The weights are not loadable with stock `transformers`. Use `behemoth_server` from the Behemoth-qtip repo. The 2-bit |
| tensors follow the reference QTIP (Cornell-RelaxML/qtip) hfized layout, but the mixed FP8 k/v projections are specific |
| to Behemoth-qtip (see its `docs/MIXED_FP8_RECONSTRUCTION.md`). |
|
|
| ## Calibration |
|
|
| - 4.19M tokens: 512 sequences of 8192 tokens. |
| - 75% chat and roleplay conversations with chain-of-thought, in the model's chat template with the reasoning inlined. |
| 25% general text: OpenWebText and SmolTalk. |
| - Hessians were collected layer by layer from the original FP8 model (FP8 weights used exactly, per-channel scale |
| applied in fp32), one Hessian group at a time: q, o, gate+up, down. |
|
|
| ## Serving |
|
|
| Build `runtime/` from the Behemoth-qtip repository (ROCm, see its README), then: |
|
|
| ```sh |
| runtime/build-rocm/behemoth_server --model /path/to/Behemoth-R1-123B-v2-QTIP-2bit \ |
| --devices 0,1 --transport zerocopy --code-layout lane --memory-fraction 0.9 \ |
| --token-budget 2048 --prefill-chunk 2048 --mixed-prefill-tokens 512 --prefill-interval 24 \ |
| --host 0.0.0.0 --port 1337 --threads 4 --profile false |
| ``` |
|
|
| Single-request decode measures about 44 ms/token at short context and about 52 ms/token at 16k on 2x R9700. |
|
|
| ## Chat template |
|
|
| Mistral `[INST]` format, the same as the base model (`chat_template.jinja`): |
|
|
| ``` |
| <s>[SYSTEM_PROMPT] ...[/SYSTEM_PROMPT][INST] ...[/INST] assistant reply</s> |
| ``` |
|
|
| ## License |
|
|
| This is a derivative of Mistral-Large-Instruct-2411 and is distributed under the |
| [Mistral AI Research License](https://mistral.ai/licenses/MRL-0.1.md): research and non-commercial use only. See the |
| base models for their terms. |
|
|
