license: other
license_name: mrl
license_link: https://mistral.ai/licenses/MRL-0.1.md
base_model: TheDrummer/Behemoth-R1-123B-v2
base_model_relation: quantized
tags:
- qtip
- 2-bit
- quantized
- rocm
- mistral
pipeline_tag: text-generation
Behemoth-R1-123B-v2 QTIP 2-bit
A 2-bit QTIP quantization of TheDrummer/Behemoth-R1-123B-v2, which is a fine-tune of Mistral-Large-Instruct-2411 (88 layers, hidden 12288, GQA 96/8). It is about 32 GB on disk and runs a 123B model on two 32 GB AMD Radeon AI PRO R9700 cards.
The quantizer and the inference runtime are in c000507/Behemoth-qtip: a
C++/HIP reimplementation of the QTIP pipeline for ROCm (gfx1201), plus behemoth_server, an OpenAI-compatible server
with QTIP trellis-decode kernels and two-GPU tensor parallelism.
Format
| q/o, gate/up/down projections | QTIP 2-bit HYB code: bitshift trellis L=16, K=2, V=2, quantlut_sym with a 2^9-entry LUT, 16x16 tiles |
| k/v projections | FP8 e4m3 with per-channel scale, kept from the source FP8 model (kv_format: fp8) |
| Embeddings, LM head, norms | bf16 |
| Layout | QTIP-hfized llama layout (trellis, SU, SV, tlut, rcp, tp_rank per linear); quip_params in config.json |
| Incoherence processing | random sign vectors + Hadamard transforms on both sides |
| Rounding | block LDLQ (16-blocks) with tail-biting Viterbi, sigma_reg 1e-2, scale override 0.9 |
The weights are not loadable with stock transformers. Use behemoth_server from the Behemoth-qtip repo. The 2-bit
tensors follow the reference QTIP (Cornell-RelaxML/qtip) hfized layout, but the mixed FP8 k/v projections are specific
to Behemoth-qtip (see its docs/MIXED_FP8_RECONSTRUCTION.md).
Calibration
- 4.19M tokens: 512 sequences of 8192 tokens.
- 75% chat and roleplay conversations with chain-of-thought, in the model's chat template with the reasoning inlined. 25% general text: OpenWebText and SmolTalk.
- Hessians were collected layer by layer from the original FP8 model (FP8 weights used exactly, per-channel scale applied in fp32), one Hessian group at a time: q, o, gate+up, down.
Serving
Build runtime/ from the Behemoth-qtip repository (ROCm, see its README), then:
runtime/build-rocm/behemoth_server --model /path/to/Behemoth-R1-123B-v2-QTIP-2bit \
--devices 0,1 --transport zerocopy --code-layout lane --memory-fraction 0.9 \
--token-budget 2048 --prefill-chunk 2048 --mixed-prefill-tokens 512 --prefill-interval 24 \
--host 0.0.0.0 --port 1337 --threads 4 --profile false
Single-request decode measures about 44 ms/token at short context and about 52 ms/token at 16k on 2x R9700.
Chat template
Mistral [INST] format, the same as the base model (chat_template.jinja):
<s>[SYSTEM_PROMPT] ...[/SYSTEM_PROMPT][INST] ...[/INST] assistant reply</s>
License
This is a derivative of Mistral-Large-Instruct-2411 and is distributed under the Mistral AI Research License: research and non-commercial use only. See the base models for their terms.