---
license: other
license_name: mrl
license_link: https://mistral.ai/licenses/MRL-0.1.md
base_model: TheDrummer/Behemoth-R1-123B-v2
base_model_relation: quantized
tags:
- qtip
- 2-bit
- quantized
- rocm
- mistral
pipeline_tag: text-generation
---

# Behemoth-R1-123B-v2 QTIP 2-bit

A 2-bit [QTIP](https://arxiv.org/abs/2406.11235) quantization of
[TheDrummer/Behemoth-R1-123B-v2](https://huggingface.co/TheDrummer/Behemoth-R1-123B-v2), which is a fine-tune of
Mistral-Large-Instruct-2411 (88 layers, hidden 12288, GQA 96/8). It is about 32 GB on disk and runs a 123B model on two
32 GB AMD Radeon AI PRO R9700 cards.

The quantizer and the inference runtime are in [c000507/Behemoth-qtip](https://github.com/c000507/Behemoth-qtip): a
C++/HIP reimplementation of the QTIP pipeline for ROCm (gfx1201), plus `behemoth_server`, an OpenAI-compatible server
with QTIP trellis-decode kernels and two-GPU tensor parallelism.

## Format

| | |
|---|---|
| q/o, gate/up/down projections | QTIP 2-bit HYB code: bitshift trellis L=16, K=2, V=2, `quantlut_sym` with a 2^9-entry LUT, 16x16 tiles |
| k/v projections | FP8 e4m3 with per-channel scale, kept from the source FP8 model (`kv_format: fp8`) |
| Embeddings, LM head, norms | bf16 |
| Layout | QTIP-hfized llama layout (`trellis`, `SU`, `SV`, `tlut`, `rcp`, `tp_rank` per linear); `quip_params` in `config.json` |
| Incoherence processing | random sign vectors + Hadamard transforms on both sides |
| Rounding | block LDLQ (16-blocks) with tail-biting Viterbi, sigma_reg 1e-2, scale override 0.9 |

The weights are not loadable with stock `transformers`. Use `behemoth_server` from the Behemoth-qtip repo. The 2-bit
tensors follow the reference QTIP (Cornell-RelaxML/qtip) hfized layout, but the mixed FP8 k/v projections are specific
to Behemoth-qtip (see its `docs/MIXED_FP8_RECONSTRUCTION.md`).

## Calibration

- 4.19M tokens: 512 sequences of 8192 tokens.
- 75% chat and roleplay conversations with chain-of-thought, in the model's chat template with the reasoning inlined.
  25% general text: OpenWebText and SmolTalk.
- Hessians were collected layer by layer from the original FP8 model (FP8 weights used exactly, per-channel scale
  applied in fp32), one Hessian group at a time: q, o, gate+up, down.

## Serving

Build `runtime/` from the Behemoth-qtip repository (ROCm, see its README), then:

```sh
runtime/build-rocm/behemoth_server --model /path/to/Behemoth-R1-123B-v2-QTIP-2bit \
  --devices 0,1 --transport zerocopy --code-layout lane --memory-fraction 0.9 \
  --token-budget 2048 --prefill-chunk 2048 --mixed-prefill-tokens 512 --prefill-interval 24 \
  --host 0.0.0.0 --port 1337 --threads 4 --profile false
```

Single-request decode measures about 44 ms/token at short context and about 52 ms/token at 16k on 2x R9700.

## Chat template

Mistral `[INST]` format, the same as the base model (`chat_template.jinja`):

```
<s>[SYSTEM_PROMPT] ...[/SYSTEM_PROMPT][INST] ...[/INST] assistant reply</s>
```

## License

This is a derivative of Mistral-Large-Instruct-2411 and is distributed under the
[Mistral AI Research License](https://mistral.ai/licenses/MRL-0.1.md): research and non-commercial use only. See the
base models for their terms.