Add model card

README.md

@@ -0,0 +1,75 @@
+---
+license: other
+license_name: mrl
+license_link: https://mistral.ai/licenses/MRL-0.1.md
+base_model: TheDrummer/Behemoth-R1-123B-v2
+base_model_relation: quantized
+tags:
+- qtip
+- 2-bit
+- quantized
+- rocm
+- mistral
+pipeline_tag: text-generation
+---
+
+# Behemoth-R1-123B-v2 QTIP 2-bit
+
+A 2-bit [QTIP](https://arxiv.org/abs/2406.11235) quantization of
+[TheDrummer/Behemoth-R1-123B-v2](https://huggingface.co/TheDrummer/Behemoth-R1-123B-v2), which is a fine-tune of
+Mistral-Large-Instruct-2411 (88 layers, hidden 12288, GQA 96/8). It is about 32 GB on disk and runs a 123B model on two
+32 GB AMD Radeon AI PRO R9700 cards.
+
+The quantizer and the inference runtime are in [c000507/Behemoth-qtip](https://github.com/c000507/Behemoth-qtip): a
+C++/HIP reimplementation of the QTIP pipeline for ROCm (gfx1201), plus `behemoth_server`, an OpenAI-compatible server
+with QTIP trellis-decode kernels and two-GPU tensor parallelism.
+
+## Format
+
+| | |
+|---|---|
+| q/o, gate/up/down projections | QTIP 2-bit HYB code: bitshift trellis L=16, K=2, V=2, `quantlut_sym` with a 2^9-entry LUT, 16x16 tiles |
+| k/v projections | FP8 e4m3 with per-channel scale, kept from the source FP8 model (`kv_format: fp8`) |
+| Embeddings, LM head, norms | bf16 |
+| Layout | QTIP-hfized llama layout (`trellis`, `SU`, `SV`, `tlut`, `rcp`, `tp_rank` per linear); `quip_params` in `config.json` |
+| Incoherence processing | random sign vectors + Hadamard transforms on both sides |
+| Rounding | block LDLQ (16-blocks) with tail-biting Viterbi, sigma_reg 1e-2, scale override 0.9 |
+
+The weights are not loadable with stock `transformers`. Use `behemoth_server` from the Behemoth-qtip repo. The 2-bit
+tensors follow the reference QTIP (Cornell-RelaxML/qtip) hfized layout, but the mixed FP8 k/v projections are specific
+to Behemoth-qtip (see its `docs/MIXED_FP8_RECONSTRUCTION.md`).
+
+## Calibration
+
+- 4.19M tokens: 512 sequences of 8192 tokens.
+- 75% chat and roleplay conversations with chain-of-thought, in the model's chat template with the reasoning inlined.
+  25% general text: OpenWebText and SmolTalk.
+- Hessians were collected layer by layer from the original FP8 model (FP8 weights used exactly, per-channel scale
+  applied in fp32), one Hessian group at a time: q, o, gate+up, down.
+
+## Serving
+
+Build `runtime/` from the Behemoth-qtip repository (ROCm, see its README), then:
+
+```sh
+runtime/build-rocm/behemoth_server --model /path/to/Behemoth-R1-123B-v2-QTIP-2bit \
+  --devices 0,1 --transport zerocopy --code-layout lane --memory-fraction 0.9 \
+  --token-budget 2048 --prefill-chunk 2048 --mixed-prefill-tokens 512 --prefill-interval 24 \
+  --host 0.0.0.0 --port 1337 --threads 4 --profile false
+```
+
+Single-request decode measures about 44 ms/token at short context and about 52 ms/token at 16k on 2x R9700.
+
+## Chat template
+
+Mistral `[INST]` format, the same as the base model (`chat_template.jinja`):
+
+```
+<s>[SYSTEM_PROMPT] ...[/SYSTEM_PROMPT][INST] ...[/INST] assistant reply</s>
+```
+
+## License
+
+This is a derivative of Mistral-Large-Instruct-2411 and is distributed under the
+[Mistral AI Research License](https://mistral.ai/licenses/MRL-0.1.md): research and non-commercial use only. See the
+base models for their terms.