File size: 5,223 Bytes
f682480 1e68b42 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 | ---
base_model:
- BAAI/AREX-2
tags:
- text-generation-inference
- qwen3_5
- agent
- deep-research
- reasoning
- tool-use
- long-context
- self-improvement
- fp8
- compressed-tensors
- llm-compressor
- vllm
- safetensors
license: apache-2.0
language:
- en
pipeline_tag: image-text-to-text
library_name: transformers
---
# **AREX-2-FP8**
> **AREX-2-FP8** is an **FP8 (W8A8) dynamic-quantized** variant of [BAAI/AREX-2](https://huggingface.co/BAAI/AREX-2), a 27B long-horizon, self-improving agent model built on a Qwen3.8-compatible multimodal dense architecture. Compression was performed with [LLM Compressor](https://github.com/vllm-project/llm-compressor) using the **FP8_DYNAMIC** scheme, which quantizes **weights to FP8 per-channel** and **activations to FP8 per-token at runtime**, so **no calibration data is required**. The result reduces memory footprint and improves inference throughput on FP8-capable GPUs, with the checkpoint stored in the **compressed-tensors** format for direct loading in **vLLM**. For model architecture, training details, intended use, evaluation results, and the citation, refer to the original model card: [BAAI/AREX-2](https://huggingface.co/BAAI/AREX-2).
## Model Overview
| Attribute | Value |
|---|---|
| **Base model** | [BAAI/AREX-2](https://huggingface.co/BAAI/AREX-2) (derived from Qwen/Qwen3.8-27B) |
| **Quantized model** | `prithivMLmods/AREX-2-FP8` |
| **Parameters** | 27B |
| **Context length** | 262,144 tokens |
| **Modality** | Image-Text-to-Text |
| **Quantization scheme** | FP8_DYNAMIC (W8A8) |
| **Weights** | FP8, per-channel |
| **Activations** | FP8, per-token, dynamic |
| **Calibration data** | Not required |
| **Format** | compressed-tensors (safetensors) |
| **Tooling** | LLM Compressor |
| **License** | Apache 2.0 |
## Quantization Details
Only **Linear** layers are quantized. The following modules are **excluded** and remain in their original precision to preserve output quality: the **language-model head**, the **token embeddings**, the entire **vision tower**, and all **linear-attention** layers.
**recipe.yaml**
```yaml
default_stage:
default_modifiers:
QuantizationModifier:
targets: [Linear]
ignore: ['re:.*lm_head', 're:.*embed_tokens$', 're:.*visual.*', 're:.*model.visual.*',
're:.*linear_attn.*']
scheme: FP8_DYNAMIC
bypass_divisibility_checks: false
requires_calibration_data: false
```
| Setting | Value |
|---|---|
| **targets** | `Linear` |
| **ignore** | `lm_head`, `embed_tokens`, `visual`, `model.visual`, `linear_attn` |
| **scheme** | `FP8_DYNAMIC` |
| **bypass_divisibility_checks** | `false` |
| **requires_calibration_data** | `false` |
## Inference with vLLM
Install a recent vLLM release with Qwen3.8 support.
```bash
pip install -U vllm
```
**Serve**
```bash
vllm serve prithivMLmods/AREX-2-FP8 \
--max-model-len 32768 \
--trust-remote-code
```
Raise `--max-model-len` toward the native 262,144 tokens as GPU memory allows. Add `--tensor-parallel-size N` for multi-GPU setups.
**Query the OpenAI-compatible endpoint**
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="prithivMLmods/AREX-2-FP8",
messages=[
{
"role": "user",
"content": "Propose a solution and explain how you would improve it over several rounds.",
}
],
max_tokens=1024,
seed=42,
)
print(response.choices[0].message.content)
```
**Offline inference**
```python
from vllm import LLM, SamplingParams
llm = LLM(
model="prithivMLmods/AREX-2-FP8",
max_model_len=32768,
trust_remote_code=True,
)
params = SamplingParams(max_tokens=1024, seed=42)
messages = [{
"role": "user",
"content": "Propose a solution and explain how you would improve it over several rounds.",
}]
outputs = llm.chat(messages, params)
print(outputs[0].outputs[0].text)
```
## Reproducing the Quantization
```bash
pip install -U llmcompressor transformers accelerate torch
```
```python
import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor
from llmcompressor import oneshot
model_id = "BAAI/AREX-2"
save_dir = "AREX-2-FP8"
model = AutoModelForMultimodalLM.from_pretrained(
model_id, dtype=torch.bfloat16, device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)
oneshot(model=model, recipe="recipe.yaml")
model.save_pretrained(save_dir, save_compressed=True)
processor.save_pretrained(save_dir)
```
## Notes
- **FP8 hardware support** is required for full speed benefits (NVIDIA Hopper, Ada Lovelace, or newer). Other GPUs may fall back to weight-only behavior with reduced gains.
- Quantization can introduce small deviations from the BF16 reference. Benchmark scores reported on the [BAAI/AREX-2](https://huggingface.co/BAAI/AREX-2) card apply to the original weights and were not re-measured for this checkpoint.
- Follow the license terms and notices of the original AREX-2 release and the Qwen base model.
## Acknowledgements
- **BAAI** for [AREX-2](https://huggingface.co/BAAI/AREX-2).
- **vLLM Project** for [LLM Compressor](https://github.com/vllm-project/llm-compressor). |