---
base_model:
- BAAI/AREX-2
tags:
- text-generation-inference
- qwen3_5
- agent
- deep-research
- reasoning
- tool-use
- long-context
- self-improvement
- fp8
- compressed-tensors
- llm-compressor
- vllm
- safetensors
license: apache-2.0
language:
- en
pipeline_tag: image-text-to-text
library_name: transformers
---

# **AREX-2-FP8**

> **AREX-2-FP8** is an **FP8 (W8A8) dynamic-quantized** variant of [BAAI/AREX-2](https://huggingface.co/BAAI/AREX-2), a 27B long-horizon, self-improving agent model built on a Qwen3.8-compatible multimodal dense architecture. Compression was performed with [LLM Compressor](https://github.com/vllm-project/llm-compressor) using the **FP8_DYNAMIC** scheme, which quantizes **weights to FP8 per-channel** and **activations to FP8 per-token at runtime**, so **no calibration data is required**. The result reduces memory footprint and improves inference throughput on FP8-capable GPUs, with the checkpoint stored in the **compressed-tensors** format for direct loading in **vLLM**. For model architecture, training details, intended use, evaluation results, and the citation, refer to the original model card: [BAAI/AREX-2](https://huggingface.co/BAAI/AREX-2).

## Model Overview

| Attribute | Value |
|---|---|
| **Base model** | [BAAI/AREX-2](https://huggingface.co/BAAI/AREX-2) (derived from Qwen/Qwen3.8-27B) |
| **Quantized model** | `prithivMLmods/AREX-2-FP8` |
| **Parameters** | 27B |
| **Context length** | 262,144 tokens |
| **Modality** | Image-Text-to-Text |
| **Quantization scheme** | FP8_DYNAMIC (W8A8) |
| **Weights** | FP8, per-channel |
| **Activations** | FP8, per-token, dynamic |
| **Calibration data** | Not required |
| **Format** | compressed-tensors (safetensors) |
| **Tooling** | LLM Compressor |
| **License** | Apache 2.0 |

## Quantization Details

Only **Linear** layers are quantized. The following modules are **excluded** and remain in their original precision to preserve output quality: the **language-model head**, the **token embeddings**, the entire **vision tower**, and all **linear-attention** layers.

**recipe.yaml**

```yaml
default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: ['re:.*lm_head', 're:.*embed_tokens$', 're:.*visual.*', 're:.*model.visual.*',
        're:.*linear_attn.*']
      scheme: FP8_DYNAMIC
      bypass_divisibility_checks: false
      requires_calibration_data: false
```

| Setting | Value |
|---|---|
| **targets** | `Linear` |
| **ignore** | `lm_head`, `embed_tokens`, `visual`, `model.visual`, `linear_attn` |
| **scheme** | `FP8_DYNAMIC` |
| **bypass_divisibility_checks** | `false` |
| **requires_calibration_data** | `false` |

## Inference with vLLM

Install a recent vLLM release with Qwen3.8 support.

```bash
pip install -U vllm
```

**Serve**

```bash
vllm serve prithivMLmods/AREX-2-FP8 \
  --max-model-len 32768 \
  --trust-remote-code
```

Raise `--max-model-len` toward the native 262,144 tokens as GPU memory allows. Add `--tensor-parallel-size N` for multi-GPU setups.

**Query the OpenAI-compatible endpoint**

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="prithivMLmods/AREX-2-FP8",
    messages=[
        {
            "role": "user",
            "content": "Propose a solution and explain how you would improve it over several rounds.",
        }
    ],
    max_tokens=1024,
    seed=42,
)
print(response.choices[0].message.content)
```

**Offline inference**

```python
from vllm import LLM, SamplingParams

llm = LLM(
    model="prithivMLmods/AREX-2-FP8",
    max_model_len=32768,
    trust_remote_code=True,
)

params = SamplingParams(max_tokens=1024, seed=42)
messages = [{
    "role": "user",
    "content": "Propose a solution and explain how you would improve it over several rounds.",
}]

outputs = llm.chat(messages, params)
print(outputs[0].outputs[0].text)
```

## Reproducing the Quantization

```bash
pip install -U llmcompressor transformers accelerate torch
```

```python
import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor
from llmcompressor import oneshot

model_id = "BAAI/AREX-2"
save_dir = "AREX-2-FP8"

model = AutoModelForMultimodalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)

oneshot(model=model, recipe="recipe.yaml")

model.save_pretrained(save_dir, save_compressed=True)
processor.save_pretrained(save_dir)
```

## Notes

- **FP8 hardware support** is required for full speed benefits (NVIDIA Hopper, Ada Lovelace, or newer). Other GPUs may fall back to weight-only behavior with reduced gains.
- Quantization can introduce small deviations from the BF16 reference. Benchmark scores reported on the [BAAI/AREX-2](https://huggingface.co/BAAI/AREX-2) card apply to the original weights and were not re-measured for this checkpoint.
- Follow the license terms and notices of the original AREX-2 release and the Qwen base model.

## Acknowledgements

- **BAAI** for [AREX-2](https://huggingface.co/BAAI/AREX-2).
- **vLLM Project** for [LLM Compressor](https://github.com/vllm-project/llm-compressor).