Instructions to use amd/MiniMax-M3-EAGLE3.1-Quark with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use amd/MiniMax-M3-EAGLE3.1-Quark with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="amd/MiniMax-M3-EAGLE3.1-Quark")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, LlamaForCausalLMEagle3

tokenizer = AutoTokenizer.from_pretrained("amd/MiniMax-M3-EAGLE3.1-Quark")
model = LlamaForCausalLMEagle3.from_pretrained("amd/MiniMax-M3-EAGLE3.1-Quark", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use amd/MiniMax-M3-EAGLE3.1-Quark with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "amd/MiniMax-M3-EAGLE3.1-Quark"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/MiniMax-M3-EAGLE3.1-Quark",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/amd/MiniMax-M3-EAGLE3.1-Quark

SGLang

How to use amd/MiniMax-M3-EAGLE3.1-Quark with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "amd/MiniMax-M3-EAGLE3.1-Quark" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/MiniMax-M3-EAGLE3.1-Quark",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "amd/MiniMax-M3-EAGLE3.1-Quark" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/MiniMax-M3-EAGLE3.1-Quark",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use amd/MiniMax-M3-EAGLE3.1-Quark with Docker Model Runner:
```
docker model run hf.co/amd/MiniMax-M3-EAGLE3.1-Quark
```

MiniMax-M3-EAGLE3.1-Quark / README.md

larryli2

Model card: document native AITER throughput scaling across concurrency

cadf8c9 2 months ago

preview code

Raw History Blame Contribute Delete

5.57 kB

---
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
- speculative-decoding
- eagle3
- draft-model
- minimax-m3
- amd-quark
- vllm
base_model:
- amd/MiniMax-M3-MXFP4
---
# MiniMax-M3-EAGLE3.1

## Model Overview

* **Model Architecture:** `LlamaForCausalLMEagle3` (EAGLE3 speculative-decoding draft model)
  * **Input:** Text
  * **Output:** Text
* **Draft variant:** EAGLE3.1 (single decoder layer, full vocabulary, BF16)
* **Target model:** [amd/MiniMax-M3-MXFP4](https://huggingface.co/amd/MiniMax-M3-MXFP4)
* **Supported Hardware Microarchitecture:** AMD Instinct MI350X / MI355X
* **Inference Engine:** vLLM (ROCm)
* **Trained by:** the AMD Quark team

**MiniMax-M3-EAGLE3.1** is an EAGLE3 draft model for accelerating inference of
[amd/MiniMax-M3-MXFP4](https://huggingface.co/amd/MiniMax-M3-MXFP4) with speculative
decoding. It was trained **from scratch (cold-start)** entirely on AMD Instinct
MI350X GPUs with a vLLM-centric pipeline (on-policy data generated by the target,
hidden-state extraction from the target, and in-loop serve-evaluation). Speculative
decoding is **lossless** — every draft token is verified by the target, so the
target's output distribution is preserved.

## Intended Use

This model is intended to be used as an EAGLE3 draft model for speculative decoding
with [amd/MiniMax-M3-MXFP4](https://huggingface.co/amd/MiniMax-M3-MXFP4) as the
target model. It reuses the target's tokenizer, so no tokenizer files are shipped
with the draft.

## Acceptance Length

Evaluated on the official **SPEED-Bench** harness (`specdec_bench`) at TP=8 on
vLLM ROCm v0.27.1 with native AITER MXFP4 MoE kernels, using
`num_speculative_tokens=3` and temperature 0. Qualitative results are the mean of
two complete runs; each fixed-context Throughput split is one complete run.
Acceptance length (AL) is the mean number of tokens emitted per target-model
verification step (AL = 1 means no speedup); higher is better.

### Acceptance length by domain (SPEED-Bench qualitative)

| Domain        | Acceptance length (AL) |
| ------------- | ---------------------- |
| Coding        | 3.33                   |
| Math          | 3.17                   |
| RAG           | 3.14                   |
| Multilingual  | 3.07                   |
| Reasoning     | 2.91                   |
| Summarization | 2.88                   |
| STEM          | 2.84                   |
| Humanities    | 2.70                   |
| QA            | 2.55                   |
| Writing       | 2.33                   |
| Roleplay      | 2.02                   |
| **Overall**   | **2.81**               |

### Acceptance length by context length

| Context length | Acceptance length (AL) |
| -------------- | ---------------------- |
| 1K             | 2.71                   |
| 8K             | 2.71                   |
| 16K            | 2.72                   |
| 32K            | 2.71                   |

Acceptance length stays essentially flat at 2.71–2.72 from 1K to 32K context.

## Native AITER Serving Results

On SPEED-Bench Qualitative at concurrency 48, EAGLE3 improves output throughput
and end-to-end latency over a matched non-speculative baseline:

| Metric                | EAGLE3 average | Non-speculative | Change         |
| --------------------- | -------------- | --------------- | -------------- |
| Output throughput/GPU | 415.2 tok/s    | 281.7 tok/s     | 1.47×          |
| Median E2E latency    | 9.68 s         | 13.15 s         | 1.36× faster   |
| Median TTFT           | 154 ms         | 130 ms          | +18.5% latency |

The EAGLE3 values are means of two complete runs. Both configurations use vLLM
ROCm v0.27.1, native AITER MXFP4, TP=8, temperature 0, a 12,288-token maximum
model length, and a 2,048-token output cap. The TTFT result shows the first-token
latency trade-off.

![Native AITER per-GPU output throughput across concurrency on SPEED-Bench Qualitative](assets/eagle3-throughput-concurrency.png)

Across concurrency 1–128, EAGLE3 improves per-GPU output throughput at every
tested point, scaling from 29 to 652 tok/s/GPU versus 16 to 465 tok/s/GPU for
the non-speculative baseline. The relative gain narrows from 1.77× at
concurrency 1 to 1.40× at concurrency 128 as the baseline approaches saturation,
while the absolute throughput advantage grows from approximately 13 to
187 tok/s/GPU. Figure labels are rounded; speedup ratios use the underlying
unrounded benchmark values.

## Serving with vLLM

Serve the [amd/MiniMax-M3-MXFP4](https://huggingface.co/amd/MiniMax-M3-MXFP4) target
with this draft as the EAGLE3 speculative model at TP=8. See the
[target model card](https://huggingface.co/amd/MiniMax-M3-MXFP4) for the
ROCm/vLLM runtime image and setup.

```bash
export VLLM_ROCM_USE_AITER=1
vllm serve amd/MiniMax-M3-MXFP4 --trust-remote-code --tensor-parallel-size 8 \
    --block-size 128 --attention-backend TRITON_ATTN --moe-backend aiter \
    --speculative-config '{"method":"eagle3","model":"amd/MiniMax-M3-EAGLE3.1","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN"}'
```

## Citation and Acknowledgements

Trained by the AMD Quark team as the EAGLE3 draft for
[amd/MiniMax-M3-MXFP4](https://huggingface.co/amd/MiniMax-M3-MXFP4). Please validate
quality and acceptance length in your own serving stack.

## License

This draft targets MiniMax-M3; see the bundled `MiniMax M3 LICENSE.txt` (MiniMax Community License) for the terms that apply to the target model and its derivatives.

Modifications Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.

	---
	license: mit
	library_name: transformers
	pipeline_tag: text-generation
	tags:
	- speculative-decoding
	- eagle3
	- draft-model
	- minimax-m3
	- amd-quark
	- vllm
	base_model:
	- amd/MiniMax-M3-MXFP4
	---

	# MiniMax-M3-EAGLE3.1

	## Model Overview

	* Model Architecture: `LlamaForCausalLMEagle3` (EAGLE3 speculative-decoding draft model)
	* Input: Text
	* Output: Text
	* Draft variant: EAGLE3.1 (single decoder layer, full vocabulary, BF16)
	* Target model: [amd/MiniMax-M3-MXFP4](https://huggingface.co/amd/MiniMax-M3-MXFP4)
	* Supported Hardware Microarchitecture: AMD Instinct MI350X / MI355X
	* Inference Engine: vLLM (ROCm)
	* Trained by: the AMD Quark team

	MiniMax-M3-EAGLE3.1 is an EAGLE3 draft model for accelerating inference of
	[amd/MiniMax-M3-MXFP4](https://huggingface.co/amd/MiniMax-M3-MXFP4) with speculative
	decoding. It was trained from scratch (cold-start) entirely on AMD Instinct
	MI350X GPUs with a vLLM-centric pipeline (on-policy data generated by the target,
	hidden-state extraction from the target, and in-loop serve-evaluation). Speculative
	decoding is lossless — every draft token is verified by the target, so the
	target's output distribution is preserved.

	## Intended Use

	This model is intended to be used as an EAGLE3 draft model for speculative decoding
	with [amd/MiniMax-M3-MXFP4](https://huggingface.co/amd/MiniMax-M3-MXFP4) as the
	target model. It reuses the target's tokenizer, so no tokenizer files are shipped
	with the draft.

	## Acceptance Length

	Evaluated on the official SPEED-Bench harness (`specdec_bench`) at TP=8 on
	vLLM ROCm v0.27.1 with native AITER MXFP4 MoE kernels, using
	`num_speculative_tokens=3` and temperature 0. Qualitative results are the mean of
	two complete runs; each fixed-context Throughput split is one complete run.
	Acceptance length (AL) is the mean number of tokens emitted per target-model
	verification step (AL = 1 means no speedup); higher is better.

	### Acceptance length by domain (SPEED-Bench qualitative)

	\| Domain \| Acceptance length (AL) \|
	\| ------------- \| ---------------------- \|
	\| Coding \| 3.33 \|
	\| Math \| 3.17 \|
	\| RAG \| 3.14 \|
	\| Multilingual \| 3.07 \|
	\| Reasoning \| 2.91 \|
	\| Summarization \| 2.88 \|
	\| STEM \| 2.84 \|
	\| Humanities \| 2.70 \|
	\| QA \| 2.55 \|
	\| Writing \| 2.33 \|
	\| Roleplay \| 2.02 \|
	\| Overall \| 2.81 \|

	### Acceptance length by context length

	\| Context length \| Acceptance length (AL) \|
	\| -------------- \| ---------------------- \|
	\| 1K \| 2.71 \|
	\| 8K \| 2.71 \|
	\| 16K \| 2.72 \|
	\| 32K \| 2.71 \|

	Acceptance length stays essentially flat at 2.71–2.72 from 1K to 32K context.

	## Native AITER Serving Results

	On SPEED-Bench Qualitative at concurrency 48, EAGLE3 improves output throughput
	and end-to-end latency over a matched non-speculative baseline:

	\| Metric \| EAGLE3 average \| Non-speculative \| Change \|
	\| --------------------- \| -------------- \| --------------- \| -------------- \|
	\| Output throughput/GPU \| 415.2 tok/s \| 281.7 tok/s \| 1.47× \|
	\| Median E2E latency \| 9.68 s \| 13.15 s \| 1.36× faster \|
	\| Median TTFT \| 154 ms \| 130 ms \| +18.5% latency \|

	The EAGLE3 values are means of two complete runs. Both configurations use vLLM
	ROCm v0.27.1, native AITER MXFP4, TP=8, temperature 0, a 12,288-token maximum
	model length, and a 2,048-token output cap. The TTFT result shows the first-token
	latency trade-off.

	![Native AITER per-GPU output throughput across concurrency on SPEED-Bench Qualitative](assets/eagle3-throughput-concurrency.png)

	Across concurrency 1–128, EAGLE3 improves per-GPU output throughput at every
	tested point, scaling from 29 to 652 tok/s/GPU versus 16 to 465 tok/s/GPU for
	the non-speculative baseline. The relative gain narrows from 1.77× at
	concurrency 1 to 1.40× at concurrency 128 as the baseline approaches saturation,
	while the absolute throughput advantage grows from approximately 13 to
	187 tok/s/GPU. Figure labels are rounded; speedup ratios use the underlying
	unrounded benchmark values.

	## Serving with vLLM

	Serve the [amd/MiniMax-M3-MXFP4](https://huggingface.co/amd/MiniMax-M3-MXFP4) target
	with this draft as the EAGLE3 speculative model at TP=8. See the
	[target model card](https://huggingface.co/amd/MiniMax-M3-MXFP4) for the
	ROCm/vLLM runtime image and setup.

	```bash
	export VLLM_ROCM_USE_AITER=1
	vllm serve amd/MiniMax-M3-MXFP4 --trust-remote-code --tensor-parallel-size 8 \
	--block-size 128 --attention-backend TRITON_ATTN --moe-backend aiter \
	--speculative-config '{"method":"eagle3","model":"amd/MiniMax-M3-EAGLE3.1","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN"}'
	```

	## Citation and Acknowledgements

	Trained by the AMD Quark team as the EAGLE3 draft for
	[amd/MiniMax-M3-MXFP4](https://huggingface.co/amd/MiniMax-M3-MXFP4). Please validate
	quality and acceptance length in your own serving stack.

	## License

	This draft targets MiniMax-M3; see the bundled `MiniMax M3 LICENSE.txt` (MiniMax Community License) for the terms that apply to the target model and its derivatives.

	Modifications Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.