Mixtral-8x7B-Instruct-v0.1-w8a8-llmcompressor

Model Overview

• Model Architecture: MixtralForCausalLM
  • Input: Text
  • Output: Text
• Source Model: Mixtral-8x7B-Instruct-v0.1
• Supported Hardware: AMD EPYC (CPU inference)
• Preferred Operating System: Linux
• Inference Engine: vLLM v0.31.0+cpu
• Quantization Framework: LLM Compressor v0.12.0
• Quantization Method: 8-bit Weight, 8-bit Dynamic Activation Quantization (W8A8)
• Compatible Stack:
  • ZenDNN v6.1.0
  • ZenTorch v2.13.0.3
  • PyTorch v2.13.0+cpu
  • LLM Compressor v0.12.0
  • vLLM v0.31.0+cpu
• Published with: LLM Compressor v0.12.0

This is a quantized version of Mixtral-8x7B-Instruct-v0.1 created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from Mixtral-8x7B-Instruct-v0.1 using LLM Compressor via the Round-to-Nearest (RTN) algorithm. This reduces the model weights from 87.0 GiB to 43.8 GiB on disk (~50% reduction).

• Method: 8-bit Weight, 8-bit Dynamic Activation Quantization (W8A8)
• Config: compressed-tensors, num_bits=8, type=int, symmetric=true
• Weights: INT8, symmetric, per-channel (static)
• Activations: INT8, symmetric, per-token (dynamic)
• Quantized: all 8 routed experts in each of the 32 layers (block_sparse_moe.experts.*.{w1,w2,w3}) and self_attn.{q,k,v,o}_proj
• Kept in BF16: the MoE router (block_sparse_moe.gate), lm_head, embed_tokens, and the layer norms

The recipe needs only two ignore entries. lm_head is standard, and the router is skipped because it is a tiny Linear whose logits decide expert assignment, where an 8-bit rounding error can flip the top-2 selection and change which experts run. Unlike Gemma-4, Mixtral stores its experts as real nn.Linear modules rather than packed 3D parameters, so targets=["Linear"] reaches them directly. Those experts are the bulk of the weights and the whole reason to quantize an MoE, which is why the footprint lands close to the full 50% an INT8 pass should give.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
output_dir = "./Mixtral-8x7B-Instruct-v0.1-w8a8-llmcompressor"

# Step 1: Load the BF16 model and tokenizer.
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="cpu",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Step 2: Define the W8A8 recipe. The routed experts and attention are all
# quantized; only lm_head and the MoE router are skipped.
recipe = QuantizationModifier(
    scheme="W8A8",
    targets=["Linear"],
    ignore=[
        "lm_head",
        r"re:.*lm_head",
        # MoE router / gate - the single most important layer to skip.
        r"re:.*block_sparse_moe\.gate$",
    ],
)

# Step 3: One-shot quantize and save in compressed-tensors format. W8A8 here is
# data-free (RTN), so no calibration dataset is needed.
oneshot(
    model=model,
    recipe=recipe,
    tokenizer=tokenizer,
    output_dir=output_dir,
    trust_remote_code_model=True,
)

# Smoke test
inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Quick Start

Use with vLLM

from vllm import LLM, SamplingParams

model = LLM(
    model="amd/Mixtral-8x7B-Instruct-v0.1-w8a8-llmcompressor",
    dtype="bfloat16",
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

torch==2.13.0+cpu
zentorch==2.13.0.3
vllm==0.31.0+cpu
llmcompressor==0.12.0

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.

Benchmark	BF16 Baseline	W8A8 (this model)	Recovery
GSM8K (5-shot)	0.6619	0.6520	98.50%

Evaluation Command

lm_eval \
    --model vllm \
    --model_args pretrained=amd/Mixtral-8x7B-Instruct-v0.1-w8a8-llmcompressor,dtype=bfloat16 \
    --tasks gsm8k \
    --batch_size auto \
    --trust_remote_code \
    --num_fewshot 5 \
    --apply_chat_template \
    --log_samples \
    --gen_kwargs "max_gen_toks=2048" \
    --output_path .

Limitations

• Version Lock: This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.13.0.3 / PyTorch v2.13.0+cpu. It may not load correctly on other versions.
• CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.

Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.