Model Overview

Model Architecture: Llama-3.3
- Input: Text
- Output: Text
Supported Hardware Microarchitecture: AMD MI350/MI355
ROCm: 7.0
PyTorch: 2.8.0
Transformers: 4.53.0
Operating System(s): Linux
Inference Engine: vLLM
Model Optimizer: AMD-Quark (V0.9)
- Weight quantization: OCP MXFP4, Static
- Activation quantization: OCP MXFP4, Dynamic
- KV cache quantization: OCP FP8, Static
Calibration Dataset: Pile

This model was built with Meta Llama by applying AMD-Quark for MXFP4 quantization.

Model Quantization

This model was obtained by quantizing Llama-3.3-70B-Instruct's weights and activations to MXFP4 and KV caches to FP8, using AutoSmoothQuant algorithm in AMD-Quark.

Quantization scripts:

cd Quark/examples/torch/language_modeling/llm_ptq/
python3 quantize_quark.py --model_dir meta-llama/Llama-3.3-70B-Instruct \
                          --quant_scheme w_mxfp4_a_mxfp4 \
                          --kv_cache_dtype fp8 \
                          --num_calib_data 128 \
                          --multi_gpu \
                          --quant_algo autosmoothquant 
                          --model_export hf_format \
                          --output_dir amd/Llama-3.3-70B-Instruct-WMXFP4-AMXFP4-KVFP8-Scale-UINT8-ASQ

Deployment

Use with vLLM

This model can be deployed efficiently using the vLLM backend.

Evaluation

The model was evaluated on MMLU, GSM8K_COT, ARC Challenge and IFEVAL. Evaluation was conducted using the framework lm-evaluation-harness and the vLLM engine.

Accuracy

Benchmark	Llama-3.3-70B-Instruct	Llama-3.3-70B-Instruct-MXFP4(this model)	Recovery
MMLU (5-shot)	83.29	80.99	97.24%
GSM8K_COT (8-shot, strict-match)	93.18	92.12	98.86%
ARC Challenge (0-shot)	94.25	93.05	98.73%
IFEVAL (0-shot, (inst_level_strict_acc+prompt_level_strict_acc)/2)	89.8	88.00	98.00%

Reproduction

The results were obtained using the following commands:

MMLU

lm_eval \
    --model vllm \
    --model_args pretrained=amd/Llama-3.3-70B-Instruct-WMXFP4-AMXFP4-KVFP8-Scale-UINT8-ASQ,gpu_memory_utilization=0.8,tensor_parallel_size=4,kv_cache_dtype='fp8' \
    --tasks mmlu_llama \
    --fewshot_as_multiturn \
    --apply_chat_template \
    --num_fewshot 5 \
    --batch_size auto

GSM8K_COT

lm_eval \
    --model vllm \
    --model_args pretrained=amd/Llama-3.3-70B-Instruct-WMXFP4-AMXFP4-KVFP8-Scale-UINT8-ASQ,gpu_memory_utilization=0.8,tensor_parallel_size=4,kv_cache_dtype='fp8' \
    --tasks gsm8k_llama \
    --fewshot_as_multiturn \
    --apply_chat_template \
    --num_fewshot 8 \
    --batch_size auto

ARC Challenge

lm_eval \
    --model vllm \
    --model_args pretrained=amd/Llama-3.3-70B-Instruct-WMXFP4-AMXFP4-KVFP8-Scale-UINT8-ASQ,gpu_memory_utilization=0.8,tensor_parallel_size=4,kv_cache_dtype='fp8' \
    --tasks arc_challenge_llama \
    --fewshot_as_multiturn \
    --apply_chat_template \
    --num_fewshot 0 \
    --batch_size auto

IFEVAL

lm_eval \
    --model vllm \
    --model_args pretrained=amd/Llama-3.3-70B-Instruct-WMXFP4-AMXFP4-KVFP8-Scale-UINT8-ASQ,gpu_memory_utilization=0.8,tensor_parallel_size=4,kv_cache_dtype='fp8' \
    --tasks ifeval \
    --fewshot_as_multiturn \
    --apply_chat_template \
    --num_fewshot 0 \
    --batch_size auto