| --- |
| license: apache-2.0 |
| base_model: thinkingmachines/Inkling-2-Small-MXFP8 |
| base_model_relation: quantized |
| library_name: vllm |
| pipeline_tag: image-text-to-text |
| tags: |
| - fp8 |
| - rocm |
| - mi300x |
| - inkling2 |
| - moe |
| - dspark |
| - vision |
| --- |
| |
| # Inkling 2 Small FP8 for AMD MI300X |
|
|
| A row-scaled E4M3FNUZ conversion of |
| [Thinking Machines' Inkling 2 Small MXFP8](https://huggingface.co/thinkingmachines/Inkling-2-Small-MXFP8), |
| revision `3febc62259ee92ea36e2fb6e8303371c4eb3922c`. |
| The original model was developed by Thinking Machines Lab. This repository |
| contains a precision conversion, not a new trained model. The upstream model |
| card is preserved in [UPSTREAM_README.md](UPSTREAM_README.md). |
|
|
| ## Precision and files |
|
|
| The 96 expert projection tensors use native AMD **E4M3FNUZ FP8**, with one FP32 |
| scale per expert output channel. Activations use dynamic per-token E4M3FNUZ |
| quantization. Weights are dequantized from the original MXFP8 block scales and |
| requantized with `scale = max(abs(row)) / 240`. Attention, dense projections, |
| vision, and the DSpark draft retain the upstream precision. This is mixed |
| precision inference. |
|
|
| All 48 target weight shards, tokenizer/configuration files, vision weights, |
| and `mtp.safetensors` are included. The checkpoint is approximately 605.14 GB. |
| [conversion_manifest.json](conversion_manifest.json) records the source revision, |
| per-shard tensor sizes, and serialization verification fingerprints. |
| The original checkpoint is unchanged. |
|
|
| `hf_quant_config.json` marks converted experts with `compute_dtype: fp8`; |
| `config.json` selects `expert_fp8_scale_mode: row`. Expert scale tensors retain |
| the `_weight_scale` suffix and have shape `[384, output_channels]`, dtype FP32. |
| The loader reads the converted weights directly without requantizing them. |
|
|
| ## Serving |
|
|
| Use the ROCm vLLM image built by the Seekr GitLab pipeline from the source tree |
| containing Inkling 2 and prequantized row-FP8 support. The Docker build validates |
| that the wheel and installed package contain the current Inkling implementation, |
| renderer, and model registrations. No separate patch is distributed here. |
| This checkpoint requires that implementation; an unmodified vLLM installation |
| is not a supported loader for this format. |
|
|
| Validated hardware: **8 AMD MI300X GPUs (gfx942), 192 GiB per GPU**. |
| Other GPU architectures have not been validated. |
|
|
| Inside the deployment image: |
|
|
| ```bash |
| export OMP_NUM_THREADS=1 |
| export VLLM_ROCM_USE_AITER=1 |
| export VLLM_ROCM_USE_AITER_CUSTOM_AR=1 |
| |
| vllm serve seekr-org/Inkling-2-Small-FP8-ROCm \ |
| --quantization inkling2 --tensor-parallel-size 8 \ |
| --max-model-len 262144 --max-num-batched-tokens 2048 --max-num-seqs 8 \ |
| --no-enable-prefix-caching \ |
| --reasoning-parser inkling --limit-mm-per-prompt '{"image":2}' \ |
| --gpu-memory-utilization 0.9 \ |
| --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}' \ |
| --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","decode_cudagraph_capture_request_sizes":[1,2,4,8],"use_breakable_cudagraph":false}' \ |
| --kernel-config '{"enable_jit_warmup":false}' |
| ``` |
|
|
| The image's entrypoint is `vllm serve`. With Docker, expose `/dev/kfd` and |
| `/dev/dri`, use host IPC, mount a persistent Hugging Face cache, and pass the |
| model name and flags above as container arguments. Set `HF_TOKEN` in the |
| container environment if required; no credentials are included in this repo. |
|
|
| Images use standard `image_url` chat content parts. To request direct answers, |
| set `chat_template_kwargs.reasoning_effort` to `0`. The default permits reasoning. |
|
|
| ## Measurements and accuracy |
|
|
| Prior end-to-end measurements of this row-FP8 quantization used 24 distinct |
| math/programming/prose tasks, about 2K input and 512 completion tokens per |
| request, temperature 1, default reasoning, prefix caching disabled, and the |
| seven-token probabilistic DSpark configuration above: |
|
|
| | Metric | Row FP8 | Original MXFP8 scales | |
| |---|---:|---:| |
| | C1 output tokens/s | 247.47 | 141.20 | |
| | C8 aggregate output tokens/s | 800.03 | 348.82 | |
| | GSM8K, fixed 50-problem sample | 48/50 | 48/50 | |
| | HumanEval, fixed 20-problem sample | 19/20 | 20/20 | |
|
|
| Throughput includes prefill, the final partially occupied batch, and reasoning |
| tokens. These are short-prompt results, not throughput at 256K input. The C8 |
| 1,000-token/s goal was not reached. Prose alone averaged 198.89 tokens/s at C1. |
|
|
| The paired accuracy check used temperature 0, reasoning effort 0, and a |
| 2,048-token generation limit for each mode. The row-FP8 coding miss exhausted |
| its output budget. One GSM8K reference answer is arithmetically incorrect; |
| raw scores preserve that label. These small samples do not establish full |
| benchmark parity, and greedy continuations can vary with batching. |
|
|
| Both quantization paths passed a 261,040-input-token recall check and text, |
| image colors, OCR, multiple-image, and streaming checks before serialization. |
| The exported checkpoint's additional load validation is recorded in |
| [validation_results.json](validation_results.json). |
|
|
| ## License and attribution |
|
|
| Apache License 2.0, inherited from the upstream checkpoint. See [LICENSE](LICENSE). |
| Original model: Thinking Machines Lab. This repository converts the expert |
| precision for AMD MI300X deployment and retains the upstream model files and |
| attribution. Model purpose and upstream information are in UPSTREAM_README.md. |
| |
