Blame for README.md

---
license: apache-2.0
base_model: thinkingmachines/Inkling-2-Small-MXFP8
base_model_relation: quantized
library_name: vllm
pipeline_tag: image-text-to-text
tags:
- fp8
- rocm
- mi300x
- inkling2
- moe
- dspark
- vision
---

# Inkling 2 Small FP8 for AMD MI300X

A row-scaled E4M3FNUZ conversion of
[Thinking Machines' Inkling 2 Small MXFP8](https://huggingface.co/thinkingmachines/Inkling-2-Small-MXFP8),
revision `3febc62259ee92ea36e2fb6e8303371c4eb3922c`.
The original model was developed by Thinking Machines Lab. This repository
contains a precision conversion, not a new trained model. The upstream model
card is preserved in [UPSTREAM_README.md](UPSTREAM_README.md).

## Precision and files

The 96 expert projection tensors use native AMD **E4M3FNUZ FP8**, with one FP32
scale per expert output channel. Activations use dynamic per-token E4M3FNUZ
quantization. Weights are dequantized from the original MXFP8 block scales and
requantized with `scale = max(abs(row)) / 240`. Attention, dense projections,
vision, and the DSpark draft retain the upstream precision. This is mixed
precision inference.

All 48 target weight shards, tokenizer/configuration files, vision weights,
and `mtp.safetensors` are included. The checkpoint is approximately 605.14 GB.
[conversion_manifest.json](conversion_manifest.json) records the source revision,
per-shard tensor sizes, and serialization verification fingerprints.
The original checkpoint is unchanged.

`hf_quant_config.json` marks converted experts with `compute_dtype: fp8`;
`config.json` selects `expert_fp8_scale_mode: row`. Expert scale tensors retain
the `_weight_scale` suffix and have shape `[384, output_channels]`, dtype FP32.
The loader reads the converted weights directly without requantizing them.

## Serving

Use the ROCm vLLM image built by the Seekr GitLab pipeline from the source tree
containing Inkling 2 and prequantized row-FP8 support. The Docker build validates
that the wheel and installed package contain the current Inkling implementation,
renderer, and model registrations. No separate patch is distributed here.
This checkpoint requires that implementation; an unmodified vLLM installation
is not a supported loader for this format.

Validated hardware: **8 AMD MI300X GPUs (gfx942), 192 GiB per GPU**.
Other GPU architectures have not been validated.

Inside the deployment image:

```bash
export OMP_NUM_THREADS=1
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_CUSTOM_AR=1

vllm serve seekr-org/Inkling-2-Small-FP8-ROCm \
  --quantization inkling2 --tensor-parallel-size 8 \
  --max-model-len 262144 --max-num-batched-tokens 2048 --max-num-seqs 8 \
  --no-enable-prefix-caching \
  --reasoning-parser inkling --limit-mm-per-prompt '{"image":2}' \
  --gpu-memory-utilization 0.9 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}' \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","decode_cudagraph_capture_request_sizes":[1,2,4,8],"use_breakable_cudagraph":false}' \
  --kernel-config '{"enable_jit_warmup":false}'
```

The image's entrypoint is `vllm serve`. With Docker, expose `/dev/kfd` and
`/dev/dri`, use host IPC, mount a persistent Hugging Face cache, and pass the
model name and flags above as container arguments. Set `HF_TOKEN` in the
container environment if required; no credentials are included in this repo.

Images use standard `image_url` chat content parts. To request direct answers,
set `chat_template_kwargs.reasoning_effort` to `0`. The default permits reasoning.

## Measurements and accuracy

Prior end-to-end measurements of this row-FP8 quantization used 24 distinct
math/programming/prose tasks, about 2K input and 512 completion tokens per
request, temperature 1, default reasoning, prefix caching disabled, and the
seven-token probabilistic DSpark configuration above:

| Metric | Row FP8 | Original MXFP8 scales |
|---|---:|---:|
| C1 output tokens/s | 247.47 | 141.20 |
| C8 aggregate output tokens/s | 800.03 | 348.82 |
| GSM8K, fixed 50-problem sample | 48/50 | 48/50 |
| HumanEval, fixed 20-problem sample | 19/20 | 20/20 |

Throughput includes prefill, the final partially occupied batch, and reasoning
tokens. These are short-prompt results, not throughput at 256K input. The C8
1,000-token/s goal was not reached. Prose alone averaged 198.89 tokens/s at C1.

The paired accuracy check used temperature 0, reasoning effort 0, and a
2,048-token generation limit for each mode. The row-FP8 coding miss exhausted
its output budget. One GSM8K reference answer is arithmetically incorrect;
raw scores preserve that label. These small samples do not establish full
benchmark parity, and greedy continuations can vary with batching.

Both quantization paths passed a 261,040-input-token recall check and text,
image colors, OCR, multiple-image, and streaming checks before serialization.
The exported checkpoint's additional load validation is recorded in
[validation_results.json](validation_results.json).

## License and attribution

Apache License 2.0, inherited from the upstream checkpoint. See [LICENSE](LICENSE).
Original model: Thinking Machines Lab. This repository converts the expert
precision for AMD MI300X deployment and retains the upstream model files and
attribution. Model purpose and upstream information are in UPSTREAM_README.md.