# GPT-2 125M through FIR: inference benchmark

The model runs from `model.fir` only (`runtime graph`): one GPU session per token, KV cache
resident on the GPU. Q4 is the default (resident Q4 linears); `--fir-f32` keeps the linears
in fp32 as the correctness reference.

## English

### Intel Iris Plus Graphics 640

Measured in chat mode (`-cnv`, temperature 0.8, chat template from `chat_template.jinja`).

| Backend | Weights | Prompt tokens | Prompt eval | Generated tokens | Decode |
|---|---|---:|---:|---:|---:|
| Metal | Q4 | 13 | 0.583 s, 22.31 tok/s | 168 | 19.596 s, 8.57 tok/s |
| Vulkan | Q4 | 15 | 1.048 s, 14.31 tok/s | 35 | 3.071 s, 11.40 tok/s |
| Metal | fp32 | 10 | 0.732 s, 13.67 tok/s | 21 | 1.948 s, 10.78 tok/s |
| Vulkan | fp32 | 11 | 0.955 s, 11.52 tok/s | 51 | 5.974 s, 8.54 tok/s |

```sh
other/formers.n/formers-cli --models models/gpt2-125m -cnv --temp 0.8 --backend metal
other/formers.n/formers-cli --models models/gpt2-125m -cnv --temp 0.8 --backend vulkan
other/formers.n/formers-cli --models models/gpt2-125m -cnv --fir-f32 --temp 0.8 --backend metal
other/formers.n/formers-cli --models models/gpt2-125m -cnv --fir-f32 --temp 0.8 --backend vulkan
```

## Tiếng Việt

### Intel Iris Plus Graphics 640

Đo ở chế độ chat (`-cnv`, temperature 0.8, template từ `chat_template.jinja`). GPT-2 125M

| Backend | Weight | Token prompt | Prompt eval | Token sinh ra | Decode |
|---|---|---:|---:|---:|---:|
| Metal | Q4 | 13 | 0,583 s, 22,31 tok/s | 168 | 19,596 s, 8,57 tok/s |
| Vulkan | Q4 | 15 | 1,048 s, 14,31 tok/s | 35 | 3,071 s, 11,40 tok/s |
| Metal | fp32 | 10 | 0,732 s, 13,67 tok/s | 21 | 1,948 s, 10,78 tok/s |
| Vulkan | fp32 | 11 | 0,955 s, 11,52 tok/s | 51 | 5,974 s, 8,54 tok/s |

```sh
other/formers.n/formers-cli --models models/gpt2-125m -cnv --temp 0.8 --backend metal
other/formers.n/formers-cli --models models/gpt2-125m -cnv --temp 0.8 --backend vulkan
other/formers.n/formers-cli --models models/gpt2-125m -cnv --fir-f32 --temp 0.8 --backend metal
other/formers.n/formers-cli --models models/gpt2-125m -cnv --fir-f32 --temp 0.8 --backend vulkan
```