GPT-2 125M through FIR: inference benchmark
The model runs from model.fir only (runtime graph): one GPU session per token, KV cache
resident on the GPU. Q4 is the default (resident Q4 linears); --fir-f32 keeps the linears
in fp32 as the correctness reference.
English
Intel Iris Plus Graphics 640
Measured in chat mode (-cnv, temperature 0.8, chat template from chat_template.jinja).
| Backend | Weights | Prompt tokens | Prompt eval | Generated tokens | Decode |
|---|---|---|---|---|---|
| Metal | Q4 | 13 | 0.583 s, 22.31 tok/s | 168 | 19.596 s, 8.57 tok/s |
| Vulkan | Q4 | 15 | 1.048 s, 14.31 tok/s | 35 | 3.071 s, 11.40 tok/s |
| Metal | fp32 | 10 | 0.732 s, 13.67 tok/s | 21 | 1.948 s, 10.78 tok/s |
| Vulkan | fp32 | 11 | 0.955 s, 11.52 tok/s | 51 | 5.974 s, 8.54 tok/s |
other/formers.n/formers-cli --models models/gpt2-125m -cnv --temp 0.8 --backend metal
other/formers.n/formers-cli --models models/gpt2-125m -cnv --temp 0.8 --backend vulkan
other/formers.n/formers-cli --models models/gpt2-125m -cnv --fir-f32 --temp 0.8 --backend metal
other/formers.n/formers-cli --models models/gpt2-125m -cnv --fir-f32 --temp 0.8 --backend vulkan
Tiếng Việt
Intel Iris Plus Graphics 640
Đo ở chế độ chat (-cnv, temperature 0.8, template từ chat_template.jinja). GPT-2 125M
| Backend | Weight | Token prompt | Prompt eval | Token sinh ra | Decode |
|---|---|---|---|---|---|
| Metal | Q4 | 13 | 0,583 s, 22,31 tok/s | 168 | 19,596 s, 8,57 tok/s |
| Vulkan | Q4 | 15 | 1,048 s, 14,31 tok/s | 35 | 3,071 s, 11,40 tok/s |
| Metal | fp32 | 10 | 0,732 s, 13,67 tok/s | 21 | 1,948 s, 10,78 tok/s |
| Vulkan | fp32 | 11 | 0,955 s, 11,52 tok/s | 51 | 5,974 s, 8,54 tok/s |
other/formers.n/formers-cli --models models/gpt2-125m -cnv --temp 0.8 --backend metal
other/formers.n/formers-cli --models models/gpt2-125m -cnv --temp 0.8 --backend vulkan
other/formers.n/formers-cli --models models/gpt2-125m -cnv --fir-f32 --temp 0.8 --backend metal
other/formers.n/formers-cli --models models/gpt2-125m -cnv --fir-f32 --temp 0.8 --backend vulkan