Measured performance
Browse files
CARD.md
CHANGED
|
@@ -71,6 +71,29 @@ Weight encoding: ternary `{-1, 0, +1}` -> 2-bit codes `{1, 2, 3}` packed four
|
|
| 71 |
per byte (decode is `byte - 2`), the same packing used by
|
| 72 |
phanerozoic/bitnet-tc and Microsoft's bitnet.cpp.
|
| 73 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 74 |
## Requirements
|
| 75 |
|
| 76 |
- `K` divisible by 32.
|
|
|
|
| 71 |
per byte (decode is `byte - 2`), the same packing used by
|
| 72 |
phanerozoic/bitnet-tc and Microsoft's bitnet.cpp.
|
| 73 |
|
| 74 |
+
## Performance
|
| 75 |
+
|
| 76 |
+
Measured on a 16 vCPU x86-64 host (AVX2, no VNNI; HF Jobs `cpu-xl`),
|
| 77 |
+
torch 2.12 CPU, median of 15:
|
| 78 |
+
|
| 79 |
+
| shape (M, N, K) | torch bf16 matmul | bitnet-cpu | speedup |
|
| 80 |
+
|---|---|---|---|
|
| 81 |
+
| 1, 6912, 2560 | 0.24 ms | 0.37 ms | 0.7x |
|
| 82 |
+
| 1, 11008, 4096 | 0.52 ms | 0.42 ms | 1.2x |
|
| 83 |
+
| 1, 128256, 2560 | 9.86 ms | 2.01 ms | 4.9x |
|
| 84 |
+
| 16, 6912, 2560 | 2.76 ms | 0.87 ms | 3.2x |
|
| 85 |
+
| 128, 6912, 2560 | 22.83 ms | 1.66 ms | 13.8x |
|
| 86 |
+
| 512, 11008, 4096 | 351.6 ms | 13.25 ms | 26.5x |
|
| 87 |
+
|
| 88 |
+
End-to-end `microsoft/bitnet-b1.58-2B-4T-bf16` through transformers with all
|
| 89 |
+
210 BitLinear layers routed to the kernel: decode throughput 0.78 -> 3.86
|
| 90 |
+
tok/s (5.0x) versus the stock forward on the same host, with identical greedy
|
| 91 |
+
output over the measured window. AVX-VNNI and AVX-512 VNNI hosts run faster
|
| 92 |
+
than this AVX2 baseline.
|
| 93 |
+
|
| 94 |
+
The integer path is exact; output differs from an f32 reference only by bf16
|
| 95 |
+
output rounding (max relative error ~4e-3 across all dispatch paths).
|
| 96 |
+
|
| 97 |
## Requirements
|
| 98 |
|
| 99 |
- `K` divisible by 32.
|
README.md
CHANGED
|
@@ -71,6 +71,29 @@ Weight encoding: ternary `{-1, 0, +1}` -> 2-bit codes `{1, 2, 3}` packed four
|
|
| 71 |
per byte (decode is `byte - 2`), the same packing used by
|
| 72 |
phanerozoic/bitnet-tc and Microsoft's bitnet.cpp.
|
| 73 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 74 |
## Requirements
|
| 75 |
|
| 76 |
- `K` divisible by 32.
|
|
|
|
| 71 |
per byte (decode is `byte - 2`), the same packing used by
|
| 72 |
phanerozoic/bitnet-tc and Microsoft's bitnet.cpp.
|
| 73 |
|
| 74 |
+
## Performance
|
| 75 |
+
|
| 76 |
+
Measured on a 16 vCPU x86-64 host (AVX2, no VNNI; HF Jobs `cpu-xl`),
|
| 77 |
+
torch 2.12 CPU, median of 15:
|
| 78 |
+
|
| 79 |
+
| shape (M, N, K) | torch bf16 matmul | bitnet-cpu | speedup |
|
| 80 |
+
|---|---|---|---|
|
| 81 |
+
| 1, 6912, 2560 | 0.24 ms | 0.37 ms | 0.7x |
|
| 82 |
+
| 1, 11008, 4096 | 0.52 ms | 0.42 ms | 1.2x |
|
| 83 |
+
| 1, 128256, 2560 | 9.86 ms | 2.01 ms | 4.9x |
|
| 84 |
+
| 16, 6912, 2560 | 2.76 ms | 0.87 ms | 3.2x |
|
| 85 |
+
| 128, 6912, 2560 | 22.83 ms | 1.66 ms | 13.8x |
|
| 86 |
+
| 512, 11008, 4096 | 351.6 ms | 13.25 ms | 26.5x |
|
| 87 |
+
|
| 88 |
+
End-to-end `microsoft/bitnet-b1.58-2B-4T-bf16` through transformers with all
|
| 89 |
+
210 BitLinear layers routed to the kernel: decode throughput 0.78 -> 3.86
|
| 90 |
+
tok/s (5.0x) versus the stock forward on the same host, with identical greedy
|
| 91 |
+
output over the measured window. AVX-VNNI and AVX-512 VNNI hosts run faster
|
| 92 |
+
than this AVX2 baseline.
|
| 93 |
+
|
| 94 |
+
The integer path is exact; output differs from an f32 reference only by bf16
|
| 95 |
+
output rounding (max relative error ~4e-3 across all dispatch paths).
|
| 96 |
+
|
| 97 |
## Requirements
|
| 98 |
|
| 99 |
- `K` divisible by 32.
|