flint_cpu_opt / benchmarks /profile-prefill.txt
HelloSun's picture
flint CPU opt: IQ4_XS AVX512-VNNI vec_dot, Q5_K mask merge, rebased on de5122d; re-measured benchmarks and KLD
d35f8cb verified
Raw History Blame Contribute Delete
8.52 kB
| model | size | params | backend | threads | n_batch | n_ubatch | fa | lm | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | --: | ---------: | --------------: | -------------------: |
| flint ?B Q4_K - Medium | 23.33 GiB | 39.79 B | CPU | 16 | 1024 | 1024 | 1 | mlock | pp2048 | 309.71 ± 0.00 |
[GGML_PROF] total_us=13581874
[GGML_PROF] GET_ROWS f32 <q8_0 [ 2048, 1024, 1] 5.90 ms 0.04% calls=4 us/call=1476.0
[GGML_PROF] RMS_NORM f32 <f32 [ 2048, 1024, 1] 125.01 ms 0.92% calls=388 us/call=322.2
[GGML_PROF] MUL_MAT f32 <q8_0 [ 4096, 1024, 1] 732.99 ms 5.40% calls=168 us/call=4363.0
[GGML_PROF] RMS_NORM f32 <f32 [ 128, 32,1024] 139.90 ms 1.03% calls=196 us/call=713.8
[GGML_PROF] MUL_MAT f32 <q8_0 [ 512, 1024, 1] 245.74 ms 1.81% calls=336 us/call=731.4
[GGML_PROF] RMS_NORM f32 <f32 [ 128, 4,1024] 50.24 ms 0.37% calls=196 us/call=256.3
[GGML_PROF] SET_ROWS f16 <f32 [ 512, 2048, 1] 19.08 ms 0.14% calls=112 us/call=170.4
[GGML_PROF] FLASH_ATTN_EXT f32 <f32 [ 128, 32,1024] 3199.22 ms 23.56% calls=196 us/call=16322.6
[GGML_PROF] MUL_MAT f32 <q8_0 [ 2048, 1024, 1] 948.76 ms 6.99% calls=308 us/call=3080.4
[GGML_PROF] ADD f32 <f32 [ 2048, 1024, 1] 88.97 ms 0.66% calls=1488 us/call=59.8
[GGML_PROF] MUL_MAT f32 <q8_0 [ 6144, 1024, 1] 138.57 ms 1.02% calls=16 us/call=8660.9
[GGML_PROF] GLU f32 <f32 [ 6144, 1024, 1] 12.83 ms 0.09% calls=8 us/call=1603.5
[GGML_PROF] ROPE f32 <f32 [ 128, 32,1024] 42.53 ms 0.31% calls=140 us/call=303.8
[GGML_PROF] ROPE f32 <f32 [ 128, 4,1024] 10.87 ms 0.08% calls=140 us/call=77.6
[GGML_PROF] SET_ROWS f16 <f32 [ 512, 1792, 1] 3.89 ms 0.03% calls=280 us/call=13.9
[GGML_PROF] MUL_MAT f32 <bf16 [ 256, 1024, 1] 197.14 ms 1.45% calls=184 us/call=1071.4
[GGML_PROF] UNARY f32 <f32 [ 256, 1024, 1] 11.94 ms 0.09% calls=184 us/call=64.9
[GGML_PROF] ARGSORT i32 <f32 [ 256, 1024, 1] 150.49 ms 1.11% calls=184 us/call=817.9
[GGML_PROF] GET_ROWS f32 <f32 [ 1, 6,1024] 2.21 ms 0.02% calls=184 us/call=12.0
[GGML_PROF] SUM_ROWS f32 <f32 [ 1, 1024, 1] 1.02 ms 0.01% calls=184 us/call=5.6
[GGML_PROF] CLAMP f32 <f32 [ 1, 1024, 1] 1.94 ms 0.01% calls=184 us/call=10.5
[GGML_PROF] DIV f32 <f32 [ 6, 1024, 1] 0.65 ms 0.00% calls=184 us/call=3.5
[GGML_PROF] MUL_MAT_ID f32 <iq3_s [ 512, 6,1024] 111.13 ms 0.82% calls=8 us/call=13891.1
[GGML_PROF] GLU f32 <f32 [ 512, 6,1024] 41.05 ms 0.30% calls=184 us/call=223.1
[GGML_PROF] UNARY f32 <f32 [ 512, 6,1024] 14.72 ms 0.11% calls=184 us/call=80.0
[GGML_PROF] POOL_1D f32 <f32 [ 1, 6,1024] 717.08 ms 5.28% calls=184 us/call=3897.2
[GGML_PROF] SCALE f32 <f32 [ 1, 6,1024] 6.38 ms 0.05% calls=552 us/call=11.6
[GGML_PROF] LOG f32 <f32 [ 1, 6,1024] 1.32 ms 0.01% calls=184 us/call=7.2
[GGML_PROF] UNARY f32 <f32 [ 1, 6,1024] 1.79 ms 0.01% calls=368 us/call=4.9
[GGML_PROF] CLAMP f32 <f32 [ 1, 6,1024] 8.17 ms 0.06% calls=184 us/call=44.4
[GGML_PROF] DIV f32 <f32 [ 512, 6,1024] 48.76 ms 0.36% calls=184 us/call=265.0
[GGML_PROF] MUL_MAT_ID f32 <q5_K [ 2048, 6,1024] 1450.95 ms 10.68% calls=116 us/call=12508.2
[GGML_PROF] MUL f32 <f32 [ 2048, 6,1024] 388.11 ms 2.86% calls=368 us/call=1054.7
[GGML_PROF] MUL_MAT f32 <q8_0 [ 1024, 1024, 1] 292.28 ms 2.15% calls=264 us/call=1107.1
[GGML_PROF] GLU f32 <f32 [ 1024, 1024, 1] 9.64 ms 0.07% calls=184 us/call=52.4
[GGML_PROF] MUL_MAT_ID f32 <q5_K [ 512, 6,1024] 1415.40 ms 10.42% calls=144 us/call=9829.2
[GGML_PROF] MUL_MAT_ID f32 <q8_0 [ 2048, 6,1024] 105.06 ms 0.77% calls=4 us/call=26264.0
[GGML_PROF] MUL_MAT_ID f32 <q6_K [ 2048, 6,1024] 62.41 ms 0.46% calls=4 us/call=15601.8
[GGML_PROF] MUL_MAT f32 <q5_K [ 4096, 1024, 1] 140.26 ms 1.03% calls=28 us/call=5009.4
[GGML_PROF] MUL_MAT f32 <q5_K [ 512, 1024, 1] 48.46 ms 0.36% calls=56 us/call=865.3
[GGML_PROF] MUL_MAT f32 <q5_K [ 2048, 1024, 1] 215.82 ms 1.59% calls=76 us/call=2839.8
[GGML_PROF] MUL_MAT_ID f32 <iq4_xs [ 512, 6,1024] 1376.81 ms 10.14% calls=192 us/call=7170.9
[GGML_PROF] MUL_MAT_ID f32 <iq4_xs [ 2048, 6,1024] 631.64 ms 4.65% calls=60 us/call=10527.4
[GGML_PROF] MUL_MAT_ID f32 <q4_K [ 512, 6,1024] 194.15 ms 1.43% calls=24 us/call=8089.7
[GGML_PROF] MUL_MAT f32 <q5_K [ 1024, 1024, 1] 146.41 ms 1.08% calls=96 us/call=1525.1
[GGML_PROF] MUL_MAT f32 <iq4_xs [ 1024, 1024, 1] 10.24 ms 0.08% calls=8 us/call=1280.6
[GGML_PROF] MUL_MAT f32 <iq4_xs [ 2048, 1024, 1] 4.50 ms 0.03% calls=4 us/call=1126.0
[GGML_PROF] GET_ROWS f32 <f32 [ 2048, 1, 1] 0.02 ms 0.00% calls=8 us/call=2.1
[GGML_PROF] ADD f32 <f32 [ 2048, 1, 1] 0.05 ms 0.00% calls=32 us/call=1.5
[GGML_PROF] RMS_NORM f32 <f32 [ 2048, 1, 1] 0.04 ms 0.00% calls=8 us/call=4.5
[GGML_PROF] MUL_MAT f32 <bf16 [ 256, 1, 1] 0.07 ms 0.00% calls=4 us/call=16.8
[GGML_PROF] UNARY f32 <f32 [ 256, 1, 1] 0.01 ms 0.00% calls=4 us/call=2.5
[GGML_PROF] ARGSORT i32 <f32 [ 256, 1, 1] 0.06 ms 0.00% calls=4 us/call=13.8
[GGML_PROF] GET_ROWS f32 <f32 [ 1, 6, 1] 0.01 ms 0.00% calls=4 us/call=1.2
[GGML_PROF] SUM_ROWS f32 <f32 [ 1, 1, 1] 0.01 ms 0.00% calls=4 us/call=1.5
[GGML_PROF] CLAMP f32 <f32 [ 1, 1, 1] 0.00 ms 0.00% calls=4 us/call=1.0
[GGML_PROF] DIV f32 <f32 [ 6, 1, 1] 0.00 ms 0.00% calls=4 us/call=1.0
[GGML_PROF] MUL_MAT_ID f32 <iq4_xs [ 512, 6, 1] 0.25 ms 0.00% calls=8 us/call=30.9
[GGML_PROF] GLU f32 <f32 [ 512, 6, 1] 0.01 ms 0.00% calls=4 us/call=2.2
[GGML_PROF] UNARY f32 <f32 [ 512, 6, 1] 0.01 ms 0.00% calls=4 us/call=1.5
[GGML_PROF] POOL_1D f32 <f32 [ 1, 6, 1] 0.02 ms 0.00% calls=4 us/call=5.8
[GGML_PROF] SCALE f32 <f32 [ 1, 6, 1] 0.01 ms 0.00% calls=12 us/call=1.1
[GGML_PROF] LOG f32 <f32 [ 1, 6, 1] 0.01 ms 0.00% calls=4 us/call=1.8
[GGML_PROF] UNARY f32 <f32 [ 1, 6, 1] 0.01 ms 0.00% calls=8 us/call=1.4
[GGML_PROF] CLAMP f32 <f32 [ 1, 6, 1] 0.00 ms 0.00% calls=4 us/call=1.0
[GGML_PROF] DIV f32 <f32 [ 512, 6, 1] 0.01 ms 0.00% calls=4 us/call=1.2
[GGML_PROF] MUL_MAT_ID f32 <iq4_xs [ 2048, 6, 1] 0.14 ms 0.00% calls=4 us/call=35.0
[GGML_PROF] MUL f32 <f32 [ 2048, 6, 1] 0.03 ms 0.00% calls=8 us/call=3.5
[GGML_PROF] MUL_MAT f32 <q8_0 [ 1024, 1, 1] 0.15 ms 0.00% calls=8 us/call=18.6
[GGML_PROF] GLU f32 <f32 [ 1024, 1, 1] 0.01 ms 0.00% calls=4 us/call=2.0
[GGML_PROF] MUL_MAT f32 <q8_0 [ 2048, 1, 1] 0.07 ms 0.00% calls=4 us/call=17.5
[GGML_PROF] MUL_MAT f32 <q8_0 [150144, 1, 1] 8.43 ms 0.06% calls=4 us/call=2106.2
build: de5122d87 (1)