lingtiny30vq3 / results /bench.tsv
HelloSun's picture
LUT-VQ3-64: machine-tuned (threads/NUMA/ubatch), full 128k ctx data, one-click build, results
511ba50 verified
Raw History Blame Contribute Delete
1.54 kB
| model | size | params | backend | threads | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| bailingmoe3 7.9B.A1.3B BF16 | 14.71 GiB | 7.89 B | CPU | 64 | pp512 | 425.22 ± 19.90 |
| bailingmoe3 7.9B.A1.3B BF16 | 14.71 GiB | 7.89 B | CPU | 64 | tg128 | 26.89 ± 0.06 |
| bailingmoe3 7.9B.A1.3B Q8_0 | 7.82 GiB | 7.89 B | CPU | 64 | pp512 | 503.99 ± 14.85 |
| bailingmoe3 7.9B.A1.3B Q8_0 | 7.82 GiB | 7.89 B | CPU | 64 | tg128 | 32.99 ± 1.51 |
| bailingmoe3 7.9B.A1.3B Q4_K - Medium | 4.49 GiB | 7.89 B | CPU | 64 | pp512 | 424.16 ± 1.63 |
| bailingmoe3 7.9B.A1.3B Q4_K - Medium | 4.49 GiB | 7.89 B | CPU | 64 | tg128 | 33.70 ± 0.84 |
| bailingmoe3 7.9B.A1.3B Q8_0 | 7.30 GiB | 7.89 B | CPU | 64 | pp512 | 457.76 ± 3.86 |
| bailingmoe3 7.9B.A1.3B Q8_0 | 7.30 GiB | 7.89 B | CPU | 64 | tg128 | 34.19 ± 0.35 |
| bailingmoe3 7.9B.A1.3B Q8_0 | 4.34 GiB | 7.89 B | CPU | 64 | pp512 | 368.65 ± 2.86 |
| bailingmoe3 7.9B.A1.3B Q8_0 | 4.34 GiB | 7.89 B | CPU | 64 | tg128 | 32.84 ± 1.06 |
build: 61dbf49 (1)