| model | size | params | backend | threads | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: | | bailingmoe3 7.9B.A1.3B BF16 | 14.71 GiB | 7.89 B | CPU | 64 | pp512 | 425.22 ± 19.90 | | bailingmoe3 7.9B.A1.3B BF16 | 14.71 GiB | 7.89 B | CPU | 64 | tg128 | 26.89 ± 0.06 | | bailingmoe3 7.9B.A1.3B Q8_0 | 7.82 GiB | 7.89 B | CPU | 64 | pp512 | 503.99 ± 14.85 | | bailingmoe3 7.9B.A1.3B Q8_0 | 7.82 GiB | 7.89 B | CPU | 64 | tg128 | 32.99 ± 1.51 | | bailingmoe3 7.9B.A1.3B Q4_K - Medium | 4.49 GiB | 7.89 B | CPU | 64 | pp512 | 424.16 ± 1.63 | | bailingmoe3 7.9B.A1.3B Q4_K - Medium | 4.49 GiB | 7.89 B | CPU | 64 | tg128 | 33.70 ± 0.84 | | bailingmoe3 7.9B.A1.3B Q8_0 | 7.30 GiB | 7.89 B | CPU | 64 | pp512 | 457.76 ± 3.86 | | bailingmoe3 7.9B.A1.3B Q8_0 | 7.30 GiB | 7.89 B | CPU | 64 | tg128 | 34.19 ± 0.35 | | bailingmoe3 7.9B.A1.3B Q8_0 | 4.34 GiB | 7.89 B | CPU | 64 | pp512 | 368.65 ± 2.86 | | bailingmoe3 7.9B.A1.3B Q8_0 | 4.34 GiB | 7.89 B | CPU | 64 | tg128 | 32.84 ± 1.06 | build: 61dbf49 (1)