eval_all_test.md

Evaluation report

Models: smart_turn_v3.2_cpu, smart_turn_v3.2_gpu, tamil_base_int8, base_fp32, base_int8, tiny_fp32, tiny_int8

CPU latency

model	latency (ms, 1 thread, batch 1)
smart_turn_v3.2_cpu	79.9
smart_turn_v3.2_gpu	90.0
tamil_base_int8	143.3
base_fp32	207.5
base_int8	142.4
tiny_fp32	83.7
tiny_int8	83.0

Clean test split

rows: 20431

By language

language	n	pos%	model	acc 95% CI	AUC 95% CI	prec(inc)	rec(inc)	rec(comp)
Assamese	639	67	smart_turn_v3.2_cpu	67.8 [63.7, 71.4]	0.696 [0.649, 0.736]	50.4	56.2	73.3
Assamese	639	67	smart_turn_v3.2_gpu	55.4 [51.3, 59.3]	0.678 [0.634, 0.723]	40.4	77.4	44.8
Assamese	639	67	tamil_base_int8	68.1 [64.5, 71.8]	0.738 [0.694, 0.779]	50.8	64.9	69.6
Assamese	639	67	base_fp32	83.6 [80.9, 86.5]	0.892 [0.864, 0.918]	74.6	75.0	87.7
Assamese	639	67	base_int8	76.8 [73.6, 80.0]	0.849 [0.816, 0.882]	62.0	74.5	78.0
Assamese	639	67	tiny_fp32	81.4 [78.4, 84.4]	0.846 [0.813, 0.880]	73.1	67.8	87.9
Assamese	639	67	tiny_int8	80.4 [77.3, 83.4]	0.837 [0.801, 0.870]	71.1	67.3	86.8
Bengali	1482	57	smart_turn_v3.2_cpu	81.2 [79.2, 83.2]	0.870 [0.851, 0.889]	82.5	72.0	88.3
Bengali	1482	57	smart_turn_v3.2_gpu	75.9 [73.8, 78.1]	0.893 [0.878, 0.908]	67.4	85.8	68.3
Bengali	1482	57	tamil_base_int8	62.7 [60.2, 65.0]	0.638 [0.608, 0.666]	66.1	28.5	88.8
Bengali	1482	57	base_fp32	86.6 [84.8, 88.3]	0.934 [0.920, 0.946]	84.8	84.1	88.5
Bengali	1482	57	base_int8	81.0 [78.9, 83.0]	0.898 [0.881, 0.914]	76.2	81.6	80.5
Bengali	1482	57	tiny_fp32	85.4 [83.5, 87.1]	0.934 [0.921, 0.946]	83.5	82.6	87.5
Bengali	1482	57	tiny_int8	84.3 [82.5, 86.0]	0.916 [0.899, 0.930]	84.0	78.7	88.6
English	7820	49	smart_turn_v3.2_cpu	92.9 [92.4, 93.4]	0.978 [0.976, 0.981]	93.8	92.1	93.6
English	7820	49	smart_turn_v3.2_gpu	94.7 [94.2, 95.2]	0.986 [0.983, 0.988]	95.4	94.2	95.3
English	7820	49	tamil_base_int8	77.7 [76.8, 78.7]	0.882 [0.875, 0.889]	89.9	63.5	92.6
English	7820	49	base_fp32	94.9 [94.4, 95.4]	0.988 [0.986, 0.990]	95.3	94.6	95.2
English	7820	49	base_int8	86.2 [85.4, 86.9]	0.971 [0.968, 0.975]	80.1	96.9	75.0
English	7820	49	tiny_fp32	93.7 [93.2, 94.3]	0.979 [0.977, 0.982]	94.8	92.8	94.7
English	7820	49	tiny_int8	91.7 [91.1, 92.4]	0.973 [0.970, 0.976]	91.4	92.6	90.9
Gujarati	498	75	smart_turn_v3.2_cpu	73.5 [69.5, 77.3]	0.765 [0.715, 0.812]	47.7	58.4	78.6
Gujarati	498	75	smart_turn_v3.2_gpu	58.4 [54.2, 62.7]	0.759 [0.706, 0.811]	36.0	84.0	49.9
Gujarati	498	75	tamil_base_int8	73.3 [69.3, 77.1]	0.771 [0.721, 0.820]	47.5	61.6	77.2
Gujarati	498	75	base_fp32	84.7 [81.3, 88.0]	0.895 [0.858, 0.927]	68.1	73.6	88.5
Gujarati	498	75	base_int8	81.1 [77.9, 84.3]	0.861 [0.825, 0.898]	60.8	69.6	85.0
Gujarati	498	75	tiny_fp32	82.5 [79.5, 85.7]	0.856 [0.816, 0.895]	64.6	67.2	87.7
Gujarati	498	75	tiny_int8	80.9 [77.5, 84.1]	0.839 [0.791, 0.879]	60.6	68.8	85.0
Hindi	1744	58	smart_turn_v3.2_cpu	84.7 [83.0, 86.5]	0.932 [0.921, 0.943]	85.9	75.7	91.2
Hindi	1744	58	smart_turn_v3.2_gpu	83.0 [81.2, 85.0]	0.943 [0.932, 0.953]	73.8	91.6	77.0
Hindi	1744	58	tamil_base_int8	67.7 [65.4, 69.8]	0.710 [0.685, 0.733]	78.2	30.7	93.9
Hindi	1744	58	base_fp32	91.0 [89.6, 92.3]	0.964 [0.954, 0.973]	88.0	90.7	91.2
Hindi	1744	58	base_int8	86.8 [85.0, 88.4]	0.949 [0.939, 0.958]	80.9	89.1	85.1
Hindi	1744	58	tiny_fp32	90.2 [88.8, 91.6]	0.958 [0.948, 0.967]	87.6	89.0	91.1
Hindi	1744	58	tiny_int8	89.4 [88.0, 90.8]	0.955 [0.945, 0.964]	88.1	86.0	91.8
Kannada	385	81	smart_turn_v3.2_cpu	77.7 [73.5, 81.8]	0.770 [0.707, 0.835]	44.6	60.0	81.9
Kannada	385	81	smart_turn_v3.2_gpu	56.6 [51.9, 61.6]	0.795 [0.732, 0.854]	28.9	84.0	50.0
Kannada	385	81	tamil_base_int8	76.6 [72.2, 80.5]	0.800 [0.730, 0.864]	43.6	68.0	78.7
Kannada	385	81	base_fp32	88.6 [85.2, 91.7]	0.921 [0.879, 0.956]	67.0	81.3	90.3
Kannada	385	81	base_int8	83.9 [80.0, 87.5]	0.900 [0.852, 0.940]	56.0	81.3	84.5
Kannada	385	81	tiny_fp32	82.3 [78.4, 86.0]	0.876 [0.832, 0.916]	53.8	65.3	86.5
Kannada	385	81	tiny_int8	82.9 [79.2, 86.8]	0.878 [0.833, 0.919]	54.5	73.3	85.2
Malayalam	472	74	smart_turn_v3.2_cpu	66.9 [62.5, 71.0]	0.657 [0.600, 0.713]	38.9	45.2	74.7
Malayalam	472	74	smart_turn_v3.2_gpu	49.6 [45.1, 54.2]	0.667 [0.606, 0.728]	31.6	79.0	39.1
Malayalam	472	74	tamil_base_int8	75.6 [71.4, 79.2]	0.763 [0.714, 0.809]	53.2	59.7	81.3
Malayalam	472	74	base_fp32	84.5 [81.1, 87.9]	0.859 [0.815, 0.902]	70.1	71.8	89.1
Malayalam	472	74	base_int8	76.1 [72.0, 80.1]	0.801 [0.754, 0.847]	53.4	70.2	78.2
Malayalam	472	74	tiny_fp32	81.4 [77.8, 85.0]	0.833 [0.785, 0.878]	63.4	68.5	85.9
Malayalam	472	74	tiny_int8	80.9 [77.3, 84.5]	0.823 [0.776, 0.865]	62.9	66.9	85.9
Marathi	1311	60	smart_turn_v3.2_cpu	77.9 [75.7, 80.0]	0.851 [0.830, 0.871]	75.4	66.2	85.7
Marathi	1311	60	smart_turn_v3.2_gpu	75.8 [73.5, 78.2]	0.880 [0.863, 0.899]	64.7	86.6	68.7
Marathi	1311	60	tamil_base_int8	67.7 [65.1, 70.3]	0.655 [0.623, 0.687]	70.7	32.3	91.1
Marathi	1311	60	base_fp32	89.6 [88.0, 91.4]	0.952 [0.940, 0.962]	86.3	88.0	90.7
Marathi	1311	60	base_int8	84.3 [82.3, 86.1]	0.920 [0.903, 0.934]	79.6	81.5	86.2
Marathi	1311	60	tiny_fp32	88.1 [86.3, 89.9]	0.943 [0.930, 0.956]	85.0	85.3	90.0
Marathi	1311	60	tiny_int8	84.1 [82.2, 86.2]	0.924 [0.910, 0.938]	80.0	80.3	86.7
Odia	331	72	smart_turn_v3.2_cpu	74.6 [70.1, 79.2]	0.767 [0.706, 0.824]	54.5	58.1	81.1
Odia	331	72	smart_turn_v3.2_gpu	60.4 [55.6, 65.3]	0.785 [0.728, 0.841]	40.0	81.7	52.1
Odia	331	72	tamil_base_int8	81.6 [77.3, 85.5]	0.861 [0.816, 0.899]	66.7	68.8	86.6
Odia	331	72	base_fp32	88.5 [84.9, 91.9]	0.938 [0.906, 0.965]	82.4	75.3	93.7
Odia	331	72	base_int8	86.4 [82.5, 90.0]	0.896 [0.851, 0.939]	73.5	80.6	88.7
Odia	331	72	tiny_fp32	84.9 [81.0, 89.1]	0.885 [0.839, 0.927]	75.9	67.7	91.6
Odia	331	72	tiny_int8	86.4 [82.8, 90.0]	0.896 [0.851, 0.933]	80.0	68.8	93.3
Punjabi	556	71	smart_turn_v3.2_cpu	69.2 [65.5, 73.2]	0.732 [0.686, 0.778]	47.6	60.9	72.7
Punjabi	556	71	smart_turn_v3.2_gpu	56.7 [52.5, 60.8]	0.748 [0.702, 0.792]	38.8	86.3	44.6
Punjabi	556	71	tamil_base_int8	73.2 [69.2, 76.6]	0.787 [0.741, 0.829]	53.0	65.8	76.2
Punjabi	556	71	base_fp32	85.3 [82.4, 88.3]	0.903 [0.873, 0.932]	74.2	75.2	89.4
Punjabi	556	71	base_int8	76.6 [73.2, 80.2]	0.829 [0.787, 0.869]	57.6	72.7	78.2
Punjabi	556	71	tiny_fp32	83.1 [79.9, 86.2]	0.869 [0.831, 0.900]	69.8	73.3	87.1
Punjabi	556	71	tiny_int8	80.2 [76.6, 83.5]	0.834 [0.790, 0.874]	64.7	69.6	84.6
Tamil	4468	65	smart_turn_v3.2_cpu	70.5 [69.3, 71.8]	0.744 [0.729, 0.759]	58.1	60.6	76.0
Tamil	4468	65	smart_turn_v3.2_gpu	58.6 [57.2, 60.0]	0.744 [0.728, 0.759]	45.4	81.3	46.1
Tamil	4468	65	tamil_base_int8	85.8 [84.8, 86.8]	0.918 [0.908, 0.926]	83.7	74.6	92.0
Tamil	4468	65	base_fp32	86.2 [85.2, 87.2]	0.918 [0.909, 0.927]	83.1	76.7	91.4
Tamil	4468	65	base_int8	81.3 [80.1, 82.4]	0.877 [0.867, 0.888]	73.5	73.9	85.3
Tamil	4468	65	tiny_fp32	83.4 [82.3, 84.4]	0.885 [0.875, 0.896]	79.0	72.3	89.5
Tamil	4468	65	tiny_int8	82.1 [81.1, 83.2]	0.876 [0.865, 0.886]	76.3	71.9	87.7
Telugu	725	69	smart_turn_v3.2_cpu	73.4 [70.3, 76.4]	0.750 [0.713, 0.790]	56.6	59.4	79.6
Telugu	725	69	smart_turn_v3.2_gpu	58.1 [54.3, 61.4]	0.773 [0.734, 0.809]	41.2	83.9	46.5
Telugu	725	69	tamil_base_int8	80.7 [77.9, 83.6]	0.852 [0.819, 0.882]	66.9	74.1	83.6
Telugu	725	69	base_fp32	85.7 [83.2, 88.0]	0.908 [0.879, 0.930]	76.5	77.2	89.4
Telugu	725	69	base_int8	81.4 [78.5, 84.0]	0.871 [0.842, 0.898]	71.5	66.1	88.2
Telugu	725	69	tiny_fp32	85.0 [82.5, 87.4]	0.895 [0.869, 0.919]	77.0	73.2	90.2
Telugu	725	69	tiny_int8	85.1 [82.6, 87.6]	0.898 [0.871, 0.924]	74.8	78.1	88.2

By dataset

dataset	n	pos%	model	acc 95% CI	AUC 95% CI	prec(inc)	rec(inc)	rec(comp)
indicvoices_asm	503	86	smart_turn_v3.2_cpu	69.4 [65.2, 73.2]	0.633 [0.559, 0.706]	22.3	45.8	73.3
indicvoices_asm	503	86	smart_turn_v3.2_gpu	48.3 [43.9, 52.5]	0.598 [0.532, 0.668]	17.4	69.4	44.8
indicvoices_asm	503	86	tamil_base_int8	67.2 [62.8, 71.0]	0.688 [0.614, 0.749]	22.5	52.8	69.6
indicvoices_asm	503	86	base_fp32	85.5 [82.5, 88.5]	0.870 [0.813, 0.917]	49.5	72.2	87.7
indicvoices_asm	503	86	base_int8	76.7 [73.0, 80.3]	0.819 [0.759, 0.873]	34.5	69.4	78.0
indicvoices_asm	503	86	tiny_fp32	83.9 [80.7, 86.9]	0.809 [0.740, 0.863]	45.3	59.7	87.9
indicvoices_asm	503	86	tiny_int8	82.3 [79.1, 85.7]	0.776 [0.709, 0.833]	41.2	55.6	86.8
indicvoices_asm_pausecut	136	0	smart_turn_v3.2_cpu	61.8 [52.9, 69.9]	n/a	100.0	61.8	0.0
indicvoices_asm_pausecut	136	0	smart_turn_v3.2_gpu	81.6 [75.0, 88.2]	n/a	100.0	81.6	0.0
indicvoices_asm_pausecut	136	0	tamil_base_int8	71.3 [64.0, 78.7]	n/a	100.0	71.3	0.0
indicvoices_asm_pausecut	136	0	base_fp32	76.5 [69.1, 83.8]	n/a	100.0	76.5	0.0
indicvoices_asm_pausecut	136	0	base_int8	77.2 [69.9, 83.8]	n/a	100.0	77.2	0.0
indicvoices_asm_pausecut	136	0	tiny_fp32	72.1 [64.7, 79.4]	n/a	100.0	72.1	0.0
indicvoices_asm_pausecut	136	0	tiny_int8	73.5 [66.2, 80.9]	n/a	100.0	73.5	0.0
indicvoices_ben	383	91	smart_turn_v3.2_cpu	81.2 [77.3, 84.9]	0.725 [0.615, 0.831]	24.0	54.5	83.7
indicvoices_ben	383	91	smart_turn_v3.2_gpu	48.3 [43.3, 53.0]	0.749 [0.640, 0.850]	12.3	81.8	45.1
indicvoices_ben	383	91	tamil_base_int8	77.0 [72.6, 81.2]	0.785 [0.686, 0.865]	22.2	66.7	78.0
indicvoices_ben	383	91	base_fp32	88.8 [85.1, 91.6]	0.850 [0.770, 0.923]	40.4	63.6	91.1
indicvoices_ben	383	91	base_int8	84.3 [80.4, 88.0]	0.829 [0.734, 0.904]	31.5	69.7	85.7
indicvoices_ben	383	91	tiny_fp32	90.6 [87.7, 93.5]	0.855 [0.761, 0.930]	46.5	60.6	93.4
indicvoices_ben	383	91	tiny_int8	90.9 [88.0, 93.5]	0.849 [0.766, 0.920]	47.7	63.6	93.4
indicvoices_ben_pausecut	99	0	smart_turn_v3.2_cpu	65.7 [55.6, 74.7]	n/a	100.0	65.7	0.0
indicvoices_ben_pausecut	99	0	smart_turn_v3.2_gpu	89.9 [82.8, 94.9]	n/a	100.0	89.9	0.0
indicvoices_ben_pausecut	99	0	tamil_base_int8	67.7 [58.6, 76.8]	n/a	100.0	67.7	0.0
indicvoices_ben_pausecut	99	0	base_fp32	83.8 [76.8, 90.9]	n/a	100.0	83.8	0.0
indicvoices_ben_pausecut	99	0	base_int8	85.9 [78.8, 91.9]	n/a	100.0	85.9	0.0
indicvoices_ben_pausecut	99	0	tiny_fp32	80.8 [72.7, 88.9]	n/a	100.0	80.8	0.0
indicvoices_ben_pausecut	99	0	tiny_int8	79.8 [71.7, 86.9]	n/a	100.0	79.8	0.0
indicvoices_guj	409	91	smart_turn_v3.2_cpu	74.8 [70.4, 78.7]	0.644 [0.550, 0.734]	14.0	36.1	78.6
indicvoices_guj	409	91	smart_turn_v3.2_gpu	51.8 [46.9, 56.7]	0.613 [0.516, 0.708]	12.2	72.2	49.9
indicvoices_guj	409	91	tamil_base_int8	73.6 [68.9, 77.5]	0.668 [0.583, 0.751]	13.3	36.1	77.2
indicvoices_guj	409	91	base_fp32	85.6 [82.2, 89.0]	0.818 [0.740, 0.888]	31.7	55.6	88.5
indicvoices_guj	409	91	base_int8	82.4 [79.0, 85.8]	0.766 [0.684, 0.841]	26.3	55.6	85.0
indicvoices_guj	409	91	tiny_fp32	83.6 [80.0, 87.0]	0.766 [0.688, 0.836]	24.6	41.7	87.7
indicvoices_guj	409	91	tiny_int8	81.9 [78.2, 85.6]	0.754 [0.664, 0.835]	24.3	50.0	85.0
indicvoices_guj_pausecut	89	0	smart_turn_v3.2_cpu	67.4 [58.4, 77.5]	n/a	100.0	67.4	0.0
indicvoices_guj_pausecut	89	0	smart_turn_v3.2_gpu	88.8 [82.0, 95.5]	n/a	100.0	88.8	0.0
indicvoices_guj_pausecut	89	0	tamil_base_int8	71.9 [61.8, 80.9]	n/a	100.0	71.9	0.0
indicvoices_guj_pausecut	89	0	base_fp32	80.9 [71.9, 88.8]	n/a	100.0	80.9	0.0
indicvoices_guj_pausecut	89	0	base_int8	75.3 [66.3, 84.3]	n/a	100.0	75.3	0.0
indicvoices_guj_pausecut	89	0	tiny_fp32	77.5 [67.4, 85.4]	n/a	100.0	77.5	0.0
indicvoices_guj_pausecut	89	0	tiny_int8	76.4 [67.4, 85.4]	n/a	100.0	76.4	0.0
indicvoices_hin	384	91	smart_turn_v3.2_cpu	75.8 [71.4, 79.9]	0.712 [0.620, 0.787]	16.7	45.5	78.6
indicvoices_hin	384	91	smart_turn_v3.2_gpu	49.0 [43.8, 53.9]	0.723 [0.640, 0.805]	12.1	78.8	46.2
indicvoices_hin	384	91	tamil_base_int8	81.0 [77.1, 84.9]	0.749 [0.649, 0.835]	22.2	48.5	84.0
indicvoices_hin	384	91	base_fp32	84.9 [81.2, 88.5]	0.748 [0.631, 0.861]	30.2	57.6	87.5
indicvoices_hin	384	91	base_int8	80.2 [76.0, 84.4]	0.797 [0.720, 0.870]	23.5	57.6	82.3
indicvoices_hin	384	91	tiny_fp32	84.9 [81.8, 88.3]	0.728 [0.628, 0.823]	25.5	39.4	89.2
indicvoices_hin	384	91	tiny_int8	86.2 [82.6, 89.6]	0.762 [0.671, 0.846]	29.2	42.4	90.3
indicvoices_hin_pausecut	76	0	smart_turn_v3.2_cpu	61.8 [50.0, 72.4]	n/a	100.0	61.8	0.0
indicvoices_hin_pausecut	76	0	smart_turn_v3.2_gpu	90.8 [82.9, 96.1]	n/a	100.0	90.8	0.0
indicvoices_hin_pausecut	76	0	tamil_base_int8	59.2 [48.7, 69.7]	n/a	100.0	59.2	0.0
indicvoices_hin_pausecut	76	0	base_fp32	71.1 [60.5, 81.6]	n/a	100.0	71.1	0.0
indicvoices_hin_pausecut	76	0	base_int8	81.6 [72.4, 89.5]	n/a	100.0	81.6	0.0
indicvoices_hin_pausecut	76	0	tiny_fp32	77.6 [67.1, 85.5]	n/a	100.0	77.6	0.0
indicvoices_hin_pausecut	76	0	tiny_int8	73.7 [63.1, 82.9]	n/a	100.0	73.7	0.0
indicvoices_kan	323	96	smart_turn_v3.2_cpu	80.5 [76.5, 85.1]	0.764 [0.657, 0.870]	9.7	46.2	81.9
indicvoices_kan	323	96	smart_turn_v3.2_gpu	52.0 [46.1, 57.6]	0.797 [0.700, 0.875]	7.7	100.0	50.0
indicvoices_kan	323	96	tamil_base_int8	78.0 [73.4, 82.7]	0.754 [0.581, 0.916]	10.8	61.5	78.7
indicvoices_kan	323	96	base_fp32	89.5 [85.8, 92.9]	0.845 [0.671, 0.962]	23.1	69.2	90.3
indicvoices_kan	323	96	base_int8	84.2 [80.2, 87.9]	0.858 [0.699, 0.971]	17.2	76.9	84.5
indicvoices_kan	323	96	tiny_fp32	85.4 [81.1, 89.2]	0.863 [0.777, 0.933]	16.0	61.5	86.5
indicvoices_kan	323	96	tiny_int8	83.9 [79.9, 87.6]	0.868 [0.797, 0.926]	13.2	53.8	85.2
indicvoices_kan_pausecut	62	0	smart_turn_v3.2_cpu	62.9 [51.6, 75.8]	n/a	100.0	62.9	0.0
indicvoices_kan_pausecut	62	0	smart_turn_v3.2_gpu	80.6 [70.9, 90.3]	n/a	100.0	80.6	0.0
indicvoices_kan_pausecut	62	0	tamil_base_int8	69.4 [58.1, 80.6]	n/a	100.0	69.4	0.0
indicvoices_kan_pausecut	62	0	base_fp32	83.9 [74.2, 91.9]	n/a	100.0	83.9	0.0
indicvoices_kan_pausecut	62	0	base_int8	82.3 [72.6, 91.9]	n/a	100.0	82.3	0.0
indicvoices_kan_pausecut	62	0	tiny_fp32	66.1 [54.8, 77.4]	n/a	100.0	66.1	0.0
indicvoices_kan_pausecut	62	0	tiny_int8	77.4 [66.1, 87.1]	n/a	100.0	77.4	0.0
indicvoices_mal	396	88	smart_turn_v3.2_cpu	71.0 [66.2, 75.3]	0.643 [0.554, 0.730]	19.3	43.8	74.7
indicvoices_mal	396	88	smart_turn_v3.2_gpu	43.4 [38.4, 48.5]	0.628 [0.531, 0.722]	14.5	75.0	39.1
indicvoices_mal	396	88	tamil_base_int8	77.8 [73.7, 81.6]	0.707 [0.628, 0.779]	27.8	52.1	81.3
indicvoices_mal	396	88	base_fp32	85.6 [82.3, 89.1]	0.821 [0.739, 0.893]	43.3	60.4	89.1
indicvoices_mal	396	88	base_int8	76.8 [72.2, 80.8]	0.771 [0.691, 0.837]	29.6	66.7	78.2
indicvoices_mal	396	88	tiny_fp32	82.3 [78.5, 86.1]	0.747 [0.663, 0.831]	35.5	56.2	85.9
indicvoices_mal	396	88	tiny_int8	82.3 [78.5, 85.9]	0.772 [0.695, 0.847]	35.5	56.2	85.9
indicvoices_mal_pausecut	76	0	smart_turn_v3.2_cpu	46.1 [35.5, 57.9]	n/a	100.0	46.1	0.0
indicvoices_mal_pausecut	76	0	smart_turn_v3.2_gpu	81.6 [72.4, 89.5]	n/a	100.0	81.6	0.0
indicvoices_mal_pausecut	76	0	tamil_base_int8	64.5 [53.9, 75.0]	n/a	100.0	64.5	0.0
indicvoices_mal_pausecut	76	0	base_fp32	78.9 [69.7, 88.2]	n/a	100.0	78.9	0.0
indicvoices_mal_pausecut	76	0	base_int8	72.4 [63.2, 81.6]	n/a	100.0	72.4	0.0
indicvoices_mal_pausecut	76	0	tiny_fp32	76.3 [65.8, 85.5]	n/a	100.0	76.3	0.0
indicvoices_mal_pausecut	76	0	tiny_int8	73.7 [63.2, 84.2]	n/a	100.0	73.7	0.0
indicvoices_mar	444	89	smart_turn_v3.2_cpu	75.5 [71.4, 79.5]	0.766 [0.690, 0.836]	25.0	61.2	77.2
indicvoices_mar	444	89	smart_turn_v3.2_gpu	51.8 [47.1, 56.1]	0.782 [0.718, 0.847]	17.1	87.8	47.3
indicvoices_mar	444	89	tamil_base_int8	79.7 [76.1, 83.3]	0.799 [0.728, 0.863]	29.3	59.2	82.3
indicvoices_mar	444	89	base_fp32	89.6 [86.7, 92.3]	0.919 [0.879, 0.954]	52.1	77.6	91.1
indicvoices_mar	444	89	base_int8	86.5 [83.3, 89.6]	0.881 [0.822, 0.933]	43.4	73.5	88.1
indicvoices_mar	444	89	tiny_fp32	89.4 [86.5, 92.1]	0.903 [0.851, 0.948]	51.4	77.6	90.9
indicvoices_mar	444	89	tiny_int8	85.8 [82.7, 89.0]	0.882 [0.818, 0.937]	41.5	69.4	87.8
indicvoices_mar_pausecut	93	0	smart_turn_v3.2_cpu	66.7 [57.0, 77.4]	n/a	100.0	66.7	0.0
indicvoices_mar_pausecut	93	0	smart_turn_v3.2_gpu	88.2 [81.7, 94.6]	n/a	100.0	88.2	0.0
indicvoices_mar_pausecut	93	0	tamil_base_int8	73.1 [64.5, 81.7]	n/a	100.0	73.1	0.0
indicvoices_mar_pausecut	93	0	base_fp32	84.9 [77.4, 91.4]	n/a	100.0	84.9	0.0
indicvoices_mar_pausecut	93	0	base_int8	79.6 [71.0, 87.1]	n/a	100.0	79.6	0.0
indicvoices_mar_pausecut	93	0	tiny_fp32	84.9 [77.4, 92.5]	n/a	100.0	84.9	0.0
indicvoices_mar_pausecut	93	0	tiny_int8	82.8 [75.3, 89.2]	n/a	100.0	82.8	0.0
indicvoices_ori	265	90	smart_turn_v3.2_cpu	77.4 [72.4, 82.6]	0.700 [0.587, 0.807]	21.1	44.4	81.1
indicvoices_ori	265	90	smart_turn_v3.2_gpu	54.3 [48.3, 60.0]	0.731 [0.609, 0.845]	14.9	74.1	52.1
indicvoices_ori	265	90	tamil_base_int8	84.5 [80.0, 88.7]	0.870 [0.804, 0.930]	36.0	66.7	86.6
indicvoices_ori	265	90	base_fp32	89.8 [86.0, 93.6]	0.907 [0.849, 0.955]	50.0	55.6	93.7
indicvoices_ori	265	90	base_int8	87.2 [83.0, 90.9]	0.868 [0.778, 0.945]	42.6	74.1	88.7
indicvoices_ori	265	90	tiny_fp32	87.5 [83.4, 91.3]	0.807 [0.701, 0.902]	41.2	51.9	91.6
indicvoices_ori	265	90	tiny_int8	88.7 [84.9, 92.5]	0.837 [0.750, 0.916]	44.8	48.1	93.3
indicvoices_ori_pausecut	66	0	smart_turn_v3.2_cpu	63.6 [51.5, 74.2]	n/a	100.0	63.6	0.0
indicvoices_ori_pausecut	66	0	smart_turn_v3.2_gpu	84.8 [77.3, 93.9]	n/a	100.0	84.8	0.0
indicvoices_ori_pausecut	66	0	tamil_base_int8	69.7 [59.1, 80.3]	n/a	100.0	69.7	0.0
indicvoices_ori_pausecut	66	0	base_fp32	83.3 [74.2, 90.9]	n/a	100.0	83.3	0.0
indicvoices_ori_pausecut	66	0	base_int8	83.3 [74.2, 92.4]	n/a	100.0	83.3	0.0
indicvoices_ori_pausecut	66	0	tiny_fp32	74.2 [63.6, 83.3]	n/a	100.0	74.2	0.0
indicvoices_ori_pausecut	66	0	tiny_int8	77.3 [66.7, 86.4]	n/a	100.0	77.3	0.0
indicvoices_pan	444	89	smart_turn_v3.2_cpu	70.3 [66.0, 74.1]	0.672 [0.583, 0.756]	18.8	51.0	72.7
indicvoices_pan	444	89	smart_turn_v3.2_gpu	48.0 [43.5, 52.5]	0.689 [0.606, 0.768]	14.5	75.5	44.6
indicvoices_pan	444	89	tamil_base_int8	74.1 [70.3, 77.9]	0.722 [0.635, 0.797]	23.0	57.1	76.2
indicvoices_pan	444	89	base_fp32	86.7 [83.3, 89.9]	0.855 [0.793, 0.912]	43.2	65.3	89.4
indicvoices_pan	444	89	base_int8	76.1 [72.1, 80.2]	0.764 [0.675, 0.843]	25.2	59.2	78.2
indicvoices_pan	444	89	tiny_fp32	84.0 [80.6, 87.4]	0.792 [0.708, 0.868]	36.2	59.2	87.1
indicvoices_pan	444	89	tiny_int8	82.4 [78.6, 85.8]	0.756 [0.652, 0.846]	34.4	65.3	84.6
indicvoices_pan_pausecut	112	0	smart_turn_v3.2_cpu	65.2 [56.2, 73.2]	n/a	100.0	65.2	0.0
indicvoices_pan_pausecut	112	0	smart_turn_v3.2_gpu	91.1 [85.7, 95.5]	n/a	100.0	91.1	0.0
indicvoices_pan_pausecut	112	0	tamil_base_int8	69.6 [60.7, 78.6]	n/a	100.0	69.6	0.0
indicvoices_pan_pausecut	112	0	base_fp32	79.5 [71.4, 86.6]	n/a	100.0	79.5	0.0
indicvoices_pan_pausecut	112	0	base_int8	78.6 [70.5, 85.7]	n/a	100.0	78.6	0.0
indicvoices_pan_pausecut	112	0	tiny_fp32	79.5 [72.3, 86.6]	n/a	100.0	79.5	0.0
indicvoices_pan_pausecut	112	0	tiny_int8	71.4 [62.5, 79.5]	n/a	100.0	71.4	0.0
indicvoices_tam	269	94	smart_turn_v3.2_cpu	72.5 [66.5, 77.7]	0.698 [0.507, 0.855]	11.7	60.0	73.2
indicvoices_tam	269	94	smart_turn_v3.2_gpu	42.0 [36.1, 48.3]	0.739 [0.529, 0.899]	7.3	80.0	39.8
indicvoices_tam	269	94	tamil_base_int8	87.4 [83.3, 91.1]	0.788 [0.646, 0.920]	22.9	53.3	89.4
indicvoices_tam	269	94	base_fp32	91.1 [87.4, 94.4]	0.901 [0.804, 0.981]	35.5	73.3	92.1
indicvoices_tam	269	94	base_int8	89.2 [85.5, 92.6]	0.886 [0.755, 0.971]	31.6	80.0	89.8
indicvoices_tam	269	94	tiny_fp32	90.0 [86.2, 93.3]	0.912 [0.840, 0.965]	31.2	66.7	91.3
indicvoices_tam	269	94	tiny_int8	87.0 [82.9, 91.1]	0.836 [0.699, 0.943]	25.0	66.7	88.2
indicvoices_tam_pausecut	31	0	smart_turn_v3.2_cpu	71.0 [54.8, 87.1]	n/a	100.0	71.0	0.0
indicvoices_tam_pausecut	31	0	smart_turn_v3.2_gpu	93.5 [83.9, 100.0]	n/a	100.0	93.5	0.0
indicvoices_tam_pausecut	31	0	tamil_base_int8	71.0 [54.8, 87.1]	n/a	100.0	71.0	0.0
indicvoices_tam_pausecut	31	0	base_fp32	83.9 [71.0, 96.8]	n/a	100.0	83.9	0.0
indicvoices_tam_pausecut	31	0	base_int8	80.6 [64.5, 93.5]	n/a	100.0	80.6	0.0
indicvoices_tam_pausecut	31	0	tiny_fp32	71.0 [54.8, 87.1]	n/a	100.0	71.0	0.0
indicvoices_tam_pausecut	31	0	tiny_int8	77.4 [61.3, 90.3]	n/a	100.0	77.4	0.0
indicvoices_tel	567	88	smart_turn_v3.2_cpu	75.7 [72.1, 79.0]	0.651 [0.572, 0.725]	22.7	45.5	79.6
indicvoices_tel	567	88	smart_turn_v3.2_gpu	50.1 [45.7, 54.3]	0.692 [0.620, 0.763]	16.0	77.3	46.5
indicvoices_tel	567	88	tamil_base_int8	80.8 [77.1, 83.8]	0.778 [0.704, 0.839]	32.2	59.1	83.6
indicvoices_tel	567	88	base_fp32	86.6 [84.0, 89.2]	0.859 [0.798, 0.913]	44.8	65.2	89.4
indicvoices_tel	567	88	base_int8	84.1 [81.0, 86.9]	0.794 [0.729, 0.852]	37.2	53.0	88.2
indicvoices_tel	567	88	tiny_fp32	86.4 [83.6, 89.2]	0.847 [0.788, 0.903]	43.7	57.6	90.2
indicvoices_tel	567	88	tiny_int8	85.2 [82.0, 87.7]	0.832 [0.776, 0.892]	41.0	62.1	88.2
indicvoices_tel_pausecut	158	0	smart_turn_v3.2_cpu	65.2 [57.6, 72.8]	n/a	100.0	65.2	0.0
indicvoices_tel_pausecut	158	0	smart_turn_v3.2_gpu	86.7 [81.0, 91.8]	n/a	100.0	86.7	0.0
indicvoices_tel_pausecut	158	0	tamil_base_int8	80.4 [74.1, 86.1]	n/a	100.0	80.4	0.0
indicvoices_tel_pausecut	158	0	base_fp32	82.3 [75.9, 88.0]	n/a	100.0	82.3	0.0
indicvoices_tel_pausecut	158	0	base_int8	71.5 [64.5, 78.5]	n/a	100.0	71.5	0.0
indicvoices_tel_pausecut	158	0	tiny_fp32	79.7 [73.4, 86.1]	n/a	100.0	79.7	0.0
indicvoices_tel_pausecut	158	0	tiny_int8	84.8 [79.1, 89.9]	n/a	100.0	84.8	0.0
smart-turn-data-v3.2-test:train	10878	50	smart_turn_v3.2_cpu	90.6 [90.0, 91.1]	0.968 [0.965, 0.970]	93.7	87.3	94.0
smart-turn-data-v3.2-test:train	10878	50	smart_turn_v3.2_gpu	93.1 [92.6, 93.6]	0.982 [0.979, 0.984]	93.7	92.6	93.7
smart-turn-data-v3.2-test:train	10878	50	tamil_base_int8	72.9 [72.1, 73.7]	0.810 [0.801, 0.818]	90.3	52.0	94.3
smart-turn-data-v3.2-test:train	10878	50	base_fp32	93.6 [93.2, 94.1]	0.983 [0.981, 0.985]	93.9	93.5	93.8
smart-turn-data-v3.2-test:train	10878	50	base_int8	85.7 [85.1, 86.3]	0.956 [0.953, 0.960]	80.8	93.9	77.3
smart-turn-data-v3.2-test:train	10878	50	tiny_fp32	92.3 [91.8, 92.7]	0.976 [0.973, 0.978]	93.0	91.6	92.9
smart-turn-data-v3.2-test:train	10878	50	tiny_int8	90.2 [89.7, 90.8]	0.966 [0.963, 0.969]	90.4	90.2	90.2
tamil-eot:test	4168	63	smart_turn_v3.2_cpu	70.4 [68.9, 71.7]	0.743 [0.728, 0.759]	59.8	60.4	76.2
tamil-eot:test	4168	63	smart_turn_v3.2_gpu	59.4 [58.1, 60.8]	0.743 [0.726, 0.759]	47.1	81.1	46.7
tamil-eot:test	4168	63	tamil_base_int8	85.8 [84.7, 86.9]	0.921 [0.912, 0.930]	85.0	74.9	92.2
tamil-eot:test	4168	63	base_fp32	85.9 [84.9, 87.0]	0.917 [0.909, 0.927]	83.8	76.5	91.4
tamil-eot:test	4168	63	base_int8	80.8 [79.6, 81.9]	0.876 [0.864, 0.887]	74.1	73.7	84.9
tamil-eot:test	4168	63	tiny_fp32	83.0 [81.9, 84.1]	0.885 [0.874, 0.895]	79.8	72.4	89.3
tamil-eot:test	4168	63	tiny_int8	81.8 [80.6, 83.0]	0.875 [0.862, 0.886]	77.3	71.9	87.7

Overall

model	acc	AUC
smart_turn_v3.2_cpu	81.7	0.894
smart_turn_v3.2_gpu	76.5	0.901
tamil_base_int8	76.5	0.823
base_fp32	90.1	0.955
base_int8	83.5	0.923
tiny_fp32	88.3	0.938
tiny_int8	86.7	0.930