karna-large / eval /stock_1b.json
Darayut's picture
Add v1 evaluation results (student, fine-tuned teacher, stock 1B, inference time)
f370090 verified
Raw History Blame Contribute Delete
10.5 kB
{
"ckpt": "/run/media/pc/disk1/ml-data/models/pretrained/omniASR-CTC-1B/omniASR-CTC-1B.omniprune.pt",
"params_total": 975065300,
"params_encoder": 962496128.0,
"gmacs_per_audio_second": 50.277134336,
"dtype": "bf16",
"results": {
"ddd_dev": {
"cer": 0.07939672561805132,
"n": 400,
"rtf": 0.006409313296584102,
"examples": [
{
"ref": "αž˜αŸ’αž…αžΆαžŸαŸ‹αž’αžΆαž•αžΆαžαž˜αž·αž“αžαŸ‚αž„αžαŸ‚αž‡αŸ’αžšαžΎαžŸαžšαžΎαžŸαž‚αŸ’αžšαžΏαž„αžŸαž„αŸ’αž αžΆαžšαž·αž˜αžŠαŸ‚αž›αž˜αžΆαž“αž‚αž»αžŽαž—αžΆαž–αžαŸ’αž–αžŸαŸ‹ αž“αž·αž„αž˜αžΆαž“αžŸαŸ„αž—αŸαžŽαž—αžΆαž–αžŸαŸ’αžšαžŸαŸ‹αžŸαŸ’αž’αžΆαžαž”αŸ†αž•αž»αž",
"hyp": "αž˜αŸ’αž…αžΆαžŸαŸ‹ αž’αžΆαž•αžΆαžαž˜αž·αž“ αžαŸ‚αž„αžαŸ‚ αž‡αŸ’αžšαžΎαžŸαžšαžΎαžŸ αž‚αŸ’αžšαžΏαž„ αžŸαž„αŸ’αž αžΆαžšαžΉαž˜ αžŠαŸ‚αž› αž˜αžΆαž“ αž‚αž»αžŽαž—αžΆαž– αžαŸ’αž–αžŸαŸ‹ αž“αž·αž„ αž˜αžΆαž“ αžŸαŸ„αž—αŸαžŽαŸ’αž—αžΆαž– αžŸαŸ’αžšαžŸαŸ‹ αžŸαŸ’αž’αžΆαž αž”αŸ†αž•αž»αž"
},
{
"ref": "αžŸαŸ’αžαžΆαž”αžαŸ’αž™αž€αž˜αŸ’αž˜αž’αžΆαžŽαžΆαž“αž·αž‚αž˜αž”αžΆαžšαžΆαŸ†αž„αžŠαŸ‚αž›αž™αžΎαž„αžƒαžΎαž‰αž˜αžΆαž“αž“αŸ…αž—αŸ’αž“αŸ†αž–αŸαž‰ αž“αž·αž„αž”αžΆαžαŸ‹αžŠαŸ†αž”αž„ αž˜αžΆαž“αž›αž€αŸ’αžαžŽαŸˆαž–αž·αžŸαŸαžŸαž˜αž½αž™αž…αŸ†αž“αž½αž“αžŠαŸ‚αž›αž’αŸ’αžœαžΎαž±αŸ’αž™αžœαžΆαž›αŸαž…αž’αŸ’αž›αŸ„",
"hyp": "αžŸαŸ’αžαžΆαž”αžšαž·αž™αž€αž˜αŸ’αž˜ αž’αžΆαžŽαžΆαž“αž·αž‚αž˜ αž”αžΆαžšαžΆαŸ†αž„ αžŠαŸ‚αž› αž™αžΎαž„ αžƒαžΎαž‰ αž˜αžΆαž“ αž“αŸ… αž—αŸ’αž“αŸ†αž–αŸαž‰ αž“αž·αž„ αž”αžΆαžαŸ‹αžŠαŸ†αž”αž„ αž˜αžΆαž“ αž›αž€αŸ’αžαžŽαŸˆ αž–αž·αžŸαŸαžŸ αž˜αž½αž™ αž…αŸ†αž“αž½αž“ αžŠαŸ‚αž› αž’αŸ’αžœαžΎ αž²αŸ’αž™ វអ αž›αŸαž… αž’αŸ’αž›αŸ„αžš"
},
{
"ref": "αž–αž½αž€αž‚αŸαž€αŸαž˜αžΆαž“αž‡αŸ†αž“αžΏαž›αžΎαžœαž·αž‰αŸ’αž‰αžΆαžŽαž’αž˜αŸ’αž˜αž‡αžΆαžαž· αž“αž·αž„αž˜αžΆαž“αž–αž·αž’αžΈαž”αžΌαž‡αžΆαžŠαŸ†αžšαžΈ αž“αž·αž„αžšαž”αžšαž…αž·αž‰αŸ’αž…αžΉαž˜αžŠαŸ†αžšαžΈ αž€αžΆαžšαž’αŸ’αžœαžΎαž…αž˜αŸ’αž€αžΆαžšαž€αžΆαž αŸ’αžœαŸ αž“αž·αž„αžŠαŸ†αžŽαžΆαŸ†αž•αŸ’αžŸαŸαž„ αž€αŸαž‡αžΆαž•αŸ’αž“αŸ‚αž€αž˜αž½αž™αž“αŸƒαž‡αžΈαžœαž—αžΆαž–αžšαž”αžŸαŸ‹αž–αž½αž€αž‚αŸαžŠαŸ‚αžš",
"hyp": "αž–αž½αž€αž‚αŸ αž€αŸ αž˜αžΆαž“ αž‡αŸ†αž“αžΏ αž›αžΎ αžœαž·αž‰αŸ’αž‰αžΆαžŽ αž’αž˜αŸ’αž˜αž‡αžΆαžαž· αž“αž·αž„ αž˜αžΆαž“ αž–αž·αž’αžΈ αž”αžΌ αž‡αžΆαžαž˜αŸ’αžšαžΈ αž“αž·αž„ αžšαž”αžš αž…αž·αž‰αŸ’αž…αžΉαž˜ αžαž˜αŸ’αžšαžΈ αž€αžΆαžš αž’αŸ’αžœαžΎ αž…αŸ†αž€αžΆαžš αž€αžΆαž αŸ’αžœαŸ αž“αž·αž„ αžαŸ†αžŽαžΆαŸ† αž•αŸ’αžŸαŸ’αžŸαŸαž„ αž€αŸ αž‡αžΆ αž•αŸ’αž“αŸ‚αž€ αž˜αž½αž™ αž“αŸƒ αž‡αžΈαžœαž—αžΆαž– αžšαž”αžŸαŸ‹ αž–αž½αž€αž‚αŸ αžŠαŸ‚αžš"
}
]
},
"ddd_test": {
"cer": 0.05946630924820275,
"n": 1993,
"rtf": 0.0060830656866102135,
"examples": [
{
"ref": "αž›αŸ’αž–αŸ…αžŸαž˜αŸ’αž”αžΌαžšαž‡αžΆαžαž·αžŸαžšαžŸαŸƒ αžŠαŸ‚αž›αž‡αž½αž™αž‡αŸ†αžšαž»αž‰αž…αž›αž“αžΆαž–αŸ„αŸ‡αžœαŸ€αž“αž²αŸ’αž™αž˜αžΆαž“αž›αž€αŸ’αžαžŽαŸˆαž’αž˜αŸ’αž˜αžαžΆ αž€αžΆαžšαž–αžΆαžšαž€αžΆαžšαž‘αž›αŸ‹αž›αžΆαž˜αž€ αž“αž·αž„αžšαž€αŸ’αžŸαžΆαž”αŸ’αžšαž–αŸαž“αŸ’αž’αžšαŸ†αž›αžΆαž™αž’αžΆαž αžΆαžšαž²αŸ’αž™αž˜αžΆαž“αžŸαž»αžαž—αžΆαž–αž›αŸ’αž’",
"hyp": "αž›αŸ’αž–αŸ… αžŸαŸ†αŸ’αž”αžΌαžš αž‡αžΆαžαž· αžŸαžšαžŸαŸƒ αžŠαŸ‚αž› αž‡αž½αž™ αž‡αŸ†αžšαž½αž‰ αž…αž›αžŽαžΆ αž–αžœαŸ€αž“ αž±αŸ’αž™ αž˜αžΆαž“ αž›αž€αŸ’αžαžŽαŸˆ αž’αž˜αŸ’αž˜αžαžΆ αž€αžΆαž›αž–αžΆαžš αž€αžΆαžšαž‘αž»αŸ‹αž›αžΈαž˜αž½αž αž“αž·αž„ αžšαž€αŸ’αžŸαžΆ αž”αŸ’αžšαž–αŸαž“αŸ’αž’ αžšαŸ†αž›αžΆαž€ ធអហអរ αž±αŸ’αž™ αž˜αžΆαž“ αžŸαž»αžαž—αžΆαž– αž›αŸ’αž’"
},
{
"ref": "αž€αžΆαžšαž”αž“αŸ’αžαŸ‚αž˜ αžαŸ’αž‘αž·αŸ‡αžŠαžΌαž„αž‡αž½αž™αž²αŸ’αž™αžŸαž„αŸ‹αžαŸ’αž™αžΆαž˜αžΆαž“αžŸαž—αžΆαž–αž‘αž“αŸ‹αž›αŸ’αž˜αžΎαž™ αž αžΎαž™αžŸαŸ’αž›αžΉαž€αžαžΎαž™αž€αŸαž‡αž½αž™αž”αž“αŸ’αžαŸ‚αž˜αž“αžΌαžœαž€αŸ’αž›αž·αž“αž€αŸ’αžšαž’αžΌαž”αž”αŸ’αžšαž αžΎαžšαž•αž„αžŠαŸ‚αžš",
"hyp": "αž€αžΆαžšαž”αž“αŸ’αžαŸ‚αž˜ αžαŸ’αž‘αž·αŸ‡αžŠαžΌαž„ αž‡αž½αž™ αž±αŸ’αž™ αžŸαž„αŸ‹αžαŸ’αž…αžΆαžšαŸ’αž™ αž˜αžΆαž“ αž—αžΆαž– αž‘αž“αŸ‹ αž›αŸ’αž˜αžΎαž™ αž αžΎαž™ αžŸαŸ’αž›αžΉαž€αžαžΎαž™ αž€αŸ αž‡αž½αž™ αž”αž“αŸ’αžαŸ‚αž˜ αž“αžΌαžœ αž€αŸ’αž›αž·αž“ αž€αŸ’αžšαž’αžΌαž” αž”αŸ’αžšαž αžΎαžš αž•αž„ αžŠαŸ‚αžš"
},
{
"ref": "αž‘αžΉαž€αž›αŸ’αž–αŸ…αž”αžΆαž“αž€αŸ’αž›αžΆαž™αž‡αžΆαž—αŸαžŸαž‡αŸ’αž‡αŸˆαžŠαŸαž–αŸαž‰αž“αž·αž™αž˜αž˜αž½αž™ αž€αŸ’αž“αž»αž„αž…αŸ†αžŽαŸ„αž˜αž’αŸ’αž“αž€αžŠαŸ‚αž›αž™αž€αž…αž·αžαŸ’αžαž‘αž»αž€αžŠαžΆαž€αŸ‹αž›αžΎαžŸαž»αžαž—αžΆαž– αž“αž·αž„αžŸαŸ’αžœαŸ‚αž„αžšαž€αž—αŸαžŸαž‡αŸ’αž‡αŸˆαžŠαŸ‚αž›αž˜αžΆαž“αž‚αž»αžŽαž”αŸ’αžšαž™αŸ„αž‡αž“αŸαžαŸ’αž–αžŸαŸ‹",
"hyp": "αž‘αžΉαž€ αž›αŸ’αž–αŸ… αž”αžΆαž“ αž€αŸ’αž›αžΆαž™ αž‡αžΆ αž—αŸαžŸαž‡αŸ’αž‡αŸˆ ដ៏ αž–αŸ αž“αž·αž™αž˜ αž˜αž½αž™ αž€αŸ’αž“αž»αž„ αž…αŸ†αžŽαŸ„αž˜ αž’αŸ’αž“αž€ αžŠαŸ‚αž› αž™αž€ αž…αž·αžαŸ’αž αž‘αž»αž€αžŠαžΆαž€αŸ‹ αž›αžΎ αžŸαž»αžαž—αžΆαž– αž“αž·αž„ αžŸαŸ’αžœαŸ‚αž„αžšαž€ αž—αŸαžŸαž‡αŸ’αž‡αŸˆ αžŠαŸ‚αž› αž˜αžΆαž“ αž‚αž»αžŽ αž”αŸ’αžšαž™αŸ„αž‡αž“αŸ αžαŸ’αž–αžŸαŸ‹"
}
]
},
"fleurs_km_test": {
"cer": 0.11262976629766297,
"n": 771,
"rtf": 0.006777807880311398,
"examples": [
{
"ref": "αž˜αž“αž»αžŸαŸ’αžŸαž˜αž½αž™αž…αŸ†αž“αž½αž“αž‚αž·αžαžαžΆ αž‚αžΆαžαŸ‹αž“αž·αž™αžΆαž™αžαŸ’αžšαžΌαžœ αž”αŸ‰αž»αž“αŸ’αžαŸ‚αž˜αž“αž»αžŸαŸ’αžŸαž‡αžΆαž…αŸ’αžšαžΎαž“αž‡αžΏαžαžΆαž‚αžΆαžαŸ‹αž“αž·αž™αžΆαž™αž˜αž·αž“αžαŸ’αžšαžΌαžœαž‘αŸ αž‚αžΆαžαŸ‹αžαžΆαž”αŸ’αžšαž–αŸαž“αŸ’αž’αž–αŸ’αžšαŸ‡αž’αžΆαž‘αž·αžαŸ’αž™ αž’αŸ’αžœαžΎαžŠαŸ†αžŽαžΎαžšαžœαž·αž›αž‡αž»αŸ†αžœαž·αž‰αž•αŸ‚αž“αžŠαžΈ αžšαž½αž˜αž‘αžΆαŸ†αž„αž–αŸ’αžšαŸ‡αž’αžΆαž‘αž·αžαŸ’αž™ αž“αž·αž„αž•αŸ’αž€αžΆαž™αžŠαž‘αŸƒαž‘αŸ€αžαž•αž„αžŠαŸ‚αžš",
"hyp": "αž˜αž“αž»αžŸαŸ’αžŸαž˜αž½αž™αž…αŸ†αž“αž½αž“αž‚αž·αžαžαžΆαž‚αžΆαžαŸ‹αž“αž·αž™αžΆαž™αžαŸ’αžšαžΌαžœ αž”αŸ‰αž»αž“αŸ’αžαŸ‚αž˜αž“αž»αžŸαŸ’αžŸαž‡αžΆαž…αŸ’αžšαžΎαž“αž‡αžΏαžαžΆαž‚αžΆαžαŸ‹αž“αž·αž™αžΆαž™αž˜αž·αž“αžαŸ’αžšαžΌαžœαž‘αŸ αž‚αžΆαžαŸ‹αžαžΆαž”αŸ’αžšαž–αŸαž“αŸ’αž’αž–αŸ’αžšαŸ‡αž’αžΆαž‘αž·αžαŸ’αž™αž’αŸ’αžœαžΎαžŠαŸ†αžŽαžΎαžšαžœαž·αž›αž‡αŸ…αžœαž·αž‰αž•αŸ‚αž“αžŠαžΈαžšαž½αž˜αž‘αžΆαŸ†αž„αž–αŸ’αžšαŸ‡αž’αžΆαž‘αž·αžαŸ’αž™αž“αž·αž„αž•αŸ’αž€αžΆαž™αžŠαŸαž’αž‘αŸƒαž‘αŸ€αžαž•αž„αžŠαŸ‚αžš"
},
{
"ref": "αžŸαŸ†αžŽαžΎαž˜αž“αŸ…αž›αžΎαžŠαŸƒαžšαž”αžŸαŸ‹αž’αŸ’αž“αž€αž“αžΉαž„αž˜αžΆαž“αž”αŸ’αžšαžαž·αž€αž˜αŸ’αž˜αž‡αžΆαž˜αž½αž™αž“αžΉαž„αžŸαŸ’αžšαž‘αžΆαž”αŸ‹αžαžΆαž„αž€αŸ’αžšαŸ… αžŠαŸ‚αž›αž“αžΉαž„αž’αžΆαž…αžŠαžΉαž„αž‡αžΆαž˜αž»αž“αž”αžΆαž“ αžšαž½αž…αž”αž„αŸ’αž€αžΎαžαž‡αžΆαžŸαŸ†αž”αž€αž˜αž½αž™αž”αŸ’αžšαž—αŸαž‘",
"hyp": "αžŸαŸ†αžŽαžΎαž˜ αž“αŸ…αž›αžΎαžŠαŸƒαžšαž”αžŸαŸ‹αž’αŸ’αž“αž€αž“αžΉαž„αž˜αžΆαž“αž”αŸ’αžšαžΉαžαŸ’αžαž·αž€αž˜αŸ’αž˜αž‡αžΆαž˜αž½αž™αž“αžΉαž„αžŸαŸ’αžšαž‘αŸαž–αžαžΆαž„αž€αŸ’αžšαŸ… αžŠαŸ‚αž›αž“αžΉαž„αž’αžΆαž…αžŠαžΉαž„αž‡αžΆαž˜αž»αž“αž”αžΆαž“ αžšαž½αž…αž”αž„αŸ’αž€αžΎαžαž‡αžΆαžŸαŸ†αž”αž€αž˜αž½αž™αž”αŸ’αžšαž—αŸαž‘"
},
{
"ref": "αž‘αžΈαž€αŸ’αžšαž»αž„αž“αŸαŸ‡αž€αŸαž‡αžΆαž˜αžΌαž›αžŠαŸ’αž‹αžΆαž“αžŠαžΎαž˜αŸ’αž”αžΈαž‘αžΎαž„αž—αŸ’αž“αŸ†αž—αŸ’αž›αžΎαž„ yiragongo αžšαž½αž˜αž‡αžΆαž˜αž½αž™αž“αžΉαž„αž€αžΆαžšαžαžΆαž˜ountain orilla αžŠαŸ‚αž›αž˜αžΆαž“αžαž˜αŸ’αž›αŸƒαžαŸ„αž€αž”αŸ†αž•αž»αžαž“αŸ…αž’αžΆαž αŸ’αžœαŸ’αžšαž·αž€",
"hyp": "αž‘αžΈαž€αŸ’αžšαž»αž„αž“αŸαŸ‡αž€αŸαž‡αžΆαž˜αžΌαž›αžŠαŸ’αž‹αžΆαž“αžŠαžΎαž˜αŸ’αž”αžΈαž‘αžΎαž„αž—αŸ’αž“αŸ†αž—αŸ’αž›αžΎαž„ αž“αžΈαžšαŸ‰αžΆαŸ’αž€oαž“αž€αžΌ αžšαž½αž˜αž‡αžΆαž˜αž½αž™αž“αžΉαž„αž€αžΆαžšαžαžΆαž˜αž˜αŸ‰αŸ„αž“tαŸ‚αž“ αž€αžΌαžšαžΈαž‘αžΆ αžŠαŸ‚αž›αž˜αžΆαž“αžαž˜αŸ’αž›αŸƒαžαž”αž”αŸ†αž•αž»αžαž“αŸ…αž’αžΆαž αž αŸ’αžœαŸ’αžšαž·αž€"
}
]
},
"libri_dev_clean": {
"cer": 0.015860143522683594,
"n": 2703,
"wer": 0.04521892577478769,
"rtf": 0.006160100618348849,
"examples": [
{
"ref": "mister quilter is the apostle of the middle classes and we are glad to welcome his gospel",
"hyp": "mr quilter is the apostle of the middle classes and we are glad to welcome his gospel"
},
{
"ref": "nor is mister quilter's manner less interesting than his matter",
"hyp": "nor is mr quilter's manner less interesting than his matter"
},
{
"ref": "he tells us that at this festive season of the year with christmas and roast beef looming before us similes drawn from eating and its results occur most readily to the mind",
"hyp": "he tells us that at this festive season of the year with christmas and rose beef looming before us similes drawn from eating and its results occur most readily to the mind"
}
]
},
"libri_test_clean": {
"cer": 0.01636625873370931,
"n": 2620,
"wer": 0.04397443700547778,
"rtf": 0.006210090116807237,
"examples": [
{
"ref": "he hoped there would be stew for dinner turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick peppered flour fattened sauce",
"hyp": "he hoped there would be stew for dinner turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick peppered flower-fattened sauce"
},
{
"ref": "stuff it into you his belly counselled him",
"hyp": "stuff it into you his belly counselled him"
},
{
"ref": "after early nightfall the yellow lamps would light up here and there the squalid quarter of the brothels",
"hyp": "after early nightfall the yellow lamps would light up here and there the squalid quarter of the brofles"
}
]
},
"libri_test_other": {
"cer": 0.036180234067892225,
"n": 2939,
"wer": 0.08870336052576276,
"rtf": 0.006097428758073933,
"examples": [
{
"ref": "there's iron they say in all our blood and a grain or two perhaps is good but his he makes me harshly feel has got a little too much of steel anon",
"hyp": "there's iron they say in all our blood and a grain or two perhaps is good but his he makes me harshly feel has got a little too much of steel anon"
},
{
"ref": "margaret said mister hale as he returned from showing his guest downstairs i could not help watching your face with some anxiety when mister thornton made his confession of having been a shop boy",
"hyp": "margaret said mr hal as he returned from showing his guest downstairs i could not help watching your face with some anxiety when mr thounton made his confession of having been a shop-boy"
},
{
"ref": "you don't mean that you thought me so silly",
"hyp": "you don't mean that you thought me so silly"
}
]
}
}
}