Publish audited public-eight Open ASR H200 results and training disclosure

Browse files

Files changed (6) hide show

.eval_results/open_asr_leaderboard.yaml +145 -0
README.md +29 -1
evaluation/open_asr_20260918/official-scorer-output.txt +34 -0
evaluation/open_asr_20260918/public-eight-h200-results.json +1530 -0
evaluation/open_asr_20260918/public-eight-h200-results.md +16 -0
evaluation/open_asr_20260918/training-disclosure.md +15 -0

.eval_results/open_asr_leaderboard.yaml ADDED Viewed

	@@ -0,0 +1,145 @@

+# Monsoon has no registered Hub task; its measured result is retained in the companion JSON.
+- dataset:
+    id: hf-audio/open-asr-leaderboard
+    task_id: mean_wer
+    revision: b6bdcd0beb34f8975dc659796176d88f43aff502
+  value: 4.49
+  date: '2026-09-18'
+  source:
+    url: https://huggingface.co/oruk/orukeet/blob/main/evaluation/open_asr_20260918/public-eight-h200-results.json
+    name: Orukeet H200 evaluation receipts
+    user: oruk
+  notes: Public English eight only, including Monsoon; no private-set result or official
+    acceptance claimed. NVIDIA H200, torch.bfloat16; raw-output scoring at evaluator
+    48e6f0d75ad7dad68c962112356fec44776f3bbb. LibriSpeech test-other was used for
+    r3 training and selection. Monsoon English informed prior training-partition selection.
+    Remaining inherited exposure is incompletely verified. Equal-weight mean of eight
+    two-decimal WER values.
+- dataset:
+    id: hf-audio/open-asr-leaderboard
+    task_id: rtfx
+    revision: b6bdcd0beb34f8975dc659796176d88f43aff502
+  value: 5545.68
+  date: '2026-09-18'
+  source:
+    url: https://huggingface.co/oruk/orukeet/blob/main/evaluation/open_asr_20260918/public-eight-h200-results.json
+    name: Orukeet H200 evaluation receipts
+    user: oruk
+  notes: Public English eight only, including Monsoon; no private-set result or official
+    acceptance claimed. NVIDIA H200, torch.bfloat16; raw-output scoring at evaluator
+    48e6f0d75ad7dad68c962112356fec44776f3bbb. LibriSpeech test-other was used for
+    r3 training and selection. Monsoon English informed prior training-partition selection.
+    Remaining inherited exposure is incompletely verified. Sum of measured eligible
+    audio seconds / sum of measured transcribe seconds; warmup excluded.
+- dataset:
+    id: hf-audio/open-asr-leaderboard
+    task_id: ami_wer
+    revision: b6bdcd0beb34f8975dc659796176d88f43aff502
+  value: 8.68
+  date: '2026-09-18'
+  source:
+    url: https://huggingface.co/oruk/orukeet/blob/main/evaluation/open_asr_20260918/public-eight-h200-results.json
+    name: Orukeet H200 evaluation receipts
+    user: oruk
+  notes: 'Public English eight only, including Monsoon; no private-set result or official
+    acceptance claimed. NVIDIA H200, torch.bfloat16; raw-output scoring at evaluator
+    48e6f0d75ad7dad68c962112356fec44776f3bbb. LibriSpeech test-other was used for
+    r3 training and selection. Monsoon English informed prior training-partition selection.
+    Remaining inherited exposure is incompletely verified. Exact dataset key: ami_cleaned;
+    see companion JSON for pins and coverage.'
+- dataset:
+    id: hf-audio/open-asr-leaderboard
+    task_id: earnings22_wer
+    revision: b6bdcd0beb34f8975dc659796176d88f43aff502
+  value: 5.74
+  date: '2026-09-18'
+  source:
+    url: https://huggingface.co/oruk/orukeet/blob/main/evaluation/open_asr_20260918/public-eight-h200-results.json
+    name: Orukeet H200 evaluation receipts
+    user: oruk
+  notes: 'Public English eight only, including Monsoon; no private-set result or official
+    acceptance claimed. NVIDIA H200, torch.bfloat16; raw-output scoring at evaluator
+    48e6f0d75ad7dad68c962112356fec44776f3bbb. LibriSpeech test-other was used for
+    r3 training and selection. Monsoon English informed prior training-partition selection.
+    Remaining inherited exposure is incompletely verified. Exact dataset key: earnings22_cleaned_aa_chunked;
+    see companion JSON for pins and coverage.'
+- dataset:
+    id: hf-audio/open-asr-leaderboard
+    task_id: gigaspeech_wer
+    revision: b6bdcd0beb34f8975dc659796176d88f43aff502
+  value: 7.48
+  date: '2026-09-18'
+  source:
+    url: https://huggingface.co/oruk/orukeet/blob/main/evaluation/open_asr_20260918/public-eight-h200-results.json
+    name: Orukeet H200 evaluation receipts
+    user: oruk
+  notes: 'Public English eight only, including Monsoon; no private-set result or official
+    acceptance claimed. NVIDIA H200, torch.bfloat16; raw-output scoring at evaluator
+    48e6f0d75ad7dad68c962112356fec44776f3bbb. LibriSpeech test-other was used for
+    r3 training and selection. Monsoon English informed prior training-partition selection.
+    Remaining inherited exposure is incompletely verified. Exact dataset key: gigaspeech_cleaned;
+    see companion JSON for pins and coverage.'
+- dataset:
+    id: hf-audio/open-asr-leaderboard
+    task_id: librispeech_clean_wer
+    revision: b6bdcd0beb34f8975dc659796176d88f43aff502
+  value: 1.47
+  date: '2026-09-18'
+  source:
+    url: https://huggingface.co/oruk/orukeet/blob/main/evaluation/open_asr_20260918/public-eight-h200-results.json
+    name: Orukeet H200 evaluation receipts
+    user: oruk
+  notes: 'Public English eight only, including Monsoon; no private-set result or official
+    acceptance claimed. NVIDIA H200, torch.bfloat16; raw-output scoring at evaluator
+    48e6f0d75ad7dad68c962112356fec44776f3bbb. LibriSpeech test-other was used for
+    r3 training and selection. Monsoon English informed prior training-partition selection.
+    Remaining inherited exposure is incompletely verified. Exact dataset key: librispeech_test_clean;
+    see companion JSON for pins and coverage.'
+- dataset:
+    id: hf-audio/open-asr-leaderboard
+    task_id: librispeech_other_wer
+    revision: b6bdcd0beb34f8975dc659796176d88f43aff502
+  value: 2.84
+  date: '2026-09-18'
+  source:
+    url: https://huggingface.co/oruk/orukeet/blob/main/evaluation/open_asr_20260918/public-eight-h200-results.json
+    name: Orukeet H200 evaluation receipts
+    user: oruk
+  notes: 'Public English eight only, including Monsoon; no private-set result or official
+    acceptance claimed. NVIDIA H200, torch.bfloat16; raw-output scoring at evaluator
+    48e6f0d75ad7dad68c962112356fec44776f3bbb. LibriSpeech test-other was used for
+    r3 training and selection. Monsoon English informed prior training-partition selection.
+    Remaining inherited exposure is incompletely verified. Exact dataset key: librispeech_test_other;
+    see companion JSON for pins and coverage.'
+- dataset:
+    id: hf-audio/open-asr-leaderboard
+    task_id: spgispeech_wer
+    revision: b6bdcd0beb34f8975dc659796176d88f43aff502
+  value: 3.38
+  date: '2026-09-18'
+  source:
+    url: https://huggingface.co/oruk/orukeet/blob/main/evaluation/open_asr_20260918/public-eight-h200-results.json
+    name: Orukeet H200 evaluation receipts
+    user: oruk
+  notes: 'Public English eight only, including Monsoon; no private-set result or official
+    acceptance claimed. NVIDIA H200, torch.bfloat16; raw-output scoring at evaluator
+    48e6f0d75ad7dad68c962112356fec44776f3bbb. LibriSpeech test-other was used for
+    r3 training and selection. Monsoon English informed prior training-partition selection.
+    Remaining inherited exposure is incompletely verified. Exact dataset key: spgispeech;
+    see companion JSON for pins and coverage.'
+- dataset:
+    id: hf-audio/open-asr-leaderboard
+    task_id: voxpopuli_cleaned_aa_wer
+    revision: b6bdcd0beb34f8975dc659796176d88f43aff502
+  value: 2.46
+  date: '2026-09-18'
+  source:
+    url: https://huggingface.co/oruk/orukeet/blob/main/evaluation/open_asr_20260918/public-eight-h200-results.json
+    name: Orukeet H200 evaluation receipts
+    user: oruk
+  notes: 'Public English eight only, including Monsoon; no private-set result or official
+    acceptance claimed. NVIDIA H200, torch.bfloat16; raw-output scoring at evaluator
+    48e6f0d75ad7dad68c962112356fec44776f3bbb. LibriSpeech test-other was used for
+    r3 training and selection. Monsoon English informed prior training-partition selection.
+    Remaining inherited exposure is incompletely verified. Exact dataset key: voxpopuli_cleaned_aa;
+    see companion JSON for pins and coverage.'

README.md CHANGED Viewed

@@ -101,10 +101,38 @@ Both models decode identical recordings with NeMo greedy-batch TDT, FP32 weights
 | Accents/domains pooled, 47 splits | 12,006 | 16.72% | **15.25%** |
 | Accents/domains English, 20 splits | 5,120 | 9.51% | **8.84%** |
-Orukeet improves 25 of 27 complete LibriSpeech/FLEURS splits and 36 of 47 accent/domain splits, including all 20 English accent/domain splits. The accent/domain sample contains 256 recordings per split and all 230 Lesbos recordings; the preceding adaptation includes 6,118 sampled recordings. Read speech and accents/domains have separate pooled results. Every recording contributes to the scores.
 [All 74 paired WER/CER scores and edit counts](docs/current-checkpoint-benchmarks.md) · [Methods](docs/technical-report.md) · [Technical report](orukeet-technical-report.pdf)
 ## sherpa-onnx inference
 The [ONNX INT8 archive](https://huggingface.co/oruk/orukeet/resolve/55a984d46f68323301837194ce647c702f55facc/onnx/sherpa-onnx-orukeet-v0.1.0-int8.tar.bz2) uses the standard Parakeet TDT v3 layout: `encoder.int8.onnx`, `decoder.int8.onnx`, `joiner.int8.onnx` and `tokens.txt`. It also includes the BPE vocabulary, weight license and attribution. Gabor filters are ordinary convolution weights; the model uses sherpa-onnx's existing offline transducer loader.

 | Accents/domains pooled, 47 splits | 12,006 | 16.72% | **15.25%** |
 | Accents/domains English, 20 splits | 5,120 | 9.51% | **8.84%** |
+Orukeet improves 25 of 27 complete LibriSpeech/FLEURS splits and 36 of 47 accent/domain splits, including all 20 English accent/domain splits. The accent/domain sample contains 256 recordings per split and all 230 Lesbos recordings; the preceding FT-4035 continuation trained on 223,452 recordings across 24 complete selected partitions (371.47 hours); the 6,118 examples previously reported were a sampled follow-up fit diagnostic. Read speech and accents/domains have separate pooled results. Every recording contributes to the scores.
 [All 74 paired WER/CER scores and edit counts](docs/current-checkpoint-benchmarks.md) · [Methods](docs/technical-report.md) · [Technical report](orukeet-technical-report.pdf)
+<!-- orukeet-open-asr-20260918:start -->
+## Open ASR public English evaluation, September 18, 2026
+The released r3 checkpoint was evaluated on all eight current public English Open ASR datasets using the official NeMo H200 container. Mean WER is **4.49%**, with aggregate throughput **5545.68 RTFx**. These are self-reported public-set results; official leaderboard inclusion and private-set verification are pending.
+**Training exposure:** r3 trained on all 2,939 LibriSpeech test-other recordings for three passes and used that split for selection. Monsoon English also informed earlier training-partition selection. The aggregate includes these datasets and is not a held-out generalization estimate. [Training and benchmark exposure](https://huggingface.co/oruk/orukeet/blob/main/evaluation/open_asr_20260918/training-disclosure.md).
+| Dataset | WER (%) | H200 RTFx |
+|:--|--:|--:|
+| AMI-Cleaned | 8.68 | 3741.22 |
+| Earnings22-Cleaned-AA | 5.74 | 2982.17 |
+| GigaSpeech-Cleaned | 7.48 | 5346.49 |
+| LibriSpeech test-clean | 1.47 | 4180.11 |
+| LibriSpeech test-other (trained on) | 2.84 | 4159.42 |
+| SPGISpeech | 3.38 | 6419.21 |
+| Monsoon English (prior selection exposure) | 3.84 | 4835.62 |
+| VoxPopuli-Cleaned-AA | 2.46 | 2488.16 |
+| **Public-eight aggregate** | **4.49** | **5545.68** |
+The run uses H200, NeMo 2.7.2, Torch 2.8.0, BF16, batch size 128, and the same greedy-batch TDT decoder with `max_symbols=10` across all datasets. Timing covers the complete transcription pass after up to four warmup batches, with CUDA synchronization. RTFx is total evaluated audio divided by total measured transcription time; it excludes model/data acquisition and warmup. Batch size follows the existing Parakeet profile; exhaustive throughput optimization is not claimed.
+All 74,443 eligible inputs from 74,544 source records are retained. The official reference rule excludes 90 AMI and 11 GigaSpeech records; Earnings22's 341 chunks are merged into six full calls for scoring. The exact official normalizer and compound-aware `kaldialign==0.12.0` scorer are used. The mean averages the eight two-decimal dataset WERs. The earlier CPU FP32 comparison remains a separate run.
+[Complete metrics and execution hashes](https://huggingface.co/oruk/orukeet/blob/main/evaluation/open_asr_20260918/public-eight-h200-results.json) · [Pinned reproduction code](https://huggingface.co/spaces/oruk/open-asr-leaderboard-orukeet/tree/171ab5e79454cc3772a8700479cb150cfe4cc84f) · [Hub evaluation YAML](.eval_results/open_asr_leaderboard.yaml)
+Monsoon currently lacks a registered Hub metric task. Its score is retained in the complete report and eight-dataset aggregate; the Hub YAML contains the seven registered per-dataset entries.
+<!-- orukeet-open-asr-20260918:end -->
 ## sherpa-onnx inference
 The [ONNX INT8 archive](https://huggingface.co/oruk/orukeet/resolve/55a984d46f68323301837194ce647c702f55facc/onnx/sherpa-onnx-orukeet-v0.1.0-int8.tar.bz2) uses the standard Parakeet TDT v3 layout: `encoder.int8.onnx`, `decoder.int8.onnx`, `joiner.int8.onnx` and `tokens.txt`. It also includes the BPE vocabulary, weight license and attribution. Gabor filters are ordinary convolution weights; the model uses sherpa-onnx's existing offline transducer loader.

evaluation/open_asr_20260918/official-scorer-output.txt ADDED Viewed

	@@ -0,0 +1,34 @@

+Filtering models by id: oruk/orukeet
+********************************************************************************
+Results per dataset:
+********************************************************************************
+oruk/orukeet | ArtificialAnalysis-Earnings22-Cleaned-AA-chunked__test: WER = 5.74 %, RTFx = 2982.17
+oruk/orukeet | VoiceArena-Monsoon_en_IN_test__test: WER = 3.84 %, RTFx = 4835.62
+oruk/orukeet | hf-audio-open-asr-leaderboard_ami_cleaned_test: WER = 8.68 %, RTFx = 3741.22
+oruk/orukeet | hf-audio-open-asr-leaderboard_gigaspeech_cleaned_test: WER = 7.48 %, RTFx = 5346.49
+oruk/orukeet | hf-audio-open-asr-leaderboard_librispeech_test.clean: WER = 1.47 %, RTFx = 4180.11
+oruk/orukeet | hf-audio-open-asr-leaderboard_librispeech_test.other: WER = 2.84 %, RTFx = 4159.42
+oruk/orukeet | hf-audio-open-asr-leaderboard_spgispeech_test: WER = 3.38 %, RTFx = 6419.21
+oruk/orukeet | hf-audio-open-asr-leaderboard_voxpopuli_cleaned_aa_test: WER = 2.46 %, RTFx = 2488.16
+********************************************************************************
+Composite Results:
+********************************************************************************
+oruk/orukeet: WER = 4.49 %
+oruk/orukeet: RTFx = 5545.68
+********************************************************************************
+********************************************************************************
+CSV Summary (Public):
+********************************************************************************
+avg WER (oruk/orukeet) = 4.49
+model,RTFx,License,Size (B),# Languages,Encoder,Decoder,AMI-Cleaned WER,Earnings22-Cleaned-AA-chunked WER,Gigaspeech-Cleaned WER,LS Clean WER,LS Other WER,SPGISpeech WER,Voice-Arena-Moonsoon WER,Voxpopuli-Cleaned-AA WER,AMI-Cleaned RTFx,Earnings22-Cleaned-AA-chunked RTFx,Gigaspeech-Cleaned RTFx,LS Clean RTFx,LS Other RTFx,SPGISpeech RTFx,Voice-Arena-Moonsoon RTFx,Voxpopuli-Cleaned-AA RTFx
+oruk/orukeet,5545.68,,,,,,8.68,5.74,7.48,1.47,2.84,3.38,3.84,2.46,3741.2159,2982.1706,5346.4904,4180.1148,4159.4244,6419.2142,4835.625,2488.1643
+********************************************************************************
+********************************************************************************
+CSV Summary (Extra):
+********************************************************************************
+model,AMI WER,Earnings22 WER,Gigaspeech WER,Voxpopuli WER
+oruk/orukeet,,,,
+********************************************************************************

evaluation/open_asr_20260918/public-eight-h200-results.json ADDED Viewed

	@@ -0,0 +1,1530 @@

+{
+  "schema_version": 1,
+  "complete_public_eight_metrics": true,
+  "complete_per_dataset_hub_task_coverage": false,
+  "missing_registered_tasks": [
+    "voicearena_monsoon_en_in"
+  ],
+  "model": {
+    "model_id": "oruk/orukeet",
+    "parameter_count": 627008134,
+    "size_billions": 0.627008134,
+    "supported_language_count": 25,
+    "evaluated_languages": [
+      "en"
+    ],
+    "license": "cc-by-sa-4.0",
+    "encoder": "FastConformer",
+    "decoder": "Token-and-Duration Transducer (TDT)",
+    "checkpoint_file": "orukeet-v0.1.0.nemo",
+    "checkpoint_sha256": "031c8ddab4845aeced904a7cde8e8aa57993b2e344716cf83a545b079c473b56",
+    "parameter_scope": "total parameters, not trainable parameters only",
+    "known_exposure": [
+      "All 2939 LibriSpeech test-other recordings used for r3 training and selection.",
+      "Monsoon English informed earlier training-partition selection.",
+      "Other inherited training exposure is incompletely verified."
+    ]
+  },
+  "evaluation_date": "2026-09-18",
+  "aggregate": {
+    "mean_wer_percent": 4.49,
+    "mean_of_two_decimal_dataset_wer": 4.48625,
+    "rtfx": 5545.68,
+    "rtfx_unrounded": 5545.67879726494,
+    "audio_seconds": 588203.26,
+    "transcription_seconds": 106.06515117501839
+  },
+  "datasets": {
+    "ami_cleaned": {
+      "wer": 8.68,
+      "audio_length": 28727.5903125,
+      "inference_time": 7.678677501156925,
+      "rtfx": 3741.2159,
+      "ins": 1873,
+      "del": 2187,
+      "sub": 3085
+    },
+    "earnings22_cleaned_aa_chunked": {
+      "wer": 5.74,
+      "audio_length": 6900.9376250000005,
+      "inference_time": 2.3140653409063816,
+      "rtfx": 2982.1706,
+      "ins": 101,
+      "del": 237,
+      "sub": 545
+    },
+    "gigaspeech_cleaned": {
+      "wer": 7.48,
+      "audio_length": 126515.196,
+      "inference_time": 23.663223169045523,
+      "rtfx": 5346.4904,
+      "ins": 8004,
+      "del": 4624,
+      "sub": 17079
+    },
+    "librispeech_test_clean": {
+      "wer": 1.47,
+      "audio_length": 19452.480625,
+      "inference_time": 4.653575702803209,
+      "rtfx": 4180.1148,
+      "ins": 72,
+      "del": 65,
+      "sub": 640
+    },
+    "librispeech_test_other": {
+      "wer": 2.84,
+      "audio_length": 19229.570125,
+      "inference_time": 4.6231324940454215,
+      "rtfx": 4159.4244,
+      "ins": 132,
+      "del": 102,
+      "sub": 1265
+    },
+    "spgispeech": {
+      "wer": 3.38,
+      "audio_length": 360007.4995,
+      "inference_time": 56.082798800896846,
+      "rtfx": 6419.2142,
+      "ins": 17251,
+      "del": 2741,
+      "sub": 12728
+    },
+    "voicearena_monsoon_en_in": {
+      "wer": 3.84,
+      "audio_length": 20247.6074375,
+      "inference_time": 4.187174894148484,
+      "rtfx": 4835.625,
+      "ins": 481,
+      "del": 299,
+      "sub": 1095
+    },
+    "voxpopuli_cleaned_aa": {
+      "wer": 2.46,
+      "audio_length": 7122.378375,
+      "inference_time": 2.8625032720156014,
+      "rtfx": 2488.1643,
+      "ins": 99,
+      "del": 147,
+      "sub": 191
+    }
+  },
+  "runs": {
+    "ami_cleaned": {
+      "completion_sha256": "867369d9353d5006fb9219b0cb870bd7591383d54a862a4fd843657301610b31",
+      "manifest_sha256": "79c7419a85acda2054114234299ff7574efcf3cb0693d40e1c7bd5a6e69d416b",
+      "run_sha256": "823d21dbdd15fa3cd20c3fd002f2ba42bef6b0a0e8ee1d31382fc54099d13246",
+      "inputs_sha256": "e50554e60ce197c9c6d96a4efb86768d58c48e1bb50a57c05bce7780dcd3f28b",
+      "model_revision": "555136b50265a132d4cea0d35560c26fc4f657ab",
+      "model_sha256": "031c8ddab4845aeced904a7cde8e8aa57993b2e344716cf83a545b079c473b56",
+      "dataset": {
+        "dataset_path": "hf-audio/open-asr-leaderboard",
+        "dataset": "ami_cleaned",
+        "split": "test",
+        "dataset_revision": "b6bdcd0beb34f8975dc659796176d88f43aff502",
+        "parent_dataset_revision": null
+      },
+      "counts": {
+        "source_count": 7805,
+        "eligible_count": 7715,
+        "prediction_count": 7715
+      },
+      "runtime": {
+        "gpu_name": "NVIDIA H200",
+        "device": "cuda:0",
+        "dtype": "torch.bfloat16",
+        "batch_size": 128,
+        "num_workers": 1,
+        "packages": {
+          "torch": "2.8.0+cu128",
+          "nemo_toolkit": "2.7.2",
+          "datasets": "5.0.0",
+          "torchcodec": "0.6.0+cu128",
+          "soundfile": "0.14.0",
+          "numpy": "2.4.6",
+          "huggingface_hub": "0.36.2"
+        },
+        "decoding": {
+          "model_type": "tdt",
+          "strategy": "greedy_batch",
+          "compute_hypothesis_token_set": false,
+          "preserve_alignments": null,
+          "tdt_include_token_duration": null,
+          "confidence_cfg": {
+            "preserve_frame_confidence": false,
+            "preserve_token_confidence": false,
+            "preserve_word_confidence": false,
+            "exclude_blank": true,
+            "aggregation": "min",
+            "tdt_include_duration": false,
+            "method_cfg": {
+              "name": "entropy",
+              "entropy_type": "tsallis",
+              "alpha": 0.33,
+              "entropy_norm": "exp",
+              "temperature": "DEPRECATED"
+            }
+          },
+          "fused_batch_size": null,
+          "compute_timestamps": null,
+          "compute_langs": false,
+          "word_seperator": " ",
+          "segment_seperators": [
+            ".",
+            "!",
+            "?"
+          ],
+          "segment_gap_threshold": null,
+          "rnnt_timestamp_type": "all",
+          "greedy": {
+            "max_symbols_per_step": 10,
+            "preserve_alignments": false,
+            "preserve_frame_confidence": false,
+            "tdt_include_token_duration": false,
+            "tdt_include_duration_confidence": false,
+            "confidence_method_cfg": {
+              "name": "entropy",
+              "entropy_type": "tsallis",
+              "alpha": 0.33,
+              "entropy_norm": "exp",
+              "temperature": "DEPRECATED"
+            },
+            "loop_labels": true,
+            "use_cuda_graph_decoder": true,
+            "ngram_lm_model": null,
+            "ngram_lm_alpha": 0.0,
+            "boosting_tree": {
+              "model_path": null,
+              "key_phrases_file": null,
+              "key_phrases_list": null,
+              "key_phrase_items_list": null,
+              "context_score": 1.0,
+              "depth_scaling": 2.0,
+              "unk_score": 0.0,
+              "final_eos_score": 1.0,
+              "score_per_phrase": 0.0,
+              "source_lang": "en",
+              "use_triton": true,
+              "uniform_weights": false,
+              "use_bpe_dropout": false,
+              "num_of_transcriptions": 5,
+              "bpe_alpha": 0.3
+            },
+            "boosting_tree_alpha": 0.0,
+            "enable_per_stream_biasing": false,
+            "max_symbols": 10
+          },
+          "beam": {
+            "beam_size": 2,
+            "search_type": "default",
+            "score_norm": true,
+            "return_best_hypothesis": false,
+            "tsd_max_sym_exp_per_step": 50,
+            "alsd_max_target_len": 2.0,
+            "nsc_max_timesteps_expansion": 1,
+            "nsc_prefix_alpha": 1,
+            "maes_num_steps": 2,
+            "maes_prefix_alpha": 1,
+            "maes_expansion_gamma": 2.3,
+            "maes_expansion_beta": 2,
+            "language_model": null,
+            "softmax_temperature": 1.0,
+            "preserve_alignments": false,
+            "ngram_lm_model": null,
+            "ngram_lm_alpha": 0.0,
+            "boosting_tree": {
+              "model_path": null,
+              "key_phrases_file": null,
+              "key_phrases_list": null,
+              "key_phrase_items_list": null,
+              "context_score": 1.0,
+              "depth_scaling": 2.0,
+              "unk_score": 0.0,
+              "final_eos_score": 1.0,
+              "score_per_phrase": 0.0,
+              "source_lang": "en",
+              "use_triton": true,
+              "uniform_weights": false,
+              "use_bpe_dropout": false,
+              "num_of_transcriptions": 5,
+              "bpe_alpha": 0.3
+            },
+            "boosting_tree_alpha": 0.0,
+            "hat_subtract_ilm": false,
+            "hat_ilm_weight": 0.0,
+            "max_symbols_per_step": 10,
+            "blank_lm_score_mode": "LM_WEIGHTED_FULL",
+            "pruning_mode": "LATE",
+            "allow_cuda_graphs": true,
+            "tsd_max_sym_exp": 50
+          },
+          "temperature": 1.0,
+          "durations": [
+            0,
+            1,
+            2,
+            3,
+            4
+          ],
+          "big_blank_durations": []
+        },
+        "parameter_dtypes": [
+          "torch.bfloat16"
+        ],
+        "parameter_devices": [
+          "cuda:0"
+        ],
+        "warmup_batches": 4,
+        "cuda_synchronized": true,
+        "clock": "time.perf_counter",
+        "segmentation": "unchanged source clips; no added chunking or truncation"
+      },
+      "timing": {
+        "audio_seconds": 28727.5903125,
+        "transcription_seconds": 7.678677501156926,
+        "rtfx_unrounded": 3741.2158940353584,
+        "timing_scope": "full transcribe pass including file loading and decoding; four-batch warmup excluded; CUDA synchronized"
+      }
+    },
+    "earnings22_cleaned_aa_chunked": {
+      "completion_sha256": "09fe168223cb28d0f57fc7d33d660e7a23769dbb2e6f00e20010fe25364c1125",
+      "manifest_sha256": "6d5361d4b3946158906882ff454050ed31b2ec7e3458044a7a06ed4acd938229",
+      "run_sha256": "d09c8781619a5a6d279bc5900a29e5825308298ec0e8b23c517558e969936a67",
+      "inputs_sha256": "2ef1afa2dab81292b15a4c2e43fcded0ae925009f795e50136cbf86fafbd657e",
+      "model_revision": "555136b50265a132d4cea0d35560c26fc4f657ab",
+      "model_sha256": "031c8ddab4845aeced904a7cde8e8aa57993b2e344716cf83a545b079c473b56",
+      "dataset": {
+        "dataset_path": "ArtificialAnalysis/Earnings22-Cleaned-AA-chunked",
+        "dataset": "",
+        "split": "test",
+        "dataset_revision": "27f7d0056d593231974a752ce03c1e965da69b3f",
+        "parent_dataset_revision": "5999936ee3d8d3cde160e3c65a8cc6ff5b7c5276"
+      },
+      "counts": {
+        "source_count": 341,
+        "eligible_count": 341,
+        "prediction_count": 341
+      },
+      "runtime": {
+        "gpu_name": "NVIDIA H200",
+        "device": "cuda:0",
+        "dtype": "torch.bfloat16",
+        "batch_size": 128,
+        "num_workers": 1,
+        "packages": {
+          "torch": "2.8.0+cu128",
+          "nemo_toolkit": "2.7.2",
+          "datasets": "5.0.0",
+          "torchcodec": "0.6.0+cu128",
+          "soundfile": "0.14.0",
+          "numpy": "2.4.6",
+          "huggingface_hub": "0.36.2"
+        },
+        "decoding": {
+          "model_type": "tdt",
+          "strategy": "greedy_batch",
+          "compute_hypothesis_token_set": false,
+          "preserve_alignments": null,
+          "tdt_include_token_duration": null,
+          "confidence_cfg": {
+            "preserve_frame_confidence": false,
+            "preserve_token_confidence": false,
+            "preserve_word_confidence": false,
+            "exclude_blank": true,
+            "aggregation": "min",
+            "tdt_include_duration": false,
+            "method_cfg": {
+              "name": "entropy",
+              "entropy_type": "tsallis",
+              "alpha": 0.33,
+              "entropy_norm": "exp",
+              "temperature": "DEPRECATED"
+            }
+          },
+          "fused_batch_size": null,
+          "compute_timestamps": null,
+          "compute_langs": false,
+          "word_seperator": " ",
+          "segment_seperators": [
+            ".",
+            "!",
+            "?"
+          ],
+          "segment_gap_threshold": null,
+          "rnnt_timestamp_type": "all",
+          "greedy": {
+            "max_symbols_per_step": 10,
+            "preserve_alignments": false,
+            "preserve_frame_confidence": false,
+            "tdt_include_token_duration": false,
+            "tdt_include_duration_confidence": false,
+            "confidence_method_cfg": {
+              "name": "entropy",
+              "entropy_type": "tsallis",
+              "alpha": 0.33,
+              "entropy_norm": "exp",
+              "temperature": "DEPRECATED"
+            },
+            "loop_labels": true,
+            "use_cuda_graph_decoder": true,
+            "ngram_lm_model": null,
+            "ngram_lm_alpha": 0.0,
+            "boosting_tree": {
+              "model_path": null,
+              "key_phrases_file": null,
+              "key_phrases_list": null,
+              "key_phrase_items_list": null,
+              "context_score": 1.0,
+              "depth_scaling": 2.0,
+              "unk_score": 0.0,
+              "final_eos_score": 1.0,
+              "score_per_phrase": 0.0,
+              "source_lang": "en",
+              "use_triton": true,
+              "uniform_weights": false,
+              "use_bpe_dropout": false,
+              "num_of_transcriptions": 5,
+              "bpe_alpha": 0.3
+            },
+            "boosting_tree_alpha": 0.0,
+            "enable_per_stream_biasing": false,
+            "max_symbols": 10
+          },
+          "beam": {
+            "beam_size": 2,
+            "search_type": "default",
+            "score_norm": true,
+            "return_best_hypothesis": false,
+            "tsd_max_sym_exp_per_step": 50,
+            "alsd_max_target_len": 2.0,
+            "nsc_max_timesteps_expansion": 1,
+            "nsc_prefix_alpha": 1,
+            "maes_num_steps": 2,
+            "maes_prefix_alpha": 1,
+            "maes_expansion_gamma": 2.3,
+            "maes_expansion_beta": 2,
+            "language_model": null,
+            "softmax_temperature": 1.0,
+            "preserve_alignments": false,
+            "ngram_lm_model": null,
+            "ngram_lm_alpha": 0.0,
+            "boosting_tree": {
+              "model_path": null,
+              "key_phrases_file": null,
+              "key_phrases_list": null,
+              "key_phrase_items_list": null,
+              "context_score": 1.0,
+              "depth_scaling": 2.0,
+              "unk_score": 0.0,
+              "final_eos_score": 1.0,
+              "score_per_phrase": 0.0,
+              "source_lang": "en",
+              "use_triton": true,
+              "uniform_weights": false,
+              "use_bpe_dropout": false,
+              "num_of_transcriptions": 5,
+              "bpe_alpha": 0.3
+            },
+            "boosting_tree_alpha": 0.0,
+            "hat_subtract_ilm": false,
+            "hat_ilm_weight": 0.0,
+            "max_symbols_per_step": 10,
+            "blank_lm_score_mode": "LM_WEIGHTED_FULL",
+            "pruning_mode": "LATE",
+            "allow_cuda_graphs": true,
+            "tsd_max_sym_exp": 50
+          },
+          "temperature": 1.0,
+          "durations": [
+            0,
+            1,
+            2,
+            3,
+            4
+          ],
+          "big_blank_durations": []
+        },
+        "parameter_dtypes": [
+          "torch.bfloat16"
+        ],
+        "parameter_devices": [
+          "cuda:0"
+        ],
+        "warmup_batches": 4,
+        "cuda_synchronized": true,
+        "clock": "time.perf_counter",
+        "segmentation": "unchanged source clips; no added chunking or truncation"
+      },
+      "timing": {
+        "audio_seconds": 6900.937625,
+        "transcription_seconds": 2.3140653409063816,
+        "rtfx_unrounded": 2982.1705995116868,
+        "timing_scope": "full transcribe pass including file loading and decoding; four-batch warmup excluded; CUDA synchronized"
+      }
+    },
+    "gigaspeech_cleaned": {
+      "completion_sha256": "49e46bcdd03efe2eb8398417a6427ebb34babec98a17b8f30990fb676757b128",
+      "manifest_sha256": "b2904a6c1e25d4146ec46f416f6837ee6b751d8d2f5b412bd8f0e990357f6c6c",
+      "run_sha256": "e9ac3a765118471602ef13709383dae3ba4f316c083bc3f231cc8b37d4cbaa99",
+      "inputs_sha256": "91a933d2a4128e506f7f7faec4c0a218b804fd3d1298d775092729e9520ff6b3",
+      "model_revision": "555136b50265a132d4cea0d35560c26fc4f657ab",
+      "model_sha256": "031c8ddab4845aeced904a7cde8e8aa57993b2e344716cf83a545b079c473b56",
+      "dataset": {
+        "dataset_path": "hf-audio/open-asr-leaderboard",
+        "dataset": "gigaspeech_cleaned",
+        "split": "test",
+        "dataset_revision": "b6bdcd0beb34f8975dc659796176d88f43aff502",
+        "parent_dataset_revision": null
+      },
+      "counts": {
+        "source_count": 18768,
+        "eligible_count": 18757,
+        "prediction_count": 18757
+      },
+      "runtime": {
+        "gpu_name": "NVIDIA H200",
+        "device": "cuda:0",
+        "dtype": "torch.bfloat16",
+        "batch_size": 128,
+        "num_workers": 1,
+        "packages": {
+          "torch": "2.8.0+cu128",
+          "nemo_toolkit": "2.7.2",
+          "datasets": "5.0.0",
+          "torchcodec": "0.6.0+cu128",
+          "soundfile": "0.14.0",
+          "numpy": "2.4.6",
+          "huggingface_hub": "0.36.2"
+        },
+        "decoding": {
+          "model_type": "tdt",
+          "strategy": "greedy_batch",
+          "compute_hypothesis_token_set": false,
+          "preserve_alignments": null,
+          "tdt_include_token_duration": null,
+          "confidence_cfg": {
+            "preserve_frame_confidence": false,
+            "preserve_token_confidence": false,
+            "preserve_word_confidence": false,
+            "exclude_blank": true,
+            "aggregation": "min",
+            "tdt_include_duration": false,
+            "method_cfg": {
+              "name": "entropy",
+              "entropy_type": "tsallis",
+              "alpha": 0.33,
+              "entropy_norm": "exp",
+              "temperature": "DEPRECATED"
+            }
+          },
+          "fused_batch_size": null,
+          "compute_timestamps": null,
+          "compute_langs": false,
+          "word_seperator": " ",
+          "segment_seperators": [
+            ".",
+            "!",
+            "?"
+          ],
+          "segment_gap_threshold": null,
+          "rnnt_timestamp_type": "all",
+          "greedy": {
+            "max_symbols_per_step": 10,
+            "preserve_alignments": false,
+            "preserve_frame_confidence": false,
+            "tdt_include_token_duration": false,
+            "tdt_include_duration_confidence": false,
+            "confidence_method_cfg": {
+              "name": "entropy",
+              "entropy_type": "tsallis",
+              "alpha": 0.33,
+              "entropy_norm": "exp",
+              "temperature": "DEPRECATED"
+            },
+            "loop_labels": true,
+            "use_cuda_graph_decoder": true,
+            "ngram_lm_model": null,
+            "ngram_lm_alpha": 0.0,
+            "boosting_tree": {
+              "model_path": null,
+              "key_phrases_file": null,
+              "key_phrases_list": null,
+              "key_phrase_items_list": null,
+              "context_score": 1.0,
+              "depth_scaling": 2.0,
+              "unk_score": 0.0,
+              "final_eos_score": 1.0,
+              "score_per_phrase": 0.0,
+              "source_lang": "en",
+              "use_triton": true,
+              "uniform_weights": false,
+              "use_bpe_dropout": false,
+              "num_of_transcriptions": 5,
+              "bpe_alpha": 0.3
+            },
+            "boosting_tree_alpha": 0.0,
+            "enable_per_stream_biasing": false,
+            "max_symbols": 10
+          },
+          "beam": {
+            "beam_size": 2,
+            "search_type": "default",
+            "score_norm": true,
+            "return_best_hypothesis": false,
+            "tsd_max_sym_exp_per_step": 50,
+            "alsd_max_target_len": 2.0,
+            "nsc_max_timesteps_expansion": 1,
+            "nsc_prefix_alpha": 1,
+            "maes_num_steps": 2,
+            "maes_prefix_alpha": 1,
+            "maes_expansion_gamma": 2.3,
+            "maes_expansion_beta": 2,
+            "language_model": null,
+            "softmax_temperature": 1.0,
+            "preserve_alignments": false,
+            "ngram_lm_model": null,
+            "ngram_lm_alpha": 0.0,
+            "boosting_tree": {
+              "model_path": null,
+              "key_phrases_file": null,
+              "key_phrases_list": null,
+              "key_phrase_items_list": null,
+              "context_score": 1.0,
+              "depth_scaling": 2.0,
+              "unk_score": 0.0,
+              "final_eos_score": 1.0,
+              "score_per_phrase": 0.0,
+              "source_lang": "en",
+              "use_triton": true,
+              "uniform_weights": false,
+              "use_bpe_dropout": false,
+              "num_of_transcriptions": 5,
+              "bpe_alpha": 0.3
+            },
+            "boosting_tree_alpha": 0.0,
+            "hat_subtract_ilm": false,
+            "hat_ilm_weight": 0.0,
+            "max_symbols_per_step": 10,
+            "blank_lm_score_mode": "LM_WEIGHTED_FULL",
+            "pruning_mode": "LATE",
+            "allow_cuda_graphs": true,
+            "tsd_max_sym_exp": 50
+          },
+          "temperature": 1.0,
+          "durations": [
+            0,
+            1,
+            2,
+            3,
+            4
+          ],
+          "big_blank_durations": []
+        },
+        "parameter_dtypes": [
+          "torch.bfloat16"
+        ],
+        "parameter_devices": [
+          "cuda:0"
+        ],
+        "warmup_batches": 4,
+        "cuda_synchronized": true,
+        "clock": "time.perf_counter",
+        "segmentation": "unchanged source clips; no added chunking or truncation"
+      },
+      "timing": {
+        "audio_seconds": 126515.196,
+        "transcription_seconds": 23.663223169045523,
+        "rtfx_unrounded": 5346.490420861086,
+        "timing_scope": "full transcribe pass including file loading and decoding; four-batch warmup excluded; CUDA synchronized"
+      }
+    },
+    "librispeech_test_clean": {
+      "completion_sha256": "454e79fc47aeac5e5c3f20a0e333abb63aef21bab587d52fb881ed1b132c3ad6",
+      "manifest_sha256": "ef41dea7fd36011ea044679c1d5f5dd419fd2660a86408db1265f72f381ba8d0",
+      "run_sha256": "035e817cb45c3b2ff60a0501196871ec830e192b0706352376980bf5cf586952",
+      "inputs_sha256": "d3d9dc61b5ac17c6c66f66e0da3d25c15acbbc36fb69ee4fd0cd60ee1ddd4c6b",
+      "model_revision": "555136b50265a132d4cea0d35560c26fc4f657ab",
+      "model_sha256": "031c8ddab4845aeced904a7cde8e8aa57993b2e344716cf83a545b079c473b56",
+      "dataset": {
+        "dataset_path": "hf-audio/open-asr-leaderboard",
+        "dataset": "librispeech",
+        "split": "test.clean",
+        "dataset_revision": "b6bdcd0beb34f8975dc659796176d88f43aff502",
+        "parent_dataset_revision": null
+      },
+      "counts": {
+        "source_count": 2620,
+        "eligible_count": 2620,
+        "prediction_count": 2620
+      },
+      "runtime": {
+        "gpu_name": "NVIDIA H200",
+        "device": "cuda:0",
+        "dtype": "torch.bfloat16",
+        "batch_size": 128,
+        "num_workers": 1,
+        "packages": {
+          "torch": "2.8.0+cu128",
+          "nemo_toolkit": "2.7.2",
+          "datasets": "5.0.0",
+          "torchcodec": "0.6.0+cu128",
+          "soundfile": "0.14.0",
+          "numpy": "2.4.6",
+          "huggingface_hub": "0.36.2"
+        },
+        "decoding": {
+          "model_type": "tdt",
+          "strategy": "greedy_batch",
+          "compute_hypothesis_token_set": false,
+          "preserve_alignments": null,
+          "tdt_include_token_duration": null,
+          "confidence_cfg": {
+            "preserve_frame_confidence": false,
+            "preserve_token_confidence": false,
+            "preserve_word_confidence": false,
+            "exclude_blank": true,
+            "aggregation": "min",
+            "tdt_include_duration": false,
+            "method_cfg": {
+              "name": "entropy",
+              "entropy_type": "tsallis",
+              "alpha": 0.33,
+              "entropy_norm": "exp",
+              "temperature": "DEPRECATED"
+            }
+          },
+          "fused_batch_size": null,
+          "compute_timestamps": null,
+          "compute_langs": false,
+          "word_seperator": " ",
+          "segment_seperators": [
+            ".",
+            "!",
+            "?"
+          ],
+          "segment_gap_threshold": null,
+          "rnnt_timestamp_type": "all",
+          "greedy": {
+            "max_symbols_per_step": 10,
+            "preserve_alignments": false,
+            "preserve_frame_confidence": false,
+            "tdt_include_token_duration": false,
+            "tdt_include_duration_confidence": false,
+            "confidence_method_cfg": {
+              "name": "entropy",
+              "entropy_type": "tsallis",
+              "alpha": 0.33,
+              "entropy_norm": "exp",
+              "temperature": "DEPRECATED"
+            },
+            "loop_labels": true,
+            "use_cuda_graph_decoder": true,
+            "ngram_lm_model": null,
+            "ngram_lm_alpha": 0.0,
+            "boosting_tree": {
+              "model_path": null,
+              "key_phrases_file": null,
+              "key_phrases_list": null,
+              "key_phrase_items_list": null,
+              "context_score": 1.0,
+              "depth_scaling": 2.0,
+              "unk_score": 0.0,
+              "final_eos_score": 1.0,
+              "score_per_phrase": 0.0,
+              "source_lang": "en",
+              "use_triton": true,
+              "uniform_weights": false,
+              "use_bpe_dropout": false,
+              "num_of_transcriptions": 5,
+              "bpe_alpha": 0.3
+            },
+            "boosting_tree_alpha": 0.0,
+            "enable_per_stream_biasing": false,
+            "max_symbols": 10
+          },
+          "beam": {
+            "beam_size": 2,
+            "search_type": "default",
+            "score_norm": true,
+            "return_best_hypothesis": false,
+            "tsd_max_sym_exp_per_step": 50,
+            "alsd_max_target_len": 2.0,
+            "nsc_max_timesteps_expansion": 1,
+            "nsc_prefix_alpha": 1,
+            "maes_num_steps": 2,
+            "maes_prefix_alpha": 1,
+            "maes_expansion_gamma": 2.3,
+            "maes_expansion_beta": 2,
+            "language_model": null,
+            "softmax_temperature": 1.0,
+            "preserve_alignments": false,
+            "ngram_lm_model": null,
+            "ngram_lm_alpha": 0.0,
+            "boosting_tree": {
+              "model_path": null,
+              "key_phrases_file": null,
+              "key_phrases_list": null,
+              "key_phrase_items_list": null,
+              "context_score": 1.0,
+              "depth_scaling": 2.0,
+              "unk_score": 0.0,
+              "final_eos_score": 1.0,
+              "score_per_phrase": 0.0,
+              "source_lang": "en",
+              "use_triton": true,
+              "uniform_weights": false,
+              "use_bpe_dropout": false,
+              "num_of_transcriptions": 5,
+              "bpe_alpha": 0.3
+            },
+            "boosting_tree_alpha": 0.0,
+            "hat_subtract_ilm": false,
+            "hat_ilm_weight": 0.0,
+            "max_symbols_per_step": 10,
+            "blank_lm_score_mode": "LM_WEIGHTED_FULL",
+            "pruning_mode": "LATE",
+            "allow_cuda_graphs": true,
+            "tsd_max_sym_exp": 50
+          },
+          "temperature": 1.0,
+          "durations": [
+            0,
+            1,
+            2,
+            3,
+            4
+          ],
+          "big_blank_durations": []
+        },
+        "parameter_dtypes": [
+          "torch.bfloat16"
+        ],
+        "parameter_devices": [
+          "cuda:0"
+        ],
+        "warmup_batches": 4,
+        "cuda_synchronized": true,
+        "clock": "time.perf_counter",
+        "segmentation": "unchanged source clips; no added chunking or truncation"
+      },
+      "timing": {
+        "audio_seconds": 19452.480625,
+        "transcription_seconds": 4.653575702803209,
+        "rtfx_unrounded": 4180.114790715076,
+        "timing_scope": "full transcribe pass including file loading and decoding; four-batch warmup excluded; CUDA synchronized"
+      }
+    },
+    "librispeech_test_other": {
+      "completion_sha256": "b996b7eddeacb73cb4bd4574098a79cac229697b2d5f517fc8984cb4276bf484",
+      "manifest_sha256": "0358a449ca0adbd031371a8c342189b67330d3ec66f8dbb8ca898829521c215e",
+      "run_sha256": "4c5ecfb0c6ce492a92634607b786e9292b5c43df654d9bf13b27f91b3aa2391f",
+      "inputs_sha256": "c695e3fea3f000b019ed0a7a5bea10a3036f0f4952927887174f0f1466337c63",
+      "model_revision": "555136b50265a132d4cea0d35560c26fc4f657ab",
+      "model_sha256": "031c8ddab4845aeced904a7cde8e8aa57993b2e344716cf83a545b079c473b56",
+      "dataset": {
+        "dataset_path": "hf-audio/open-asr-leaderboard",
+        "dataset": "librispeech",
+        "split": "test.other",
+        "dataset_revision": "b6bdcd0beb34f8975dc659796176d88f43aff502",
+        "parent_dataset_revision": null
+      },
+      "counts": {
+        "source_count": 2939,
+        "eligible_count": 2939,
+        "prediction_count": 2939
+      },
+      "runtime": {
+        "gpu_name": "NVIDIA H200",
+        "device": "cuda:0",
+        "dtype": "torch.bfloat16",
+        "batch_size": 128,
+        "num_workers": 1,
+        "packages": {
+          "torch": "2.8.0+cu128",
+          "nemo_toolkit": "2.7.2",
+          "datasets": "5.0.0",
+          "torchcodec": "0.6.0+cu128",
+          "soundfile": "0.14.0",
+          "numpy": "2.4.6",
+          "huggingface_hub": "0.36.2"
+        },
+        "decoding": {
+          "model_type": "tdt",
+          "strategy": "greedy_batch",
+          "compute_hypothesis_token_set": false,
+          "preserve_alignments": null,
+          "tdt_include_token_duration": null,
+          "confidence_cfg": {
+            "preserve_frame_confidence": false,
+            "preserve_token_confidence": false,
+            "preserve_word_confidence": false,
+            "exclude_blank": true,
+            "aggregation": "min",
+            "tdt_include_duration": false,
+            "method_cfg": {
+              "name": "entropy",
+              "entropy_type": "tsallis",
+              "alpha": 0.33,
+              "entropy_norm": "exp",
+              "temperature": "DEPRECATED"
+            }
+          },
+          "fused_batch_size": null,
+          "compute_timestamps": null,
+          "compute_langs": false,
+          "word_seperator": " ",
+          "segment_seperators": [
+            ".",
+            "!",
+            "?"
+          ],
+          "segment_gap_threshold": null,
+          "rnnt_timestamp_type": "all",
+          "greedy": {
+            "max_symbols_per_step": 10,
+            "preserve_alignments": false,
+            "preserve_frame_confidence": false,
+            "tdt_include_token_duration": false,
+            "tdt_include_duration_confidence": false,
+            "confidence_method_cfg": {
+              "name": "entropy",
+              "entropy_type": "tsallis",
+              "alpha": 0.33,
+              "entropy_norm": "exp",
+              "temperature": "DEPRECATED"
+            },
+            "loop_labels": true,
+            "use_cuda_graph_decoder": true,
+            "ngram_lm_model": null,
+            "ngram_lm_alpha": 0.0,
+            "boosting_tree": {
+              "model_path": null,
+              "key_phrases_file": null,
+              "key_phrases_list": null,
+              "key_phrase_items_list": null,
+              "context_score": 1.0,
+              "depth_scaling": 2.0,
+              "unk_score": 0.0,
+              "final_eos_score": 1.0,
+              "score_per_phrase": 0.0,
+              "source_lang": "en",
+              "use_triton": true,
+              "uniform_weights": false,
+              "use_bpe_dropout": false,
+              "num_of_transcriptions": 5,
+              "bpe_alpha": 0.3
+            },
+            "boosting_tree_alpha": 0.0,
+            "enable_per_stream_biasing": false,
+            "max_symbols": 10
+          },
+          "beam": {
+            "beam_size": 2,
+            "search_type": "default",
+            "score_norm": true,
+            "return_best_hypothesis": false,
+            "tsd_max_sym_exp_per_step": 50,
+            "alsd_max_target_len": 2.0,
+            "nsc_max_timesteps_expansion": 1,
+            "nsc_prefix_alpha": 1,
+            "maes_num_steps": 2,
+            "maes_prefix_alpha": 1,
+            "maes_expansion_gamma": 2.3,
+            "maes_expansion_beta": 2,
+            "language_model": null,
+            "softmax_temperature": 1.0,
+            "preserve_alignments": false,
+            "ngram_lm_model": null,
+            "ngram_lm_alpha": 0.0,
+            "boosting_tree": {
+              "model_path": null,
+              "key_phrases_file": null,
+              "key_phrases_list": null,
+              "key_phrase_items_list": null,
+              "context_score": 1.0,
+              "depth_scaling": 2.0,
+              "unk_score": 0.0,
+              "final_eos_score": 1.0,
+              "score_per_phrase": 0.0,
+              "source_lang": "en",
+              "use_triton": true,
+              "uniform_weights": false,
+              "use_bpe_dropout": false,
+              "num_of_transcriptions": 5,
+              "bpe_alpha": 0.3
+            },
+            "boosting_tree_alpha": 0.0,
+            "hat_subtract_ilm": false,
+            "hat_ilm_weight": 0.0,
+            "max_symbols_per_step": 10,
+            "blank_lm_score_mode": "LM_WEIGHTED_FULL",
+            "pruning_mode": "LATE",
+            "allow_cuda_graphs": true,
+            "tsd_max_sym_exp": 50
+          },
+          "temperature": 1.0,
+          "durations": [
+            0,
+            1,
+            2,
+            3,
+            4
+          ],
+          "big_blank_durations": []
+        },
+        "parameter_dtypes": [
+          "torch.bfloat16"
+        ],
+        "parameter_devices": [
+          "cuda:0"
+        ],
+        "warmup_batches": 4,
+        "cuda_synchronized": true,
+        "clock": "time.perf_counter",
+        "segmentation": "unchanged source clips; no added chunking or truncation"
+      },
+      "timing": {
+        "audio_seconds": 19229.570125,
+        "transcription_seconds": 4.6231324940454215,
+        "rtfx_unrounded": 4159.424405371816,
+        "timing_scope": "full transcribe pass including file loading and decoding; four-batch warmup excluded; CUDA synchronized"
+      }
+    },
+    "spgispeech": {
+      "completion_sha256": "1e24a2e56d62cb37ea9b5d320aed0a536f83924fdaf7d6bcca7f3441f8c0f370",
+      "manifest_sha256": "04cee652d2ec9734bbf22918f6d848a3e812465e09e3dd5e4d49f373e189f4e3",
+      "run_sha256": "014659bc814f3b8a61e234d65fdeeb0ce0095ff23d9a267f933ec02309870963",
+      "inputs_sha256": "7a60c54feeedfe78479c1f2782948dd902374e3fda3d7a1082e16cea20191ca2",
+      "model_revision": "555136b50265a132d4cea0d35560c26fc4f657ab",
+      "model_sha256": "031c8ddab4845aeced904a7cde8e8aa57993b2e344716cf83a545b079c473b56",
+      "dataset": {
+        "dataset_path": "hf-audio/open-asr-leaderboard",
+        "dataset": "spgispeech",
+        "split": "test",
+        "dataset_revision": "b6bdcd0beb34f8975dc659796176d88f43aff502",
+        "parent_dataset_revision": null
+      },
+      "counts": {
+        "source_count": 39341,
+        "eligible_count": 39341,
+        "prediction_count": 39341
+      },
+      "runtime": {
+        "gpu_name": "NVIDIA H200",
+        "device": "cuda:0",
+        "dtype": "torch.bfloat16",
+        "batch_size": 128,
+        "num_workers": 1,
+        "packages": {
+          "torch": "2.8.0+cu128",
+          "nemo_toolkit": "2.7.2",
+          "datasets": "5.0.0",
+          "torchcodec": "0.6.0+cu128",
+          "soundfile": "0.14.0",
+          "numpy": "2.4.6",
+          "huggingface_hub": "0.36.2"
+        },
+        "decoding": {
+          "model_type": "tdt",
+          "strategy": "greedy_batch",
+          "compute_hypothesis_token_set": false,
+          "preserve_alignments": null,
+          "tdt_include_token_duration": null,
+          "confidence_cfg": {
+            "preserve_frame_confidence": false,
+            "preserve_token_confidence": false,
+            "preserve_word_confidence": false,
+            "exclude_blank": true,
+            "aggregation": "min",
+            "tdt_include_duration": false,
+            "method_cfg": {
+              "name": "entropy",
+              "entropy_type": "tsallis",
+              "alpha": 0.33,
+              "entropy_norm": "exp",
+              "temperature": "DEPRECATED"
+            }
+          },
+          "fused_batch_size": null,
+          "compute_timestamps": null,
+          "compute_langs": false,
+          "word_seperator": " ",
+          "segment_seperators": [
+            ".",
+            "!",
+            "?"
+          ],
+          "segment_gap_threshold": null,
+          "rnnt_timestamp_type": "all",
+          "greedy": {
+            "max_symbols_per_step": 10,
+            "preserve_alignments": false,
+            "preserve_frame_confidence": false,
+            "tdt_include_token_duration": false,
+            "tdt_include_duration_confidence": false,
+            "confidence_method_cfg": {
+              "name": "entropy",
+              "entropy_type": "tsallis",
+              "alpha": 0.33,
+              "entropy_norm": "exp",
+              "temperature": "DEPRECATED"
+            },
+            "loop_labels": true,
+            "use_cuda_graph_decoder": true,
+            "ngram_lm_model": null,
+            "ngram_lm_alpha": 0.0,
+            "boosting_tree": {
+              "model_path": null,
+              "key_phrases_file": null,
+              "key_phrases_list": null,
+              "key_phrase_items_list": null,
+              "context_score": 1.0,
+              "depth_scaling": 2.0,
+              "unk_score": 0.0,
+              "final_eos_score": 1.0,
+              "score_per_phrase": 0.0,
+              "source_lang": "en",
+              "use_triton": true,
+              "uniform_weights": false,
+              "use_bpe_dropout": false,
+              "num_of_transcriptions": 5,
+              "bpe_alpha": 0.3
+            },
+            "boosting_tree_alpha": 0.0,
+            "enable_per_stream_biasing": false,
+            "max_symbols": 10
+          },
+          "beam": {
+            "beam_size": 2,
+            "search_type": "default",
+            "score_norm": true,
+            "return_best_hypothesis": false,
+            "tsd_max_sym_exp_per_step": 50,
+            "alsd_max_target_len": 2.0,
+            "nsc_max_timesteps_expansion": 1,
+            "nsc_prefix_alpha": 1,
+            "maes_num_steps": 2,
+            "maes_prefix_alpha": 1,
+            "maes_expansion_gamma": 2.3,
+            "maes_expansion_beta": 2,
+            "language_model": null,
+            "softmax_temperature": 1.0,
+            "preserve_alignments": false,
+            "ngram_lm_model": null,
+            "ngram_lm_alpha": 0.0,
+            "boosting_tree": {
+              "model_path": null,
+              "key_phrases_file": null,
+              "key_phrases_list": null,
+              "key_phrase_items_list": null,
+              "context_score": 1.0,
+              "depth_scaling": 2.0,
+              "unk_score": 0.0,
+              "final_eos_score": 1.0,
+              "score_per_phrase": 0.0,
+              "source_lang": "en",
+              "use_triton": true,
+              "uniform_weights": false,
+              "use_bpe_dropout": false,
+              "num_of_transcriptions": 5,
+              "bpe_alpha": 0.3
+            },
+            "boosting_tree_alpha": 0.0,
+            "hat_subtract_ilm": false,
+            "hat_ilm_weight": 0.0,
+            "max_symbols_per_step": 10,
+            "blank_lm_score_mode": "LM_WEIGHTED_FULL",
+            "pruning_mode": "LATE",
+            "allow_cuda_graphs": true,
+            "tsd_max_sym_exp": 50
+          },
+          "temperature": 1.0,
+          "durations": [
+            0,
+            1,
+            2,
+            3,
+            4
+          ],
+          "big_blank_durations": []
+        },
+        "parameter_dtypes": [
+          "torch.bfloat16"
+        ],
+        "parameter_devices": [
+          "cuda:0"
+        ],
+        "warmup_batches": 4,
+        "cuda_synchronized": true,
+        "clock": "time.perf_counter",
+        "segmentation": "unchanged source clips; no added chunking or truncation"
+      },
+      "timing": {
+        "audio_seconds": 360007.4995,
+        "transcription_seconds": 56.08279880089685,
+        "rtfx_unrounded": 6419.214218928084,
+        "timing_scope": "full transcribe pass including file loading and decoding; four-batch warmup excluded; CUDA synchronized"
+      }
+    },
+    "voicearena_monsoon_en_in": {
+      "completion_sha256": "cd6b3b08c5a98e5c9c981305790c7c862a3f808575fa53c6c1129e2fb815d756",
+      "manifest_sha256": "c4d6afde9f6506cfa313d99b9c3019368c88c947ef05e95240fcc780c7f1b34b",
+      "run_sha256": "13fe7db18c2c944c0def9456fedd5b6adff6c2e5ed15c06dbcf80ba1143e7e42",
+      "inputs_sha256": "13845efbda33d013912cc7ffeba467c7a763fbb4ab17fb45770d26256414f512",
+      "model_revision": "555136b50265a132d4cea0d35560c26fc4f657ab",
+      "model_sha256": "031c8ddab4845aeced904a7cde8e8aa57993b2e344716cf83a545b079c473b56",
+      "dataset": {
+        "dataset_path": "VoiceArena/Monsoon_en_IN_test",
+        "dataset": "",
+        "split": "test",
+        "dataset_revision": "bc1da7b42ef6e2853123c97bf6d22067e4802d11",
+        "parent_dataset_revision": null
+      },
+      "counts": {
+        "source_count": 2102,
+        "eligible_count": 2102,
+        "prediction_count": 2102
+      },
+      "runtime": {
+        "gpu_name": "NVIDIA H200",
+        "device": "cuda:0",
+        "dtype": "torch.bfloat16",
+        "batch_size": 128,
+        "num_workers": 1,
+        "packages": {
+          "torch": "2.8.0+cu128",
+          "nemo_toolkit": "2.7.2",
+          "datasets": "5.0.0",
+          "torchcodec": "0.6.0+cu128",
+          "soundfile": "0.14.0",
+          "numpy": "2.4.6",
+          "huggingface_hub": "0.36.2"
+        },
+        "decoding": {
+          "model_type": "tdt",
+          "strategy": "greedy_batch",
+          "compute_hypothesis_token_set": false,
+          "preserve_alignments": null,
+          "tdt_include_token_duration": null,
+          "confidence_cfg": {
+            "preserve_frame_confidence": false,
+            "preserve_token_confidence": false,
+            "preserve_word_confidence": false,
+            "exclude_blank": true,
+            "aggregation": "min",
+            "tdt_include_duration": false,
+            "method_cfg": {
+              "name": "entropy",
+              "entropy_type": "tsallis",
+              "alpha": 0.33,
+              "entropy_norm": "exp",
+              "temperature": "DEPRECATED"
+            }
+          },
+          "fused_batch_size": null,
+          "compute_timestamps": null,
+          "compute_langs": false,
+          "word_seperator": " ",
+          "segment_seperators": [
+            ".",
+            "!",
+            "?"
+          ],
+          "segment_gap_threshold": null,
+          "rnnt_timestamp_type": "all",
+          "greedy": {
+            "max_symbols_per_step": 10,
+            "preserve_alignments": false,
+            "preserve_frame_confidence": false,
+            "tdt_include_token_duration": false,
+            "tdt_include_duration_confidence": false,
+            "confidence_method_cfg": {
+              "name": "entropy",
+              "entropy_type": "tsallis",
+              "alpha": 0.33,
+              "entropy_norm": "exp",
+              "temperature": "DEPRECATED"
+            },
+            "loop_labels": true,
+            "use_cuda_graph_decoder": true,
+            "ngram_lm_model": null,
+            "ngram_lm_alpha": 0.0,
+            "boosting_tree": {
+              "model_path": null,
+              "key_phrases_file": null,
+              "key_phrases_list": null,
+              "key_phrase_items_list": null,
+              "context_score": 1.0,
+              "depth_scaling": 2.0,
+              "unk_score": 0.0,
+              "final_eos_score": 1.0,
+              "score_per_phrase": 0.0,
+              "source_lang": "en",
+              "use_triton": true,
+              "uniform_weights": false,
+              "use_bpe_dropout": false,
+              "num_of_transcriptions": 5,
+              "bpe_alpha": 0.3
+            },
+            "boosting_tree_alpha": 0.0,
+            "enable_per_stream_biasing": false,
+            "max_symbols": 10
+          },
+          "beam": {
+            "beam_size": 2,
+            "search_type": "default",
+            "score_norm": true,
+            "return_best_hypothesis": false,
+            "tsd_max_sym_exp_per_step": 50,
+            "alsd_max_target_len": 2.0,
+            "nsc_max_timesteps_expansion": 1,
+            "nsc_prefix_alpha": 1,
+            "maes_num_steps": 2,
+            "maes_prefix_alpha": 1,
+            "maes_expansion_gamma": 2.3,
+            "maes_expansion_beta": 2,
+            "language_model": null,
+            "softmax_temperature": 1.0,
+            "preserve_alignments": false,
+            "ngram_lm_model": null,
+            "ngram_lm_alpha": 0.0,
+            "boosting_tree": {
+              "model_path": null,
+              "key_phrases_file": null,
+              "key_phrases_list": null,
+              "key_phrase_items_list": null,
+              "context_score": 1.0,
+              "depth_scaling": 2.0,
+              "unk_score": 0.0,
+              "final_eos_score": 1.0,
+              "score_per_phrase": 0.0,
+              "source_lang": "en",
+              "use_triton": true,
+              "uniform_weights": false,
+              "use_bpe_dropout": false,
+              "num_of_transcriptions": 5,
+              "bpe_alpha": 0.3
+            },
+            "boosting_tree_alpha": 0.0,
+            "hat_subtract_ilm": false,
+            "hat_ilm_weight": 0.0,
+            "max_symbols_per_step": 10,
+            "blank_lm_score_mode": "LM_WEIGHTED_FULL",
+            "pruning_mode": "LATE",
+            "allow_cuda_graphs": true,
+            "tsd_max_sym_exp": 50
+          },
+          "temperature": 1.0,
+          "durations": [
+            0,
+            1,
+            2,
+            3,
+            4
+          ],
+          "big_blank_durations": []
+        },
+        "parameter_dtypes": [
+          "torch.bfloat16"
+        ],
+        "parameter_devices": [
+          "cuda:0"
+        ],
+        "warmup_batches": 4,
+        "cuda_synchronized": true,
+        "clock": "time.perf_counter",
+        "segmentation": "unchanged source clips; no added chunking or truncation"
+      },
+      "timing": {
+        "audio_seconds": 20247.6074375,
+        "transcription_seconds": 4.187174894148484,
+        "rtfx_unrounded": 4835.624961784075,
+        "timing_scope": "full transcribe pass including file loading and decoding; four-batch warmup excluded; CUDA synchronized"
+      }
+    },
+    "voxpopuli_cleaned_aa": {
+      "completion_sha256": "92805a8d186411aeb61b1591222be24878c95669fa80bc52335ce95f1ad960b7",
+      "manifest_sha256": "4ea41ceff0db011f6b5caa45825a5613489a4043f1c95c7b1e2a0b7c63009519",
+      "run_sha256": "2f6ab7c29a8f5912e020444aefee81beec6239f926ed4a3185bc8528a0fdea2c",
+      "inputs_sha256": "98255266f2e023d8bb891e76c17b67494a4c3a9a6b1bdaff848739992f9cc078",
+      "model_revision": "555136b50265a132d4cea0d35560c26fc4f657ab",
+      "model_sha256": "031c8ddab4845aeced904a7cde8e8aa57993b2e344716cf83a545b079c473b56",
+      "dataset": {
+        "dataset_path": "hf-audio/open-asr-leaderboard",
+        "dataset": "voxpopuli_cleaned_aa",
+        "split": "test",
+        "dataset_revision": "b6bdcd0beb34f8975dc659796176d88f43aff502",
+        "parent_dataset_revision": null
+      },
+      "counts": {
+        "source_count": 628,
+        "eligible_count": 628,
+        "prediction_count": 628
+      },
+      "runtime": {
+        "gpu_name": "NVIDIA H200",
+        "device": "cuda:0",
+        "dtype": "torch.bfloat16",
+        "batch_size": 128,
+        "num_workers": 1,
+        "packages": {
+          "torch": "2.8.0+cu128",
+          "nemo_toolkit": "2.7.2",
+          "datasets": "5.0.0",
+          "torchcodec": "0.6.0+cu128",
+          "soundfile": "0.14.0",
+          "numpy": "2.4.6",
+          "huggingface_hub": "0.36.2"
+        },
+        "decoding": {
+          "model_type": "tdt",
+          "strategy": "greedy_batch",
+          "compute_hypothesis_token_set": false,
+          "preserve_alignments": null,
+          "tdt_include_token_duration": null,
+          "confidence_cfg": {
+            "preserve_frame_confidence": false,
+            "preserve_token_confidence": false,
+            "preserve_word_confidence": false,
+            "exclude_blank": true,
+            "aggregation": "min",
+            "tdt_include_duration": false,
+            "method_cfg": {
+              "name": "entropy",
+              "entropy_type": "tsallis",
+              "alpha": 0.33,
+              "entropy_norm": "exp",
+              "temperature": "DEPRECATED"
+            }
+          },
+          "fused_batch_size": null,
+          "compute_timestamps": null,
+          "compute_langs": false,
+          "word_seperator": " ",
+          "segment_seperators": [
+            ".",
+            "!",
+            "?"
+          ],
+          "segment_gap_threshold": null,
+          "rnnt_timestamp_type": "all",
+          "greedy": {
+            "max_symbols_per_step": 10,
+            "preserve_alignments": false,
+            "preserve_frame_confidence": false,
+            "tdt_include_token_duration": false,
+            "tdt_include_duration_confidence": false,
+            "confidence_method_cfg": {
+              "name": "entropy",
+              "entropy_type": "tsallis",
+              "alpha": 0.33,
+              "entropy_norm": "exp",
+              "temperature": "DEPRECATED"
+            },
+            "loop_labels": true,
+            "use_cuda_graph_decoder": true,
+            "ngram_lm_model": null,
+            "ngram_lm_alpha": 0.0,
+            "boosting_tree": {
+              "model_path": null,
+              "key_phrases_file": null,
+              "key_phrases_list": null,
+              "key_phrase_items_list": null,
+              "context_score": 1.0,
+              "depth_scaling": 2.0,
+              "unk_score": 0.0,
+              "final_eos_score": 1.0,
+              "score_per_phrase": 0.0,
+              "source_lang": "en",
+              "use_triton": true,
+              "uniform_weights": false,
+              "use_bpe_dropout": false,
+              "num_of_transcriptions": 5,
+              "bpe_alpha": 0.3
+            },
+            "boosting_tree_alpha": 0.0,
+            "enable_per_stream_biasing": false,
+            "max_symbols": 10
+          },
+          "beam": {
+            "beam_size": 2,
+            "search_type": "default",
+            "score_norm": true,
+            "return_best_hypothesis": false,
+            "tsd_max_sym_exp_per_step": 50,
+            "alsd_max_target_len": 2.0,
+            "nsc_max_timesteps_expansion": 1,
+            "nsc_prefix_alpha": 1,
+            "maes_num_steps": 2,
+            "maes_prefix_alpha": 1,
+            "maes_expansion_gamma": 2.3,
+            "maes_expansion_beta": 2,
+            "language_model": null,
+            "softmax_temperature": 1.0,
+            "preserve_alignments": false,
+            "ngram_lm_model": null,
+            "ngram_lm_alpha": 0.0,
+            "boosting_tree": {
+              "model_path": null,
+              "key_phrases_file": null,
+              "key_phrases_list": null,
+              "key_phrase_items_list": null,
+              "context_score": 1.0,
+              "depth_scaling": 2.0,
+              "unk_score": 0.0,
+              "final_eos_score": 1.0,
+              "score_per_phrase": 0.0,
+              "source_lang": "en",
+              "use_triton": true,
+              "uniform_weights": false,
+              "use_bpe_dropout": false,
+              "num_of_transcriptions": 5,
+              "bpe_alpha": 0.3
+            },
+            "boosting_tree_alpha": 0.0,
+            "hat_subtract_ilm": false,
+            "hat_ilm_weight": 0.0,
+            "max_symbols_per_step": 10,
+            "blank_lm_score_mode": "LM_WEIGHTED_FULL",
+            "pruning_mode": "LATE",
+            "allow_cuda_graphs": true,
+            "tsd_max_sym_exp": 50
+          },
+          "temperature": 1.0,
+          "durations": [
+            0,
+            1,
+            2,
+            3,
+            4
+          ],
+          "big_blank_durations": []
+        },
+        "parameter_dtypes": [
+          "torch.bfloat16"
+        ],
+        "parameter_devices": [
+          "cuda:0"
+        ],
+        "warmup_batches": 4,
+        "cuda_synchronized": true,
+        "clock": "time.perf_counter",
+        "segmentation": "unchanged source clips; no added chunking or truncation"
+      },
+      "timing": {
+        "audio_seconds": 7122.378375,
+        "transcription_seconds": 2.8625032720156014,
+        "rtfx_unrounded": 2488.1642737773545,
+        "timing_scope": "full transcribe pass including file loading and decoding; four-batch warmup excluded; CUDA synchronized"
+      }
+    }
+  },
+  "source_url": "https://huggingface.co/oruk/orukeet/blob/main/evaluation/open_asr_20260918/public-eight-h200-results.json",
+  "prepared_by_sha256": "600b58ad29df47ef1bbc833b25d99c6dd4fff3ed40cbf1ad002efa467cab0b33",
+  "bundle_sha256": "0cb403d1553052f0396770be499558cdd2a0d88dd29e73481121b54f715af5cd",
+  "task_map_sha256": "27d565b719a9f98e5cfd58e185404d3c129329ebfc5b73cb3de0399ee6a34137",
+  "hub_yaml": {
+    "path": ".eval_results/open_asr_leaderboard.yaml",
+    "sha256": "b2aa8144405d83c39278ff022e6eb16193a0b769d11b2775b52253db815d25e7"
+  },
+  "known_exposure": "LibriSpeech test-other was used for r3 training and selection. Monsoon English informed prior training-partition selection. Remaining inherited exposure is incompletely verified."
+}

evaluation/open_asr_20260918/public-eight-h200-results.md ADDED Viewed

	@@ -0,0 +1,16 @@

+| Dataset | WER (%) | H200 RTFx |
+|:--|--:|--:|
+| ami_cleaned | 8.68 | 3741.22 |
+| earnings22_cleaned_aa_chunked | 5.74 | 2982.17 |
+| gigaspeech_cleaned | 7.48 | 5346.49 |
+| librispeech_test_clean | 1.47 | 4180.11 |
+| librispeech_test_other | 2.84 | 4159.42 |
+| spgispeech | 3.38 | 6419.21 |
+| voicearena_monsoon_en_in | 3.84 | 4835.62 |
+| voxpopuli_cleaned_aa | 2.46 | 2488.16 |
+| **Public-eight aggregate** | **4.49** | **5545.68** |
+LibriSpeech test-other was used for r3 training and selection. Monsoon English informed prior training-partition selection. Remaining inherited exposure is incompletely verified.
+Monsoon has no registered Hub metric task. All eight results are retained above; no task ID is invented.
+RTFx aggregate is total audio divided by total measured inference time, not an arithmetic mean of per-dataset RTFx.

evaluation/open_asr_20260918/training-disclosure.md ADDED Viewed

	@@ -0,0 +1,15 @@

+# Orukeet r3: training and benchmark exposure
+This disclosure applies to [Orukeet](https://huggingface.co/oruk/orukeet), native NeMo artifact `orukeet-v0.1.0.nemo` at revision `555136b50265a132d4cea0d35560c26fc4f657ab`. Its SHA-256 is `031c8ddab4845aeced904a7cde8e8aa57993b2e344716cf83a545b079c473b56`.
+**LibriSpeech test-other was used directly for training and model selection.** The released r3 checkpoint trained on all 2,939 recordings for three passes, totaling 8,817 training presentations. The same split informed candidate selection. Its reported WER is therefore a result on exposed data, not a held-out generalization estimate. Exact source-ID and decoded-waveform matching confirmed all 2,939 records against the current evaluation partition.
+**The preceding FT-4035 continuation used 223,452 records across 24 complete selected partitions, totaling 371.47 hours for one pass.** The previously reported 6,118 examples were a sampled follow-up fit diagnostic; they were not the full training population. Those partitions comprised EuroSpeech Bulgarian, Greek and Italian; 17 English GigaSpeechBench domains; Golos crowd Russian; Lesbos Greek; and NST Danish and Swedish. These names describe that continuation stage, not a complete accounting of upstream Parakeet or earlier Orukeet training.
+**Voice Arena Monsoon English has prior evaluation and selection exposure.** All 2,102 records from the same source revision used in the current benchmark participated in the earlier evaluation that informed selection of FT-4035 training partitions. Monsoon itself was not among the 24 partitions directly trained on in that continuation. LibriSpeech test-clean was also evaluated earlier; evaluation alone is not presented as proof of direct training or of an additional selection decision.
+An audit of the retained FT-4035 source and training waveform fingerprints found no exact whole-waveform matches to the eight current public partitions. This limited comparison does not rule out inherited exposure, partial overlap, different segmentation or transformations of the same recording. GigaSpeechBench domain training and GigaSpeech-Cleaned evaluation are distinct source definitions; their similar names do not establish exact overlap or independence.
+The local eight-dataset mean includes test-other. A separately reported seven-dataset sensitivity omits that split, but still includes Monsoon selection exposure and does not certify the remaining audio as unseen. We have not selected a new subset retrospectively.
+We ask the leaderboard maintainers how this known exposure should be represented and whether they require a different checkpoint for inclusion. We do not assume that disclosure guarantees acceptance or that the published guidelines automatically exclude this model.