evals/methods.md · twangodev/Ming-3.0-audio-tiny-NVFP4 at main

Ming-3.0-audio-tiny-NVFP4 / evals /methods.md

docs: update cards with complete local evaluation results

ef7cda1 verified 9 days ago

4.81 kB

> Historical methods for the earlier 25,159-example subset and throughput study. The current model card uses the [full-suite evaluation methods](https://huggingface.co/twangodev/Ming-3.0-audio-tiny-NVFP4/blob/main/evals/full-suite/methods.md).

# Methods and interpretation

The eight-task **adapted local evaluation** covers FLEURS en_us/cmn_hans_cn test, LibriSpeech test-clean/test-other, MMAR, public MMAU test-mini, MMSU and MMLU-Pro test with five-shot validation demonstrations. It has 25,159 acceptance and 1,024 earlier development examples. Selection used inspected outcomes, including acceptance outcomes: these are descriptive revalidations, not an untouched holdout, published-score reproduction, equivalence test or coverage of Ming's other tasks.
Greedy text output caps are ASR 512, audio multiple-choice 1,024 and MMLU-Pro 2,048 new tokens. The evaluation uses 10-gram blocking (a local adaptation for MC/text); Ming internal thinking is disabled for MMLU-Pro. ASR uses corpus WER/CER. Primary MC accuracy keeps every example and counts capped/malformed answers wrong. MMAR/MMAU canonicalize anchored unambiguous labels to option text. MMSU counts malformed answers wrong although its upstream scorer drops them. Raw-parsed accuracy is a separate diagnostic.
Tiny reuses historical native-vLLM BF16 reference against the corrected NVFP4 candidate: 16 routed NVFP4 layers, with BF16 routed layers 5, 10, 15, 18, 20, 21 and 22. Both arms used input limit 8,192, client concurrency 512 and stream interval 8 on separate schedules. The HF-only RoPE correction does not imply an HF RoPE error in the native BF16 reference. Schedule-dependent greedy responses and reused outcomes prevent isolated quantization attribution or sub-0.5 pp reproducibility certification.
`results/quality.csv` uses percentages; deltas and intervals use percentage points. MC intervals are conservative exact discordance-marginal Clopper-Pearson differences with exact two-sided McNemar p-values and paired counts. Tiny's Holm p-values cover only four MC tests. ASR uses 20,000 paired percentile corpus-rate bootstrap replicates, seed 20261025: utterances for FLEURS (speaker IDs unavailable), speakers for LibriSpeech. Tiny's LibriSpeech speaker clusters number 40/33. The CSV reports the primary speaker sensitivity; source reports also retain utterance sensitivity. Nominal 95% and Bonferroni-eight columns assume exchangeable units; MC requires independent pairs and bootstrap coverage is approximate. Multiplicity corrections do not repair selection bias; zero-width intervals do not establish equivalence.
Hardware: NVIDIA RTX PRO 6000 Blackwell, exact SM120 (12.0), measured physical97887MiB/95.593GiB; host RAM about125GiB. Serving: torch 2.13.0, vllm 0.30.0, vllm_omni 0.1.dev3352+g41edeb7fe, flashinfer_python 0.6.18.post1, triton 3.7.1. Preparation: torch 2.8.0, transformers 5.14.1, accelerate 1.10.1, nvidia_modelopt 0.47.0, safetensors 0.8.0. Serving/preparation Torch versions describe distinct environments. Required serving plugins: `ming_nvfp4`, `vllm_omni_register_models`. Native `VLLM_CUTLASS` executes NVFP4 on SM120; Tiny's retained BF16 routed layers use Triton. Portability was not validated by these reports.

Quality profiles: Tiny context 16,384 / server capacity 256, graphs/async, queued client concurrency 512.

`results/performance.csv` gives median output tokens/s over three fixed-workload trials; audio means audio input/text output, not speech synthesis. Tiny: context 16,384 / capacity 1,024; text client concurrency 512 with 256 output tokens, audio concurrency 1,024 with 128 output tokens; 2,048 warmup plus 2,048 timed requests each.

Thinker model allocation is Tiny BF16 15.97 GiB and Tiny NVFP4 9.50 GiB. It excludes KV cache, workspace, CUDA graphs and other allocations; it is neither checkpoint size nor total GPU memory. Workload hashes identify fixed inputs. `evidence_id` resolves to relative local artifact names/SHA-256 in `provenance.json`; original source files remain unchanged and are not distributed. This export checks report hashes/completion, counts, deltas and shared primary values. It does not rerun inference, rehash bulk weights/audio or newly validate every prediction. Only aggregate data is included.

Benchmark references from the local catalog: [FLEURS](https://huggingface.co/datasets/google/fleurs), [LibriSpeech](https://huggingface.co/datasets/openslr/librispeech_asr), [MMAR](https://github.com/ddlBoJack/MMAR), [MMAU](https://github.com/Sakshi113/MMAU/blob/110127f54c0dfba3faa5ec9feee4a7e4148679c5/README.md), [MMSU](https://github.com/dingdongwang/MMSU), [MMLU-Pro](https://github.com/TIGER-AI-Lab/MMLU-Pro/blob/f418b116db00b065c2aea046518d8fcf74d39872/README.md). Source datasets retain their own terms; no examples are redistributed.

	> Historical methods for the earlier 25,159-example subset and throughput study. The current model card uses the [full-suite evaluation methods](https://huggingface.co/twangodev/Ming-3.0-audio-tiny-NVFP4/blob/main/evals/full-suite/methods.md).

	# Methods and interpretation

	The eight-task adapted local evaluation covers FLEURS en_us/cmn_hans_cn test, LibriSpeech test-clean/test-other, MMAR, public MMAU test-mini, MMSU and MMLU-Pro test with five-shot validation demonstrations. It has 25,159 acceptance and 1,024 earlier development examples. Selection used inspected outcomes, including acceptance outcomes: these are descriptive revalidations, not an untouched holdout, published-score reproduction, equivalence test or coverage of Ming's other tasks.

	Greedy text output caps are ASR 512, audio multiple-choice 1,024 and MMLU-Pro 2,048 new tokens. The evaluation uses 10-gram blocking (a local adaptation for MC/text); Ming internal thinking is disabled for MMLU-Pro. ASR uses corpus WER/CER. Primary MC accuracy keeps every example and counts capped/malformed answers wrong. MMAR/MMAU canonicalize anchored unambiguous labels to option text. MMSU counts malformed answers wrong although its upstream scorer drops them. Raw-parsed accuracy is a separate diagnostic.

	Tiny reuses historical native-vLLM BF16 reference against the corrected NVFP4 candidate: 16 routed NVFP4 layers, with BF16 routed layers 5, 10, 15, 18, 20, 21 and 22. Both arms used input limit 8,192, client concurrency 512 and stream interval 8 on separate schedules. The HF-only RoPE correction does not imply an HF RoPE error in the native BF16 reference. Schedule-dependent greedy responses and reused outcomes prevent isolated quantization attribution or sub-0.5 pp reproducibility certification.

	`results/quality.csv` uses percentages; deltas and intervals use percentage points. MC intervals are conservative exact discordance-marginal Clopper-Pearson differences with exact two-sided McNemar p-values and paired counts. Tiny's Holm p-values cover only four MC tests. ASR uses 20,000 paired percentile corpus-rate bootstrap replicates, seed 20261025: utterances for FLEURS (speaker IDs unavailable), speakers for LibriSpeech. Tiny's LibriSpeech speaker clusters number 40/33. The CSV reports the primary speaker sensitivity; source reports also retain utterance sensitivity. Nominal 95% and Bonferroni-eight columns assume exchangeable units; MC requires independent pairs and bootstrap coverage is approximate. Multiplicity corrections do not repair selection bias; zero-width intervals do not establish equivalence.

	Hardware: NVIDIA RTX PRO 6000 Blackwell, exact SM120 (12.0), measured physical97887MiB/95.593GiB; host RAM about125GiB. Serving: torch 2.13.0, vllm 0.30.0, vllm_omni 0.1.dev3352+g41edeb7fe, flashinfer_python 0.6.18.post1, triton 3.7.1. Preparation: torch 2.8.0, transformers 5.14.1, accelerate 1.10.1, nvidia_modelopt 0.47.0, safetensors 0.8.0. Serving/preparation Torch versions describe distinct environments. Required serving plugins: `ming_nvfp4`, `vllm_omni_register_models`. Native `VLLM_CUTLASS` executes NVFP4 on SM120; Tiny's retained BF16 routed layers use Triton. Portability was not validated by these reports.

	Quality profiles: Tiny context 16,384 / server capacity 256, graphs/async, queued client concurrency 512.

	`results/performance.csv` gives median output tokens/s over three fixed-workload trials; audio means audio input/text output, not speech synthesis. Tiny: context 16,384 / capacity 1,024; text client concurrency 512 with 256 output tokens, audio concurrency 1,024 with 128 output tokens; 2,048 warmup plus 2,048 timed requests each.

	Thinker model allocation is Tiny BF16 15.97 GiB and Tiny NVFP4 9.50 GiB. It excludes KV cache, workspace, CUDA graphs and other allocations; it is neither checkpoint size nor total GPU memory. Workload hashes identify fixed inputs. `evidence_id` resolves to relative local artifact names/SHA-256 in `provenance.json`; original source files remain unchanged and are not distributed. This export checks report hashes/completion, counts, deltas and shared primary values. It does not rerun inference, rehash bulk weights/audio or newly validate every prediction. Only aggregate data is included.

	Benchmark references from the local catalog: [FLEURS](https://huggingface.co/datasets/google/fleurs), [LibriSpeech](https://huggingface.co/datasets/openslr/librispeech_asr), [MMAR](https://github.com/ddlBoJack/MMAR), [MMAU](https://github.com/Sakshi113/MMAU/blob/110127f54c0dfba3faa5ec9feee4a7e4148679c5/README.md), [MMSU](https://github.com/dingdongwang/MMSU), [MMLU-Pro](https://github.com/TIGER-AI-Lab/MMLU-Pro/blob/f418b116db00b065c2aea046518d8fcf74d39872/README.md). Source datasets retain their own terms; no examples are redistributed.