Add the MMLU-Pro dataset block; both records now validate
Browse filesdataset.name: MMLU-Pro (TIGER-Lab/MMLU-Pro, MIT licence, public and ungated).
sample_count is provisionally 1 pending the real figure.
Both records now pass llm-generative.schema.yaml.
Still outstanding: runtime.version (the ExecuTorch version). Unlike created_at and sample_count, it forms part of the report identity URN
urn:ai-portal:perf:{runtime.name}:{runtime.version}:{dataset.name}:{dataset.version}:{batch_size}:{target.name}:{profile}
so ingesting without it and adding it later would insert a duplicate report rather than update the existing one. Worth supplying before first ingest.
benchmarks/gemma-4-e2b-vivo-x300-bf16.yaml
CHANGED
|
@@ -12,6 +12,10 @@ context:
|
|
| 12 |
optimisations:
|
| 13 |
- XNNPACK
|
| 14 |
- KleidiAI
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
benchmark:
|
| 16 |
batch_size: 1
|
| 17 |
num_runs: 20
|
|
|
|
| 12 |
optimisations:
|
| 13 |
- XNNPACK
|
| 14 |
- KleidiAI
|
| 15 |
+
dataset:
|
| 16 |
+
name: MMLU-Pro
|
| 17 |
+
reference_url: https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
|
| 18 |
+
sample_count: 1
|
| 19 |
benchmark:
|
| 20 |
batch_size: 1
|
| 21 |
num_runs: 20
|
benchmarks/gemma-4-e2b-vivo-x300-int4.yaml
CHANGED
|
@@ -12,6 +12,10 @@ context:
|
|
| 12 |
optimisations:
|
| 13 |
- XNNPACK
|
| 14 |
- KleidiAI
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
benchmark:
|
| 16 |
batch_size: 1
|
| 17 |
num_runs: 20
|
|
|
|
| 12 |
optimisations:
|
| 13 |
- XNNPACK
|
| 14 |
- KleidiAI
|
| 15 |
+
dataset:
|
| 16 |
+
name: MMLU-Pro
|
| 17 |
+
reference_url: https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
|
| 18 |
+
sample_count: 1
|
| 19 |
benchmark:
|
| 20 |
batch_size: 1
|
| 21 |
num_runs: 20
|