BMartin1494 commited on
Commit
55a8363
·
verified ·
1 Parent(s): 11592ec

Add the MMLU-Pro dataset block; both records now validate

Browse files

dataset.name: MMLU-Pro (TIGER-Lab/MMLU-Pro, MIT licence, public and ungated).
sample_count is provisionally 1 pending the real figure.

Both records now pass llm-generative.schema.yaml.

Still outstanding: runtime.version (the ExecuTorch version). Unlike created_at and sample_count, it forms part of the report identity URN
urn:ai-portal:perf:{runtime.name}:{runtime.version}:{dataset.name}:{dataset.version}:{batch_size}:{target.name}:{profile}
so ingesting without it and adding it later would insert a duplicate report rather than update the existing one. Worth supplying before first ingest.

benchmarks/gemma-4-e2b-vivo-x300-bf16.yaml CHANGED
@@ -12,6 +12,10 @@ context:
12
  optimisations:
13
  - XNNPACK
14
  - KleidiAI
 
 
 
 
15
  benchmark:
16
  batch_size: 1
17
  num_runs: 20
 
12
  optimisations:
13
  - XNNPACK
14
  - KleidiAI
15
+ dataset:
16
+ name: MMLU-Pro
17
+ reference_url: https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
18
+ sample_count: 1
19
  benchmark:
20
  batch_size: 1
21
  num_runs: 20
benchmarks/gemma-4-e2b-vivo-x300-int4.yaml CHANGED
@@ -12,6 +12,10 @@ context:
12
  optimisations:
13
  - XNNPACK
14
  - KleidiAI
 
 
 
 
15
  benchmark:
16
  batch_size: 1
17
  num_runs: 20
 
12
  optimisations:
13
  - XNNPACK
14
  - KleidiAI
15
+ dataset:
16
+ name: MMLU-Pro
17
+ reference_url: https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
18
+ sample_count: 1
19
  benchmark:
20
  batch_size: 1
21
  num_runs: 20