DedeProGames commited on
Commit
929cfc1
·
verified ·
1 Parent(s): b9d2370

Card wording: GPQA sampling temperature; LiveBench rounding

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -76,13 +76,13 @@ Held-out loss on assistant tokens (smol-smoltalk test split): 1.791 at the end o
76
 
77
  ## Evaluation
78
 
79
- Benchmarks follow the protocol of the [Kiyo-230M-Instruct](https://huggingface.co/DedeProGames/Kiyo-230M-Instruct) card: FP16, the model's own chat template, no system prompt, new-token limits IFEval 1,024, GPQA 384, LiveBench 512, and the model's full 2,048-token context. The main setting uses temperature 0.1, top-k 50 and repetition penalty 1.05, as on the card. The second setting is the recommended one shipped in `generation_config.json`.
80
 
81
  | Benchmark | Metric | Wisp-5M-Instruct, protocol | Wisp-5M-Instruct, recommended sampling |
82
  |---|---|---:|---:|
83
  | GPQA Diamond | pass@1 (198 × 5 samples, T = 0.6) | 9.80 | not run |
84
  | IFEval | prompt-level strict | 12.38 | 11.65 |
85
- | LiveBench instruction following | strict, 200 questions | 16.52 | not run |
86
  | CaseReportBench | token F1 | not run (not on the Hub) | not run |
87
  | EOS rate (IFEval) | share of answers that stop | 0.82 | 0.98 |
88
 
 
76
 
77
  ## Evaluation
78
 
79
+ Benchmarks follow the protocol of the [Kiyo-230M-Instruct](https://huggingface.co/DedeProGames/Kiyo-230M-Instruct) card: FP16, the model's own chat template, no system prompt, new-token limits IFEval 1,024, GPQA 384, LiveBench 512, and the model's full 2,048-token context. The main setting uses temperature 0.1, top-k 50 and repetition penalty 1.05, as on the card (GPQA is sampled at T = 0.6, as on the card). The second setting is the recommended one shipped in `generation_config.json`.
80
 
81
  | Benchmark | Metric | Wisp-5M-Instruct, protocol | Wisp-5M-Instruct, recommended sampling |
82
  |---|---|---:|---:|
83
  | GPQA Diamond | pass@1 (198 × 5 samples, T = 0.6) | 9.80 | not run |
84
  | IFEval | prompt-level strict | 12.38 | 11.65 |
85
+ | LiveBench instruction following | strict, 200 questions | 16.53 | not run |
86
  | CaseReportBench | token F1 | not run (not on the Hub) | not run |
87
  | EOS rate (IFEval) | share of answers that stop | 0.82 | 0.98 |
88