| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
| [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel. |
| overall n=6,000 acc=0.650 ece=0.060 nll=0.851 T={choice:0.818, noul:0.818, |
| score:1.192} |
| by_kind |
| ┏━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┓ |
| ┃ group ┃ n ┃ accuracy ┃ ece ┃ mean conf ┃ |
| ┡━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━┩ |
| │ noul │ 1,104 │ 0.625 │ 0.194 │ 0.441 │ |
| │ choice │ 3,467 │ 0.730 │ 0.038 │ 0.710 │ |
| │ score │ 1,429 │ 0.474 │ 0.013 │ 0.463 │ |
| └────────┴───────┴──────────┴───────┴───────────┘ |
| by_task |
| ┏━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┓ |
| ┃ group ┃ n ┃ accuracy ┃ ece ┃ mean conf ┃ |
| ┡━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━┩ |
| │ emotion │ 1,714 │ 0.567 │ 0.051 │ 0.526 │ |
| │ hate_severity │ 1,429 │ 0.474 │ 0.013 │ 0.463 │ |
| │ massive_intent │ 1,753 │ 0.889 │ 0.029 │ 0.890 │ |
| │ sarcasm │ 1,104 │ 0.625 │ 0.194 │ 0.441 │ |
| └────────────────┴───────┴──────────┴───────┴───────────┘ |
| by_confidence_band |
| ┏━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┓ |
| ┃ group ┃ n ┃ accuracy ┃ ece ┃ mean conf ┃ |
| ┡━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━┩ |
| │ low(<0.5) │ 2,784 │ 0.474 │ 0.105 │ 0.369 │ |
| │ mid(0.5-0.9) │ 1,847 │ 0.682 │ 0.029 │ 0.677 │ |
| │ high(>=0.9) │ 1,369 │ 0.964 │ 0.011 │ 0.975 │ |
| └──────────────┴───────┴──────────┴───────┴───────────┘ |
| report -> reports/recipe_heldout.json |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
| [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel. |
| checkpoints/v23-v21b-s5 (0 rows over the window skipped) |
| boardgame 0.821 (n=900) |
| contractnli 0.865 (n=1,026) |
| hotpotqa (held out) 0.746 (n=959) |
| musique 0.882 (n=1,199) |
| musique, answerable 0.886 (n=599) |
| musique, unanswerable 0.877 (n=600) |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
| [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel. |
| checkpoints/v23-v21b-s5 (0 rows over the window skipped) |
| gen:all_conditions 0.824 (n=34) |
| gen:dates 0.871 (n=31) |
| gen:exception 0.946 (n=37) |
| gen:lookup_chain 0.800 (n=40) |
| gen:numbers 0.848 (n=33) |
| gen:precedence 0.917 (n=36) |
| gen:rubric 0.794 (n=34) |
| gen:surface_trap 0.958 (n=48) |
| gen:tradeoff 0.700 (n=40) |
| gen:underdetermined 0.765 (n=17) |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
| [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel. |
| checkpoints/v23-v21b-s5 (0 rows over the window skipped) |
| gen:all_conditions 0.889 (n=9) |
| gen:borderline_rubric 0.538 (n=13) |
| gen:cross_artefact 0.677 (n=31) |
| gen:dates 0.800 (n=10) |
| gen:definition_scope 0.667 (n=12) |
| gen:exception 1.000 (n=15) |
| gen:judge_action 0.714 (n=7) |
| gen:lookup_chain 0.700 (n=10) |
| gen:missing_fact 0.621 (n=29) |
| gen:nested_exception 0.692 (n=13) |
| gen:numbers 1.000 (n=1) |
| gen:precedence 0.909 (n=11) |
| gen:rubric 0.667 (n=3) |
| gen:running_total 0.826 (n=23) |
| gen:surface_trap 0.000 (n=1) |
| gen:tie_break 0.826 (n=23) |
| gen:underdetermined 1.000 (n=1) |
| gen:version_in_force 0.704 (n=27) |
| gen:withdrawn_instruction 0.750 (n=8) |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
| [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel. |
| checkpoints/v23-v21b-s5 (0 rows over the window skipped) |
| adequacy_hs2 0.739 (n=234) |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
| [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel. |
| checkpoints/v23-v21b-s5 (0 rows over the window skipped) |
| gen:adequacy 0.788 (n=302) |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
| [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel. |
| {"file": "data/para_pairs_v16_eval.jsonl", "pairs": 339, "acc_first": 0.8525, "acc_second": 0.8437, "same_answer": 0.9794, "both_right": 0.8378} |
| {"file": "data/para_pairs_v18_eval.jsonl", "pairs": 242, "acc_first": 0.7438, "acc_second": 0.7479, "same_answer": 0.9835, "both_right": 0.7397} |
| {"file": "data/flips_v20_eval.jsonl", "pairs": 60, "acc_first": 0.5, "acc_second": 0.65, "same_answer": 0.85, "both_right": 0.15} |
|
|
