Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel. overall n=6,000 acc=0.650 ece=0.060 nll=0.851 T={choice:0.818, noul:0.818, score:1.192} by_kind ┏━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┓ ┃ group ┃ n ┃ accuracy ┃ ece ┃ mean conf ┃ ┡━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━┩ │ noul │ 1,104 │ 0.625 │ 0.194 │ 0.441 │ │ choice │ 3,467 │ 0.730 │ 0.038 │ 0.710 │ │ score │ 1,429 │ 0.474 │ 0.013 │ 0.463 │ └────────┴───────┴──────────┴───────┴───────────┘ by_task ┏━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┓ ┃ group ┃ n ┃ accuracy ┃ ece ┃ mean conf ┃ ┡━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━┩ │ emotion │ 1,714 │ 0.567 │ 0.051 │ 0.526 │ │ hate_severity │ 1,429 │ 0.474 │ 0.013 │ 0.463 │ │ massive_intent │ 1,753 │ 0.889 │ 0.029 │ 0.890 │ │ sarcasm │ 1,104 │ 0.625 │ 0.194 │ 0.441 │ └────────────────┴───────┴──────────┴───────┴───────────┘ by_confidence_band ┏━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┓ ┃ group ┃ n ┃ accuracy ┃ ece ┃ mean conf ┃ ┡━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━┩ │ low(<0.5) │ 2,784 │ 0.474 │ 0.105 │ 0.369 │ │ mid(0.5-0.9) │ 1,847 │ 0.682 │ 0.029 │ 0.677 │ │ high(>=0.9) │ 1,369 │ 0.964 │ 0.011 │ 0.975 │ └──────────────┴───────┴──────────┴───────┴───────────┘ report -> reports/recipe_heldout.json Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel. checkpoints/v23-v21b-s5 (0 rows over the window skipped) boardgame 0.821 (n=900) contractnli 0.865 (n=1,026) hotpotqa (held out) 0.746 (n=959) musique 0.882 (n=1,199) musique, answerable 0.886 (n=599) musique, unanswerable 0.877 (n=600) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel. checkpoints/v23-v21b-s5 (0 rows over the window skipped) gen:all_conditions 0.824 (n=34) gen:dates 0.871 (n=31) gen:exception 0.946 (n=37) gen:lookup_chain 0.800 (n=40) gen:numbers 0.848 (n=33) gen:precedence 0.917 (n=36) gen:rubric 0.794 (n=34) gen:surface_trap 0.958 (n=48) gen:tradeoff 0.700 (n=40) gen:underdetermined 0.765 (n=17) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel. checkpoints/v23-v21b-s5 (0 rows over the window skipped) gen:all_conditions 0.889 (n=9) gen:borderline_rubric 0.538 (n=13) gen:cross_artefact 0.677 (n=31) gen:dates 0.800 (n=10) gen:definition_scope 0.667 (n=12) gen:exception 1.000 (n=15) gen:judge_action 0.714 (n=7) gen:lookup_chain 0.700 (n=10) gen:missing_fact 0.621 (n=29) gen:nested_exception 0.692 (n=13) gen:numbers 1.000 (n=1) gen:precedence 0.909 (n=11) gen:rubric 0.667 (n=3) gen:running_total 0.826 (n=23) gen:surface_trap 0.000 (n=1) gen:tie_break 0.826 (n=23) gen:underdetermined 1.000 (n=1) gen:version_in_force 0.704 (n=27) gen:withdrawn_instruction 0.750 (n=8) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel. checkpoints/v23-v21b-s5 (0 rows over the window skipped) adequacy_hs2 0.739 (n=234) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel. checkpoints/v23-v21b-s5 (0 rows over the window skipped) gen:adequacy 0.788 (n=302) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel. {"file": "data/para_pairs_v16_eval.jsonl", "pairs": 339, "acc_first": 0.8525, "acc_second": 0.8437, "same_answer": 0.9794, "both_right": 0.8378} {"file": "data/para_pairs_v18_eval.jsonl", "pairs": 242, "acc_first": 0.7438, "acc_second": 0.7479, "same_answer": 0.9835, "both_right": 0.7397} {"file": "data/flips_v20_eval.jsonl", "pairs": 60, "acc_first": 0.5, "acc_second": 0.65, "same_answer": 0.85, "both_right": 0.15}