[11:01:43] longeval: T=16384 prompt_len=16360 max_new=24 n=100 greedy=1 gpus=0,1,2,3 [11:01:43] haystack: Data_Loader/Data/flux_val_000000.bin [11:01:43] prompt signature changed -- regenerating prompts and clearing state [11:01:43] generating prompts: niah_single niah_single 100 x 16360 tok -> longeval_runs/prompts/niah_single_prompts.bin [11:01:44] generating prompts: niah_multikey niah_multikey 100 x 16360 tok -> longeval_runs/prompts/niah_multikey_prompts.bin [11:01:44] generating prompts: niah_multivalue niah_multivalue 100 x 16360 tok -> longeval_runs/prompts/niah_multivalue_prompts.bin [11:01:45] generating prompts: vt vt 100 x 16360 tok -> longeval_runs/prompts/vt_prompts.bin [11:01:45] generating prompts: val_backward val_backward 100 x 16360 tok -> longeval_runs/prompts/val_backward_prompts.bin [11:01:45] fan-out: 4 GPU(s), NP=1 -> 4 concurrent job(s) [11:01:45] 20 job(s) to run [11:01:45] [gpu 0] base / niah_single [11:01:45] [gpu 1] base / niah_multikey [11:01:45] [gpu 2] base / niah_multivalue [11:01:45] [gpu 3] base / vt [11:16:16] [gpu 0] base / val_backward [11:16:31] [gpu 1] arm_a / niah_single [11:16:41] [gpu 2] arm_a / niah_multikey [11:20:51] [gpu 3] arm_a / niah_multivalue [11:29:10] [gpu 0] arm_a / vt [11:29:16] [gpu 1] arm_a / val_backward [11:29:33] [gpu 2] arm_b / niah_single [11:33:38] longeval: T=16384 prompt_len=16360 max_new=24 n=100 greedy=1 gpus=0,1,2,3 [11:33:38] haystack: Data_Loader/Data/flux_val_000000.bin [11:33:38] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s) [11:33:38] base/niah_single already generated [11:33:38] base/niah_multikey already generated [11:33:38] base/niah_multivalue already generated [11:33:38] base/vt already generated [11:33:38] base/val_backward already generated [11:33:38] arm_a/niah_single already generated [11:33:38] arm_a/niah_multikey already generated [11:33:38] 13 job(s) to run [11:33:38] [gpu 0] arm_a / niah_multivalue [11:33:38] [gpu 1] arm_a / vt [11:33:38] [gpu 2] arm_a / val_backward [11:33:38] [gpu 3] arm_b / niah_single [11:33:38] [gpu 0] arm_b / niah_multikey [11:33:38] [gpu 1] arm_b / niah_multivalue [11:33:38] [gpu 2] arm_b / vt [11:33:38] [gpu 3] arm_b / val_backward [11:54:02] [gpu 0] arm_c / niah_single [11:54:15] [gpu 1] arm_c / niah_multikey [11:55:52] [gpu 2] arm_c / niah_multivalue [11:56:31] [gpu 3] arm_c / vt [11:56:41] [gpu 0] arm_c / val_backward [12:16:44] scoring base / niah_single === niah_single (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 1.00 0/20 0.0% --- 2 sample generations --- [MISS] expect=8056020 got=:\nQuestion: What is the Devil?\n [MISS] expect=5343902 got= not known, what the period modes: transportation of the cactus?\n wrote longeval_runs/scores/base_niah_single.csv [12:16:44] scoring base / niah_multikey === niah_multikey (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 1.00 0/20 0.0% --- 2 sample generations --- [MISS] expect=6088743 got= the first, and the magic number is made by the same.\n [MISS] expect=9934935 got= the following:\n wrote longeval_runs/scores/base_niah_multikey.csv [12:16:44] scoring base / niah_multivalue === niah_multivalue (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 1.00 0/20 0.0% --- 2 sample generations --- [MISS] expect=8056020|1679215|5343902 got= the magic, from zero-time, the magic.\nQuestion: What is the wild?\n [MISS] expect=8995970|7007071|4664860 got= the same thing.\nWhere does the religious events of the same age are tundra?\n wrote longeval_runs/scores/base_niah_multivalue.csv [12:16:44] scoring base / vt === vt (prompt_len=16360, n=100) === accuracy: 1/100 = 1.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 0.80 1/20 5.0% --- 2 sample generations --- [MISS] expect=5370335 got= “VAR”, “California”, “California night” is a perfect setting. [MISS] expect=6319163 got= VAR TV, VAR is not just like VAR.\n wrote longeval_runs/scores/base_vt.csv [12:16:45] scoring base / val_backward === val_backward (prompt_len=16360, n=100) === accuracy: 1/100 = 1.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 1.00 1/20 5.0% --- 2 sample generations --- [MISS] expect=yarrow got= a magical, but we use 80560 to make a magical, open and peaceful. You may get the magic of [MISS] expect=cactus got= the special lifestyle.\n wrote longeval_runs/scores/base_val_backward.csv [12:16:45] scoring arm_a / niah_single === niah_single (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 1.00 0/20 0.0% --- 2 sample generations --- [MISS] expect=8056020 got= the same.\n [MISS] expect=5343902 got= the special of the cactus.\n wrote longeval_runs/scores/arm_a_niah_single.csv [12:16:45] scoring arm_a / niah_multikey === niah_multikey (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 1.00 0/20 0.0% --- 2 sample generations --- [MISS] expect=6088743 got= the same.\n [MISS] expect=9934935 got= the same for the car.\n wrote longeval_runs/scores/arm_a_niah_multikey.csv [12:16:45] scoring arm_a / niah_multivalue === niah_multivalue (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 1.00 0/20 0.0% --- 2 sample generations --- [MISS] expect=8056020|1679215|5343902 got= the same.\nQuestion: Juan Rodriguez Cabrillo\nNorto Cabrillo\n [MISS] expect=8995970|7007071|4664860 got= the same for tundra.\nQuestion: The other tundra is the same of the tundra\n wrote longeval_runs/scores/arm_a_niah_multivalue.csv [12:16:45] scoring arm_a / vt === vt (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 0.80 0/20 0.0% --- 2 sample generations --- [MISS] expect=5370335 got=\n“VAR HOL88,” the VAR, and the VAR, and the VAR, [MISS] expect=6319163 got=\nVAR YAR\n wrote longeval_runs/scores/arm_a_vt.csv [12:16:46] scoring arm_a / val_backward === val_backward (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 1.00 0/20 0.0% --- 2 sample generations --- [MISS] expect=yarrow got= the one of the the same.\n [MISS] expect=cactus got= the\nHow many of the special, the same of thin old grey stone Georgian mansion is the\n wrote longeval_runs/scores/arm_a_val_backward.csv [12:16:46] scoring arm_b / niah_single === niah_single (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 1.00 0/20 0.0% --- 2 sample generations --- [MISS] expect=8056020 got= the, and the only fight is the same.\n [MISS] expect=5343902 got= the.\nDo the language cactus, that is the other.\n wrote longeval_runs/scores/arm_b_niah_single.csv [12:16:46] scoring arm_b / niah_multikey === niah_multikey (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 1.00 0/20 0.0% --- 2 sample generations --- [MISS] expect=6088743 got= the same. Steam was the stone in the C.\n [MISS] expect=9934935 got= the\nHow rumorian\n wrote longeval_runs/scores/arm_b_niah_multikey.csv [12:16:46] scoring arm_b / niah_multivalue === niah_multivalue (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 1.00 0/20 0.0% --- 2 sample generations --- [MISS] expect=8056020|1679215|5343902 got= the.\nThe name silk in the e-Crookeeng, the<|endoftext|>Possar, and the name of [MISS] expect=8995970|7007071|4664860 got= the.\nPhrenoettes of the same\nTermany measurements of the same\n wrote longeval_runs/scores/arm_b_niah_multivalue.csv [12:16:46] scoring arm_b / vt === vt (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 0.80 0/20 0.0% --- 2 sample generations --- [MISS] expect=5370335 got= a lot of the experience.\n [MISS] expect=6319163 got=\nDebt the language of VAR. The Cock House of the same\n wrote longeval_runs/scores/arm_b_vt.csv [12:16:47] scoring arm_b / val_backward === val_backward (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 1.00 0/20 0.0% --- 2 sample generations --- [MISS] expect=yarrow got= the most.\n [MISS] expect=cactus got=,\nPropver Indian Old Women\n wrote longeval_runs/scores/arm_b_val_backward.csv [12:16:47] scoring arm_c / niah_single === niah_single (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 1.00 0/20 0.0% --- 2 sample generations --- [MISS] expect=8056020 got= the same, the best time.\n [MISS] expect=5343902 got= the\nFirst of the book\n wrote longeval_runs/scores/arm_c_niah_single.csv [12:16:47] scoring arm_c / niah_multikey === niah_multikey (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 1.00 0/20 0.0% --- 2 sample generations --- [MISS] expect=6088743 got= the\nThe first time of the Cabrillo\n [MISS] expect=9934935 got= a time.\n wrote longeval_runs/scores/arm_c_niah_multikey.csv [12:16:47] scoring arm_c / niah_multivalue === niah_multivalue (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 1.00 0/20 0.0% --- 2 sample generations --- [MISS] expect=8056020|1679215|5343902 got= the first time.\nThe timberline in the first of the world, the the first of the world, the the [MISS] expect=8995970|7007071|4664860 got= the way!\nIs the way a tundra\nIf you are the tundra\n wrote longeval_runs/scores/arm_c_niah_multivalue.csv [12:16:47] scoring arm_c / vt === vt (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 0.80 0/20 0.0% --- 2 sample generations --- [MISS] expect=5370335 got= the first time\n [MISS] expect=6319163 got= V-!\n wrote longeval_runs/scores/arm_c_vt.csv [12:16:48] scoring arm_c / val_backward === val_backward (prompt_len=16360, n=100) === accuracy: 0/100 = 0.0% depth acc (needle position in context) 0.00 0/20 0.0% 0.25 0/20 0.0% 0.50 0/20 0.0% 0.75 0/20 0.0% 1.00 0/20 0.0% --- 2 sample generations --- [MISS] expect=yarrow got= the same.\n [MISS] expect=cactus got= the.\nQuestion:\n wrote longeval_runs/scores/arm_c_val_backward.csv [12:16:48] building summary === per-task accuracy % (secondary/descriptive: ~10 per depth cell) === task arm_a arm_b arm_c base niah_multikey 0.0 0.0 0.0 0.0 niah_multivalue 0.0 0.0 0.0 0.0 niah_single 0.0 0.0 0.0 0.0 val_backward 0.0 0.0 0.0 1.0 vt 0.0 0.0 0.0 1.0 EXCLUDED from pooling (<=1% for every model): ['niah_multikey', 'niah_multivalue', 'niah_single', 'val_backward', 'vt'] === PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) === depth arm_a arm_b arm_c base wrote longeval_runs/summary.csv and longeval_runs/depth_summary.csv [12:16:48] DONE -> longeval_runs/summary.csv