Hugging Face's logo

unparallelled
/
BluScriptCP

Safetensors
Model card Files
xet
 Community
BluScriptCP / logs /longeval_16k /master.log
unparallelled's picture
unparallelled
Upload folder using huggingface_hub
c18743b verified
Raw History Blame Contribute Delete
14 kB
[11:01:43] longeval: T=16384 prompt_len=16360 max_new=24 n=100 greedy=1 gpus=0,1,2,3
[11:01:43] haystack: Data_Loader/Data/flux_val_000000.bin
[11:01:43] prompt signature changed -- regenerating prompts and clearing state
[11:01:43] generating prompts: niah_single
niah_single 100 x 16360 tok -> longeval_runs/prompts/niah_single_prompts.bin
[11:01:44] generating prompts: niah_multikey
niah_multikey 100 x 16360 tok -> longeval_runs/prompts/niah_multikey_prompts.bin
[11:01:44] generating prompts: niah_multivalue
niah_multivalue 100 x 16360 tok -> longeval_runs/prompts/niah_multivalue_prompts.bin
[11:01:45] generating prompts: vt
vt 100 x 16360 tok -> longeval_runs/prompts/vt_prompts.bin
[11:01:45] generating prompts: val_backward
val_backward 100 x 16360 tok -> longeval_runs/prompts/val_backward_prompts.bin
[11:01:45] fan-out: 4 GPU(s), NP=1 -> 4 concurrent job(s)
[11:01:45] 20 job(s) to run
[11:01:45] [gpu 0] base / niah_single
[11:01:45] [gpu 1] base / niah_multikey
[11:01:45] [gpu 2] base / niah_multivalue
[11:01:45] [gpu 3] base / vt
[11:16:16] [gpu 0] base / val_backward
[11:16:31] [gpu 1] arm_a / niah_single
[11:16:41] [gpu 2] arm_a / niah_multikey
[11:20:51] [gpu 3] arm_a / niah_multivalue
[11:29:10] [gpu 0] arm_a / vt
[11:29:16] [gpu 1] arm_a / val_backward
[11:29:33] [gpu 2] arm_b / niah_single
[11:33:38] longeval: T=16384 prompt_len=16360 max_new=24 n=100 greedy=1 gpus=0,1,2,3
[11:33:38] haystack: Data_Loader/Data/flux_val_000000.bin
[11:33:38] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
[11:33:38] base/niah_single already generated
[11:33:38] base/niah_multikey already generated
[11:33:38] base/niah_multivalue already generated
[11:33:38] base/vt already generated
[11:33:38] base/val_backward already generated
[11:33:38] arm_a/niah_single already generated
[11:33:38] arm_a/niah_multikey already generated
[11:33:38] 13 job(s) to run
[11:33:38] [gpu 0] arm_a / niah_multivalue
[11:33:38] [gpu 1] arm_a / vt
[11:33:38] [gpu 2] arm_a / val_backward
[11:33:38] [gpu 3] arm_b / niah_single
[11:33:38] [gpu 0] arm_b / niah_multikey
[11:33:38] [gpu 1] arm_b / niah_multivalue
[11:33:38] [gpu 2] arm_b / vt
[11:33:38] [gpu 3] arm_b / val_backward
[11:54:02] [gpu 0] arm_c / niah_single
[11:54:15] [gpu 1] arm_c / niah_multikey
[11:55:52] [gpu 2] arm_c / niah_multivalue
[11:56:31] [gpu 3] arm_c / vt
[11:56:41] [gpu 0] arm_c / val_backward
[12:16:44] scoring base / niah_single
=== niah_single (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=8056020
got=:\nQuestion: What is the Devil?\n
[MISS] expect=5343902
got= not known, what the period modes: transportation of the cactus?\n
wrote longeval_runs/scores/base_niah_single.csv
[12:16:44] scoring base / niah_multikey
=== niah_multikey (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=6088743
got= the first, and the magic number is made by the same.\n
[MISS] expect=9934935
got= the following:\n
wrote longeval_runs/scores/base_niah_multikey.csv
[12:16:44] scoring base / niah_multivalue
=== niah_multivalue (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=8056020|1679215|5343902
got= the magic, from zero-time, the magic.\nQuestion: What is the wild?\n
[MISS] expect=8995970|7007071|4664860
got= the same thing.\nWhere does the religious events of the same age are tundra?\n
wrote longeval_runs/scores/base_niah_multivalue.csv
[12:16:44] scoring base / vt
=== vt (prompt_len=16360, n=100) ===
accuracy: 1/100 = 1.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
0.80 1/20 5.0%
--- 2 sample generations ---
[MISS] expect=5370335
got= “VAR”, “California”, “California night” is a perfect setting.
[MISS] expect=6319163
got= VAR TV, VAR is not just like VAR.\n
wrote longeval_runs/scores/base_vt.csv
[12:16:45] scoring base / val_backward
=== val_backward (prompt_len=16360, n=100) ===
accuracy: 1/100 = 1.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 1/20 5.0%
--- 2 sample generations ---
[MISS] expect=yarrow
got= a magical, but we use 80560 to make a magical, open and peaceful. You may get the magic of
[MISS] expect=cactus
got= the special lifestyle.\n
wrote longeval_runs/scores/base_val_backward.csv
[12:16:45] scoring arm_a / niah_single
=== niah_single (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=8056020
got= the same.\n
[MISS] expect=5343902
got= the special of the cactus.\n
wrote longeval_runs/scores/arm_a_niah_single.csv
[12:16:45] scoring arm_a / niah_multikey
=== niah_multikey (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=6088743
got= the same.\n
[MISS] expect=9934935
got= the same for the car.\n
wrote longeval_runs/scores/arm_a_niah_multikey.csv
[12:16:45] scoring arm_a / niah_multivalue
=== niah_multivalue (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=8056020|1679215|5343902
got= the same.\nQuestion: Juan Rodriguez Cabrillo\nNorto Cabrillo\n
[MISS] expect=8995970|7007071|4664860
got= the same for tundra.\nQuestion: The other tundra is the same of the tundra\n
wrote longeval_runs/scores/arm_a_niah_multivalue.csv
[12:16:45] scoring arm_a / vt
=== vt (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
0.80 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=5370335
got=\n“VAR HOL88,” the VAR, and the VAR, and the VAR,
[MISS] expect=6319163
got=\nVAR YAR\n
wrote longeval_runs/scores/arm_a_vt.csv
[12:16:46] scoring arm_a / val_backward
=== val_backward (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=yarrow
got= the one of the the same.\n
[MISS] expect=cactus
got= the\nHow many of the special, the same of thin old grey stone Georgian mansion is the\n
wrote longeval_runs/scores/arm_a_val_backward.csv
[12:16:46] scoring arm_b / niah_single
=== niah_single (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=8056020
got= the, and the only fight is the same.\n
[MISS] expect=5343902
got= the.\nDo the language cactus, that is the other.\n
wrote longeval_runs/scores/arm_b_niah_single.csv
[12:16:46] scoring arm_b / niah_multikey
=== niah_multikey (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=6088743
got= the same. Steam was the stone in the C.\n
[MISS] expect=9934935
got= the\nHow rumorian\n
wrote longeval_runs/scores/arm_b_niah_multikey.csv
[12:16:46] scoring arm_b / niah_multivalue
=== niah_multivalue (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=8056020|1679215|5343902
got= the.\nThe name silk in the e-Crookeeng, the<|endoftext|>Possar, and the name of
[MISS] expect=8995970|7007071|4664860
got= the.\nPhrenoettes of the same\nTermany measurements of the same\n
wrote longeval_runs/scores/arm_b_niah_multivalue.csv
[12:16:46] scoring arm_b / vt
=== vt (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
0.80 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=5370335
got= a lot of the experience.\n
[MISS] expect=6319163
got=\nDebt the language of VAR. The Cock House of the same\n
wrote longeval_runs/scores/arm_b_vt.csv
[12:16:47] scoring arm_b / val_backward
=== val_backward (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=yarrow
got= the most.\n
[MISS] expect=cactus
got=,\nPropver Indian Old Women\n
wrote longeval_runs/scores/arm_b_val_backward.csv
[12:16:47] scoring arm_c / niah_single
=== niah_single (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=8056020
got= the same, the best time.\n
[MISS] expect=5343902
got= the\nFirst of the book\n
wrote longeval_runs/scores/arm_c_niah_single.csv
[12:16:47] scoring arm_c / niah_multikey
=== niah_multikey (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=6088743
got= the\nThe first time of the Cabrillo\n
[MISS] expect=9934935
got= a time.\n
wrote longeval_runs/scores/arm_c_niah_multikey.csv
[12:16:47] scoring arm_c / niah_multivalue
=== niah_multivalue (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=8056020|1679215|5343902
got= the first time.\nThe timberline in the first of the world, the the first of the world, the the
[MISS] expect=8995970|7007071|4664860
got= the way!\nIs the way a tundra\nIf you are the tundra\n
wrote longeval_runs/scores/arm_c_niah_multivalue.csv
[12:16:47] scoring arm_c / vt
=== vt (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
0.80 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=5370335
got= the first time\n
[MISS] expect=6319163
got= V-!\n
wrote longeval_runs/scores/arm_c_vt.csv
[12:16:48] scoring arm_c / val_backward
=== val_backward (prompt_len=16360, n=100) ===
accuracy: 0/100 = 0.0%
depth acc (needle position in context)
0.00 0/20 0.0%
0.25 0/20 0.0%
0.50 0/20 0.0%
0.75 0/20 0.0%
1.00 0/20 0.0%
--- 2 sample generations ---
[MISS] expect=yarrow
got= the same.\n
[MISS] expect=cactus
got= the.\nQuestion:\n
wrote longeval_runs/scores/arm_c_val_backward.csv
[12:16:48] building summary
=== per-task accuracy % (secondary/descriptive: ~10 per depth cell) ===
task arm_a arm_b arm_c base
niah_multikey 0.0 0.0 0.0 0.0
niah_multivalue 0.0 0.0 0.0 0.0
niah_single 0.0 0.0 0.0 0.0
val_backward 0.0 0.0 0.0 1.0
vt 0.0 0.0 0.0 1.0
EXCLUDED from pooling (<=1% for every model): ['niah_multikey', 'niah_multivalue', 'niah_single', 'val_backward', 'vt']
=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
depth arm_a arm_b arm_c base
wrote longeval_runs/summary.csv and longeval_runs/depth_summary.csv
[12:16:48] DONE -> longeval_runs/summary.csv