Upload folder using huggingface_hub

c7ed289 verified 2 months ago

5.03 kB

[09:27:23] longeval: T=16384 prompt_len=16360 max_new=24 n=100 greedy=1 gpus=0,1,2,3
[09:27:23] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b3val_haystack.bin
[09:27:23] prompt signature changed -- regenerating prompts and clearing state
[09:27:23] generating prompts: niah_single
  niah_single      100 x 16360 tok -> lenb_len_16360/prompts/niah_single_prompts.bin
[09:27:24] generating prompts: val_backward
  val_backward     100 x 16360 tok -> lenb_len_16360/prompts/val_backward_prompts.bin
[09:27:24] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
[09:27:24] 6 job(s) to run
[09:27:24]   [gpu 0] armA_c / niah_single
[09:27:24]   [gpu 1] armA_c / val_backward
[09:27:24]   [gpu 2] l1 / niah_single
[09:27:24]   [gpu 3] l1 / val_backward
[09:27:24]   [gpu 0] l2 / niah_single
[09:27:24]   [gpu 1] l2 / val_backward
[09:45:18]   scoring armA_c / niah_single

=== niah_single  (prompt_len=16360, n=100) ===
  accuracy: 33/100 = 33.0%
    depth      acc   (needle position in context)
     0.00   0/20    0.0%
     0.25   0/20    0.0%
     0.50   4/20   20.0%
     0.75  11/20   55.0%
     1.00  18/20   90.0%

  --- 2 sample generations ---
  [MISS] expect=8056020
         got= the number of the Ghost-Type Pokemon that has been summoned to the Zangoose's lair.\n
  [MISS] expect=5343902
         got= the number of the “original sin” of the so-called primitive accumulation, which is the so-called

  wrote lenb_len_16360/scores/armA_c_niah_single.csv
[09:45:18]   scoring armA_c / val_backward

=== val_backward  (prompt_len=16360, n=100) ===
  accuracy: 1/100 = 1.0%
    depth      acc   (needle position in context)
     0.00   0/20    0.0%
     0.25   0/20    0.0%
     0.50   0/20    0.0%
     0.75   0/20    0.0%
     1.00   1/20    5.0%

  --- 2 sample generations ---
  [MISS] expect=yarrow
         got= the Ghost Dog Pokemon.\n
  [MISS] expect=cactus
         got= the “West”.\n

  wrote lenb_len_16360/scores/armA_c_val_backward.csv
[09:45:18]   scoring l1 / niah_single

=== niah_single  (prompt_len=16360, n=100) ===
  accuracy: 39/100 = 39.0%
    depth      acc   (needle position in context)
     0.00   0/20    0.0%
     0.25   0/20    0.0%
     0.50   6/20   30.0%
     0.75  15/20   75.0%
     1.00  18/20   90.0%

  --- 2 sample generations ---
  [MISS] expect=8056020
         got= the number of the Ghost Dog Pokemon that can be summoned to the Zangoose's lair.\n
  [MISS] expect=5343902
         got= the number of the “centripetal” or “internal” elements of the “West

  wrote lenb_len_16360/scores/l1_niah_single.csv
[09:45:19]   scoring l1 / val_backward

=== val_backward  (prompt_len=16360, n=100) ===
  accuracy: 1/100 = 1.0%
    depth      acc   (needle position in context)
     0.00   0/20    0.0%
     0.25   0/20    0.0%
     0.50   0/20    0.0%
     0.75   0/20    0.0%
     1.00   1/20    5.0%

  --- 2 sample generations ---
  [MISS] expect=yarrow
         got= the Ghost Dog Pokemon.\n
  [MISS] expect=cactus
         got= the “exact number” of the “exact number” of the “exact number

  wrote lenb_len_16360/scores/l1_val_backward.csv
[09:45:19]   scoring l2 / niah_single

=== niah_single  (prompt_len=16360, n=100) ===
  accuracy: 48/100 = 48.0%
    depth      acc   (needle position in context)
     0.00   0/20    0.0%
     0.25   6/20   30.0%
     0.50  10/20   50.0%
     0.75  14/20   70.0%
     1.00  18/20   90.0%

  --- 2 sample generations ---
  [MISS] expect=8056020
         got= the number of the Zangoose's soul.\n
  [MISS] expect=5343902
         got= the number of the “West” and the “Rest” in the world.\n

  wrote lenb_len_16360/scores/l2_niah_single.csv
[09:45:19]   scoring l2 / val_backward

=== val_backward  (prompt_len=16360, n=100) ===
  accuracy: 0/100 = 0.0%
    depth      acc   (needle position in context)
     0.00   0/20    0.0%
     0.25   0/20    0.0%
     0.50   0/20    0.0%
     0.75   0/20    0.0%
     1.00   0/20    0.0%

  --- 2 sample generations ---
  [MISS] expect=yarrow
         got= the Seviper's treasure bag.\n
  [MISS] expect=cactus
         got= the “national question”.\n

  wrote lenb_len_16360/scores/l2_val_backward.csv
[09:45:19] building summary

=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
  task                 armA_c       l1       l2
  niah_single            33.0     39.0     48.0
  val_backward            1.0      1.0      0.0

  degenerate (<=1% for every model): ['val_backward']

=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
  depth             armA_c              l1              l2
  0.00           0.0+- 8.1       0.0+- 8.1       0.0+- 8.1
  0.25           0.0+- 8.1       0.0+- 8.1      30.0+-18.7
  0.50          20.0+-16.8      30.0+-18.7      50.0+-20.1
  0.75          55.0+-20.0      75.0+-17.8      70.0+-18.7
  1.00          90.0+-13.7      90.0+-13.7      90.0+-13.7

  wrote lenb_len_16360/summary.csv and lenb_len_16360/depth_summary.csv
[09:45:19] DONE -> lenb_len_16360/summary.csv

	[09:27:23] longeval: T=16384 prompt_len=16360 max_new=24 n=100 greedy=1 gpus=0,1,2,3
	[09:27:23] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b3val_haystack.bin
	[09:27:23] prompt signature changed -- regenerating prompts and clearing state
	[09:27:23] generating prompts: niah_single
	niah_single 100 x 16360 tok -> lenb_len_16360/prompts/niah_single_prompts.bin
	[09:27:24] generating prompts: val_backward
	val_backward 100 x 16360 tok -> lenb_len_16360/prompts/val_backward_prompts.bin
	[09:27:24] fan-out: 4 GPU(s), NP=1 -> 8 concurrent job(s)
	[09:27:24] 6 job(s) to run
	[09:27:24] [gpu 0] armA_c / niah_single
	[09:27:24] [gpu 1] armA_c / val_backward
	[09:27:24] [gpu 2] l1 / niah_single
	[09:27:24] [gpu 3] l1 / val_backward
	[09:27:24] [gpu 0] l2 / niah_single
	[09:27:24] [gpu 1] l2 / val_backward
	[09:45:18] scoring armA_c / niah_single

	=== niah_single (prompt_len=16360, n=100) ===
	accuracy: 33/100 = 33.0%
	depth acc (needle position in context)
	0.00 0/20 0.0%
	0.25 0/20 0.0%
	0.50 4/20 20.0%
	0.75 11/20 55.0%
	1.00 18/20 90.0%

	--- 2 sample generations ---
	[MISS] expect=8056020
	got= the number of the Ghost-Type Pokemon that has been summoned to the Zangoose's lair.\n
	[MISS] expect=5343902
	got= the number of the “original sin” of the so-called primitive accumulation, which is the so-called

	wrote lenb_len_16360/scores/armA_c_niah_single.csv
	[09:45:18] scoring armA_c / val_backward

	=== val_backward (prompt_len=16360, n=100) ===
	accuracy: 1/100 = 1.0%
	depth acc (needle position in context)
	0.00 0/20 0.0%
	0.25 0/20 0.0%
	0.50 0/20 0.0%
	0.75 0/20 0.0%
	1.00 1/20 5.0%

	--- 2 sample generations ---
	[MISS] expect=yarrow
	got= the Ghost Dog Pokemon.\n
	[MISS] expect=cactus
	got= the “West”.\n

	wrote lenb_len_16360/scores/armA_c_val_backward.csv
	[09:45:18] scoring l1 / niah_single

	=== niah_single (prompt_len=16360, n=100) ===
	accuracy: 39/100 = 39.0%
	depth acc (needle position in context)
	0.00 0/20 0.0%
	0.25 0/20 0.0%
	0.50 6/20 30.0%
	0.75 15/20 75.0%
	1.00 18/20 90.0%

	--- 2 sample generations ---
	[MISS] expect=8056020
	got= the number of the Ghost Dog Pokemon that can be summoned to the Zangoose's lair.\n
	[MISS] expect=5343902
	got= the number of the “centripetal” or “internal” elements of the “West

	wrote lenb_len_16360/scores/l1_niah_single.csv
	[09:45:19] scoring l1 / val_backward

	=== val_backward (prompt_len=16360, n=100) ===
	accuracy: 1/100 = 1.0%
	depth acc (needle position in context)
	0.00 0/20 0.0%
	0.25 0/20 0.0%
	0.50 0/20 0.0%
	0.75 0/20 0.0%
	1.00 1/20 5.0%

	--- 2 sample generations ---
	[MISS] expect=yarrow
	got= the Ghost Dog Pokemon.\n
	[MISS] expect=cactus
	got= the “exact number” of the “exact number” of the “exact number

	wrote lenb_len_16360/scores/l1_val_backward.csv
	[09:45:19] scoring l2 / niah_single

	=== niah_single (prompt_len=16360, n=100) ===
	accuracy: 48/100 = 48.0%
	depth acc (needle position in context)
	0.00 0/20 0.0%
	0.25 6/20 30.0%
	0.50 10/20 50.0%
	0.75 14/20 70.0%
	1.00 18/20 90.0%

	--- 2 sample generations ---
	[MISS] expect=8056020
	got= the number of the Zangoose's soul.\n
	[MISS] expect=5343902
	got= the number of the “West” and the “Rest” in the world.\n

	wrote lenb_len_16360/scores/l2_niah_single.csv
	[09:45:19] scoring l2 / val_backward

	=== val_backward (prompt_len=16360, n=100) ===
	accuracy: 0/100 = 0.0%
	depth acc (needle position in context)
	0.00 0/20 0.0%
	0.25 0/20 0.0%
	0.50 0/20 0.0%
	0.75 0/20 0.0%
	1.00 0/20 0.0%

	--- 2 sample generations ---
	[MISS] expect=yarrow
	got= the Seviper's treasure bag.\n
	[MISS] expect=cactus
	got= the “national question”.\n

	wrote lenb_len_16360/scores/l2_val_backward.csv
	[09:45:19] building summary

	=== per-task accuracy % (secondary/descriptive: ~10 per depth cell) ===
	task armA_c l1 l2
	niah_single 33.0 39.0 48.0
	val_backward 1.0 1.0 0.0

	degenerate (<=1% for every model): ['val_backward']

	=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
	depth armA_c l1 l2
	0.00 0.0+- 8.1 0.0+- 8.1 0.0+- 8.1
	0.25 0.0+- 8.1 0.0+- 8.1 30.0+-18.7
	0.50 20.0+-16.8 30.0+-18.7 50.0+-20.1
	0.75 55.0+-20.0 75.0+-17.8 70.0+-18.7
	1.00 90.0+-13.7 90.0+-13.7 90.0+-13.7

	wrote lenb_len_16360/summary.csv and lenb_len_16360/depth_summary.csv
	[09:45:19] DONE -> lenb_len_16360/summary.csv