Blame for train.log

logging stdout/stderr to research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/train.log
seed=42  flash SDP  TF32
using measured preliminary seq=1024 layout batch=16 accum=8; CPT lock remains seq=2048
train layout batch=16 accum=8 (128 seq/step, 131072 tok/step) checkpointing=off liger=True
loading tokenizer research/results/embedding_init/gemma_mapping_byteify
full token census over local corpus
using full token census from /workspace/ugrip-kd/research/scripts/embedding_initialization/results/corpus_token_counts.json (cached tokenizer /workspace/ugrip-kd/research/results/expansion/tokenizer; current research/results/embedding_init/gemma_mapping_byteify)
  am census: 476898623 tokens (1959.7 tok/doc, 243349 docs)
  ta census: 4369273220 tokens (924.0 tok/doc, 4728460 docs)
  kk census: 4464108618 tokens (1632.8 tok/doc, 2733982 docs)
  tl census: 306736374 tokens (880.3 tok/doc, 348453 docs)
  en census: 1542145528 tokens (771.1 tok/doc, 2000000 docs)
splits
  am: full 243349  eval 256  pool 243093
  ta: full 4728460  eval 256  pool 4728204
  kk: full 2733982  eval 256  pool 2733726
  tl: full 348453  eval 256  pool 348197
  en: full 2000000  eval 256  pool 1999744
mix weights {'am': 0.0446, 'ta': 0.4089, 'kk': 0.4178, 'tl': 0.0287, 'en': 0.1} (corpus-proportional; no document repeat)
lang token budgets {'am': 49803264, 'ta': 456292352, 'kk': 466195456, 'tl': 32032768, 'en': 111591424}
packing via HuggingFace datasets.map (batched, num_proc; not cached)
    am: datasets.map n=35332 batched=1024 num_proc=8

hf-map am (num_proc=8):   0%|          | 0/35332 [00:00<?, ? examples/s]
hf-map am (num_proc=8):   3%|2         | 1024/35332 [00:07<04:09, 137.64 examples/s]
hf-map am (num_proc=8):   9%|8         | 3072/35332 [00:07<01:03, 511.12 examples/s]
hf-map am (num_proc=8):  14%|#4        | 5120/35332 [00:08<00:34, 879.97 examples/s]
hf-map am (num_proc=8):  20%|##        | 7168/35332 [00:08<00:19, 1450.16 examples/s]
hf-map am (num_proc=8):  26%|##6       | 9216/35332 [00:11<00:26, 1000.86 examples/s]
hf-map am (num_proc=8):  32%|###1      | 11264/35332 [00:11<00:16, 1498.20 examples/s]
hf-map am (num_proc=8):  38%|###7      | 13312/35332 [00:12<00:11, 1955.47 examples/s]
hf-map am (num_proc=8):  41%|####      | 14336/35332 [00:12<00:09, 2104.40 examples/s]
hf-map am (num_proc=8):  43%|####3     | 15360/35332 [00:12<00:07, 2524.41 examples/s]
hf-map am (num_proc=8):  49%|####9     | 17408/35332 [00:15<00:13, 1303.45 examples/s]
hf-map am (num_proc=8):  52%|#####2    | 18432/35332 [00:15<00:10, 1585.01 examples/s]
hf-map am (num_proc=8):  55%|#####5    | 19456/35332 [00:15<00:08, 1947.46 examples/s]
hf-map am (num_proc=8):  61%|######    | 21504/35332 [00:15<00:04, 2909.92 examples/s]
hf-map am (num_proc=8):  64%|######3   | 22528/35332 [00:15<00:03, 3422.96 examples/s]
hf-map am (num_proc=8):  67%|######6   | 23552/35332 [00:16<00:03, 3747.85 examples/s]
hf-map am (num_proc=8):  70%|######9   | 24576/35332 [00:16<00:02, 4062.43 examples/s]
hf-map am (num_proc=8):  72%|#######2  | 25600/35332 [00:19<00:08, 1140.56 examples/s]
hf-map am (num_proc=8):  84%|########4 | 29696/35332 [00:19<00:02, 2679.47 examples/s]
hf-map am (num_proc=8):  87%|########6 | 30720/35332 [00:19<00:01, 2686.69 examples/s]
hf-map am (num_proc=8):  90%|########9 | 31744/35332 [00:19<00:01, 2871.15 examples/s]
hf-map am (num_proc=8):  93%|#########2| 32768/35332 [00:20<00:00, 2821.29 examples/s]
hf-map am (num_proc=8):  95%|#########4| 33409/35332 [00:20<00:00, 2900.15 examples/s]
hf-map am (num_proc=8):  96%|#########6| 34049/35332 [00:20<00:00, 3219.02 examples/s]
hf-map am (num_proc=8):  98%|#########8| 34691/35332 [00:20<00:00, 3011.10 examples/s]
hf-map am (num_proc=8): 100%|##########| 35332/35332 [00:21<00:00, 2249.48 examples/s]
hf-map am (num_proc=8): 100%|##########| 35332/35332 [00:21<00:00, 1640.80 examples/s]
    am: datasets.map num_proc=8 done 22s
    am: 25000 docs  48394736 tokens  27s
  packed am: 49804449 tokens (1.000 of budget)  rows=62355 docs=25725  pad-per-doc (HF map)
    ta: datasets.map n=667659 batched=1024 num_proc=8

hf-map ta (num_proc=8):   0%|          | 0/667659 [00:00<?, ? examples/s]
hf-map ta (num_proc=8):   0%|          | 1024/667659 [00:07<1:23:31, 133.03 examples/s]
hf-map ta (num_proc=8):   0%|          | 2048/667659 [00:07<35:16, 314.53 examples/s]  
hf-map ta (num_proc=8):   1%|          | 4096/667659 [00:08<14:04, 785.91 examples/s]
hf-map ta (num_proc=8):   1%|          | 6144/667659 [00:08<07:48, 1412.93 examples/s]
hf-map ta (num_proc=8):   1%|1         | 8192/667659 [00:08<04:51, 2258.58 examples/s]
hf-map ta (num_proc=8):   2%|1         | 10240/667659 [00:12<11:46, 930.61 examples/s]
hf-map ta (num_proc=8):   2%|1         | 11264/667659 [00:13<09:43, 1124.24 examples/s]
hf-map ta (num_proc=8):   2%|1         | 13312/667659 [00:13<06:26, 1691.19 examples/s]
hf-map ta (num_proc=8):   2%|2         | 14336/667659 [00:13<05:30, 1973.79 examples/s]
hf-map ta (num_proc=8):   2%|2         | 16384/667659 [00:13<04:29, 2417.89 examples/s]
hf-map ta (num_proc=8):   3%|2         | 17408/667659 [00:17<10:44, 1009.57 examples/s]
hf-map ta (num_proc=8):   3%|3         | 20480/667659 [00:17<06:05, 1769.33 examples/s]
hf-map ta (num_proc=8):   3%|3         | 21504/667659 [00:17<05:23, 1996.45 examples/s]
hf-map ta (num_proc=8):   3%|3         | 22528/667659 [00:18<05:09, 2085.43 examples/s]
hf-map ta (num_proc=8):   4%|3         | 24576/667659 [00:18<04:20, 2469.19 examples/s]
hf-map ta (num_proc=8):   4%|3         | 25600/667659 [00:21<09:26, 1133.33 examples/s]
hf-map ta (num_proc=8):   4%|3         | 26624/667659 [00:21<07:32, 1417.30 examples/s]
hf-map ta (num_proc=8):   4%|4         | 28672/667659 [00:21<04:59, 2131.90 examples/s]
hf-map ta (num_proc=8):   4%|4         | 29696/667659 [00:21<04:06, 2584.08 examples/s]
hf-map ta (num_proc=8):   5%|4         | 30720/667659 [00:22<04:39, 2281.78 examples/s]
hf-map ta (num_proc=8):   5%|4         | 31744/667659 [00:22<03:43, 2843.01 examples/s]
hf-map ta (num_proc=8):   5%|4         | 32768/667659 [00:22<03:48, 2773.30 examples/s]
hf-map ta (num_proc=8):   5%|5         | 33792/667659 [00:25<09:00, 1171.93 examples/s]
hf-map ta (num_proc=8):   5%|5         | 34816/667659 [00:25<07:36, 1386.96 examples/s]
hf-map ta (num_proc=8):   6%|5         | 36864/667659 [00:25<04:37, 2275.25 examples/s]
hf-map ta (num_proc=8):   6%|5         | 37888/667659 [00:25<03:45, 2789.32 examples/s]
hf-map ta (num_proc=8):   6%|5         | 38912/667659 [00:26<04:38, 2261.40 examples/s]
hf-map ta (num_proc=8):   6%|5         | 39936/667659 [00:26<03:42, 2825.94 examples/s]
hf-map ta (num_proc=8):   6%|6         | 40960/667659 [00:27<04:02, 2585.99 examples/s]
hf-map ta (num_proc=8):   6%|6         | 41984/667659 [00:29<08:26, 1234.11 examples/s]
hf-map ta (num_proc=8):   6%|6         | 43008/667659 [00:29<07:01, 1481.08 examples/s]
hf-map ta (num_proc=8):   7%|6         | 45056/667659 [00:29<04:45, 2181.82 examples/s]
hf-map ta (num_proc=8):   7%|6         | 46080/667659 [00:29<04:04, 2539.72 examples/s]
hf-map ta (num_proc=8):   7%|7         | 47104/667659 [00:30<04:22, 2367.29 examples/s]
hf-map ta (num_proc=8):   7%|7         | 49152/667659 [00:31<04:03, 2540.93 examples/s]
hf-map ta (num_proc=8):   8%|7         | 50176/667659 [00:32<07:03, 1457.61 examples/s]
hf-map ta (num_proc=8):   8%|7         | 51200/667659 [00:33<05:45, 1785.28 examples/s]
hf-map ta (num_proc=8):   8%|7         | 52224/667659 [00:33<04:46, 2149.76 examples/s]
hf-map ta (num_proc=8):   8%|7         | 53248/667659 [00:33<04:08, 2468.38 examples/s]
hf-map ta (num_proc=8):   8%|8         | 54272/667659 [00:33<03:42, 2758.85 examples/s]
hf-map ta (num_proc=8):   8%|8         | 55296/667659 [00:34<03:53, 2622.30 examples/s]
hf-map ta (num_proc=8):   9%|8         | 57344/667659 [00:35<04:20, 2346.01 examples/s]
hf-map ta (num_proc=8):   9%|8         | 58368/667659 [00:36<07:07, 1425.81 examples/s]
hf-map ta (num_proc=8):   9%|8         | 59392/667659 [00:36<05:37, 1801.96 examples/s]
hf-map ta (num_proc=8):   9%|9         | 61440/667659 [00:37<03:41, 2737.97 examples/s]
hf-map ta (num_proc=8):   9%|9         | 62464/667659 [00:37<04:14, 2376.40 examples/s]
hf-map ta (num_proc=8):  10%|9         | 64512/667659 [00:37<02:53, 3467.85 examples/s]
hf-map ta (num_proc=8):  10%|9         | 65536/667659 [00:38<04:22, 2292.68 examples/s]
hf-map ta (num_proc=8):  10%|9         | 66560/667659 [00:40<06:45, 1481.69 examples/s]
hf-map ta (num_proc=8):  10%|#         | 67584/667659 [00:40<05:23, 1855.89 examples/s]
hf-map ta (num_proc=8):  10%|#         | 68608/667659 [00:40<04:26, 2249.87 examples/s]
hf-map ta (num_proc=8):  10%|#         | 69632/667659 [00:40<03:45, 2651.25 examples/s]
hf-map ta (num_proc=8):  11%|#         | 70656/667659 [00:41<04:31, 2198.99 examples/s]
hf-map ta (num_proc=8):  11%|#         | 72704/667659 [00:41<02:53, 3428.70 examples/s]
hf-map ta (num_proc=8):  11%|#1        | 73728/667659 [00:42<04:51, 2039.28 examples/s]
hf-map ta (num_proc=8):  11%|#1        | 74752/667659 [00:44<07:10, 1376.43 examples/s]
hf-map ta (num_proc=8):  12%|#1        | 77824/667659 [00:44<04:12, 2337.10 examples/s]
hf-map ta (num_proc=8):  12%|#1        | 78848/667659 [00:45<04:04, 2407.01 examples/s]
hf-map ta (num_proc=8):  12%|#1        | 79872/667659 [00:45<03:40, 2667.60 examples/s]
hf-map ta (num_proc=8):  12%|#2        | 81920/667659 [00:46<04:29, 2175.72 examples/s]
hf-map ta (num_proc=8):  12%|#2        | 82944/667659 [00:48<06:19, 1539.10 examples/s]
hf-map ta (num_proc=8):  13%|#2        | 83968/667659 [00:48<05:10, 1878.62 examples/s]
hf-map ta (num_proc=8):  13%|#2        | 86016/667659 [00:48<04:03, 2384.41 examples/s]
hf-map ta (num_proc=8):  13%|#3        | 87040/667659 [00:49<04:04, 2377.90 examples/s]
hf-map ta (num_proc=8):  13%|#3        | 88064/667659 [00:49<03:39, 2642.93 examples/s]
hf-map ta (num_proc=8):  13%|#3        | 90112/667659 [00:50<04:37, 2080.09 examples/s]
hf-map ta (num_proc=8):  14%|#3        | 91136/667659 [00:51<05:50, 1643.44 examples/s]
hf-map ta (num_proc=8):  14%|#3        | 92160/667659 [00:52<04:56, 1942.04 examples/s]
hf-map ta (num_proc=8):  14%|#3        | 93184/667659 [00:52<03:58, 2406.08 examples/s]
hf-map ta (num_proc=8):  14%|#4        | 94208/667659 [00:52<03:18, 2883.69 examples/s]
hf-map ta (num_proc=8):  14%|#4        | 95232/667659 [00:52<03:38, 2624.45 examples/s]
hf-map ta (num_proc=8):  14%|#4        | 96256/667659 [00:53<03:07, 3040.40 examples/s]
hf-map ta (num_proc=8):  15%|#4        | 98304/667659 [00:54<04:47, 1980.29 examples/s]
hf-map ta (num_proc=8):  15%|#4        | 99328/667659 [00:56<07:33, 1253.13 examples/s]
hf-map ta (num_proc=8):  15%|#5        | 102400/667659 [00:56<04:02, 2327.00 examples/s]
hf-map ta (num_proc=8):  15%|#5        | 103424/667659 [00:57<04:22, 2145.81 examples/s]
hf-map ta (num_proc=8):  16%|#5        | 105472/667659 [00:57<03:06, 3009.65 examples/s]
hf-map ta (num_proc=8):  16%|#5        | 106496/667659 [00:59<05:50, 1601.49 examples/s]
hf-map ta (num_proc=8):  16%|#6        | 107520/667659 [00:59<05:28, 1707.23 examples/s]
hf-map ta (num_proc=8):  16%|#6        | 109568/667659 [00:59<03:31, 2638.49 examples/s]
hf-map ta (num_proc=8):  17%|#6        | 110592/667659 [01:00<03:28, 2676.06 examples/s]
hf-map ta (num_proc=8):  17%|#6        | 111616/667659 [01:00<03:57, 2339.34 examples/s]
hf-map ta (num_proc=8):  17%|#7        | 113664/667659 [01:00<02:38, 3489.23 examples/s]
hf-map ta (num_proc=8):  17%|#7        | 114688/667659 [01:02<05:52, 1567.47 examples/s]
hf-map ta (num_proc=8):  17%|#7        | 115712/667659 [01:02<04:52, 1888.70 examples/s]
hf-map ta (num_proc=8):  17%|#7        | 116736/667659 [01:03<04:09, 2205.94 examples/s]
hf-map ta (num_proc=8):  18%|#7        | 117760/667659 [01:03<03:30, 2618.40 examples/s]
hf-map ta (num_proc=8):  18%|#7        | 119808/667659 [01:04<03:39, 2495.38 examples/s]
hf-map ta (num_proc=8):  18%|#8        | 121856/667659 [01:04<02:33, 3560.45 examples/s]
hf-map ta (num_proc=8):  18%|#8        | 122880/667659 [01:05<05:01, 1809.79 examples/s]
hf-map ta (num_proc=8):  19%|#8        | 123904/667659 [01:06<04:19, 2099.28 examples/s]
hf-map ta (num_proc=8):  19%|#8        | 125952/667659 [01:06<03:01, 2989.85 examples/s]
hf-map ta (num_proc=8):  19%|#9        | 126976/667659 [01:06<02:49, 3180.95 examples/s]
hf-map ta (num_proc=8):  19%|#9        | 128000/667659 [01:07<03:07, 2878.74 examples/s]
hf-map ta (num_proc=8):  19%|#9        | 129024/667659 [01:07<03:15, 2759.14 examples/s]
hf-map ta (num_proc=8):  19%|#9        | 130048/667659 [01:07<02:57, 3026.56 examples/s]
hf-map ta (num_proc=8):  20%|#9        | 131072/667659 [01:09<05:37, 1590.13 examples/s]
hf-map ta (num_proc=8):  20%|#9        | 132096/667659 [01:09<04:21, 2044.34 examples/s]
hf-map ta (num_proc=8):  20%|#9        | 133120/667659 [01:09<03:43, 2392.14 examples/s]
hf-map ta (num_proc=8):  20%|##        | 135168/667659 [01:09<02:19, 3817.89 examples/s]
hf-map ta (num_proc=8):  20%|##        | 136192/667659 [01:10<02:40, 3314.63 examples/s]
hf-map ta (num_proc=8):  21%|##        | 137216/667659 [01:10<03:19, 2652.62 examples/s]
hf-map ta (num_proc=8):  21%|##        | 138240/667659 [01:11<02:52, 3071.68 examples/s]
hf-map ta (num_proc=8):  21%|##        | 139264/667659 [01:12<05:18, 1659.16 examples/s]
hf-map ta (num_proc=8):  21%|##1       | 141312/667659 [01:12<03:11, 2745.73 examples/s]
hf-map ta (num_proc=8):  21%|##1       | 142336/667659 [01:12<02:45, 3172.99 examples/s]
hf-map ta (num_proc=8):  21%|##1       | 143360/667659 [01:13<02:39, 3278.73 examples/s]
hf-map ta (num_proc=8):  22%|##1       | 144384/667659 [01:13<03:34, 2442.12 examples/s]
hf-map ta (num_proc=8):  22%|##1       | 145408/667659 [01:14<03:32, 2453.26 examples/s]
hf-map ta (num_proc=8):  22%|##2       | 147456/667659 [01:15<04:33, 1903.32 examples/s]
hf-map ta (num_proc=8):  22%|##2       | 149504/667659 [01:15<03:01, 2849.42 examples/s]
hf-map ta (num_proc=8):  23%|##2       | 150528/667659 [01:15<02:40, 3225.96 examples/s]
hf-map ta (num_proc=8):  23%|##2       | 151552/667659 [01:16<02:45, 3124.81 examples/s]
hf-map ta (num_proc=8):  23%|##2       | 152576/667659 [01:16<03:24, 2514.61 examples/s]
hf-map ta (num_proc=8):  23%|##3       | 153600/667659 [01:17<03:17, 2602.29 examples/s]
hf-map ta (num_proc=8):  23%|##3       | 154624/667659 [01:17<03:28, 2457.21 examples/s]
hf-map ta (num_proc=8):  23%|##3       | 155648/667659 [01:18<04:11, 2032.03 examples/s]
hf-map ta (num_proc=8):  23%|##3       | 156672/667659 [01:18<03:18, 2568.75 examples/s]
hf-map ta (num_proc=8):  24%|##3       | 157696/667659 [01:18<02:51, 2978.58 examples/s]
hf-map ta (num_proc=8):  24%|##3       | 159744/667659 [01:19<02:54, 2912.51 examples/s]
hf-map ta (num_proc=8):  24%|##4       | 160768/667659 [01:19<02:47, 3018.60 examples/s]
hf-map ta (num_proc=8):  24%|##4       | 161792/667659 [01:19<02:34, 3272.88 examples/s]
hf-map ta (num_proc=8):  24%|##4       | 162816/667659 [01:20<03:12, 2625.37 examples/s]
hf-map ta (num_proc=8):  25%|##4       | 163840/667659 [01:21<03:38, 2303.23 examples/s]
hf-map ta (num_proc=8):  25%|##4       | 164864/667659 [01:21<03:02, 2748.32 examples/s]
hf-map ta (num_proc=8):  25%|##4       | 165888/667659 [01:21<03:24, 2448.54 examples/s]
hf-map ta (num_proc=8):  25%|##5       | 167936/667659 [01:22<02:44, 3029.95 examples/s]
hf-map ta (num_proc=8):  25%|##5       | 168960/667659 [01:22<02:45, 3012.84 examples/s]
hf-map ta (num_proc=8):  26%|##5       | 171008/667659 [01:23<02:50, 2920.26 examples/s]
hf-map ta (num_proc=8):  26%|##5       | 172032/667659 [01:24<03:19, 2485.02 examples/s]
hf-map ta (num_proc=8):  26%|##5       | 173056/667659 [01:24<02:54, 2839.62 examples/s]
hf-map ta (num_proc=8):  26%|##6       | 174080/667659 [01:24<02:51, 2870.06 examples/s]
hf-map ta (num_proc=8):  26%|##6       | 176128/667659 [01:25<02:41, 3051.75 examples/s]
hf-map ta (num_proc=8):  27%|##6       | 177152/667659 [01:25<02:56, 2776.24 examples/s]
hf-map ta (num_proc=8):  27%|##6       | 179200/667659 [01:26<02:37, 3099.76 examples/s]
hf-map ta (num_proc=8):  27%|##6       | 180224/667659 [01:26<03:11, 2551.51 examples/s]
hf-map ta (num_proc=8):  27%|##7       | 181248/667659 [01:27<02:45, 2937.18 examples/s]
hf-map ta (num_proc=8):  27%|##7       | 182272/667659 [01:27<02:57, 2740.51 examples/s]
hf-map ta (num_proc=8):  28%|##7       | 184320/667659 [01:28<03:02, 2654.55 examples/s]
hf-map ta (num_proc=8):  28%|##7       | 186368/667659 [01:28<02:19, 3461.52 examples/s]
hf-map ta (num_proc=8):  28%|##8       | 187392/667659 [01:29<02:45, 2899.98 examples/s]
hf-map ta (num_proc=8):  28%|##8       | 188416/667659 [01:29<03:00, 2654.55 examples/s]
hf-map ta (num_proc=8):  29%|##8       | 190464/667659 [01:30<02:50, 2797.61 examples/s]
hf-map ta (num_proc=8):  29%|##8       | 191488/667659 [01:30<02:35, 3068.71 examples/s]
hf-map ta (num_proc=8):  29%|##8       | 192512/667659 [01:31<03:06, 2544.37 examples/s]
hf-map ta (num_proc=8):  29%|##8       | 193536/667659 [01:31<02:34, 3072.09 examples/s]
hf-map ta (num_proc=8):  29%|##9       | 194560/667659 [01:31<02:14, 3510.55 examples/s]
hf-map ta (num_proc=8):  29%|##9       | 195584/667659 [01:32<03:06, 2536.62 examples/s]
hf-map ta (num_proc=8):  29%|##9       | 196608/667659 [01:32<02:56, 2672.47 examples/s]
hf-map ta (num_proc=8):  30%|##9       | 197632/667659 [01:32<02:23, 3264.73 examples/s]
hf-map ta (num_proc=8):  30%|##9       | 198656/667659 [01:33<02:43, 2864.29 examples/s]
hf-map ta (num_proc=8):  30%|##9       | 199680/667659 [01:33<02:34, 3024.48 examples/s]
hf-map ta (num_proc=8):  30%|###       | 200704/667659 [01:34<03:49, 2038.27 examples/s]
hf-map ta (num_proc=8):  30%|###       | 202752/667659 [01:34<02:25, 3187.23 examples/s]
hf-map ta (num_proc=8):  31%|###       | 203776/667659 [01:34<02:33, 3017.34 examples/s]
hf-map ta (num_proc=8):  31%|###       | 204800/667659 [01:35<03:07, 2467.61 examples/s]
hf-map ta (num_proc=8):  31%|###       | 206848/667659 [01:36<02:39, 2884.28 examples/s]
hf-map ta (num_proc=8):  31%|###1      | 207872/667659 [01:36<02:41, 2841.52 examples/s]
hf-map ta (num_proc=8):  31%|###1      | 208896/667659 [01:37<03:26, 2223.05 examples/s]
hf-map ta (num_proc=8):  32%|###1      | 210944/667659 [01:37<02:12, 3454.86 examples/s]
hf-map ta (num_proc=8):  32%|###1      | 211968/667659 [01:38<03:13, 2350.62 examples/s]
hf-map ta (num_proc=8):  32%|###2      | 214016/667659 [01:38<02:05, 3611.45 examples/s]
hf-map ta (num_proc=8):  32%|###2      | 215040/667659 [01:39<02:41, 2803.54 examples/s]
hf-map ta (num_proc=8):  32%|###2      | 216064/667659 [01:39<02:46, 2706.24 examples/s]
hf-map ta (num_proc=8):  33%|###2      | 217088/667659 [01:40<03:47, 1983.85 examples/s]
hf-map ta (num_proc=8):  33%|###2      | 218112/667659 [01:40<03:02, 2467.88 examples/s]
hf-map ta (num_proc=8):  33%|###2      | 220160/667659 [01:41<02:57, 2520.04 examples/s]
hf-map ta (num_proc=8):  33%|###3      | 221184/667659 [01:41<02:29, 2986.89 examples/s]
hf-map ta (num_proc=8):  33%|###3      | 222208/667659 [01:41<02:33, 2897.21 examples/s]
hf-map ta (num_proc=8):  33%|###3      | 223232/667659 [01:41<02:07, 3495.86 examples/s]
hf-map ta (num_proc=8):  34%|###3      | 224256/667659 [01:42<02:18, 3201.34 examples/s]
hf-map ta (num_proc=8):  34%|###3      | 225280/667659 [01:43<03:18, 2223.94 examples/s]
hf-map ta (num_proc=8):  34%|###3      | 226304/667659 [01:43<02:44, 2687.58 examples/s]
hf-map ta (num_proc=8):  34%|###4      | 228352/667659 [01:44<02:50, 2579.39 examples/s]
hf-map ta (num_proc=8):  34%|###4      | 229376/667659 [01:44<02:43, 2674.62 examples/s]
hf-map ta (num_proc=8):  35%|###4      | 230400/667659 [01:44<02:23, 3057.35 examples/s]
hf-map ta (num_proc=8):  35%|###4      | 231424/667659 [01:44<02:02, 3565.12 examples/s]
hf-map ta (num_proc=8):  35%|###4      | 232448/667659 [01:45<02:04, 3490.43 examples/s]
hf-map ta (num_proc=8):  35%|###4      | 233472/667659 [01:46<03:18, 2183.02 examples/s]
hf-map ta (num_proc=8):  35%|###5      | 234496/667659 [01:46<02:57, 2443.25 examples/s]
hf-map ta (num_proc=8):  35%|###5      | 235520/667659 [01:46<02:18, 3126.50 examples/s]
hf-map ta (num_proc=8):  35%|###5      | 236544/667659 [01:47<02:50, 2533.59 examples/s]
hf-map ta (num_proc=8):  36%|###5      | 237568/667659 [01:47<02:29, 2869.09 examples/s]
hf-map ta (num_proc=8):  36%|###5      | 238592/667659 [01:47<02:30, 2848.97 examples/s]
hf-map ta (num_proc=8):  36%|###5      | 239616/667659 [01:47<02:10, 3291.27 examples/s]
hf-map ta (num_proc=8):  36%|###6      | 240640/667659 [01:48<01:51, 3825.29 examples/s]
hf-map ta (num_proc=8):  36%|###6      | 241664/667659 [01:48<02:52, 2475.74 examples/s]
hf-map ta (num_proc=8):  36%|###6      | 242688/667659 [01:49<02:57, 2389.84 examples/s]
hf-map ta (num_proc=8):  37%|###6      | 243712/667659 [01:49<02:26, 2892.72 examples/s]
hf-map ta (num_proc=8):  37%|###6      | 244736/667659 [01:50<03:25, 2053.73 examples/s]
hf-map ta (num_proc=8):  37%|###6      | 245760/667659 [01:50<02:47, 2514.64 examples/s]
hf-map ta (num_proc=8):  37%|###6      | 246784/667659 [01:50<02:44, 2562.32 examples/s]
hf-map ta (num_proc=8):  37%|###7      | 247808/667659 [01:51<02:23, 2935.70 examples/s]
hf-map ta (num_proc=8):  37%|###7      | 249856/667659 [01:52<03:08, 2211.06 examples/s]
hf-map ta (num_proc=8):  38%|###7      | 250880/667659 [01:52<02:45, 2521.21 examples/s]
hf-map ta (num_proc=8):  38%|###7      | 251904/667659 [01:53<03:13, 2144.89 examples/s]
hf-map ta (num_proc=8):  38%|###7      | 252928/667659 [01:53<03:28, 1992.68 examples/s]
hf-map ta (num_proc=8):  38%|###8      | 253952/667659 [01:54<02:55, 2352.87 examples/s]
hf-map ta (num_proc=8):  38%|###8      | 254976/667659 [01:54<02:43, 2518.02 examples/s]
hf-map ta (num_proc=8):  38%|###8      | 256000/667659 [01:54<02:24, 2848.09 examples/s]
hf-map ta (num_proc=8):  38%|###8      | 257024/667659 [01:54<01:55, 3557.97 examples/s]
hf-map ta (num_proc=8):  39%|###8      | 258048/667659 [01:56<03:47, 1796.82 examples/s]
hf-map ta (num_proc=8):  39%|###8      | 260096/667659 [01:56<02:34, 2636.21 examples/s]
hf-map ta (num_proc=8):  39%|###9      | 261120/667659 [01:57<03:21, 2019.77 examples/s]
hf-map ta (num_proc=8):  39%|###9      | 262144/667659 [01:57<02:48, 2410.08 examples/s]
hf-map ta (num_proc=8):  39%|###9      | 263168/667659 [01:57<03:02, 2221.52 examples/s]
hf-map ta (num_proc=8):  40%|###9      | 264192/667659 [01:58<02:43, 2467.93 examples/s]
hf-map ta (num_proc=8):  40%|###9      | 266240/667659 [01:58<02:33, 2619.03 examples/s]
hf-map ta (num_proc=8):  40%|####      | 267264/667659 [01:59<02:27, 2722.64 examples/s]
hf-map ta (num_proc=8):  40%|####      | 268288/667659 [01:59<01:59, 3341.03 examples/s]
hf-map ta (num_proc=8):  40%|####      | 269312/667659 [01:59<02:17, 2892.53 examples/s]
hf-map ta (num_proc=8):  40%|####      | 270336/667659 [02:00<02:33, 2592.23 examples/s]
hf-map ta (num_proc=8):  41%|####      | 271360/667659 [02:00<02:06, 3130.78 examples/s]
hf-map ta (num_proc=8):  41%|####      | 272384/667659 [02:00<02:07, 3103.42 examples/s]
hf-map ta (num_proc=8):  41%|####      | 273408/667659 [02:01<01:49, 3606.25 examples/s]
hf-map ta (num_proc=8):  41%|####1     | 274432/667659 [02:01<02:41, 2435.45 examples/s]
hf-map ta (num_proc=8):  41%|####1     | 275456/667659 [02:02<02:22, 2761.65 examples/s]
hf-map ta (num_proc=8):  42%|####1     | 277504/667659 [02:02<01:58, 3301.88 examples/s]
hf-map ta (num_proc=8):  42%|####1     | 278528/667659 [02:03<02:34, 2522.11 examples/s]
hf-map ta (num_proc=8):  42%|####2     | 280576/667659 [02:03<01:46, 3641.13 examples/s]
hf-map ta (num_proc=8):  42%|####2     | 281600/667659 [02:03<02:01, 3174.59 examples/s]
hf-map ta (num_proc=8):  42%|####2     | 282624/667659 [02:04<02:40, 2404.17 examples/s]
hf-map ta (num_proc=8):  42%|####2     | 283648/667659 [02:04<02:13, 2884.67 examples/s]
hf-map ta (num_proc=8):  43%|####2     | 285696/667659 [02:05<01:55, 3300.61 examples/s]
hf-map ta (num_proc=8):  43%|####2     | 286720/667659 [02:05<02:10, 2929.36 examples/s]
hf-map ta (num_proc=8):  43%|####3     | 287744/667659 [02:05<01:50, 3424.00 examples/s]
hf-map ta (num_proc=8):  43%|####3     | 288768/667659 [02:06<01:39, 3798.99 examples/s]
hf-map ta (num_proc=8):  43%|####3     | 289792/667659 [02:06<01:56, 3233.83 examples/s]
hf-map ta (num_proc=8):  44%|####3     | 290816/667659 [02:07<02:43, 2308.21 examples/s]
hf-map ta (num_proc=8):  44%|####3     | 291840/667659 [02:07<02:32, 2469.27 examples/s]
hf-map ta (num_proc=8):  44%|####4     | 293888/667659 [02:08<01:55, 3237.94 examples/s]
hf-map ta (num_proc=8):  44%|####4     | 294912/667659 [02:08<01:51, 3353.35 examples/s]
hf-map ta (num_proc=8):  44%|####4     | 295936/667659 [02:08<01:37, 3815.30 examples/s]
hf-map ta (num_proc=8):  44%|####4     | 296960/667659 [02:08<01:40, 3682.50 examples/s]
hf-map ta (num_proc=8):  45%|####4     | 297984/667659 [02:09<02:14, 2751.66 examples/s]
hf-map ta (num_proc=8):  45%|####4     | 299008/667659 [02:10<02:50, 2156.59 examples/s]
hf-map ta (num_proc=8):  45%|####4     | 300032/667659 [02:10<02:24, 2543.89 examples/s]
hf-map ta (num_proc=8):  45%|####5     | 302080/667659 [02:10<01:33, 3895.49 examples/s]
hf-map ta (num_proc=8):  45%|####5     | 303104/667659 [02:11<01:53, 3214.20 examples/s]
hf-map ta (num_proc=8):  46%|####5     | 305152/667659 [02:11<01:33, 3865.81 examples/s]
hf-map ta (num_proc=8):  46%|####5     | 306176/667659 [02:12<02:20, 2565.20 examples/s]
hf-map ta (num_proc=8):  46%|####6     | 307200/667659 [02:12<02:37, 2290.35 examples/s]
hf-map ta (num_proc=8):  46%|####6     | 308224/667659 [02:13<02:16, 2627.93 examples/s]
hf-map ta (num_proc=8):  46%|####6     | 310272/667659 [02:13<01:39, 3587.52 examples/s]
hf-map ta (num_proc=8):  47%|####6     | 312320/667659 [02:14<01:55, 3077.99 examples/s]
hf-map ta (num_proc=8):  47%|####7     | 314368/667659 [02:14<02:00, 2938.23 examples/s]
hf-map ta (num_proc=8):  47%|####7     | 315392/667659 [02:15<02:13, 2638.47 examples/s]
hf-map ta (num_proc=8):  47%|####7     | 316416/667659 [02:15<01:57, 2980.17 examples/s]
hf-map ta (num_proc=8):  48%|####7     | 317440/667659 [02:16<01:54, 3066.55 examples/s]
hf-map ta (num_proc=8):  48%|####7     | 319488/667659 [02:16<01:18, 4445.36 examples/s]
hf-map ta (num_proc=8):  48%|####8     | 320512/667659 [02:17<02:03, 2800.26 examples/s]
hf-map ta (num_proc=8):  48%|####8     | 322560/667659 [02:17<01:53, 3040.48 examples/s]
hf-map ta (num_proc=8):  48%|####8     | 323584/667659 [02:17<01:51, 3089.04 examples/s]
hf-map ta (num_proc=8):  49%|####8     | 324608/667659 [02:18<01:35, 3584.75 examples/s]
hf-map ta (num_proc=8):  49%|####8     | 325632/667659 [02:18<02:01, 2812.46 examples/s]
hf-map ta (num_proc=8):  49%|####8     | 326656/667659 [02:18<01:41, 3370.29 examples/s]
hf-map ta (num_proc=8):  49%|####9     | 327680/667659 [02:19<01:44, 3252.97 examples/s]
hf-map ta (num_proc=8):  49%|####9     | 328704/667659 [02:19<01:52, 3009.43 examples/s]
hf-map ta (num_proc=8):  49%|####9     | 329728/667659 [02:19<01:33, 3602.36 examples/s]
hf-map ta (num_proc=8):  50%|####9     | 330752/667659 [02:19<01:30, 3718.42 examples/s]
hf-map ta (num_proc=8):  50%|####9     | 331776/667659 [02:20<01:53, 2963.09 examples/s]
hf-map ta (num_proc=8):  50%|####9     | 333824/667659 [02:21<01:49, 3039.81 examples/s]
hf-map ta (num_proc=8):  50%|#####     | 334848/667659 [02:21<01:42, 3243.22 examples/s]
hf-map ta (num_proc=8):  50%|#####     | 335872/667659 [02:21<01:36, 3443.45 examples/s]
hf-map ta (num_proc=8):  50%|#####     | 336896/667659 [02:22<01:47, 3068.21 examples/s]
hf-map ta (num_proc=8):  51%|#####     | 337920/667659 [02:22<01:38, 3356.08 examples/s]
hf-map ta (num_proc=8):  51%|#####     | 339968/667659 [02:23<01:51, 2929.13 examples/s]
hf-map ta (num_proc=8):  51%|#####1    | 340992/667659 [02:23<01:43, 3167.35 examples/s]
hf-map ta (num_proc=8):  51%|#####1    | 342016/667659 [02:23<01:31, 3558.10 examples/s]
hf-map ta (num_proc=8):  51%|#####1    | 343040/667659 [02:24<01:53, 2870.45 examples/s]
hf-map ta (num_proc=8):  52%|#####1    | 345088/667659 [02:24<01:44, 3077.71 examples/s]
hf-map ta (num_proc=8):  52%|#####1    | 346112/667659 [02:24<01:33, 3441.08 examples/s]
hf-map ta (num_proc=8):  52%|#####1    | 347136/667659 [02:24<01:18, 4106.76 examples/s]
hf-map ta (num_proc=8):  52%|#####2    | 348160/667659 [02:25<02:07, 2514.77 examples/s]
hf-map ta (num_proc=8):  52%|#####2    | 349184/667659 [02:26<02:00, 2646.25 examples/s]
hf-map ta (num_proc=8):  52%|#####2    | 350208/667659 [02:26<01:41, 3142.73 examples/s]
hf-map ta (num_proc=8):  53%|#####2    | 351232/667659 [02:26<01:22, 3826.59 examples/s]
hf-map ta (num_proc=8):  53%|#####2    | 352256/667659 [02:26<01:29, 3508.10 examples/s]
hf-map ta (num_proc=8):  53%|#####2    | 353280/667659 [02:27<01:54, 2751.41 examples/s]
hf-map ta (num_proc=8):  53%|#####3    | 354304/667659 [02:27<01:53, 2750.51 examples/s]
hf-map ta (num_proc=8):  53%|#####3    | 356352/667659 [02:28<02:06, 2458.58 examples/s]
hf-map ta (num_proc=8):  54%|#####3    | 357376/667659 [02:28<01:52, 2752.00 examples/s]
hf-map ta (num_proc=8):  54%|#####3    | 359424/667659 [02:29<01:34, 3255.16 examples/s]
hf-map ta (num_proc=8):  54%|#####3    | 360448/667659 [02:29<01:42, 3001.49 examples/s]
hf-map ta (num_proc=8):  54%|#####4    | 361472/667659 [02:29<01:28, 3445.48 examples/s]
hf-map ta (num_proc=8):  54%|#####4    | 362496/667659 [02:30<01:21, 3733.67 examples/s]
hf-map ta (num_proc=8):  54%|#####4    | 363520/667659 [02:30<01:15, 4033.06 examples/s]
hf-map ta (num_proc=8):  55%|#####4    | 364544/667659 [02:31<02:05, 2405.89 examples/s]
hf-map ta (num_proc=8):  55%|#####4    | 365568/667659 [02:31<01:55, 2614.95 examples/s]
hf-map ta (num_proc=8):  55%|#####4    | 366592/667659 [02:31<01:46, 2828.23 examples/s]
hf-map ta (num_proc=8):  55%|#####5    | 367616/667659 [02:31<01:31, 3295.96 examples/s]
hf-map ta (num_proc=8):  55%|#####5    | 368640/667659 [02:32<01:33, 3188.39 examples/s]
hf-map ta (num_proc=8):  55%|#####5    | 369664/667659 [02:32<01:26, 3453.42 examples/s]
hf-map ta (num_proc=8):  56%|#####5    | 370688/667659 [02:32<01:20, 3678.92 examples/s]
hf-map ta (num_proc=8):  56%|#####5    | 371712/667659 [02:32<01:12, 4080.74 examples/s]
hf-map ta (num_proc=8):  56%|#####5    | 372736/667659 [02:34<02:31, 1940.72 examples/s]
hf-map ta (num_proc=8):  56%|#####5    | 373760/667659 [02:34<02:00, 2443.30 examples/s]
hf-map ta (num_proc=8):  56%|#####6    | 374784/667659 [02:34<01:44, 2815.34 examples/s]
hf-map ta (num_proc=8):  56%|#####6    | 375808/667659 [02:35<02:10, 2237.90 examples/s]
hf-map ta (num_proc=8):  56%|#####6    | 376832/667659 [02:35<02:05, 2314.18 examples/s]
hf-map ta (num_proc=8):  57%|#####6    | 377856/667659 [02:36<02:05, 2308.55 examples/s]
hf-map ta (num_proc=8):  57%|#####6    | 378880/667659 [02:36<02:03, 2331.87 examples/s]
hf-map ta (num_proc=8):  57%|#####6    | 379904/667659 [02:36<01:38, 2921.07 examples/s]
hf-map ta (num_proc=8):  57%|#####7    | 380928/667659 [02:38<03:02, 1571.73 examples/s]
hf-map ta (num_proc=8):  57%|#####7    | 382976/667659 [02:38<01:51, 2558.76 examples/s]
hf-map ta (num_proc=8):  58%|#####7    | 384000/667659 [02:38<01:31, 3090.11 examples/s]
hf-map ta (num_proc=8):  58%|#####7    | 385024/667659 [02:38<01:44, 2707.57 examples/s]
hf-map ta (num_proc=8):  58%|#####7    | 386048/667659 [02:39<01:31, 3083.59 examples/s]
hf-map ta (num_proc=8):  58%|#####7    | 387072/667659 [02:39<01:46, 2639.45 examples/s]
hf-map ta (num_proc=8):  58%|#####8    | 389120/667659 [02:41<02:37, 1772.81 examples/s]
hf-map ta (num_proc=8):  59%|#####8    | 391168/667659 [02:41<01:49, 2534.32 examples/s]
hf-map ta (num_proc=8):  59%|#####8    | 392192/667659 [02:41<01:42, 2692.04 examples/s]
hf-map ta (num_proc=8):  59%|#####8    | 393216/667659 [02:42<01:39, 2753.57 examples/s]
hf-map ta (num_proc=8):  59%|#####9    | 394240/667659 [02:42<01:50, 2471.69 examples/s]
hf-map ta (num_proc=8):  59%|#####9    | 395264/667659 [02:42<01:32, 2938.62 examples/s]
hf-map ta (num_proc=8):  59%|#####9    | 396288/667659 [02:42<01:17, 3515.48 examples/s]
hf-map ta (num_proc=8):  60%|#####9    | 397312/667659 [02:44<02:19, 1941.91 examples/s]
hf-map ta (num_proc=8):  60%|#####9    | 399360/667659 [02:44<01:22, 3234.07 examples/s]
hf-map ta (num_proc=8):  60%|#####9    | 400384/667659 [02:44<01:32, 2885.55 examples/s]
hf-map ta (num_proc=8):  60%|######    | 401408/667659 [02:44<01:19, 3336.73 examples/s]
hf-map ta (num_proc=8):  60%|######    | 402432/667659 [02:45<01:26, 3063.39 examples/s]
hf-map ta (num_proc=8):  60%|######    | 403456/667659 [02:45<01:27, 3020.24 examples/s]
hf-map ta (num_proc=8):  61%|######    | 405504/667659 [02:46<01:53, 2301.76 examples/s]
hf-map ta (num_proc=8):  61%|######1   | 407552/667659 [02:46<01:18, 3325.50 examples/s]
hf-map ta (num_proc=8):  61%|######1   | 408576/667659 [02:47<01:32, 2811.59 examples/s]
hf-map ta (num_proc=8):  62%|######1   | 410624/667659 [02:48<01:33, 2756.90 examples/s]
hf-map ta (num_proc=8):  62%|######1   | 412672/667659 [02:48<01:17, 3279.48 examples/s]
hf-map ta (num_proc=8):  62%|######1   | 413696/667659 [02:49<01:18, 3242.54 examples/s]
hf-map ta (num_proc=8):  62%|######2   | 414720/667659 [02:49<01:31, 2756.02 examples/s]
hf-map ta (num_proc=8):  62%|######2   | 416768/667659 [02:49<01:13, 3397.01 examples/s]
hf-map ta (num_proc=8):  63%|######2   | 417792/667659 [02:50<01:07, 3677.07 examples/s]
hf-map ta (num_proc=8):  63%|######2   | 418816/667659 [02:51<02:18, 1793.25 examples/s]
hf-map ta (num_proc=8):  63%|######3   | 420864/667659 [02:52<01:38, 2517.09 examples/s]
hf-map ta (num_proc=8):  63%|######3   | 422912/667659 [02:52<01:11, 3404.94 examples/s]
hf-map ta (num_proc=8):  63%|######3   | 423936/667659 [02:52<01:10, 3459.17 examples/s]
hf-map ta (num_proc=8):  64%|######3   | 424960/667659 [02:52<01:02, 3853.99 examples/s]
hf-map ta (num_proc=8):  64%|######3   | 425984/667659 [02:52<01:01, 3926.44 examples/s]
hf-map ta (num_proc=8):  64%|######3   | 427008/667659 [02:54<02:12, 1809.50 examples/s]
hf-map ta (num_proc=8):  64%|######4   | 429056/667659 [02:54<01:36, 2468.66 examples/s]
hf-map ta (num_proc=8):  65%|######4   | 431104/667659 [02:55<01:11, 3300.71 examples/s]
hf-map ta (num_proc=8):  65%|######4   | 432128/667659 [02:55<01:17, 3025.01 examples/s]
hf-map ta (num_proc=8):  65%|######5   | 434176/667659 [02:55<00:56, 4111.60 examples/s]
hf-map ta (num_proc=8):  65%|######5   | 435200/667659 [02:56<01:36, 2397.87 examples/s]
hf-map ta (num_proc=8):  65%|######5   | 437248/667659 [02:57<01:28, 2599.41 examples/s]
hf-map ta (num_proc=8):  66%|######5   | 438272/667659 [02:57<01:21, 2815.28 examples/s]
hf-map ta (num_proc=8):  66%|######5   | 439296/667659 [02:58<01:17, 2945.32 examples/s]
hf-map ta (num_proc=8):  66%|######5   | 440320/667659 [02:58<01:10, 3227.15 examples/s]
hf-map ta (num_proc=8):  66%|######6   | 441344/667659 [02:58<00:58, 3859.12 examples/s]
hf-map ta (num_proc=8):  66%|######6   | 442368/667659 [02:58<01:12, 3098.73 examples/s]
hf-map ta (num_proc=8):  66%|######6   | 443392/667659 [02:59<01:31, 2443.38 examples/s]
hf-map ta (num_proc=8):  67%|######6   | 444416/667659 [02:59<01:21, 2727.26 examples/s]
hf-map ta (num_proc=8):  67%|######6   | 445440/667659 [03:00<01:53, 1955.07 examples/s]
hf-map ta (num_proc=8):  67%|######6   | 446464/667659 [03:01<01:40, 2211.71 examples/s]
hf-map ta (num_proc=8):  67%|######7   | 449536/667659 [03:01<00:50, 4318.61 examples/s]
hf-map ta (num_proc=8):  67%|######7   | 450560/667659 [03:02<01:31, 2383.38 examples/s]
hf-map ta (num_proc=8):  68%|######7   | 451584/667659 [03:02<01:33, 2315.15 examples/s]
hf-map ta (num_proc=8):  68%|######7   | 453632/667659 [03:03<01:37, 2186.54 examples/s]
hf-map ta (num_proc=8):  68%|######8   | 454656/667659 [03:04<01:31, 2323.73 examples/s]
hf-map ta (num_proc=8):  68%|######8   | 455680/667659 [03:04<01:44, 2021.51 examples/s]
hf-map ta (num_proc=8):  68%|######8   | 456704/667659 [03:04<01:23, 2540.07 examples/s]
hf-map ta (num_proc=8):  69%|######8   | 457728/667659 [03:05<01:13, 2860.37 examples/s]
hf-map ta (num_proc=8):  69%|######8   | 458752/667659 [03:06<01:44, 2004.68 examples/s]
hf-map ta (num_proc=8):  69%|######9   | 460800/667659 [03:06<01:15, 2724.38 examples/s]
hf-map ta (num_proc=8):  69%|######9   | 461824/667659 [03:06<01:15, 2723.90 examples/s]
hf-map ta (num_proc=8):  69%|######9   | 463872/667659 [03:07<01:16, 2662.07 examples/s]
hf-map ta (num_proc=8):  70%|######9   | 464896/667659 [03:08<01:13, 2767.31 examples/s]
hf-map ta (num_proc=8):  70%|######9   | 466944/667659 [03:08<01:05, 3064.61 examples/s]
hf-map ta (num_proc=8):  70%|#######   | 467968/667659 [03:08<00:55, 3596.19 examples/s]
hf-map ta (num_proc=8):  70%|#######   | 468992/667659 [03:09<01:06, 2977.10 examples/s]
hf-map ta (num_proc=8):  71%|#######   | 471040/667659 [03:09<00:55, 3572.40 examples/s]
hf-map ta (num_proc=8):  71%|#######   | 472064/667659 [03:10<01:04, 3040.51 examples/s]
hf-map ta (num_proc=8):  71%|#######   | 473088/667659 [03:10<01:08, 2820.15 examples/s]
hf-map ta (num_proc=8):  71%|#######1  | 475136/667659 [03:11<00:59, 3261.33 examples/s]
hf-map ta (num_proc=8):  71%|#######1  | 476160/667659 [03:11<00:59, 3225.99 examples/s]
hf-map ta (num_proc=8):  71%|#######1  | 477184/667659 [03:11<00:55, 3423.08 examples/s]
hf-map ta (num_proc=8):  72%|#######1  | 478208/667659 [03:11<00:55, 3406.23 examples/s]
hf-map ta (num_proc=8):  72%|#######1  | 479232/667659 [03:12<00:48, 3914.25 examples/s]
hf-map ta (num_proc=8):  72%|#######1  | 480256/667659 [03:12<00:44, 4191.74 examples/s]
hf-map ta (num_proc=8):  72%|#######2  | 481280/667659 [03:12<01:03, 2938.00 examples/s]
hf-map ta (num_proc=8):  72%|#######2  | 482304/667659 [03:13<01:00, 3082.47 examples/s]
hf-map ta (num_proc=8):  72%|#######2  | 483328/667659 [03:13<00:56, 3285.05 examples/s]
hf-map ta (num_proc=8):  73%|#######2  | 484352/667659 [03:13<01:01, 2996.36 examples/s]
hf-map ta (num_proc=8):  73%|#######2  | 485376/667659 [03:14<00:54, 3317.10 examples/s]
hf-map ta (num_proc=8):  73%|#######2  | 486400/667659 [03:14<00:47, 3825.03 examples/s]
hf-map ta (num_proc=8):  73%|#######3  | 487424/667659 [03:14<00:41, 4345.52 examples/s]
hf-map ta (num_proc=8):  73%|#######3  | 488448/667659 [03:14<00:43, 4102.37 examples/s]
hf-map ta (num_proc=8):  73%|#######3  | 489472/667659 [03:15<01:06, 2699.65 examples/s]
hf-map ta (num_proc=8):  73%|#######3  | 490496/667659 [03:15<01:07, 2619.60 examples/s]
hf-map ta (num_proc=8):  74%|#######3  | 492544/667659 [03:16<01:03, 2762.26 examples/s]
hf-map ta (num_proc=8):  74%|#######4  | 494592/667659 [03:16<00:47, 3606.26 examples/s]
hf-map ta (num_proc=8):  74%|#######4  | 496640/667659 [03:17<00:44, 3808.42 examples/s]
hf-map ta (num_proc=8):  75%|#######4  | 497664/667659 [03:18<01:03, 2689.94 examples/s]
hf-map ta (num_proc=8):  75%|#######4  | 498688/667659 [03:18<01:08, 2468.44 examples/s]
hf-map ta (num_proc=8):  75%|#######4  | 500736/667659 [03:19<01:00, 2769.16 examples/s]
hf-map ta (num_proc=8):  75%|#######5  | 502784/667659 [03:19<00:45, 3645.51 examples/s]
hf-map ta (num_proc=8):  76%|#######5  | 504832/667659 [03:19<00:40, 4011.51 examples/s]
hf-map ta (num_proc=8):  76%|#######5  | 505856/667659 [03:21<01:09, 2320.56 examples/s]
hf-map ta (num_proc=8):  76%|#######5  | 506880/667659 [03:21<01:02, 2554.57 examples/s]
hf-map ta (num_proc=8):  76%|#######6  | 507904/667659 [03:21<00:54, 2940.82 examples/s]
hf-map ta (num_proc=8):  76%|#######6  | 508928/667659 [03:21<00:54, 2914.01 examples/s]
hf-map ta (num_proc=8):  76%|#######6  | 509952/667659 [03:22<00:47, 3316.66 examples/s]
hf-map ta (num_proc=8):  77%|#######6  | 510976/667659 [03:22<00:40, 3855.70 examples/s]
hf-map ta (num_proc=8):  77%|#######6  | 512000/667659 [03:22<00:33, 4583.74 examples/s]
hf-map ta (num_proc=8):  77%|#######6  | 513024/667659 [03:22<00:39, 3906.29 examples/s]
hf-map ta (num_proc=8):  77%|#######6  | 514048/667659 [03:23<00:59, 2560.93 examples/s]
hf-map ta (num_proc=8):  77%|#######7  | 515072/667659 [03:23<01:03, 2389.14 examples/s]
hf-map ta (num_proc=8):  77%|#######7  | 517120/667659 [03:24<00:51, 2912.22 examples/s]
hf-map ta (num_proc=8):  78%|#######7  | 518144/667659 [03:24<00:47, 3119.45 examples/s]
hf-map ta (num_proc=8):  78%|#######7  | 519168/667659 [03:24<00:40, 3682.43 examples/s]
hf-map ta (num_proc=8):  78%|#######7  | 520192/667659 [03:25<00:37, 3954.50 examples/s]
hf-map ta (num_proc=8):  78%|#######8  | 521216/667659 [03:25<00:42, 3484.37 examples/s]
hf-map ta (num_proc=8):  78%|#######8  | 522240/667659 [03:26<00:54, 2690.67 examples/s]
hf-map ta (num_proc=8):  78%|#######8  | 523264/667659 [03:26<00:45, 3183.90 examples/s]
hf-map ta (num_proc=8):  79%|#######8  | 524288/667659 [03:26<00:43, 3310.64 examples/s]
hf-map ta (num_proc=8):  79%|#######8  | 525312/667659 [03:26<00:35, 4053.04 examples/s]
hf-map ta (num_proc=8):  79%|#######8  | 526336/667659 [03:26<00:34, 4148.33 examples/s]
hf-map ta (num_proc=8):  79%|#######8  | 527360/667659 [03:27<00:30, 4564.33 examples/s]
hf-map ta (num_proc=8):  79%|#######9  | 528384/667659 [03:27<00:35, 3881.13 examples/s]
hf-map ta (num_proc=8):  79%|#######9  | 529408/667659 [03:27<00:47, 2893.49 examples/s]
hf-map ta (num_proc=8):  79%|#######9  | 530432/667659 [03:28<00:52, 2620.49 examples/s]
hf-map ta (num_proc=8):  80%|#######9  | 531456/667659 [03:28<00:55, 2460.53 examples/s]
hf-map ta (num_proc=8):  80%|#######9  | 533504/667659 [03:29<00:37, 3534.41 examples/s]
hf-map ta (num_proc=8):  80%|########  | 534528/667659 [03:29<00:38, 3423.34 examples/s]
hf-map ta (num_proc=8):  80%|########  | 535552/667659 [03:29<00:34, 3780.21 examples/s]
hf-map ta (num_proc=8):  80%|########  | 536576/667659 [03:29<00:28, 4530.96 examples/s]
hf-map ta (num_proc=8):  81%|########  | 537600/667659 [03:30<00:43, 3010.18 examples/s]
hf-map ta (num_proc=8):  81%|########  | 538624/667659 [03:30<00:38, 3313.35 examples/s]
hf-map ta (num_proc=8):  81%|########  | 539648/667659 [03:31<01:00, 2112.05 examples/s]
hf-map ta (num_proc=8):  81%|########1 | 541696/667659 [03:31<00:37, 3333.44 examples/s]
hf-map ta (num_proc=8):  81%|########1 | 542720/667659 [03:32<00:39, 3195.04 examples/s]
hf-map ta (num_proc=8):  82%|########1 | 544768/667659 [03:32<00:33, 3628.17 examples/s]
hf-map ta (num_proc=8):  82%|########1 | 545792/667659 [03:33<00:41, 2921.54 examples/s]
hf-map ta (num_proc=8):  82%|########2 | 547840/667659 [03:34<00:43, 2750.30 examples/s]
hf-map ta (num_proc=8):  82%|########2 | 548864/667659 [03:34<00:41, 2862.01 examples/s]
hf-map ta (num_proc=8):  82%|########2 | 549888/667659 [03:34<00:36, 3255.23 examples/s]
hf-map ta (num_proc=8):  83%|########2 | 550912/667659 [03:34<00:29, 3907.50 examples/s]
hf-map ta (num_proc=8):  83%|########2 | 551936/667659 [03:34<00:28, 4030.93 examples/s]
hf-map ta (num_proc=8):  83%|########2 | 552960/667659 [03:34<00:25, 4528.25 examples/s]
hf-map ta (num_proc=8):  83%|########2 | 553984/667659 [03:35<00:43, 2639.05 examples/s]
hf-map ta (num_proc=8):  83%|########3 | 555008/667659 [03:35<00:36, 3062.08 examples/s]
hf-map ta (num_proc=8):  83%|########3 | 556032/667659 [03:36<00:47, 2366.14 examples/s]
hf-map ta (num_proc=8):  84%|########3 | 558080/667659 [03:36<00:30, 3559.90 examples/s]
hf-map ta (num_proc=8):  84%|########3 | 560128/667659 [03:37<00:25, 4287.39 examples/s]
hf-map ta (num_proc=8):  84%|########4 | 561152/667659 [03:37<00:30, 3513.18 examples/s]
hf-map ta (num_proc=8):  84%|########4 | 562176/667659 [03:38<00:43, 2452.30 examples/s]
hf-map ta (num_proc=8):  84%|########4 | 563200/667659 [03:38<00:35, 2983.90 examples/s]
hf-map ta (num_proc=8):  85%|########4 | 564224/667659 [03:39<00:36, 2865.09 examples/s]
hf-map ta (num_proc=8):  85%|########4 | 565248/667659 [03:39<00:28, 3533.52 examples/s]
hf-map ta (num_proc=8):  85%|########4 | 566272/667659 [03:39<00:30, 3305.16 examples/s]
hf-map ta (num_proc=8):  85%|########5 | 568320/667659 [03:39<00:24, 4051.42 examples/s]
hf-map ta (num_proc=8):  85%|########5 | 569344/667659 [03:40<00:28, 3478.01 examples/s]
hf-map ta (num_proc=8):  85%|########5 | 570368/667659 [03:40<00:29, 3275.07 examples/s]
hf-map ta (num_proc=8):  86%|########5 | 571392/667659 [03:40<00:28, 3406.81 examples/s]
hf-map ta (num_proc=8):  86%|########5 | 572416/667659 [03:41<00:27, 3443.72 examples/s]
hf-map ta (num_proc=8):  86%|########5 | 573440/667659 [03:41<00:27, 3417.38 examples/s]
hf-map ta (num_proc=8):  86%|########6 | 574464/667659 [03:41<00:30, 3007.92 examples/s]
hf-map ta (num_proc=8):  86%|########6 | 576512/667659 [03:42<00:21, 4208.12 examples/s]
hf-map ta (num_proc=8):  87%|########6 | 577536/667659 [03:42<00:30, 2975.13 examples/s]
hf-map ta (num_proc=8):  87%|########6 | 578560/667659 [03:43<00:26, 3308.75 examples/s]
hf-map ta (num_proc=8):  87%|########6 | 579584/667659 [03:43<00:30, 2842.68 examples/s]
hf-map ta (num_proc=8):  87%|########6 | 580608/667659 [03:43<00:31, 2785.84 examples/s]
hf-map ta (num_proc=8):  87%|########7 | 581632/667659 [03:44<00:24, 3489.86 examples/s]
hf-map ta (num_proc=8):  87%|########7 | 582656/667659 [03:44<00:22, 3701.29 examples/s]
hf-map ta (num_proc=8):  87%|########7 | 583680/667659 [03:44<00:18, 4464.58 examples/s]
hf-map ta (num_proc=8):  88%|########7 | 584704/667659 [03:44<00:19, 4317.72 examples/s]
hf-map ta (num_proc=8):  88%|########7 | 585728/667659 [03:45<00:26, 3104.99 examples/s]
hf-map ta (num_proc=8):  88%|########7 | 586752/667659 [03:45<00:30, 2686.12 examples/s]
hf-map ta (num_proc=8):  88%|########8 | 587776/667659 [03:46<00:27, 2865.72 examples/s]
hf-map ta (num_proc=8):  88%|########8 | 588800/667659 [03:46<00:27, 2854.92 examples/s]
hf-map ta (num_proc=8):  88%|########8 | 589824/667659 [03:46<00:26, 2964.58 examples/s]
hf-map ta (num_proc=8):  88%|########8 | 590848/667659 [03:46<00:22, 3368.81 examples/s]
hf-map ta (num_proc=8):  89%|########8 | 591872/667659 [03:47<00:22, 3338.48 examples/s]
hf-map ta (num_proc=8):  89%|########8 | 592896/667659 [03:47<00:18, 4001.26 examples/s]
hf-map ta (num_proc=8):  89%|########8 | 593920/667659 [03:47<00:19, 3706.41 examples/s]
hf-map ta (num_proc=8):  89%|########9 | 594944/667659 [03:48<00:20, 3499.21 examples/s]
hf-map ta (num_proc=8):  89%|########9 | 595968/667659 [03:48<00:22, 3175.20 examples/s]
hf-map ta (num_proc=8):  89%|########9 | 596992/667659 [03:48<00:23, 2990.83 examples/s]
hf-map ta (num_proc=8):  90%|########9 | 598016/667659 [03:49<00:23, 2943.84 examples/s]
hf-map ta (num_proc=8):  90%|########9 | 599040/667659 [03:49<00:20, 3299.88 examples/s]
hf-map ta (num_proc=8):  90%|########9 | 600064/667659 [03:49<00:19, 3501.33 examples/s]
hf-map ta (num_proc=8):  90%|######### | 601088/667659 [03:49<00:16, 4059.22 examples/s]
hf-map ta (num_proc=8):  90%|######### | 602112/667659 [03:50<00:24, 2684.58 examples/s]
hf-map ta (num_proc=8):  90%|######### | 603136/667659 [03:50<00:20, 3168.13 examples/s]
hf-map ta (num_proc=8):  90%|######### | 604160/667659 [03:51<00:20, 3137.17 examples/s]
hf-map ta (num_proc=8):  91%|######### | 605184/667659 [03:51<00:16, 3763.88 examples/s]
hf-map ta (num_proc=8):  91%|######### | 606208/667659 [03:51<00:18, 3340.32 examples/s]
hf-map ta (num_proc=8):  91%|######### | 607232/667659 [03:51<00:18, 3225.19 examples/s]
hf-map ta (num_proc=8):  91%|#########1| 608256/667659 [03:52<00:15, 3785.78 examples/s]
hf-map ta (num_proc=8):  91%|#########1| 610304/667659 [03:52<00:17, 3337.96 examples/s]
hf-map ta (num_proc=8):  92%|#########1| 611328/667659 [03:53<00:21, 2592.50 examples/s]
hf-map ta (num_proc=8):  92%|#########1| 612352/667659 [03:53<00:20, 2752.08 examples/s]
hf-map ta (num_proc=8):  92%|#########2| 614400/667659 [03:54<00:14, 3586.11 examples/s]
hf-map ta (num_proc=8):  92%|#########2| 615424/667659 [03:54<00:15, 3275.03 examples/s]
hf-map ta (num_proc=8):  92%|#########2| 616448/667659 [03:54<00:13, 3887.22 examples/s]
hf-map ta (num_proc=8):  92%|#########2| 617472/667659 [03:54<00:11, 4543.88 examples/s]
hf-map ta (num_proc=8):  93%|#########2| 618496/667659 [03:55<00:17, 2790.61 examples/s]
hf-map ta (num_proc=8):  93%|#########2| 619520/667659 [03:55<00:19, 2526.48 examples/s]
hf-map ta (num_proc=8):  93%|#########2| 620544/667659 [03:56<00:15, 2954.23 examples/s]
hf-map ta (num_proc=8):  93%|#########3| 621568/667659 [03:56<00:12, 3630.44 examples/s]
hf-map ta (num_proc=8):  93%|#########3| 622592/667659 [03:56<00:13, 3271.90 examples/s]
hf-map ta (num_proc=8):  93%|#########3| 623616/667659 [03:56<00:11, 3803.55 examples/s]
hf-map ta (num_proc=8):  94%|#########3| 624640/667659 [03:57<00:11, 3690.66 examples/s]
hf-map ta (num_proc=8):  94%|#########3| 625664/667659 [03:57<00:09, 4385.57 examples/s]
hf-map ta (num_proc=8):  94%|#########3| 626688/667659 [03:57<00:11, 3713.66 examples/s]
hf-map ta (num_proc=8):  94%|#########4| 627712/667659 [03:58<00:15, 2542.38 examples/s]
hf-map ta (num_proc=8):  94%|#########4| 628736/667659 [03:58<00:12, 3036.90 examples/s]
hf-map ta (num_proc=8):  94%|#########4| 629760/667659 [03:58<00:10, 3679.91 examples/s]
hf-map ta (num_proc=8):  94%|#########4| 630784/667659 [03:59<00:12, 2946.47 examples/s]
hf-map ta (num_proc=8):  95%|#########4| 631808/667659 [03:59<00:12, 2764.68 examples/s]
hf-map ta (num_proc=8):  95%|#########4| 633856/667659 [03:59<00:08, 3778.87 examples/s]
hf-map ta (num_proc=8):  95%|#########5| 634880/667659 [04:00<00:07, 4143.95 examples/s]
hf-map ta (num_proc=8):  95%|#########5| 635904/667659 [04:00<00:10, 2985.78 examples/s]
hf-map ta (num_proc=8):  95%|#########5| 636928/667659 [04:01<00:10, 2893.30 examples/s]
hf-map ta (num_proc=8):  96%|#########5| 637952/667659 [04:01<00:10, 2809.77 examples/s]
hf-map ta (num_proc=8):  96%|#########5| 640000/667659 [04:01<00:06, 4022.89 examples/s]
hf-map ta (num_proc=8):  96%|#########6| 641024/667659 [04:01<00:06, 4200.82 examples/s]
hf-map ta (num_proc=8):  96%|#########6| 642048/667659 [04:02<00:06, 3796.41 examples/s]
hf-map ta (num_proc=8):  96%|#########6| 643072/667659 [04:02<00:05, 4283.49 examples/s]
hf-map ta (num_proc=8):  96%|#########6| 644096/667659 [04:03<00:09, 2409.66 examples/s]
hf-map ta (num_proc=8):  97%|#########6| 645120/667659 [04:03<00:08, 2788.73 examples/s]
hf-map ta (num_proc=8):  97%|#########6| 646144/667659 [04:03<00:07, 2743.61 examples/s]
hf-map ta (num_proc=8):  97%|#########6| 647168/667659 [04:04<00:06, 3244.58 examples/s]
hf-map ta (num_proc=8):  97%|#########7| 648192/667659 [04:04<00:05, 3491.78 examples/s]
hf-map ta (num_proc=8):  97%|#########7| 649216/667659 [04:04<00:05, 3569.10 examples/s]
hf-map ta (num_proc=8):  97%|#########7| 650240/667659 [04:04<00:04, 3591.43 examples/s]
hf-map ta (num_proc=8):  98%|#########7| 651264/667659 [04:05<00:04, 3735.37 examples/s]
hf-map ta (num_proc=8):  98%|#########7| 652288/667659 [04:05<00:05, 2930.53 examples/s]
hf-map ta (num_proc=8):  98%|#########7| 653312/667659 [04:05<00:04, 3200.61 examples/s]
hf-map ta (num_proc=8):  98%|#########8| 654336/667659 [04:06<00:04, 3032.67 examples/s]
hf-map ta (num_proc=8):  98%|#########8| 655873/667659 [04:06<00:03, 3908.12 examples/s]
hf-map ta (num_proc=8):  98%|#########8| 656897/667659 [04:06<00:02, 3955.46 examples/s]
hf-map ta (num_proc=8):  99%|#########8| 657921/667659 [04:06<00:02, 4435.01 examples/s]
hf-map ta (num_proc=8):  99%|#########8| 658434/667659 [04:07<00:03, 2958.97 examples/s]
hf-map ta (num_proc=8):  99%|#########8| 659458/667659 [04:07<00:02, 3269.23 examples/s]
hf-map ta (num_proc=8):  99%|#########8| 660482/667659 [04:08<00:03, 2259.55 examples/s]
hf-map ta (num_proc=8):  99%|#########9| 662530/667659 [04:08<00:01, 3731.75 examples/s]
hf-map ta (num_proc=8):  99%|#########9| 664068/667659 [04:09<00:01, 3430.39 examples/s]
hf-map ta (num_proc=8): 100%|#########9| 665092/667659 [04:09<00:00, 3726.88 examples/s]
hf-map ta (num_proc=8): 100%|#########9| 666119/667659 [04:09<00:00, 3109.26 examples/s]
hf-map ta (num_proc=8): 100%|#########9| 667146/667659 [04:10<00:00, 2599.27 examples/s]
hf-map ta (num_proc=8): 100%|##########| 667659/667659 [04:10<00:00, 2480.57 examples/s]
hf-map ta (num_proc=8): 100%|##########| 667659/667659 [04:11<00:00, 2659.26 examples/s]
    ta: datasets.map num_proc=8 done 251s
    ta: 25000 docs  23325458 tokens  256s
    ta: 50000 docs  46092495 tokens  258s
    ta: 75000 docs  69234775 tokens  260s
    ta: 100000 docs  92178764 tokens  262s
    ta: 125000 docs  114953253 tokens  264s
    ta: 150000 docs  138281352 tokens  266s
    ta: 175000 docs  161404187 tokens  268s
    ta: 200000 docs  184541026 tokens  270s
    ta: 225000 docs  207872624 tokens  273s
    ta: 250000 docs  230922341 tokens  275s
    ta: 275000 docs  254309793 tokens  277s
    ta: 300000 docs  277730139 tokens  279s
    ta: 325000 docs  301150397 tokens  281s
    ta: 350000 docs  324210916 tokens  283s
    ta: 375000 docs  346690917 tokens  286s
    ta: 400000 docs  369202603 tokens  288s
    ta: 425000 docs  392444860 tokens  290s
    ta: 450000 docs  416252337 tokens  292s
    ta: 475000 docs  439421148 tokens  294s
  packed ta: 456292825 tokens (1.000 of budget)  rows=728619 docs=493028  pad-per-doc (HF map)
    kk: datasets.map n=386470 batched=1024 num_proc=8

hf-map kk (num_proc=8):   0%|          | 0/386470 [00:00<?, ? examples/s]
hf-map kk (num_proc=8):   0%|          | 1024/386470 [00:13<1:27:24, 73.50 examples/s]
hf-map kk (num_proc=8):   1%|          | 2048/386470 [00:14<38:50, 164.97 examples/s] 
hf-map kk (num_proc=8):   2%|1         | 6144/386470 [00:14<09:02, 700.65 examples/s]
hf-map kk (num_proc=8):   2%|2         | 8192/386470 [00:15<05:55, 1065.45 examples/s]
hf-map kk (num_proc=8):   3%|2         | 10240/386470 [00:20<09:34, 654.52 examples/s]
hf-map kk (num_proc=8):   3%|3         | 12288/386470 [00:20<06:48, 916.47 examples/s]
hf-map kk (num_proc=8):   4%|3         | 15360/386470 [00:21<04:08, 1493.18 examples/s]
hf-map kk (num_proc=8):   5%|4         | 17408/386470 [00:24<05:53, 1044.96 examples/s]
hf-map kk (num_proc=8):   5%|4         | 18432/386470 [00:25<05:38, 1085.97 examples/s]
hf-map kk (num_proc=8):   5%|5         | 19456/386470 [00:25<05:06, 1196.74 examples/s]
hf-map kk (num_proc=8):   6%|5         | 21504/386470 [00:25<03:23, 1793.12 examples/s]
hf-map kk (num_proc=8):   6%|6         | 23552/386470 [00:26<02:39, 2271.86 examples/s]
hf-map kk (num_proc=8):   6%|6         | 24576/386470 [00:26<02:15, 2670.27 examples/s]
hf-map kk (num_proc=8):   7%|6         | 25600/386470 [00:29<06:08, 980.30 examples/s] 
hf-map kk (num_proc=8):   7%|6         | 26624/386470 [00:30<05:22, 1115.50 examples/s]
hf-map kk (num_proc=8):   7%|7         | 27648/386470 [00:30<04:35, 1301.00 examples/s]
hf-map kk (num_proc=8):   7%|7         | 28672/386470 [00:31<03:42, 1607.32 examples/s]
hf-map kk (num_proc=8):   8%|7         | 30720/386470 [00:31<02:14, 2635.22 examples/s]
hf-map kk (num_proc=8):   8%|8         | 32768/386470 [00:31<01:59, 2948.15 examples/s]
hf-map kk (num_proc=8):   9%|8         | 33792/386470 [00:34<05:11, 1132.08 examples/s]
hf-map kk (num_proc=8):   9%|9         | 34816/386470 [00:34<04:23, 1334.48 examples/s]
hf-map kk (num_proc=8):   9%|9         | 35840/386470 [00:35<04:18, 1356.87 examples/s]
hf-map kk (num_proc=8):  10%|9         | 36864/386470 [00:35<03:22, 1725.55 examples/s]
hf-map kk (num_proc=8):  10%|#         | 38912/386470 [00:35<02:03, 2825.20 examples/s]
hf-map kk (num_proc=8):  10%|#         | 39936/386470 [00:36<02:04, 2781.68 examples/s]
hf-map kk (num_proc=8):  11%|#         | 40960/386470 [00:36<01:52, 3064.07 examples/s]
hf-map kk (num_proc=8):  11%|#         | 41984/386470 [00:39<05:50, 983.56 examples/s] 
hf-map kk (num_proc=8):  11%|#1        | 43008/386470 [00:39<04:25, 1293.32 examples/s]
hf-map kk (num_proc=8):  11%|#1        | 44032/386470 [00:40<03:56, 1449.78 examples/s]
hf-map kk (num_proc=8):  12%|#1        | 45056/386470 [00:40<03:05, 1844.89 examples/s]
hf-map kk (num_proc=8):  12%|#2        | 48128/386470 [00:40<01:43, 3270.51 examples/s]
hf-map kk (num_proc=8):  13%|#2        | 49152/386470 [00:41<01:41, 3319.47 examples/s]
hf-map kk (num_proc=8):  13%|#2        | 50176/386470 [00:44<05:23, 1038.95 examples/s]
hf-map kk (num_proc=8):  13%|#3        | 51200/386470 [00:44<04:21, 1282.43 examples/s]
hf-map kk (num_proc=8):  14%|#3        | 53248/386470 [00:44<02:39, 2092.97 examples/s]
hf-map kk (num_proc=8):  14%|#4        | 54272/386470 [00:45<02:33, 2164.98 examples/s]
hf-map kk (num_proc=8):  15%|#4        | 57344/386470 [00:45<01:44, 3141.49 examples/s]
hf-map kk (num_proc=8):  15%|#5        | 58368/386470 [00:48<04:53, 1119.55 examples/s]
hf-map kk (num_proc=8):  16%|#5        | 61440/386470 [00:49<02:48, 1932.21 examples/s]
hf-map kk (num_proc=8):  16%|#6        | 62464/386470 [00:49<02:38, 2038.72 examples/s]
hf-map kk (num_proc=8):  16%|#6        | 63488/386470 [00:49<02:17, 2346.18 examples/s]
hf-map kk (num_proc=8):  17%|#6        | 65536/386470 [00:50<01:52, 2850.02 examples/s]
hf-map kk (num_proc=8):  17%|#7        | 66560/386470 [00:53<04:48, 1110.13 examples/s]
hf-map kk (num_proc=8):  17%|#7        | 67584/386470 [00:53<04:07, 1289.70 examples/s]
hf-map kk (num_proc=8):  18%|#8        | 69632/386470 [00:53<02:35, 2034.12 examples/s]
hf-map kk (num_proc=8):  19%|#8        | 71680/386470 [00:54<02:00, 2611.87 examples/s]
hf-map kk (num_proc=8):  19%|#8        | 72704/386470 [00:54<01:56, 2695.09 examples/s]
hf-map kk (num_proc=8):  19%|#9        | 73728/386470 [00:54<01:39, 3131.34 examples/s]
hf-map kk (num_proc=8):  19%|#9        | 74752/386470 [00:57<04:56, 1052.86 examples/s]
hf-map kk (num_proc=8):  20%|#9        | 76800/386470 [00:57<03:00, 1719.23 examples/s]
hf-map kk (num_proc=8):  20%|##        | 77824/386470 [00:57<02:33, 2012.41 examples/s]
hf-map kk (num_proc=8):  20%|##        | 78848/386470 [00:57<02:04, 2475.90 examples/s]
hf-map kk (num_proc=8):  21%|##        | 79872/386470 [00:58<01:50, 2778.64 examples/s]
hf-map kk (num_proc=8):  21%|##        | 80896/386470 [00:58<01:59, 2554.15 examples/s]
hf-map kk (num_proc=8):  21%|##1       | 81920/386470 [00:58<01:43, 2944.49 examples/s]
hf-map kk (num_proc=8):  21%|##1       | 82944/386470 [01:01<04:32, 1113.93 examples/s]
hf-map kk (num_proc=8):  22%|##1       | 83968/386470 [01:01<03:29, 1441.38 examples/s]
hf-map kk (num_proc=8):  22%|##1       | 84992/386470 [01:01<02:49, 1777.49 examples/s]
hf-map kk (num_proc=8):  22%|##2       | 86016/386470 [01:01<02:17, 2183.23 examples/s]
hf-map kk (num_proc=8):  23%|##2       | 87040/386470 [01:02<01:46, 2799.24 examples/s]
hf-map kk (num_proc=8):  23%|##2       | 88064/386470 [01:02<01:53, 2618.30 examples/s]
hf-map kk (num_proc=8):  23%|##3       | 89088/386470 [01:02<01:44, 2846.58 examples/s]
hf-map kk (num_proc=8):  23%|##3       | 90112/386470 [01:03<01:35, 3091.38 examples/s]
hf-map kk (num_proc=8):  24%|##3       | 91136/386470 [01:05<04:05, 1203.56 examples/s]
hf-map kk (num_proc=8):  24%|##3       | 92160/386470 [01:05<03:42, 1325.25 examples/s]
hf-map kk (num_proc=8):  24%|##4       | 94208/386470 [01:05<02:14, 2175.49 examples/s]
hf-map kk (num_proc=8):  25%|##4       | 95232/386470 [01:06<01:56, 2505.50 examples/s]
hf-map kk (num_proc=8):  25%|##4       | 96256/386470 [01:06<02:07, 2275.47 examples/s]
hf-map kk (num_proc=8):  25%|##5       | 98304/386470 [01:06<01:20, 3580.45 examples/s]
hf-map kk (num_proc=8):  26%|##5       | 99328/386470 [01:09<03:37, 1318.81 examples/s]
hf-map kk (num_proc=8):  26%|##5       | 100352/386470 [01:09<02:59, 1596.51 examples/s]
hf-map kk (num_proc=8):  26%|##6       | 101376/386470 [01:09<02:23, 1989.13 examples/s]
hf-map kk (num_proc=8):  26%|##6       | 102400/386470 [01:09<02:09, 2194.40 examples/s]
hf-map kk (num_proc=8):  27%|##6       | 103424/386470 [01:10<02:03, 2290.34 examples/s]
hf-map kk (num_proc=8):  27%|##7       | 104448/386470 [01:10<01:38, 2851.88 examples/s]
hf-map kk (num_proc=8):  28%|##7       | 106496/386470 [01:10<01:05, 4250.97 examples/s]
hf-map kk (num_proc=8):  28%|##7       | 107520/386470 [01:13<03:33, 1304.65 examples/s]
hf-map kk (num_proc=8):  28%|##8       | 109568/386470 [01:13<02:21, 1960.78 examples/s]
hf-map kk (num_proc=8):  29%|##8       | 110592/386470 [01:13<02:19, 1976.68 examples/s]
hf-map kk (num_proc=8):  29%|##8       | 111616/386470 [01:14<02:21, 1944.74 examples/s]
hf-map kk (num_proc=8):  29%|##9       | 113664/386470 [01:14<01:37, 2784.24 examples/s]
hf-map kk (num_proc=8):  30%|##9       | 115712/386470 [01:16<02:35, 1740.61 examples/s]
hf-map kk (num_proc=8):  30%|###       | 116736/386470 [01:16<02:12, 2039.44 examples/s]
hf-map kk (num_proc=8):  30%|###       | 117760/386470 [01:17<02:23, 1866.54 examples/s]
hf-map kk (num_proc=8):  31%|###       | 118784/386470 [01:17<02:03, 2161.10 examples/s]
hf-map kk (num_proc=8):  31%|###1      | 119808/386470 [01:18<01:52, 2367.72 examples/s]
hf-map kk (num_proc=8):  31%|###1      | 120832/386470 [01:18<01:33, 2852.06 examples/s]
hf-map kk (num_proc=8):  32%|###1      | 121856/386470 [01:18<01:42, 2576.70 examples/s]
hf-map kk (num_proc=8):  32%|###1      | 122880/386470 [01:19<01:33, 2820.25 examples/s]
hf-map kk (num_proc=8):  32%|###2      | 123904/386470 [01:20<02:38, 1660.01 examples/s]
hf-map kk (num_proc=8):  33%|###2      | 125952/386470 [01:21<02:03, 2104.10 examples/s]
hf-map kk (num_proc=8):  33%|###2      | 126976/386470 [01:21<02:08, 2021.72 examples/s]
hf-map kk (num_proc=8):  33%|###3      | 129024/386470 [01:22<01:36, 2665.31 examples/s]
hf-map kk (num_proc=8):  34%|###3      | 130048/386470 [01:22<01:45, 2436.04 examples/s]
hf-map kk (num_proc=8):  34%|###4      | 132096/386470 [01:23<02:09, 1970.71 examples/s]
hf-map kk (num_proc=8):  35%|###4      | 134144/386470 [01:24<02:02, 2057.95 examples/s]
hf-map kk (num_proc=8):  35%|###4      | 135168/386470 [01:24<01:43, 2422.96 examples/s]
hf-map kk (num_proc=8):  36%|###5      | 137216/386470 [01:25<01:19, 3143.38 examples/s]
hf-map kk (num_proc=8):  36%|###5      | 138240/386470 [01:26<01:43, 2403.10 examples/s]
hf-map kk (num_proc=8):  36%|###6      | 139264/386470 [01:26<01:28, 2797.15 examples/s]
hf-map kk (num_proc=8):  36%|###6      | 140288/386470 [01:27<01:52, 2179.11 examples/s]
hf-map kk (num_proc=8):  37%|###6      | 141312/386470 [01:27<01:36, 2543.62 examples/s]
hf-map kk (num_proc=8):  37%|###6      | 142336/386470 [01:28<02:24, 1690.77 examples/s]
hf-map kk (num_proc=8):  37%|###7      | 143360/386470 [01:28<01:50, 2196.51 examples/s]
hf-map kk (num_proc=8):  37%|###7      | 144384/386470 [01:28<01:30, 2677.78 examples/s]
hf-map kk (num_proc=8):  38%|###7      | 146432/386470 [01:29<01:25, 2818.89 examples/s]
hf-map kk (num_proc=8):  38%|###8      | 147456/386470 [01:29<01:29, 2683.43 examples/s]
hf-map kk (num_proc=8):  38%|###8      | 148480/386470 [01:30<01:39, 2388.00 examples/s]
hf-map kk (num_proc=8):  39%|###8      | 149504/386470 [01:30<01:44, 2266.38 examples/s]
hf-map kk (num_proc=8):  39%|###8      | 150528/386470 [01:31<02:08, 1842.23 examples/s]
hf-map kk (num_proc=8):  39%|###9      | 152576/386470 [01:32<01:27, 2662.83 examples/s]
hf-map kk (num_proc=8):  40%|####      | 154624/386470 [01:32<01:25, 2718.53 examples/s]
hf-map kk (num_proc=8):  40%|####      | 155648/386470 [01:33<01:27, 2629.08 examples/s]
hf-map kk (num_proc=8):  41%|####      | 156672/386470 [01:34<01:51, 2064.80 examples/s]
hf-map kk (num_proc=8):  41%|####1     | 158720/386470 [01:34<01:42, 2222.77 examples/s]
hf-map kk (num_proc=8):  41%|####1     | 159744/386470 [01:35<01:36, 2339.18 examples/s]
hf-map kk (num_proc=8):  42%|####1     | 160768/386470 [01:35<01:22, 2723.53 examples/s]
hf-map kk (num_proc=8):  42%|####2     | 162816/386470 [01:36<01:18, 2841.45 examples/s]
hf-map kk (num_proc=8):  42%|####2     | 163840/386470 [01:36<01:23, 2650.97 examples/s]
hf-map kk (num_proc=8):  43%|####2     | 164864/386470 [01:37<01:35, 2309.12 examples/s]
hf-map kk (num_proc=8):  43%|####2     | 165888/386470 [01:37<01:22, 2658.35 examples/s]
hf-map kk (num_proc=8):  43%|####3     | 166912/386470 [01:38<01:50, 1979.99 examples/s]
hf-map kk (num_proc=8):  43%|####3     | 167936/386470 [01:38<01:41, 2147.85 examples/s]
hf-map kk (num_proc=8):  44%|####3     | 168960/386470 [01:38<01:27, 2495.68 examples/s]
hf-map kk (num_proc=8):  44%|####4     | 171008/386470 [01:39<01:10, 3061.33 examples/s]
hf-map kk (num_proc=8):  45%|####4     | 172032/386470 [01:39<01:18, 2731.42 examples/s]
hf-map kk (num_proc=8):  45%|####4     | 173056/386470 [01:40<01:44, 2036.23 examples/s]
hf-map kk (num_proc=8):  45%|####5     | 174080/386470 [01:40<01:25, 2497.14 examples/s]
hf-map kk (num_proc=8):  45%|####5     | 175104/386470 [01:41<01:52, 1880.88 examples/s]
hf-map kk (num_proc=8):  46%|####5     | 177152/386470 [01:42<01:20, 2609.13 examples/s]
hf-map kk (num_proc=8):  46%|####6     | 178176/386470 [01:42<01:11, 2923.25 examples/s]
hf-map kk (num_proc=8):  46%|####6     | 179200/386470 [01:42<01:05, 3188.24 examples/s]
hf-map kk (num_proc=8):  47%|####6     | 180224/386470 [01:43<01:23, 2460.20 examples/s]
hf-map kk (num_proc=8):  47%|####6     | 181248/386470 [01:43<01:06, 3107.32 examples/s]
hf-map kk (num_proc=8):  47%|####7     | 182272/386470 [01:44<01:26, 2354.94 examples/s]
hf-map kk (num_proc=8):  47%|####7     | 183296/386470 [01:44<01:30, 2240.05 examples/s]
hf-map kk (num_proc=8):  48%|####7     | 184320/386470 [01:44<01:19, 2536.04 examples/s]
hf-map kk (num_proc=8):  48%|####7     | 185344/386470 [01:45<01:19, 2542.54 examples/s]
hf-map kk (num_proc=8):  48%|####8     | 186368/386470 [01:45<01:14, 2684.49 examples/s]
hf-map kk (num_proc=8):  48%|####8     | 187392/386470 [01:45<01:12, 2745.50 examples/s]
hf-map kk (num_proc=8):  49%|####8     | 188416/386470 [01:46<01:17, 2560.22 examples/s]
hf-map kk (num_proc=8):  49%|####9     | 189440/386470 [01:46<01:10, 2795.96 examples/s]
hf-map kk (num_proc=8):  49%|####9     | 190464/386470 [01:47<01:26, 2259.37 examples/s]
hf-map kk (num_proc=8):  50%|####9     | 191488/386470 [01:47<01:17, 2511.28 examples/s]
hf-map kk (num_proc=8):  50%|####9     | 192512/386470 [01:48<01:19, 2442.65 examples/s]
hf-map kk (num_proc=8):  50%|#####     | 193536/386470 [01:48<01:05, 2946.38 examples/s]
hf-map kk (num_proc=8):  50%|#####     | 194560/386470 [01:48<01:14, 2589.37 examples/s]
hf-map kk (num_proc=8):  51%|#####     | 195584/386470 [01:49<01:13, 2580.28 examples/s]
hf-map kk (num_proc=8):  51%|#####     | 196608/386470 [01:49<01:16, 2495.68 examples/s]
hf-map kk (num_proc=8):  51%|#####1    | 197632/386470 [01:49<01:08, 2741.18 examples/s]
hf-map kk (num_proc=8):  51%|#####1    | 198656/386470 [01:50<01:36, 1938.52 examples/s]
hf-map kk (num_proc=8):  52%|#####1    | 199680/386470 [01:51<01:20, 2318.39 examples/s]
hf-map kk (num_proc=8):  52%|#####1    | 200704/386470 [01:51<01:12, 2565.01 examples/s]
hf-map kk (num_proc=8):  52%|#####2    | 201728/386470 [01:51<01:00, 3053.11 examples/s]
hf-map kk (num_proc=8):  52%|#####2    | 202752/386470 [01:51<01:03, 2902.05 examples/s]
hf-map kk (num_proc=8):  53%|#####2    | 203776/386470 [01:52<01:05, 2782.61 examples/s]
hf-map kk (num_proc=8):  53%|#####2    | 204800/386470 [01:52<01:13, 2462.08 examples/s]
hf-map kk (num_proc=8):  53%|#####3    | 205824/386470 [01:53<01:19, 2261.91 examples/s]
hf-map kk (num_proc=8):  54%|#####3    | 206848/386470 [01:53<01:17, 2310.94 examples/s]
hf-map kk (num_proc=8):  54%|#####3    | 207872/386470 [01:54<01:09, 2588.10 examples/s]
hf-map kk (num_proc=8):  54%|#####4    | 208896/386470 [01:54<01:18, 2256.35 examples/s]
hf-map kk (num_proc=8):  54%|#####4    | 209920/386470 [01:54<01:02, 2843.29 examples/s]
hf-map kk (num_proc=8):  55%|#####4    | 210944/386470 [01:55<01:10, 2480.34 examples/s]
hf-map kk (num_proc=8):  55%|#####4    | 211968/386470 [01:55<01:18, 2230.62 examples/s]
hf-map kk (num_proc=8):  55%|#####5    | 212992/386470 [01:56<01:16, 2261.80 examples/s]
hf-map kk (num_proc=8):  55%|#####5    | 214016/386470 [01:56<01:21, 2107.79 examples/s]
hf-map kk (num_proc=8):  56%|#####5    | 215040/386470 [01:57<01:02, 2732.98 examples/s]
hf-map kk (num_proc=8):  56%|#####5    | 216064/386470 [01:57<01:11, 2385.76 examples/s]
hf-map kk (num_proc=8):  56%|#####6    | 217088/386470 [01:58<01:09, 2423.54 examples/s]
hf-map kk (num_proc=8):  56%|#####6    | 218112/386470 [01:58<01:00, 2799.65 examples/s]
hf-map kk (num_proc=8):  57%|#####6    | 220160/386470 [01:59<01:35, 1739.16 examples/s]
hf-map kk (num_proc=8):  57%|#####7    | 221184/386470 [02:00<01:21, 2022.93 examples/s]
hf-map kk (num_proc=8):  57%|#####7    | 222208/386470 [02:00<01:08, 2396.76 examples/s]
hf-map kk (num_proc=8):  58%|#####8    | 224256/386470 [02:01<01:03, 2573.51 examples/s]
hf-map kk (num_proc=8):  59%|#####8    | 226304/386470 [02:01<00:45, 3485.89 examples/s]
hf-map kk (num_proc=8):  59%|#####8    | 227328/386470 [02:01<00:47, 3372.18 examples/s]
hf-map kk (num_proc=8):  59%|#####9    | 228352/386470 [02:03<01:31, 1733.88 examples/s]
hf-map kk (num_proc=8):  60%|#####9    | 230400/386470 [02:03<01:12, 2148.15 examples/s]
hf-map kk (num_proc=8):  60%|######    | 232448/386470 [02:04<01:02, 2445.85 examples/s]
hf-map kk (num_proc=8):  60%|######    | 233472/386470 [02:04<00:56, 2699.96 examples/s]
hf-map kk (num_proc=8):  61%|######    | 234496/386470 [02:04<00:47, 3166.55 examples/s]
hf-map kk (num_proc=8):  61%|######    | 235520/386470 [02:05<00:54, 2794.61 examples/s]
hf-map kk (num_proc=8):  61%|######1   | 236544/386470 [02:06<01:17, 1930.31 examples/s]
hf-map kk (num_proc=8):  61%|######1   | 237568/386470 [02:06<01:04, 2293.26 examples/s]
hf-map kk (num_proc=8):  62%|######1   | 238592/386470 [02:07<01:11, 2059.62 examples/s]
hf-map kk (num_proc=8):  62%|######2   | 239616/386470 [02:07<01:07, 2186.21 examples/s]
hf-map kk (num_proc=8):  63%|######2   | 241664/386470 [02:08<00:49, 2954.11 examples/s]
hf-map kk (num_proc=8):  63%|######3   | 243712/386470 [02:08<00:42, 3358.76 examples/s]
hf-map kk (num_proc=8):  63%|######3   | 244736/386470 [02:09<00:58, 2430.56 examples/s]
hf-map kk (num_proc=8):  64%|######3   | 245760/386470 [02:09<00:52, 2684.90 examples/s]
hf-map kk (num_proc=8):  64%|######3   | 246784/386470 [02:10<01:09, 2015.62 examples/s]
hf-map kk (num_proc=8):  64%|######4   | 247808/386470 [02:10<01:02, 2235.62 examples/s]
hf-map kk (num_proc=8):  64%|######4   | 248832/386470 [02:10<00:49, 2802.29 examples/s]
hf-map kk (num_proc=8):  65%|######4   | 249856/386470 [02:11<00:47, 2874.18 examples/s]
hf-map kk (num_proc=8):  65%|######4   | 250880/386470 [02:11<00:40, 3371.23 examples/s]
hf-map kk (num_proc=8):  65%|######5   | 251904/386470 [02:11<00:40, 3307.89 examples/s]
hf-map kk (num_proc=8):  65%|######5   | 252928/386470 [02:12<00:57, 2337.37 examples/s]
hf-map kk (num_proc=8):  66%|######5   | 253952/386470 [02:13<00:59, 2224.09 examples/s]
hf-map kk (num_proc=8):  66%|######5   | 254976/386470 [02:13<01:06, 1969.98 examples/s]
hf-map kk (num_proc=8):  66%|######6   | 256000/386470 [02:14<01:00, 2164.73 examples/s]
hf-map kk (num_proc=8):  67%|######6   | 258048/386470 [02:14<00:48, 2665.29 examples/s]
hf-map kk (num_proc=8):  67%|######7   | 259072/386470 [02:14<00:41, 3048.57 examples/s]
hf-map kk (num_proc=8):  67%|######7   | 260096/386470 [02:14<00:35, 3556.39 examples/s]
hf-map kk (num_proc=8):  68%|######7   | 261120/386470 [02:15<00:42, 2950.48 examples/s]
hf-map kk (num_proc=8):  68%|######7   | 262144/386470 [02:15<00:46, 2688.99 examples/s]
hf-map kk (num_proc=8):  68%|######8   | 263168/386470 [02:16<00:58, 2111.68 examples/s]
hf-map kk (num_proc=8):  68%|######8   | 264192/386470 [02:17<01:04, 1888.06 examples/s]
hf-map kk (num_proc=8):  69%|######8   | 265216/386470 [02:17<00:50, 2396.85 examples/s]
hf-map kk (num_proc=8):  69%|######9   | 267264/386470 [02:18<00:46, 2549.89 examples/s]
hf-map kk (num_proc=8):  70%|######9   | 269312/386470 [02:18<00:39, 2989.00 examples/s]
hf-map kk (num_proc=8):  70%|######9   | 270336/386470 [02:19<00:42, 2748.76 examples/s]
hf-map kk (num_proc=8):  70%|#######   | 271360/386470 [02:20<00:54, 2123.21 examples/s]
hf-map kk (num_proc=8):  70%|#######   | 272384/386470 [02:20<00:55, 2058.14 examples/s]
hf-map kk (num_proc=8):  71%|#######   | 273408/386470 [02:20<00:49, 2272.89 examples/s]
hf-map kk (num_proc=8):  71%|#######1  | 274432/386470 [02:21<00:41, 2715.98 examples/s]
hf-map kk (num_proc=8):  71%|#######1  | 275456/386470 [02:21<00:39, 2782.17 examples/s]
hf-map kk (num_proc=8):  72%|#######1  | 277504/386470 [02:22<00:35, 3113.04 examples/s]
hf-map kk (num_proc=8):  72%|#######2  | 278528/386470 [02:22<00:36, 2931.94 examples/s]
hf-map kk (num_proc=8):  72%|#######2  | 279552/386470 [02:23<00:54, 1951.74 examples/s]
hf-map kk (num_proc=8):  73%|#######2  | 280576/386470 [02:23<00:44, 2373.18 examples/s]
hf-map kk (num_proc=8):  73%|#######2  | 281600/386470 [02:24<00:45, 2312.16 examples/s]
hf-map kk (num_proc=8):  73%|#######3  | 282624/386470 [02:24<00:37, 2749.52 examples/s]
hf-map kk (num_proc=8):  73%|#######3  | 283648/386470 [02:24<00:33, 3046.97 examples/s]
hf-map kk (num_proc=8):  74%|#######3  | 284672/386470 [02:24<00:31, 3202.40 examples/s]
hf-map kk (num_proc=8):  74%|#######3  | 285696/386470 [02:24<00:26, 3843.27 examples/s]
hf-map kk (num_proc=8):  74%|#######4  | 286720/386470 [02:25<00:32, 3063.18 examples/s]
hf-map kk (num_proc=8):  74%|#######4  | 287744/386470 [02:26<00:53, 1837.16 examples/s]
hf-map kk (num_proc=8):  75%|#######4  | 289792/386470 [02:27<00:41, 2311.58 examples/s]
hf-map kk (num_proc=8):  75%|#######5  | 290816/386470 [02:27<00:40, 2341.67 examples/s]
hf-map kk (num_proc=8):  76%|#######5  | 292864/386470 [02:27<00:29, 3126.50 examples/s]
hf-map kk (num_proc=8):  76%|#######6  | 294912/386470 [02:28<00:30, 2996.70 examples/s]
hf-map kk (num_proc=8):  77%|#######6  | 295936/386470 [02:29<00:39, 2305.95 examples/s]
hf-map kk (num_proc=8):  77%|#######6  | 296960/386470 [02:29<00:33, 2668.50 examples/s]
hf-map kk (num_proc=8):  77%|#######7  | 297984/386470 [02:30<00:43, 2052.21 examples/s]
hf-map kk (num_proc=8):  78%|#######7  | 300032/386470 [02:30<00:27, 3141.02 examples/s]
hf-map kk (num_proc=8):  78%|#######8  | 302080/386470 [02:31<00:26, 3173.89 examples/s]
hf-map kk (num_proc=8):  79%|#######8  | 304128/386470 [02:32<00:36, 2264.10 examples/s]
hf-map kk (num_proc=8):  79%|#######8  | 305152/386470 [02:32<00:33, 2437.36 examples/s]
hf-map kk (num_proc=8):  79%|#######9  | 306176/386470 [02:33<00:40, 1958.95 examples/s]
hf-map kk (num_proc=8):  80%|########  | 309248/386470 [02:34<00:22, 3363.98 examples/s]
hf-map kk (num_proc=8):  80%|########  | 310272/386470 [02:34<00:22, 3454.48 examples/s]
hf-map kk (num_proc=8):  81%|########  | 311296/386470 [02:34<00:25, 2920.61 examples/s]
hf-map kk (num_proc=8):  81%|########  | 312320/386470 [02:36<00:40, 1832.97 examples/s]
hf-map kk (num_proc=8):  81%|########1 | 313344/386470 [02:36<00:36, 2023.79 examples/s]
hf-map kk (num_proc=8):  81%|########1 | 314368/386470 [02:36<00:34, 2108.83 examples/s]
hf-map kk (num_proc=8):  82%|########1 | 315392/386470 [02:37<00:27, 2608.92 examples/s]
hf-map kk (num_proc=8):  82%|########2 | 317440/386470 [02:37<00:17, 3855.09 examples/s]
hf-map kk (num_proc=8):  82%|########2 | 318464/386470 [02:38<00:26, 2613.62 examples/s]
hf-map kk (num_proc=8):  83%|########2 | 319488/386470 [02:38<00:26, 2550.80 examples/s]
hf-map kk (num_proc=8):  83%|########2 | 320512/386470 [02:39<00:34, 1927.63 examples/s]
hf-map kk (num_proc=8):  83%|########3 | 321536/386470 [02:39<00:29, 2227.43 examples/s]
hf-map kk (num_proc=8):  83%|########3 | 322560/386470 [02:40<00:28, 2265.67 examples/s]
hf-map kk (num_proc=8):  84%|########3 | 324608/386470 [02:40<00:19, 3190.75 examples/s]
hf-map kk (num_proc=8):  85%|########4 | 326656/386470 [02:41<00:20, 2879.39 examples/s]
hf-map kk (num_proc=8):  85%|########4 | 327680/386470 [02:42<00:27, 2163.88 examples/s]
hf-map kk (num_proc=8):  85%|########5 | 328704/386470 [02:42<00:29, 1940.02 examples/s]
hf-map kk (num_proc=8):  86%|########5 | 330752/386470 [02:43<00:21, 2649.87 examples/s]
hf-map kk (num_proc=8):  86%|########5 | 331776/386470 [02:43<00:17, 3106.14 examples/s]
hf-map kk (num_proc=8):  86%|########6 | 332800/386470 [02:43<00:15, 3504.84 examples/s]
hf-map kk (num_proc=8):  86%|########6 | 333824/386470 [02:43<00:13, 3969.77 examples/s]
hf-map kk (num_proc=8):  87%|########6 | 334848/386470 [02:45<00:30, 1674.19 examples/s]
hf-map kk (num_proc=8):  87%|########6 | 335872/386470 [02:45<00:23, 2156.26 examples/s]
hf-map kk (num_proc=8):  87%|########7 | 336896/386470 [02:46<00:27, 1823.73 examples/s]
hf-map kk (num_proc=8):  87%|########7 | 337920/386470 [02:46<00:26, 1810.99 examples/s]
hf-map kk (num_proc=8):  88%|########7 | 338944/386470 [02:46<00:22, 2134.70 examples/s]
hf-map kk (num_proc=8):  88%|########7 | 339968/386470 [02:47<00:21, 2197.47 examples/s]
hf-map kk (num_proc=8):  89%|########8 | 343040/386470 [02:48<00:17, 2443.29 examples/s]
hf-map kk (num_proc=8):  89%|########9 | 344064/386470 [02:48<00:14, 2886.87 examples/s]
hf-map kk (num_proc=8):  89%|########9 | 345088/386470 [02:49<00:21, 1953.38 examples/s]
hf-map kk (num_proc=8):  90%|########9 | 346112/386470 [02:50<00:19, 2044.12 examples/s]
hf-map kk (num_proc=8):  90%|########9 | 347136/386470 [02:50<00:17, 2281.40 examples/s]
hf-map kk (num_proc=8):  90%|######### | 348160/386470 [02:50<00:13, 2797.28 examples/s]
hf-map kk (num_proc=8):  90%|######### | 349184/386470 [02:50<00:12, 3076.47 examples/s]
hf-map kk (num_proc=8):  91%|######### | 350208/386470 [02:51<00:09, 3732.38 examples/s]
hf-map kk (num_proc=8):  91%|######### | 351232/386470 [02:51<00:13, 2690.88 examples/s]
hf-map kk (num_proc=8):  91%|#########1| 352256/386470 [02:51<00:11, 2890.45 examples/s]
hf-map kk (num_proc=8):  91%|#########1| 353280/386470 [02:53<00:21, 1529.03 examples/s]
hf-map kk (num_proc=8):  92%|#########1| 354304/386470 [02:53<00:19, 1677.19 examples/s]
hf-map kk (num_proc=8):  92%|#########2| 356352/386470 [02:54<00:12, 2487.70 examples/s]
hf-map kk (num_proc=8):  92%|#########2| 357376/386470 [02:54<00:09, 3019.36 examples/s]
hf-map kk (num_proc=8):  93%|#########2| 358400/386470 [02:54<00:09, 2920.16 examples/s]
hf-map kk (num_proc=8):  93%|#########3| 359424/386470 [02:54<00:08, 3332.41 examples/s]
hf-map kk (num_proc=8):  93%|#########3| 360448/386470 [02:55<00:08, 2905.96 examples/s]
hf-map kk (num_proc=8):  94%|#########3| 361472/386470 [02:56<00:16, 1495.15 examples/s]
hf-map kk (num_proc=8):  94%|#########3| 362496/386470 [02:57<00:13, 1742.61 examples/s]
hf-map kk (num_proc=8):  94%|#########4| 363520/386470 [02:57<00:10, 2116.65 examples/s]
hf-map kk (num_proc=8):  95%|#########4| 365568/386470 [02:57<00:06, 3296.48 examples/s]
hf-map kk (num_proc=8):  95%|#########4| 366592/386470 [02:57<00:05, 3578.60 examples/s]
hf-map kk (num_proc=8):  95%|#########5| 367616/386470 [02:58<00:04, 3826.08 examples/s]
hf-map kk (num_proc=8):  95%|#########5| 368640/386470 [02:58<00:05, 2981.63 examples/s]
hf-map kk (num_proc=8):  96%|#########5| 369664/386470 [03:00<00:11, 1520.69 examples/s]
hf-map kk (num_proc=8):  96%|#########5| 370688/386470 [03:00<00:09, 1679.72 examples/s]
hf-map kk (num_proc=8):  96%|#########6| 372736/386470 [03:00<00:05, 2735.75 examples/s]
hf-map kk (num_proc=8):  97%|#########6| 373760/386470 [03:01<00:04, 2867.58 examples/s]
hf-map kk (num_proc=8):  97%|#########6| 374784/386470 [03:01<00:05, 2172.29 examples/s]
hf-map kk (num_proc=8):  98%|#########7| 376832/386470 [03:02<00:03, 2965.05 examples/s]
hf-map kk (num_proc=8):  98%|#########7| 378037/386470 [03:03<00:04, 1710.54 examples/s]
hf-map kk (num_proc=8):  98%|#########8| 379061/386470 [03:03<00:03, 2138.73 examples/s]
hf-map kk (num_proc=8):  98%|#########8| 380085/386470 [03:03<00:02, 2649.09 examples/s]
hf-map kk (num_proc=8):  99%|#########8| 381109/386470 [03:04<00:01, 3190.75 examples/s]
hf-map kk (num_proc=8):  99%|#########8| 382133/386470 [03:04<00:01, 3334.09 examples/s]
hf-map kk (num_proc=8):  99%|#########9| 383700/386470 [03:04<00:00, 3813.59 examples/s]
hf-map kk (num_proc=8): 100%|#########9| 385085/386470 [03:05<00:00, 3200.81 examples/s]
hf-map kk (num_proc=8): 100%|#########9| 386290/386470 [03:06<00:00, 1927.83 examples/s]
hf-map kk (num_proc=8): 100%|##########| 386470/386470 [03:07<00:00, 2060.79 examples/s]
    kk: datasets.map num_proc=8 done 188s
    kk: 25000 docs  40985213 tokens  193s
    kk: 50000 docs  81894926 tokens  197s
    kk: 75000 docs  121953883 tokens  200s
    kk: 100000 docs  163346527 tokens  204s
    kk: 125000 docs  204439098 tokens  207s
    kk: 150000 docs  245100472 tokens  210s
    kk: 175000 docs  284907275 tokens  214s
    kk: 200000 docs  326211845 tokens  217s
    kk: 225000 docs  368265583 tokens  221s
    kk: 250000 docs  408948851 tokens  224s
    kk: 275000 docs  449889902 tokens  227s
  packed kk: 466196065 tokens (1.000 of budget)  rows=602321 docs=285249  pad-per-doc (HF map)
    tl: datasets.map n=50150 batched=1024 num_proc=8

hf-map tl (num_proc=8):   0%|          | 0/50150 [00:00<?, ? examples/s]
hf-map tl (num_proc=8):   2%|2         | 1024/50150 [00:06<04:47, 170.67 examples/s]
hf-map tl (num_proc=8):   8%|8         | 4096/50150 [00:06<00:52, 875.04 examples/s]
hf-map tl (num_proc=8):  14%|#4        | 7168/50150 [00:06<00:24, 1789.33 examples/s]
hf-map tl (num_proc=8):  18%|#8        | 9216/50150 [00:08<00:30, 1359.78 examples/s]
hf-map tl (num_proc=8):  25%|##4       | 12288/50150 [00:08<00:16, 2266.57 examples/s]
hf-map tl (num_proc=8):  29%|##8       | 14336/50150 [00:08<00:11, 3011.36 examples/s]
hf-map tl (num_proc=8):  33%|###2      | 16384/50150 [00:08<00:08, 3800.45 examples/s]
hf-map tl (num_proc=8):  37%|###6      | 18432/50150 [00:10<00:15, 2113.45 examples/s]
hf-map tl (num_proc=8):  45%|####4     | 22528/50150 [00:11<00:07, 3643.83 examples/s]
hf-map tl (num_proc=8):  49%|####9     | 24576/50150 [00:11<00:05, 4355.40 examples/s]
hf-map tl (num_proc=8):  53%|#####3    | 26624/50150 [00:12<00:08, 2616.28 examples/s]
hf-map tl (num_proc=8):  57%|#####7    | 28672/50150 [00:13<00:06, 3164.06 examples/s]
hf-map tl (num_proc=8):  63%|######3   | 31744/50150 [00:13<00:03, 4609.14 examples/s]
hf-map tl (num_proc=8):  67%|######7   | 33792/50150 [00:15<00:06, 2689.02 examples/s]
hf-map tl (num_proc=8):  69%|######9   | 34816/50150 [00:15<00:05, 2799.48 examples/s]
hf-map tl (num_proc=8):  76%|#######5  | 37888/50150 [00:15<00:02, 4384.19 examples/s]
hf-map tl (num_proc=8):  80%|#######9  | 39936/50150 [00:15<00:02, 4976.26 examples/s]
hf-map tl (num_proc=8):  84%|########3 | 41984/50150 [00:16<00:02, 3212.45 examples/s]
hf-map tl (num_proc=8):  86%|########6 | 43133/50150 [00:17<00:02, 3249.85 examples/s]
hf-map tl (num_proc=8):  90%|######### | 45181/50150 [00:17<00:01, 4234.28 examples/s]
hf-map tl (num_proc=8):  95%|#########4| 47478/50150 [00:17<00:00, 5464.28 examples/s]
hf-map tl (num_proc=8):  99%|#########9| 49775/50150 [00:17<00:00, 6862.21 examples/s]
hf-map tl (num_proc=8): 100%|##########| 50150/50150 [00:18<00:00, 2760.63 examples/s]
    tl: datasets.map num_proc=8 done 18s
    tl: 25000 docs  22165685 tokens  21s
  packed tl: 32043235 tokens (1.000 of budget)  rows=55838 docs=36428  pad-per-doc (HF map)
    en: datasets.map n=196400 batched=1024 num_proc=8

hf-map en (num_proc=8):   0%|          | 0/196400 [00:00<?, ? examples/s]
hf-map en (num_proc=8):   1%|          | 1024/196400 [00:12<40:14, 80.91 examples/s]
hf-map en (num_proc=8):   1%|1         | 2048/196400 [00:12<16:43, 193.73 examples/s]
hf-map en (num_proc=8):   2%|2         | 4096/196400 [00:13<06:23, 500.83 examples/s]
hf-map en (num_proc=8):   3%|3         | 6144/196400 [00:13<03:32, 895.24 examples/s]
hf-map en (num_proc=8):   5%|4         | 9216/196400 [00:16<03:32, 879.41 examples/s]
hf-map en (num_proc=8):   6%|5         | 11264/196400 [00:17<02:26, 1260.46 examples/s]
hf-map en (num_proc=8):   6%|6         | 12288/196400 [00:17<02:09, 1421.44 examples/s]
hf-map en (num_proc=8):   7%|6         | 13312/196400 [00:17<01:47, 1705.39 examples/s]
hf-map en (num_proc=8):   8%|7         | 15360/196400 [00:17<01:10, 2574.29 examples/s]
hf-map en (num_proc=8):   9%|8         | 17408/196400 [00:21<02:42, 1102.69 examples/s]
hf-map en (num_proc=8):  10%|#         | 20480/196400 [00:21<01:34, 1862.45 examples/s]
hf-map en (num_proc=8):  11%|#1        | 22528/196400 [00:21<01:09, 2495.28 examples/s]
hf-map en (num_proc=8):  13%|#2        | 24576/196400 [00:21<00:51, 3335.75 examples/s]
hf-map en (num_proc=8):  14%|#3        | 26624/196400 [00:25<01:59, 1417.96 examples/s]
hf-map en (num_proc=8):  15%|#5        | 29696/196400 [00:25<01:14, 2241.43 examples/s]
hf-map en (num_proc=8):  16%|#6        | 31744/196400 [00:25<01:00, 2700.21 examples/s]
hf-map en (num_proc=8):  17%|#7        | 33792/196400 [00:29<02:01, 1341.40 examples/s]
hf-map en (num_proc=8):  18%|#8        | 35840/196400 [00:29<01:28, 1812.85 examples/s]
hf-map en (num_proc=8):  19%|#9        | 37888/196400 [00:29<01:07, 2349.23 examples/s]
hf-map en (num_proc=8):  21%|##1       | 41984/196400 [00:32<01:31, 1683.67 examples/s]
hf-map en (num_proc=8):  22%|##1       | 43008/196400 [00:32<01:20, 1898.44 examples/s]
hf-map en (num_proc=8):  23%|##3       | 46080/196400 [00:33<00:51, 2917.22 examples/s]
hf-map en (num_proc=8):  25%|##4       | 48128/196400 [00:33<00:41, 3534.71 examples/s]
hf-map en (num_proc=8):  26%|##5       | 50176/196400 [00:36<01:26, 1682.03 examples/s]
hf-map en (num_proc=8):  27%|##7       | 53248/196400 [00:36<00:56, 2543.31 examples/s]
hf-map en (num_proc=8):  28%|##8       | 55296/196400 [00:36<00:44, 3146.69 examples/s]
hf-map en (num_proc=8):  30%|##9       | 58368/196400 [00:39<01:12, 1913.38 examples/s]
hf-map en (num_proc=8):  30%|###       | 59392/196400 [00:39<01:04, 2111.71 examples/s]
hf-map en (num_proc=8):  31%|###1      | 61440/196400 [00:39<00:51, 2640.83 examples/s]
hf-map en (num_proc=8):  33%|###2      | 64512/196400 [00:40<00:36, 3587.43 examples/s]
hf-map en (num_proc=8):  34%|###3      | 66560/196400 [00:42<01:06, 1949.15 examples/s]
hf-map en (num_proc=8):  35%|###5      | 69632/196400 [00:42<00:43, 2928.25 examples/s]
hf-map en (num_proc=8):  36%|###6      | 71680/196400 [00:42<00:35, 3469.58 examples/s]
hf-map en (num_proc=8):  37%|###7      | 72704/196400 [00:43<00:35, 3481.27 examples/s]
hf-map en (num_proc=8):  38%|###7      | 73728/196400 [00:43<00:31, 3841.77 examples/s]
hf-map en (num_proc=8):  38%|###8      | 74752/196400 [00:45<01:09, 1750.94 examples/s]
hf-map en (num_proc=8):  39%|###8      | 75776/196400 [00:45<00:58, 2058.27 examples/s]
hf-map en (num_proc=8):  40%|###9      | 77824/196400 [00:45<00:38, 3059.25 examples/s]
hf-map en (num_proc=8):  41%|####      | 79872/196400 [00:45<00:27, 4289.44 examples/s]
hf-map en (num_proc=8):  41%|####1     | 80896/196400 [00:46<00:29, 3877.40 examples/s]
hf-map en (num_proc=8):  42%|####1     | 81920/196400 [00:46<00:28, 4067.37 examples/s]
hf-map en (num_proc=8):  42%|####2     | 82944/196400 [00:47<00:54, 2071.37 examples/s]
hf-map en (num_proc=8):  43%|####2     | 83968/196400 [00:47<00:49, 2263.37 examples/s]
hf-map en (num_proc=8):  43%|####3     | 84992/196400 [00:48<00:40, 2751.85 examples/s]
hf-map en (num_proc=8):  44%|####3     | 86016/196400 [00:48<00:31, 3451.05 examples/s]
hf-map en (num_proc=8):  44%|####4     | 87040/196400 [00:48<00:28, 3808.40 examples/s]
hf-map en (num_proc=8):  45%|####5     | 89088/196400 [00:48<00:21, 5033.38 examples/s]
hf-map en (num_proc=8):  46%|####5     | 90112/196400 [00:48<00:25, 4187.36 examples/s]
hf-map en (num_proc=8):  46%|####6     | 91136/196400 [00:50<00:49, 2130.17 examples/s]
hf-map en (num_proc=8):  47%|####6     | 92160/196400 [00:50<00:45, 2278.84 examples/s]
hf-map en (num_proc=8):  48%|####7     | 94208/196400 [00:50<00:33, 3082.60 examples/s]
hf-map en (num_proc=8):  48%|####8     | 95232/196400 [00:51<00:35, 2814.50 examples/s]
hf-map en (num_proc=8):  50%|####9     | 97280/196400 [00:51<00:25, 3925.77 examples/s]
hf-map en (num_proc=8):  50%|#####     | 98304/196400 [00:51<00:23, 4089.60 examples/s]
hf-map en (num_proc=8):  51%|#####     | 99328/196400 [00:52<00:41, 2352.61 examples/s]
hf-map en (num_proc=8):  51%|#####1    | 100352/196400 [00:52<00:34, 2795.14 examples/s]
hf-map en (num_proc=8):  52%|#####2    | 102400/196400 [00:53<00:28, 3264.54 examples/s]
hf-map en (num_proc=8):  53%|#####2    | 103424/196400 [00:53<00:30, 3097.70 examples/s]
hf-map en (num_proc=8):  53%|#####3    | 104448/196400 [00:54<00:28, 3209.95 examples/s]
hf-map en (num_proc=8):  54%|#####4    | 106496/196400 [00:54<00:23, 3829.30 examples/s]
hf-map en (num_proc=8):  55%|#####4    | 107520/196400 [00:55<00:29, 2996.18 examples/s]
hf-map en (num_proc=8):  55%|#####5    | 108544/196400 [00:55<00:29, 2974.30 examples/s]
hf-map en (num_proc=8):  56%|#####6    | 110592/196400 [00:56<00:28, 3028.31 examples/s]
hf-map en (num_proc=8):  57%|#####6    | 111616/196400 [00:56<00:24, 3502.07 examples/s]
hf-map en (num_proc=8):  57%|#####7    | 112640/196400 [00:56<00:25, 3292.08 examples/s]
hf-map en (num_proc=8):  58%|#####7    | 113664/196400 [00:56<00:22, 3626.02 examples/s]
hf-map en (num_proc=8):  58%|#####8    | 114688/196400 [00:57<00:24, 3274.06 examples/s]
hf-map en (num_proc=8):  59%|#####8    | 115712/196400 [00:57<00:25, 3107.08 examples/s]
hf-map en (num_proc=8):  59%|#####9    | 116736/196400 [00:57<00:28, 2806.75 examples/s]
hf-map en (num_proc=8):  60%|#####9    | 117760/196400 [00:58<00:23, 3301.06 examples/s]
hf-map en (num_proc=8):  60%|######    | 118784/196400 [00:58<00:25, 3053.52 examples/s]
hf-map en (num_proc=8):  62%|######1   | 120832/196400 [00:58<00:17, 4320.32 examples/s]
hf-map en (num_proc=8):  62%|######2   | 121856/196400 [00:59<00:21, 3523.73 examples/s]
hf-map en (num_proc=8):  63%|######2   | 122880/196400 [00:59<00:19, 3773.68 examples/s]
hf-map en (num_proc=8):  63%|######3   | 123904/196400 [01:00<00:29, 2477.06 examples/s]
hf-map en (num_proc=8):  64%|######3   | 124928/196400 [01:00<00:28, 2535.53 examples/s]
hf-map en (num_proc=8):  65%|######4   | 126976/196400 [01:00<00:18, 3783.20 examples/s]
hf-map en (num_proc=8):  65%|######5   | 128000/196400 [01:00<00:15, 4441.07 examples/s]
hf-map en (num_proc=8):  66%|######5   | 129024/196400 [01:01<00:15, 4338.10 examples/s]
hf-map en (num_proc=8):  66%|######6   | 130048/196400 [01:01<00:19, 3338.82 examples/s]
hf-map en (num_proc=8):  67%|######6   | 131072/196400 [01:01<00:17, 3805.35 examples/s]
hf-map en (num_proc=8):  67%|######7   | 132096/196400 [01:02<00:19, 3259.11 examples/s]
hf-map en (num_proc=8):  68%|######7   | 133120/196400 [01:03<00:32, 1918.76 examples/s]
hf-map en (num_proc=8):  69%|######8   | 135168/196400 [01:03<00:19, 3090.30 examples/s]
hf-map en (num_proc=8):  70%|######9   | 137216/196400 [01:03<00:14, 3997.83 examples/s]
hf-map en (num_proc=8):  70%|#######   | 138240/196400 [01:04<00:16, 3558.11 examples/s]
hf-map en (num_proc=8):  71%|#######   | 139264/196400 [01:04<00:14, 4060.22 examples/s]
hf-map en (num_proc=8):  71%|#######1  | 140288/196400 [01:04<00:13, 4280.15 examples/s]
hf-map en (num_proc=8):  72%|#######1  | 141312/196400 [01:05<00:28, 1948.07 examples/s]
hf-map en (num_proc=8):  72%|#######2  | 142336/196400 [01:05<00:21, 2498.19 examples/s]
hf-map en (num_proc=8):  74%|#######3  | 144384/196400 [01:06<00:13, 3757.51 examples/s]
hf-map en (num_proc=8):  74%|#######4  | 145408/196400 [01:06<00:16, 3155.35 examples/s]
hf-map en (num_proc=8):  75%|#######4  | 146432/196400 [01:06<00:13, 3696.41 examples/s]
hf-map en (num_proc=8):  75%|#######5  | 147456/196400 [01:07<00:12, 3880.24 examples/s]
hf-map en (num_proc=8):  76%|#######5  | 148480/196400 [01:07<00:12, 3989.36 examples/s]
hf-map en (num_proc=8):  76%|#######6  | 149504/196400 [01:08<00:21, 2198.52 examples/s]
hf-map en (num_proc=8):  77%|#######7  | 151552/196400 [01:08<00:15, 2963.51 examples/s]
hf-map en (num_proc=8):  78%|#######8  | 153600/196400 [01:09<00:12, 3305.29 examples/s]
hf-map en (num_proc=8):  79%|#######8  | 154624/196400 [01:09<00:11, 3547.79 examples/s]
hf-map en (num_proc=8):  80%|#######9  | 156672/196400 [01:10<00:13, 3001.74 examples/s]
hf-map en (num_proc=8):  80%|########  | 157696/196400 [01:10<00:13, 2856.76 examples/s]
hf-map en (num_proc=8):  81%|########  | 158720/196400 [01:10<00:12, 3096.08 examples/s]
hf-map en (num_proc=8):  81%|########1 | 159744/196400 [01:11<00:13, 2691.68 examples/s]
hf-map en (num_proc=8):  82%|########2 | 161792/196400 [01:11<00:09, 3474.05 examples/s]
hf-map en (num_proc=8):  83%|########2 | 162816/196400 [01:11<00:09, 3708.86 examples/s]
hf-map en (num_proc=8):  83%|########3 | 163840/196400 [01:12<00:07, 4306.74 examples/s]
hf-map en (num_proc=8):  84%|########3 | 164864/196400 [01:12<00:11, 2728.86 examples/s]
hf-map en (num_proc=8):  84%|########4 | 165888/196400 [01:13<00:09, 3120.21 examples/s]
hf-map en (num_proc=8):  85%|########4 | 166912/196400 [01:13<00:11, 2631.80 examples/s]
hf-map en (num_proc=8):  86%|########5 | 167936/196400 [01:13<00:09, 3147.10 examples/s]
hf-map en (num_proc=8):  86%|########6 | 168960/196400 [01:14<00:08, 3182.87 examples/s]
hf-map en (num_proc=8):  87%|########6 | 169984/196400 [01:14<00:07, 3691.70 examples/s]
hf-map en (num_proc=8):  87%|########7 | 171008/196400 [01:14<00:07, 3227.44 examples/s]
hf-map en (num_proc=8):  88%|########7 | 172032/196400 [01:14<00:06, 3920.18 examples/s]
hf-map en (num_proc=8):  88%|########8 | 173056/196400 [01:15<00:07, 3246.64 examples/s]
hf-map en (num_proc=8):  89%|########8 | 174080/196400 [01:15<00:07, 3180.97 examples/s]
hf-map en (num_proc=8):  89%|########9 | 175104/196400 [01:15<00:06, 3067.93 examples/s]
hf-map en (num_proc=8):  90%|########9 | 176128/196400 [01:16<00:07, 2698.93 examples/s]
hf-map en (num_proc=8):  90%|######### | 177152/196400 [01:16<00:06, 2923.65 examples/s]
hf-map en (num_proc=8):  91%|######### | 178176/196400 [01:16<00:05, 3447.93 examples/s]
hf-map en (num_proc=8):  91%|#########1| 179200/196400 [01:17<00:05, 3289.20 examples/s]
hf-map en (num_proc=8):  92%|#########2| 181248/196400 [01:18<00:05, 2847.28 examples/s]
hf-map en (num_proc=8):  93%|#########3| 183296/196400 [01:18<00:04, 3251.79 examples/s]
hf-map en (num_proc=8):  94%|#########3| 184320/196400 [01:19<00:04, 2573.89 examples/s]
hf-map en (num_proc=8):  95%|#########4| 186368/196400 [01:19<00:02, 3830.33 examples/s]
hf-map en (num_proc=8):  95%|#########5| 187392/196400 [01:19<00:02, 3409.61 examples/s]
hf-map en (num_proc=8):  96%|#########6| 189414/196400 [01:20<00:02, 2880.59 examples/s]
hf-map en (num_proc=8):  97%|#########6| 190412/196400 [01:20<00:01, 3168.23 examples/s]
hf-map en (num_proc=8):  97%|#########7| 191410/196400 [01:21<00:01, 2818.54 examples/s]
hf-map en (num_proc=8):  98%|#########7| 192408/196400 [01:21<00:01, 2895.92 examples/s]
hf-map en (num_proc=8):  99%|#########8| 194404/196400 [01:21<00:00, 4486.65 examples/s]
hf-map en (num_proc=8):  99%|#########9| 195402/196400 [01:22<00:00, 3688.30 examples/s]
hf-map en (num_proc=8): 100%|##########| 196400/196400 [01:22<00:00, 2380.94 examples/s]
    en: datasets.map num_proc=8 done 83s
    en: 25000 docs  19355384 tokens  85s
    en: 50000 docs  38502040 tokens  87s
    en: 75000 docs  57868565 tokens  89s
    en: 100000 docs  77221904 tokens  91s
    en: 125000 docs  96715846 tokens  93s
  packed en: 111592053 tokens (1.000 of budget)  rows=196029 docs=144428  pad-per-doc (HF map)
full mix budget 1115916236 (en=10%); packed 1645162 sequences, 984858 docs, 1115928627 tokens
eval every 100 steps; resume checkpoint-{step} every 500 steps, keep last 2; analysis model-{step} every 1000 steps (weights+tokenizer; upload SeragAmin/gemma_mapping_byteify_cpt_seq1024)
starting from step 0 (no resumable checkpoint-{step})
loading research/results/embedding_init/gemma_mapping_byteify
liger OLMo2 kernels (fused linear CE, RMSNorm, SwiGLU, RoPE)

Loading weights:   0%|          | 0/179 [00:00<?, ?it/s]
Loading weights:   1%|          | 1/179 [00:03<10:15,  3.46s/it]
Loading weights:  21%|##1       | 38/179 [00:03<00:10, 14.04it/s]
Loading weights:  26%|##6       | 47/179 [00:03<00:07, 16.96it/s]
Loading weights:  30%|###       | 54/179 [00:04<00:06, 19.81it/s]
Loading weights:  34%|###3      | 60/179 [00:04<00:06, 17.96it/s]
Loading weights:  38%|###7      | 68/179 [00:04<00:05, 20.68it/s]
Loading weights:  40%|####      | 72/179 [00:05<00:05, 19.21it/s]
Loading weights:  44%|####4     | 79/179 [00:05<00:05, 19.57it/s]
Loading weights:  46%|####5     | 82/179 [00:05<00:05, 17.66it/s]
Loading weights:  50%|#####     | 90/179 [00:05<00:04, 20.71it/s]
Loading weights:  52%|#####1    | 93/179 [00:06<00:04, 18.20it/s]
Loading weights:  56%|#####5    | 100/179 [00:06<00:03, 23.44it/s]
Loading weights:  58%|#####7    | 103/179 [00:06<00:03, 19.25it/s]
Loading weights:  61%|######    | 109/179 [00:06<00:02, 24.41it/s]
Loading weights:  63%|######3   | 113/179 [00:07<00:03, 18.09it/s]
Loading weights:  65%|######4   | 116/179 [00:07<00:03, 18.53it/s]
Loading weights:  69%|######8   | 123/179 [00:07<00:02, 26.02it/s]
Loading weights:  71%|#######   | 127/179 [00:07<00:02, 19.96it/s]
Loading weights:  75%|#######5  | 135/179 [00:08<00:02, 17.76it/s]
Loading weights:  77%|#######7  | 138/179 [00:08<00:02, 18.14it/s]
Loading weights:  81%|########1 | 145/179 [00:08<00:01, 25.18it/s]
Loading weights:  83%|########3 | 149/179 [00:08<00:01, 16.71it/s]
Loading weights:  88%|########7 | 157/179 [00:09<00:01, 19.52it/s]
Loading weights:  89%|########9 | 160/179 [00:09<00:01, 17.27it/s]
Loading weights:  94%|#########3| 168/179 [00:09<00:00, 20.38it/s]
Loading weights:  96%|#########5| 171/179 [00:09<00:00, 20.80it/s]
Loading weights: 100%|##########| 179/179 [00:10<00:00, 17.87it/s]
gradient_checkpointing=off  liger=on
seed=42  flash SDP  TF32
eval at step 0
Backend impl 'liger_kernel.ops.backends._cutedsl.rope' for op 'rope' could not be imported: ImportError: cutedsl backend requires the NVIDIA CUTLASS Python DSL (CuTe DSL). Install it with `pip install nvidia-cutlass-dsl`, or when installing Liger-Kernel use `pip install 'liger-kernel[cutedsl]'`.
Backend impl 'liger_kernel.ops.backends._ascend.rope' for op 'rope' could not be imported: ImportError: cannot import name 'tanh' from 'triton.language.math' (/usr/local/lib/python3.12/dist-packages/triton/language/math.py)
  am   nll=11.4879  ppl=97524.63  bpb=4.0397
  ta   nll=11.4029  ppl=89584.75  bpb=1.6562
  kk   nll=8.5479  ppl=5155.85  bpb=2.8784
  tl   nll=2.8077  ppl=16.57  bpb=1.3322
  en   nll=2.8266  ppl=16.89  bpb=0.8801
eval full_cpt step 0  01:27 (86.6s)
full_cpt: 1645162 sequences → 12852 steps  batch=16 accum=8  real_tokens=1115858680  liger fused CE, labels=-100 on pad, no attn mask
full_cpt step     1/12852 (  0.0%)  nll=9.0161  ppl=8234.82  grad=44.750  lr=1.56e-07  7489 tok/s  11.42s/it  [00:11<40:46:51]  mem=28.42GB
full_cpt step    10/12852 (  0.1%)  nll=8.7396  ppl=6245.61  grad=43.500  lr=1.56e-06  7554 tok/s  11.55s/it  [01:55<41:11:12]  mem=34.07GB
full_cpt step    20/12852 (  0.2%)  nll=9.1986  ppl=9883.72  grad=43.250  lr=3.13e-06  7500 tok/s  11.57s/it  [03:51<41:14:50]  mem=34.07GB
full_cpt step    30/12852 (  0.2%)  nll=7.9113  ppl=2727.83  grad=27.875  lr=4.69e-06  7431 tok/s  11.58s/it  [05:47<41:14:42]  mem=34.07GB
full_cpt step    40/12852 (  0.3%)  nll=7.9068  ppl=2715.78  grad=19.500  lr=6.25e-06  7487 tok/s  11.59s/it  [07:44<41:15:03]  mem=34.07GB
full_cpt step    50/12852 (  0.4%)  nll=6.6994  ppl=811.90  grad=9.750  lr=7.81e-06  7488 tok/s  11.60s/it  [09:40<41:14:13]  mem=34.07GB
full_cpt step    60/12852 (  0.5%)  nll=6.7456  ppl=850.32  grad=6.844  lr=9.38e-06  7481 tok/s  11.60s/it  [11:36<41:12:34]  mem=34.07GB
full_cpt step    70/12852 (  0.5%)  nll=5.8483  ppl=346.64  grad=3.312  lr=1.09e-05  7468 tok/s  11.60s/it  [13:32<41:10:42]  mem=34.07GB
full_cpt step    80/12852 (  0.6%)  nll=5.6926  ppl=296.68  grad=2.453  lr=1.25e-05  7461 tok/s  11.60s/it  [15:28<41:08:49]  mem=34.07GB
full_cpt step    90/12852 (  0.7%)  nll=5.4719  ppl=237.91  grad=2.078  lr=1.41e-05  7458 tok/s  11.60s/it  [17:24<41:07:00]  mem=34.07GB
full_cpt step   100/12852 (  0.8%)  nll=5.0722  ppl=159.52  grad=1.828  lr=1.56e-05  7460 tok/s  11.60s/it  [19:20<41:05:16]  mem=34.07GB
  am   nll=5.8568  ppl=349.59  bpb=2.0595
  ta   nll=5.9974  ppl=402.40  bpb=0.8711
  kk   nll=4.8469  ppl=127.34  bpb=1.6322
  tl   nll=2.8098  ppl=16.61  bpb=1.3332
  en   nll=2.8291  ppl=16.93  bpb=0.8809
eval full_cpt step 100  01:27 (86.5s)
full_cpt step   110/12852 (  0.9%)  nll=4.9608  ppl=142.71  grad=1.773  lr=1.72e-05  6998 tok/s  12.39s/it  [22:43<43:50:44]  mem=34.07GB
full_cpt step   120/12852 (  0.9%)  nll=4.8165  ppl=123.54  grad=1.359  lr=1.88e-05  7029 tok/s  12.32s/it  [24:39<43:34:49]  mem=34.07GB
full_cpt step   130/12852 (  1.0%)  nll=4.7707  ppl=118.01  grad=1.320  lr=2.00e-05  7062 tok/s  12.27s/it  [26:35<43:21:10]  mem=34.07GB
full_cpt step   140/12852 (  1.1%)  nll=4.6013  ppl=99.61  grad=1.422  lr=2.00e-05  7101 tok/s  12.22s/it  [28:31<43:09:25]  mem=34.07GB
full_cpt step   150/12852 (  1.2%)  nll=4.3607  ppl=78.31  grad=1.312  lr=2.00e-05  7132 tok/s  12.18s/it  [30:27<42:58:54]  mem=34.07GB
full_cpt step   160/12852 (  1.2%)  nll=4.1826  ppl=65.53  grad=0.961  lr=2.00e-05  7156 tok/s  12.15s/it  [32:23<42:49:24]  mem=34.07GB
full_cpt step   170/12852 (  1.3%)  nll=4.4758  ppl=87.87  grad=1.016  lr=2.00e-05  7160 tok/s  12.11s/it  [34:19<42:40:31]  mem=34.07GB
full_cpt step   180/12852 (  1.4%)  nll=4.4099  ppl=82.26  grad=1.117  lr=2.00e-05  7172 tok/s  12.09s/it  [36:15<42:32:29]  mem=34.07GB
full_cpt step   190/12852 (  1.5%)  nll=4.1856  ppl=65.73  grad=1.062  lr=2.00e-05  7182 tok/s  12.06s/it  [38:11<42:25:04]  mem=34.07GB
full_cpt step   200/12852 (  1.6%)  nll=4.2869  ppl=72.74  grad=1.062  lr=2.00e-05  7190 tok/s  12.04s/it  [40:07<42:18:15]  mem=34.07GB
  am   nll=4.7848  ppl=119.68  bpb=1.6826
  ta   nll=5.0247  ppl=152.13  bpb=0.7298
  kk   nll=3.7758  ppl=43.63  bpb=1.2715
  tl   nll=2.7740  ppl=16.02  bpb=1.3162
  en   nll=2.8229  ppl=16.82  bpb=0.8789
eval full_cpt step 200  01:27 (86.6s)
full_cpt step   210/12852 (  1.6%)  nll=4.1040  ppl=60.58  grad=0.969  lr=2.00e-05  6963 tok/s  12.43s/it  [43:30<43:38:47]  mem=34.07GB
full_cpt step   220/12852 (  1.7%)  nll=4.2026  ppl=66.86  grad=1.156  lr=2.00e-05  6982 tok/s  12.39s/it  [45:26<43:28:49]  mem=34.07GB
full_cpt step   230/12852 (  1.8%)  nll=4.1434  ppl=63.02  grad=1.172  lr=2.00e-05  7002 tok/s  12.36s/it  [47:22<43:19:33]  mem=34.07GB
full_cpt step   240/12852 (  1.9%)  nll=4.1126  ppl=61.11  grad=0.996  lr=2.00e-05  7028 tok/s  12.33s/it  [49:18<43:10:57]  mem=34.07GB
full_cpt step   250/12852 (  1.9%)  nll=4.0005  ppl=54.62  grad=0.953  lr=2.00e-05  7042 tok/s  12.30s/it  [51:14<43:02:48]  mem=34.07GB
full_cpt step   260/12852 (  2.0%)  nll=4.1169  ppl=61.37  grad=0.961  lr=2.00e-05  7061 tok/s  12.27s/it  [53:11<42:55:23]  mem=34.07GB
full_cpt step   270/12852 (  2.1%)  nll=4.0314  ppl=56.34  grad=0.977  lr=2.00e-05  7074 tok/s  12.25s/it  [55:07<42:48:14]  mem=34.07GB
full_cpt step   280/12852 (  2.2%)  nll=4.0677  ppl=58.43  grad=0.953  lr=2.00e-05  7088 tok/s  12.22s/it  [57:03<42:41:28]  mem=34.07GB
full_cpt step   290/12852 (  2.3%)  nll=4.0905  ppl=59.77  grad=0.988  lr=2.00e-05  7097 tok/s  12.20s/it  [58:59<42:35:00]  mem=34.07GB
full_cpt step   300/12852 (  2.3%)  nll=3.9160  ppl=50.20  grad=0.926  lr=2.00e-05  7104 tok/s  12.18s/it  [1:00:55<42:28:45]  mem=34.07GB
  am   nll=4.6131  ppl=100.80  bpb=1.6222
  ta   nll=4.7709  ppl=118.02  bpb=0.6929
  kk   nll=3.4599  ppl=31.81  bpb=1.1651
  tl   nll=2.7481  ppl=15.61  bpb=1.3039
  en   nll=2.8202  ppl=16.78  bpb=0.8781
eval full_cpt step 300  01:28 (88.3s)
full_cpt step   310/12852 (  2.4%)  nll=3.9002  ppl=49.41  grad=1.086  lr=2.00e-05  6953 tok/s  12.45s/it  [1:04:19<43:22:26]  mem=34.07GB
full_cpt step   320/12852 (  2.5%)  nll=3.9939  ppl=54.27  grad=1.133  lr=2.00e-05  6969 tok/s  12.42s/it  [1:06:16<43:14:58]  mem=34.07GB
full_cpt step   330/12852 (  2.6%)  nll=3.9894  ppl=54.02  grad=1.102  lr=2.00e-05  6987 tok/s  12.40s/it  [1:08:12<43:07:51]  mem=34.07GB
full_cpt step   340/12852 (  2.6%)  nll=3.8941  ppl=49.11  grad=0.871  lr=2.00e-05  7002 tok/s  12.38s/it  [1:10:08<43:01:02]  mem=34.07GB
full_cpt step   350/12852 (  2.7%)  nll=3.8811  ppl=48.48  grad=1.078  lr=2.00e-05  7011 tok/s  12.36s/it  [1:12:04<42:54:23]  mem=34.07GB
full_cpt step   360/12852 (  2.8%)  nll=3.8757  ppl=48.22  grad=0.902  lr=2.00e-05  7029 tok/s  12.34s/it  [1:14:01<42:48:10]  mem=34.07GB
full_cpt step   370/12852 (  2.9%)  nll=3.8915  ppl=48.98  grad=1.086  lr=2.00e-05  7043 tok/s  12.32s/it  [1:15:57<42:42:09]  mem=34.07GB
full_cpt step   380/12852 (  3.0%)  nll=3.8985  ppl=49.33  grad=1.039  lr=2.00e-05  7055 tok/s  12.30s/it  [1:17:53<42:36:17]  mem=34.07GB
full_cpt step   390/12852 (  3.0%)  nll=3.9137  ppl=50.08  grad=0.973  lr=2.00e-05  7060 tok/s  12.28s/it  [1:19:49<42:30:32]  mem=34.07GB
full_cpt step   400/12852 (  3.1%)  nll=3.8371  ppl=46.39  grad=0.957  lr=2.00e-05  7066 tok/s  12.26s/it  [1:21:45<42:24:59]  mem=34.07GB
  am   nll=4.5228  ppl=92.10  bpb=1.5905
  ta   nll=4.6198  ppl=101.47  bpb=0.6710
  kk   nll=3.3079  ppl=27.33  bpb=1.1139
  tl   nll=2.7356  ppl=15.42  bpb=1.2980
  en   nll=2.8197  ppl=16.77  bpb=0.8779
eval full_cpt step 400  01:27 (87.2s)
full_cpt step   410/12852 (  3.2%)  nll=3.8910  ppl=48.96  grad=0.988  lr=2.00e-05  6951 tok/s  12.46s/it  [1:25:08<43:03:39]  mem=34.07GB
full_cpt step   420/12852 (  3.3%)  nll=3.6862  ppl=39.89  grad=0.965  lr=2.00e-05  6960 tok/s  12.44s/it  [1:27:04<42:57:23]  mem=34.07GB
full_cpt step   430/12852 (  3.3%)  nll=3.8859  ppl=48.71  grad=0.906  lr=2.00e-05  6972 tok/s  12.42s/it  [1:29:01<42:51:20]  mem=34.07GB
full_cpt step   440/12852 (  3.4%)  nll=3.8437  ppl=46.70  grad=0.961  lr=2.00e-05  6977 tok/s  12.40s/it  [1:30:56<42:45:23]  mem=34.07GB
full_cpt step   450/12852 (  3.5%)  nll=3.8862  ppl=48.72  grad=0.902  lr=2.00e-05  6988 tok/s  12.38s/it  [1:32:53<42:39:42]  mem=34.07GB
full_cpt step   460/12852 (  3.6%)  nll=3.7112  ppl=40.90  grad=1.008  lr=2.00e-05  6998 tok/s  12.37s/it  [1:34:49<42:34:09]  mem=34.07GB
full_cpt step   470/12852 (  3.7%)  nll=3.8068  ppl=45.01  grad=1.023  lr=2.00e-05  7010 tok/s  12.35s/it  [1:36:45<42:28:49]  mem=34.07GB
full_cpt step   480/12852 (  3.7%)  nll=3.6855  ppl=39.86  grad=0.891  lr=2.00e-05  7023 tok/s  12.34s/it  [1:38:41<42:23:40]  mem=34.07GB
full_cpt step   490/12852 (  3.8%)  nll=3.8315  ppl=46.13  grad=1.039  lr=2.00e-05  7029 tok/s  12.32s/it  [1:40:37<42:18:32]  mem=34.07GB
full_cpt step   500/12852 (  3.9%)  nll=3.6173  ppl=37.24  grad=0.988  lr=2.00e-05  7036 tok/s  12.31s/it  [1:42:33<42:13:32]  mem=34.07GB
  am   nll=4.4628  ppl=86.73  bpb=1.5693
  ta   nll=4.5147  ppl=91.35  bpb=0.6557
  kk   nll=3.2084  ppl=24.74  bpb=1.0804
  tl   nll=2.7281  ppl=15.30  bpb=1.2944
  en   nll=2.8188  ppl=16.76  bpb=0.8777
eval full_cpt step 500  01:29 (88.6s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:03<00:00,  3.02s/it]
Writing model shards: 100%|##########| 1/1 [00:03<00:00,  3.02s/it]
kept last 1 resume checkpoints: checkpoint-500
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-500 at step 500 next_row=64000
full_cpt step   510/12852 (  4.0%)  nll=3.7847  ppl=44.02  grad=0.906  lr=2.00e-05  6900 tok/s  12.55s/it  [1:46:40<43:01:16]  mem=34.07GB
full_cpt step   520/12852 (  4.0%)  nll=3.6834  ppl=39.78  grad=0.926  lr=2.00e-05  6913 tok/s  12.53s/it  [1:48:36<42:55:33]  mem=34.07GB
full_cpt step   530/12852 (  4.1%)  nll=3.6554  ppl=38.68  grad=0.895  lr=2.00e-05  6922 tok/s  12.51s/it  [1:50:32<42:49:57]  mem=34.07GB
full_cpt step   540/12852 (  4.2%)  nll=3.6495  ppl=38.46  grad=0.914  lr=2.00e-05  6933 tok/s  12.50s/it  [1:52:29<42:44:31]  mem=34.07GB
full_cpt step   550/12852 (  4.3%)  nll=3.7495  ppl=42.50  grad=0.984  lr=2.00e-05  6942 tok/s  12.48s/it  [1:54:25<42:39:10]  mem=34.07GB
full_cpt step   560/12852 (  4.4%)  nll=3.6277  ppl=37.63  grad=0.910  lr=1.99e-05  6948 tok/s  12.47s/it  [1:56:21<42:33:55]  mem=34.07GB
full_cpt step   570/12852 (  4.4%)  nll=3.6966  ppl=40.31  grad=1.031  lr=1.99e-05  6954 tok/s  12.45s/it  [1:58:17<42:28:45]  mem=34.07GB
full_cpt step   580/12852 (  4.5%)  nll=3.7028  ppl=40.56  grad=1.062  lr=1.99e-05  6957 tok/s  12.44s/it  [2:00:13<42:23:41]  mem=34.07GB
full_cpt step   590/12852 (  4.6%)  nll=3.7709  ppl=43.42  grad=0.883  lr=1.99e-05  6965 tok/s  12.42s/it  [2:02:09<42:18:45]  mem=34.07GB
full_cpt step   600/12852 (  4.7%)  nll=3.7626  ppl=43.06  grad=0.918  lr=1.99e-05  6974 tok/s  12.41s/it  [2:04:06<42:13:59]  mem=34.07GB
  am   nll=4.4064  ppl=81.97  bpb=1.5495
  ta   nll=4.4384  ppl=84.64  bpb=0.6446
  kk   nll=3.1384  ppl=23.07  bpb=1.0568
  tl   nll=2.7212  ppl=15.20  bpb=1.2912
  en   nll=2.8186  ppl=16.75  bpb=0.8776
eval full_cpt step 600  01:28 (87.9s)
full_cpt step   610/12852 (  4.7%)  nll=3.7003  ppl=40.46  grad=0.883  lr=1.99e-05  6904 tok/s  12.54s/it  [2:07:30<42:38:42]  mem=34.07GB
full_cpt step   620/12852 (  4.8%)  nll=3.5746  ppl=35.68  grad=1.008  lr=1.99e-05  6913 tok/s  12.53s/it  [2:09:26<42:33:37]  mem=34.07GB
full_cpt step   630/12852 (  4.9%)  nll=3.5998  ppl=36.59  grad=1.086  lr=1.99e-05  6922 tok/s  12.51s/it  [2:11:22<42:28:38]  mem=34.07GB
full_cpt step   640/12852 (  5.0%)  nll=3.7162  ppl=41.11  grad=0.992  lr=1.99e-05  6930 tok/s  12.50s/it  [2:13:19<42:23:43]  mem=34.07GB
full_cpt step   650/12852 (  5.1%)  nll=3.4840  ppl=32.59  grad=1.039  lr=1.99e-05  6937 tok/s  12.48s/it  [2:15:15<42:18:52]  mem=34.07GB
full_cpt step   660/12852 (  5.1%)  nll=3.7686  ppl=43.32  grad=0.902  lr=1.99e-05  6945 tok/s  12.47s/it  [2:17:11<42:14:08]  mem=34.07GB
full_cpt step   670/12852 (  5.2%)  nll=3.6015  ppl=36.65  grad=1.047  lr=1.99e-05  6951 tok/s  12.46s/it  [2:19:07<42:09:28]  mem=34.07GB
full_cpt step   680/12852 (  5.3%)  nll=3.5772  ppl=35.77  grad=0.969  lr=1.99e-05  6956 tok/s  12.45s/it  [2:21:03<42:04:53]  mem=34.07GB
full_cpt step   690/12852 (  5.4%)  nll=3.5768  ppl=35.76  grad=0.969  lr=1.99e-05  6961 tok/s  12.43s/it  [2:22:59<42:00:23]  mem=34.07GB
full_cpt step   700/12852 (  5.4%)  nll=3.6700  ppl=39.25  grad=0.984  lr=1.99e-05  6967 tok/s  12.42s/it  [2:24:56<41:55:57]  mem=34.07GB
  am   nll=4.3600  ppl=78.26  bpb=1.5332
  ta   nll=4.3733  ppl=79.31  bpb=0.6352
  kk   nll=3.0876  ppl=21.92  bpb=1.0397
  tl   nll=2.7138  ppl=15.09  bpb=1.2877
  en   nll=2.8185  ppl=16.75  bpb=0.8776
eval full_cpt step 700  01:31 (90.9s)
full_cpt step   710/12852 (  5.5%)  nll=3.5730  ppl=35.62  grad=0.957  lr=1.99e-05  6903 tok/s  12.54s/it  [2:28:23<42:17:28]  mem=34.07GB
full_cpt step   720/12852 (  5.6%)  nll=3.6783  ppl=39.58  grad=1.031  lr=1.99e-05  6910 tok/s  12.53s/it  [2:30:19<42:12:48]  mem=34.07GB
full_cpt step   730/12852 (  5.7%)  nll=3.6362  ppl=37.95  grad=0.887  lr=1.99e-05  6917 tok/s  12.51s/it  [2:32:15<42:08:14]  mem=34.07GB
full_cpt step   740/12852 (  5.8%)  nll=3.6386  ppl=38.04  grad=0.871  lr=1.99e-05  6925 tok/s  12.50s/it  [2:34:11<42:03:43]  mem=34.07GB
full_cpt step   750/12852 (  5.8%)  nll=3.6336  ppl=37.85  grad=0.852  lr=1.99e-05  6930 tok/s  12.49s/it  [2:36:08<41:59:15]  mem=34.07GB
full_cpt step   760/12852 (  5.9%)  nll=3.6625  ppl=38.96  grad=0.887  lr=1.99e-05  6938 tok/s  12.48s/it  [2:38:04<41:54:54]  mem=34.07GB
full_cpt step   770/12852 (  6.0%)  nll=3.6674  ppl=39.15  grad=0.871  lr=1.99e-05  6944 tok/s  12.47s/it  [2:40:00<41:50:35]  mem=34.07GB
full_cpt step   780/12852 (  6.1%)  nll=3.5545  ppl=34.97  grad=0.953  lr=1.99e-05  6949 tok/s  12.46s/it  [2:41:56<41:46:20]  mem=34.07GB
full_cpt step   790/12852 (  6.1%)  nll=3.6614  ppl=38.92  grad=0.992  lr=1.99e-05  6954 tok/s  12.45s/it  [2:43:53<41:42:08]  mem=34.07GB
full_cpt step   800/12852 (  6.2%)  nll=3.5055  ppl=33.30  grad=0.938  lr=1.99e-05  6961 tok/s  12.44s/it  [2:45:49<41:38:01]  mem=34.07GB
  am   nll=4.3257  ppl=75.62  bpb=1.5211
  ta   nll=4.3230  ppl=75.42  bpb=0.6279
  kk   nll=3.0481  ppl=21.08  bpb=1.0264
  tl   nll=2.7088  ppl=15.01  bpb=1.2853
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 800  01:29 (89.5s)
full_cpt step   810/12852 (  6.3%)  nll=3.5105  ppl=33.46  grad=0.898  lr=1.99e-05  6906 tok/s  12.54s/it  [2:49:15<41:56:05]  mem=34.07GB
full_cpt step   820/12852 (  6.4%)  nll=3.4255  ppl=30.74  grad=0.840  lr=1.99e-05  6912 tok/s  12.53s/it  [2:51:11<41:51:46]  mem=34.07GB
full_cpt step   830/12852 (  6.5%)  nll=3.5034  ppl=33.23  grad=0.898  lr=1.99e-05  6919 tok/s  12.51s/it  [2:53:07<41:47:31]  mem=34.07GB
full_cpt step   840/12852 (  6.5%)  nll=3.5037  ppl=33.24  grad=0.945  lr=1.99e-05  6925 tok/s  12.50s/it  [2:55:03<41:43:19]  mem=34.07GB
full_cpt step   850/12852 (  6.6%)  nll=3.6228  ppl=37.44  grad=0.898  lr=1.99e-05  6931 tok/s  12.49s/it  [2:57:00<41:39:09]  mem=34.07GB
full_cpt step   860/12852 (  6.7%)  nll=3.7638  ppl=43.11  grad=0.984  lr=1.99e-05  6938 tok/s  12.48s/it  [2:58:56<41:35:05]  mem=34.07GB
full_cpt step   870/12852 (  6.8%)  nll=3.6368  ppl=37.97  grad=0.883  lr=1.98e-05  6943 tok/s  12.47s/it  [3:00:52<41:31:01]  mem=34.07GB
full_cpt step   880/12852 (  6.8%)  nll=3.6277  ppl=37.63  grad=0.879  lr=1.98e-05  6947 tok/s  12.46s/it  [3:02:48<41:27:00]  mem=34.07GB
full_cpt step   890/12852 (  6.9%)  nll=3.7619  ppl=43.03  grad=0.883  lr=1.98e-05  6952 tok/s  12.45s/it  [3:04:45<41:23:02]  mem=34.07GB
full_cpt step   900/12852 (  7.0%)  nll=3.6840  ppl=39.80  grad=1.016  lr=1.98e-05  6957 tok/s  12.45s/it  [3:06:41<41:19:08]  mem=34.07GB
  am   nll=4.2913  ppl=73.06  bpb=1.5090
  ta   nll=4.2802  ppl=72.25  bpb=0.6217
  kk   nll=3.0145  ppl=20.38  bpb=1.0151
  tl   nll=2.7040  ppl=14.94  bpb=1.2830
  en   nll=2.8180  ppl=16.74  bpb=0.8774
eval full_cpt step 900  01:30 (89.7s)
full_cpt step   910/12852 (  7.1%)  nll=3.5601  ppl=35.17  grad=0.980  lr=1.98e-05  6907 tok/s  12.53s/it  [3:10:07<41:34:52]  mem=34.07GB
full_cpt step   920/12852 (  7.2%)  nll=3.6371  ppl=37.98  grad=0.891  lr=1.98e-05  6910 tok/s  12.52s/it  [3:12:03<41:30:47]  mem=34.07GB
full_cpt step   930/12852 (  7.2%)  nll=3.5700  ppl=35.52  grad=0.867  lr=1.98e-05  6915 tok/s  12.51s/it  [3:13:59<41:26:43]  mem=34.07GB
full_cpt step   940/12852 (  7.3%)  nll=3.6872  ppl=39.93  grad=0.922  lr=1.98e-05  6921 tok/s  12.51s/it  [3:15:55<41:22:44]  mem=34.07GB
full_cpt step   950/12852 (  7.4%)  nll=3.4510  ppl=31.53  grad=0.836  lr=1.98e-05  6927 tok/s  12.50s/it  [3:17:51<41:18:47]  mem=34.07GB
full_cpt step   960/12852 (  7.5%)  nll=3.5752  ppl=35.70  grad=0.973  lr=1.98e-05  6934 tok/s  12.49s/it  [3:19:47<41:14:52]  mem=34.07GB
full_cpt step   970/12852 (  7.5%)  nll=3.5482  ppl=34.75  grad=0.996  lr=1.98e-05  6938 tok/s  12.48s/it  [3:21:43<41:10:59]  mem=34.07GB
full_cpt step   980/12852 (  7.6%)  nll=3.5146  ppl=33.60  grad=0.957  lr=1.98e-05  6943 tok/s  12.47s/it  [3:23:39<41:07:08]  mem=34.07GB
full_cpt step   990/12852 (  7.7%)  nll=3.3437  ppl=28.32  grad=0.914  lr=1.98e-05  6948 tok/s  12.46s/it  [3:25:36<41:03:22]  mem=34.07GB
full_cpt step  1000/12852 (  7.8%)  nll=3.6479  ppl=38.39  grad=0.871  lr=1.98e-05  6952 tok/s  12.45s/it  [3:27:32<40:59:36]  mem=34.07GB
  am   nll=4.2638  ppl=71.08  bpb=1.4994
  ta   nll=4.2455  ppl=69.79  bpb=0.6166
  kk   nll=2.9858  ppl=19.80  bpb=1.0055
  tl   nll=2.7004  ppl=14.89  bpb=1.2813
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 1000  01:28 (88.3s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.35s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.35s/it]
kept last 2 resume checkpoints: checkpoint-500 checkpoint-1000
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-1000 at step 1000 next_row=128000

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:03<00:00,  3.86s/it]
Writing model shards: 100%|##########| 1/1 [00:03<00:00,  3.88s/it]
wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-1000 (weights+expanded tokenizer, no optimizer)
full_cpt step  1010/12852 (  7.9%)  nll=3.5644  ppl=35.32  grad=0.879  lr=1.98e-05  6845 tok/s  12.65s/it  [3:32:53<41:36:05]  mem=34.07GB
full_cpt step  1020/12852 (  7.9%)  nll=3.5107  ppl=33.47  grad=0.922  lr=1.98e-05  6850 tok/s  12.64s/it  [3:34:50<41:32:00]  mem=34.07GB
full_cpt step  1030/12852 (  8.0%)  nll=3.4797  ppl=32.45  grad=0.840  lr=1.98e-05  6856 tok/s  12.63s/it  [3:36:46<41:27:58]  mem=34.07GB
full_cpt step  1040/12852 (  8.1%)  nll=3.5795  ppl=35.86  grad=0.891  lr=1.98e-05  6864 tok/s  12.62s/it  [3:38:42<41:24:00]  mem=34.07GB
full_cpt step  1050/12852 (  8.2%)  nll=3.5643  ppl=35.31  grad=0.914  lr=1.98e-05  6870 tok/s  12.61s/it  [3:40:39<41:20:03]  mem=34.07GB
full_cpt step  1060/12852 (  8.2%)  nll=3.5734  ppl=35.64  grad=0.961  lr=1.98e-05  6875 tok/s  12.60s/it  [3:42:35<41:16:06]  mem=34.07GB
full_cpt step  1070/12852 (  8.3%)  nll=3.5230  ppl=33.89  grad=0.887  lr=1.98e-05  6880 tok/s  12.59s/it  [3:44:31<41:12:12]  mem=34.07GB
full_cpt step  1080/12852 (  8.4%)  nll=3.4551  ppl=31.66  grad=0.945  lr=1.98e-05  6885 tok/s  12.58s/it  [3:46:27<41:08:19]  mem=34.07GB
full_cpt step  1090/12852 (  8.5%)  nll=3.4399  ppl=31.18  grad=0.961  lr=1.97e-05  6890 tok/s  12.57s/it  [3:48:23<41:04:28]  mem=34.07GB
full_cpt step  1100/12852 (  8.6%)  nll=3.4967  ppl=33.01  grad=0.895  lr=1.97e-05  6895 tok/s  12.56s/it  [3:50:19<41:00:40]  mem=34.07GB
  am   nll=4.2356  ppl=69.11  bpb=1.4895
  ta   nll=4.2159  ppl=67.76  bpb=0.6123
  kk   nll=2.9628  ppl=19.35  bpb=0.9977
  tl   nll=2.6950  ppl=14.81  bpb=1.2787
  en   nll=2.8180  ppl=16.74  bpb=0.8774
eval full_cpt step 1100  01:28 (88.3s)
full_cpt step  1110/12852 (  8.6%)  nll=3.5617  ppl=35.22  grad=0.918  lr=1.97e-05  6856 tok/s  12.63s/it  [3:53:44<41:12:26]  mem=34.07GB
full_cpt step  1120/12852 (  8.7%)  nll=3.4824  ppl=32.54  grad=0.879  lr=1.97e-05  6861 tok/s  12.62s/it  [3:55:40<41:08:32]  mem=34.07GB
full_cpt step  1130/12852 (  8.8%)  nll=3.4419  ppl=31.25  grad=1.047  lr=1.97e-05  6864 tok/s  12.62s/it  [3:57:36<41:04:39]  mem=34.07GB
full_cpt step  1140/12852 (  8.9%)  nll=3.5142  ppl=33.59  grad=0.879  lr=1.97e-05  6869 tok/s  12.61s/it  [3:59:32<41:00:49]  mem=34.07GB
full_cpt step  1150/12852 (  8.9%)  nll=3.5591  ppl=35.13  grad=0.898  lr=1.97e-05  6874 tok/s  12.60s/it  [4:01:28<40:57:03]  mem=34.07GB
full_cpt step  1160/12852 (  9.0%)  nll=3.5064  ppl=33.33  grad=0.859  lr=1.97e-05  6880 tok/s  12.59s/it  [4:03:24<40:53:18]  mem=34.07GB
full_cpt step  1170/12852 (  9.1%)  nll=3.6826  ppl=39.75  grad=0.887  lr=1.97e-05  6884 tok/s  12.58s/it  [4:05:20<40:49:35]  mem=34.07GB
full_cpt step  1180/12852 (  9.2%)  nll=3.4707  ppl=32.16  grad=0.871  lr=1.97e-05  6890 tok/s  12.57s/it  [4:07:16<40:45:54]  mem=34.07GB
full_cpt step  1190/12852 (  9.3%)  nll=3.5170  ppl=33.68  grad=0.852  lr=1.97e-05  6894 tok/s  12.56s/it  [4:09:12<40:42:13]  mem=34.07GB
full_cpt step  1200/12852 (  9.3%)  nll=3.5983  ppl=36.54  grad=0.879  lr=1.97e-05  6900 tok/s  12.56s/it  [4:11:09<40:38:37]  mem=34.07GB
  am   nll=4.2136  ppl=67.60  bpb=1.4817
  ta   nll=4.1897  ppl=66.00  bpb=0.6085
  kk   nll=2.9418  ppl=18.95  bpb=0.9906
  tl   nll=2.6909  ppl=14.75  bpb=1.2768
  en   nll=2.8173  ppl=16.73  bpb=0.8772
eval full_cpt step 1200  01:29 (88.9s)
full_cpt step  1210/12852 (  9.4%)  nll=3.4936  ppl=32.91  grad=0.863  lr=1.97e-05  6865 tok/s  12.62s/it  [4:14:34<40:49:15]  mem=34.07GB
full_cpt step  1220/12852 (  9.5%)  nll=3.5528  ppl=34.91  grad=0.953  lr=1.97e-05  6872 tok/s  12.61s/it  [4:16:30<40:45:34]  mem=34.07GB
full_cpt step  1230/12852 (  9.6%)  nll=3.5026  ppl=33.20  grad=0.875  lr=1.97e-05  6876 tok/s  12.61s/it  [4:18:26<40:41:52]  mem=34.07GB
full_cpt step  1240/12852 (  9.6%)  nll=3.5098  ppl=33.44  grad=0.883  lr=1.97e-05  6882 tok/s  12.60s/it  [4:20:22<40:38:14]  mem=34.07GB
full_cpt step  1250/12852 (  9.7%)  nll=3.5573  ppl=35.07  grad=0.906  lr=1.97e-05  6887 tok/s  12.59s/it  [4:22:18<40:34:38]  mem=34.07GB
full_cpt step  1260/12852 (  9.8%)  nll=3.3521  ppl=28.56  grad=0.887  lr=1.97e-05  6891 tok/s  12.58s/it  [4:24:14<40:31:01]  mem=34.07GB
full_cpt step  1270/12852 (  9.9%)  nll=3.3965  ppl=29.86  grad=0.902  lr=1.96e-05  6895 tok/s  12.58s/it  [4:26:11<40:27:26]  mem=34.07GB
full_cpt step  1280/12852 ( 10.0%)  nll=3.4612  ppl=31.86  grad=0.898  lr=1.96e-05  6900 tok/s  12.57s/it  [4:28:07<40:23:54]  mem=34.07GB
full_cpt step  1290/12852 ( 10.0%)  nll=3.4444  ppl=31.32  grad=0.848  lr=1.96e-05  6904 tok/s  12.56s/it  [4:30:03<40:20:22]  mem=34.07GB
full_cpt step  1300/12852 ( 10.1%)  nll=3.5590  ppl=35.13  grad=0.969  lr=1.96e-05  6907 tok/s  12.55s/it  [4:31:59<40:16:51]  mem=34.07GB
  am   nll=4.1931  ppl=66.23  bpb=1.4745
  ta   nll=4.1677  ppl=64.57  bpb=0.6053
  kk   nll=2.9234  ppl=18.60  bpb=0.9844
  tl   nll=2.6876  ppl=14.70  bpb=1.2752
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 1300  01:30 (90.3s)
full_cpt step  1310/12852 ( 10.2%)  nll=3.4727  ppl=32.22  grad=0.891  lr=1.96e-05  6872 tok/s  12.61s/it  [4:35:25<40:26:37]  mem=34.07GB
full_cpt step  1320/12852 ( 10.3%)  nll=3.5907  ppl=36.26  grad=0.926  lr=1.96e-05  6876 tok/s  12.61s/it  [4:37:21<40:23:03]  mem=34.07GB
full_cpt step  1330/12852 ( 10.3%)  nll=3.4014  ppl=30.01  grad=0.895  lr=1.96e-05  6878 tok/s  12.60s/it  [4:39:17<40:19:29]  mem=34.07GB
full_cpt step  1340/12852 ( 10.4%)  nll=3.5272  ppl=34.03  grad=0.895  lr=1.96e-05  6883 tok/s  12.59s/it  [4:41:13<40:15:59]  mem=34.07GB
full_cpt step  1350/12852 ( 10.5%)  nll=3.5428  ppl=34.56  grad=0.914  lr=1.96e-05  6886 tok/s  12.58s/it  [4:43:09<40:12:29]  mem=34.07GB
full_cpt step  1360/12852 ( 10.6%)  nll=3.5278  ppl=34.05  grad=0.930  lr=1.96e-05  6889 tok/s  12.58s/it  [4:45:05<40:09:01]  mem=34.07GB
full_cpt step  1370/12852 ( 10.7%)  nll=3.4274  ppl=30.80  grad=0.871  lr=1.96e-05  6894 tok/s  12.57s/it  [4:47:02<40:05:35]  mem=34.07GB
full_cpt step  1380/12852 ( 10.7%)  nll=3.4210  ppl=30.60  grad=0.910  lr=1.96e-05  6898 tok/s  12.56s/it  [4:48:58<40:02:10]  mem=34.07GB
full_cpt step  1390/12852 ( 10.8%)  nll=3.3496  ppl=28.49  grad=0.898  lr=1.96e-05  6902 tok/s  12.56s/it  [4:50:54<39:58:46]  mem=34.07GB
full_cpt step  1400/12852 ( 10.9%)  nll=3.5792  ppl=35.84  grad=0.902  lr=1.96e-05  6906 tok/s  12.55s/it  [4:52:50<39:55:23]  mem=34.07GB
  am   nll=4.1741  ppl=64.98  bpb=1.4678
  ta   nll=4.1479  ppl=63.30  bpb=0.6024
  kk   nll=2.9068  ppl=18.30  bpb=0.9788
  tl   nll=2.6858  ppl=14.67  bpb=1.2744
  en   nll=2.8172  ppl=16.73  bpb=0.8772
eval full_cpt step 1400  01:28 (87.8s)
full_cpt step  1410/12852 ( 11.0%)  nll=3.3860  ppl=29.55  grad=0.844  lr=1.96e-05  6876 tok/s  12.61s/it  [4:56:14<40:03:54]  mem=34.07GB
full_cpt step  1420/12852 ( 11.0%)  nll=3.4304  ppl=30.89  grad=0.812  lr=1.95e-05  6880 tok/s  12.60s/it  [4:58:10<40:00:28]  mem=34.07GB
full_cpt step  1430/12852 ( 11.1%)  nll=3.4062  ppl=30.15  grad=0.984  lr=1.95e-05  6884 tok/s  12.59s/it  [5:00:06<39:57:03]  mem=34.07GB
full_cpt step  1440/12852 ( 11.2%)  nll=3.4192  ppl=30.55  grad=0.863  lr=1.95e-05  6888 tok/s  12.58s/it  [5:02:02<39:53:38]  mem=34.07GB
full_cpt step  1450/12852 ( 11.3%)  nll=3.4368  ppl=31.09  grad=0.926  lr=1.95e-05  6891 tok/s  12.58s/it  [5:03:58<39:50:15]  mem=34.07GB
full_cpt step  1460/12852 ( 11.4%)  nll=3.3653  ppl=28.94  grad=0.953  lr=1.95e-05  6895 tok/s  12.57s/it  [5:05:54<39:46:54]  mem=34.07GB
full_cpt step  1470/12852 ( 11.4%)  nll=3.4071  ppl=30.18  grad=0.914  lr=1.95e-05  6898 tok/s  12.56s/it  [5:07:50<39:43:33]  mem=34.07GB
full_cpt step  1480/12852 ( 11.5%)  nll=3.3605  ppl=28.80  grad=0.953  lr=1.95e-05  6901 tok/s  12.56s/it  [5:09:46<39:40:14]  mem=34.07GB
full_cpt step  1490/12852 ( 11.6%)  nll=3.3500  ppl=28.50  grad=0.848  lr=1.95e-05  6904 tok/s  12.55s/it  [5:11:43<39:36:57]  mem=34.07GB
full_cpt step  1500/12852 ( 11.7%)  nll=3.4439  ppl=31.31  grad=0.863  lr=1.95e-05  6908 tok/s  12.55s/it  [5:13:39<39:33:41]  mem=34.07GB
  am   nll=4.1580  ppl=63.94  bpb=1.4622
  ta   nll=4.1290  ppl=62.12  bpb=0.5997
  kk   nll=2.8925  ppl=18.04  bpb=0.9740
  tl   nll=2.6840  ppl=14.64  bpb=1.2735
  en   nll=2.8175  ppl=16.74  bpb=0.8773
eval full_cpt step 1500  01:28 (88.2s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.43s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.43s/it]
removing old checkpoint checkpoint-500
kept last 2 resume checkpoints: checkpoint-1000 checkpoint-1500
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-1500 at step 1500 next_row=192000
full_cpt step  1510/12852 ( 11.7%)  nll=3.5295  ppl=34.11  grad=0.832  lr=1.95e-05  6848 tok/s  12.66s/it  [5:18:33<39:52:39]  mem=34.07GB
full_cpt step  1520/12852 ( 11.8%)  nll=3.5562  ppl=35.03  grad=0.859  lr=1.95e-05  6852 tok/s  12.65s/it  [5:20:29<39:49:16]  mem=34.07GB
full_cpt step  1530/12852 ( 11.9%)  nll=3.5207  ppl=33.81  grad=0.855  lr=1.95e-05  6856 tok/s  12.64s/it  [5:22:25<39:45:54]  mem=34.07GB
full_cpt step  1540/12852 ( 12.0%)  nll=3.3791  ppl=29.34  grad=0.941  lr=1.95e-05  6859 tok/s  12.64s/it  [5:24:21<39:42:33]  mem=34.07GB
full_cpt step  1550/12852 ( 12.1%)  nll=3.4615  ppl=31.87  grad=0.855  lr=1.95e-05  6863 tok/s  12.63s/it  [5:26:18<39:39:13]  mem=34.07GB
full_cpt step  1560/12852 ( 12.1%)  nll=3.6005  ppl=36.62  grad=0.879  lr=1.94e-05  6868 tok/s  12.62s/it  [5:28:14<39:35:54]  mem=34.07GB
full_cpt step  1570/12852 ( 12.2%)  nll=3.5305  ppl=34.14  grad=0.867  lr=1.94e-05  6872 tok/s  12.62s/it  [5:30:10<39:32:36]  mem=34.07GB
full_cpt step  1580/12852 ( 12.3%)  nll=3.4097  ppl=30.26  grad=1.016  lr=1.94e-05  6874 tok/s  12.61s/it  [5:32:06<39:29:19]  mem=34.07GB
full_cpt step  1590/12852 ( 12.4%)  nll=3.3741  ppl=29.20  grad=0.914  lr=1.94e-05  6878 tok/s  12.61s/it  [5:34:03<39:26:02]  mem=34.07GB
full_cpt step  1600/12852 ( 12.4%)  nll=3.4263  ppl=30.76  grad=0.844  lr=1.94e-05  6882 tok/s  12.60s/it  [5:35:59<39:22:48]  mem=34.07GB
  am   nll=4.1428  ppl=62.98  bpb=1.4568
  ta   nll=4.1132  ppl=61.14  bpb=0.5974
  kk   nll=2.8815  ppl=17.84  bpb=0.9703
  tl   nll=2.6822  ppl=14.62  bpb=1.2727
  en   nll=2.8175  ppl=16.74  bpb=0.8772
eval full_cpt step 1600  01:29 (89.2s)
full_cpt step  1610/12852 ( 12.5%)  nll=3.5164  ppl=33.66  grad=0.992  lr=1.94e-05  6855 tok/s  12.65s/it  [5:39:24<39:29:55]  mem=34.07GB
full_cpt step  1620/12852 ( 12.6%)  nll=3.4835  ppl=32.57  grad=0.969  lr=1.94e-05  6858 tok/s  12.64s/it  [5:41:20<39:26:37]  mem=34.07GB
full_cpt step  1630/12852 ( 12.7%)  nll=3.3300  ppl=27.94  grad=0.902  lr=1.94e-05  6862 tok/s  12.64s/it  [5:43:17<39:23:21]  mem=34.07GB
full_cpt step  1640/12852 ( 12.8%)  nll=3.3863  ppl=29.56  grad=0.980  lr=1.94e-05  6866 tok/s  12.63s/it  [5:45:13<39:20:05]  mem=34.07GB
full_cpt step  1650/12852 ( 12.8%)  nll=3.4368  ppl=31.09  grad=0.895  lr=1.94e-05  6868 tok/s  12.62s/it  [5:47:09<39:16:49]  mem=34.07GB
full_cpt step  1660/12852 ( 12.9%)  nll=3.4265  ppl=30.77  grad=0.984  lr=1.94e-05  6872 tok/s  12.62s/it  [5:49:05<39:13:35]  mem=34.07GB
full_cpt step  1670/12852 ( 13.0%)  nll=3.5378  ppl=34.39  grad=0.879  lr=1.94e-05  6875 tok/s  12.61s/it  [5:51:01<39:10:22]  mem=34.07GB
full_cpt step  1680/12852 ( 13.1%)  nll=3.3791  ppl=29.34  grad=1.023  lr=1.93e-05  6878 tok/s  12.61s/it  [5:52:57<39:07:09]  mem=34.07GB
full_cpt step  1690/12852 ( 13.1%)  nll=3.3743  ppl=29.20  grad=0.906  lr=1.93e-05  6882 tok/s  12.60s/it  [5:54:54<39:03:59]  mem=34.07GB
full_cpt step  1700/12852 ( 13.2%)  nll=3.3287  ppl=27.90  grad=0.859  lr=1.93e-05  6884 tok/s  12.59s/it  [5:56:50<39:00:48]  mem=34.07GB
  am   nll=4.1298  ppl=62.16  bpb=1.4522
  ta   nll=4.0989  ppl=60.27  bpb=0.5953
  kk   nll=2.8689  ppl=17.62  bpb=0.9661
  tl   nll=2.6805  ppl=14.59  bpb=1.2718
  en   nll=2.8175  ppl=16.73  bpb=0.8772
eval full_cpt step 1700  01:27 (87.4s)
full_cpt step  1710/12852 ( 13.3%)  nll=3.4315  ppl=30.92  grad=0.887  lr=1.93e-05  6860 tok/s  12.64s/it  [6:00:13<39:07:07]  mem=34.07GB
full_cpt step  1720/12852 ( 13.4%)  nll=3.4934  ppl=32.90  grad=0.957  lr=1.93e-05  6863 tok/s  12.63s/it  [6:02:09<39:03:55]  mem=34.07GB
full_cpt step  1730/12852 ( 13.5%)  nll=3.1659  ppl=23.71  grad=0.910  lr=1.93e-05  6866 tok/s  12.63s/it  [6:04:05<39:00:42]  mem=34.07GB
full_cpt step  1740/12852 ( 13.5%)  nll=3.4504  ppl=31.51  grad=0.887  lr=1.93e-05  6868 tok/s  12.62s/it  [6:06:02<38:57:31]  mem=34.07GB
full_cpt step  1750/12852 ( 13.6%)  nll=3.4786  ppl=32.42  grad=0.977  lr=1.93e-05  6871 tok/s  12.62s/it  [6:07:58<38:54:21]  mem=34.07GB
full_cpt step  1760/12852 ( 13.7%)  nll=3.3750  ppl=29.22  grad=0.840  lr=1.93e-05  6873 tok/s  12.61s/it  [6:09:54<38:51:11]  mem=34.07GB
full_cpt step  1770/12852 ( 13.8%)  nll=3.4869  ppl=32.68  grad=0.855  lr=1.93e-05  6876 tok/s  12.60s/it  [6:11:50<38:48:03]  mem=34.07GB
full_cpt step  1780/12852 ( 13.8%)  nll=3.5554  ppl=35.00  grad=0.914  lr=1.93e-05  6880 tok/s  12.60s/it  [6:13:46<38:44:56]  mem=34.07GB
full_cpt step  1790/12852 ( 13.9%)  nll=3.5641  ppl=35.31  grad=0.934  lr=1.93e-05  6883 tok/s  12.59s/it  [6:15:42<38:41:50]  mem=34.07GB
full_cpt step  1800/12852 ( 14.0%)  nll=3.3180  ppl=27.60  grad=0.875  lr=1.92e-05  6886 tok/s  12.59s/it  [6:17:39<38:38:45]  mem=34.07GB
  am   nll=4.1164  ppl=61.34  bpb=1.4476
  ta   nll=4.0870  ppl=59.56  bpb=0.5936
  kk   nll=2.8596  ppl=17.46  bpb=0.9630
  tl   nll=2.6785  ppl=14.56  bpb=1.2709
  en   nll=2.8175  ppl=16.73  bpb=0.8772
eval full_cpt step 1800  01:29 (88.6s)
full_cpt step  1810/12852 ( 14.1%)  nll=3.3135  ppl=27.48  grad=0.812  lr=1.92e-05  6864 tok/s  12.63s/it  [6:21:04<38:44:42]  mem=34.07GB
full_cpt step  1820/12852 ( 14.2%)  nll=3.4622  ppl=31.89  grad=0.891  lr=1.92e-05  6867 tok/s  12.63s/it  [6:23:00<38:41:35]  mem=34.07GB
full_cpt step  1830/12852 ( 14.2%)  nll=3.3685  ppl=29.04  grad=0.871  lr=1.92e-05  6870 tok/s  12.62s/it  [6:24:56<38:38:28]  mem=34.07GB
full_cpt step  1840/12852 ( 14.3%)  nll=3.5130  ppl=33.55  grad=0.898  lr=1.92e-05  6873 tok/s  12.62s/it  [6:26:53<38:35:22]  mem=34.07GB
full_cpt step  1850/12852 ( 14.4%)  nll=3.3123  ppl=27.45  grad=0.844  lr=1.92e-05  6877 tok/s  12.61s/it  [6:28:49<38:32:17]  mem=34.07GB
full_cpt step  1860/12852 ( 14.5%)  nll=3.4580  ppl=31.75  grad=0.871  lr=1.92e-05  6880 tok/s  12.60s/it  [6:30:45<38:29:13]  mem=34.07GB
full_cpt step  1870/12852 ( 14.6%)  nll=3.2985  ppl=27.07  grad=0.941  lr=1.92e-05  6883 tok/s  12.60s/it  [6:32:41<38:26:10]  mem=34.07GB
full_cpt step  1880/12852 ( 14.6%)  nll=3.4683  ppl=32.08  grad=0.863  lr=1.92e-05  6886 tok/s  12.59s/it  [6:34:38<38:23:07]  mem=34.07GB
full_cpt step  1890/12852 ( 14.7%)  nll=3.4725  ppl=32.22  grad=0.969  lr=1.92e-05  6890 tok/s  12.59s/it  [6:36:34<38:20:06]  mem=34.07GB
full_cpt step  1900/12852 ( 14.8%)  nll=3.3387  ppl=28.18  grad=0.902  lr=1.92e-05  6893 tok/s  12.58s/it  [6:38:30<38:17:04]  mem=34.07GB
  am   nll=4.1057  ppl=60.68  bpb=1.4438
  ta   nll=4.0737  ppl=58.77  bpb=0.5917
  kk   nll=2.8500  ppl=17.29  bpb=0.9597
  tl   nll=2.6766  ppl=14.54  bpb=1.2700
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 1900  01:30 (89.7s)
full_cpt step  1910/12852 ( 14.9%)  nll=3.4400  ppl=31.19  grad=0.938  lr=1.91e-05  6870 tok/s  12.63s/it  [6:41:56<38:22:35]  mem=34.07GB
full_cpt step  1920/12852 ( 14.9%)  nll=3.4093  ppl=30.24  grad=0.871  lr=1.91e-05  6873 tok/s  12.62s/it  [6:43:52<38:19:31]  mem=34.07GB
full_cpt step  1930/12852 ( 15.0%)  nll=3.3329  ppl=28.02  grad=0.852  lr=1.91e-05  6876 tok/s  12.62s/it  [6:45:48<38:16:27]  mem=34.07GB
full_cpt step  1940/12852 ( 15.1%)  nll=3.5041  ppl=33.25  grad=0.914  lr=1.91e-05  6878 tok/s  12.61s/it  [6:47:44<38:13:24]  mem=34.07GB
full_cpt step  1950/12852 ( 15.2%)  nll=3.3569  ppl=28.70  grad=0.902  lr=1.91e-05  6881 tok/s  12.61s/it  [6:49:40<38:10:21]  mem=34.07GB
full_cpt step  1960/12852 ( 15.3%)  nll=3.4603  ppl=31.83  grad=0.922  lr=1.91e-05  6884 tok/s  12.60s/it  [6:51:36<38:07:20]  mem=34.07GB
full_cpt step  1970/12852 ( 15.3%)  nll=3.4304  ppl=30.89  grad=0.867  lr=1.91e-05  6887 tok/s  12.59s/it  [6:53:32<38:04:19]  mem=34.07GB
full_cpt step  1980/12852 ( 15.4%)  nll=3.3635  ppl=28.89  grad=0.828  lr=1.91e-05  6890 tok/s  12.59s/it  [6:55:28<38:01:18]  mem=34.07GB
full_cpt step  1990/12852 ( 15.5%)  nll=3.3724  ppl=29.15  grad=0.887  lr=1.91e-05  6892 tok/s  12.59s/it  [6:57:24<37:58:19]  mem=34.07GB
full_cpt step  2000/12852 ( 15.6%)  nll=3.4527  ppl=31.58  grad=0.902  lr=1.91e-05  6895 tok/s  12.58s/it  [6:59:20<37:55:20]  mem=34.07GB
  am   nll=4.0939  ppl=59.97  bpb=1.4396
  ta   nll=4.0632  ppl=58.16  bpb=0.5901
  kk   nll=2.8424  ppl=17.16  bpb=0.9572
  tl   nll=2.6758  ppl=14.52  bpb=1.2696
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 2000  01:27 (87.3s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.27s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.27s/it]
removing old checkpoint checkpoint-1000
kept last 2 resume checkpoints: checkpoint-1500 checkpoint-2000
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-2000 at step 2000 next_row=256000

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.74s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.74s/it]
wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-2000 (weights+expanded tokenizer, no optimizer)
full_cpt step  2010/12852 ( 15.6%)  nll=3.3616  ppl=28.83  grad=0.922  lr=1.90e-05  6848 tok/s  12.67s/it  [7:04:22<38:09:01]  mem=34.07GB
full_cpt step  2020/12852 ( 15.7%)  nll=3.4129  ppl=30.35  grad=0.891  lr=1.90e-05  6851 tok/s  12.66s/it  [7:06:18<38:05:58]  mem=34.07GB
full_cpt step  2030/12852 ( 15.8%)  nll=3.4853  ppl=32.63  grad=0.891  lr=1.90e-05  6853 tok/s  12.66s/it  [7:08:14<38:02:56]  mem=34.07GB
full_cpt step  2040/12852 ( 15.9%)  nll=3.4149  ppl=30.41  grad=0.859  lr=1.90e-05  6856 tok/s  12.65s/it  [7:10:10<37:59:54]  mem=34.07GB
full_cpt step  2050/12852 ( 16.0%)  nll=3.4357  ppl=31.05  grad=0.926  lr=1.90e-05  6859 tok/s  12.65s/it  [7:12:06<37:56:53]  mem=34.07GB
full_cpt step  2060/12852 ( 16.0%)  nll=3.4271  ppl=30.79  grad=0.965  lr=1.90e-05  6861 tok/s  12.64s/it  [7:14:02<37:53:52]  mem=34.07GB
full_cpt step  2070/12852 ( 16.1%)  nll=3.3403  ppl=28.23  grad=0.863  lr=1.90e-05  6864 tok/s  12.64s/it  [7:15:59<37:50:52]  mem=34.07GB
full_cpt step  2080/12852 ( 16.2%)  nll=3.2685  ppl=26.27  grad=0.898  lr=1.90e-05  6866 tok/s  12.63s/it  [7:17:55<37:47:53]  mem=34.07GB
full_cpt step  2090/12852 ( 16.3%)  nll=3.4011  ppl=30.00  grad=0.895  lr=1.90e-05  6868 tok/s  12.63s/it  [7:19:51<37:44:53]  mem=34.07GB
full_cpt step  2100/12852 ( 16.3%)  nll=3.3971  ppl=29.88  grad=0.930  lr=1.90e-05  6871 tok/s  12.62s/it  [7:21:47<37:41:55]  mem=34.07GB
  am   nll=4.0848  ppl=59.43  bpb=1.4364
  ta   nll=4.0533  ppl=57.59  bpb=0.5887
  kk   nll=2.8351  ppl=17.03  bpb=0.9547
  tl   nll=2.6750  ppl=14.51  bpb=1.2692
  en   nll=2.8172  ppl=16.73  bpb=0.8772
eval full_cpt step 2100  01:28 (87.7s)
full_cpt step  2110/12852 ( 16.4%)  nll=3.3259  ppl=27.82  grad=1.047  lr=1.89e-05  6851 tok/s  12.66s/it  [7:25:11<37:46:24]  mem=34.07GB
full_cpt step  2120/12852 ( 16.5%)  nll=3.6365  ppl=37.96  grad=0.922  lr=1.89e-05  6853 tok/s  12.65s/it  [7:27:07<37:43:24]  mem=34.07GB
full_cpt step  2130/12852 ( 16.6%)  nll=3.3676  ppl=29.01  grad=0.941  lr=1.89e-05  6857 tok/s  12.65s/it  [7:29:03<37:40:26]  mem=34.07GB
full_cpt step  2140/12852 ( 16.7%)  nll=3.4597  ppl=31.81  grad=0.883  lr=1.89e-05  6860 tok/s  12.64s/it  [7:30:59<37:37:28]  mem=34.07GB
full_cpt step  2150/12852 ( 16.7%)  nll=3.3435  ppl=28.32  grad=0.883  lr=1.89e-05  6862 tok/s  12.64s/it  [7:32:55<37:34:30]  mem=34.07GB
full_cpt step  2160/12852 ( 16.8%)  nll=3.4069  ppl=30.17  grad=0.848  lr=1.89e-05  6864 tok/s  12.63s/it  [7:34:51<37:31:32]  mem=34.07GB
full_cpt step  2170/12852 ( 16.9%)  nll=3.3236  ppl=27.76  grad=0.984  lr=1.89e-05  6867 tok/s  12.63s/it  [7:36:47<37:28:34]  mem=34.07GB
full_cpt step  2180/12852 ( 17.0%)  nll=3.3631  ppl=28.88  grad=0.891  lr=1.89e-05  6869 tok/s  12.63s/it  [7:38:43<37:25:38]  mem=34.07GB
full_cpt step  2190/12852 ( 17.0%)  nll=3.4803  ppl=32.47  grad=0.863  lr=1.89e-05  6872 tok/s  12.62s/it  [7:40:40<37:22:43]  mem=34.07GB
full_cpt step  2200/12852 ( 17.1%)  nll=3.4452  ppl=31.35  grad=0.926  lr=1.88e-05  6874 tok/s  12.62s/it  [7:42:36<37:19:47]  mem=34.07GB
  am   nll=4.0786  ppl=59.06  bpb=1.4342
  ta   nll=4.0440  ppl=57.05  bpb=0.5874
  kk   nll=2.8280  ppl=16.91  bpb=0.9523
  tl   nll=2.6738  ppl=14.49  bpb=1.2686
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 2200  01:27 (86.9s)
full_cpt step  2210/12852 ( 17.2%)  nll=3.4542  ppl=31.63  grad=1.203  lr=1.88e-05  6855 tok/s  12.65s/it  [7:45:58<37:23:51]  mem=34.07GB
full_cpt step  2220/12852 ( 17.3%)  nll=3.3421  ppl=28.28  grad=0.918  lr=1.88e-05  6858 tok/s  12.65s/it  [7:47:55<37:20:54]  mem=34.07GB
full_cpt step  2230/12852 ( 17.4%)  nll=3.3061  ppl=27.28  grad=0.918  lr=1.88e-05  6860 tok/s  12.64s/it  [7:49:51<37:17:58]  mem=34.07GB
full_cpt step  2240/12852 ( 17.4%)  nll=3.3513  ppl=28.54  grad=0.891  lr=1.88e-05  6863 tok/s  12.64s/it  [7:51:47<37:15:03]  mem=34.07GB
full_cpt step  2250/12852 ( 17.5%)  nll=3.3691  ppl=29.05  grad=0.855  lr=1.88e-05  6864 tok/s  12.63s/it  [7:53:43<37:12:08]  mem=34.07GB
full_cpt step  2260/12852 ( 17.6%)  nll=3.3389  ppl=28.19  grad=0.926  lr=1.88e-05  6867 tok/s  12.63s/it  [7:55:39<37:09:13]  mem=34.07GB
full_cpt step  2270/12852 ( 17.7%)  nll=3.2810  ppl=26.60  grad=0.867  lr=1.88e-05  6870 tok/s  12.62s/it  [7:57:35<37:06:20]  mem=34.07GB
full_cpt step  2280/12852 ( 17.7%)  nll=3.4405  ppl=31.20  grad=0.910  lr=1.88e-05  6873 tok/s  12.62s/it  [7:59:31<37:03:27]  mem=34.07GB
full_cpt step  2290/12852 ( 17.8%)  nll=3.4135  ppl=30.37  grad=0.906  lr=1.87e-05  6875 tok/s  12.61s/it  [8:01:27<37:00:34]  mem=34.07GB
full_cpt step  2300/12852 ( 17.9%)  nll=3.3808  ppl=29.39  grad=0.859  lr=1.87e-05  6878 tok/s  12.61s/it  [8:03:23<36:57:42]  mem=34.07GB
  am   nll=4.0700  ppl=58.56  bpb=1.4312
  ta   nll=4.0363  ppl=56.61  bpb=0.5862
  kk   nll=2.8222  ppl=16.81  bpb=0.9504
  tl   nll=2.6725  ppl=14.48  bpb=1.2680
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 2300  01:27 (86.7s)
full_cpt step  2310/12852 ( 18.0%)  nll=3.3639  ppl=28.90  grad=0.891  lr=1.87e-05  6859 tok/s  12.64s/it  [8:06:46<37:01:25]  mem=34.07GB
full_cpt step  2320/12852 ( 18.1%)  nll=3.3103  ppl=27.39  grad=0.863  lr=1.87e-05  6861 tok/s  12.64s/it  [8:08:42<36:58:32]  mem=34.07GB
full_cpt step  2330/12852 ( 18.1%)  nll=3.3149  ppl=27.52  grad=0.887  lr=1.87e-05  6864 tok/s  12.63s/it  [8:10:38<36:55:39]  mem=34.07GB
full_cpt step  2340/12852 ( 18.2%)  nll=3.1849  ppl=24.17  grad=0.930  lr=1.87e-05  6865 tok/s  12.63s/it  [8:12:34<36:52:45]  mem=34.07GB
full_cpt step  2350/12852 ( 18.3%)  nll=3.4273  ppl=30.79  grad=0.855  lr=1.87e-05  6868 tok/s  12.63s/it  [8:14:30<36:49:54]  mem=34.07GB
full_cpt step  2360/12852 ( 18.4%)  nll=3.3490  ppl=28.47  grad=1.055  lr=1.87e-05  6870 tok/s  12.62s/it  [8:16:26<36:47:02]  mem=34.07GB
full_cpt step  2370/12852 ( 18.4%)  nll=3.5643  ppl=35.32  grad=0.879  lr=1.87e-05  6872 tok/s  12.62s/it  [8:18:22<36:44:11]  mem=34.07GB
full_cpt step  2380/12852 ( 18.5%)  nll=3.4909  ppl=32.82  grad=0.859  lr=1.86e-05  6874 tok/s  12.61s/it  [8:20:18<36:41:20]  mem=34.07GB
full_cpt step  2390/12852 ( 18.6%)  nll=3.3274  ppl=27.87  grad=0.934  lr=1.86e-05  6877 tok/s  12.61s/it  [8:22:14<36:38:30]  mem=34.07GB
full_cpt step  2400/12852 ( 18.7%)  nll=3.3946  ppl=29.80  grad=0.930  lr=1.86e-05  6879 tok/s  12.60s/it  [8:24:10<36:35:41]  mem=34.07GB
  am   nll=4.0635  ppl=58.18  bpb=1.4289
  ta   nll=4.0282  ppl=56.16  bpb=0.5851
  kk   nll=2.8159  ppl=16.71  bpb=0.9482
  tl   nll=2.6715  ppl=14.46  bpb=1.2676
  en   nll=2.8173  ppl=16.73  bpb=0.8772
eval full_cpt step 2400  01:27 (86.8s)
full_cpt step  2410/12852 ( 18.8%)  nll=3.3297  ppl=27.93  grad=0.941  lr=1.86e-05  6862 tok/s  12.64s/it  [8:27:33<36:39:07]  mem=34.07GB
full_cpt step  2420/12852 ( 18.8%)  nll=3.3200  ppl=27.66  grad=0.922  lr=1.86e-05  6864 tok/s  12.63s/it  [8:29:29<36:36:17]  mem=34.07GB
full_cpt step  2430/12852 ( 18.9%)  nll=3.3080  ppl=27.33  grad=0.906  lr=1.86e-05  6866 tok/s  12.63s/it  [8:31:25<36:33:27]  mem=34.07GB
full_cpt step  2440/12852 ( 19.0%)  nll=3.4604  ppl=31.83  grad=1.008  lr=1.86e-05  6868 tok/s  12.62s/it  [8:33:22<36:30:37]  mem=34.07GB
full_cpt step  2450/12852 ( 19.1%)  nll=3.3371  ppl=28.14  grad=0.848  lr=1.86e-05  6871 tok/s  12.62s/it  [8:35:18<36:27:47]  mem=34.07GB
full_cpt step  2460/12852 ( 19.1%)  nll=3.4006  ppl=29.98  grad=0.840  lr=1.85e-05  6872 tok/s  12.62s/it  [8:37:13<36:24:58]  mem=34.07GB
full_cpt step  2470/12852 ( 19.2%)  nll=3.3719  ppl=29.13  grad=0.941  lr=1.85e-05  6874 tok/s  12.61s/it  [8:39:09<36:22:09]  mem=34.07GB
full_cpt step  2480/12852 ( 19.3%)  nll=3.2523  ppl=25.85  grad=0.934  lr=1.85e-05  6876 tok/s  12.61s/it  [8:41:05<36:19:20]  mem=34.07GB
full_cpt step  2490/12852 ( 19.4%)  nll=3.1766  ppl=23.97  grad=0.844  lr=1.85e-05  6878 tok/s  12.60s/it  [8:43:02<36:16:33]  mem=34.07GB
full_cpt step  2500/12852 ( 19.5%)  nll=3.3358  ppl=28.10  grad=1.016  lr=1.85e-05  6880 tok/s  12.60s/it  [8:44:58<36:13:46]  mem=34.07GB
  am   nll=4.0553  ppl=57.70  bpb=1.4260
  ta   nll=4.0217  ppl=55.79  bpb=0.5841
  kk   nll=2.8106  ppl=16.62  bpb=0.9464
  tl   nll=2.6706  ppl=14.45  bpb=1.2671
  en   nll=2.8175  ppl=16.74  bpb=0.8772
eval full_cpt step 2500  01:27 (87.4s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.40s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.40s/it]
removing old checkpoint checkpoint-1500
kept last 2 resume checkpoints: checkpoint-2000 checkpoint-2500
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-2500 at step 2500 next_row=320000
full_cpt step  2510/12852 ( 19.5%)  nll=3.5412  ppl=34.51  grad=0.852  lr=1.85e-05  6849 tok/s  12.66s/it  [8:49:30<36:21:43]  mem=34.07GB
full_cpt step  2520/12852 ( 19.6%)  nll=3.3689  ppl=29.05  grad=0.918  lr=1.85e-05  6852 tok/s  12.65s/it  [8:51:26<36:18:54]  mem=34.07GB
full_cpt step  2530/12852 ( 19.7%)  nll=3.4242  ppl=30.70  grad=0.871  lr=1.85e-05  6854 tok/s  12.65s/it  [8:53:22<36:16:05]  mem=34.07GB
full_cpt step  2540/12852 ( 19.8%)  nll=3.3884  ppl=29.62  grad=0.945  lr=1.85e-05  6856 tok/s  12.65s/it  [8:55:18<36:13:16]  mem=34.07GB
full_cpt step  2550/12852 ( 19.8%)  nll=3.3614  ppl=28.83  grad=0.887  lr=1.84e-05  6858 tok/s  12.64s/it  [8:57:14<36:10:27]  mem=34.07GB
full_cpt step  2560/12852 ( 19.9%)  nll=3.3251  ppl=27.80  grad=0.910  lr=1.84e-05  6860 tok/s  12.64s/it  [8:59:10<36:07:39]  mem=34.07GB
full_cpt step  2570/12852 ( 20.0%)  nll=3.3147  ppl=27.51  grad=0.852  lr=1.84e-05  6862 tok/s  12.63s/it  [9:01:07<36:04:52]  mem=34.07GB
full_cpt step  2580/12852 ( 20.1%)  nll=3.3842  ppl=29.49  grad=0.867  lr=1.84e-05  6865 tok/s  12.63s/it  [9:03:03<36:02:05]  mem=34.07GB
full_cpt step  2590/12852 ( 20.2%)  nll=3.3318  ppl=27.99  grad=0.883  lr=1.84e-05  6867 tok/s  12.63s/it  [9:04:59<35:59:18]  mem=34.07GB
full_cpt step  2600/12852 ( 20.2%)  nll=3.3510  ppl=28.53  grad=0.898  lr=1.84e-05  6870 tok/s  12.62s/it  [9:06:55<35:56:32]  mem=34.07GB
  am   nll=4.0492  ppl=57.35  bpb=1.4239
  ta   nll=4.0158  ppl=55.47  bpb=0.5833
  kk   nll=2.8058  ppl=16.54  bpb=0.9448
  tl   nll=2.6697  ppl=14.44  bpb=1.2667
  en   nll=2.8175  ppl=16.73  bpb=0.8772
eval full_cpt step 2600  01:27 (87.4s)
full_cpt step  2610/12852 ( 20.3%)  nll=3.4266  ppl=30.77  grad=0.902  lr=1.84e-05  6854 tok/s  12.65s/it  [9:10:18<35:59:29]  mem=34.07GB
full_cpt step  2620/12852 ( 20.4%)  nll=3.5226  ppl=33.87  grad=0.910  lr=1.83e-05  6856 tok/s  12.65s/it  [9:12:14<35:56:41]  mem=34.07GB
full_cpt step  2630/12852 ( 20.5%)  nll=3.4147  ppl=30.41  grad=0.934  lr=1.83e-05  6858 tok/s  12.64s/it  [9:14:11<35:53:54]  mem=34.07GB
full_cpt step  2640/12852 ( 20.5%)  nll=3.3771  ppl=29.29  grad=0.914  lr=1.83e-05  6860 tok/s  12.64s/it  [9:16:06<35:51:07]  mem=34.07GB
full_cpt step  2650/12852 ( 20.6%)  nll=3.2391  ppl=25.51  grad=0.957  lr=1.83e-05  6862 tok/s  12.63s/it  [9:18:02<35:48:21]  mem=34.07GB
full_cpt step  2660/12852 ( 20.7%)  nll=3.3965  ppl=29.86  grad=0.961  lr=1.83e-05  6864 tok/s  12.63s/it  [9:19:59<35:45:36]  mem=34.07GB
full_cpt step  2670/12852 ( 20.8%)  nll=3.3516  ppl=28.55  grad=0.898  lr=1.83e-05  6867 tok/s  12.63s/it  [9:21:55<35:42:50]  mem=34.07GB
full_cpt step  2680/12852 ( 20.9%)  nll=3.4038  ppl=30.08  grad=0.879  lr=1.83e-05  6869 tok/s  12.62s/it  [9:23:51<35:40:05]  mem=34.07GB
full_cpt step  2690/12852 ( 20.9%)  nll=3.2725  ppl=26.38  grad=0.961  lr=1.83e-05  6870 tok/s  12.62s/it  [9:25:47<35:37:20]  mem=34.07GB
full_cpt step  2700/12852 ( 21.0%)  nll=3.3709  ppl=29.11  grad=0.875  lr=1.82e-05  6872 tok/s  12.62s/it  [9:27:43<35:34:36]  mem=34.07GB
  am   nll=4.0434  ppl=57.02  bpb=1.4219
  ta   nll=4.0096  ppl=55.12  bpb=0.5824
  kk   nll=2.8017  ppl=16.47  bpb=0.9434
  tl   nll=2.6688  ppl=14.42  bpb=1.2663
  en   nll=2.8171  ppl=16.73  bpb=0.8771
eval full_cpt step 2700  01:27 (87.2s)
full_cpt step  2710/12852 ( 21.1%)  nll=3.2168  ppl=24.95  grad=0.910  lr=1.82e-05  6858 tok/s  12.64s/it  [9:31:06<35:37:19]  mem=34.07GB
full_cpt step  2720/12852 ( 21.2%)  nll=3.3410  ppl=28.25  grad=0.836  lr=1.82e-05  6860 tok/s  12.64s/it  [9:33:02<35:34:34]  mem=34.07GB
full_cpt step  2730/12852 ( 21.2%)  nll=3.3651  ppl=28.94  grad=0.887  lr=1.82e-05  6862 tok/s  12.64s/it  [9:34:58<35:31:49]  mem=34.07GB
full_cpt step  2740/12852 ( 21.3%)  nll=3.4179  ppl=30.50  grad=0.926  lr=1.82e-05  6863 tok/s  12.63s/it  [9:36:54<35:29:05]  mem=34.07GB
full_cpt step  2750/12852 ( 21.4%)  nll=3.3009  ppl=27.14  grad=0.871  lr=1.82e-05  6865 tok/s  12.63s/it  [9:38:50<35:26:21]  mem=34.07GB
full_cpt step  2760/12852 ( 21.5%)  nll=3.4507  ppl=31.52  grad=0.879  lr=1.82e-05  6867 tok/s  12.63s/it  [9:40:47<35:23:37]  mem=34.07GB
full_cpt step  2770/12852 ( 21.6%)  nll=3.3995  ppl=29.95  grad=0.879  lr=1.82e-05  6869 tok/s  12.62s/it  [9:42:43<35:20:54]  mem=34.07GB
full_cpt step  2780/12852 ( 21.6%)  nll=3.3821  ppl=29.43  grad=0.859  lr=1.81e-05  6871 tok/s  12.62s/it  [9:44:39<35:18:11]  mem=34.07GB
full_cpt step  2790/12852 ( 21.7%)  nll=3.4893  ppl=32.76  grad=0.953  lr=1.81e-05  6873 tok/s  12.61s/it  [9:46:35<35:15:28]  mem=34.07GB
full_cpt step  2800/12852 ( 21.8%)  nll=3.3556  ppl=28.66  grad=0.855  lr=1.81e-05  6874 tok/s  12.61s/it  [9:48:31<35:12:45]  mem=34.07GB
  am   nll=4.0375  ppl=56.68  bpb=1.4198
  ta   nll=4.0043  ppl=54.84  bpb=0.5816
  kk   nll=2.7974  ppl=16.40  bpb=0.9420
  tl   nll=2.6681  ppl=14.41  bpb=1.2659
  en   nll=2.8179  ppl=16.74  bpb=0.8774
eval full_cpt step 2800  01:29 (88.6s)
full_cpt step  2810/12852 ( 21.9%)  nll=3.2925  ppl=26.91  grad=0.828  lr=1.81e-05  6859 tok/s  12.64s/it  [9:51:55<35:15:20]  mem=34.07GB
full_cpt step  2820/12852 ( 21.9%)  nll=3.2135  ppl=24.86  grad=0.871  lr=1.81e-05  6862 tok/s  12.64s/it  [9:53:51<35:12:37]  mem=34.07GB
full_cpt step  2830/12852 ( 22.0%)  nll=3.3102  ppl=27.39  grad=0.848  lr=1.81e-05  6864 tok/s  12.63s/it  [9:55:47<35:09:54]  mem=34.07GB
full_cpt step  2840/12852 ( 22.1%)  nll=3.3786  ppl=29.33  grad=0.867  lr=1.81e-05  6866 tok/s  12.63s/it  [9:57:43<35:07:11]  mem=34.07GB
full_cpt step  2850/12852 ( 22.2%)  nll=3.4983  ppl=33.06  grad=0.844  lr=1.80e-05  6867 tok/s  12.62s/it  [9:59:39<35:04:29]  mem=34.07GB
full_cpt step  2860/12852 ( 22.3%)  nll=3.4347  ppl=31.02  grad=0.875  lr=1.80e-05  6869 tok/s  12.62s/it  [10:01:35<35:01:47]  mem=34.07GB
full_cpt step  2870/12852 ( 22.3%)  nll=3.4857  ppl=32.65  grad=0.992  lr=1.80e-05  6870 tok/s  12.62s/it  [10:03:31<34:59:05]  mem=34.07GB
full_cpt step  2880/12852 ( 22.4%)  nll=3.2174  ppl=24.96  grad=0.879  lr=1.80e-05  6872 tok/s  12.61s/it  [10:05:28<34:56:24]  mem=34.07GB
full_cpt step  2890/12852 ( 22.5%)  nll=3.2848  ppl=26.70  grad=0.930  lr=1.80e-05  6874 tok/s  12.61s/it  [10:07:24<34:53:44]  mem=34.07GB
full_cpt step  2900/12852 ( 22.6%)  nll=3.3325  ppl=28.01  grad=0.852  lr=1.80e-05  6876 tok/s  12.61s/it  [10:09:20<34:51:04]  mem=34.07GB
  am   nll=4.0324  ppl=56.40  bpb=1.4180
  ta   nll=3.9994  ppl=54.56  bpb=0.5809
  kk   nll=2.7936  ppl=16.34  bpb=0.9407
  tl   nll=2.6673  ppl=14.40  bpb=1.2656
  en   nll=2.8178  ppl=16.74  bpb=0.8773
eval full_cpt step 2900  01:27 (87.5s)
full_cpt step  2910/12852 ( 22.6%)  nll=3.1911  ppl=24.31  grad=0.891  lr=1.80e-05  6862 tok/s  12.63s/it  [10:12:43<34:53:22]  mem=34.07GB
full_cpt step  2920/12852 ( 22.7%)  nll=3.4941  ppl=32.92  grad=0.934  lr=1.79e-05  6863 tok/s  12.63s/it  [10:14:39<34:50:40]  mem=34.07GB
full_cpt step  2930/12852 ( 22.8%)  nll=3.4455  ppl=31.36  grad=0.898  lr=1.79e-05  6865 tok/s  12.63s/it  [10:16:36<34:48:00]  mem=34.07GB
full_cpt step  2940/12852 ( 22.9%)  nll=3.5284  ppl=34.07  grad=0.871  lr=1.79e-05  6867 tok/s  12.62s/it  [10:18:32<34:45:19]  mem=34.07GB
full_cpt step  2950/12852 ( 23.0%)  nll=3.2054  ppl=24.66  grad=0.875  lr=1.79e-05  6869 tok/s  12.62s/it  [10:20:28<34:42:39]  mem=34.07GB
full_cpt step  2960/12852 ( 23.0%)  nll=3.4266  ppl=30.77  grad=0.879  lr=1.79e-05  6871 tok/s  12.62s/it  [10:22:24<34:39:59]  mem=34.07GB
full_cpt step  2970/12852 ( 23.1%)  nll=3.2708  ppl=26.33  grad=0.945  lr=1.79e-05  6873 tok/s  12.61s/it  [10:24:20<34:37:19]  mem=34.07GB
full_cpt step  2980/12852 ( 23.2%)  nll=3.2657  ppl=26.20  grad=0.812  lr=1.79e-05  6874 tok/s  12.61s/it  [10:26:16<34:34:39]  mem=34.07GB
full_cpt step  2990/12852 ( 23.3%)  nll=3.3059  ppl=27.27  grad=0.906  lr=1.78e-05  6876 tok/s  12.61s/it  [10:28:12<34:32:01]  mem=34.07GB
full_cpt step  3000/12852 ( 23.3%)  nll=3.2788  ppl=26.54  grad=0.859  lr=1.78e-05  6878 tok/s  12.60s/it  [10:30:08<34:29:22]  mem=34.07GB
  am   nll=4.0275  ppl=56.12  bpb=1.4163
  ta   nll=3.9949  ppl=54.32  bpb=0.5802
  kk   nll=2.7903  ppl=16.29  bpb=0.9396
  tl   nll=2.6668  ppl=14.39  bpb=1.2654
  en   nll=2.8175  ppl=16.74  bpb=0.8773
eval full_cpt step 3000  01:27 (87.4s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.33s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.34s/it]
removing old checkpoint checkpoint-2000
kept last 2 resume checkpoints: checkpoint-2500 checkpoint-3000
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-3000 at step 3000 next_row=384000

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.88s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.88s/it]
wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-3000 (weights+expanded tokenizer, no optimizer)
full_cpt step  3010/12852 ( 23.4%)  nll=3.2212  ppl=25.06  grad=0.898  lr=1.78e-05  6847 tok/s  12.66s/it  [10:35:07<34:36:40]  mem=34.07GB
full_cpt step  3020/12852 ( 23.5%)  nll=3.5225  ppl=33.87  grad=0.953  lr=1.78e-05  6849 tok/s  12.66s/it  [10:37:03<34:33:59]  mem=34.07GB
full_cpt step  3030/12852 ( 23.6%)  nll=3.3544  ppl=28.63  grad=0.965  lr=1.78e-05  6851 tok/s  12.65s/it  [10:38:59<34:31:18]  mem=34.07GB
full_cpt step  3040/12852 ( 23.7%)  nll=3.2071  ppl=24.71  grad=0.867  lr=1.78e-05  6853 tok/s  12.65s/it  [10:40:55<34:28:38]  mem=34.07GB
full_cpt step  3050/12852 ( 23.7%)  nll=3.3298  ppl=27.93  grad=0.824  lr=1.78e-05  6855 tok/s  12.65s/it  [10:42:51<34:25:58]  mem=34.07GB
full_cpt step  3060/12852 ( 23.8%)  nll=3.2161  ppl=24.93  grad=0.895  lr=1.77e-05  6857 tok/s  12.64s/it  [10:44:47<34:23:19]  mem=34.07GB
full_cpt step  3070/12852 ( 23.9%)  nll=3.3118  ppl=27.43  grad=0.867  lr=1.77e-05  6858 tok/s  12.64s/it  [10:46:43<34:20:40]  mem=34.07GB
full_cpt step  3080/12852 ( 24.0%)  nll=3.3263  ppl=27.83  grad=0.871  lr=1.77e-05  6860 tok/s  12.64s/it  [10:48:39<34:18:00]  mem=34.07GB
full_cpt step  3090/12852 ( 24.0%)  nll=3.3137  ppl=27.49  grad=0.906  lr=1.77e-05  6861 tok/s  12.63s/it  [10:50:35<34:15:21]  mem=34.07GB
full_cpt step  3100/12852 ( 24.1%)  nll=3.3402  ppl=28.22  grad=0.906  lr=1.77e-05  6863 tok/s  12.63s/it  [10:52:31<34:12:43]  mem=34.07GB
  am   nll=4.0230  ppl=55.87  bpb=1.4147
  ta   nll=3.9910  ppl=54.11  bpb=0.5797
  kk   nll=2.7872  ppl=16.23  bpb=0.9386
  tl   nll=2.6665  ppl=14.39  bpb=1.2652
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 3100  01:29 (88.8s)
full_cpt step  3110/12852 ( 24.2%)  nll=3.3203  ppl=27.67  grad=0.926  lr=1.77e-05  6849 tok/s  12.65s/it  [10:55:56<34:14:42]  mem=34.07GB
full_cpt step  3120/12852 ( 24.3%)  nll=3.3614  ppl=28.83  grad=0.887  lr=1.77e-05  6851 tok/s  12.65s/it  [10:57:52<34:12:03]  mem=34.07GB
full_cpt step  3130/12852 ( 24.4%)  nll=3.3152  ppl=27.53  grad=0.977  lr=1.76e-05  6853 tok/s  12.65s/it  [10:59:48<34:09:24]  mem=34.07GB
full_cpt step  3140/12852 ( 24.4%)  nll=3.2643  ppl=26.16  grad=0.895  lr=1.76e-05  6855 tok/s  12.64s/it  [11:01:45<34:06:46]  mem=34.07GB
full_cpt step  3150/12852 ( 24.5%)  nll=3.3826  ppl=29.45  grad=0.949  lr=1.76e-05  6857 tok/s  12.64s/it  [11:03:41<34:04:08]  mem=34.07GB
full_cpt step  3160/12852 ( 24.6%)  nll=3.1855  ppl=24.18  grad=0.863  lr=1.76e-05  6859 tok/s  12.64s/it  [11:05:37<34:01:30]  mem=34.07GB
full_cpt step  3170/12852 ( 24.7%)  nll=3.3632  ppl=28.88  grad=0.895  lr=1.76e-05  6860 tok/s  12.64s/it  [11:07:33<33:58:52]  mem=34.07GB
full_cpt step  3180/12852 ( 24.7%)  nll=3.3905  ppl=29.68  grad=0.867  lr=1.76e-05  6863 tok/s  12.63s/it  [11:09:29<33:56:15]  mem=34.07GB
full_cpt step  3190/12852 ( 24.8%)  nll=3.3715  ppl=29.12  grad=0.852  lr=1.75e-05  6864 tok/s  12.63s/it  [11:11:25<33:53:37]  mem=34.07GB
full_cpt step  3200/12852 ( 24.9%)  nll=3.3026  ppl=27.18  grad=0.898  lr=1.75e-05  6866 tok/s  12.63s/it  [11:13:21<33:51:00]  mem=34.07GB
  am   nll=4.0185  ppl=55.62  bpb=1.4131
  ta   nll=3.9863  ppl=53.86  bpb=0.5790
  kk   nll=2.7841  ppl=16.18  bpb=0.9375
  tl   nll=2.6662  ppl=14.39  bpb=1.2651
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 3200  01:26 (86.5s)
full_cpt step  3210/12852 ( 25.0%)  nll=3.4051  ppl=30.12  grad=0.957  lr=1.75e-05  6852 tok/s  12.65s/it  [11:16:44<33:52:43]  mem=34.07GB
full_cpt step  3220/12852 ( 25.1%)  nll=3.3297  ppl=27.93  grad=0.930  lr=1.75e-05  6854 tok/s  12.65s/it  [11:18:40<33:50:05]  mem=34.07GB
full_cpt step  3230/12852 ( 25.1%)  nll=3.4816  ppl=32.51  grad=0.973  lr=1.75e-05  6857 tok/s  12.64s/it  [11:20:36<33:47:28]  mem=34.07GB
full_cpt step  3240/12852 ( 25.2%)  nll=3.3376  ppl=28.15  grad=0.930  lr=1.75e-05  6858 tok/s  12.64s/it  [11:22:32<33:44:51]  mem=34.07GB
full_cpt step  3250/12852 ( 25.3%)  nll=3.2956  ppl=26.99  grad=0.832  lr=1.75e-05  6860 tok/s  12.64s/it  [11:24:28<33:42:15]  mem=34.07GB
full_cpt step  3260/12852 ( 25.4%)  nll=3.2708  ppl=26.33  grad=0.918  lr=1.74e-05  6862 tok/s  12.63s/it  [11:26:24<33:39:38]  mem=34.07GB
full_cpt step  3270/12852 ( 25.4%)  nll=3.4030  ppl=30.05  grad=0.957  lr=1.74e-05  6863 tok/s  12.63s/it  [11:28:20<33:37:02]  mem=34.07GB
full_cpt step  3280/12852 ( 25.5%)  nll=3.4093  ppl=30.24  grad=1.000  lr=1.74e-05  6865 tok/s  12.63s/it  [11:30:17<33:34:26]  mem=34.07GB
full_cpt step  3290/12852 ( 25.6%)  nll=3.3485  ppl=28.46  grad=0.836  lr=1.74e-05  6867 tok/s  12.62s/it  [11:32:13<33:31:50]  mem=34.07GB
full_cpt step  3300/12852 ( 25.7%)  nll=3.3036  ppl=27.21  grad=0.922  lr=1.74e-05  6870 tok/s  12.62s/it  [11:34:09<33:29:15]  mem=34.07GB
  am   nll=4.0155  ppl=55.45  bpb=1.4121
  ta   nll=3.9827  ppl=53.66  bpb=0.5785
  kk   nll=2.7813  ppl=16.14  bpb=0.9366
  tl   nll=2.6652  ppl=14.37  bpb=1.2646
  en   nll=2.8179  ppl=16.74  bpb=0.8774
eval full_cpt step 3300  01:28 (87.6s)
full_cpt step  3310/12852 ( 25.8%)  nll=3.1741  ppl=23.91  grad=0.867  lr=1.74e-05  6857 tok/s  12.64s/it  [11:37:33<33:30:52]  mem=34.07GB
full_cpt step  3320/12852 ( 25.8%)  nll=3.3239  ppl=27.77  grad=0.840  lr=1.73e-05  6858 tok/s  12.64s/it  [11:39:29<33:28:16]  mem=34.07GB
full_cpt step  3330/12852 ( 25.9%)  nll=3.3961  ppl=29.85  grad=0.871  lr=1.73e-05  6860 tok/s  12.64s/it  [11:41:25<33:25:40]  mem=34.07GB
full_cpt step  3340/12852 ( 26.0%)  nll=3.2743  ppl=26.43  grad=0.902  lr=1.73e-05  6861 tok/s  12.63s/it  [11:43:21<33:23:03]  mem=34.07GB
full_cpt step  3350/12852 ( 26.1%)  nll=3.3892  ppl=29.64  grad=0.895  lr=1.73e-05  6863 tok/s  12.63s/it  [11:45:17<33:20:28]  mem=34.07GB
full_cpt step  3360/12852 ( 26.1%)  nll=3.2678  ppl=26.25  grad=0.945  lr=1.73e-05  6864 tok/s  12.63s/it  [11:47:13<33:17:52]  mem=34.07GB
full_cpt step  3370/12852 ( 26.2%)  nll=3.3735  ppl=29.18  grad=0.906  lr=1.73e-05  6866 tok/s  12.63s/it  [11:49:09<33:15:17]  mem=34.07GB
full_cpt step  3380/12852 ( 26.3%)  nll=3.3481  ppl=28.45  grad=0.906  lr=1.73e-05  6868 tok/s  12.62s/it  [11:51:05<33:12:42]  mem=34.07GB
full_cpt step  3390/12852 ( 26.4%)  nll=3.4800  ppl=32.46  grad=0.855  lr=1.72e-05  6869 tok/s  12.62s/it  [11:53:01<33:10:08]  mem=34.07GB
full_cpt step  3400/12852 ( 26.5%)  nll=3.4799  ppl=32.46  grad=0.914  lr=1.72e-05  6870 tok/s  12.62s/it  [11:54:57<33:07:33]  mem=34.07GB
  am   nll=4.0123  ppl=55.27  bpb=1.4109
  ta   nll=3.9802  ppl=53.53  bpb=0.5781
  kk   nll=2.7792  ppl=16.11  bpb=0.9359
  tl   nll=2.6655  ppl=14.37  bpb=1.2647
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 3400  01:27 (87.2s)
full_cpt step  3410/12852 ( 26.5%)  nll=3.2276  ppl=25.22  grad=0.891  lr=1.72e-05  6858 tok/s  12.64s/it  [11:58:20<33:09:00]  mem=34.07GB
full_cpt step  3420/12852 ( 26.6%)  nll=3.3239  ppl=27.77  grad=0.879  lr=1.72e-05  6860 tok/s  12.64s/it  [12:00:16<33:06:25]  mem=34.07GB
full_cpt step  3430/12852 ( 26.7%)  nll=3.3931  ppl=29.76  grad=0.941  lr=1.72e-05  6861 tok/s  12.63s/it  [12:02:12<33:03:51]  mem=34.07GB
full_cpt step  3440/12852 ( 26.8%)  nll=3.3246  ppl=27.79  grad=0.863  lr=1.72e-05  6863 tok/s  12.63s/it  [12:04:08<33:01:16]  mem=34.07GB
full_cpt step  3450/12852 ( 26.8%)  nll=3.2674  ppl=26.24  grad=0.871  lr=1.71e-05  6864 tok/s  12.63s/it  [12:06:04<32:58:42]  mem=34.07GB
full_cpt step  3460/12852 ( 26.9%)  nll=3.3705  ppl=29.09  grad=0.820  lr=1.71e-05  6865 tok/s  12.62s/it  [12:08:00<32:56:08]  mem=34.07GB
full_cpt step  3470/12852 ( 27.0%)  nll=3.3075  ppl=27.32  grad=0.844  lr=1.71e-05  6867 tok/s  12.62s/it  [12:09:56<32:53:34]  mem=34.07GB
full_cpt step  3480/12852 ( 27.1%)  nll=3.3699  ppl=29.07  grad=0.855  lr=1.71e-05  6869 tok/s  12.62s/it  [12:11:52<32:51:01]  mem=34.07GB
full_cpt step  3490/12852 ( 27.2%)  nll=3.4049  ppl=30.11  grad=0.910  lr=1.71e-05  6870 tok/s  12.62s/it  [12:13:48<32:48:27]  mem=34.07GB
full_cpt step  3500/12852 ( 27.2%)  nll=3.2355  ppl=25.42  grad=0.844  lr=1.71e-05  6872 tok/s  12.61s/it  [12:15:45<32:45:54]  mem=34.07GB
  am   nll=4.0086  ppl=55.07  bpb=1.4096
  ta   nll=3.9767  ppl=53.34  bpb=0.5776
  kk   nll=2.7766  ppl=16.06  bpb=0.9350
  tl   nll=2.6644  ppl=14.36  bpb=1.2642
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 3500  01:27 (86.7s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.38s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.38s/it]
removing old checkpoint checkpoint-2500
kept last 2 resume checkpoints: checkpoint-3000 checkpoint-3500
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-3500 at step 3500 next_row=448000
full_cpt step  3510/12852 ( 27.3%)  nll=3.1593  ppl=23.55  grad=0.871  lr=1.70e-05  6848 tok/s  12.66s/it  [12:20:25<32:50:38]  mem=34.07GB
full_cpt step  3520/12852 ( 27.4%)  nll=3.2587  ppl=26.01  grad=0.906  lr=1.70e-05  6849 tok/s  12.65s/it  [12:22:21<32:48:04]  mem=34.07GB
full_cpt step  3530/12852 ( 27.5%)  nll=3.3581  ppl=28.73  grad=0.941  lr=1.70e-05  6850 tok/s  12.65s/it  [12:24:17<32:45:29]  mem=34.07GB
full_cpt step  3540/12852 ( 27.5%)  nll=3.1733  ppl=23.89  grad=0.930  lr=1.70e-05  6852 tok/s  12.65s/it  [12:26:13<32:42:55]  mem=34.07GB
full_cpt step  3550/12852 ( 27.6%)  nll=3.3385  ppl=28.18  grad=0.867  lr=1.70e-05  6853 tok/s  12.64s/it  [12:28:09<32:40:21]  mem=34.07GB
full_cpt step  3560/12852 ( 27.7%)  nll=3.2603  ppl=26.06  grad=0.852  lr=1.70e-05  6855 tok/s  12.64s/it  [12:30:05<32:37:48]  mem=34.07GB
full_cpt step  3570/12852 ( 27.8%)  nll=3.3034  ppl=27.20  grad=0.910  lr=1.69e-05  6856 tok/s  12.64s/it  [12:32:01<32:35:14]  mem=34.07GB
full_cpt step  3580/12852 ( 27.9%)  nll=3.3650  ppl=28.93  grad=0.871  lr=1.69e-05  6858 tok/s  12.64s/it  [12:33:57<32:32:41]  mem=34.07GB
full_cpt step  3590/12852 ( 27.9%)  nll=3.1906  ppl=24.30  grad=0.855  lr=1.69e-05  6859 tok/s  12.63s/it  [12:35:53<32:30:08]  mem=34.07GB
full_cpt step  3600/12852 ( 28.0%)  nll=3.1864  ppl=24.20  grad=0.871  lr=1.69e-05  6860 tok/s  12.63s/it  [12:37:49<32:27:35]  mem=34.07GB
  am   nll=4.0069  ppl=54.98  bpb=1.4090
  ta   nll=3.9750  ppl=53.25  bpb=0.5773
  kk   nll=2.7747  ppl=16.03  bpb=0.9344
  tl   nll=2.6645  ppl=14.36  bpb=1.2642
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 3600  01:27 (87.3s)
full_cpt step  3610/12852 ( 28.1%)  nll=3.4219  ppl=30.63  grad=0.918  lr=1.69e-05  6849 tok/s  12.65s/it  [12:41:12<32:28:46]  mem=34.07GB
full_cpt step  3620/12852 ( 28.2%)  nll=3.3176  ppl=27.59  grad=0.879  lr=1.69e-05  6850 tok/s  12.65s/it  [12:43:08<32:26:13]  mem=34.07GB
full_cpt step  3630/12852 ( 28.2%)  nll=3.2296  ppl=25.27  grad=0.879  lr=1.68e-05  6851 tok/s  12.65s/it  [12:45:04<32:23:40]  mem=34.07GB
full_cpt step  3640/12852 ( 28.3%)  nll=3.4400  ppl=31.19  grad=0.855  lr=1.68e-05  6853 tok/s  12.64s/it  [12:47:00<32:21:07]  mem=34.07GB
full_cpt step  3650/12852 ( 28.4%)  nll=3.3133  ppl=27.48  grad=0.875  lr=1.68e-05  6855 tok/s  12.64s/it  [12:48:57<32:18:35]  mem=34.07GB
full_cpt step  3660/12852 ( 28.5%)  nll=3.3801  ppl=29.37  grad=0.859  lr=1.68e-05  6856 tok/s  12.64s/it  [12:50:53<32:16:02]  mem=34.07GB
full_cpt step  3670/12852 ( 28.6%)  nll=3.4741  ppl=32.27  grad=0.922  lr=1.68e-05  6857 tok/s  12.63s/it  [12:52:49<32:13:30]  mem=34.07GB
full_cpt step  3680/12852 ( 28.6%)  nll=3.3065  ppl=27.29  grad=0.863  lr=1.68e-05  6859 tok/s  12.63s/it  [12:54:45<32:10:58]  mem=34.07GB
full_cpt step  3690/12852 ( 28.7%)  nll=3.3576  ppl=28.72  grad=0.809  lr=1.67e-05  6861 tok/s  12.63s/it  [12:56:41<32:08:27]  mem=34.07GB
full_cpt step  3700/12852 ( 28.8%)  nll=3.3239  ppl=27.77  grad=0.855  lr=1.67e-05  6863 tok/s  12.63s/it  [12:58:37<32:05:55]  mem=34.07GB
  am   nll=4.0026  ppl=54.74  bpb=1.4075
  ta   nll=3.9726  ppl=53.12  bpb=0.5770
  kk   nll=2.7729  ppl=16.00  bpb=0.9337
  tl   nll=2.6637  ppl=14.35  bpb=1.2639
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 3700  01:27 (87.1s)
full_cpt step  3710/12852 ( 28.9%)  nll=3.2854  ppl=26.72  grad=0.879  lr=1.67e-05  6852 tok/s  12.65s/it  [13:02:00<32:06:59]  mem=34.07GB
full_cpt step  3720/12852 ( 28.9%)  nll=3.3596  ppl=28.78  grad=0.922  lr=1.67e-05  6854 tok/s  12.64s/it  [13:03:56<32:04:27]  mem=34.07GB
full_cpt step  3730/12852 ( 29.0%)  nll=3.3704  ppl=29.09  grad=0.879  lr=1.67e-05  6855 tok/s  12.64s/it  [13:05:52<32:01:55]  mem=34.07GB
full_cpt step  3740/12852 ( 29.1%)  nll=3.3628  ppl=28.87  grad=0.824  lr=1.67e-05  6856 tok/s  12.64s/it  [13:07:49<31:59:23]  mem=34.07GB
full_cpt step  3750/12852 ( 29.2%)  nll=3.3194  ppl=27.64  grad=0.867  lr=1.66e-05  6858 tok/s  12.64s/it  [13:09:45<31:56:52]  mem=34.07GB
full_cpt step  3760/12852 ( 29.3%)  nll=3.3252  ppl=27.80  grad=0.891  lr=1.66e-05  6860 tok/s  12.63s/it  [13:11:41<31:54:21]  mem=34.07GB
full_cpt step  3770/12852 ( 29.3%)  nll=3.2963  ppl=27.01  grad=0.867  lr=1.66e-05  6861 tok/s  12.63s/it  [13:13:37<31:51:50]  mem=34.07GB
full_cpt step  3780/12852 ( 29.4%)  nll=3.4094  ppl=30.25  grad=0.930  lr=1.66e-05  6863 tok/s  12.63s/it  [13:15:33<31:49:20]  mem=34.07GB
full_cpt step  3790/12852 ( 29.5%)  nll=3.3182  ppl=27.61  grad=0.875  lr=1.66e-05  6864 tok/s  12.63s/it  [13:17:29<31:46:49]  mem=34.07GB
full_cpt step  3800/12852 ( 29.6%)  nll=3.3543  ppl=28.62  grad=0.898  lr=1.65e-05  6865 tok/s  12.62s/it  [13:19:25<31:44:18]  mem=34.07GB
  am   nll=4.0004  ppl=54.62  bpb=1.4067
  ta   nll=3.9692  ppl=52.94  bpb=0.5765
  kk   nll=2.7712  ppl=15.98  bpb=0.9332
  tl   nll=2.6634  ppl=14.35  bpb=1.2637
  en   nll=2.8179  ppl=16.74  bpb=0.8774
eval full_cpt step 3800  01:27 (87.0s)
full_cpt step  3810/12852 ( 29.6%)  nll=3.3173  ppl=27.58  grad=0.879  lr=1.65e-05  6854 tok/s  12.64s/it  [13:22:48<31:45:14]  mem=34.07GB
full_cpt step  3820/12852 ( 29.7%)  nll=3.3041  ppl=27.22  grad=0.930  lr=1.65e-05  6855 tok/s  12.64s/it  [13:24:44<31:42:43]  mem=34.07GB
full_cpt step  3830/12852 ( 29.8%)  nll=3.3351  ppl=28.08  grad=0.891  lr=1.65e-05  6857 tok/s  12.64s/it  [13:26:40<31:40:13]  mem=34.07GB
full_cpt step  3840/12852 ( 29.9%)  nll=3.3208  ppl=27.68  grad=0.945  lr=1.65e-05  6858 tok/s  12.63s/it  [13:28:37<31:37:42]  mem=34.07GB
full_cpt step  3850/12852 ( 30.0%)  nll=3.3384  ppl=28.18  grad=0.914  lr=1.65e-05  6859 tok/s  12.63s/it  [13:30:33<31:35:12]  mem=34.07GB
full_cpt step  3860/12852 ( 30.0%)  nll=3.2677  ppl=26.25  grad=0.906  lr=1.64e-05  6861 tok/s  12.63s/it  [13:32:29<31:32:42]  mem=34.07GB
full_cpt step  3870/12852 ( 30.1%)  nll=3.2883  ppl=26.80  grad=0.949  lr=1.64e-05  6863 tok/s  12.63s/it  [13:34:25<31:30:12]  mem=34.07GB
full_cpt step  3880/12852 ( 30.2%)  nll=3.3466  ppl=28.41  grad=0.859  lr=1.64e-05  6864 tok/s  12.62s/it  [13:36:21<31:27:42]  mem=34.07GB
full_cpt step  3890/12852 ( 30.3%)  nll=3.3232  ppl=27.75  grad=0.863  lr=1.64e-05  6866 tok/s  12.62s/it  [13:38:17<31:25:13]  mem=34.07GB
full_cpt step  3900/12852 ( 30.3%)  nll=3.2203  ppl=25.04  grad=0.809  lr=1.64e-05  6867 tok/s  12.62s/it  [13:40:13<31:22:43]  mem=34.07GB
  am   nll=3.9965  ppl=54.41  bpb=1.4054
  ta   nll=3.9671  ppl=52.83  bpb=0.5762
  kk   nll=2.7691  ppl=15.94  bpb=0.9325
  tl   nll=2.6629  ppl=14.34  bpb=1.2635
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 3900  01:26 (86.4s)
full_cpt step  3910/12852 ( 30.4%)  nll=3.3329  ppl=28.02  grad=0.898  lr=1.64e-05  6856 tok/s  12.64s/it  [13:43:36<31:23:31]  mem=34.07GB
full_cpt step  3920/12852 ( 30.5%)  nll=3.3226  ppl=27.73  grad=0.902  lr=1.63e-05  6858 tok/s  12.64s/it  [13:45:32<31:21:02]  mem=34.07GB
full_cpt step  3930/12852 ( 30.6%)  nll=3.3971  ppl=29.88  grad=0.871  lr=1.63e-05  6860 tok/s  12.63s/it  [13:47:28<31:18:32]  mem=34.07GB
full_cpt step  3940/12852 ( 30.7%)  nll=3.4773  ppl=32.37  grad=0.895  lr=1.63e-05  6861 tok/s  12.63s/it  [13:49:24<31:16:03]  mem=34.07GB
full_cpt step  3950/12852 ( 30.7%)  nll=3.3994  ppl=29.95  grad=0.879  lr=1.63e-05  6863 tok/s  12.63s/it  [13:51:20<31:13:33]  mem=34.07GB
full_cpt step  3960/12852 ( 30.8%)  nll=3.2809  ppl=26.60  grad=0.875  lr=1.63e-05  6864 tok/s  12.63s/it  [13:53:16<31:11:04]  mem=34.07GB
full_cpt step  3970/12852 ( 30.9%)  nll=3.2901  ppl=26.85  grad=0.867  lr=1.62e-05  6865 tok/s  12.62s/it  [13:55:12<31:08:35]  mem=34.07GB
full_cpt step  3980/12852 ( 31.0%)  nll=3.1977  ppl=24.48  grad=0.887  lr=1.62e-05  6867 tok/s  12.62s/it  [13:57:08<31:06:07]  mem=34.07GB
full_cpt step  3990/12852 ( 31.0%)  nll=3.2513  ppl=25.82  grad=0.875  lr=1.62e-05  6868 tok/s  12.62s/it  [13:59:04<31:03:38]  mem=34.07GB
full_cpt step  4000/12852 ( 31.1%)  nll=3.2619  ppl=26.10  grad=0.883  lr=1.62e-05  6870 tok/s  12.62s/it  [14:01:01<31:01:09]  mem=34.07GB
  am   nll=3.9948  ppl=54.32  bpb=1.4048
  ta   nll=3.9647  ppl=52.70  bpb=0.5758
  kk   nll=2.7678  ppl=15.92  bpb=0.9321
  tl   nll=2.6626  ppl=14.33  bpb=1.2633
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 4000  01:27 (87.1s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.38s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.38s/it]
removing old checkpoint checkpoint-3000
kept last 2 resume checkpoints: checkpoint-3500 checkpoint-4000
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-4000 at step 4000 next_row=512000

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.66s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.66s/it]
wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-4000 (weights+expanded tokenizer, no optimizer)
full_cpt step  4010/12852 ( 31.2%)  nll=3.3249  ppl=27.79  grad=0.926  lr=1.62e-05  6848 tok/s  12.65s/it  [14:05:46<31:04:54]  mem=34.07GB
full_cpt step  4020/12852 ( 31.3%)  nll=3.4192  ppl=30.55  grad=1.094  lr=1.62e-05  6849 tok/s  12.65s/it  [14:07:42<31:02:24]  mem=34.07GB
full_cpt step  4030/12852 ( 31.4%)  nll=3.2768  ppl=26.49  grad=0.879  lr=1.61e-05  6851 tok/s  12.65s/it  [14:09:38<30:59:55]  mem=34.07GB
full_cpt step  4040/12852 ( 31.4%)  nll=3.2381  ppl=25.49  grad=0.859  lr=1.61e-05  6852 tok/s  12.65s/it  [14:11:34<30:57:25]  mem=34.07GB
full_cpt step  4050/12852 ( 31.5%)  nll=3.2846  ppl=26.70  grad=0.902  lr=1.61e-05  6853 tok/s  12.64s/it  [14:13:30<30:54:56]  mem=34.07GB
full_cpt step  4060/12852 ( 31.6%)  nll=3.4232  ppl=30.67  grad=0.953  lr=1.61e-05  6855 tok/s  12.64s/it  [14:15:26<30:52:27]  mem=34.07GB
full_cpt step  4070/12852 ( 31.7%)  nll=3.3701  ppl=29.08  grad=0.848  lr=1.61e-05  6856 tok/s  12.64s/it  [14:17:22<30:49:59]  mem=34.07GB
full_cpt step  4080/12852 ( 31.7%)  nll=3.5206  ppl=33.80  grad=0.969  lr=1.60e-05  6858 tok/s  12.64s/it  [14:19:18<30:47:30]  mem=34.07GB
full_cpt step  4090/12852 ( 31.8%)  nll=3.3677  ppl=29.01  grad=0.938  lr=1.60e-05  6859 tok/s  12.63s/it  [14:21:14<30:45:02]  mem=34.07GB
full_cpt step  4100/12852 ( 31.9%)  nll=3.3089  ppl=27.36  grad=0.898  lr=1.60e-05  6861 tok/s  12.63s/it  [14:23:10<30:42:34]  mem=34.07GB
  am   nll=3.9919  ppl=54.16  bpb=1.4038
  ta   nll=3.9636  ppl=52.64  bpb=0.5757
  kk   nll=2.7659  ppl=15.89  bpb=0.9314
  tl   nll=2.6623  ppl=14.33  bpb=1.2632
  en   nll=2.8175  ppl=16.74  bpb=0.8773
eval full_cpt step 4100  01:27 (86.9s)
full_cpt step  4110/12852 ( 32.0%)  nll=3.3334  ppl=28.03  grad=0.941  lr=1.60e-05  6851 tok/s  12.65s/it  [14:26:33<30:43:10]  mem=34.07GB
full_cpt step  4120/12852 ( 32.1%)  nll=3.2013  ppl=24.56  grad=0.895  lr=1.60e-05  6852 tok/s  12.65s/it  [14:28:30<30:40:42]  mem=34.07GB
full_cpt step  4130/12852 ( 32.1%)  nll=3.3952  ppl=29.82  grad=0.926  lr=1.60e-05  6854 tok/s  12.65s/it  [14:30:26<30:38:14]  mem=34.07GB
full_cpt step  4140/12852 ( 32.2%)  nll=3.3306  ppl=27.96  grad=0.914  lr=1.59e-05  6855 tok/s  12.64s/it  [14:32:22<30:35:45]  mem=34.07GB
full_cpt step  4150/12852 ( 32.3%)  nll=3.2908  ppl=26.86  grad=0.836  lr=1.59e-05  6856 tok/s  12.64s/it  [14:34:18<30:33:17]  mem=34.07GB
full_cpt step  4160/12852 ( 32.4%)  nll=3.2927  ppl=26.91  grad=0.930  lr=1.59e-05  6857 tok/s  12.64s/it  [14:36:14<30:30:49]  mem=34.07GB
full_cpt step  4170/12852 ( 32.4%)  nll=3.1692  ppl=23.79  grad=0.852  lr=1.59e-05  6859 tok/s  12.64s/it  [14:38:10<30:28:21]  mem=34.07GB
full_cpt step  4180/12852 ( 32.5%)  nll=3.2696  ppl=26.30  grad=0.930  lr=1.59e-05  6860 tok/s  12.63s/it  [14:40:06<30:25:53]  mem=34.07GB
full_cpt step  4190/12852 ( 32.6%)  nll=3.3850  ppl=29.52  grad=0.867  lr=1.58e-05  6861 tok/s  12.63s/it  [14:42:02<30:23:26]  mem=34.07GB
full_cpt step  4200/12852 ( 32.7%)  nll=3.2205  ppl=25.04  grad=0.875  lr=1.58e-05  6862 tok/s  12.63s/it  [14:43:58<30:20:58]  mem=34.07GB
  am   nll=3.9900  ppl=54.05  bpb=1.4031
  ta   nll=3.9612  ppl=52.52  bpb=0.5753
  kk   nll=2.7642  ppl=15.87  bpb=0.9308
  tl   nll=2.6623  ppl=14.33  bpb=1.2632
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 4200  01:27 (86.7s)
full_cpt step  4210/12852 ( 32.8%)  nll=3.3693  ppl=29.06  grad=0.859  lr=1.58e-05  6852 tok/s  12.65s/it  [14:47:21<30:21:29]  mem=34.07GB
full_cpt step  4220/12852 ( 32.8%)  nll=3.3485  ppl=28.46  grad=0.879  lr=1.58e-05  6853 tok/s  12.64s/it  [14:49:17<30:19:01]  mem=34.07GB
full_cpt step  4230/12852 ( 32.9%)  nll=3.3073  ppl=27.31  grad=0.828  lr=1.58e-05  6855 tok/s  12.64s/it  [14:51:13<30:16:34]  mem=34.07GB
full_cpt step  4240/12852 ( 33.0%)  nll=3.2101  ppl=24.78  grad=0.910  lr=1.57e-05  6856 tok/s  12.64s/it  [14:53:09<30:14:07]  mem=34.07GB
full_cpt step  4250/12852 ( 33.1%)  nll=3.4040  ppl=30.08  grad=0.879  lr=1.57e-05  6858 tok/s  12.64s/it  [14:55:05<30:11:40]  mem=34.07GB
full_cpt step  4260/12852 ( 33.1%)  nll=3.4427  ppl=31.27  grad=0.867  lr=1.57e-05  6859 tok/s  12.63s/it  [14:57:01<30:09:12]  mem=34.07GB
full_cpt step  4270/12852 ( 33.2%)  nll=3.2428  ppl=25.61  grad=0.906  lr=1.57e-05  6860 tok/s  12.63s/it  [14:58:57<30:06:45]  mem=34.07GB
logging stdout/stderr to research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/train.log
seed=42  flash SDP  TF32
using measured preliminary seq=1024 layout batch=16 accum=8; CPT lock remains seq=2048
train layout batch=16 accum=8 (128 seq/step, 131072 tok/step) checkpointing=off liger=True
loading tokenizer research/results/embedding_init/gemma_mapping_byteify
full token census over local corpus
using full token census from /workspace/ugrip-kd/research/scripts/embedding_initialization/results/corpus_token_counts.json (cached tokenizer /workspace/ugrip-kd/research/results/expansion/tokenizer; current research/results/embedding_init/gemma_mapping_byteify)
  am census: 476898623 tokens (1959.7 tok/doc, 243349 docs)
  ta census: 4369273220 tokens (924.0 tok/doc, 4728460 docs)
  kk census: 4464108618 tokens (1632.8 tok/doc, 2733982 docs)
  tl census: 306736374 tokens (880.3 tok/doc, 348453 docs)
  en census: 1542145528 tokens (771.1 tok/doc, 2000000 docs)
splits
  am: full 243349  eval 256  pool 243093
  ta: full 4728460  eval 256  pool 4728204
  kk: full 2733982  eval 256  pool 2733726
  tl: full 348453  eval 256  pool 348197
  en: full 2000000  eval 256  pool 1999744
mix weights {'am': 0.0446, 'ta': 0.4089, 'kk': 0.4178, 'tl': 0.0287, 'en': 0.1} (corpus-proportional; no document repeat)
lang token budgets {'am': 49803264, 'ta': 456292352, 'kk': 466195456, 'tl': 32032768, 'en': 111591424}
packing via HuggingFace datasets.map (batched, num_proc; not cached)
    am: datasets.map n=35332 batched=1024 num_proc=8

hf-map am (num_proc=8):   0%|          | 0/35332 [00:00<?, ? examples/s]
hf-map am (num_proc=8):   3%|2         | 1024/35332 [00:06<03:32, 161.71 examples/s]
hf-map am (num_proc=8):   6%|5         | 2048/35332 [00:06<01:31, 364.95 examples/s]
hf-map am (num_proc=8):  14%|#4        | 5120/35332 [00:06<00:24, 1244.58 examples/s]
hf-map am (num_proc=8):  20%|##        | 7168/35332 [00:06<00:14, 1930.95 examples/s]
hf-map am (num_proc=8):  26%|##6       | 9216/35332 [00:10<00:24, 1053.42 examples/s]
hf-map am (num_proc=8):  32%|###1      | 11264/35332 [00:10<00:16, 1464.56 examples/s]
hf-map am (num_proc=8):  43%|####3     | 15360/35332 [00:11<00:07, 2688.24 examples/s]
hf-map am (num_proc=8):  49%|####9     | 17408/35332 [00:14<00:12, 1474.63 examples/s]
hf-map am (num_proc=8):  55%|#####5    | 19456/35332 [00:14<00:08, 1956.44 examples/s]
hf-map am (num_proc=8):  61%|######    | 21504/35332 [00:14<00:05, 2451.33 examples/s]
hf-map am (num_proc=8):  70%|######9   | 24576/35332 [00:14<00:02, 3663.52 examples/s]
hf-map am (num_proc=8):  75%|#######5  | 26624/35332 [00:17<00:04, 1804.26 examples/s]
hf-map am (num_proc=8):  78%|#######8  | 27648/35332 [00:17<00:03, 2047.79 examples/s]
hf-map am (num_proc=8):  84%|########4 | 29696/35332 [00:17<00:02, 2657.47 examples/s]
hf-map am (num_proc=8):  90%|########9 | 31744/35332 [00:18<00:01, 3105.52 examples/s]
hf-map am (num_proc=8):  94%|#########3| 33088/35332 [00:18<00:00, 3121.10 examples/s]
hf-map am (num_proc=8):  95%|#########5| 33728/35332 [00:18<00:00, 3326.03 examples/s]
hf-map am (num_proc=8):  97%|#########7| 34370/35332 [00:19<00:00, 3293.26 examples/s]
hf-map am (num_proc=8):  99%|#########9| 35012/35332 [00:19<00:00, 2574.26 examples/s]
hf-map am (num_proc=8): 100%|##########| 35332/35332 [00:19<00:00, 1780.96 examples/s]
    am: datasets.map num_proc=8 done 20s
    am: 25000 docs  48394736 tokens  25s
  packed am: 49804449 tokens (1.000 of budget)  rows=62355 docs=25725  pad-per-doc (HF map)
    ta: datasets.map n=667659 batched=1024 num_proc=8

hf-map ta (num_proc=8):   0%|          | 0/667659 [00:00<?, ? examples/s]
hf-map ta (num_proc=8):   0%|          | 1024/667659 [00:07<1:25:15, 130.31 examples/s]
hf-map ta (num_proc=8):   0%|          | 3072/667659 [00:08<22:40, 488.46 examples/s]  
hf-map ta (num_proc=8):   1%|          | 4096/667659 [00:08<15:17, 723.38 examples/s]
hf-map ta (num_proc=8):   1%|          | 5120/667659 [00:08<10:55, 1009.99 examples/s]
hf-map ta (num_proc=8):   1%|1         | 8192/667659 [00:08<04:57, 2219.47 examples/s]
hf-map ta (num_proc=8):   1%|1         | 9216/667659 [00:12<13:31, 811.72 examples/s] 
hf-map ta (num_proc=8):   2%|1         | 10240/667659 [00:13<11:33, 948.13 examples/s]
hf-map ta (num_proc=8):   2%|1         | 12288/667659 [00:13<07:30, 1454.00 examples/s]
hf-map ta (num_proc=8):   2%|1         | 13312/667659 [00:13<06:28, 1686.45 examples/s]
hf-map ta (num_proc=8):   2%|2         | 14336/667659 [00:13<05:12, 2088.40 examples/s]
hf-map ta (num_proc=8):   2%|2         | 16384/667659 [00:14<03:19, 3256.84 examples/s]
hf-map ta (num_proc=8):   3%|2         | 17408/667659 [00:17<11:08, 973.30 examples/s] 
hf-map ta (num_proc=8):   3%|2         | 18432/667659 [00:17<09:22, 1155.02 examples/s]
hf-map ta (num_proc=8):   3%|2         | 19456/667659 [00:18<08:18, 1300.04 examples/s]
hf-map ta (num_proc=8):   3%|3         | 20480/667659 [00:18<06:35, 1638.14 examples/s]
hf-map ta (num_proc=8):   3%|3         | 21504/667659 [00:19<05:52, 1834.15 examples/s]
hf-map ta (num_proc=8):   4%|3         | 24576/667659 [00:19<03:08, 3419.41 examples/s]
hf-map ta (num_proc=8):   4%|3         | 25600/667659 [00:22<09:43, 1101.08 examples/s]
hf-map ta (num_proc=8):   4%|3         | 26624/667659 [00:23<09:04, 1177.97 examples/s]
hf-map ta (num_proc=8):   4%|4         | 27648/667659 [00:23<07:42, 1382.66 examples/s]
hf-map ta (num_proc=8):   4%|4         | 28672/667659 [00:24<06:39, 1600.28 examples/s]
hf-map ta (num_proc=8):   4%|4         | 29696/667659 [00:24<05:29, 1935.33 examples/s]
hf-map ta (num_proc=8):   5%|4         | 31744/667659 [00:24<03:28, 3047.60 examples/s]
hf-map ta (num_proc=8):   5%|4         | 32768/667659 [00:24<02:57, 3584.55 examples/s]
hf-map ta (num_proc=8):   5%|5         | 33792/667659 [00:27<10:08, 1041.12 examples/s]
hf-map ta (num_proc=8):   5%|5         | 34816/667659 [00:28<10:02, 1050.70 examples/s]
hf-map ta (num_proc=8):   5%|5         | 35840/667659 [00:28<07:33, 1394.29 examples/s]
hf-map ta (num_proc=8):   6%|5         | 36864/667659 [00:28<06:08, 1713.30 examples/s]
hf-map ta (num_proc=8):   6%|5         | 37888/667659 [00:29<04:49, 2172.10 examples/s]
hf-map ta (num_proc=8):   6%|5         | 38912/667659 [00:29<05:05, 2054.81 examples/s]
hf-map ta (num_proc=8):   6%|6         | 40960/667659 [00:29<03:04, 3392.48 examples/s]
hf-map ta (num_proc=8):   6%|6         | 41984/667659 [00:32<08:52, 1174.66 examples/s]
hf-map ta (num_proc=8):   6%|6         | 43008/667659 [00:33<10:05, 1032.02 examples/s]
hf-map ta (num_proc=8):   7%|6         | 45056/667659 [00:33<06:02, 1718.59 examples/s]
hf-map ta (num_proc=8):   7%|6         | 46080/667659 [00:34<05:22, 1926.19 examples/s]
hf-map ta (num_proc=8):   7%|7         | 47104/667659 [00:34<05:19, 1943.58 examples/s]
hf-map ta (num_proc=8):   7%|7         | 48128/667659 [00:34<04:43, 2186.30 examples/s]
hf-map ta (num_proc=8):   8%|7         | 50176/667659 [00:37<07:21, 1399.58 examples/s]
hf-map ta (num_proc=8):   8%|7         | 51200/667659 [00:38<08:35, 1194.85 examples/s]
hf-map ta (num_proc=8):   8%|7         | 52224/667659 [00:38<06:47, 1509.42 examples/s]
hf-map ta (num_proc=8):   8%|7         | 53248/667659 [00:38<05:22, 1902.84 examples/s]
hf-map ta (num_proc=8):   8%|8         | 54272/667659 [00:38<04:37, 2206.48 examples/s]
hf-map ta (num_proc=8):   8%|8         | 55296/667659 [00:39<04:52, 2092.04 examples/s]
hf-map ta (num_proc=8):   8%|8         | 56320/667659 [00:39<04:04, 2497.15 examples/s]
hf-map ta (num_proc=8):   9%|8         | 58368/667659 [00:41<06:09, 1650.95 examples/s]
hf-map ta (num_proc=8):   9%|8         | 59392/667659 [00:42<07:57, 1274.16 examples/s]
hf-map ta (num_proc=8):   9%|9         | 61440/667659 [00:43<05:24, 1866.88 examples/s]
hf-map ta (num_proc=8):   9%|9         | 62464/667659 [00:43<05:22, 1878.76 examples/s]
hf-map ta (num_proc=8):  10%|9         | 63488/667659 [00:43<04:21, 2306.69 examples/s]
hf-map ta (num_proc=8):  10%|9         | 65536/667659 [00:44<03:10, 3159.93 examples/s]
hf-map ta (num_proc=8):  10%|9         | 66560/667659 [00:45<05:43, 1748.37 examples/s]
hf-map ta (num_proc=8):  10%|#         | 67584/667659 [00:46<06:35, 1517.54 examples/s]
hf-map ta (num_proc=8):  10%|#         | 68608/667659 [00:46<05:18, 1882.12 examples/s]
hf-map ta (num_proc=8):  10%|#         | 69632/667659 [00:47<05:11, 1917.03 examples/s]
hf-map ta (num_proc=8):  11%|#         | 70656/667659 [00:47<05:14, 1899.26 examples/s]
hf-map ta (num_proc=8):  11%|#         | 71680/667659 [00:47<04:12, 2361.11 examples/s]
hf-map ta (num_proc=8):  11%|#         | 72704/667659 [00:48<03:45, 2636.80 examples/s]
hf-map ta (num_proc=8):  11%|#1        | 73728/667659 [00:48<03:45, 2638.38 examples/s]
hf-map ta (num_proc=8):  11%|#1        | 74752/667659 [00:50<06:39, 1485.69 examples/s]
hf-map ta (num_proc=8):  11%|#1        | 75776/667659 [00:50<07:12, 1367.41 examples/s]
hf-map ta (num_proc=8):  12%|#1        | 76800/667659 [00:51<05:34, 1768.38 examples/s]
hf-map ta (num_proc=8):  12%|#1        | 77824/667659 [00:51<05:42, 1723.20 examples/s]
hf-map ta (num_proc=8):  12%|#1        | 78848/667659 [00:51<04:37, 2119.67 examples/s]
hf-map ta (num_proc=8):  12%|#1        | 79872/667659 [00:52<04:16, 2294.58 examples/s]
hf-map ta (num_proc=8):  12%|#2        | 80896/667659 [00:52<04:15, 2295.98 examples/s]
hf-map ta (num_proc=8):  12%|#2        | 81920/667659 [00:53<03:38, 2675.98 examples/s]
hf-map ta (num_proc=8):  12%|#2        | 82944/667659 [00:53<05:18, 1836.10 examples/s]
hf-map ta (num_proc=8):  13%|#2        | 83968/667659 [00:55<07:18, 1331.16 examples/s]
hf-map ta (num_proc=8):  13%|#2        | 86016/667659 [00:55<05:19, 1820.54 examples/s]
hf-map ta (num_proc=8):  13%|#3        | 87040/667659 [00:56<05:02, 1917.91 examples/s]
hf-map ta (num_proc=8):  13%|#3        | 88064/667659 [00:56<04:39, 2071.43 examples/s]
hf-map ta (num_proc=8):  13%|#3        | 89088/667659 [00:57<04:19, 2231.25 examples/s]
hf-map ta (num_proc=8):  13%|#3        | 90112/667659 [00:57<03:55, 2456.83 examples/s]
hf-map ta (num_proc=8):  14%|#3        | 91136/667659 [00:57<03:25, 2810.83 examples/s]
hf-map ta (num_proc=8):  14%|#3        | 92160/667659 [00:58<05:04, 1892.78 examples/s]
hf-map ta (num_proc=8):  14%|#3        | 93184/667659 [00:59<05:07, 1866.94 examples/s]
hf-map ta (num_proc=8):  14%|#4        | 94208/667659 [00:59<05:05, 1878.62 examples/s]
hf-map ta (num_proc=8):  14%|#4        | 95232/667659 [01:00<05:09, 1848.70 examples/s]
hf-map ta (num_proc=8):  14%|#4        | 96256/667659 [01:00<04:14, 2242.27 examples/s]
hf-map ta (num_proc=8):  15%|#4        | 97280/667659 [01:01<04:24, 2158.00 examples/s]
hf-map ta (num_proc=8):  15%|#4        | 98304/667659 [01:01<04:52, 1949.34 examples/s]
hf-map ta (num_proc=8):  15%|#4        | 99328/667659 [01:01<04:04, 2321.59 examples/s]
hf-map ta (num_proc=8):  15%|#5        | 100352/667659 [01:03<06:22, 1484.46 examples/s]
hf-map ta (num_proc=8):  15%|#5        | 101376/667659 [01:03<06:03, 1558.56 examples/s]
hf-map ta (num_proc=8):  15%|#5        | 102400/667659 [01:04<05:16, 1787.01 examples/s]
hf-map ta (num_proc=8):  15%|#5        | 103424/667659 [01:04<05:31, 1701.30 examples/s]
hf-map ta (num_proc=8):  16%|#5        | 104448/667659 [01:05<04:48, 1954.74 examples/s]
hf-map ta (num_proc=8):  16%|#5        | 105472/667659 [01:05<03:51, 2432.96 examples/s]
hf-map ta (num_proc=8):  16%|#5        | 106496/667659 [01:05<03:27, 2704.25 examples/s]
hf-map ta (num_proc=8):  16%|#6        | 107520/667659 [01:06<03:49, 2436.60 examples/s]
hf-map ta (num_proc=8):  16%|#6        | 108544/667659 [01:06<03:56, 2362.46 examples/s]
hf-map ta (num_proc=8):  16%|#6        | 109568/667659 [01:07<04:48, 1935.24 examples/s]
hf-map ta (num_proc=8):  17%|#6        | 110592/667659 [01:07<04:29, 2063.94 examples/s]
hf-map ta (num_proc=8):  17%|#6        | 111616/667659 [01:08<05:32, 1672.12 examples/s]
hf-map ta (num_proc=8):  17%|#6        | 112640/667659 [01:08<04:24, 2100.86 examples/s]
hf-map ta (num_proc=8):  17%|#7        | 114688/667659 [01:09<02:49, 3253.72 examples/s]
hf-map ta (num_proc=8):  17%|#7        | 115712/667659 [01:10<05:15, 1747.97 examples/s]
hf-map ta (num_proc=8):  17%|#7        | 116736/667659 [01:10<04:20, 2116.95 examples/s]
hf-map ta (num_proc=8):  18%|#7        | 117760/667659 [01:11<05:14, 1747.65 examples/s]
hf-map ta (num_proc=8):  18%|#7        | 118784/667659 [01:12<05:05, 1797.06 examples/s]
hf-map ta (num_proc=8):  18%|#7        | 119808/667659 [01:12<05:52, 1556.33 examples/s]
hf-map ta (num_proc=8):  19%|#8        | 123904/667659 [01:14<04:35, 1971.37 examples/s]
hf-map ta (num_proc=8):  19%|#8        | 124928/667659 [01:14<04:09, 2173.59 examples/s]
hf-map ta (num_proc=8):  19%|#8        | 125952/667659 [01:15<05:07, 1760.21 examples/s]
hf-map ta (num_proc=8):  19%|#9        | 126976/667659 [01:16<04:19, 2083.34 examples/s]
hf-map ta (num_proc=8):  19%|#9        | 128000/667659 [01:16<04:29, 2005.32 examples/s]
hf-map ta (num_proc=8):  19%|#9        | 129024/667659 [01:16<03:58, 2259.25 examples/s]
hf-map ta (num_proc=8):  19%|#9        | 130048/667659 [01:17<03:12, 2789.65 examples/s]
hf-map ta (num_proc=8):  20%|#9        | 131072/667659 [01:17<03:40, 2438.52 examples/s]
hf-map ta (num_proc=8):  20%|#9        | 132096/667659 [01:18<05:16, 1691.70 examples/s]
hf-map ta (num_proc=8):  20%|#9        | 133120/667659 [01:19<04:24, 2024.72 examples/s]
hf-map ta (num_proc=8):  20%|##        | 134144/667659 [01:19<05:29, 1618.99 examples/s]
hf-map ta (num_proc=8):  20%|##        | 135168/667659 [01:20<04:21, 2035.84 examples/s]
hf-map ta (num_proc=8):  20%|##        | 136192/667659 [01:20<04:04, 2173.98 examples/s]
hf-map ta (num_proc=8):  21%|##        | 137216/667659 [01:20<03:24, 2589.71 examples/s]
hf-map ta (num_proc=8):  21%|##        | 139264/667659 [01:21<03:10, 2775.84 examples/s]
hf-map ta (num_proc=8):  21%|##1       | 140288/667659 [01:22<04:43, 1858.56 examples/s]
hf-map ta (num_proc=8):  21%|##1       | 141312/667659 [01:22<03:50, 2283.71 examples/s]
hf-map ta (num_proc=8):  21%|##1       | 142336/667659 [01:23<04:00, 2188.63 examples/s]
hf-map ta (num_proc=8):  21%|##1       | 143360/667659 [01:23<04:21, 2004.83 examples/s]
hf-map ta (num_proc=8):  22%|##1       | 144384/667659 [01:24<04:37, 1883.41 examples/s]
hf-map ta (num_proc=8):  22%|##1       | 145408/667659 [01:24<04:31, 1921.81 examples/s]
hf-map ta (num_proc=8):  22%|##2       | 147456/667659 [01:25<02:55, 2971.64 examples/s]
hf-map ta (num_proc=8):  22%|##2       | 148480/667659 [01:26<04:55, 1754.43 examples/s]
hf-map ta (num_proc=8):  22%|##2       | 149504/667659 [01:26<04:03, 2130.31 examples/s]
hf-map ta (num_proc=8):  23%|##2       | 150528/667659 [01:27<04:17, 2007.41 examples/s]
hf-map ta (num_proc=8):  23%|##2       | 151552/667659 [01:27<04:39, 1844.78 examples/s]
hf-map ta (num_proc=8):  23%|##2       | 152576/667659 [01:28<04:48, 1784.56 examples/s]
hf-map ta (num_proc=8):  23%|##3       | 153600/667659 [01:29<04:32, 1887.24 examples/s]
hf-map ta (num_proc=8):  23%|##3       | 155648/667659 [01:29<03:23, 2511.35 examples/s]
hf-map ta (num_proc=8):  23%|##3       | 156672/667659 [01:30<04:36, 1844.79 examples/s]
hf-map ta (num_proc=8):  24%|##3       | 157696/667659 [01:31<04:21, 1946.43 examples/s]
hf-map ta (num_proc=8):  24%|##3       | 159744/667659 [01:31<03:30, 2407.99 examples/s]
hf-map ta (num_proc=8):  24%|##4       | 160768/667659 [01:32<04:05, 2068.30 examples/s]
hf-map ta (num_proc=8):  24%|##4       | 161792/667659 [01:33<04:36, 1828.23 examples/s]
hf-map ta (num_proc=8):  24%|##4       | 162816/667659 [01:33<03:36, 2327.17 examples/s]
hf-map ta (num_proc=8):  25%|##4       | 163840/667659 [01:33<03:08, 2675.04 examples/s]
hf-map ta (num_proc=8):  25%|##4       | 164864/667659 [01:34<03:55, 2131.61 examples/s]
hf-map ta (num_proc=8):  25%|##4       | 165888/667659 [01:34<04:01, 2075.61 examples/s]
hf-map ta (num_proc=8):  25%|##5       | 167936/667659 [01:35<03:57, 2100.09 examples/s]
hf-map ta (num_proc=8):  25%|##5       | 168960/667659 [01:36<04:07, 2013.21 examples/s]
hf-map ta (num_proc=8):  25%|##5       | 169984/667659 [01:36<03:37, 2292.63 examples/s]
hf-map ta (num_proc=8):  26%|##5       | 171008/667659 [01:36<03:21, 2460.87 examples/s]
hf-map ta (num_proc=8):  26%|##5       | 172032/667659 [01:37<03:16, 2517.79 examples/s]
hf-map ta (num_proc=8):  26%|##5       | 173056/667659 [01:37<03:45, 2196.83 examples/s]
hf-map ta (num_proc=8):  26%|##6       | 174080/667659 [01:38<03:57, 2082.11 examples/s]
hf-map ta (num_proc=8):  26%|##6       | 175104/667659 [01:38<03:15, 2524.50 examples/s]
hf-map ta (num_proc=8):  26%|##6       | 176128/667659 [01:39<04:51, 1684.86 examples/s]
hf-map ta (num_proc=8):  27%|##6       | 177152/667659 [01:39<04:13, 1934.76 examples/s]
hf-map ta (num_proc=8):  27%|##6       | 179200/667659 [01:40<02:51, 2850.58 examples/s]
hf-map ta (num_proc=8):  27%|##6       | 180224/667659 [01:40<03:07, 2603.00 examples/s]
hf-map ta (num_proc=8):  27%|##7       | 181248/667659 [01:41<03:14, 2507.09 examples/s]
hf-map ta (num_proc=8):  27%|##7       | 182272/667659 [01:41<03:44, 2161.69 examples/s]
hf-map ta (num_proc=8):  27%|##7       | 183296/667659 [01:42<03:24, 2369.91 examples/s]
hf-map ta (num_proc=8):  28%|##7       | 184320/667659 [01:43<04:11, 1919.63 examples/s]
hf-map ta (num_proc=8):  28%|##7       | 185344/667659 [01:43<03:13, 2491.32 examples/s]
hf-map ta (num_proc=8):  28%|##7       | 186368/667659 [01:43<02:54, 2765.23 examples/s]
hf-map ta (num_proc=8):  28%|##8       | 187392/667659 [01:43<02:25, 3301.03 examples/s]
hf-map ta (num_proc=8):  28%|##8       | 188416/667659 [01:44<03:39, 2186.60 examples/s]
hf-map ta (num_proc=8):  28%|##8       | 189440/667659 [01:44<03:20, 2383.47 examples/s]
hf-map ta (num_proc=8):  29%|##8       | 190464/667659 [01:45<03:27, 2295.26 examples/s]
hf-map ta (num_proc=8):  29%|##8       | 191488/667659 [01:45<03:37, 2186.84 examples/s]
hf-map ta (num_proc=8):  29%|##8       | 192512/667659 [01:46<04:19, 1829.53 examples/s]
hf-map ta (num_proc=8):  29%|##8       | 193536/667659 [01:46<03:56, 2008.28 examples/s]
hf-map ta (num_proc=8):  29%|##9       | 195584/667659 [01:47<02:20, 3350.73 examples/s]
hf-map ta (num_proc=8):  29%|##9       | 196608/667659 [01:48<04:28, 1756.19 examples/s]
hf-map ta (num_proc=8):  30%|##9       | 198656/667659 [01:49<03:37, 2160.19 examples/s]
hf-map ta (num_proc=8):  30%|##9       | 199680/667659 [01:49<03:34, 2183.33 examples/s]
hf-map ta (num_proc=8):  30%|###       | 200704/667659 [01:50<03:44, 2079.79 examples/s]
hf-map ta (num_proc=8):  30%|###       | 201728/667659 [01:50<03:15, 2381.74 examples/s]
hf-map ta (num_proc=8):  30%|###       | 202752/667659 [01:50<02:52, 2692.11 examples/s]
hf-map ta (num_proc=8):  31%|###       | 203776/667659 [01:50<02:29, 3100.19 examples/s]
hf-map ta (num_proc=8):  31%|###       | 204800/667659 [01:51<04:04, 1889.91 examples/s]
hf-map ta (num_proc=8):  31%|###       | 205824/667659 [01:52<03:14, 2371.77 examples/s]
hf-map ta (num_proc=8):  31%|###       | 206848/667659 [01:52<03:21, 2286.64 examples/s]
hf-map ta (num_proc=8):  31%|###1      | 207872/667659 [01:53<03:32, 2158.92 examples/s]
hf-map ta (num_proc=8):  31%|###1      | 208896/667659 [01:53<03:42, 2066.07 examples/s]
hf-map ta (num_proc=8):  31%|###1      | 209920/667659 [01:53<03:03, 2497.41 examples/s]
hf-map ta (num_proc=8):  32%|###1      | 210944/667659 [01:54<02:51, 2662.54 examples/s]
hf-map ta (num_proc=8):  32%|###1      | 211968/667659 [01:54<02:43, 2790.83 examples/s]
hf-map ta (num_proc=8):  32%|###1      | 212992/667659 [01:55<03:28, 2177.51 examples/s]
hf-map ta (num_proc=8):  32%|###2      | 214016/667659 [01:55<03:20, 2268.14 examples/s]
hf-map ta (num_proc=8):  32%|###2      | 215040/667659 [01:55<02:37, 2870.04 examples/s]
hf-map ta (num_proc=8):  32%|###2      | 216064/667659 [01:56<03:10, 2371.85 examples/s]
hf-map ta (num_proc=8):  33%|###2      | 217088/667659 [01:56<03:18, 2269.34 examples/s]
hf-map ta (num_proc=8):  33%|###2      | 219136/667659 [01:57<02:27, 3045.01 examples/s]
hf-map ta (num_proc=8):  33%|###2      | 220160/667659 [01:58<03:38, 2050.72 examples/s]
hf-map ta (num_proc=8):  33%|###3      | 221184/667659 [01:58<03:30, 2124.81 examples/s]
hf-map ta (num_proc=8):  33%|###3      | 222208/667659 [01:59<03:20, 2222.62 examples/s]
hf-map ta (num_proc=8):  33%|###3      | 223232/667659 [01:59<03:48, 1944.59 examples/s]
hf-map ta (num_proc=8):  34%|###3      | 224256/667659 [02:00<03:24, 2169.62 examples/s]
hf-map ta (num_proc=8):  34%|###3      | 225280/667659 [02:00<02:58, 2480.16 examples/s]
hf-map ta (num_proc=8):  34%|###3      | 226304/667659 [02:00<02:54, 2526.67 examples/s]
hf-map ta (num_proc=8):  34%|###4      | 228352/667659 [02:02<03:58, 1845.35 examples/s]
hf-map ta (num_proc=8):  35%|###4      | 230400/667659 [02:02<02:49, 2583.93 examples/s]
hf-map ta (num_proc=8):  35%|###4      | 231424/667659 [02:03<03:37, 2003.96 examples/s]
hf-map ta (num_proc=8):  35%|###4      | 232448/667659 [02:03<03:13, 2249.34 examples/s]
hf-map ta (num_proc=8):  35%|###4      | 233472/667659 [02:04<03:00, 2399.84 examples/s]
hf-map ta (num_proc=8):  35%|###5      | 234496/667659 [02:04<02:37, 2750.30 examples/s]
hf-map ta (num_proc=8):  35%|###5      | 235520/667659 [02:04<02:48, 2570.75 examples/s]
hf-map ta (num_proc=8):  35%|###5      | 236544/667659 [02:05<04:08, 1732.18 examples/s]
hf-map ta (num_proc=8):  36%|###5      | 237568/667659 [02:05<03:14, 2212.43 examples/s]
hf-map ta (num_proc=8):  36%|###5      | 238592/667659 [02:06<02:51, 2508.52 examples/s]
hf-map ta (num_proc=8):  36%|###5      | 239616/667659 [02:06<03:06, 2292.79 examples/s]
hf-map ta (num_proc=8):  36%|###6      | 240640/667659 [02:07<03:06, 2287.61 examples/s]
hf-map ta (num_proc=8):  36%|###6      | 241664/667659 [02:07<02:36, 2726.80 examples/s]
hf-map ta (num_proc=8):  36%|###6      | 242688/667659 [02:07<02:11, 3220.82 examples/s]
hf-map ta (num_proc=8):  37%|###6      | 243712/667659 [02:08<02:52, 2456.29 examples/s]
hf-map ta (num_proc=8):  37%|###6      | 244736/667659 [02:09<04:04, 1730.28 examples/s]
hf-map ta (num_proc=8):  37%|###6      | 245760/667659 [02:09<03:07, 2247.34 examples/s]
hf-map ta (num_proc=8):  37%|###6      | 246784/667659 [02:09<02:57, 2372.03 examples/s]
hf-map ta (num_proc=8):  37%|###7      | 247808/667659 [02:09<02:27, 2842.10 examples/s]
hf-map ta (num_proc=8):  37%|###7      | 248832/667659 [02:10<02:54, 2406.94 examples/s]
hf-map ta (num_proc=8):  38%|###7      | 251904/667659 [02:11<02:44, 2527.24 examples/s]
hf-map ta (num_proc=8):  38%|###7      | 252928/667659 [02:12<03:33, 1938.33 examples/s]
hf-map ta (num_proc=8):  38%|###8      | 253952/667659 [02:13<03:09, 2180.56 examples/s]
hf-map ta (num_proc=8):  38%|###8      | 254976/667659 [02:13<02:32, 2706.45 examples/s]
hf-map ta (num_proc=8):  38%|###8      | 257024/667659 [02:14<02:47, 2445.72 examples/s]
hf-map ta (num_proc=8):  39%|###8      | 259072/667659 [02:14<02:01, 3371.23 examples/s]
hf-map ta (num_proc=8):  39%|###8      | 260096/667659 [02:15<02:58, 2281.03 examples/s]
hf-map ta (num_proc=8):  39%|###9      | 261120/667659 [02:16<03:40, 1845.88 examples/s]
hf-map ta (num_proc=8):  39%|###9      | 263168/667659 [02:16<02:39, 2528.10 examples/s]
hf-map ta (num_proc=8):  40%|###9      | 264192/667659 [02:16<02:14, 2993.73 examples/s]
hf-map ta (num_proc=8):  40%|###9      | 265216/667659 [02:17<02:56, 2277.86 examples/s]
hf-map ta (num_proc=8):  40%|###9      | 266240/667659 [02:17<02:37, 2547.62 examples/s]
hf-map ta (num_proc=8):  40%|####      | 268288/667659 [02:18<02:41, 2468.43 examples/s]
hf-map ta (num_proc=8):  40%|####      | 269312/667659 [02:19<03:31, 1882.10 examples/s]
hf-map ta (num_proc=8):  40%|####      | 270336/667659 [02:19<03:00, 2201.86 examples/s]
hf-map ta (num_proc=8):  41%|####      | 272384/667659 [02:20<02:14, 2932.69 examples/s]
hf-map ta (num_proc=8):  41%|####      | 273408/667659 [02:20<02:36, 2526.42 examples/s]
hf-map ta (num_proc=8):  41%|####1     | 274432/667659 [02:21<02:36, 2506.41 examples/s]
hf-map ta (num_proc=8):  41%|####1     | 276480/667659 [02:22<03:00, 2172.74 examples/s]
hf-map ta (num_proc=8):  42%|####1     | 277504/667659 [02:22<03:07, 2080.29 examples/s]
hf-map ta (num_proc=8):  42%|####1     | 278528/667659 [02:23<02:39, 2446.77 examples/s]
hf-map ta (num_proc=8):  42%|####2     | 280576/667659 [02:23<01:49, 3521.53 examples/s]
hf-map ta (num_proc=8):  42%|####2     | 281600/667659 [02:23<01:57, 3276.49 examples/s]
hf-map ta (num_proc=8):  42%|####2     | 282624/667659 [02:24<02:17, 2801.81 examples/s]
hf-map ta (num_proc=8):  43%|####2     | 284672/667659 [02:25<02:59, 2131.15 examples/s]
hf-map ta (num_proc=8):  43%|####2     | 285696/667659 [02:25<02:52, 2208.49 examples/s]
hf-map ta (num_proc=8):  43%|####2     | 286720/667659 [02:26<02:29, 2543.31 examples/s]
hf-map ta (num_proc=8):  43%|####3     | 289792/667659 [02:26<01:24, 4495.72 examples/s]
hf-map ta (num_proc=8):  44%|####3     | 290816/667659 [02:26<01:41, 3708.07 examples/s]
hf-map ta (num_proc=8):  44%|####3     | 291840/667659 [02:27<01:43, 3614.03 examples/s]
hf-map ta (num_proc=8):  44%|####3     | 292864/667659 [02:28<03:43, 1680.51 examples/s]
hf-map ta (num_proc=8):  44%|####4     | 293888/667659 [02:29<03:24, 1830.62 examples/s]
hf-map ta (num_proc=8):  44%|####4     | 295936/667659 [02:29<02:12, 2809.77 examples/s]
hf-map ta (num_proc=8):  44%|####4     | 296960/667659 [02:29<02:03, 2997.70 examples/s]
hf-map ta (num_proc=8):  45%|####4     | 297984/667659 [02:30<02:18, 2673.66 examples/s]
hf-map ta (num_proc=8):  45%|####4     | 300032/667659 [02:30<01:41, 3624.87 examples/s]
hf-map ta (num_proc=8):  45%|####5     | 301056/667659 [02:32<03:34, 1712.11 examples/s]
hf-map ta (num_proc=8):  45%|####5     | 302080/667659 [02:32<02:56, 2075.60 examples/s]
hf-map ta (num_proc=8):  45%|####5     | 303104/667659 [02:32<02:35, 2345.07 examples/s]
hf-map ta (num_proc=8):  46%|####5     | 304128/667659 [02:32<02:14, 2706.80 examples/s]
hf-map ta (num_proc=8):  46%|####5     | 305152/667659 [02:32<02:03, 2946.73 examples/s]
hf-map ta (num_proc=8):  46%|####5     | 306176/667659 [02:33<01:41, 3566.46 examples/s]
hf-map ta (num_proc=8):  46%|####6     | 307200/667659 [02:33<01:50, 3268.57 examples/s]
hf-map ta (num_proc=8):  46%|####6     | 308224/667659 [02:33<01:29, 4033.78 examples/s]
hf-map ta (num_proc=8):  46%|####6     | 309248/667659 [02:34<03:04, 1945.15 examples/s]
hf-map ta (num_proc=8):  46%|####6     | 310272/667659 [02:35<02:37, 2268.42 examples/s]
hf-map ta (num_proc=8):  47%|####6     | 311296/667659 [02:35<02:06, 2814.65 examples/s]
hf-map ta (num_proc=8):  47%|####6     | 312320/667659 [02:35<01:55, 3085.54 examples/s]
hf-map ta (num_proc=8):  47%|####6     | 313344/667659 [02:35<01:51, 3167.09 examples/s]
hf-map ta (num_proc=8):  47%|####7     | 314368/667659 [02:35<01:41, 3465.39 examples/s]
hf-map ta (num_proc=8):  47%|####7     | 315392/667659 [02:36<01:33, 3786.29 examples/s]
hf-map ta (num_proc=8):  48%|####7     | 317440/667659 [02:37<02:46, 2105.41 examples/s]
hf-map ta (num_proc=8):  48%|####7     | 319488/667659 [02:38<02:02, 2850.14 examples/s]
hf-map ta (num_proc=8):  48%|####8     | 320512/667659 [02:38<01:44, 3335.50 examples/s]
hf-map ta (num_proc=8):  48%|####8     | 321536/667659 [02:38<01:29, 3849.60 examples/s]
hf-map ta (num_proc=8):  48%|####8     | 322560/667659 [02:38<01:49, 3152.81 examples/s]
hf-map ta (num_proc=8):  49%|####8     | 324608/667659 [02:39<01:22, 4148.53 examples/s]
hf-map ta (num_proc=8):  49%|####8     | 325632/667659 [02:40<02:35, 2200.70 examples/s]
hf-map ta (num_proc=8):  49%|####8     | 326656/667659 [02:40<02:22, 2397.80 examples/s]
hf-map ta (num_proc=8):  49%|####9     | 328704/667659 [02:40<01:30, 3747.31 examples/s]
hf-map ta (num_proc=8):  49%|####9     | 329728/667659 [02:40<01:22, 4117.37 examples/s]
hf-map ta (num_proc=8):  50%|####9     | 330752/667659 [02:41<01:50, 3049.90 examples/s]
hf-map ta (num_proc=8):  50%|####9     | 332800/667659 [02:42<01:59, 2810.69 examples/s]
hf-map ta (num_proc=8):  50%|####9     | 333824/667659 [02:42<02:11, 2540.05 examples/s]
hf-map ta (num_proc=8):  50%|#####     | 334848/667659 [02:43<02:05, 2643.38 examples/s]
hf-map ta (num_proc=8):  50%|#####     | 336896/667659 [02:43<01:26, 3824.75 examples/s]
hf-map ta (num_proc=8):  51%|#####     | 337920/667659 [02:43<01:14, 4443.37 examples/s]
hf-map ta (num_proc=8):  51%|#####     | 338944/667659 [02:43<01:40, 3263.89 examples/s]
hf-map ta (num_proc=8):  51%|#####     | 339968/667659 [02:44<01:31, 3590.91 examples/s]
hf-map ta (num_proc=8):  51%|#####1    | 340992/667659 [02:44<02:16, 2387.72 examples/s]
hf-map ta (num_proc=8):  51%|#####1    | 342016/667659 [02:45<02:12, 2456.08 examples/s]
hf-map ta (num_proc=8):  51%|#####1    | 343040/667659 [02:45<02:10, 2487.44 examples/s]
hf-map ta (num_proc=8):  52%|#####1    | 344064/667659 [02:46<01:53, 2855.21 examples/s]
hf-map ta (num_proc=8):  52%|#####1    | 345088/667659 [02:46<01:30, 3565.34 examples/s]
hf-map ta (num_proc=8):  52%|#####1    | 346112/667659 [02:46<01:18, 4077.94 examples/s]
hf-map ta (num_proc=8):  52%|#####1    | 347136/667659 [02:46<01:42, 3116.75 examples/s]
hf-map ta (num_proc=8):  52%|#####2    | 348160/667659 [02:47<01:51, 2860.48 examples/s]
hf-map ta (num_proc=8):  52%|#####2    | 349184/667659 [02:47<01:59, 2670.01 examples/s]
hf-map ta (num_proc=8):  52%|#####2    | 350208/667659 [02:48<01:57, 2700.34 examples/s]
hf-map ta (num_proc=8):  53%|#####2    | 351232/667659 [02:48<02:01, 2595.77 examples/s]
hf-map ta (num_proc=8):  53%|#####2    | 353280/667659 [02:48<01:22, 3811.40 examples/s]
hf-map ta (num_proc=8):  53%|#####3    | 354304/667659 [02:49<01:25, 3648.10 examples/s]
hf-map ta (num_proc=8):  53%|#####3    | 355328/667659 [02:49<01:19, 3947.77 examples/s]
hf-map ta (num_proc=8):  53%|#####3    | 356352/667659 [02:49<01:29, 3486.85 examples/s]
hf-map ta (num_proc=8):  54%|#####3    | 357376/667659 [02:50<02:00, 2584.98 examples/s]
hf-map ta (num_proc=8):  54%|#####3    | 359424/667659 [02:51<01:58, 2604.83 examples/s]
hf-map ta (num_proc=8):  54%|#####4    | 361472/667659 [02:51<01:38, 3111.62 examples/s]
hf-map ta (num_proc=8):  54%|#####4    | 362496/667659 [02:51<01:26, 3522.82 examples/s]
hf-map ta (num_proc=8):  55%|#####4    | 364544/667659 [02:52<01:30, 3345.82 examples/s]
hf-map ta (num_proc=8):  55%|#####4    | 365568/667659 [02:52<01:49, 2754.61 examples/s]
hf-map ta (num_proc=8):  55%|#####4    | 366592/667659 [02:53<01:49, 2740.54 examples/s]
hf-map ta (num_proc=8):  55%|#####5    | 367616/667659 [02:53<01:50, 2706.74 examples/s]
hf-map ta (num_proc=8):  55%|#####5    | 368640/667659 [02:53<01:31, 3283.68 examples/s]
hf-map ta (num_proc=8):  55%|#####5    | 369664/667659 [02:54<01:32, 3236.52 examples/s]
hf-map ta (num_proc=8):  56%|#####5    | 370688/667659 [02:54<01:24, 3513.73 examples/s]
hf-map ta (num_proc=8):  56%|#####5    | 372736/667659 [02:55<01:27, 3356.90 examples/s]
hf-map ta (num_proc=8):  56%|#####5    | 373760/667659 [02:55<01:46, 2757.42 examples/s]
hf-map ta (num_proc=8):  56%|#####6    | 374784/667659 [02:56<02:05, 2330.34 examples/s]
hf-map ta (num_proc=8):  57%|#####6    | 377856/667659 [02:56<01:18, 3714.39 examples/s]
hf-map ta (num_proc=8):  57%|#####6    | 378880/667659 [02:57<01:31, 3150.75 examples/s]
hf-map ta (num_proc=8):  57%|#####7    | 380928/667659 [02:57<01:32, 3089.29 examples/s]
hf-map ta (num_proc=8):  57%|#####7    | 381952/667659 [02:58<02:04, 2300.51 examples/s]
hf-map ta (num_proc=8):  57%|#####7    | 382976/667659 [02:59<02:10, 2182.04 examples/s]
hf-map ta (num_proc=8):  58%|#####7    | 385024/667659 [02:59<01:31, 3078.57 examples/s]
hf-map ta (num_proc=8):  58%|#####7    | 386048/667659 [02:59<01:33, 3013.44 examples/s]
hf-map ta (num_proc=8):  58%|#####7    | 387072/667659 [03:00<01:40, 2786.29 examples/s]
hf-map ta (num_proc=8):  58%|#####8    | 388096/667659 [03:00<01:24, 3312.24 examples/s]
hf-map ta (num_proc=8):  58%|#####8    | 389120/667659 [03:00<01:35, 2911.84 examples/s]
hf-map ta (num_proc=8):  58%|#####8    | 390144/667659 [03:01<02:11, 2114.67 examples/s]
hf-map ta (num_proc=8):  59%|#####8    | 391168/667659 [03:02<02:19, 1982.19 examples/s]
hf-map ta (num_proc=8):  59%|#####8    | 393216/667659 [03:02<01:40, 2720.64 examples/s]
hf-map ta (num_proc=8):  59%|#####9    | 394240/667659 [03:03<01:34, 2884.05 examples/s]
hf-map ta (num_proc=8):  59%|#####9    | 395264/667659 [03:03<01:41, 2675.13 examples/s]
hf-map ta (num_proc=8):  59%|#####9    | 396288/667659 [03:04<01:53, 2391.87 examples/s]
hf-map ta (num_proc=8):  60%|#####9    | 397312/667659 [03:04<02:01, 2228.27 examples/s]
hf-map ta (num_proc=8):  60%|#####9    | 398336/667659 [03:04<01:43, 2606.83 examples/s]
hf-map ta (num_proc=8):  60%|#####9    | 399360/667659 [03:05<01:39, 2699.96 examples/s]
hf-map ta (num_proc=8):  60%|#####9    | 400384/667659 [03:05<01:25, 3133.80 examples/s]
hf-map ta (num_proc=8):  60%|######    | 401408/667659 [03:06<01:52, 2373.03 examples/s]
hf-map ta (num_proc=8):  60%|######    | 402432/667659 [03:06<01:41, 2620.33 examples/s]
hf-map ta (num_proc=8):  60%|######    | 403456/667659 [03:06<01:18, 3362.79 examples/s]
hf-map ta (num_proc=8):  61%|######    | 404480/667659 [03:07<02:11, 1997.25 examples/s]
hf-map ta (num_proc=8):  61%|######    | 405504/667659 [03:07<02:03, 2129.85 examples/s]
hf-map ta (num_proc=8):  61%|######    | 406528/667659 [03:08<01:42, 2540.10 examples/s]
hf-map ta (num_proc=8):  61%|######1   | 407552/667659 [03:08<01:39, 2625.98 examples/s]
hf-map ta (num_proc=8):  61%|######1   | 409600/667659 [03:09<01:31, 2806.16 examples/s]
hf-map ta (num_proc=8):  62%|######1   | 410624/667659 [03:09<01:26, 2964.56 examples/s]
hf-map ta (num_proc=8):  62%|######1   | 411648/667659 [03:09<01:19, 3221.54 examples/s]
hf-map ta (num_proc=8):  62%|######1   | 412672/667659 [03:10<01:58, 2144.18 examples/s]
hf-map ta (num_proc=8):  62%|######1   | 413696/667659 [03:10<01:36, 2630.31 examples/s]
hf-map ta (num_proc=8):  62%|######2   | 414720/667659 [03:11<01:40, 2509.10 examples/s]
hf-map ta (num_proc=8):  62%|######2   | 415744/667659 [03:11<01:30, 2787.77 examples/s]
hf-map ta (num_proc=8):  63%|######2   | 417792/667659 [03:12<01:44, 2391.61 examples/s]
hf-map ta (num_proc=8):  63%|######2   | 418816/667659 [03:12<01:36, 2581.27 examples/s]
hf-map ta (num_proc=8):  63%|######2   | 419840/667659 [03:12<01:25, 2881.68 examples/s]
hf-map ta (num_proc=8):  63%|######3   | 420864/667659 [03:13<01:29, 2762.93 examples/s]
hf-map ta (num_proc=8):  63%|######3   | 422912/667659 [03:14<01:28, 2759.79 examples/s]
hf-map ta (num_proc=8):  63%|######3   | 423936/667659 [03:14<01:13, 3322.06 examples/s]
hf-map ta (num_proc=8):  64%|######3   | 424960/667659 [03:14<01:06, 3636.02 examples/s]
hf-map ta (num_proc=8):  64%|######3   | 425984/667659 [03:15<01:34, 2548.29 examples/s]
hf-map ta (num_proc=8):  64%|######3   | 427008/667659 [03:15<01:20, 2998.49 examples/s]
hf-map ta (num_proc=8):  64%|######4   | 428032/667659 [03:15<01:06, 3619.38 examples/s]
hf-map ta (num_proc=8):  64%|######4   | 429056/667659 [03:16<01:33, 2539.28 examples/s]
hf-map ta (num_proc=8):  65%|######4   | 431104/667659 [03:17<01:40, 2358.64 examples/s]
hf-map ta (num_proc=8):  65%|######4   | 433152/667659 [03:17<01:17, 3010.73 examples/s]
hf-map ta (num_proc=8):  65%|######5   | 434176/667659 [03:18<01:29, 2617.25 examples/s]
hf-map ta (num_proc=8):  65%|######5   | 435200/667659 [03:18<01:35, 2424.81 examples/s]
hf-map ta (num_proc=8):  65%|######5   | 437248/667659 [03:19<01:20, 2867.62 examples/s]
hf-map ta (num_proc=8):  66%|######5   | 438272/667659 [03:19<01:26, 2651.31 examples/s]
hf-map ta (num_proc=8):  66%|######5   | 439296/667659 [03:20<01:31, 2483.15 examples/s]
hf-map ta (num_proc=8):  66%|######6   | 441344/667659 [03:20<01:17, 2919.60 examples/s]
hf-map ta (num_proc=8):  66%|######6   | 442368/667659 [03:20<01:05, 3432.16 examples/s]
hf-map ta (num_proc=8):  66%|######6   | 443392/667659 [03:21<01:29, 2512.33 examples/s]
hf-map ta (num_proc=8):  67%|######6   | 444416/667659 [03:22<01:58, 1889.32 examples/s]
hf-map ta (num_proc=8):  67%|######6   | 445440/667659 [03:22<01:45, 2098.82 examples/s]
hf-map ta (num_proc=8):  67%|######6   | 446464/667659 [03:22<01:27, 2527.15 examples/s]
hf-map ta (num_proc=8):  67%|######7   | 447488/667659 [03:23<01:49, 2011.32 examples/s]
hf-map ta (num_proc=8):  67%|######7   | 449536/667659 [03:23<01:06, 3294.93 examples/s]
hf-map ta (num_proc=8):  67%|######7   | 450560/667659 [03:24<01:18, 2757.77 examples/s]
hf-map ta (num_proc=8):  68%|######7   | 451584/667659 [03:24<01:15, 2847.13 examples/s]
hf-map ta (num_proc=8):  68%|######7   | 452608/667659 [03:25<01:42, 2104.53 examples/s]
hf-map ta (num_proc=8):  68%|######7   | 453632/667659 [03:25<01:26, 2488.53 examples/s]
hf-map ta (num_proc=8):  68%|######8   | 455680/667659 [03:26<01:21, 2613.35 examples/s]
hf-map ta (num_proc=8):  68%|######8   | 456704/667659 [03:26<01:17, 2723.05 examples/s]
hf-map ta (num_proc=8):  69%|######8   | 457728/667659 [03:27<01:12, 2888.15 examples/s]
hf-map ta (num_proc=8):  69%|######8   | 459776/667659 [03:28<01:42, 2032.91 examples/s]
hf-map ta (num_proc=8):  69%|######9   | 460800/667659 [03:28<01:24, 2460.77 examples/s]
hf-map ta (num_proc=8):  69%|######9   | 461824/667659 [03:28<01:09, 2982.19 examples/s]
hf-map ta (num_proc=8):  69%|######9   | 462848/667659 [03:29<01:45, 1946.50 examples/s]
hf-map ta (num_proc=8):  70%|######9   | 464896/667659 [03:30<01:20, 2516.31 examples/s]
hf-map ta (num_proc=8):  70%|######9   | 465920/667659 [03:30<01:15, 2676.01 examples/s]
hf-map ta (num_proc=8):  70%|######9   | 466944/667659 [03:30<01:04, 3131.20 examples/s]
hf-map ta (num_proc=8):  70%|#######   | 467968/667659 [03:31<01:02, 3174.59 examples/s]
hf-map ta (num_proc=8):  70%|#######   | 468992/667659 [03:31<00:59, 3363.88 examples/s]
hf-map ta (num_proc=8):  70%|#######   | 470016/667659 [03:31<01:00, 3265.28 examples/s]
hf-map ta (num_proc=8):  71%|#######   | 471040/667659 [03:32<01:26, 2284.40 examples/s]
hf-map ta (num_proc=8):  71%|#######   | 472064/667659 [03:32<01:13, 2673.75 examples/s]
hf-map ta (num_proc=8):  71%|#######   | 473088/667659 [03:33<01:18, 2481.18 examples/s]
hf-map ta (num_proc=8):  71%|#######1  | 475136/667659 [03:33<00:53, 3615.16 examples/s]
hf-map ta (num_proc=8):  71%|#######1  | 476160/667659 [03:33<00:45, 4193.61 examples/s]
hf-map ta (num_proc=8):  71%|#######1  | 477184/667659 [03:33<00:45, 4150.20 examples/s]
hf-map ta (num_proc=8):  72%|#######1  | 478208/667659 [03:34<00:51, 3653.82 examples/s]
hf-map ta (num_proc=8):  72%|#######1  | 479232/667659 [03:34<01:15, 2485.10 examples/s]
hf-map ta (num_proc=8):  72%|#######1  | 480256/667659 [03:35<01:24, 2216.67 examples/s]
hf-map ta (num_proc=8):  72%|#######2  | 481280/667659 [03:35<01:19, 2335.98 examples/s]
hf-map ta (num_proc=8):  72%|#######2  | 482304/667659 [03:36<01:01, 2995.82 examples/s]
hf-map ta (num_proc=8):  72%|#######2  | 483328/667659 [03:36<00:58, 3169.37 examples/s]
hf-map ta (num_proc=8):  73%|#######2  | 484352/667659 [03:36<00:47, 3826.42 examples/s]
hf-map ta (num_proc=8):  73%|#######2  | 485376/667659 [03:36<00:55, 3272.10 examples/s]
hf-map ta (num_proc=8):  73%|#######3  | 487424/667659 [03:38<01:14, 2417.46 examples/s]
hf-map ta (num_proc=8):  73%|#######3  | 488448/667659 [03:38<01:12, 2463.11 examples/s]
hf-map ta (num_proc=8):  73%|#######3  | 489472/667659 [03:38<01:05, 2737.80 examples/s]
hf-map ta (num_proc=8):  73%|#######3  | 490496/667659 [03:39<01:04, 2759.11 examples/s]
hf-map ta (num_proc=8):  74%|#######3  | 492544/667659 [03:39<00:39, 4414.79 examples/s]
hf-map ta (num_proc=8):  74%|#######3  | 493568/667659 [03:39<00:46, 3720.44 examples/s]
hf-map ta (num_proc=8):  74%|#######4  | 494592/667659 [03:39<00:48, 3592.68 examples/s]
hf-map ta (num_proc=8):  74%|#######4  | 495616/667659 [03:40<01:08, 2507.93 examples/s]
hf-map ta (num_proc=8):  74%|#######4  | 496640/667659 [03:41<01:15, 2252.84 examples/s]
hf-map ta (num_proc=8):  75%|#######4  | 498688/667659 [03:41<00:58, 2912.84 examples/s]
hf-map ta (num_proc=8):  75%|#######4  | 499712/667659 [03:41<00:47, 3508.40 examples/s]
hf-map ta (num_proc=8):  75%|#######4  | 500736/667659 [03:42<00:51, 3266.34 examples/s]
hf-map ta (num_proc=8):  75%|#######5  | 501760/667659 [03:42<00:44, 3697.84 examples/s]
hf-map ta (num_proc=8):  75%|#######5  | 502784/667659 [03:42<00:39, 4226.77 examples/s]
hf-map ta (num_proc=8):  75%|#######5  | 503808/667659 [03:43<01:08, 2402.26 examples/s]
hf-map ta (num_proc=8):  76%|#######5  | 504832/667659 [03:44<01:20, 2020.04 examples/s]
hf-map ta (num_proc=8):  76%|#######5  | 506880/667659 [03:44<00:54, 2961.47 examples/s]
hf-map ta (num_proc=8):  76%|#######6  | 507904/667659 [03:44<00:53, 2979.61 examples/s]
hf-map ta (num_proc=8):  76%|#######6  | 508928/667659 [03:44<00:47, 3321.67 examples/s]
hf-map ta (num_proc=8):  76%|#######6  | 509952/667659 [03:45<01:11, 2208.74 examples/s]
hf-map ta (num_proc=8):  77%|#######6  | 510976/667659 [03:46<01:04, 2412.95 examples/s]
hf-map ta (num_proc=8):  77%|#######6  | 513024/667659 [03:46<00:56, 2717.53 examples/s]
hf-map ta (num_proc=8):  77%|#######6  | 514048/667659 [03:47<00:53, 2866.12 examples/s]
hf-map ta (num_proc=8):  77%|#######7  | 516096/667659 [03:47<00:43, 3477.76 examples/s]
hf-map ta (num_proc=8):  78%|#######7  | 518144/667659 [03:48<00:54, 2763.59 examples/s]
hf-map ta (num_proc=8):  78%|#######7  | 520192/667659 [03:48<00:45, 3258.33 examples/s]
hf-map ta (num_proc=8):  78%|#######8  | 521216/667659 [03:49<00:55, 2641.36 examples/s]
hf-map ta (num_proc=8):  78%|#######8  | 523264/667659 [03:49<00:38, 3743.41 examples/s]
hf-map ta (num_proc=8):  79%|#######8  | 524288/667659 [03:49<00:39, 3633.54 examples/s]
hf-map ta (num_proc=8):  79%|#######8  | 525312/667659 [03:50<00:41, 3415.69 examples/s]
hf-map ta (num_proc=8):  79%|#######8  | 526336/667659 [03:50<00:41, 3386.65 examples/s]
hf-map ta (num_proc=8):  79%|#######8  | 527360/667659 [03:51<00:43, 3218.43 examples/s]
hf-map ta (num_proc=8):  79%|#######9  | 528384/667659 [03:51<00:54, 2547.95 examples/s]
hf-map ta (num_proc=8):  79%|#######9  | 529408/667659 [03:52<00:56, 2451.37 examples/s]
hf-map ta (num_proc=8):  79%|#######9  | 530432/667659 [03:52<00:45, 2988.90 examples/s]
hf-map ta (num_proc=8):  80%|#######9  | 531456/667659 [03:52<00:37, 3639.06 examples/s]
hf-map ta (num_proc=8):  80%|#######9  | 532480/667659 [03:52<00:40, 3336.09 examples/s]
hf-map ta (num_proc=8):  80%|########  | 534528/667659 [03:53<00:34, 3872.36 examples/s]
hf-map ta (num_proc=8):  80%|########  | 535552/667659 [03:53<00:31, 4202.15 examples/s]
hf-map ta (num_proc=8):  80%|########  | 536576/667659 [03:54<00:47, 2759.45 examples/s]
hf-map ta (num_proc=8):  81%|########  | 537600/667659 [03:54<00:57, 2256.36 examples/s]
hf-map ta (num_proc=8):  81%|########  | 540672/667659 [03:55<00:41, 3049.86 examples/s]
hf-map ta (num_proc=8):  81%|########1 | 541696/667659 [03:55<00:37, 3387.06 examples/s]
hf-map ta (num_proc=8):  81%|########1 | 543744/667659 [03:55<00:27, 4528.33 examples/s]
hf-map ta (num_proc=8):  82%|########1 | 544768/667659 [03:56<00:40, 3014.87 examples/s]
hf-map ta (num_proc=8):  82%|########1 | 545792/667659 [03:57<00:45, 2663.39 examples/s]
hf-map ta (num_proc=8):  82%|########1 | 546816/667659 [03:57<00:38, 3134.23 examples/s]
hf-map ta (num_proc=8):  82%|########2 | 547840/667659 [03:57<00:33, 3620.57 examples/s]
hf-map ta (num_proc=8):  82%|########2 | 548864/667659 [03:58<00:42, 2764.43 examples/s]
hf-map ta (num_proc=8):  83%|########2 | 550912/667659 [03:58<00:27, 4170.44 examples/s]
hf-map ta (num_proc=8):  83%|########2 | 551936/667659 [03:59<00:44, 2615.44 examples/s]
hf-map ta (num_proc=8):  83%|########2 | 553984/667659 [03:59<00:42, 2669.16 examples/s]
hf-map ta (num_proc=8):  83%|########3 | 556032/667659 [04:00<00:33, 3342.32 examples/s]
hf-map ta (num_proc=8):  83%|########3 | 557056/667659 [04:00<00:39, 2825.53 examples/s]
hf-map ta (num_proc=8):  84%|########3 | 558080/667659 [04:00<00:34, 3207.86 examples/s]
hf-map ta (num_proc=8):  84%|########3 | 559104/667659 [04:01<00:35, 3050.07 examples/s]
hf-map ta (num_proc=8):  84%|########3 | 560128/667659 [04:01<00:42, 2559.03 examples/s]
hf-map ta (num_proc=8):  84%|########4 | 562176/667659 [04:02<00:36, 2906.97 examples/s]
hf-map ta (num_proc=8):  85%|########4 | 564224/667659 [04:03<00:32, 3150.49 examples/s]
hf-map ta (num_proc=8):  85%|########4 | 566272/667659 [04:03<00:24, 4064.72 examples/s]
hf-map ta (num_proc=8):  85%|########4 | 567296/667659 [04:03<00:25, 3980.69 examples/s]
hf-map ta (num_proc=8):  85%|########5 | 568320/667659 [04:04<00:32, 3086.29 examples/s]
hf-map ta (num_proc=8):  85%|########5 | 570368/667659 [04:04<00:29, 3347.56 examples/s]
hf-map ta (num_proc=8):  86%|########5 | 571392/667659 [04:04<00:28, 3385.59 examples/s]
hf-map ta (num_proc=8):  86%|########5 | 572416/667659 [04:05<00:29, 3271.67 examples/s]
hf-map ta (num_proc=8):  86%|########5 | 573440/667659 [04:05<00:25, 3714.25 examples/s]
hf-map ta (num_proc=8):  86%|########6 | 574464/667659 [04:05<00:23, 3889.37 examples/s]
hf-map ta (num_proc=8):  86%|########6 | 575488/667659 [04:05<00:21, 4312.07 examples/s]
hf-map ta (num_proc=8):  86%|########6 | 576512/667659 [04:06<00:29, 3051.84 examples/s]
hf-map ta (num_proc=8):  87%|########6 | 578560/667659 [04:06<00:22, 3973.35 examples/s]
hf-map ta (num_proc=8):  87%|########6 | 579584/667659 [04:07<00:23, 3824.06 examples/s]
hf-map ta (num_proc=8):  87%|########6 | 580608/667659 [04:07<00:29, 2975.60 examples/s]
hf-map ta (num_proc=8):  87%|########7 | 582656/667659 [04:07<00:21, 3878.75 examples/s]
hf-map ta (num_proc=8):  87%|########7 | 583680/667659 [04:08<00:21, 3913.47 examples/s]
hf-map ta (num_proc=8):  88%|########7 | 584704/667659 [04:08<00:24, 3318.70 examples/s]
hf-map ta (num_proc=8):  88%|########7 | 585728/667659 [04:08<00:21, 3891.89 examples/s]
hf-map ta (num_proc=8):  88%|########7 | 586752/667659 [04:08<00:19, 4104.98 examples/s]
hf-map ta (num_proc=8):  88%|########8 | 587776/667659 [04:09<00:20, 3944.62 examples/s]
hf-map ta (num_proc=8):  88%|########8 | 588800/667659 [04:09<00:25, 3148.11 examples/s]
hf-map ta (num_proc=8):  88%|########8 | 590848/667659 [04:10<00:20, 3816.53 examples/s]
hf-map ta (num_proc=8):  89%|########8 | 591872/667659 [04:10<00:23, 3180.65 examples/s]
hf-map ta (num_proc=8):  89%|########8 | 592896/667659 [04:10<00:22, 3292.95 examples/s]
hf-map ta (num_proc=8):  89%|########9 | 594944/667659 [04:11<00:16, 4401.50 examples/s]
hf-map ta (num_proc=8):  89%|########9 | 595968/667659 [04:11<00:20, 3474.47 examples/s]
hf-map ta (num_proc=8):  89%|########9 | 596992/667659 [04:11<00:19, 3557.91 examples/s]
hf-map ta (num_proc=8):  90%|########9 | 598016/667659 [04:12<00:22, 3068.53 examples/s]
hf-map ta (num_proc=8):  90%|########9 | 599040/667659 [04:12<00:20, 3391.05 examples/s]
hf-map ta (num_proc=8):  90%|########9 | 600064/667659 [04:13<00:29, 2305.07 examples/s]
hf-map ta (num_proc=8):  90%|######### | 601088/667659 [04:13<00:24, 2713.78 examples/s]
hf-map ta (num_proc=8):  90%|######### | 604160/667659 [04:14<00:16, 3927.87 examples/s]
hf-map ta (num_proc=8):  91%|######### | 605184/667659 [04:14<00:16, 3802.29 examples/s]
hf-map ta (num_proc=8):  91%|######### | 606208/667659 [04:14<00:18, 3297.43 examples/s]
hf-map ta (num_proc=8):  91%|######### | 607232/667659 [04:15<00:16, 3616.12 examples/s]
hf-map ta (num_proc=8):  91%|#########1| 608256/667659 [04:15<00:21, 2794.31 examples/s]
hf-map ta (num_proc=8):  91%|#########1| 610304/667659 [04:15<00:14, 4052.22 examples/s]
hf-map ta (num_proc=8):  92%|#########1| 612352/667659 [04:16<00:13, 4010.24 examples/s]
hf-map ta (num_proc=8):  92%|#########2| 614400/667659 [04:17<00:15, 3372.52 examples/s]
hf-map ta (num_proc=8):  92%|#########2| 616448/667659 [04:18<00:17, 2967.15 examples/s]
hf-map ta (num_proc=8):  92%|#########2| 617472/667659 [04:18<00:15, 3193.91 examples/s]
hf-map ta (num_proc=8):  93%|#########2| 620544/667659 [04:18<00:09, 4883.89 examples/s]
hf-map ta (num_proc=8):  93%|#########3| 621568/667659 [04:19<00:13, 3525.04 examples/s]
hf-map ta (num_proc=8):  93%|#########3| 622592/667659 [04:19<00:11, 3772.58 examples/s]
hf-map ta (num_proc=8):  93%|#########3| 623616/667659 [04:19<00:12, 3499.71 examples/s]
hf-map ta (num_proc=8):  94%|#########3| 624640/667659 [04:20<00:16, 2642.26 examples/s]
hf-map ta (num_proc=8):  94%|#########3| 625664/667659 [04:20<00:13, 3147.48 examples/s]
hf-map ta (num_proc=8):  94%|#########4| 628736/667659 [04:20<00:06, 5725.47 examples/s]
hf-map ta (num_proc=8):  94%|#########4| 629760/667659 [04:21<00:11, 3359.33 examples/s]
hf-map ta (num_proc=8):  94%|#########4| 630784/667659 [04:22<00:12, 2882.51 examples/s]
hf-map ta (num_proc=8):  95%|#########4| 631808/667659 [04:22<00:11, 3184.68 examples/s]
hf-map ta (num_proc=8):  95%|#########4| 632832/667659 [04:22<00:14, 2461.94 examples/s]
hf-map ta (num_proc=8):  95%|#########4| 633856/667659 [04:23<00:12, 2662.90 examples/s]
hf-map ta (num_proc=8):  96%|#########5| 637952/667659 [04:24<00:08, 3365.61 examples/s]
hf-map ta (num_proc=8):  96%|#########5| 638976/667659 [04:24<00:08, 3303.83 examples/s]
hf-map ta (num_proc=8):  96%|#########5| 640000/667659 [04:24<00:07, 3801.32 examples/s]
hf-map ta (num_proc=8):  96%|#########6| 641024/667659 [04:25<00:10, 2580.22 examples/s]
hf-map ta (num_proc=8):  96%|#########6| 642048/667659 [04:25<00:08, 3002.66 examples/s]
hf-map ta (num_proc=8):  96%|#########6| 644096/667659 [04:25<00:05, 4426.72 examples/s]
hf-map ta (num_proc=8):  97%|#########6| 645120/667659 [04:26<00:05, 4231.28 examples/s]
hf-map ta (num_proc=8):  97%|#########6| 646144/667659 [04:26<00:07, 2923.57 examples/s]
hf-map ta (num_proc=8):  97%|#########7| 647681/667659 [04:27<00:05, 3543.78 examples/s]
hf-map ta (num_proc=8):  97%|#########7| 649218/667659 [04:27<00:03, 4729.36 examples/s]
hf-map ta (num_proc=8):  97%|#########7| 650242/667659 [04:27<00:05, 3173.39 examples/s]
hf-map ta (num_proc=8):  98%|#########7| 651780/667659 [04:28<00:04, 3484.06 examples/s]
hf-map ta (num_proc=8):  98%|#########7| 652804/667659 [04:28<00:04, 3131.52 examples/s]
hf-map ta (num_proc=8):  98%|#########7| 653828/667659 [04:29<00:05, 2471.18 examples/s]
hf-map ta (num_proc=8):  98%|#########8| 655876/667659 [04:30<00:05, 2042.68 examples/s]
hf-map ta (num_proc=8):  98%|#########8| 657413/667659 [04:30<00:03, 2594.04 examples/s]
hf-map ta (num_proc=8):  99%|#########8| 658437/667659 [04:31<00:03, 2557.36 examples/s]
hf-map ta (num_proc=8):  99%|#########8| 659461/667659 [04:31<00:03, 2285.86 examples/s]
hf-map ta (num_proc=8):  99%|#########8| 660485/667659 [04:32<00:04, 1702.28 examples/s]
hf-map ta (num_proc=8):  99%|#########9| 662533/667659 [04:33<00:02, 2161.76 examples/s]
hf-map ta (num_proc=8):  99%|#########9| 663047/667659 [04:34<00:02, 1755.95 examples/s]
hf-map ta (num_proc=8):  99%|#########9| 664071/667659 [04:34<00:02, 1738.68 examples/s]
hf-map ta (num_proc=8): 100%|#########9| 665095/667659 [04:35<00:01, 1818.25 examples/s]
hf-map ta (num_proc=8): 100%|#########9| 665609/667659 [04:36<00:01, 1305.53 examples/s]
hf-map ta (num_proc=8): 100%|#########9| 667146/667659 [04:36<00:00, 1881.48 examples/s]
hf-map ta (num_proc=8): 100%|##########| 667659/667659 [04:37<00:00, 1258.97 examples/s]
hf-map ta (num_proc=8): 100%|##########| 667659/667659 [04:38<00:00, 2399.81 examples/s]
    ta: datasets.map num_proc=8 done 278s
    ta: 25000 docs  23325458 tokens  283s
    ta: 50000 docs  46092495 tokens  285s
    ta: 75000 docs  69234775 tokens  287s
    ta: 100000 docs  92178764 tokens  289s
    ta: 125000 docs  114953253 tokens  291s
    ta: 150000 docs  138281352 tokens  294s
    ta: 175000 docs  161404187 tokens  296s
    ta: 200000 docs  184541026 tokens  299s
    ta: 225000 docs  207872624 tokens  301s
    ta: 250000 docs  230922341 tokens  303s
    ta: 275000 docs  254309793 tokens  305s
    ta: 300000 docs  277730139 tokens  307s
    ta: 325000 docs  301150397 tokens  309s
    ta: 350000 docs  324210916 tokens  311s
    ta: 375000 docs  346690917 tokens  313s
    ta: 400000 docs  369202603 tokens  315s
    ta: 425000 docs  392444860 tokens  317s
    ta: 450000 docs  416252337 tokens  319s
    ta: 475000 docs  439421148 tokens  321s
  packed ta: 456292825 tokens (1.000 of budget)  rows=728619 docs=493028  pad-per-doc (HF map)
    kk: datasets.map n=386470 batched=1024 num_proc=8

hf-map kk (num_proc=8):   0%|          | 0/386470 [00:00<?, ? examples/s]
hf-map kk (num_proc=8):   0%|          | 1024/386470 [00:09<58:52, 109.11 examples/s]
hf-map kk (num_proc=8):   1%|          | 2048/386470 [00:09<24:55, 257.07 examples/s]
hf-map kk (num_proc=8):   1%|          | 3072/386470 [00:09<13:53, 460.23 examples/s]
hf-map kk (num_proc=8):   1%|1         | 4096/386470 [00:09<08:37, 738.35 examples/s]
hf-map kk (num_proc=8):   2%|1         | 6144/386470 [00:09<04:13, 1501.61 examples/s]
hf-map kk (num_proc=8):   2%|2         | 8192/386470 [00:10<02:39, 2372.76 examples/s]
hf-map kk (num_proc=8):   3%|2         | 10240/386470 [00:14<07:05, 884.35 examples/s]
hf-map kk (num_proc=8):   3%|2         | 11264/386470 [00:15<06:06, 1023.22 examples/s]
hf-map kk (num_proc=8):   3%|3         | 13312/386470 [00:15<03:55, 1586.09 examples/s]
hf-map kk (num_proc=8):   4%|3         | 15360/386470 [00:15<02:46, 2222.78 examples/s]
hf-map kk (num_proc=8):   4%|4         | 16384/386470 [00:15<02:21, 2617.52 examples/s]
hf-map kk (num_proc=8):   5%|4         | 17408/386470 [00:19<06:14, 986.31 examples/s] 
hf-map kk (num_proc=8):   5%|4         | 18432/386470 [00:19<05:13, 1172.90 examples/s]
hf-map kk (num_proc=8):   5%|5         | 19456/386470 [00:19<04:50, 1262.59 examples/s]
hf-map kk (num_proc=8):   5%|5         | 20480/386470 [00:20<03:49, 1596.45 examples/s]
hf-map kk (num_proc=8):   6%|5         | 22528/386470 [00:20<02:19, 2617.19 examples/s]
hf-map kk (num_proc=8):   6%|6         | 23552/386470 [00:20<01:55, 3149.20 examples/s]
hf-map kk (num_proc=8):   6%|6         | 24576/386470 [00:20<02:16, 2654.03 examples/s]
hf-map kk (num_proc=8):   7%|6         | 25600/386470 [00:23<05:17, 1135.09 examples/s]
hf-map kk (num_proc=8):   7%|6         | 26624/386470 [00:23<04:47, 1252.95 examples/s]
hf-map kk (num_proc=8):   7%|7         | 27648/386470 [00:24<03:43, 1606.15 examples/s]
hf-map kk (num_proc=8):   7%|7         | 28672/386470 [00:24<03:38, 1636.84 examples/s]
hf-map kk (num_proc=8):   8%|8         | 31744/386470 [00:24<01:46, 3319.22 examples/s]
hf-map kk (num_proc=8):   8%|8         | 32768/386470 [00:25<02:14, 2623.91 examples/s]
hf-map kk (num_proc=8):   9%|8         | 33792/386470 [00:27<04:32, 1293.03 examples/s]
hf-map kk (num_proc=8):   9%|9         | 34816/386470 [00:28<04:40, 1252.73 examples/s]
hf-map kk (num_proc=8):  10%|9         | 36864/386470 [00:29<03:51, 1509.45 examples/s]
hf-map kk (num_proc=8):  10%|9         | 37888/386470 [00:29<03:14, 1789.20 examples/s]
hf-map kk (num_proc=8):  10%|#         | 39936/386470 [00:29<02:05, 2755.68 examples/s]
hf-map kk (num_proc=8):  11%|#         | 40960/386470 [00:30<02:29, 2317.85 examples/s]
hf-map kk (num_proc=8):  11%|#         | 41984/386470 [00:32<04:24, 1301.03 examples/s]
hf-map kk (num_proc=8):  11%|#1        | 43008/386470 [00:33<04:38, 1235.10 examples/s]
hf-map kk (num_proc=8):  11%|#1        | 44032/386470 [00:33<04:00, 1422.90 examples/s]
hf-map kk (num_proc=8):  12%|#1        | 45056/386470 [00:34<04:02, 1407.00 examples/s]
hf-map kk (num_proc=8):  12%|#2        | 48128/386470 [00:34<02:03, 2741.80 examples/s]
hf-map kk (num_proc=8):  13%|#2        | 49152/386470 [00:35<02:03, 2739.93 examples/s]
hf-map kk (num_proc=8):  13%|#2        | 50176/386470 [00:37<04:15, 1315.02 examples/s]
hf-map kk (num_proc=8):  13%|#3        | 51200/386470 [00:37<03:53, 1434.38 examples/s]
hf-map kk (num_proc=8):  14%|#3        | 52224/386470 [00:38<03:54, 1422.81 examples/s]
hf-map kk (num_proc=8):  14%|#3        | 53248/386470 [00:38<03:06, 1781.98 examples/s]
hf-map kk (num_proc=8):  14%|#4        | 54272/386470 [00:38<02:30, 2202.27 examples/s]
hf-map kk (num_proc=8):  14%|#4        | 55296/386470 [00:39<01:58, 2801.37 examples/s]
hf-map kk (num_proc=8):  15%|#5        | 58368/386470 [00:41<03:18, 1650.80 examples/s]
hf-map kk (num_proc=8):  15%|#5        | 59392/386470 [00:41<03:03, 1784.32 examples/s]
hf-map kk (num_proc=8):  16%|#5        | 60416/386470 [00:42<02:58, 1827.34 examples/s]
hf-map kk (num_proc=8):  16%|#5        | 61440/386470 [00:42<02:38, 2048.37 examples/s]
hf-map kk (num_proc=8):  16%|#6        | 63488/386470 [00:42<01:45, 3068.16 examples/s]
hf-map kk (num_proc=8):  17%|#7        | 66560/386470 [00:45<03:10, 1676.78 examples/s]
hf-map kk (num_proc=8):  17%|#7        | 67584/386470 [00:45<02:48, 1896.82 examples/s]
hf-map kk (num_proc=8):  18%|#7        | 68608/386470 [00:46<02:42, 1952.81 examples/s]
hf-map kk (num_proc=8):  18%|#8        | 69632/386470 [00:47<02:57, 1786.81 examples/s]
hf-map kk (num_proc=8):  18%|#8        | 70656/386470 [00:47<02:24, 2191.52 examples/s]
hf-map kk (num_proc=8):  19%|#8        | 71680/386470 [00:47<01:54, 2740.61 examples/s]
hf-map kk (num_proc=8):  19%|#8        | 72704/386470 [00:47<01:33, 3361.10 examples/s]
hf-map kk (num_proc=8):  19%|#9        | 73728/386470 [00:48<01:56, 2674.92 examples/s]
hf-map kk (num_proc=8):  19%|#9        | 74752/386470 [00:50<04:07, 1258.36 examples/s]
hf-map kk (num_proc=8):  20%|#9        | 75776/386470 [00:50<03:29, 1480.42 examples/s]
hf-map kk (num_proc=8):  20%|#9        | 76800/386470 [00:51<04:07, 1251.26 examples/s]
hf-map kk (num_proc=8):  20%|##        | 77824/386470 [00:51<03:09, 1628.33 examples/s]
hf-map kk (num_proc=8):  21%|##        | 79872/386470 [00:52<02:03, 2479.80 examples/s]
hf-map kk (num_proc=8):  21%|##        | 80896/386470 [00:52<01:40, 3035.51 examples/s]
hf-map kk (num_proc=8):  21%|##1       | 81920/386470 [00:52<02:13, 2273.38 examples/s]
hf-map kk (num_proc=8):  21%|##1       | 82944/386470 [00:54<03:41, 1369.74 examples/s]
hf-map kk (num_proc=8):  22%|##1       | 83968/386470 [00:54<03:04, 1642.25 examples/s]
hf-map kk (num_proc=8):  22%|##1       | 84992/386470 [00:56<04:18, 1167.97 examples/s]
hf-map kk (num_proc=8):  22%|##2       | 86016/386470 [00:56<03:11, 1565.17 examples/s]
hf-map kk (num_proc=8):  23%|##2       | 88064/386470 [00:56<02:07, 2340.19 examples/s]
hf-map kk (num_proc=8):  23%|##3       | 89088/386470 [00:56<01:49, 2710.42 examples/s]
hf-map kk (num_proc=8):  23%|##3       | 90112/386470 [00:57<01:57, 2531.87 examples/s]
hf-map kk (num_proc=8):  24%|##3       | 91136/386470 [00:58<03:23, 1452.17 examples/s]
hf-map kk (num_proc=8):  24%|##3       | 92160/386470 [00:59<03:16, 1500.29 examples/s]
hf-map kk (num_proc=8):  24%|##4       | 93184/386470 [01:00<03:45, 1299.89 examples/s]
hf-map kk (num_proc=8):  25%|##4       | 95232/386470 [01:00<02:10, 2233.47 examples/s]
hf-map kk (num_proc=8):  25%|##4       | 96256/386470 [01:00<01:49, 2643.89 examples/s]
hf-map kk (num_proc=8):  25%|##5       | 97280/386470 [01:01<02:06, 2279.20 examples/s]
hf-map kk (num_proc=8):  25%|##5       | 98304/386470 [01:01<01:54, 2519.03 examples/s]
hf-map kk (num_proc=8):  26%|##5       | 99328/386470 [01:03<03:20, 1429.44 examples/s]
hf-map kk (num_proc=8):  26%|##5       | 100352/386470 [01:03<02:41, 1768.57 examples/s]
hf-map kk (num_proc=8):  26%|##6       | 101376/386470 [01:04<03:28, 1366.41 examples/s]
hf-map kk (num_proc=8):  26%|##6       | 102400/386470 [01:04<02:42, 1749.83 examples/s]
hf-map kk (num_proc=8):  27%|##6       | 103424/386470 [01:05<02:05, 2252.20 examples/s]
hf-map kk (num_proc=8):  27%|##7       | 104448/386470 [01:05<02:01, 2323.63 examples/s]
hf-map kk (num_proc=8):  27%|##7       | 105472/386470 [01:05<01:50, 2545.62 examples/s]
hf-map kk (num_proc=8):  28%|##7       | 106496/386470 [01:06<02:54, 1605.13 examples/s]
hf-map kk (num_proc=8):  28%|##7       | 107520/386470 [01:07<02:53, 1605.30 examples/s]
hf-map kk (num_proc=8):  28%|##8       | 108544/386470 [01:08<02:40, 1734.81 examples/s]
hf-map kk (num_proc=8):  28%|##8       | 109568/386470 [01:09<03:38, 1264.76 examples/s]
hf-map kk (num_proc=8):  29%|##8       | 111616/386470 [01:09<02:16, 2007.83 examples/s]
hf-map kk (num_proc=8):  29%|##9       | 112640/386470 [01:10<02:28, 1849.51 examples/s]
hf-map kk (num_proc=8):  29%|##9       | 113664/386470 [01:10<01:57, 2318.74 examples/s]
hf-map kk (num_proc=8):  30%|##9       | 114688/386470 [01:11<02:05, 2172.54 examples/s]
hf-map kk (num_proc=8):  30%|##9       | 115712/386470 [01:11<01:59, 2256.66 examples/s]
hf-map kk (num_proc=8):  30%|###       | 116736/386470 [01:11<01:47, 2505.84 examples/s]
hf-map kk (num_proc=8):  30%|###       | 117760/386470 [01:12<02:41, 1659.22 examples/s]
hf-map kk (num_proc=8):  31%|###       | 118784/386470 [01:13<02:10, 2044.96 examples/s]
hf-map kk (num_proc=8):  31%|###1      | 119808/386470 [01:13<01:44, 2556.93 examples/s]
hf-map kk (num_proc=8):  31%|###1      | 120832/386470 [01:13<01:45, 2528.11 examples/s]
hf-map kk (num_proc=8):  32%|###1      | 121856/386470 [01:13<01:33, 2821.64 examples/s]
hf-map kk (num_proc=8):  32%|###1      | 122880/386470 [01:14<01:31, 2879.98 examples/s]
hf-map kk (num_proc=8):  32%|###2      | 123904/386470 [01:14<01:22, 3165.58 examples/s]
hf-map kk (num_proc=8):  32%|###2      | 124928/386470 [01:15<01:39, 2641.60 examples/s]
hf-map kk (num_proc=8):  33%|###2      | 125952/386470 [01:16<02:34, 1681.92 examples/s]
hf-map kk (num_proc=8):  33%|###3      | 128000/386470 [01:17<02:09, 2000.00 examples/s]
hf-map kk (num_proc=8):  33%|###3      | 129024/386470 [01:17<02:00, 2138.02 examples/s]
hf-map kk (num_proc=8):  34%|###3      | 131072/386470 [01:18<01:47, 2382.19 examples/s]
hf-map kk (num_proc=8):  34%|###4      | 133120/386470 [01:18<01:42, 2482.40 examples/s]
hf-map kk (num_proc=8):  35%|###4      | 134144/386470 [01:19<02:14, 1872.54 examples/s]
hf-map kk (num_proc=8):  35%|###5      | 136192/386470 [01:21<02:37, 1584.37 examples/s]
hf-map kk (num_proc=8):  36%|###5      | 137216/386470 [01:21<02:10, 1905.17 examples/s]
hf-map kk (num_proc=8):  36%|###6      | 139264/386470 [01:22<01:43, 2398.21 examples/s]
hf-map kk (num_proc=8):  37%|###6      | 141312/386470 [01:22<01:23, 2933.49 examples/s]
hf-map kk (num_proc=8):  37%|###6      | 142336/386470 [01:24<02:30, 1623.72 examples/s]
hf-map kk (num_proc=8):  37%|###7      | 143360/386470 [01:24<02:13, 1824.64 examples/s]
hf-map kk (num_proc=8):  37%|###7      | 144384/386470 [01:25<02:28, 1635.26 examples/s]
hf-map kk (num_proc=8):  38%|###7      | 146432/386470 [01:25<01:33, 2560.74 examples/s]
hf-map kk (num_proc=8):  38%|###8      | 147456/386470 [01:26<01:39, 2408.24 examples/s]
hf-map kk (num_proc=8):  39%|###8      | 149504/386470 [01:26<01:14, 3193.36 examples/s]
hf-map kk (num_proc=8):  39%|###8      | 150528/386470 [01:28<02:46, 1419.27 examples/s]
hf-map kk (num_proc=8):  39%|###9      | 151552/386470 [01:28<02:16, 1717.29 examples/s]
hf-map kk (num_proc=8):  39%|###9      | 152576/386470 [01:29<02:11, 1776.69 examples/s]
hf-map kk (num_proc=8):  40%|###9      | 153600/386470 [01:29<01:50, 2109.57 examples/s]
hf-map kk (num_proc=8):  40%|####      | 155648/386470 [01:29<01:17, 2997.48 examples/s]
hf-map kk (num_proc=8):  41%|####      | 156672/386470 [01:30<01:08, 3347.04 examples/s]
hf-map kk (num_proc=8):  41%|####      | 157696/386470 [01:30<01:05, 3518.54 examples/s]
hf-map kk (num_proc=8):  41%|####1     | 158720/386470 [01:32<02:28, 1537.87 examples/s]
hf-map kk (num_proc=8):  41%|####1     | 159744/386470 [01:32<02:19, 1621.54 examples/s]
hf-map kk (num_proc=8):  42%|####1     | 160768/386470 [01:33<02:13, 1684.41 examples/s]
hf-map kk (num_proc=8):  42%|####1     | 161792/386470 [01:33<01:47, 2086.00 examples/s]
hf-map kk (num_proc=8):  42%|####2     | 162816/386470 [01:33<01:23, 2674.55 examples/s]
hf-map kk (num_proc=8):  43%|####2     | 164864/386470 [01:33<00:59, 3720.38 examples/s]
hf-map kk (num_proc=8):  43%|####2     | 165888/386470 [01:33<00:56, 3911.86 examples/s]
hf-map kk (num_proc=8):  43%|####3     | 166912/386470 [01:36<02:38, 1389.49 examples/s]
hf-map kk (num_proc=8):  43%|####3     | 167936/386470 [01:36<02:31, 1438.10 examples/s]
hf-map kk (num_proc=8):  44%|####3     | 168960/386470 [01:36<02:08, 1693.12 examples/s]
hf-map kk (num_proc=8):  44%|####3     | 169984/386470 [01:37<02:08, 1679.21 examples/s]
hf-map kk (num_proc=8):  45%|####4     | 172032/386470 [01:37<01:25, 2500.79 examples/s]
hf-map kk (num_proc=8):  45%|####5     | 174080/386470 [01:38<01:01, 3457.88 examples/s]
hf-map kk (num_proc=8):  45%|####5     | 175104/386470 [01:40<02:10, 1625.46 examples/s]
hf-map kk (num_proc=8):  46%|####5     | 176128/386470 [01:40<02:11, 1604.42 examples/s]
hf-map kk (num_proc=8):  46%|####5     | 177152/386470 [01:40<01:46, 1960.61 examples/s]
hf-map kk (num_proc=8):  46%|####6     | 178176/386470 [01:41<01:55, 1806.53 examples/s]
hf-map kk (num_proc=8):  46%|####6     | 179200/386470 [01:41<01:29, 2316.46 examples/s]
hf-map kk (num_proc=8):  47%|####6     | 180224/386470 [01:41<01:16, 2698.09 examples/s]
hf-map kk (num_proc=8):  47%|####6     | 181248/386470 [01:42<01:07, 3036.98 examples/s]
hf-map kk (num_proc=8):  47%|####7     | 183296/386470 [01:43<01:59, 1698.61 examples/s]
hf-map kk (num_proc=8):  48%|####7     | 184320/386470 [01:44<01:59, 1689.83 examples/s]
hf-map kk (num_proc=8):  48%|####7     | 185344/386470 [01:44<01:46, 1897.17 examples/s]
hf-map kk (num_proc=8):  48%|####8     | 186368/386470 [01:45<01:29, 2236.37 examples/s]
hf-map kk (num_proc=8):  48%|####8     | 187392/386470 [01:45<01:28, 2254.42 examples/s]
hf-map kk (num_proc=8):  49%|####8     | 188416/386470 [01:45<01:12, 2733.52 examples/s]
hf-map kk (num_proc=8):  49%|####9     | 189440/386470 [01:46<01:16, 2592.13 examples/s]
hf-map kk (num_proc=8):  50%|####9     | 191488/386470 [01:47<01:46, 1839.04 examples/s]
hf-map kk (num_proc=8):  50%|####9     | 192512/386470 [01:48<01:42, 1892.19 examples/s]
hf-map kk (num_proc=8):  50%|#####     | 193536/386470 [01:48<01:50, 1744.98 examples/s]
hf-map kk (num_proc=8):  51%|#####     | 195584/386470 [01:49<01:13, 2603.74 examples/s]
hf-map kk (num_proc=8):  51%|#####     | 196608/386470 [01:49<01:02, 3013.80 examples/s]
hf-map kk (num_proc=8):  51%|#####1    | 197632/386470 [01:49<01:14, 2523.39 examples/s]
hf-map kk (num_proc=8):  51%|#####1    | 198656/386470 [01:50<01:22, 2287.22 examples/s]
hf-map kk (num_proc=8):  52%|#####1    | 199680/386470 [01:51<01:45, 1764.08 examples/s]
hf-map kk (num_proc=8):  52%|#####1    | 200704/386470 [01:52<01:51, 1672.10 examples/s]
hf-map kk (num_proc=8):  52%|#####2    | 201728/386470 [01:52<01:43, 1781.80 examples/s]
hf-map kk (num_proc=8):  52%|#####2    | 202752/386470 [01:52<01:26, 2129.64 examples/s]
hf-map kk (num_proc=8):  53%|#####2    | 204800/386470 [01:53<00:57, 3135.82 examples/s]
hf-map kk (num_proc=8):  53%|#####3    | 205824/386470 [01:53<01:12, 2487.21 examples/s]
hf-map kk (num_proc=8):  54%|#####3    | 206848/386470 [01:54<01:06, 2713.48 examples/s]
hf-map kk (num_proc=8):  54%|#####3    | 207872/386470 [01:54<01:25, 2084.91 examples/s]
hf-map kk (num_proc=8):  54%|#####4    | 208896/386470 [01:55<01:23, 2117.75 examples/s]
hf-map kk (num_proc=8):  54%|#####4    | 209920/386470 [01:55<01:16, 2316.70 examples/s]
hf-map kk (num_proc=8):  55%|#####4    | 210944/386470 [01:56<01:15, 2326.16 examples/s]
hf-map kk (num_proc=8):  55%|#####5    | 212992/386470 [01:56<00:55, 3121.21 examples/s]
hf-map kk (num_proc=8):  55%|#####5    | 214016/386470 [01:56<00:47, 3654.48 examples/s]
hf-map kk (num_proc=8):  56%|#####5    | 215040/386470 [01:57<00:52, 3292.80 examples/s]
hf-map kk (num_proc=8):  56%|#####5    | 216064/386470 [01:57<01:13, 2333.98 examples/s]
hf-map kk (num_proc=8):  56%|#####6    | 217088/386470 [01:58<01:24, 2004.03 examples/s]
hf-map kk (num_proc=8):  56%|#####6    | 218112/386470 [01:59<01:24, 1999.89 examples/s]
hf-map kk (num_proc=8):  57%|#####6    | 219136/386470 [01:59<01:04, 2575.85 examples/s]
hf-map kk (num_proc=8):  57%|#####6    | 220160/386470 [01:59<01:05, 2558.43 examples/s]
hf-map kk (num_proc=8):  57%|#####7    | 221184/386470 [02:00<01:18, 2095.46 examples/s]
hf-map kk (num_proc=8):  57%|#####7    | 222208/386470 [02:00<01:22, 1989.70 examples/s]
hf-map kk (num_proc=8):  58%|#####8    | 224256/386470 [02:01<00:58, 2782.42 examples/s]
hf-map kk (num_proc=8):  58%|#####8    | 225280/386470 [02:02<01:22, 1963.19 examples/s]
hf-map kk (num_proc=8):  59%|#####9    | 228352/386470 [02:02<00:47, 3334.71 examples/s]
hf-map kk (num_proc=8):  59%|#####9    | 229376/386470 [02:03<01:03, 2471.26 examples/s]
hf-map kk (num_proc=8):  60%|#####9    | 231424/386470 [02:03<00:51, 3023.97 examples/s]
hf-map kk (num_proc=8):  60%|######    | 232448/386470 [02:04<00:50, 3021.44 examples/s]
hf-map kk (num_proc=8):  60%|######    | 233472/386470 [02:05<01:17, 1962.17 examples/s]
hf-map kk (num_proc=8):  61%|######    | 235520/386470 [02:05<00:51, 2924.89 examples/s]
hf-map kk (num_proc=8):  61%|######1   | 236544/386470 [02:06<00:59, 2525.54 examples/s]
hf-map kk (num_proc=8):  61%|######1   | 237568/386470 [02:06<00:53, 2764.52 examples/s]
hf-map kk (num_proc=8):  62%|######1   | 238592/386470 [02:06<00:52, 2832.40 examples/s]
hf-map kk (num_proc=8):  62%|######2   | 239616/386470 [02:06<00:42, 3460.53 examples/s]
hf-map kk (num_proc=8):  62%|######2   | 240640/386470 [02:07<00:39, 3690.87 examples/s]
hf-map kk (num_proc=8):  63%|######2   | 241664/386470 [02:08<01:09, 2082.18 examples/s]
hf-map kk (num_proc=8):  63%|######3   | 243712/386470 [02:08<00:47, 2999.04 examples/s]
hf-map kk (num_proc=8):  63%|######3   | 244736/386470 [02:08<00:47, 2999.79 examples/s]
hf-map kk (num_proc=8):  64%|######3   | 245760/386470 [02:09<00:55, 2531.56 examples/s]
hf-map kk (num_proc=8):  64%|######3   | 246784/386470 [02:09<00:52, 2677.06 examples/s]
hf-map kk (num_proc=8):  64%|######4   | 248832/386470 [02:09<00:34, 3958.21 examples/s]
hf-map kk (num_proc=8):  65%|######4   | 249856/386470 [02:10<00:59, 2290.33 examples/s]
hf-map kk (num_proc=8):  65%|######4   | 250880/386470 [02:11<00:56, 2403.84 examples/s]
hf-map kk (num_proc=8):  65%|######5   | 252928/386470 [02:11<00:36, 3670.20 examples/s]
hf-map kk (num_proc=8):  66%|######5   | 253952/386470 [02:12<00:58, 2266.71 examples/s]
hf-map kk (num_proc=8):  66%|######5   | 254976/386470 [02:12<00:47, 2779.05 examples/s]
hf-map kk (num_proc=8):  66%|######6   | 256000/386470 [02:12<00:42, 3074.93 examples/s]
hf-map kk (num_proc=8):  67%|######6   | 257024/386470 [02:13<00:38, 3326.93 examples/s]
hf-map kk (num_proc=8):  67%|######6   | 258048/386470 [02:13<00:58, 2212.12 examples/s]
hf-map kk (num_proc=8):  67%|######7   | 259072/386470 [02:14<00:49, 2577.53 examples/s]
hf-map kk (num_proc=8):  67%|######7   | 260096/386470 [02:14<00:41, 3067.55 examples/s]
hf-map kk (num_proc=8):  68%|######7   | 261120/386470 [02:14<00:38, 3278.53 examples/s]
hf-map kk (num_proc=8):  68%|######7   | 262144/386470 [02:15<00:57, 2156.06 examples/s]
hf-map kk (num_proc=8):  68%|######8   | 263168/386470 [02:15<00:44, 2779.57 examples/s]
hf-map kk (num_proc=8):  69%|######8   | 265216/386470 [02:16<00:36, 3289.25 examples/s]
hf-map kk (num_proc=8):  69%|######8   | 266240/386470 [02:16<00:50, 2372.35 examples/s]
hf-map kk (num_proc=8):  69%|######9   | 267264/386470 [02:16<00:42, 2824.62 examples/s]
hf-map kk (num_proc=8):  70%|######9   | 269312/386470 [02:17<00:42, 2745.33 examples/s]
hf-map kk (num_proc=8):  70%|######9   | 270336/386470 [02:18<00:40, 2867.67 examples/s]
hf-map kk (num_proc=8):  70%|#######   | 271360/386470 [02:18<00:36, 3151.80 examples/s]
hf-map kk (num_proc=8):  70%|#######   | 272384/386470 [02:18<00:32, 3486.39 examples/s]
hf-map kk (num_proc=8):  71%|#######   | 273408/386470 [02:19<00:39, 2844.66 examples/s]
hf-map kk (num_proc=8):  71%|#######1  | 274432/386470 [02:19<00:45, 2468.48 examples/s]
hf-map kk (num_proc=8):  72%|#######1  | 276480/386470 [02:19<00:31, 3529.64 examples/s]
hf-map kk (num_proc=8):  72%|#######1  | 277504/386470 [02:20<00:42, 2561.35 examples/s]
hf-map kk (num_proc=8):  72%|#######2  | 278528/386470 [02:21<00:46, 2319.69 examples/s]
hf-map kk (num_proc=8):  72%|#######2  | 279552/386470 [02:21<00:37, 2843.15 examples/s]
hf-map kk (num_proc=8):  73%|#######2  | 280576/386470 [02:21<00:35, 2950.65 examples/s]
hf-map kk (num_proc=8):  73%|#######2  | 281600/386470 [02:22<00:48, 2151.63 examples/s]
hf-map kk (num_proc=8):  73%|#######3  | 282624/386470 [02:22<00:47, 2194.61 examples/s]
hf-map kk (num_proc=8):  74%|#######3  | 284672/386470 [02:23<00:44, 2309.83 examples/s]
hf-map kk (num_proc=8):  74%|#######3  | 285696/386470 [02:23<00:38, 2599.00 examples/s]
hf-map kk (num_proc=8):  74%|#######4  | 286720/386470 [02:24<00:46, 2149.70 examples/s]
hf-map kk (num_proc=8):  75%|#######4  | 288768/386470 [02:24<00:31, 3119.71 examples/s]
hf-map kk (num_proc=8):  75%|#######4  | 289792/386470 [02:25<00:39, 2478.25 examples/s]
hf-map kk (num_proc=8):  75%|#######5  | 290816/386470 [02:26<00:41, 2320.99 examples/s]
hf-map kk (num_proc=8):  76%|#######5  | 291840/386470 [02:26<00:34, 2759.10 examples/s]
hf-map kk (num_proc=8):  76%|#######5  | 292864/386470 [02:27<00:49, 1877.04 examples/s]
hf-map kk (num_proc=8):  76%|#######6  | 293888/386470 [02:27<00:46, 2005.32 examples/s]
hf-map kk (num_proc=8):  76%|#######6  | 294912/386470 [02:27<00:38, 2354.69 examples/s]
hf-map kk (num_proc=8):  77%|#######6  | 296960/386470 [02:28<00:25, 3535.96 examples/s]
hf-map kk (num_proc=8):  77%|#######7  | 297984/386470 [02:29<00:45, 1961.59 examples/s]
hf-map kk (num_proc=8):  77%|#######7  | 299008/386470 [02:29<00:38, 2262.37 examples/s]
hf-map kk (num_proc=8):  78%|#######7  | 300032/386470 [02:29<00:31, 2740.83 examples/s]
hf-map kk (num_proc=8):  78%|#######7  | 301056/386470 [02:30<00:46, 1838.79 examples/s]
hf-map kk (num_proc=8):  78%|#######8  | 302080/386470 [02:31<00:43, 1943.96 examples/s]
hf-map kk (num_proc=8):  79%|#######8  | 304128/386470 [02:31<00:27, 2955.88 examples/s]
hf-map kk (num_proc=8):  79%|#######9  | 306176/386470 [02:32<00:35, 2278.73 examples/s]
hf-map kk (num_proc=8):  79%|#######9  | 307200/386470 [02:33<00:33, 2367.77 examples/s]
hf-map kk (num_proc=8):  80%|########  | 309248/386470 [02:34<00:36, 2097.21 examples/s]
hf-map kk (num_proc=8):  80%|########  | 310272/386470 [02:34<00:31, 2439.46 examples/s]
hf-map kk (num_proc=8):  81%|########  | 311296/386470 [02:34<00:30, 2425.59 examples/s]
hf-map kk (num_proc=8):  81%|########  | 312320/386470 [02:35<00:25, 2853.84 examples/s]
hf-map kk (num_proc=8):  81%|########1 | 313344/386470 [02:35<00:24, 3032.80 examples/s]
hf-map kk (num_proc=8):  81%|########1 | 314368/386470 [02:36<00:32, 2249.82 examples/s]
hf-map kk (num_proc=8):  82%|########1 | 315392/386470 [02:36<00:31, 2289.83 examples/s]
hf-map kk (num_proc=8):  82%|########1 | 316416/386470 [02:36<00:25, 2737.53 examples/s]
hf-map kk (num_proc=8):  82%|########2 | 317440/386470 [02:37<00:38, 1801.10 examples/s]
hf-map kk (num_proc=8):  82%|########2 | 318464/386470 [02:37<00:28, 2366.06 examples/s]
hf-map kk (num_proc=8):  83%|########2 | 319488/386470 [02:38<00:30, 2224.95 examples/s]
hf-map kk (num_proc=8):  83%|########2 | 320512/386470 [02:38<00:28, 2292.58 examples/s]
hf-map kk (num_proc=8):  83%|########3 | 321536/386470 [02:39<00:25, 2519.08 examples/s]
hf-map kk (num_proc=8):  83%|########3 | 322560/386470 [02:39<00:22, 2885.82 examples/s]
hf-map kk (num_proc=8):  84%|########3 | 323584/386470 [02:40<00:26, 2332.98 examples/s]
hf-map kk (num_proc=8):  84%|########3 | 324608/386470 [02:40<00:21, 2862.58 examples/s]
hf-map kk (num_proc=8):  84%|########4 | 325632/386470 [02:41<00:29, 2039.20 examples/s]
hf-map kk (num_proc=8):  85%|########4 | 326656/386470 [02:41<00:27, 2154.18 examples/s]
hf-map kk (num_proc=8):  85%|########4 | 327680/386470 [02:42<00:31, 1846.96 examples/s]
hf-map kk (num_proc=8):  85%|########5 | 329728/386470 [02:42<00:21, 2580.03 examples/s]
hf-map kk (num_proc=8):  86%|########5 | 331776/386470 [02:43<00:17, 3063.70 examples/s]
hf-map kk (num_proc=8):  86%|########6 | 332800/386470 [02:43<00:17, 2995.79 examples/s]
hf-map kk (num_proc=8):  86%|########6 | 333824/386470 [02:43<00:17, 3048.10 examples/s]
hf-map kk (num_proc=8):  87%|########6 | 334848/386470 [02:44<00:18, 2813.75 examples/s]
hf-map kk (num_proc=8):  87%|########6 | 335872/386470 [02:44<00:22, 2253.03 examples/s]
hf-map kk (num_proc=8):  87%|########7 | 336896/386470 [02:45<00:19, 2561.03 examples/s]
hf-map kk (num_proc=8):  87%|########7 | 337920/386470 [02:45<00:17, 2752.50 examples/s]
hf-map kk (num_proc=8):  88%|########7 | 338944/386470 [02:45<00:13, 3440.64 examples/s]
hf-map kk (num_proc=8):  88%|########7 | 339968/386470 [02:45<00:13, 3356.23 examples/s]
hf-map kk (num_proc=8):  88%|########8 | 340992/386470 [02:46<00:14, 3058.95 examples/s]
hf-map kk (num_proc=8):  88%|########8 | 342016/386470 [02:46<00:12, 3614.85 examples/s]
hf-map kk (num_proc=8):  89%|########8 | 343040/386470 [02:47<00:19, 2177.11 examples/s]
hf-map kk (num_proc=8):  89%|########9 | 344064/386470 [02:47<00:20, 2050.66 examples/s]
hf-map kk (num_proc=8):  89%|########9 | 345088/386470 [02:48<00:21, 1924.65 examples/s]
hf-map kk (num_proc=8):  90%|########9 | 346112/386470 [02:48<00:17, 2358.66 examples/s]
hf-map kk (num_proc=8):  90%|######### | 348160/386470 [02:49<00:17, 2180.46 examples/s]
hf-map kk (num_proc=8):  91%|######### | 350208/386470 [02:49<00:11, 3287.72 examples/s]
hf-map kk (num_proc=8):  91%|######### | 351232/386470 [02:50<00:15, 2340.50 examples/s]
hf-map kk (num_proc=8):  91%|#########1| 352256/386470 [02:51<00:18, 1872.38 examples/s]
hf-map kk (num_proc=8):  91%|#########1| 353280/386470 [02:52<00:16, 2057.12 examples/s]
hf-map kk (num_proc=8):  92%|#########1| 354304/386470 [02:52<00:13, 2328.13 examples/s]
hf-map kk (num_proc=8):  92%|#########1| 355328/386470 [02:52<00:12, 2546.34 examples/s]
hf-map kk (num_proc=8):  92%|#########2| 356352/386470 [02:53<00:12, 2367.92 examples/s]
hf-map kk (num_proc=8):  92%|#########2| 357376/386470 [02:53<00:10, 2834.44 examples/s]
hf-map kk (num_proc=8):  93%|#########2| 358400/386470 [02:53<00:08, 3166.16 examples/s]
hf-map kk (num_proc=8):  93%|#########3| 359424/386470 [02:54<00:15, 1731.75 examples/s]
hf-map kk (num_proc=8):  93%|#########3| 360448/386470 [02:55<00:15, 1630.11 examples/s]
hf-map kk (num_proc=8):  94%|#########3| 361472/386470 [02:55<00:12, 1930.69 examples/s]
hf-map kk (num_proc=8):  94%|#########4| 363520/386470 [02:56<00:07, 3163.99 examples/s]
hf-map kk (num_proc=8):  94%|#########4| 364544/386470 [02:56<00:10, 2116.37 examples/s]
hf-map kk (num_proc=8):  95%|#########4| 365568/386470 [02:57<00:08, 2375.14 examples/s]
hf-map kk (num_proc=8):  95%|#########4| 366592/386470 [02:57<00:06, 2843.03 examples/s]
hf-map kk (num_proc=8):  95%|#########5| 367616/386470 [02:58<00:09, 2027.47 examples/s]
hf-map kk (num_proc=8):  95%|#########5| 368640/386470 [02:58<00:08, 2101.95 examples/s]
hf-map kk (num_proc=8):  96%|#########5| 369664/386470 [02:59<00:09, 1705.45 examples/s]
hf-map kk (num_proc=8):  96%|#########6| 371712/386470 [03:00<00:06, 2271.98 examples/s]
hf-map kk (num_proc=8):  97%|#########6| 373760/386470 [03:00<00:04, 2579.25 examples/s]
hf-map kk (num_proc=8):  97%|#########7| 375808/386470 [03:01<00:04, 2357.42 examples/s]
hf-map kk (num_proc=8):  98%|#########7| 376832/386470 [03:02<00:03, 2485.30 examples/s]
hf-map kk (num_proc=8):  98%|#########7| 378037/386470 [03:03<00:04, 1837.39 examples/s]
hf-map kk (num_proc=8):  98%|#########8| 380085/386470 [03:03<00:02, 2688.16 examples/s]
hf-map kk (num_proc=8):  99%|#########8| 381109/386470 [03:03<00:01, 2971.22 examples/s]
hf-map kk (num_proc=8):  99%|#########8| 382314/386470 [03:04<00:01, 2874.08 examples/s]
hf-map kk (num_proc=8):  99%|#########9| 383519/386470 [03:04<00:00, 3124.14 examples/s]
hf-map kk (num_proc=8): 100%|#########9| 384724/386470 [03:05<00:00, 2752.42 examples/s]
hf-map kk (num_proc=8): 100%|#########9| 385085/386470 [03:05<00:00, 2517.09 examples/s]
hf-map kk (num_proc=8): 100%|#########9| 386290/386470 [03:06<00:00, 1690.01 examples/s]
hf-map kk (num_proc=8): 100%|##########| 386470/386470 [03:07<00:00, 2059.14 examples/s]
    kk: datasets.map num_proc=8 done 188s
    kk: 25000 docs  40985213 tokens  194s
    kk: 50000 docs  81894926 tokens  198s
    kk: 75000 docs  121953883 tokens  201s
    kk: 100000 docs  163346527 tokens  204s
    kk: 125000 docs  204439098 tokens  208s
    kk: 150000 docs  245100472 tokens  211s
    kk: 175000 docs  284907275 tokens  214s
    kk: 200000 docs  326211845 tokens  218s
    kk: 225000 docs  368265583 tokens  221s
    kk: 250000 docs  408948851 tokens  224s
    kk: 275000 docs  449889902 tokens  228s
  packed kk: 466196065 tokens (1.000 of budget)  rows=602321 docs=285249  pad-per-doc (HF map)
    tl: datasets.map n=50150 batched=1024 num_proc=8

hf-map tl (num_proc=8):   0%|          | 0/50150 [00:00<?, ? examples/s]
hf-map tl (num_proc=8):   2%|2         | 1024/50150 [00:05<04:36, 177.77 examples/s]
hf-map tl (num_proc=8):   6%|6         | 3072/50150 [00:05<01:12, 653.14 examples/s]
hf-map tl (num_proc=8):  12%|#2        | 6144/50150 [00:06<00:26, 1633.64 examples/s]
hf-map tl (num_proc=8):  16%|#6        | 8192/50150 [00:06<00:17, 2395.63 examples/s]
hf-map tl (num_proc=8):  20%|##        | 10240/50150 [00:08<00:24, 1629.19 examples/s]
hf-map tl (num_proc=8):  25%|##4       | 12288/50150 [00:08<00:16, 2229.91 examples/s]
hf-map tl (num_proc=8):  31%|###       | 15360/50150 [00:08<00:09, 3570.79 examples/s]
hf-map tl (num_proc=8):  35%|###4      | 17408/50150 [00:10<00:15, 2133.75 examples/s]
hf-map tl (num_proc=8):  39%|###8      | 19456/50150 [00:10<00:10, 2816.14 examples/s]
hf-map tl (num_proc=8):  43%|####2     | 21504/50150 [00:10<00:07, 3652.85 examples/s]
hf-map tl (num_proc=8):  47%|####6     | 23552/50150 [00:10<00:05, 4718.59 examples/s]
hf-map tl (num_proc=8):  51%|#####1    | 25600/50150 [00:12<00:09, 2671.26 examples/s]
hf-map tl (num_proc=8):  53%|#####3    | 26624/50150 [00:12<00:08, 2887.76 examples/s]
hf-map tl (num_proc=8):  57%|#####7    | 28672/50150 [00:12<00:05, 3900.87 examples/s]
hf-map tl (num_proc=8):  61%|######1   | 30720/50150 [00:13<00:03, 4927.04 examples/s]
hf-map tl (num_proc=8):  63%|######3   | 31744/50150 [00:13<00:03, 5170.02 examples/s]
hf-map tl (num_proc=8):  67%|######7   | 33792/50150 [00:14<00:06, 2675.14 examples/s]
hf-map tl (num_proc=8):  71%|#######1  | 35840/50150 [00:15<00:04, 3225.36 examples/s]
hf-map tl (num_proc=8):  78%|#######7  | 38912/50150 [00:15<00:02, 4577.65 examples/s]
hf-map tl (num_proc=8):  80%|#######9  | 39936/50150 [00:15<00:02, 4862.49 examples/s]
hf-map tl (num_proc=8):  84%|########3 | 41984/50150 [00:16<00:02, 3331.14 examples/s]
hf-map tl (num_proc=8):  86%|########5 | 43008/50150 [00:16<00:01, 3783.25 examples/s]
hf-map tl (num_proc=8):  88%|########8 | 44282/50150 [00:17<00:01, 3431.20 examples/s]
hf-map tl (num_proc=8):  94%|#########4| 47354/50150 [00:17<00:00, 5266.06 examples/s]
hf-map tl (num_proc=8):  96%|#########6| 48378/50150 [00:17<00:00, 5631.04 examples/s]
hf-map tl (num_proc=8):  99%|#########9| 49651/50150 [00:17<00:00, 6472.24 examples/s]
hf-map tl (num_proc=8): 100%|##########| 50150/50150 [00:18<00:00, 2766.55 examples/s]
    tl: datasets.map num_proc=8 done 18s
    tl: 25000 docs  22165685 tokens  21s
  packed tl: 32043235 tokens (1.000 of budget)  rows=55838 docs=36428  pad-per-doc (HF map)
    en: datasets.map n=196400 batched=1024 num_proc=8

hf-map en (num_proc=8):   0%|          | 0/196400 [00:00<?, ? examples/s]
hf-map en (num_proc=8):   1%|          | 1024/196400 [00:11<34:59, 93.05 examples/s]
hf-map en (num_proc=8):   3%|3         | 6144/196400 [00:11<04:15, 744.78 examples/s]
hf-map en (num_proc=8):   5%|5         | 10240/196400 [00:15<03:40, 842.49 examples/s]
hf-map en (num_proc=8):   6%|6         | 12288/196400 [00:15<02:43, 1125.82 examples/s]
hf-map en (num_proc=8):   7%|7         | 14336/196400 [00:15<02:00, 1515.76 examples/s]
hf-map en (num_proc=8):   8%|8         | 16384/196400 [00:15<01:28, 2033.16 examples/s]
hf-map en (num_proc=8):   9%|9         | 18432/196400 [00:19<02:36, 1140.64 examples/s]
hf-map en (num_proc=8):  10%|#         | 20480/196400 [00:19<01:52, 1558.15 examples/s]
hf-map en (num_proc=8):  13%|#2        | 24576/196400 [00:19<01:03, 2713.98 examples/s]
hf-map en (num_proc=8):  14%|#3        | 26624/196400 [00:23<01:56, 1452.73 examples/s]
hf-map en (num_proc=8):  15%|#5        | 29696/196400 [00:23<01:19, 2100.03 examples/s]
hf-map en (num_proc=8):  17%|#6        | 32768/196400 [00:23<00:55, 2938.07 examples/s]
hf-map en (num_proc=8):  18%|#7        | 34816/196400 [00:26<01:40, 1608.36 examples/s]
hf-map en (num_proc=8):  19%|#8        | 36864/196400 [00:26<01:16, 2075.71 examples/s]
hf-map en (num_proc=8):  20%|#9        | 38912/196400 [00:27<00:59, 2643.47 examples/s]
hf-map en (num_proc=8):  21%|##        | 40960/196400 [00:27<00:45, 3447.32 examples/s]
hf-map en (num_proc=8):  22%|##1       | 43008/196400 [00:30<01:32, 1656.57 examples/s]
hf-map en (num_proc=8):  23%|##3       | 46080/196400 [00:30<00:58, 2566.47 examples/s]
hf-map en (num_proc=8):  25%|##4       | 48128/196400 [00:30<00:45, 3226.55 examples/s]
hf-map en (num_proc=8):  26%|##5       | 50176/196400 [00:32<01:14, 1967.04 examples/s]
hf-map en (num_proc=8):  26%|##6       | 51200/196400 [00:32<01:06, 2190.79 examples/s]
hf-map en (num_proc=8):  27%|##7       | 53248/196400 [00:32<00:47, 3044.22 examples/s]
hf-map en (num_proc=8):  28%|##8       | 55296/196400 [00:32<00:36, 3895.94 examples/s]
hf-map en (num_proc=8):  29%|##9       | 57344/196400 [00:33<00:28, 4934.86 examples/s]
hf-map en (num_proc=8):  30%|###       | 59392/196400 [00:35<01:03, 2163.08 examples/s]
hf-map en (num_proc=8):  32%|###1      | 62464/196400 [00:35<00:41, 3193.31 examples/s]
hf-map en (num_proc=8):  33%|###2      | 64512/196400 [00:35<00:32, 4100.27 examples/s]
hf-map en (num_proc=8):  34%|###3      | 66560/196400 [00:37<00:58, 2224.58 examples/s]
hf-map en (num_proc=8):  35%|###4      | 68608/196400 [00:38<00:47, 2713.46 examples/s]
hf-map en (num_proc=8):  36%|###6      | 71680/196400 [00:38<00:31, 3928.59 examples/s]
hf-map en (num_proc=8):  38%|###7      | 73728/196400 [00:38<00:30, 4082.41 examples/s]
hf-map en (num_proc=8):  38%|###8      | 74752/196400 [00:39<00:45, 2670.07 examples/s]
hf-map en (num_proc=8):  39%|###8      | 75776/196400 [00:40<00:45, 2641.33 examples/s]
hf-map en (num_proc=8):  39%|###9      | 76800/196400 [00:40<00:45, 2642.75 examples/s]
hf-map en (num_proc=8):  40%|###9      | 77824/196400 [00:40<00:37, 3131.09 examples/s]
hf-map en (num_proc=8):  41%|####1     | 80896/196400 [00:40<00:20, 5630.18 examples/s]
hf-map en (num_proc=8):  42%|####2     | 82944/196400 [00:42<00:36, 3136.05 examples/s]
hf-map en (num_proc=8):  43%|####2     | 83968/196400 [00:42<00:38, 2934.84 examples/s]
hf-map en (num_proc=8):  43%|####3     | 84992/196400 [00:42<00:37, 2947.40 examples/s]
hf-map en (num_proc=8):  44%|####4     | 87040/196400 [00:42<00:25, 4246.33 examples/s]
hf-map en (num_proc=8):  45%|####5     | 89088/196400 [00:43<00:20, 5263.78 examples/s]
hf-map en (num_proc=8):  46%|####6     | 91136/196400 [00:44<00:34, 3053.43 examples/s]
hf-map en (num_proc=8):  47%|####6     | 92160/196400 [00:44<00:38, 2727.04 examples/s]
hf-map en (num_proc=8):  47%|####7     | 93184/196400 [00:45<00:37, 2786.27 examples/s]
hf-map en (num_proc=8):  48%|####7     | 94208/196400 [00:45<00:35, 2919.03 examples/s]
hf-map en (num_proc=8):  50%|####9     | 97280/196400 [00:45<00:19, 5159.00 examples/s]
hf-map en (num_proc=8):  50%|#####     | 98304/196400 [00:46<00:21, 4646.30 examples/s]
hf-map en (num_proc=8):  51%|#####     | 99328/196400 [00:46<00:34, 2793.09 examples/s]
hf-map en (num_proc=8):  51%|#####1    | 100352/196400 [00:47<00:39, 2433.96 examples/s]
hf-map en (num_proc=8):  52%|#####1    | 101376/196400 [00:47<00:36, 2602.98 examples/s]
hf-map en (num_proc=8):  52%|#####2    | 102400/196400 [00:47<00:29, 3236.70 examples/s]
hf-map en (num_proc=8):  53%|#####3    | 104448/196400 [00:48<00:19, 4690.60 examples/s]
hf-map en (num_proc=8):  54%|#####4    | 106496/196400 [00:48<00:18, 4905.87 examples/s]
hf-map en (num_proc=8):  55%|#####4    | 107520/196400 [00:49<00:28, 3081.05 examples/s]
hf-map en (num_proc=8):  55%|#####5    | 108544/196400 [00:50<00:40, 2155.37 examples/s]
hf-map en (num_proc=8):  56%|#####5    | 109568/196400 [00:50<00:37, 2314.00 examples/s]
hf-map en (num_proc=8):  57%|#####7    | 112640/196400 [00:50<00:21, 3960.95 examples/s]
hf-map en (num_proc=8):  58%|#####7    | 113664/196400 [00:50<00:18, 4360.06 examples/s]
hf-map en (num_proc=8):  58%|#####8    | 114688/196400 [00:51<00:17, 4607.10 examples/s]
hf-map en (num_proc=8):  59%|#####8    | 115712/196400 [00:52<00:30, 2675.47 examples/s]
hf-map en (num_proc=8):  59%|#####9    | 116736/196400 [00:53<00:44, 1803.34 examples/s]
hf-map en (num_proc=8):  60%|######    | 118784/196400 [00:53<00:26, 2902.19 examples/s]
hf-map en (num_proc=8):  62%|######1   | 120832/196400 [00:53<00:18, 4094.07 examples/s]
hf-map en (num_proc=8):  62%|######2   | 121856/196400 [00:53<00:17, 4231.49 examples/s]
hf-map en (num_proc=8):  63%|######3   | 123904/196400 [00:54<00:25, 2809.30 examples/s]
hf-map en (num_proc=8):  64%|######3   | 124928/196400 [00:55<00:34, 2070.84 examples/s]
hf-map en (num_proc=8):  65%|######4   | 126976/196400 [00:55<00:22, 3072.02 examples/s]
hf-map en (num_proc=8):  66%|######5   | 129024/196400 [00:56<00:15, 4349.35 examples/s]
hf-map en (num_proc=8):  67%|######6   | 131072/196400 [00:56<00:15, 4182.16 examples/s]
hf-map en (num_proc=8):  67%|######7   | 132096/196400 [00:57<00:22, 2852.84 examples/s]
hf-map en (num_proc=8):  68%|######7   | 133120/196400 [00:58<00:29, 2128.36 examples/s]
hf-map en (num_proc=8):  68%|######8   | 134144/196400 [00:58<00:25, 2395.43 examples/s]
hf-map en (num_proc=8):  70%|######9   | 137216/196400 [00:58<00:14, 4216.92 examples/s]
hf-map en (num_proc=8):  70%|#######   | 138240/196400 [00:58<00:12, 4490.24 examples/s]
hf-map en (num_proc=8):  71%|#######   | 139264/196400 [00:59<00:12, 4713.23 examples/s]
hf-map en (num_proc=8):  71%|#######1  | 140288/196400 [01:00<00:27, 2035.65 examples/s]
hf-map en (num_proc=8):  72%|#######1  | 141312/196400 [01:01<00:30, 1832.96 examples/s]
hf-map en (num_proc=8):  72%|#######2  | 142336/196400 [01:01<00:23, 2286.83 examples/s]
hf-map en (num_proc=8):  73%|#######2  | 143360/196400 [01:01<00:18, 2841.71 examples/s]
hf-map en (num_proc=8):  74%|#######4  | 145408/196400 [01:01<00:12, 4207.35 examples/s]
hf-map en (num_proc=8):  75%|#######4  | 146432/196400 [01:02<00:13, 3746.82 examples/s]
hf-map en (num_proc=8):  75%|#######5  | 147456/196400 [01:02<00:11, 4279.29 examples/s]
hf-map en (num_proc=8):  76%|#######5  | 148480/196400 [01:03<00:20, 2288.76 examples/s]
hf-map en (num_proc=8):  76%|#######6  | 149504/196400 [01:03<00:22, 2069.66 examples/s]
hf-map en (num_proc=8):  77%|#######6  | 150528/196400 [01:04<00:18, 2517.61 examples/s]
hf-map en (num_proc=8):  78%|#######8  | 153600/196400 [01:04<00:11, 3621.80 examples/s]
hf-map en (num_proc=8):  79%|#######8  | 154624/196400 [01:04<00:10, 3936.79 examples/s]
hf-map en (num_proc=8):  80%|#######9  | 156672/196400 [01:05<00:14, 2794.69 examples/s]
hf-map en (num_proc=8):  80%|########  | 157696/196400 [01:06<00:15, 2523.16 examples/s]
hf-map en (num_proc=8):  81%|########1 | 159744/196400 [01:06<00:10, 3451.17 examples/s]
hf-map en (num_proc=8):  82%|########1 | 160768/196400 [01:06<00:10, 3534.73 examples/s]
hf-map en (num_proc=8):  82%|########2 | 161792/196400 [01:07<00:08, 4005.08 examples/s]
hf-map en (num_proc=8):  83%|########2 | 162816/196400 [01:07<00:08, 3960.41 examples/s]
hf-map en (num_proc=8):  83%|########3 | 163840/196400 [01:07<00:08, 3848.30 examples/s]
hf-map en (num_proc=8):  84%|########3 | 164864/196400 [01:08<00:15, 2074.71 examples/s]
hf-map en (num_proc=8):  84%|########4 | 165888/196400 [01:09<00:13, 2236.31 examples/s]
hf-map en (num_proc=8):  85%|########4 | 166912/196400 [01:09<00:10, 2877.13 examples/s]
hf-map en (num_proc=8):  86%|########5 | 167936/196400 [01:09<00:09, 2938.64 examples/s]
hf-map en (num_proc=8):  86%|########6 | 168960/196400 [01:09<00:08, 3163.31 examples/s]
hf-map en (num_proc=8):  87%|########7 | 171008/196400 [01:10<00:06, 4063.07 examples/s]
hf-map en (num_proc=8):  88%|########7 | 172032/196400 [01:10<00:06, 3621.43 examples/s]
hf-map en (num_proc=8):  88%|########8 | 173056/196400 [01:11<00:13, 1789.92 examples/s]
hf-map en (num_proc=8):  89%|########8 | 174080/196400 [01:12<00:11, 1867.96 examples/s]
hf-map en (num_proc=8):  89%|########9 | 175104/196400 [01:12<00:09, 2344.26 examples/s]
hf-map en (num_proc=8):  90%|########9 | 176128/196400 [01:12<00:07, 2569.04 examples/s]
hf-map en (num_proc=8):  90%|######### | 177152/196400 [01:12<00:06, 3170.28 examples/s]
hf-map en (num_proc=8):  91%|######### | 178176/196400 [01:13<00:04, 3671.04 examples/s]
hf-map en (num_proc=8):  91%|#########1| 179200/196400 [01:13<00:03, 4395.70 examples/s]
hf-map en (num_proc=8):  92%|#########1| 180224/196400 [01:13<00:03, 4699.68 examples/s]
hf-map en (num_proc=8):  92%|#########2| 181248/196400 [01:14<00:06, 2405.14 examples/s]
hf-map en (num_proc=8):  93%|#########2| 182272/196400 [01:14<00:06, 2320.34 examples/s]
hf-map en (num_proc=8):  93%|#########3| 183296/196400 [01:15<00:05, 2574.40 examples/s]
hf-map en (num_proc=8):  94%|#########3| 184320/196400 [01:15<00:03, 3133.96 examples/s]
hf-map en (num_proc=8):  94%|#########4| 185344/196400 [01:15<00:03, 3526.08 examples/s]
hf-map en (num_proc=8):  95%|#########5| 187392/196400 [01:15<00:02, 3839.55 examples/s]
hf-map en (num_proc=8):  96%|#########5| 188416/196400 [01:16<00:01, 4063.02 examples/s]
hf-map en (num_proc=8):  96%|#########6| 189414/196400 [01:16<00:02, 2719.96 examples/s]
hf-map en (num_proc=8):  97%|#########6| 190412/196400 [01:17<00:02, 2279.52 examples/s]
hf-map en (num_proc=8):  97%|#########7| 191410/196400 [01:17<00:01, 2684.44 examples/s]
hf-map en (num_proc=8):  98%|#########8| 193406/196400 [01:17<00:00, 3711.16 examples/s]
hf-map en (num_proc=8):  99%|#########8| 194404/196400 [01:18<00:00, 4002.99 examples/s]
hf-map en (num_proc=8):  99%|#########9| 195402/196400 [01:18<00:00, 4692.26 examples/s]
hf-map en (num_proc=8): 100%|##########| 196400/196400 [01:18<00:00, 3777.14 examples/s]
hf-map en (num_proc=8): 100%|##########| 196400/196400 [01:18<00:00, 2487.50 examples/s]
    en: datasets.map num_proc=8 done 79s
    en: 25000 docs  19355384 tokens  82s
    en: 50000 docs  38502040 tokens  84s
    en: 75000 docs  57868565 tokens  86s
    en: 100000 docs  77221904 tokens  88s
    en: 125000 docs  96715846 tokens  90s
  packed en: 111592053 tokens (1.000 of budget)  rows=196029 docs=144428  pad-per-doc (HF map)
full mix budget 1115916236 (en=10%); packed 1645162 sequences, 984858 docs, 1115928627 tokens
eval every 100 steps; resume checkpoint-{step} every 500 steps, keep last 2; analysis model-{step} every 1000 steps (weights+tokenizer; upload SeragAmin/gemma_mapping_byteify_cpt_seq1024)
resume research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-4000 after step 4000/12852
data next_row=512000 / 1645162 (mixed shuffle, perm_seed=42)
  am: rows 19525/62355  tokens 15554785/49804449
  ta: rows 226844/728619  tokens 142032738/456292825
  kk: rows 187196/602321  tokens 144792658/466196065
  tl: rows 17436/55838  tokens 10007303/32043235
  en: rows 60999/196029  tokens 34790050/111592053
loading research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-4000
liger OLMo2 kernels (fused linear CE, RMSNorm, SwiGLU, RoPE)

Loading weights:   0%|          | 0/179 [00:00<?, ?it/s]
Loading weights:  23%|##3       | 42/179 [00:00<00:00, 419.03it/s]
Loading weights:  47%|####6     | 84/179 [00:00<00:00, 311.30it/s]
Loading weights:  75%|#######4  | 134/179 [00:00<00:00, 383.65it/s]
Loading weights: 100%|##########| 179/179 [00:00<00:00, 395.03it/s]
gradient_checkpointing=off  liger=on
seed=42  flash SDP  TF32
full_cpt: 1645162 sequences → 12852 steps  batch=16 accum=8  real_tokens=1115858680  liger fused CE, labels=-100 on pad, no attn mask
full_cpt: resume after step 4000/12852  next mixed packed row 512000/1645056  (global shuffle; not per-language iterators)
Backend impl 'liger_kernel.ops.backends._cutedsl.rope' for op 'rope' could not be imported: ImportError: cutedsl backend requires the NVIDIA CUTLASS Python DSL (CuTe DSL). Install it with `pip install nvidia-cutlass-dsl`, or when installing Liger-Kernel use `pip install 'liger-kernel[cutedsl]'`.
Backend impl 'liger_kernel.ops.backends._ascend.rope' for op 'rope' could not be imported: ImportError: cannot import name 'tanh' from 'triton.language.math' (/usr/local/lib/python3.12/dist-packages/triton/language/math.py)
full_cpt step  4010/12852 ( 31.2%)  nll=3.3252  ppl=27.80  grad=0.930  lr=1.62e-05  7319 tok/s  11.52s/it  [01:55<28:17:51]  mem=34.07GB
full_cpt step  4020/12852 ( 31.3%)  nll=3.4189  ppl=30.54  grad=0.953  lr=1.62e-05  7357 tok/s  11.56s/it  [03:51<28:21:21]  mem=34.07GB
full_cpt step  4030/12852 ( 31.4%)  nll=3.2766  ppl=26.48  grad=0.879  lr=1.61e-05  7378 tok/s  11.57s/it  [05:47<28:21:25]  mem=34.07GB
full_cpt step  4040/12852 ( 31.4%)  nll=3.2380  ppl=25.48  grad=0.859  lr=1.61e-05  7403 tok/s  11.58s/it  [07:43<28:20:44]  mem=34.07GB
full_cpt step  4050/12852 ( 31.5%)  nll=3.2843  ppl=26.69  grad=0.902  lr=1.61e-05  7423 tok/s  11.59s/it  [09:39<28:19:43]  mem=34.07GB
full_cpt step  4060/12852 ( 31.6%)  nll=3.4233  ppl=30.67  grad=0.949  lr=1.61e-05  7410 tok/s  11.59s/it  [11:35<28:18:10]  mem=34.07GB
full_cpt step  4070/12852 ( 31.7%)  nll=3.3702  ppl=29.08  grad=0.848  lr=1.61e-05  7449 tok/s  11.59s/it  [13:31<28:16:45]  mem=34.07GB
full_cpt step  4080/12852 ( 31.7%)  nll=3.5208  ppl=33.81  grad=0.965  lr=1.60e-05  7444 tok/s  11.59s/it  [15:27<28:14:48]  mem=34.07GB
full_cpt step  4090/12852 ( 31.8%)  nll=3.3677  ppl=29.01  grad=0.934  lr=1.60e-05  7449 tok/s  11.59s/it  [17:23<28:12:58]  mem=34.07GB
full_cpt step  4100/12852 ( 31.9%)  nll=3.3091  ppl=27.36  grad=0.898  lr=1.60e-05  7456 tok/s  11.59s/it  [19:19<28:11:08]  mem=34.07GB
  am   nll=3.9921  ppl=54.17  bpb=1.4038
  ta   nll=3.9636  ppl=52.65  bpb=0.5757
  kk   nll=2.7664  ppl=15.90  bpb=0.9316
  tl   nll=2.6622  ppl=14.33  bpb=1.2632
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 4100  01:28 (87.7s)
full_cpt step  4110/12852 ( 32.0%)  nll=3.3330  ppl=28.02  grad=0.941  lr=1.60e-05  6984 tok/s  12.39s/it  [22:43<30:05:24]  mem=34.07GB
full_cpt step  4120/12852 ( 32.1%)  nll=3.2017  ppl=24.58  grad=0.895  lr=1.60e-05  7029 tok/s  12.33s/it  [24:39<29:53:50]  mem=34.07GB
full_cpt step  4130/12852 ( 32.1%)  nll=3.3954  ppl=29.83  grad=0.926  lr=1.60e-05  7074 tok/s  12.27s/it  [26:35<29:43:47]  mem=34.07GB
full_cpt step  4140/12852 ( 32.2%)  nll=3.3304  ppl=27.95  grad=0.910  lr=1.59e-05  7093 tok/s  12.22s/it  [28:31<29:34:40]  mem=34.07GB
full_cpt step  4150/12852 ( 32.3%)  nll=3.2906  ppl=26.86  grad=0.832  lr=1.59e-05  7121 tok/s  12.18s/it  [30:27<29:26:40]  mem=34.07GB
full_cpt step  4160/12852 ( 32.4%)  nll=3.2929  ppl=26.92  grad=0.926  lr=1.59e-05  7131 tok/s  12.14s/it  [32:23<29:19:12]  mem=34.07GB
full_cpt step  4170/12852 ( 32.4%)  nll=3.1690  ppl=23.78  grad=0.852  lr=1.59e-05  7149 tok/s  12.11s/it  [34:19<29:12:29]  mem=34.07GB
full_cpt step  4180/12852 ( 32.5%)  nll=3.2696  ppl=26.30  grad=0.938  lr=1.59e-05  7160 tok/s  12.08s/it  [36:15<29:06:16]  mem=34.07GB
full_cpt step  4190/12852 ( 32.6%)  nll=3.3850  ppl=29.52  grad=0.867  lr=1.58e-05  7170 tok/s  12.06s/it  [38:11<29:00:31]  mem=34.07GB
full_cpt step  4200/12852 ( 32.7%)  nll=3.2208  ppl=25.05  grad=0.875  lr=1.58e-05  7177 tok/s  12.03s/it  [40:07<28:55:05]  mem=34.07GB
  am   nll=3.9894  ppl=54.02  bpb=1.4029
  ta   nll=3.9611  ppl=52.52  bpb=0.5753
  kk   nll=2.7642  ppl=15.87  bpb=0.9308
  tl   nll=2.6622  ppl=14.33  bpb=1.2631
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 4200  01:27 (87.4s)
full_cpt step  4210/12852 ( 32.8%)  nll=3.3693  ppl=29.06  grad=0.859  lr=1.58e-05  6948 tok/s  12.43s/it  [43:30<29:49:59]  mem=34.07GB
full_cpt step  4220/12852 ( 32.8%)  nll=3.3484  ppl=28.46  grad=0.879  lr=1.58e-05  6967 tok/s  12.39s/it  [45:26<29:42:27]  mem=34.07GB
full_cpt step  4230/12852 ( 32.9%)  nll=3.3073  ppl=27.31  grad=0.828  lr=1.58e-05  6998 tok/s  12.36s/it  [47:22<29:35:35]  mem=34.07GB
full_cpt step  4240/12852 ( 33.0%)  nll=3.2099  ppl=24.78  grad=0.906  lr=1.57e-05  7014 tok/s  12.32s/it  [49:18<29:29:00]  mem=34.07GB
full_cpt step  4250/12852 ( 33.1%)  nll=3.4038  ppl=30.08  grad=0.879  lr=1.57e-05  7036 tok/s  12.30s/it  [51:14<29:22:49]  mem=34.07GB
full_cpt step  4260/12852 ( 33.1%)  nll=3.4426  ppl=31.27  grad=0.867  lr=1.57e-05  7052 tok/s  12.27s/it  [53:10<29:16:57]  mem=34.07GB
full_cpt step  4270/12852 ( 33.2%)  nll=3.2424  ppl=25.60  grad=0.910  lr=1.57e-05  7065 tok/s  12.24s/it  [55:06<29:11:20]  mem=34.07GB
full_cpt step  4280/12852 ( 33.3%)  nll=3.3108  ppl=27.41  grad=0.863  lr=1.57e-05  7080 tok/s  12.22s/it  [57:02<29:06:01]  mem=34.07GB
full_cpt step  4290/12852 ( 33.4%)  nll=3.2816  ppl=26.62  grad=0.875  lr=1.57e-05  7101 tok/s  12.20s/it  [58:58<29:01:02]  mem=34.07GB
full_cpt step  4300/12852 ( 33.5%)  nll=3.3467  ppl=28.41  grad=0.910  lr=1.56e-05  7112 tok/s  12.18s/it  [1:00:54<28:56:07]  mem=34.07GB
  am   nll=3.9879  ppl=53.94  bpb=1.4024
  ta   nll=3.9598  ppl=52.44  bpb=0.5751
  kk   nll=2.7632  ppl=15.85  bpb=0.9305
  tl   nll=2.6621  ppl=14.33  bpb=1.2631
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 4300  01:28 (87.6s)
full_cpt step  4310/12852 ( 33.5%)  nll=3.3887  ppl=29.63  grad=0.891  lr=1.56e-05  6960 tok/s  12.44s/it  [1:04:18<29:31:36]  mem=34.07GB
full_cpt step  4320/12852 ( 33.6%)  nll=3.2807  ppl=26.59  grad=0.871  lr=1.56e-05  6975 tok/s  12.42s/it  [1:06:14<29:25:46]  mem=34.07GB
full_cpt step  4330/12852 ( 33.7%)  nll=3.4546  ppl=31.65  grad=0.883  lr=1.56e-05  6992 tok/s  12.39s/it  [1:08:10<29:20:12]  mem=34.07GB
full_cpt step  4340/12852 ( 33.8%)  nll=3.3319  ppl=27.99  grad=0.957  lr=1.56e-05  7007 tok/s  12.37s/it  [1:10:06<29:14:51]  mem=34.07GB
full_cpt step  4350/12852 ( 33.8%)  nll=3.3390  ppl=28.19  grad=0.855  lr=1.55e-05  7014 tok/s  12.35s/it  [1:12:01<29:09:35]  mem=34.07GB
full_cpt step  4360/12852 ( 33.9%)  nll=3.3022  ppl=27.17  grad=0.922  lr=1.55e-05  7025 tok/s  12.33s/it  [1:13:57<29:04:33]  mem=34.07GB
full_cpt step  4370/12852 ( 34.0%)  nll=3.2387  ppl=25.50  grad=0.855  lr=1.55e-05  7039 tok/s  12.31s/it  [1:15:53<28:59:44]  mem=34.07GB
full_cpt step  4380/12852 ( 34.1%)  nll=3.2537  ppl=25.88  grad=0.863  lr=1.55e-05  7050 tok/s  12.29s/it  [1:17:49<28:55:04]  mem=34.07GB
full_cpt step  4390/12852 ( 34.2%)  nll=3.3776  ppl=29.30  grad=0.879  lr=1.55e-05  7063 tok/s  12.27s/it  [1:19:46<28:50:33]  mem=34.07GB
full_cpt step  4400/12852 ( 34.2%)  nll=3.2460  ppl=25.69  grad=0.902  lr=1.54e-05  7074 tok/s  12.25s/it  [1:21:42<28:46:09]  mem=34.07GB
  am   nll=3.9865  ppl=53.86  bpb=1.4018
  ta   nll=3.9582  ppl=52.36  bpb=0.5749
  kk   nll=2.7626  ppl=15.84  bpb=0.9303
  tl   nll=2.6618  ppl=14.32  bpb=1.2630
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 4400  01:27 (87.3s)
full_cpt step  4410/12852 ( 34.3%)  nll=3.4419  ppl=31.25  grad=0.945  lr=1.54e-05  6964 tok/s  12.45s/it  [1:25:05<29:11:49]  mem=34.07GB
full_cpt step  4420/12852 ( 34.4%)  nll=3.3409  ppl=28.24  grad=0.840  lr=1.54e-05  6976 tok/s  12.43s/it  [1:27:01<29:06:54]  mem=34.07GB
full_cpt step  4430/12852 ( 34.5%)  nll=3.4550  ppl=31.66  grad=0.891  lr=1.54e-05  6986 tok/s  12.41s/it  [1:28:57<29:02:05]  mem=34.07GB
full_cpt step  4440/12852 ( 34.5%)  nll=3.2877  ppl=26.78  grad=0.965  lr=1.54e-05  6995 tok/s  12.39s/it  [1:30:53<28:57:25]  mem=34.07GB
full_cpt step  4450/12852 ( 34.6%)  nll=3.2810  ppl=26.60  grad=0.883  lr=1.53e-05  7010 tok/s  12.38s/it  [1:32:49<28:52:57]  mem=34.07GB
full_cpt step  4460/12852 ( 34.7%)  nll=3.1600  ppl=23.57  grad=0.891  lr=1.53e-05  7021 tok/s  12.36s/it  [1:34:45<28:48:34]  mem=34.07GB
full_cpt step  4470/12852 ( 34.8%)  nll=3.3485  ppl=28.46  grad=0.879  lr=1.53e-05  7029 tok/s  12.34s/it  [1:36:41<28:44:14]  mem=34.07GB
full_cpt step  4480/12852 ( 34.9%)  nll=3.3101  ppl=27.39  grad=0.891  lr=1.53e-05  7039 tok/s  12.33s/it  [1:38:37<28:40:02]  mem=34.07GB
full_cpt step  4490/12852 ( 34.9%)  nll=3.2536  ppl=25.88  grad=0.883  lr=1.53e-05  7048 tok/s  12.31s/it  [1:40:33<28:35:55]  mem=34.07GB
full_cpt step  4500/12852 ( 35.0%)  nll=3.2665  ppl=26.22  grad=0.848  lr=1.52e-05  7060 tok/s  12.30s/it  [1:42:29<28:31:56]  mem=34.07GB
  am   nll=3.9849  ppl=53.78  bpb=1.4013
  ta   nll=3.9568  ppl=52.29  bpb=0.5747
  kk   nll=2.7613  ppl=15.82  bpb=0.9298
  tl   nll=2.6612  ppl=14.31  bpb=1.2627
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 4500  01:29 (88.7s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.67s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.67s/it]
removing old checkpoint checkpoint-3500
kept last 2 resume checkpoints: checkpoint-4000 checkpoint-4500
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-4500 at step 4500 next_row=576000
full_cpt step  4510/12852 ( 35.1%)  nll=3.2223  ppl=25.09  grad=0.840  lr=1.52e-05  6898 tok/s  12.59s/it  [1:47:01<29:10:19]  mem=34.07GB
full_cpt step  4520/12852 ( 35.2%)  nll=3.3475  ppl=28.43  grad=0.879  lr=1.52e-05  6911 tok/s  12.57s/it  [1:48:57<29:05:39]  mem=34.07GB
full_cpt step  4530/12852 ( 35.2%)  nll=3.2950  ppl=26.98  grad=0.922  lr=1.52e-05  6921 tok/s  12.55s/it  [1:50:53<29:01:03]  mem=34.07GB
full_cpt step  4540/12852 ( 35.3%)  nll=3.3350  ppl=28.08  grad=0.840  lr=1.52e-05  6933 tok/s  12.54s/it  [1:52:49<28:56:33]  mem=34.07GB
full_cpt step  4550/12852 ( 35.4%)  nll=3.4140  ppl=30.39  grad=0.855  lr=1.51e-05  6943 tok/s  12.52s/it  [1:54:45<28:52:08]  mem=34.07GB
full_cpt step  4560/12852 ( 35.5%)  nll=3.3828  ppl=29.45  grad=0.871  lr=1.51e-05  6952 tok/s  12.50s/it  [1:56:41<28:47:47]  mem=34.07GB
full_cpt step  4570/12852 ( 35.6%)  nll=3.3417  ppl=28.27  grad=0.910  lr=1.51e-05  6961 tok/s  12.49s/it  [1:58:37<28:43:33]  mem=34.07GB
full_cpt step  4580/12852 ( 35.6%)  nll=3.3181  ppl=27.61  grad=0.820  lr=1.51e-05  6971 tok/s  12.47s/it  [2:00:33<28:39:23]  mem=34.07GB
full_cpt step  4590/12852 ( 35.7%)  nll=3.2864  ppl=26.75  grad=0.895  lr=1.51e-05  6980 tok/s  12.46s/it  [2:02:30<28:35:19]  mem=34.07GB
full_cpt step  4600/12852 ( 35.8%)  nll=3.3052  ppl=27.25  grad=0.863  lr=1.50e-05  6988 tok/s  12.44s/it  [2:04:26<28:31:18]  mem=34.07GB
  am   nll=3.9841  ppl=53.74  bpb=1.4010
  ta   nll=3.9546  ppl=52.17  bpb=0.5744
  kk   nll=2.7603  ppl=15.81  bpb=0.9295
  tl   nll=2.6611  ppl=14.31  bpb=1.2627
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 4600  01:29 (88.6s)
full_cpt step  4610/12852 ( 35.9%)  nll=3.3785  ppl=29.33  grad=0.895  lr=1.50e-05  6913 tok/s  12.57s/it  [2:07:50<28:47:17]  mem=34.07GB
full_cpt step  4620/12852 ( 35.9%)  nll=3.2436  ppl=25.63  grad=0.855  lr=1.50e-05  6921 tok/s  12.56s/it  [2:09:46<28:43:05]  mem=34.07GB
full_cpt step  4630/12852 ( 36.0%)  nll=3.3338  ppl=28.04  grad=0.840  lr=1.50e-05  6928 tok/s  12.54s/it  [2:11:43<28:38:56]  mem=34.07GB
full_cpt step  4640/12852 ( 36.1%)  nll=3.3128  ppl=27.46  grad=0.832  lr=1.50e-05  6937 tok/s  12.53s/it  [2:13:39<28:34:54]  mem=34.07GB
full_cpt step  4650/12852 ( 36.2%)  nll=3.1552  ppl=23.46  grad=0.906  lr=1.49e-05  6945 tok/s  12.52s/it  [2:15:35<28:30:54]  mem=34.07GB
full_cpt step  4660/12852 ( 36.3%)  nll=3.2342  ppl=25.38  grad=0.867  lr=1.49e-05  6955 tok/s  12.50s/it  [2:17:32<28:26:59]  mem=34.07GB
full_cpt step  4670/12852 ( 36.3%)  nll=3.4462  ppl=31.38  grad=0.855  lr=1.49e-05  6961 tok/s  12.49s/it  [2:19:28<28:23:04]  mem=34.07GB
full_cpt step  4680/12852 ( 36.4%)  nll=3.3392  ppl=28.20  grad=0.871  lr=1.49e-05  6968 tok/s  12.48s/it  [2:21:24<28:19:14]  mem=34.07GB
full_cpt step  4690/12852 ( 36.5%)  nll=3.2266  ppl=25.19  grad=1.125  lr=1.49e-05  6975 tok/s  12.46s/it  [2:23:20<28:15:29]  mem=34.07GB
full_cpt step  4700/12852 ( 36.6%)  nll=3.2964  ppl=27.01  grad=0.887  lr=1.48e-05  6981 tok/s  12.45s/it  [2:25:16<28:11:45]  mem=34.07GB
  am   nll=3.9829  ppl=53.67  bpb=1.4006
  ta   nll=3.9546  ppl=52.17  bpb=0.5744
  kk   nll=2.7592  ppl=15.79  bpb=0.9291
  tl   nll=2.6609  ppl=14.31  bpb=1.2626
  en   nll=2.8179  ppl=16.74  bpb=0.8774
eval full_cpt step 4700  01:30 (90.0s)
full_cpt step  4710/12852 ( 36.6%)  nll=3.1793  ppl=24.03  grad=0.906  lr=1.48e-05  6917 tok/s  12.57s/it  [2:28:42<28:25:17]  mem=34.07GB
full_cpt step  4720/12852 ( 36.7%)  nll=3.1946  ppl=24.40  grad=0.875  lr=1.48e-05  6923 tok/s  12.55s/it  [2:30:38<28:21:22]  mem=34.07GB
full_cpt step  4730/12852 ( 36.8%)  nll=3.4535  ppl=31.61  grad=0.918  lr=1.48e-05  6932 tok/s  12.54s/it  [2:32:34<28:17:32]  mem=34.07GB
full_cpt step  4740/12852 ( 36.9%)  nll=3.3056  ppl=27.27  grad=0.949  lr=1.48e-05  6939 tok/s  12.53s/it  [2:34:31<28:13:45]  mem=34.07GB
full_cpt step  4750/12852 ( 37.0%)  nll=3.2337  ppl=25.37  grad=0.828  lr=1.47e-05  6946 tok/s  12.52s/it  [2:36:27<28:10:02]  mem=34.07GB
full_cpt step  4760/12852 ( 37.0%)  nll=3.3127  ppl=27.46  grad=0.824  lr=1.47e-05  6952 tok/s  12.50s/it  [2:38:23<28:06:22]  mem=34.07GB
full_cpt step  4770/12852 ( 37.1%)  nll=3.2047  ppl=24.65  grad=0.961  lr=1.47e-05  6956 tok/s  12.49s/it  [2:40:19<28:02:44]  mem=34.07GB
full_cpt step  4780/12852 ( 37.2%)  nll=3.2856  ppl=26.72  grad=0.898  lr=1.47e-05  6963 tok/s  12.48s/it  [2:42:15<27:59:10]  mem=34.07GB
full_cpt step  4790/12852 ( 37.3%)  nll=3.3269  ppl=27.85  grad=0.875  lr=1.47e-05  6967 tok/s  12.47s/it  [2:44:12<27:55:36]  mem=34.07GB
full_cpt step  4800/12852 ( 37.3%)  nll=3.2832  ppl=26.66  grad=0.871  lr=1.46e-05  6974 tok/s  12.46s/it  [2:46:08<27:52:05]  mem=34.07GB
  am   nll=3.9824  ppl=53.64  bpb=1.4004
  ta   nll=3.9526  ppl=52.07  bpb=0.5741
  kk   nll=2.7588  ppl=15.78  bpb=0.9290
  tl   nll=2.6609  ppl=14.31  bpb=1.2626
  en   nll=2.8178  ppl=16.74  bpb=0.8773
eval full_cpt step 4800  01:29 (88.9s)
full_cpt step  4810/12852 ( 37.4%)  nll=3.2861  ppl=26.74  grad=0.949  lr=1.46e-05  6919 tok/s  12.56s/it  [2:49:33<28:03:18]  mem=34.07GB
full_cpt step  4820/12852 ( 37.5%)  nll=3.2658  ppl=26.20  grad=0.859  lr=1.46e-05  6925 tok/s  12.55s/it  [2:51:29<27:59:39]  mem=34.07GB
full_cpt step  4830/12852 ( 37.6%)  nll=3.4103  ppl=30.27  grad=0.883  lr=1.46e-05  6932 tok/s  12.54s/it  [2:53:25<27:56:05]  mem=34.07GB
full_cpt step  4840/12852 ( 37.7%)  nll=3.2648  ppl=26.18  grad=0.883  lr=1.46e-05  6936 tok/s  12.53s/it  [2:55:21<27:52:31]  mem=34.07GB
full_cpt step  4850/12852 ( 37.7%)  nll=3.4691  ppl=32.11  grad=0.824  lr=1.45e-05  6943 tok/s  12.51s/it  [2:57:17<27:49:02]  mem=34.07GB
full_cpt step  4860/12852 ( 37.8%)  nll=3.2791  ppl=26.55  grad=0.855  lr=1.45e-05  6949 tok/s  12.50s/it  [2:59:13<27:45:32]  mem=34.07GB
full_cpt step  4870/12852 ( 37.9%)  nll=3.4440  ppl=31.31  grad=0.910  lr=1.45e-05  6955 tok/s  12.49s/it  [3:01:10<27:42:06]  mem=34.07GB
full_cpt step  4880/12852 ( 38.0%)  nll=3.4100  ppl=30.26  grad=0.887  lr=1.45e-05  6960 tok/s  12.48s/it  [3:03:06<27:38:40]  mem=34.07GB
full_cpt step  4890/12852 ( 38.0%)  nll=3.3922  ppl=29.73  grad=0.887  lr=1.45e-05  6964 tok/s  12.47s/it  [3:05:02<27:35:16]  mem=34.07GB
full_cpt step  4900/12852 ( 38.1%)  nll=3.4386  ppl=31.14  grad=0.922  lr=1.44e-05  6968 tok/s  12.46s/it  [3:06:58<27:31:54]  mem=34.07GB
  am   nll=3.9806  ppl=53.55  bpb=1.3998
  ta   nll=3.9517  ppl=52.02  bpb=0.5740
  kk   nll=2.7578  ppl=15.77  bpb=0.9287
  tl   nll=2.6609  ppl=14.31  bpb=1.2625
  en   nll=2.8172  ppl=16.73  bpb=0.8771
eval full_cpt step 4900  01:27 (87.4s)
full_cpt step  4910/12852 ( 38.2%)  nll=3.2977  ppl=27.05  grad=0.875  lr=1.44e-05  6918 tok/s  12.55s/it  [3:10:21<27:41:16]  mem=34.07GB
full_cpt step  4920/12852 ( 38.3%)  nll=3.3037  ppl=27.21  grad=0.852  lr=1.44e-05  6923 tok/s  12.54s/it  [3:12:17<27:37:49]  mem=34.07GB
full_cpt step  4930/12852 ( 38.4%)  nll=3.3750  ppl=29.22  grad=0.871  lr=1.44e-05  6929 tok/s  12.53s/it  [3:14:13<27:34:25]  mem=34.07GB
full_cpt step  4940/12852 ( 38.4%)  nll=3.2844  ppl=26.69  grad=0.836  lr=1.44e-05  6934 tok/s  12.52s/it  [3:16:09<27:31:01]  mem=34.07GB
full_cpt step  4950/12852 ( 38.5%)  nll=3.3602  ppl=28.80  grad=0.914  lr=1.43e-05  6937 tok/s  12.51s/it  [3:18:05<27:27:39]  mem=34.07GB
full_cpt step  4960/12852 ( 38.6%)  nll=3.2561  ppl=25.95  grad=0.859  lr=1.43e-05  6942 tok/s  12.50s/it  [3:20:01<27:24:20]  mem=34.07GB
full_cpt step  4970/12852 ( 38.7%)  nll=3.3281  ppl=27.88  grad=0.855  lr=1.43e-05  6949 tok/s  12.49s/it  [3:21:57<27:21:03]  mem=34.07GB
full_cpt step  4980/12852 ( 38.7%)  nll=3.2947  ppl=26.97  grad=0.922  lr=1.43e-05  6952 tok/s  12.48s/it  [3:23:53<27:17:47]  mem=34.07GB
full_cpt step  4990/12852 ( 38.8%)  nll=3.3853  ppl=29.53  grad=0.938  lr=1.43e-05  6956 tok/s  12.47s/it  [3:25:50<27:14:33]  mem=34.07GB
full_cpt step  5000/12852 ( 38.9%)  nll=3.4153  ppl=30.43  grad=0.852  lr=1.42e-05  6962 tok/s  12.47s/it  [3:27:46<27:11:20]  mem=34.07GB
  am   nll=3.9793  ppl=53.48  bpb=1.3993
  ta   nll=3.9515  ppl=52.01  bpb=0.5739
  kk   nll=2.7571  ppl=15.75  bpb=0.9284
  tl   nll=2.6608  ppl=14.31  bpb=1.2625
  en   nll=2.8178  ppl=16.74  bpb=0.8773
eval full_cpt step 5000  01:27 (87.2s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.29s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.29s/it]
removing old checkpoint checkpoint-4000
kept last 2 resume checkpoints: checkpoint-4500 checkpoint-5000
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-5000 at step 5000 next_row=640000

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:04<00:00,  4.21s/it]
Writing model shards: 100%|##########| 1/1 [00:04<00:00,  4.22s/it]
wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-5000 (weights+expanded tokenizer, no optimizer)
full_cpt step  5010/12852 ( 39.0%)  nll=3.3397  ppl=28.21  grad=0.883  lr=1.42e-05  6865 tok/s  12.64s/it  [3:32:48<27:32:13]  mem=34.07GB
full_cpt step  5020/12852 ( 39.1%)  nll=3.3932  ppl=29.76  grad=0.934  lr=1.42e-05  6870 tok/s  12.63s/it  [3:34:44<27:28:48]  mem=34.07GB
full_cpt step  5030/12852 ( 39.1%)  nll=3.2298  ppl=25.28  grad=0.875  lr=1.42e-05  6876 tok/s  12.62s/it  [3:36:40<27:25:24]  mem=34.07GB
full_cpt step  5040/12852 ( 39.2%)  nll=3.4147  ppl=30.41  grad=0.883  lr=1.42e-05  6883 tok/s  12.61s/it  [3:38:36<27:22:03]  mem=34.07GB
full_cpt step  5050/12852 ( 39.3%)  nll=3.1858  ppl=24.19  grad=0.836  lr=1.41e-05  6889 tok/s  12.60s/it  [3:40:32<27:18:42]  mem=34.07GB
full_cpt step  5060/12852 ( 39.4%)  nll=3.2193  ppl=25.01  grad=0.840  lr=1.41e-05  6894 tok/s  12.59s/it  [3:42:28<27:15:23]  mem=34.07GB
full_cpt step  5070/12852 ( 39.4%)  nll=3.2701  ppl=26.32  grad=0.902  lr=1.41e-05  6897 tok/s  12.58s/it  [3:44:24<27:12:04]  mem=34.07GB
full_cpt step  5080/12852 ( 39.5%)  nll=3.3942  ppl=29.79  grad=0.941  lr=1.41e-05  6903 tok/s  12.57s/it  [3:46:20<27:08:49]  mem=34.07GB
full_cpt step  5090/12852 ( 39.6%)  nll=3.2857  ppl=26.73  grad=0.863  lr=1.40e-05  6910 tok/s  12.57s/it  [3:48:17<27:05:35]  mem=34.07GB
full_cpt step  5100/12852 ( 39.7%)  nll=3.3416  ppl=28.27  grad=1.359  lr=1.40e-05  6914 tok/s  12.56s/it  [3:50:13<27:02:22]  mem=34.07GB
  am   nll=3.9787  ppl=53.45  bpb=1.3991
  ta   nll=3.9504  ppl=51.96  bpb=0.5738
  kk   nll=2.7566  ppl=15.75  bpb=0.9283
  tl   nll=2.6609  ppl=14.31  bpb=1.2625
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 5100  01:28 (87.5s)
full_cpt step  5110/12852 ( 39.8%)  nll=3.4203  ppl=30.58  grad=0.895  lr=1.40e-05  6873 tok/s  12.63s/it  [3:53:36<27:09:18]  mem=34.07GB
full_cpt step  5120/12852 ( 39.8%)  nll=3.3089  ppl=27.35  grad=0.824  lr=1.40e-05  6878 tok/s  12.62s/it  [3:55:32<27:06:01]  mem=34.07GB
full_cpt step  5130/12852 ( 39.9%)  nll=3.3672  ppl=29.00  grad=0.898  lr=1.40e-05  6884 tok/s  12.61s/it  [3:57:28<27:02:46]  mem=34.07GB
full_cpt step  5140/12852 ( 40.0%)  nll=3.2232  ppl=25.11  grad=0.852  lr=1.39e-05  6889 tok/s  12.60s/it  [3:59:24<26:59:32]  mem=34.07GB
full_cpt step  5150/12852 ( 40.1%)  nll=3.3114  ppl=27.42  grad=0.852  lr=1.39e-05  6893 tok/s  12.59s/it  [4:01:20<26:56:20]  mem=34.07GB
full_cpt step  5160/12852 ( 40.1%)  nll=3.3468  ppl=28.41  grad=0.863  lr=1.39e-05  6897 tok/s  12.58s/it  [4:03:16<26:53:08]  mem=34.07GB
full_cpt step  5170/12852 ( 40.2%)  nll=3.3987  ppl=29.93  grad=0.852  lr=1.39e-05  6901 tok/s  12.57s/it  [4:05:12<26:49:58]  mem=34.07GB
full_cpt step  5180/12852 ( 40.3%)  nll=3.3124  ppl=27.45  grad=0.859  lr=1.39e-05  6907 tok/s  12.57s/it  [4:07:08<26:46:50]  mem=34.07GB
full_cpt step  5190/12852 ( 40.4%)  nll=3.3764  ppl=29.27  grad=0.875  lr=1.38e-05  6911 tok/s  12.56s/it  [4:09:05<26:43:43]  mem=34.07GB
full_cpt step  5200/12852 ( 40.5%)  nll=3.3074  ppl=27.31  grad=0.887  lr=1.38e-05  6914 tok/s  12.55s/it  [4:11:00<26:40:36]  mem=34.07GB
  am   nll=3.9780  ppl=53.41  bpb=1.3989
  ta   nll=3.9498  ppl=51.93  bpb=0.5737
  kk   nll=2.7559  ppl=15.73  bpb=0.9280
  tl   nll=2.6605  ppl=14.30  bpb=1.2624
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 5200  01:28 (87.8s)
full_cpt step  5210/12852 ( 40.5%)  nll=3.2817  ppl=26.62  grad=0.855  lr=1.38e-05  6877 tok/s  12.61s/it  [4:14:24<26:46:43]  mem=34.07GB
full_cpt step  5220/12852 ( 40.6%)  nll=3.1863  ppl=24.20  grad=0.844  lr=1.38e-05  6881 tok/s  12.61s/it  [4:16:20<26:43:34]  mem=34.07GB
full_cpt step  5230/12852 ( 40.7%)  nll=3.4377  ppl=31.11  grad=0.867  lr=1.38e-05  6885 tok/s  12.60s/it  [4:18:16<26:40:25]  mem=34.07GB
full_cpt step  5240/12852 ( 40.8%)  nll=3.3745  ppl=29.21  grad=0.852  lr=1.37e-05  6890 tok/s  12.59s/it  [4:20:12<26:37:19]  mem=34.07GB
full_cpt step  5250/12852 ( 40.8%)  nll=3.3498  ppl=28.50  grad=0.887  lr=1.37e-05  6895 tok/s  12.58s/it  [4:22:08<26:34:14]  mem=34.07GB
full_cpt step  5260/12852 ( 40.9%)  nll=3.2113  ppl=24.81  grad=0.879  lr=1.37e-05  6899 tok/s  12.58s/it  [4:24:05<26:31:10]  mem=34.07GB
full_cpt step  5270/12852 ( 41.0%)  nll=3.4013  ppl=30.00  grad=0.863  lr=1.37e-05  6904 tok/s  12.57s/it  [4:26:01<26:28:07]  mem=34.07GB
full_cpt step  5280/12852 ( 41.1%)  nll=3.3208  ppl=27.68  grad=0.863  lr=1.36e-05  6909 tok/s  12.56s/it  [4:27:57<26:25:05]  mem=34.07GB
full_cpt step  5290/12852 ( 41.2%)  nll=3.3225  ppl=27.73  grad=0.910  lr=1.36e-05  6912 tok/s  12.55s/it  [4:29:53<26:22:03]  mem=34.07GB
full_cpt step  5300/12852 ( 41.2%)  nll=3.3065  ppl=27.29  grad=0.879  lr=1.36e-05  6916 tok/s  12.55s/it  [4:31:49<26:19:03]  mem=34.07GB
  am   nll=3.9772  ppl=53.37  bpb=1.3986
  ta   nll=3.9495  ppl=51.91  bpb=0.5736
  kk   nll=2.7554  ppl=15.73  bpb=0.9279
  tl   nll=2.6603  ppl=14.30  bpb=1.2623
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 5300  01:28 (87.9s)
full_cpt step  5310/12852 ( 41.3%)  nll=3.3144  ppl=27.51  grad=0.875  lr=1.36e-05  6883 tok/s  12.61s/it  [4:35:13<26:24:29]  mem=34.07GB
full_cpt step  5320/12852 ( 41.4%)  nll=3.3730  ppl=29.16  grad=0.867  lr=1.36e-05  6887 tok/s  12.60s/it  [4:37:09<26:21:26]  mem=34.07GB
full_cpt step  5330/12852 ( 41.5%)  nll=3.3118  ppl=27.44  grad=0.883  lr=1.35e-05  6891 tok/s  12.59s/it  [4:39:05<26:18:24]  mem=34.07GB
full_cpt step  5340/12852 ( 41.5%)  nll=3.2990  ppl=27.09  grad=0.828  lr=1.35e-05  6895 tok/s  12.58s/it  [4:41:01<26:15:23]  mem=34.07GB
full_cpt step  5350/12852 ( 41.6%)  nll=3.3497  ppl=28.49  grad=0.848  lr=1.35e-05  6900 tok/s  12.58s/it  [4:42:57<26:12:24]  mem=34.07GB
full_cpt step  5360/12852 ( 41.7%)  nll=3.3774  ppl=29.30  grad=0.906  lr=1.35e-05  6903 tok/s  12.57s/it  [4:44:53<26:09:24]  mem=34.07GB
full_cpt step  5370/12852 ( 41.8%)  nll=3.3132  ppl=27.47  grad=0.910  lr=1.35e-05  6906 tok/s  12.56s/it  [4:46:49<26:06:26]  mem=34.07GB
full_cpt step  5380/12852 ( 41.9%)  nll=3.2585  ppl=26.01  grad=0.906  lr=1.34e-05  6909 tok/s  12.55s/it  [4:48:45<26:03:28]  mem=34.07GB
full_cpt step  5390/12852 ( 41.9%)  nll=3.2479  ppl=25.74  grad=0.863  lr=1.34e-05  6913 tok/s  12.55s/it  [4:50:41<26:00:31]  mem=34.07GB
full_cpt step  5400/12852 ( 42.0%)  nll=3.2583  ppl=26.01  grad=0.852  lr=1.34e-05  6916 tok/s  12.54s/it  [4:52:37<25:57:36]  mem=34.07GB
  am   nll=3.9765  ppl=53.33  bpb=1.3984
  ta   nll=3.9485  ppl=51.86  bpb=0.5735
  kk   nll=2.7550  ppl=15.72  bpb=0.9277
  tl   nll=2.6604  ppl=14.30  bpb=1.2623
  en   nll=2.8173  ppl=16.73  bpb=0.8772
eval full_cpt step 5400  01:28 (88.0s)
full_cpt step  5410/12852 ( 42.1%)  nll=3.2110  ppl=24.80  grad=0.902  lr=1.34e-05  6887 tok/s  12.60s/it  [4:56:02<26:02:26]  mem=34.07GB
full_cpt step  5420/12852 ( 42.2%)  nll=3.2498  ppl=25.78  grad=0.816  lr=1.34e-05  6890 tok/s  12.59s/it  [4:57:58<25:59:28]  mem=34.07GB
full_cpt step  5430/12852 ( 42.3%)  nll=3.2114  ppl=24.81  grad=0.844  lr=1.33e-05  6895 tok/s  12.58s/it  [4:59:54<25:56:31]  mem=34.07GB
full_cpt step  5440/12852 ( 42.3%)  nll=3.2947  ppl=26.97  grad=0.918  lr=1.33e-05  6898 tok/s  12.58s/it  [5:01:50<25:53:35]  mem=34.07GB
full_cpt step  5450/12852 ( 42.4%)  nll=3.3932  ppl=29.76  grad=0.902  lr=1.33e-05  6901 tok/s  12.57s/it  [5:03:46<25:50:39]  mem=34.07GB
full_cpt step  5460/12852 ( 42.5%)  nll=3.1929  ppl=24.36  grad=0.918  lr=1.33e-05  6904 tok/s  12.56s/it  [5:05:42<25:47:44]  mem=34.07GB
full_cpt step  5470/12852 ( 42.6%)  nll=3.1820  ppl=24.10  grad=0.828  lr=1.32e-05  6907 tok/s  12.56s/it  [5:07:38<25:44:51]  mem=34.07GB
full_cpt step  5480/12852 ( 42.6%)  nll=3.2314  ppl=25.31  grad=0.840  lr=1.32e-05  6911 tok/s  12.55s/it  [5:09:34<25:41:58]  mem=34.07GB
full_cpt step  5490/12852 ( 42.7%)  nll=3.3164  ppl=27.56  grad=0.895  lr=1.32e-05  6915 tok/s  12.54s/it  [5:11:30<25:39:06]  mem=34.07GB
full_cpt step  5500/12852 ( 42.8%)  nll=3.2990  ppl=27.09  grad=0.926  lr=1.32e-05  6917 tok/s  12.54s/it  [5:13:26<25:36:14]  mem=34.07GB
  am   nll=3.9765  ppl=53.33  bpb=1.3983
  ta   nll=3.9486  ppl=51.86  bpb=0.5735
  kk   nll=2.7543  ppl=15.71  bpb=0.9275
  tl   nll=2.6605  ppl=14.30  bpb=1.2623
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 5500  01:27 (87.5s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.30s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.30s/it]
removing old checkpoint checkpoint-4500
kept last 2 resume checkpoints: checkpoint-5000 checkpoint-5500
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-5500 at step 5500 next_row=704000
full_cpt step  5510/12852 ( 42.9%)  nll=3.1887  ppl=24.26  grad=0.879  lr=1.32e-05  6860 tok/s  12.64s/it  [5:18:07<25:46:47]  mem=34.07GB
full_cpt step  5520/12852 ( 43.0%)  nll=3.3598  ppl=28.78  grad=0.910  lr=1.31e-05  6865 tok/s  12.63s/it  [5:20:04<25:43:52]  mem=34.07GB
full_cpt step  5530/12852 ( 43.0%)  nll=3.2223  ppl=25.09  grad=0.852  lr=1.31e-05  6868 tok/s  12.63s/it  [5:22:00<25:40:56]  mem=34.07GB
full_cpt step  5540/12852 ( 43.1%)  nll=3.2783  ppl=26.53  grad=0.828  lr=1.31e-05  6872 tok/s  12.62s/it  [5:23:56<25:38:02]  mem=34.07GB
full_cpt step  5550/12852 ( 43.2%)  nll=3.4123  ppl=30.34  grad=0.914  lr=1.31e-05  6875 tok/s  12.61s/it  [5:25:52<25:35:08]  mem=34.07GB
full_cpt step  5560/12852 ( 43.3%)  nll=3.4105  ppl=30.28  grad=0.910  lr=1.30e-05  6879 tok/s  12.61s/it  [5:27:48<25:32:15]  mem=34.07GB
full_cpt step  5570/12852 ( 43.3%)  nll=3.3011  ppl=27.14  grad=0.855  lr=1.30e-05  6882 tok/s  12.60s/it  [5:29:44<25:29:22]  mem=34.07GB
full_cpt step  5580/12852 ( 43.4%)  nll=3.3152  ppl=27.53  grad=0.953  lr=1.30e-05  6885 tok/s  12.59s/it  [5:31:40<25:26:30]  mem=34.07GB
full_cpt step  5590/12852 ( 43.5%)  nll=3.3125  ppl=27.45  grad=0.863  lr=1.30e-05  6889 tok/s  12.59s/it  [5:33:36<25:23:40]  mem=34.07GB
full_cpt step  5600/12852 ( 43.6%)  nll=3.2374  ppl=25.47  grad=0.852  lr=1.30e-05  6891 tok/s  12.58s/it  [5:35:32<25:20:49]  mem=34.07GB
  am   nll=3.9753  ppl=53.27  bpb=1.3979
  ta   nll=3.9479  ppl=51.83  bpb=0.5734
  kk   nll=2.7537  ppl=15.70  bpb=0.9273
  tl   nll=2.6601  ppl=14.30  bpb=1.2622
  en   nll=2.8175  ppl=16.73  bpb=0.8772
eval full_cpt step 5600  01:28 (88.3s)
full_cpt step  5610/12852 ( 43.7%)  nll=3.2760  ppl=26.47  grad=0.855  lr=1.29e-05  6864 tok/s  12.63s/it  [5:38:56<25:24:36]  mem=34.07GB
full_cpt step  5620/12852 ( 43.7%)  nll=3.2072  ppl=24.71  grad=0.879  lr=1.29e-05  6868 tok/s  12.62s/it  [5:40:52<25:21:44]  mem=34.07GB
full_cpt step  5630/12852 ( 43.8%)  nll=3.4398  ppl=31.18  grad=0.918  lr=1.29e-05  6872 tok/s  12.62s/it  [5:42:49<25:18:53]  mem=34.07GB
full_cpt step  5640/12852 ( 43.9%)  nll=3.3349  ppl=28.08  grad=0.902  lr=1.29e-05  6874 tok/s  12.61s/it  [5:44:45<25:16:02]  mem=34.07GB
full_cpt step  5650/12852 ( 44.0%)  nll=3.3972  ppl=29.88  grad=0.910  lr=1.29e-05  6878 tok/s  12.61s/it  [5:46:41<25:13:12]  mem=34.07GB
full_cpt step  5660/12852 ( 44.0%)  nll=3.2712  ppl=26.34  grad=0.926  lr=1.28e-05  6882 tok/s  12.60s/it  [5:48:37<25:10:23]  mem=34.07GB
full_cpt step  5670/12852 ( 44.1%)  nll=3.3413  ppl=28.26  grad=0.816  lr=1.28e-05  6884 tok/s  12.59s/it  [5:50:33<25:07:34]  mem=34.07GB
full_cpt step  5680/12852 ( 44.2%)  nll=3.3810  ppl=29.40  grad=0.875  lr=1.28e-05  6888 tok/s  12.59s/it  [5:52:29<25:04:46]  mem=34.07GB
full_cpt step  5690/12852 ( 44.3%)  nll=3.4133  ppl=30.37  grad=0.875  lr=1.28e-05  6891 tok/s  12.58s/it  [5:54:25<25:01:59]  mem=34.07GB
full_cpt step  5700/12852 ( 44.4%)  nll=3.3733  ppl=29.17  grad=0.844  lr=1.27e-05  6893 tok/s  12.58s/it  [5:56:21<24:59:12]  mem=34.07GB
  am   nll=3.9746  ppl=53.23  bpb=1.3977
  ta   nll=3.9477  ppl=51.82  bpb=0.5734
  kk   nll=2.7535  ppl=15.70  bpb=0.9272
  tl   nll=2.6602  ppl=14.30  bpb=1.2622
  en   nll=2.8173  ppl=16.73  bpb=0.8772
eval full_cpt step 5700  01:28 (87.8s)
full_cpt step  5710/12852 ( 44.4%)  nll=3.3942  ppl=29.79  grad=0.895  lr=1.27e-05  6868 tok/s  12.62s/it  [5:59:45<25:02:32]  mem=34.07GB
full_cpt step  5720/12852 ( 44.5%)  nll=3.3362  ppl=28.11  grad=0.863  lr=1.27e-05  6872 tok/s  12.62s/it  [6:01:41<24:59:44]  mem=34.07GB
full_cpt step  5730/12852 ( 44.6%)  nll=3.2601  ppl=26.05  grad=0.918  lr=1.27e-05  6875 tok/s  12.61s/it  [6:03:37<24:56:56]  mem=34.07GB
full_cpt step  5740/12852 ( 44.7%)  nll=3.3397  ppl=28.21  grad=0.926  lr=1.27e-05  6879 tok/s  12.61s/it  [6:05:33<24:54:09]  mem=34.07GB
full_cpt step  5750/12852 ( 44.7%)  nll=3.3901  ppl=29.67  grad=0.883  lr=1.26e-05  6882 tok/s  12.60s/it  [6:07:29<24:51:23]  mem=34.07GB
full_cpt step  5760/12852 ( 44.8%)  nll=3.3085  ppl=27.35  grad=0.879  lr=1.26e-05  6884 tok/s  12.59s/it  [6:09:25<24:48:37]  mem=34.07GB
full_cpt step  5770/12852 ( 44.9%)  nll=3.3332  ppl=28.03  grad=0.879  lr=1.26e-05  6887 tok/s  12.59s/it  [6:11:22<24:45:51]  mem=34.07GB
full_cpt step  5780/12852 ( 45.0%)  nll=3.2783  ppl=26.53  grad=0.891  lr=1.26e-05  6890 tok/s  12.58s/it  [6:13:18<24:43:06]  mem=34.07GB
full_cpt step  5790/12852 ( 45.1%)  nll=3.4794  ppl=32.44  grad=0.906  lr=1.25e-05  6892 tok/s  12.58s/it  [6:15:14<24:40:21]  mem=34.07GB
full_cpt step  5800/12852 ( 45.1%)  nll=3.4331  ppl=30.97  grad=0.906  lr=1.25e-05  6895 tok/s  12.57s/it  [6:17:10<24:37:38]  mem=34.07GB
  am   nll=3.9737  ppl=53.18  bpb=1.3974
  ta   nll=3.9469  ppl=51.77  bpb=0.5733
  kk   nll=2.7530  ppl=15.69  bpb=0.9270
  tl   nll=2.6600  ppl=14.30  bpb=1.2621
  en   nll=2.8173  ppl=16.73  bpb=0.8772
eval full_cpt step 5800  01:33 (93.2s)
full_cpt step  5810/12852 ( 45.2%)  nll=3.3261  ppl=27.83  grad=0.867  lr=1.25e-05  6870 tok/s  12.62s/it  [6:20:39<24:40:57]  mem=34.07GB
full_cpt step  5820/12852 ( 45.3%)  nll=3.2357  ppl=25.43  grad=0.855  lr=1.25e-05  6873 tok/s  12.61s/it  [6:22:35<24:38:12]  mem=34.07GB
full_cpt step  5830/12852 ( 45.4%)  nll=3.3501  ppl=28.51  grad=0.867  lr=1.25e-05  6876 tok/s  12.61s/it  [6:24:31<24:35:28]  mem=34.07GB
full_cpt step  5840/12852 ( 45.4%)  nll=3.2598  ppl=26.04  grad=0.832  lr=1.24e-05  6879 tok/s  12.60s/it  [6:26:28<24:32:46]  mem=34.07GB
full_cpt step  5850/12852 ( 45.5%)  nll=3.4198  ppl=30.56  grad=0.891  lr=1.24e-05  6882 tok/s  12.60s/it  [6:28:24<24:30:04]  mem=34.07GB
full_cpt step  5860/12852 ( 45.6%)  nll=3.4026  ppl=30.04  grad=0.883  lr=1.24e-05  6885 tok/s  12.59s/it  [6:30:21<24:27:21]  mem=34.07GB
full_cpt step  5870/12852 ( 45.7%)  nll=3.4840  ppl=32.59  grad=0.898  lr=1.24e-05  6888 tok/s  12.59s/it  [6:32:17<24:24:39]  mem=34.07GB
full_cpt step  5880/12852 ( 45.8%)  nll=3.3249  ppl=27.80  grad=0.895  lr=1.24e-05  6890 tok/s  12.58s/it  [6:34:13<24:21:57]  mem=34.07GB
full_cpt step  5890/12852 ( 45.8%)  nll=3.3780  ppl=29.31  grad=0.906  lr=1.23e-05  6894 tok/s  12.58s/it  [6:36:10<24:19:17]  mem=34.07GB
full_cpt step  5900/12852 ( 45.9%)  nll=3.2935  ppl=26.94  grad=0.906  lr=1.23e-05  6897 tok/s  12.57s/it  [6:38:06<24:16:37]  mem=34.07GB
  am   nll=3.9737  ppl=53.18  bpb=1.3973
  ta   nll=3.9468  ppl=51.77  bpb=0.5732
  kk   nll=2.7527  ppl=15.68  bpb=0.9270
  tl   nll=2.6601  ppl=14.30  bpb=1.2622
  en   nll=2.8175  ppl=16.73  bpb=0.8772
eval full_cpt step 5900  01:41 (100.7s)
full_cpt step  5910/12852 ( 46.0%)  nll=3.3723  ppl=29.15  grad=0.898  lr=1.23e-05  6871 tok/s  12.62s/it  [6:41:43<24:20:02]  mem=34.07GB
full_cpt step  5920/12852 ( 46.1%)  nll=3.3066  ppl=27.29  grad=0.836  lr=1.23e-05  6873 tok/s  12.61s/it  [6:43:39<24:17:20]  mem=34.07GB
full_cpt step  5930/12852 ( 46.1%)  nll=3.3472  ppl=28.42  grad=0.902  lr=1.22e-05  6877 tok/s  12.61s/it  [6:45:35<24:14:38]  mem=34.07GB
full_cpt step  5940/12852 ( 46.2%)  nll=3.3615  ppl=28.83  grad=0.871  lr=1.22e-05  6879 tok/s  12.60s/it  [6:47:31<24:11:56]  mem=34.07GB
full_cpt step  5950/12852 ( 46.3%)  nll=3.2696  ppl=26.30  grad=0.926  lr=1.22e-05  6881 tok/s  12.60s/it  [6:49:27<24:09:14]  mem=34.07GB
full_cpt step  5960/12852 ( 46.4%)  nll=3.2585  ppl=26.01  grad=0.938  lr=1.22e-05  6884 tok/s  12.59s/it  [6:51:23<24:06:33]  mem=34.07GB
full_cpt step  5970/12852 ( 46.5%)  nll=3.2932  ppl=26.93  grad=0.922  lr=1.22e-05  6886 tok/s  12.59s/it  [6:53:19<24:03:52]  mem=34.07GB
full_cpt step  5980/12852 ( 46.5%)  nll=3.2102  ppl=24.78  grad=0.871  lr=1.21e-05  6888 tok/s  12.58s/it  [6:55:15<24:01:12]  mem=34.07GB
full_cpt step  5990/12852 ( 46.6%)  nll=3.3069  ppl=27.30  grad=0.891  lr=1.21e-05  6891 tok/s  12.58s/it  [6:57:11<23:58:32]  mem=34.07GB
full_cpt step  6000/12852 ( 46.7%)  nll=3.3209  ppl=27.69  grad=0.859  lr=1.21e-05  6894 tok/s  12.57s/it  [6:59:07<23:55:53]  mem=34.07GB
  am   nll=3.9733  ppl=53.16  bpb=1.3972
  ta   nll=3.9462  ppl=51.74  bpb=0.5732
  kk   nll=2.7526  ppl=15.68  bpb=0.9269
  tl   nll=2.6599  ppl=14.30  bpb=1.2621
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 6000  01:50 (109.9s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.26s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.26s/it]
removing old checkpoint checkpoint-5000
kept last 2 resume checkpoints: checkpoint-5500 checkpoint-6000
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-6000 at step 6000 next_row=768000

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:03<00:00,  3.12s/it]
Writing model shards: 100%|##########| 1/1 [00:03<00:00,  3.12s/it]
wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-6000 (weights+expanded tokenizer, no optimizer)
full_cpt step  6010/12852 ( 46.8%)  nll=3.2879  ppl=26.79  grad=0.863  lr=1.21e-05  6841 tok/s  12.67s/it  [7:04:27<24:04:48]  mem=34.07GB
full_cpt step  6020/12852 ( 46.8%)  nll=3.2718  ppl=26.36  grad=0.934  lr=1.20e-05  6844 tok/s  12.66s/it  [7:06:23<24:02:05]  mem=34.07GB
full_cpt step  6030/12852 ( 46.9%)  nll=3.3071  ppl=27.31  grad=0.891  lr=1.20e-05  6847 tok/s  12.66s/it  [7:08:19<23:59:23]  mem=34.07GB
full_cpt step  6040/12852 ( 47.0%)  nll=3.3184  ppl=27.62  grad=0.871  lr=1.20e-05  6851 tok/s  12.65s/it  [7:10:15<23:56:42]  mem=34.07GB
full_cpt step  6050/12852 ( 47.1%)  nll=3.3727  ppl=29.16  grad=0.879  lr=1.20e-05  6854 tok/s  12.65s/it  [7:12:11<23:54:01]  mem=34.07GB
full_cpt step  6060/12852 ( 47.2%)  nll=3.2168  ppl=24.95  grad=0.914  lr=1.20e-05  6857 tok/s  12.64s/it  [7:14:07<23:51:19]  mem=34.07GB
full_cpt step  6070/12852 ( 47.2%)  nll=3.2441  ppl=25.64  grad=0.902  lr=1.19e-05  6859 tok/s  12.64s/it  [7:16:03<23:48:39]  mem=34.07GB
full_cpt step  6080/12852 ( 47.3%)  nll=3.2790  ppl=26.55  grad=0.859  lr=1.19e-05  6862 tok/s  12.63s/it  [7:17:59<23:45:58]  mem=34.07GB
full_cpt step  6090/12852 ( 47.4%)  nll=3.1845  ppl=24.15  grad=0.832  lr=1.19e-05  6864 tok/s  12.63s/it  [7:19:55<23:43:18]  mem=34.07GB
full_cpt step  6100/12852 ( 47.5%)  nll=3.2546  ppl=25.91  grad=0.883  lr=1.19e-05  6867 tok/s  12.62s/it  [7:21:51<23:40:39]  mem=34.07GB
  am   nll=3.9731  ppl=53.15  bpb=1.3972
  ta   nll=3.9463  ppl=51.74  bpb=0.5732
  kk   nll=2.7518  ppl=15.67  bpb=0.9266
  tl   nll=2.6602  ppl=14.30  bpb=1.2622
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 6100  01:28 (87.5s)
full_cpt step  6110/12852 ( 47.5%)  nll=3.3447  ppl=28.35  grad=0.805  lr=1.18e-05  6848 tok/s  12.66s/it  [7:25:14<23:42:39]  mem=34.07GB
full_cpt step  6120/12852 ( 47.6%)  nll=3.2332  ppl=25.36  grad=0.883  lr=1.18e-05  6850 tok/s  12.66s/it  [7:27:10<23:39:59]  mem=34.07GB
full_cpt step  6130/12852 ( 47.7%)  nll=3.4210  ppl=30.60  grad=0.898  lr=1.18e-05  6853 tok/s  12.65s/it  [7:29:06<23:37:19]  mem=34.07GB
full_cpt step  6140/12852 ( 47.8%)  nll=3.3735  ppl=29.18  grad=0.863  lr=1.18e-05  6855 tok/s  12.65s/it  [7:31:02<23:34:39]  mem=34.07GB
full_cpt step  6150/12852 ( 47.9%)  nll=3.3067  ppl=27.30  grad=0.848  lr=1.18e-05  6858 tok/s  12.64s/it  [7:32:58<23:32:00]  mem=34.07GB
full_cpt step  6160/12852 ( 47.9%)  nll=3.2277  ppl=25.22  grad=0.902  lr=1.17e-05  6859 tok/s  12.64s/it  [7:34:54<23:29:21]  mem=34.07GB
full_cpt step  6170/12852 ( 48.0%)  nll=3.2197  ppl=25.02  grad=0.906  lr=1.17e-05  6862 tok/s  12.63s/it  [7:36:50<23:26:42]  mem=34.07GB
full_cpt step  6180/12852 ( 48.1%)  nll=3.2985  ppl=27.07  grad=0.953  lr=1.17e-05  6865 tok/s  12.63s/it  [7:38:46<23:24:04]  mem=34.07GB
full_cpt step  6190/12852 ( 48.2%)  nll=3.3410  ppl=28.25  grad=0.879  lr=1.17e-05  6868 tok/s  12.62s/it  [7:40:42<23:21:27]  mem=34.07GB
full_cpt step  6200/12852 ( 48.2%)  nll=3.2986  ppl=27.08  grad=0.879  lr=1.16e-05  6870 tok/s  12.62s/it  [7:42:38<23:18:50]  mem=34.07GB
  am   nll=3.9731  ppl=53.15  bpb=1.3971
  ta   nll=3.9458  ppl=51.72  bpb=0.5731
  kk   nll=2.7520  ppl=15.67  bpb=0.9267
  tl   nll=2.6601  ppl=14.30  bpb=1.2621
  en   nll=2.8175  ppl=16.73  bpb=0.8772
eval full_cpt step 6200  01:27 (87.3s)
full_cpt step  6210/12852 ( 48.3%)  nll=3.2270  ppl=25.20  grad=0.820  lr=1.16e-05  6851 tok/s  12.65s/it  [7:46:01<23:20:35]  mem=34.07GB
full_cpt step  6220/12852 ( 48.4%)  nll=3.2329  ppl=25.35  grad=0.859  lr=1.16e-05  6853 tok/s  12.65s/it  [7:47:57<23:17:57]  mem=34.07GB
full_cpt step  6230/12852 ( 48.5%)  nll=3.3161  ppl=27.55  grad=0.832  lr=1.16e-05  6855 tok/s  12.64s/it  [7:49:53<23:15:19]  mem=34.07GB
full_cpt step  6240/12852 ( 48.6%)  nll=3.3070  ppl=27.30  grad=0.840  lr=1.16e-05  6858 tok/s  12.64s/it  [7:51:49<23:12:42]  mem=34.07GB
full_cpt step  6250/12852 ( 48.6%)  nll=3.4116  ppl=30.31  grad=0.820  lr=1.15e-05  6860 tok/s  12.63s/it  [7:53:45<23:10:05]  mem=34.07GB
full_cpt step  6260/12852 ( 48.7%)  nll=3.3744  ppl=29.21  grad=0.871  lr=1.15e-05  6862 tok/s  12.63s/it  [7:55:41<23:07:28]  mem=34.07GB
full_cpt step  6270/12852 ( 48.8%)  nll=3.3055  ppl=27.26  grad=0.863  lr=1.15e-05  6865 tok/s  12.62s/it  [7:57:37<23:04:52]  mem=34.07GB
full_cpt step  6280/12852 ( 48.9%)  nll=3.3385  ppl=28.18  grad=0.871  lr=1.15e-05  6867 tok/s  12.62s/it  [7:59:33<23:02:16]  mem=34.07GB
full_cpt step  6290/12852 ( 48.9%)  nll=3.3552  ppl=28.65  grad=0.934  lr=1.14e-05  6869 tok/s  12.62s/it  [8:01:29<22:59:40]  mem=34.07GB
full_cpt step  6300/12852 ( 49.0%)  nll=3.2521  ppl=25.85  grad=0.891  lr=1.14e-05  6873 tok/s  12.61s/it  [8:03:25<22:57:05]  mem=34.07GB
  am   nll=3.9728  ppl=53.14  bpb=1.3971
  ta   nll=3.9459  ppl=51.72  bpb=0.5731
  kk   nll=2.7516  ppl=15.67  bpb=0.9266
  tl   nll=2.6598  ppl=14.29  bpb=1.2620
  en   nll=2.8178  ppl=16.74  bpb=0.8773
eval full_cpt step 6300  01:27 (86.8s)
full_cpt step  6310/12852 ( 49.1%)  nll=3.2305  ppl=25.29  grad=0.816  lr=1.14e-05  6854 tok/s  12.64s/it  [8:06:47<22:58:36]  mem=34.07GB
full_cpt step  6320/12852 ( 49.2%)  nll=3.2737  ppl=26.41  grad=0.887  lr=1.14e-05  6857 tok/s  12.64s/it  [8:08:43<22:56:00]  mem=34.07GB
full_cpt step  6330/12852 ( 49.3%)  nll=3.3588  ppl=28.75  grad=0.910  lr=1.14e-05  6860 tok/s  12.63s/it  [8:10:39<22:53:24]  mem=34.07GB
full_cpt step  6340/12852 ( 49.3%)  nll=3.3837  ppl=29.48  grad=0.902  lr=1.13e-05  6861 tok/s  12.63s/it  [8:12:35<22:50:49]  mem=34.07GB
full_cpt step  6350/12852 ( 49.4%)  nll=3.2549  ppl=25.92  grad=0.863  lr=1.13e-05  6864 tok/s  12.63s/it  [8:14:31<22:48:14]  mem=34.07GB
full_cpt step  6360/12852 ( 49.5%)  nll=3.4559  ppl=31.69  grad=0.812  lr=1.13e-05  6867 tok/s  12.62s/it  [8:16:27<22:45:39]  mem=34.07GB
full_cpt step  6370/12852 ( 49.6%)  nll=3.2707  ppl=26.33  grad=0.855  lr=1.13e-05  6869 tok/s  12.62s/it  [8:18:23<22:43:05]  mem=34.07GB
full_cpt step  6380/12852 ( 49.6%)  nll=3.4500  ppl=31.50  grad=0.922  lr=1.12e-05  6871 tok/s  12.61s/it  [8:20:19<22:40:31]  mem=34.07GB
full_cpt step  6390/12852 ( 49.7%)  nll=3.3094  ppl=27.37  grad=0.859  lr=1.12e-05  6873 tok/s  12.61s/it  [8:22:15<22:37:57]  mem=34.07GB
full_cpt step  6400/12852 ( 49.8%)  nll=3.3178  ppl=27.60  grad=0.934  lr=1.12e-05  6875 tok/s  12.60s/it  [8:24:10<22:35:23]  mem=34.07GB
  am   nll=3.9727  ppl=53.13  bpb=1.3970
  ta   nll=3.9448  ppl=51.67  bpb=0.5730
  kk   nll=2.7512  ppl=15.66  bpb=0.9264
  tl   nll=2.6599  ppl=14.29  bpb=1.2621
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 6400  01:27 (87.5s)
full_cpt step  6410/12852 ( 49.9%)  nll=3.2677  ppl=26.25  grad=0.879  lr=1.12e-05  6859 tok/s  12.64s/it  [8:27:34<22:36:44]  mem=34.07GB
full_cpt step  6420/12852 ( 50.0%)  nll=3.3061  ppl=27.28  grad=0.910  lr=1.12e-05  6861 tok/s  12.63s/it  [8:29:30<22:34:10]  mem=34.07GB
full_cpt step  6430/12852 ( 50.0%)  nll=3.3911  ppl=29.70  grad=0.914  lr=1.11e-05  6863 tok/s  12.63s/it  [8:31:26<22:31:36]  mem=34.07GB
full_cpt step  6440/12852 ( 50.1%)  nll=3.3627  ppl=28.87  grad=0.832  lr=1.11e-05  6866 tok/s  12.62s/it  [8:33:22<22:29:03]  mem=34.07GB
full_cpt step  6450/12852 ( 50.2%)  nll=3.2594  ppl=26.04  grad=0.828  lr=1.11e-05  6868 tok/s  12.62s/it  [8:35:18<22:26:30]  mem=34.07GB
full_cpt step  6460/12852 ( 50.3%)  nll=3.2492  ppl=25.77  grad=0.961  lr=1.11e-05  6871 tok/s  12.62s/it  [8:37:14<22:23:57]  mem=34.07GB
full_cpt step  6470/12852 ( 50.3%)  nll=3.2841  ppl=26.69  grad=0.875  lr=1.10e-05  6873 tok/s  12.61s/it  [8:39:10<22:21:25]  mem=34.07GB
full_cpt step  6480/12852 ( 50.4%)  nll=3.3185  ppl=27.62  grad=0.871  lr=1.10e-05  6875 tok/s  12.61s/it  [8:41:06<22:18:52]  mem=34.07GB
full_cpt step  6490/12852 ( 50.5%)  nll=3.2944  ppl=26.96  grad=0.855  lr=1.10e-05  6877 tok/s  12.60s/it  [8:43:02<22:16:20]  mem=34.07GB
full_cpt step  6500/12852 ( 50.6%)  nll=3.4613  ppl=31.86  grad=0.824  lr=1.10e-05  6879 tok/s  12.60s/it  [8:44:58<22:13:49]  mem=34.07GB
  am   nll=3.9721  ppl=53.10  bpb=1.3968
  ta   nll=3.9449  ppl=51.67  bpb=0.5730
  kk   nll=2.7511  ppl=15.66  bpb=0.9264
  tl   nll=2.6599  ppl=14.29  bpb=1.2621
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 6500  01:27 (86.9s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.64s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.65s/it]
removing old checkpoint checkpoint-5500
kept last 2 resume checkpoints: checkpoint-6000 checkpoint-6500
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-6500 at step 6500 next_row=832000
full_cpt step  6510/12852 ( 50.7%)  nll=3.3151  ppl=27.53  grad=0.906  lr=1.10e-05  6846 tok/s  12.66s/it  [8:49:36<22:18:08]  mem=34.07GB
full_cpt step  6520/12852 ( 50.7%)  nll=3.3557  ppl=28.66  grad=0.859  lr=1.09e-05  6847 tok/s  12.66s/it  [8:51:32<22:15:35]  mem=34.07GB
full_cpt step  6530/12852 ( 50.8%)  nll=3.4649  ppl=31.97  grad=0.887  lr=1.09e-05  6849 tok/s  12.65s/it  [8:53:28<22:13:02]  mem=34.07GB
full_cpt step  6540/12852 ( 50.9%)  nll=3.3843  ppl=29.50  grad=0.914  lr=1.09e-05  6852 tok/s  12.65s/it  [8:55:24<22:10:30]  mem=34.07GB
full_cpt step  6550/12852 ( 51.0%)  nll=3.0930  ppl=22.04  grad=0.922  lr=1.09e-05  6854 tok/s  12.64s/it  [8:57:20<22:07:57]  mem=34.07GB
full_cpt step  6560/12852 ( 51.0%)  nll=3.3314  ppl=27.98  grad=0.824  lr=1.08e-05  6856 tok/s  12.64s/it  [8:59:16<22:05:25]  mem=34.07GB
full_cpt step  6570/12852 ( 51.1%)  nll=3.3110  ppl=27.41  grad=0.891  lr=1.08e-05  6859 tok/s  12.64s/it  [9:01:12<22:02:53]  mem=34.07GB
full_cpt step  6580/12852 ( 51.2%)  nll=3.3909  ppl=29.69  grad=0.871  lr=1.08e-05  6861 tok/s  12.63s/it  [9:03:08<22:00:22]  mem=34.07GB
full_cpt step  6590/12852 ( 51.3%)  nll=3.3570  ppl=28.70  grad=0.934  lr=1.08e-05  6862 tok/s  12.63s/it  [9:05:04<21:57:50]  mem=34.07GB
full_cpt step  6600/12852 ( 51.4%)  nll=3.3418  ppl=28.27  grad=0.887  lr=1.08e-05  6865 tok/s  12.62s/it  [9:07:00<21:55:19]  mem=34.07GB
  am   nll=3.9720  ppl=53.09  bpb=1.3968
  ta   nll=3.9447  ppl=51.66  bpb=0.5729
  kk   nll=2.7510  ppl=15.66  bpb=0.9264
  tl   nll=2.6599  ppl=14.29  bpb=1.2621
  en   nll=2.8175  ppl=16.74  bpb=0.8773
eval full_cpt step 6600  01:27 (86.9s)
full_cpt step  6610/12852 ( 51.4%)  nll=3.4597  ppl=31.81  grad=0.906  lr=1.07e-05  6849 tok/s  12.65s/it  [9:10:22<21:56:16]  mem=34.07GB
full_cpt step  6620/12852 ( 51.5%)  nll=3.2058  ppl=24.67  grad=0.844  lr=1.07e-05  6851 tok/s  12.65s/it  [9:12:18<21:53:44]  mem=34.07GB
full_cpt step  6630/12852 ( 51.6%)  nll=3.2620  ppl=26.10  grad=0.844  lr=1.07e-05  6854 tok/s  12.64s/it  [9:14:14<21:51:12]  mem=34.07GB
full_cpt step  6640/12852 ( 51.7%)  nll=3.3069  ppl=27.30  grad=0.895  lr=1.07e-05  6857 tok/s  12.64s/it  [9:16:10<21:48:42]  mem=34.07GB
full_cpt step  6650/12852 ( 51.7%)  nll=3.2561  ppl=25.95  grad=0.816  lr=1.06e-05  6859 tok/s  12.64s/it  [9:18:06<21:46:11]  mem=34.07GB
full_cpt step  6660/12852 ( 51.8%)  nll=3.1758  ppl=23.95  grad=0.840  lr=1.06e-05  6861 tok/s  12.63s/it  [9:20:02<21:43:40]  mem=34.07GB
full_cpt step  6670/12852 ( 51.9%)  nll=3.3205  ppl=27.67  grad=0.863  lr=1.06e-05  6864 tok/s  12.63s/it  [9:21:59<21:41:10]  mem=34.07GB
full_cpt step  6680/12852 ( 52.0%)  nll=3.3634  ppl=28.89  grad=0.883  lr=1.06e-05  6866 tok/s  12.62s/it  [9:23:54<21:38:40]  mem=34.07GB
full_cpt step  6690/12852 ( 52.1%)  nll=3.3397  ppl=28.21  grad=0.871  lr=1.06e-05  6868 tok/s  12.62s/it  [9:25:50<21:36:11]  mem=34.07GB
full_cpt step  6700/12852 ( 52.1%)  nll=3.3583  ppl=28.74  grad=0.930  lr=1.05e-05  6870 tok/s  12.62s/it  [9:27:46<21:33:41]  mem=34.07GB
  am   nll=3.9715  ppl=53.06  bpb=1.3966
  ta   nll=3.9444  ppl=51.64  bpb=0.5729
  kk   nll=2.7507  ppl=15.65  bpb=0.9263
  tl   nll=2.6597  ppl=14.29  bpb=1.2620
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 6700  01:29 (89.0s)
full_cpt step  6710/12852 ( 52.2%)  nll=3.2345  ppl=25.39  grad=0.934  lr=1.05e-05  6854 tok/s  12.65s/it  [9:31:11<21:34:33]  mem=34.07GB
full_cpt step  6720/12852 ( 52.3%)  nll=3.2897  ppl=26.84  grad=0.883  lr=1.05e-05  6856 tok/s  12.64s/it  [9:33:07<21:32:03]  mem=34.07GB
full_cpt step  6730/12852 ( 52.4%)  nll=3.4466  ppl=31.39  grad=0.859  lr=1.05e-05  6859 tok/s  12.64s/it  [9:35:03<21:29:34]  mem=34.07GB
full_cpt step  6740/12852 ( 52.4%)  nll=3.2301  ppl=25.28  grad=0.863  lr=1.04e-05  6861 tok/s  12.63s/it  [9:37:00<21:27:04]  mem=34.07GB
full_cpt step  6750/12852 ( 52.5%)  nll=3.3340  ppl=28.05  grad=0.906  lr=1.04e-05  6864 tok/s  12.63s/it  [9:38:56<21:24:35]  mem=34.07GB
full_cpt step  6760/12852 ( 52.6%)  nll=3.4084  ppl=30.22  grad=0.824  lr=1.04e-05  6866 tok/s  12.63s/it  [9:40:52<21:22:06]  mem=34.07GB
full_cpt step  6770/12852 ( 52.7%)  nll=3.3090  ppl=27.36  grad=0.875  lr=1.04e-05  6868 tok/s  12.62s/it  [9:42:48<21:19:37]  mem=34.07GB
full_cpt step  6780/12852 ( 52.8%)  nll=3.2311  ppl=25.31  grad=0.883  lr=1.04e-05  6870 tok/s  12.62s/it  [9:44:44<21:17:09]  mem=34.07GB
full_cpt step  6790/12852 ( 52.8%)  nll=3.3513  ppl=28.54  grad=0.891  lr=1.03e-05  6873 tok/s  12.62s/it  [9:46:40<21:14:41]  mem=34.07GB
full_cpt step  6800/12852 ( 52.9%)  nll=3.3295  ppl=27.93  grad=0.891  lr=1.03e-05  6874 tok/s  12.61s/it  [9:48:36<21:12:13]  mem=34.07GB
  am   nll=3.9713  ppl=53.05  bpb=1.3965
  ta   nll=3.9446  ppl=51.66  bpb=0.5729
  kk   nll=2.7507  ppl=15.65  bpb=0.9263
  tl   nll=2.6599  ppl=14.29  bpb=1.2620
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 6800  01:30 (89.6s)
full_cpt step  6810/12852 ( 53.0%)  nll=3.3408  ppl=28.24  grad=0.859  lr=1.03e-05  6860 tok/s  12.64s/it  [9:52:01<21:12:57]  mem=34.07GB
full_cpt step  6820/12852 ( 53.1%)  nll=3.1812  ppl=24.08  grad=0.871  lr=1.03e-05  6862 tok/s  12.64s/it  [9:53:57<21:10:29]  mem=34.07GB
full_cpt step  6830/12852 ( 53.1%)  nll=3.3399  ppl=28.22  grad=0.871  lr=1.02e-05  6864 tok/s  12.63s/it  [9:55:53<21:08:00]  mem=34.07GB
full_cpt step  6840/12852 ( 53.2%)  nll=3.3186  ppl=27.62  grad=0.859  lr=1.02e-05  6866 tok/s  12.63s/it  [9:57:49<21:05:32]  mem=34.07GB
full_cpt step  6850/12852 ( 53.3%)  nll=3.4017  ppl=30.02  grad=0.906  lr=1.02e-05  6868 tok/s  12.63s/it  [9:59:45<21:03:04]  mem=34.07GB
full_cpt step  6860/12852 ( 53.4%)  nll=3.2830  ppl=26.66  grad=0.910  lr=1.02e-05  6870 tok/s  12.62s/it  [10:01:41<21:00:36]  mem=34.07GB
full_cpt step  6870/12852 ( 53.5%)  nll=3.3422  ppl=28.28  grad=0.840  lr=1.02e-05  6872 tok/s  12.62s/it  [10:03:37<20:58:09]  mem=34.07GB
full_cpt step  6880/12852 ( 53.5%)  nll=3.3762  ppl=29.26  grad=0.898  lr=1.01e-05  6874 tok/s  12.62s/it  [10:05:33<20:55:41]  mem=34.07GB
full_cpt step  6890/12852 ( 53.6%)  nll=3.3573  ppl=28.71  grad=0.863  lr=1.01e-05  6875 tok/s  12.61s/it  [10:07:29<20:53:14]  mem=34.07GB
full_cpt step  6900/12852 ( 53.7%)  nll=3.2813  ppl=26.61  grad=0.938  lr=1.01e-05  6876 tok/s  12.61s/it  [10:09:25<20:50:47]  mem=34.07GB
  am   nll=3.9714  ppl=53.06  bpb=1.3965
  ta   nll=3.9448  ppl=51.66  bpb=0.5730
  kk   nll=2.7506  ppl=15.65  bpb=0.9262
  tl   nll=2.6599  ppl=14.29  bpb=1.2620
  en   nll=2.8173  ppl=16.73  bpb=0.8772
eval full_cpt step 6900  01:29 (89.1s)
full_cpt step  6910/12852 ( 53.8%)  nll=3.2731  ppl=26.39  grad=0.863  lr=1.01e-05  6861 tok/s  12.64s/it  [10:12:50<20:51:22]  mem=34.07GB
full_cpt step  6920/12852 ( 53.8%)  nll=3.2718  ppl=26.36  grad=0.926  lr=1.00e-05  6863 tok/s  12.63s/it  [10:14:46<20:48:54]  mem=34.07GB
full_cpt step  6930/12852 ( 53.9%)  nll=3.3421  ppl=28.28  grad=0.863  lr=1.00e-05  6865 tok/s  12.63s/it  [10:16:42<20:46:27]  mem=34.07GB
full_cpt step  6940/12852 ( 54.0%)  nll=3.3435  ppl=28.32  grad=0.996  lr=1.00e-05  6866 tok/s  12.63s/it  [10:18:38<20:44:00]  mem=34.07GB
full_cpt step  6950/12852 ( 54.1%)  nll=3.3876  ppl=29.59  grad=0.879  lr=9.98e-06  6868 tok/s  12.62s/it  [10:20:34<20:41:33]  mem=34.07GB
full_cpt step  6960/12852 ( 54.2%)  nll=3.3107  ppl=27.41  grad=0.887  lr=9.96e-06  6870 tok/s  12.62s/it  [10:22:30<20:39:07]  mem=34.07GB
full_cpt step  6970/12852 ( 54.2%)  nll=3.3488  ppl=28.47  grad=0.871  lr=9.94e-06  6872 tok/s  12.61s/it  [10:24:26<20:36:41]  mem=34.07GB
full_cpt step  6980/12852 ( 54.3%)  nll=3.2260  ppl=25.18  grad=0.906  lr=9.91e-06  6874 tok/s  12.61s/it  [10:26:22<20:34:15]  mem=34.07GB
full_cpt step  6990/12852 ( 54.4%)  nll=3.3591  ppl=28.76  grad=0.852  lr=9.89e-06  6876 tok/s  12.61s/it  [10:28:18<20:31:49]  mem=34.07GB
full_cpt step  7000/12852 ( 54.5%)  nll=3.2314  ppl=25.32  grad=0.867  lr=9.87e-06  6878 tok/s  12.60s/it  [10:30:14<20:29:23]  mem=34.07GB
  am   nll=3.9713  ppl=53.05  bpb=1.3965
  ta   nll=3.9443  ppl=51.64  bpb=0.5729
  kk   nll=2.7505  ppl=15.65  bpb=0.9262
  tl   nll=2.6598  ppl=14.29  bpb=1.2620
  en   nll=2.8175  ppl=16.74  bpb=0.8772
eval full_cpt step 7000  01:29 (89.4s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.37s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.37s/it]
removing old checkpoint checkpoint-6000
kept last 2 resume checkpoints: checkpoint-6500 checkpoint-7000
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-7000 at step 7000 next_row=896000

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:03<00:00,  3.39s/it]
Writing model shards: 100%|##########| 1/1 [00:03<00:00,  3.39s/it]
wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-7000 (weights+expanded tokenizer, no optimizer)
full_cpt step  7010/12852 ( 54.5%)  nll=3.2987  ppl=27.08  grad=0.863  lr=9.85e-06  6845 tok/s  12.66s/it  [10:35:20<20:33:06]  mem=34.07GB
full_cpt step  7020/12852 ( 54.6%)  nll=3.5208  ppl=33.81  grad=0.875  lr=9.83e-06  6847 tok/s  12.66s/it  [10:37:16<20:30:39]  mem=34.07GB
full_cpt step  7030/12852 ( 54.7%)  nll=3.2201  ppl=25.03  grad=0.895  lr=9.80e-06  6849 tok/s  12.66s/it  [10:39:12<20:28:12]  mem=34.07GB
full_cpt step  7040/12852 ( 54.8%)  nll=3.3088  ppl=27.35  grad=0.828  lr=9.78e-06  6850 tok/s  12.65s/it  [10:41:08<20:25:45]  mem=34.07GB
full_cpt step  7050/12852 ( 54.9%)  nll=3.2482  ppl=25.74  grad=0.891  lr=9.76e-06  6852 tok/s  12.65s/it  [10:43:04<20:23:19]  mem=34.07GB
full_cpt step  7060/12852 ( 54.9%)  nll=3.2973  ppl=27.04  grad=0.812  lr=9.74e-06  6854 tok/s  12.65s/it  [10:45:00<20:20:53]  mem=34.07GB
full_cpt step  7070/12852 ( 55.0%)  nll=3.3779  ppl=29.31  grad=0.910  lr=9.72e-06  6856 tok/s  12.64s/it  [10:46:57<20:18:27]  mem=34.07GB
full_cpt step  7080/12852 ( 55.1%)  nll=3.3133  ppl=27.48  grad=0.918  lr=9.69e-06  6858 tok/s  12.64s/it  [10:48:53<20:16:01]  mem=34.07GB
full_cpt step  7090/12852 ( 55.2%)  nll=3.2208  ppl=25.05  grad=0.887  lr=9.67e-06  6859 tok/s  12.64s/it  [10:50:48<20:13:34]  mem=34.07GB
full_cpt step  7100/12852 ( 55.2%)  nll=3.3418  ppl=28.27  grad=0.867  lr=9.65e-06  6862 tok/s  12.63s/it  [10:52:44<20:11:09]  mem=34.07GB
  am   nll=3.9711  ppl=53.04  bpb=1.3964
  ta   nll=3.9446  ppl=51.66  bpb=0.5729
  kk   nll=2.7507  ppl=15.65  bpb=0.9263
  tl   nll=2.6596  ppl=14.29  bpb=1.2619
  en   nll=2.8172  ppl=16.73  bpb=0.8772
eval full_cpt step 7100  01:28 (88.5s)
full_cpt step  7110/12852 ( 55.3%)  nll=3.2652  ppl=26.19  grad=0.816  lr=9.63e-06  6849 tok/s  12.66s/it  [10:56:09<20:11:27]  mem=34.07GB
full_cpt step  7120/12852 ( 55.4%)  nll=3.2273  ppl=25.21  grad=0.840  lr=9.61e-06  6851 tok/s  12.66s/it  [10:58:05<20:09:01]  mem=34.07GB
full_cpt step  7130/12852 ( 55.5%)  nll=3.3603  ppl=28.80  grad=0.910  lr=9.58e-06  6852 tok/s  12.65s/it  [11:00:01<20:06:35]  mem=34.07GB
full_cpt step  7140/12852 ( 55.6%)  nll=3.3376  ppl=28.15  grad=0.879  lr=9.56e-06  6854 tok/s  12.65s/it  [11:01:57<20:04:10]  mem=34.07GB
full_cpt step  7150/12852 ( 55.6%)  nll=3.2913  ppl=26.88  grad=0.879  lr=9.54e-06  6856 tok/s  12.65s/it  [11:03:53<20:01:45]  mem=34.07GB
full_cpt step  7160/12852 ( 55.7%)  nll=3.2475  ppl=25.73  grad=0.836  lr=9.52e-06  6859 tok/s  12.64s/it  [11:05:49<19:59:19]  mem=34.07GB
full_cpt step  7170/12852 ( 55.8%)  nll=3.4687  ppl=32.10  grad=0.867  lr=9.50e-06  6860 tok/s  12.64s/it  [11:07:45<19:56:54]  mem=34.07GB
full_cpt step  7180/12852 ( 55.9%)  nll=3.2642  ppl=26.16  grad=0.863  lr=9.47e-06  6862 tok/s  12.64s/it  [11:09:41<19:54:29]  mem=34.07GB
full_cpt step  7190/12852 ( 55.9%)  nll=3.2787  ppl=26.54  grad=0.867  lr=9.45e-06  6863 tok/s  12.63s/it  [11:11:37<19:52:04]  mem=34.07GB
full_cpt step  7200/12852 ( 56.0%)  nll=3.3572  ppl=28.71  grad=0.871  lr=9.43e-06  6865 tok/s  12.63s/it  [11:13:33<19:49:39]  mem=34.07GB
  am   nll=3.9711  ppl=53.04  bpb=1.3964
  ta   nll=3.9439  ppl=51.62  bpb=0.5728
  kk   nll=2.7501  ppl=15.64  bpb=0.9261
  tl   nll=2.6598  ppl=14.29  bpb=1.2620
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 7200  01:27 (86.8s)
full_cpt step  7210/12852 ( 56.1%)  nll=3.2478  ppl=25.73  grad=0.859  lr=9.41e-06  6852 tok/s  12.65s/it  [11:16:55<19:49:47]  mem=34.07GB
full_cpt step  7220/12852 ( 56.2%)  nll=3.3286  ppl=27.90  grad=0.875  lr=9.39e-06  6854 tok/s  12.65s/it  [11:18:51<19:47:22]  mem=34.07GB
full_cpt step  7230/12852 ( 56.3%)  nll=3.2479  ppl=25.74  grad=0.895  lr=9.36e-06  6855 tok/s  12.65s/it  [11:20:47<19:44:57]  mem=34.07GB
full_cpt step  7240/12852 ( 56.3%)  nll=3.2291  ppl=25.26  grad=0.902  lr=9.34e-06  6857 tok/s  12.64s/it  [11:22:43<19:42:32]  mem=34.07GB
full_cpt step  7250/12852 ( 56.4%)  nll=3.4218  ppl=30.62  grad=0.848  lr=9.32e-06  6859 tok/s  12.64s/it  [11:24:39<19:40:08]  mem=34.07GB
full_cpt step  7260/12852 ( 56.5%)  nll=3.3661  ppl=28.97  grad=0.887  lr=9.30e-06  6860 tok/s  12.64s/it  [11:26:35<19:37:43]  mem=34.07GB
full_cpt step  7270/12852 ( 56.6%)  nll=3.3162  ppl=27.56  grad=0.965  lr=9.28e-06  6862 tok/s  12.63s/it  [11:28:31<19:35:19]  mem=34.07GB
full_cpt step  7280/12852 ( 56.6%)  nll=3.3324  ppl=28.01  grad=0.902  lr=9.26e-06  6864 tok/s  12.63s/it  [11:30:27<19:32:55]  mem=34.07GB
full_cpt step  7290/12852 ( 56.7%)  nll=3.2756  ppl=26.46  grad=0.836  lr=9.23e-06  6866 tok/s  12.63s/it  [11:32:23<19:30:31]  mem=34.07GB
full_cpt step  7300/12852 ( 56.8%)  nll=3.1915  ppl=24.33  grad=0.887  lr=9.21e-06  6867 tok/s  12.62s/it  [11:34:19<19:28:08]  mem=34.07GB
  am   nll=3.9706  ppl=53.01  bpb=1.3963
  ta   nll=3.9442  ppl=51.63  bpb=0.5729
  kk   nll=2.7502  ppl=15.65  bpb=0.9261
  tl   nll=2.6598  ppl=14.29  bpb=1.2620
  en   nll=2.8173  ppl=16.73  bpb=0.8772
eval full_cpt step 7300  01:26 (86.4s)
full_cpt step  7310/12852 ( 56.9%)  nll=3.3443  ppl=28.34  grad=0.871  lr=9.19e-06  6855 tok/s  12.65s/it  [11:37:41<19:28:09]  mem=34.07GB
full_cpt step  7320/12852 ( 57.0%)  nll=3.2857  ppl=26.73  grad=0.840  lr=9.17e-06  6857 tok/s  12.64s/it  [11:39:37<19:25:45]  mem=34.07GB
full_cpt step  7330/12852 ( 57.0%)  nll=3.2615  ppl=26.09  grad=0.855  lr=9.15e-06  6858 tok/s  12.64s/it  [11:41:33<19:23:21]  mem=34.07GB
full_cpt step  7340/12852 ( 57.1%)  nll=3.1866  ppl=24.21  grad=0.934  lr=9.13e-06  6860 tok/s  12.64s/it  [11:43:29<19:20:57]  mem=34.07GB
full_cpt step  7350/12852 ( 57.2%)  nll=3.3548  ppl=28.64  grad=0.852  lr=9.10e-06  6861 tok/s  12.63s/it  [11:45:25<19:18:34]  mem=34.07GB
full_cpt step  7360/12852 ( 57.3%)  nll=3.3198  ppl=27.65  grad=0.871  lr=9.08e-06  6863 tok/s  12.63s/it  [11:47:21<19:16:10]  mem=34.07GB
full_cpt step  7370/12852 ( 57.3%)  nll=3.2278  ppl=25.22  grad=0.898  lr=9.06e-06  6865 tok/s  12.63s/it  [11:49:17<19:13:47]  mem=34.07GB
full_cpt step  7380/12852 ( 57.4%)  nll=3.3613  ppl=28.83  grad=0.824  lr=9.04e-06  6867 tok/s  12.63s/it  [11:51:13<19:11:24]  mem=34.07GB
full_cpt step  7390/12852 ( 57.5%)  nll=3.2545  ppl=25.91  grad=0.844  lr=9.02e-06  6869 tok/s  12.62s/it  [11:53:09<19:09:01]  mem=34.07GB
full_cpt step  7400/12852 ( 57.6%)  nll=3.3893  ppl=29.64  grad=0.859  lr=8.99e-06  6870 tok/s  12.62s/it  [11:55:05<19:06:39]  mem=34.07GB
  am   nll=3.9706  ppl=53.01  bpb=1.3963
  ta   nll=3.9437  ppl=51.61  bpb=0.5728
  kk   nll=2.7503  ppl=15.65  bpb=0.9261
  tl   nll=2.6597  ppl=14.29  bpb=1.2620
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 7400  01:27 (86.5s)
full_cpt step  7410/12852 ( 57.7%)  nll=3.2257  ppl=25.17  grad=0.879  lr=8.97e-06  6858 tok/s  12.64s/it  [11:58:27<19:06:34]  mem=34.07GB
full_cpt step  7420/12852 ( 57.7%)  nll=3.2082  ppl=24.73  grad=0.891  lr=8.95e-06  6859 tok/s  12.64s/it  [12:00:23<19:04:11]  mem=34.07GB
full_cpt step  7430/12852 ( 57.8%)  nll=3.2458  ppl=25.68  grad=0.895  lr=8.93e-06  6861 tok/s  12.64s/it  [12:02:19<19:01:48]  mem=34.07GB
full_cpt step  7440/12852 ( 57.9%)  nll=3.3389  ppl=28.19  grad=0.871  lr=8.91e-06  6863 tok/s  12.63s/it  [12:04:15<18:59:26]  mem=34.07GB
full_cpt step  7450/12852 ( 58.0%)  nll=3.2336  ppl=25.37  grad=0.844  lr=8.89e-06  6864 tok/s  12.63s/it  [12:06:11<18:57:03]  mem=34.07GB
full_cpt step  7460/12852 ( 58.0%)  nll=3.2342  ppl=25.39  grad=0.891  lr=8.87e-06  6866 tok/s  12.63s/it  [12:08:07<18:54:41]  mem=34.07GB
full_cpt step  7470/12852 ( 58.1%)  nll=3.3145  ppl=27.51  grad=0.875  lr=8.84e-06  6867 tok/s  12.62s/it  [12:10:03<18:52:19]  mem=34.07GB
full_cpt step  7480/12852 ( 58.2%)  nll=3.2484  ppl=25.75  grad=0.887  lr=8.82e-06  6868 tok/s  12.62s/it  [12:11:59<18:49:56]  mem=34.07GB
full_cpt step  7490/12852 ( 58.3%)  nll=3.3281  ppl=27.89  grad=0.887  lr=8.80e-06  6870 tok/s  12.62s/it  [12:13:55<18:47:34]  mem=34.07GB
full_cpt step  7500/12852 ( 58.4%)  nll=3.3426  ppl=28.29  grad=0.828  lr=8.78e-06  6872 tok/s  12.61s/it  [12:15:51<18:45:13]  mem=34.07GB
  am   nll=3.9705  ppl=53.01  bpb=1.3962
  ta   nll=3.9442  ppl=51.63  bpb=0.5729
  kk   nll=2.7500  ppl=15.64  bpb=0.9260
  tl   nll=2.6594  ppl=14.29  bpb=1.2619
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 7500  01:29 (88.8s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.32s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.32s/it]
removing old checkpoint checkpoint-6500
kept last 2 resume checkpoints: checkpoint-7000 checkpoint-7500
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-7500 at step 7500 next_row=960000
full_cpt step  7510/12852 ( 58.4%)  nll=3.2909  ppl=26.87  grad=0.816  lr=8.76e-06  6847 tok/s  12.66s/it  [12:20:35<18:47:08]  mem=34.07GB
full_cpt step  7520/12852 ( 58.5%)  nll=3.2871  ppl=26.76  grad=0.828  lr=8.74e-06  6849 tok/s  12.66s/it  [12:22:32<18:44:46]  mem=34.07GB
full_cpt step  7530/12852 ( 58.6%)  nll=3.2609  ppl=26.07  grad=0.840  lr=8.71e-06  6851 tok/s  12.65s/it  [12:24:28<18:42:23]  mem=34.07GB
full_cpt step  7540/12852 ( 58.7%)  nll=3.2925  ppl=26.91  grad=0.887  lr=8.69e-06  6852 tok/s  12.65s/it  [12:26:23<18:40:01]  mem=34.07GB
full_cpt step  7550/12852 ( 58.7%)  nll=3.2524  ppl=25.85  grad=0.848  lr=8.67e-06  6853 tok/s  12.65s/it  [12:28:20<18:37:38]  mem=34.07GB
full_cpt step  7560/12852 ( 58.8%)  nll=3.2874  ppl=26.77  grad=0.902  lr=8.65e-06  6855 tok/s  12.64s/it  [12:30:16<18:35:16]  mem=34.07GB
full_cpt step  7570/12852 ( 58.9%)  nll=3.3021  ppl=27.17  grad=0.914  lr=8.63e-06  6856 tok/s  12.64s/it  [12:32:12<18:32:55]  mem=34.07GB
full_cpt step  7580/12852 ( 59.0%)  nll=3.2742  ppl=26.42  grad=0.875  lr=8.61e-06  6857 tok/s  12.64s/it  [12:34:08<18:30:33]  mem=34.07GB
full_cpt step  7590/12852 ( 59.1%)  nll=3.3652  ppl=28.94  grad=0.832  lr=8.59e-06  6859 tok/s  12.64s/it  [12:36:04<18:28:11]  mem=34.07GB
full_cpt step  7600/12852 ( 59.1%)  nll=3.3812  ppl=29.41  grad=0.898  lr=8.56e-06  6860 tok/s  12.63s/it  [12:37:59<18:25:49]  mem=34.07GB
  am   nll=3.9704  ppl=53.00  bpb=1.3962
  ta   nll=3.9439  ppl=51.62  bpb=0.5728
  kk   nll=2.7499  ppl=15.64  bpb=0.9260
  tl   nll=2.6594  ppl=14.29  bpb=1.2618
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 7600  01:29 (89.1s)
full_cpt step  7610/12852 ( 59.2%)  nll=3.2511  ppl=25.82  grad=0.867  lr=8.54e-06  6847 tok/s  12.65s/it  [12:41:24<18:25:37]  mem=34.07GB
full_cpt step  7620/12852 ( 59.3%)  nll=3.4024  ppl=30.04  grad=0.875  lr=8.52e-06  6849 tok/s  12.65s/it  [12:43:20<18:23:16]  mem=34.07GB
full_cpt step  7630/12852 ( 59.4%)  nll=3.3702  ppl=29.08  grad=0.867  lr=8.50e-06  6851 tok/s  12.65s/it  [12:45:17<18:20:54]  mem=34.07GB
full_cpt step  7640/12852 ( 59.4%)  nll=3.3735  ppl=29.18  grad=0.852  lr=8.48e-06  6852 tok/s  12.65s/it  [12:47:13<18:18:33]  mem=34.07GB
full_cpt step  7650/12852 ( 59.5%)  nll=3.3221  ppl=27.72  grad=0.879  lr=8.46e-06  6854 tok/s  12.64s/it  [12:49:09<18:16:11]  mem=34.07GB
full_cpt step  7660/12852 ( 59.6%)  nll=3.2162  ppl=24.93  grad=0.844  lr=8.44e-06  6855 tok/s  12.64s/it  [12:51:05<18:13:50]  mem=34.07GB
full_cpt step  7670/12852 ( 59.7%)  nll=3.2502  ppl=25.80  grad=0.848  lr=8.41e-06  6857 tok/s  12.64s/it  [12:53:01<18:11:29]  mem=34.07GB
full_cpt step  7680/12852 ( 59.8%)  nll=3.3408  ppl=28.24  grad=0.840  lr=8.39e-06  6859 tok/s  12.63s/it  [12:54:57<18:09:08]  mem=34.07GB
full_cpt step  7690/12852 ( 59.8%)  nll=3.3039  ppl=27.22  grad=0.852  lr=8.37e-06  6860 tok/s  12.63s/it  [12:56:53<18:06:47]  mem=34.07GB
full_cpt step  7700/12852 ( 59.9%)  nll=3.3040  ppl=27.22  grad=0.852  lr=8.35e-06  6862 tok/s  12.63s/it  [12:58:49<18:04:27]  mem=34.07GB
  am   nll=3.9704  ppl=53.01  bpb=1.3962
  ta   nll=3.9439  ppl=51.62  bpb=0.5728
  kk   nll=2.7499  ppl=15.64  bpb=0.9260
  tl   nll=2.6595  ppl=14.29  bpb=1.2619
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 7700  01:26 (86.4s)
full_cpt step  7710/12852 ( 60.0%)  nll=3.3561  ppl=28.68  grad=0.840  lr=8.33e-06  6851 tok/s  12.65s/it  [13:02:11<18:04:05]  mem=34.07GB
full_cpt step  7720/12852 ( 60.1%)  nll=3.3603  ppl=28.80  grad=0.922  lr=8.31e-06  6851 tok/s  12.65s/it  [13:04:06<18:01:44]  mem=34.07GB
full_cpt step  7730/12852 ( 60.1%)  nll=3.1528  ppl=23.40  grad=0.832  lr=8.29e-06  6853 tok/s  12.64s/it  [13:06:02<17:59:23]  mem=34.07GB
full_cpt step  7740/12852 ( 60.2%)  nll=3.2494  ppl=25.77  grad=0.855  lr=8.27e-06  6855 tok/s  12.64s/it  [13:07:58<17:57:02]  mem=34.07GB
full_cpt step  7750/12852 ( 60.3%)  nll=3.2215  ppl=25.07  grad=0.898  lr=8.25e-06  6856 tok/s  12.64s/it  [13:09:54<17:54:42]  mem=34.07GB
full_cpt step  7760/12852 ( 60.4%)  nll=3.1838  ppl=24.14  grad=0.879  lr=8.22e-06  6858 tok/s  12.64s/it  [13:11:50<17:52:21]  mem=34.07GB
full_cpt step  7770/12852 ( 60.5%)  nll=3.3091  ppl=27.36  grad=0.926  lr=8.20e-06  6859 tok/s  12.63s/it  [13:13:46<17:50:01]  mem=34.07GB
full_cpt step  7780/12852 ( 60.5%)  nll=3.1512  ppl=23.36  grad=0.828  lr=8.18e-06  6861 tok/s  12.63s/it  [13:15:42<17:47:41]  mem=34.07GB
full_cpt step  7790/12852 ( 60.6%)  nll=3.3210  ppl=27.69  grad=0.902  lr=8.16e-06  6862 tok/s  12.63s/it  [13:17:38<17:45:20]  mem=34.07GB
full_cpt step  7800/12852 ( 60.7%)  nll=3.3420  ppl=28.27  grad=0.883  lr=8.14e-06  6863 tok/s  12.62s/it  [13:19:34<17:43:00]  mem=34.07GB
  am   nll=3.9705  ppl=53.01  bpb=1.3962
  ta   nll=3.9435  ppl=51.60  bpb=0.5728
  kk   nll=2.7499  ppl=15.64  bpb=0.9260
  tl   nll=2.6597  ppl=14.29  bpb=1.2620
  en   nll=2.8175  ppl=16.73  bpb=0.8772
eval full_cpt step 7800  01:26 (86.3s)
full_cpt step  7810/12852 ( 60.8%)  nll=3.3332  ppl=28.03  grad=0.832  lr=8.12e-06  6853 tok/s  12.64s/it  [13:22:56<17:42:35]  mem=34.07GB
full_cpt step  7820/12852 ( 60.8%)  nll=3.2543  ppl=25.90  grad=0.832  lr=8.10e-06  6854 tok/s  12.64s/it  [13:24:52<17:40:14]  mem=34.07GB
full_cpt step  7830/12852 ( 60.9%)  nll=3.3115  ppl=27.43  grad=0.879  lr=8.08e-06  6856 tok/s  12.64s/it  [13:26:48<17:37:54]  mem=34.07GB
full_cpt step  7840/12852 ( 61.0%)  nll=3.3011  ppl=27.14  grad=0.863  lr=8.06e-06  6857 tok/s  12.64s/it  [13:28:44<17:35:34]  mem=34.07GB
full_cpt step  7850/12852 ( 61.1%)  nll=3.2757  ppl=26.46  grad=0.953  lr=8.03e-06  6858 tok/s  12.63s/it  [13:30:40<17:33:14]  mem=34.07GB
full_cpt step  7860/12852 ( 61.2%)  nll=3.4445  ppl=31.33  grad=0.930  lr=8.01e-06  6859 tok/s  12.63s/it  [13:32:36<17:30:54]  mem=34.07GB
full_cpt step  7870/12852 ( 61.2%)  nll=3.3976  ppl=29.89  grad=0.867  lr=7.99e-06  6861 tok/s  12.63s/it  [13:34:32<17:28:34]  mem=34.07GB
full_cpt step  7880/12852 ( 61.3%)  nll=3.1665  ppl=23.72  grad=0.816  lr=7.97e-06  6863 tok/s  12.63s/it  [13:36:28<17:26:15]  mem=34.07GB
full_cpt step  7890/12852 ( 61.4%)  nll=3.2262  ppl=25.18  grad=0.867  lr=7.95e-06  6864 tok/s  12.62s/it  [13:38:24<17:23:56]  mem=34.07GB
full_cpt step  7900/12852 ( 61.5%)  nll=3.2984  ppl=27.07  grad=0.828  lr=7.93e-06  6866 tok/s  12.62s/it  [13:40:20<17:21:36]  mem=34.07GB
  am   nll=3.9700  ppl=52.99  bpb=1.3961
  ta   nll=3.9439  ppl=51.62  bpb=0.5728
  kk   nll=2.7494  ppl=15.63  bpb=0.9259
  tl   nll=2.6595  ppl=14.29  bpb=1.2619
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 7900  01:26 (86.2s)
full_cpt step  7910/12852 ( 61.5%)  nll=3.3325  ppl=28.01  grad=0.895  lr=7.91e-06  6855 tok/s  12.64s/it  [13:43:42<17:21:06]  mem=34.07GB
full_cpt step  7920/12852 ( 61.6%)  nll=3.3036  ppl=27.21  grad=0.836  lr=7.89e-06  6857 tok/s  12.64s/it  [13:45:38<17:18:47]  mem=34.07GB
full_cpt step  7930/12852 ( 61.7%)  nll=3.3334  ppl=28.03  grad=0.898  lr=7.87e-06  6859 tok/s  12.63s/it  [13:47:34<17:16:27]  mem=34.07GB
full_cpt step  7940/12852 ( 61.8%)  nll=3.3658  ppl=28.96  grad=0.852  lr=7.85e-06  6860 tok/s  12.63s/it  [13:49:30<17:14:08]  mem=34.07GB
full_cpt step  7950/12852 ( 61.9%)  nll=3.2885  ppl=26.80  grad=0.828  lr=7.83e-06  6862 tok/s  12.63s/it  [13:51:26<17:11:49]  mem=34.07GB
full_cpt step  7960/12852 ( 61.9%)  nll=3.3966  ppl=29.86  grad=0.840  lr=7.80e-06  6863 tok/s  12.63s/it  [13:53:21<17:09:29]  mem=34.07GB
full_cpt step  7970/12852 ( 62.0%)  nll=3.2600  ppl=26.05  grad=0.852  lr=7.78e-06  6864 tok/s  12.62s/it  [13:55:17<17:07:11]  mem=34.07GB
full_cpt step  7980/12852 ( 62.1%)  nll=3.2507  ppl=25.81  grad=0.863  lr=7.76e-06  6866 tok/s  12.62s/it  [13:57:14<17:04:52]  mem=34.07GB
full_cpt step  7990/12852 ( 62.2%)  nll=3.2973  ppl=27.04  grad=0.867  lr=7.74e-06  6868 tok/s  12.62s/it  [13:59:10<17:02:33]  mem=34.07GB
full_cpt step  8000/12852 ( 62.2%)  nll=3.2094  ppl=24.76  grad=0.836  lr=7.72e-06  6869 tok/s  12.62s/it  [14:01:05<17:00:15]  mem=34.07GB
  am   nll=3.9697  ppl=52.97  bpb=1.3960
  ta   nll=3.9438  ppl=51.61  bpb=0.5728
  kk   nll=2.7494  ppl=15.63  bpb=0.9258
  tl   nll=2.6594  ppl=14.29  bpb=1.2618
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 8000  01:27 (86.6s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.32s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.33s/it]
removing old checkpoint checkpoint-7000
kept last 2 resume checkpoints: checkpoint-7500 checkpoint-8000
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-8000 at step 8000 next_row=1024000

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.63s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.64s/it]
wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-8000 (weights+expanded tokenizer, no optimizer)
full_cpt step  8010/12852 ( 62.3%)  nll=3.1602  ppl=23.57  grad=0.855  lr=7.70e-06  6845 tok/s  12.66s/it  [14:06:10<17:01:44]  mem=34.07GB
full_cpt step  8020/12852 ( 62.4%)  nll=3.2906  ppl=26.86  grad=0.863  lr=7.68e-06  6846 tok/s  12.66s/it  [14:08:06<16:59:25]  mem=34.07GB
full_cpt step  8030/12852 ( 62.5%)  nll=3.3302  ppl=27.94  grad=0.840  lr=7.66e-06  6848 tok/s  12.66s/it  [14:10:02<16:57:05]  mem=34.07GB
full_cpt step  8040/12852 ( 62.6%)  nll=3.2308  ppl=25.30  grad=0.875  lr=7.64e-06  6849 tok/s  12.65s/it  [14:11:58<16:54:46]  mem=34.07GB
full_cpt step  8050/12852 ( 62.6%)  nll=3.1966  ppl=24.45  grad=0.895  lr=7.62e-06  6850 tok/s  12.65s/it  [14:13:54<16:52:27]  mem=34.07GB
full_cpt step  8060/12852 ( 62.7%)  nll=3.2061  ppl=24.68  grad=0.891  lr=7.60e-06  6852 tok/s  12.65s/it  [14:15:50<16:50:08]  mem=34.07GB
full_cpt step  8070/12852 ( 62.8%)  nll=3.2828  ppl=26.65  grad=0.898  lr=7.58e-06  6853 tok/s  12.65s/it  [14:17:46<16:47:49]  mem=34.07GB
full_cpt step  8080/12852 ( 62.9%)  nll=3.3417  ppl=28.27  grad=0.941  lr=7.56e-06  6855 tok/s  12.64s/it  [14:19:41<16:45:30]  mem=34.07GB
full_cpt step  8090/12852 ( 62.9%)  nll=3.3617  ppl=28.84  grad=0.898  lr=7.54e-06  6856 tok/s  12.64s/it  [14:21:37<16:43:11]  mem=34.07GB
full_cpt step  8100/12852 ( 63.0%)  nll=3.2946  ppl=26.97  grad=0.902  lr=7.52e-06  6857 tok/s  12.64s/it  [14:23:33<16:40:53]  mem=34.07GB
  am   nll=3.9703  ppl=53.00  bpb=1.3962
  ta   nll=3.9437  ppl=51.61  bpb=0.5728
  kk   nll=2.7496  ppl=15.64  bpb=0.9259
  tl   nll=2.6594  ppl=14.29  bpb=1.2619
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 8100  01:26 (86.3s)
full_cpt step  8110/12852 ( 63.1%)  nll=3.2212  ppl=25.06  grad=0.816  lr=7.50e-06  6848 tok/s  12.66s/it  [14:26:55<16:40:14]  mem=34.07GB
full_cpt step  8120/12852 ( 63.2%)  nll=3.2217  ppl=25.07  grad=0.867  lr=7.47e-06  6849 tok/s  12.65s/it  [14:28:51<16:37:55]  mem=34.07GB
full_cpt step  8130/12852 ( 63.3%)  nll=3.3605  ppl=28.80  grad=0.918  lr=7.45e-06  6851 tok/s  12.65s/it  [14:30:47<16:35:36]  mem=34.07GB
full_cpt step  8140/12852 ( 63.3%)  nll=3.2702  ppl=26.32  grad=0.895  lr=7.43e-06  6852 tok/s  12.65s/it  [14:32:43<16:33:18]  mem=34.07GB
full_cpt step  8150/12852 ( 63.4%)  nll=3.3433  ppl=28.31  grad=0.863  lr=7.41e-06  6853 tok/s  12.65s/it  [14:34:39<16:30:59]  mem=34.07GB
full_cpt step  8160/12852 ( 63.5%)  nll=3.3018  ppl=27.16  grad=0.910  lr=7.39e-06  6855 tok/s  12.64s/it  [14:36:35<16:28:41]  mem=34.07GB
full_cpt step  8170/12852 ( 63.6%)  nll=3.2327  ppl=25.35  grad=0.812  lr=7.37e-06  6856 tok/s  12.64s/it  [14:38:31<16:26:23]  mem=34.07GB
full_cpt step  8180/12852 ( 63.6%)  nll=3.2686  ppl=26.27  grad=0.875  lr=7.35e-06  6857 tok/s  12.64s/it  [14:40:27<16:24:05]  mem=34.07GB
full_cpt step  8190/12852 ( 63.7%)  nll=3.3119  ppl=27.44  grad=0.887  lr=7.33e-06  6858 tok/s  12.64s/it  [14:42:23<16:21:47]  mem=34.07GB
full_cpt step  8200/12852 ( 63.8%)  nll=3.2357  ppl=25.42  grad=0.859  lr=7.31e-06  6860 tok/s  12.63s/it  [14:44:19<16:19:29]  mem=34.07GB
  am   nll=3.9700  ppl=52.98  bpb=1.3960
  ta   nll=3.9436  ppl=51.61  bpb=0.5728
  kk   nll=2.7495  ppl=15.64  bpb=0.9259
  tl   nll=2.6594  ppl=14.29  bpb=1.2618
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 8200  01:26 (86.2s)
full_cpt step  8210/12852 ( 63.9%)  nll=3.3063  ppl=27.28  grad=0.910  lr=7.29e-06  6850 tok/s  12.65s/it  [14:47:41<16:18:46]  mem=34.07GB
full_cpt step  8220/12852 ( 64.0%)  nll=3.1597  ppl=23.56  grad=0.859  lr=7.27e-06  6852 tok/s  12.65s/it  [14:49:37<16:16:28]  mem=34.07GB
full_cpt step  8230/12852 ( 64.0%)  nll=3.2571  ppl=25.97  grad=0.891  lr=7.25e-06  6853 tok/s  12.65s/it  [14:51:33<16:14:10]  mem=34.07GB
full_cpt step  8240/12852 ( 64.1%)  nll=3.4470  ppl=31.41  grad=0.871  lr=7.23e-06  6854 tok/s  12.64s/it  [14:53:28<16:11:52]  mem=34.07GB
full_cpt step  8250/12852 ( 64.2%)  nll=3.2754  ppl=26.45  grad=0.852  lr=7.21e-06  6855 tok/s  12.64s/it  [14:55:24<16:09:34]  mem=34.07GB
full_cpt step  8260/12852 ( 64.3%)  nll=3.3093  ppl=27.37  grad=0.883  lr=7.19e-06  6857 tok/s  12.64s/it  [14:57:20<16:07:16]  mem=34.07GB
full_cpt step  8270/12852 ( 64.3%)  nll=3.2490  ppl=25.76  grad=0.898  lr=7.17e-06  6859 tok/s  12.64s/it  [14:59:16<16:04:59]  mem=34.07GB
full_cpt step  8280/12852 ( 64.4%)  nll=3.1901  ppl=24.29  grad=0.840  lr=7.15e-06  6860 tok/s  12.63s/it  [15:01:12<16:02:41]  mem=34.07GB
full_cpt step  8290/12852 ( 64.5%)  nll=3.3850  ppl=29.52  grad=0.906  lr=7.13e-06  6861 tok/s  12.63s/it  [15:03:08<16:00:24]  mem=34.07GB
full_cpt step  8300/12852 ( 64.6%)  nll=3.2681  ppl=26.26  grad=0.898  lr=7.11e-06  6862 tok/s  12.63s/it  [15:05:04<15:58:06]  mem=34.07GB
  am   nll=3.9696  ppl=52.97  bpb=1.3959
  ta   nll=3.9433  ppl=51.59  bpb=0.5727
  kk   nll=2.7496  ppl=15.64  bpb=0.9259
  tl   nll=2.6595  ppl=14.29  bpb=1.2619
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 8300  01:30 (89.5s)
full_cpt step  8310/12852 ( 64.7%)  nll=3.3057  ppl=27.27  grad=0.879  lr=7.09e-06  6852 tok/s  12.65s/it  [15:08:29<15:57:23]  mem=34.07GB
full_cpt step  8320/12852 ( 64.7%)  nll=3.3435  ppl=28.32  grad=0.848  lr=7.07e-06  6854 tok/s  12.64s/it  [15:10:25<15:55:06]  mem=34.07GB
full_cpt step  8330/12852 ( 64.8%)  nll=3.2883  ppl=26.80  grad=0.871  lr=7.05e-06  6855 tok/s  12.64s/it  [15:12:21<15:52:49]  mem=34.07GB
full_cpt step  8340/12852 ( 64.9%)  nll=3.3330  ppl=28.02  grad=0.836  lr=7.03e-06  6856 tok/s  12.64s/it  [15:14:17<15:50:31]  mem=34.07GB
full_cpt step  8350/12852 ( 65.0%)  nll=3.2772  ppl=26.50  grad=0.828  lr=7.01e-06  6858 tok/s  12.64s/it  [15:16:13<15:48:14]  mem=34.07GB
full_cpt step  8360/12852 ( 65.0%)  nll=3.4091  ppl=30.24  grad=0.887  lr=6.99e-06  6859 tok/s  12.64s/it  [15:18:09<15:45:57]  mem=34.07GB
full_cpt step  8370/12852 ( 65.1%)  nll=3.2744  ppl=26.43  grad=0.852  lr=6.97e-06  6860 tok/s  12.63s/it  [15:20:05<15:43:40]  mem=34.07GB
full_cpt step  8380/12852 ( 65.2%)  nll=3.3816  ppl=29.42  grad=0.938  lr=6.95e-06  6861 tok/s  12.63s/it  [15:22:01<15:41:23]  mem=34.07GB
full_cpt step  8390/12852 ( 65.3%)  nll=3.3735  ppl=29.18  grad=0.883  lr=6.93e-06  6863 tok/s  12.63s/it  [15:23:57<15:39:06]  mem=34.07GB
full_cpt step  8400/12852 ( 65.4%)  nll=3.2125  ppl=24.84  grad=0.863  lr=6.91e-06  6863 tok/s  12.63s/it  [15:25:53<15:36:49]  mem=34.07GB
  am   nll=3.9695  ppl=52.96  bpb=1.3959
  ta   nll=3.9438  ppl=51.61  bpb=0.5728
  kk   nll=2.7498  ppl=15.64  bpb=0.9260
  tl   nll=2.6594  ppl=14.29  bpb=1.2618
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 8400  01:27 (87.2s)
full_cpt step  8410/12852 ( 65.4%)  nll=3.2820  ppl=26.63  grad=0.840  lr=6.89e-06  6854 tok/s  12.64s/it  [15:29:16<15:36:00]  mem=34.07GB
full_cpt step  8420/12852 ( 65.5%)  nll=3.3200  ppl=27.66  grad=0.875  lr=6.87e-06  6855 tok/s  12.64s/it  [15:31:12<15:33:43]  mem=34.07GB
full_cpt step  8430/12852 ( 65.6%)  nll=3.2474  ppl=25.72  grad=0.832  lr=6.85e-06  6856 tok/s  12.64s/it  [15:33:07<15:31:26]  mem=34.07GB
full_cpt step  8440/12852 ( 65.7%)  nll=3.3084  ppl=27.34  grad=0.891  lr=6.83e-06  6858 tok/s  12.64s/it  [15:35:04<15:29:10]  mem=34.07GB
full_cpt step  8450/12852 ( 65.7%)  nll=3.1642  ppl=23.67  grad=0.816  lr=6.81e-06  6860 tok/s  12.63s/it  [15:37:00<15:26:53]  mem=34.07GB
full_cpt step  8460/12852 ( 65.8%)  nll=3.3078  ppl=27.33  grad=0.922  lr=6.79e-06  6861 tok/s  12.63s/it  [15:38:56<15:24:37]  mem=34.07GB
full_cpt step  8470/12852 ( 65.9%)  nll=3.2183  ppl=24.99  grad=0.863  lr=6.77e-06  6862 tok/s  12.63s/it  [15:40:52<15:22:20]  mem=34.07GB
full_cpt step  8480/12852 ( 66.0%)  nll=3.1616  ppl=23.61  grad=0.859  lr=6.75e-06  6863 tok/s  12.63s/it  [15:42:48<15:20:04]  mem=34.07GB
full_cpt step  8490/12852 ( 66.1%)  nll=3.1917  ppl=24.33  grad=0.812  lr=6.73e-06  6864 tok/s  12.62s/it  [15:44:44<15:17:48]  mem=34.07GB
full_cpt step  8500/12852 ( 66.1%)  nll=3.3693  ppl=29.06  grad=0.871  lr=6.71e-06  6866 tok/s  12.62s/it  [15:46:40<15:15:32]  mem=34.07GB
  am   nll=3.9693  ppl=52.95  bpb=1.3958
  ta   nll=3.9432  ppl=51.58  bpb=0.5727
  kk   nll=2.7498  ppl=15.64  bpb=0.9260
  tl   nll=2.6595  ppl=14.29  bpb=1.2619
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 8500  01:27 (86.7s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.62s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.63s/it]
removing old checkpoint checkpoint-7500
kept last 2 resume checkpoints: checkpoint-8000 checkpoint-8500
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-8500 at step 8500 next_row=1088000
full_cpt step  8510/12852 ( 66.2%)  nll=3.2425  ppl=25.60  grad=0.805  lr=6.70e-06  6847 tok/s  12.66s/it  [15:51:26<15:15:59]  mem=34.07GB
full_cpt step  8520/12852 ( 66.3%)  nll=3.3693  ppl=29.06  grad=0.969  lr=6.68e-06  6848 tok/s  12.66s/it  [15:53:22<15:13:42]  mem=34.07GB
full_cpt step  8530/12852 ( 66.4%)  nll=3.4402  ppl=31.19  grad=0.887  lr=6.66e-06  6849 tok/s  12.65s/it  [15:55:18<15:11:26]  mem=34.07GB
full_cpt step  8540/12852 ( 66.4%)  nll=3.3694  ppl=29.06  grad=0.883  lr=6.64e-06  6850 tok/s  12.65s/it  [15:57:14<15:09:09]  mem=34.07GB
full_cpt step  8550/12852 ( 66.5%)  nll=3.2935  ppl=26.94  grad=0.930  lr=6.62e-06  6852 tok/s  12.65s/it  [15:59:10<15:06:53]  mem=34.07GB
full_cpt step  8560/12852 ( 66.6%)  nll=3.3468  ppl=28.41  grad=0.836  lr=6.60e-06  6853 tok/s  12.65s/it  [16:01:06<15:04:37]  mem=34.07GB
full_cpt step  8570/12852 ( 66.7%)  nll=3.3175  ppl=27.59  grad=0.855  lr=6.58e-06  6854 tok/s  12.64s/it  [16:03:02<15:02:20]  mem=34.07GB
full_cpt step  8580/12852 ( 66.8%)  nll=3.3342  ppl=28.06  grad=0.945  lr=6.56e-06  6855 tok/s  12.64s/it  [16:04:58<15:00:04]  mem=34.07GB
full_cpt step  8590/12852 ( 66.8%)  nll=3.3623  ppl=28.86  grad=0.852  lr=6.54e-06  6856 tok/s  12.64s/it  [16:06:54<14:57:48]  mem=34.07GB
full_cpt step  8600/12852 ( 66.9%)  nll=3.2980  ppl=27.06  grad=0.840  lr=6.52e-06  6858 tok/s  12.64s/it  [16:08:50<14:55:32]  mem=34.07GB
  am   nll=3.9693  ppl=52.95  bpb=1.3958
  ta   nll=3.9433  ppl=51.59  bpb=0.5727
  kk   nll=2.7494  ppl=15.63  bpb=0.9258
  tl   nll=2.6593  ppl=14.29  bpb=1.2618
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 8600  01:28 (87.5s)
full_cpt step  8610/12852 ( 67.0%)  nll=3.2773  ppl=26.50  grad=0.805  lr=6.50e-06  6849 tok/s  12.65s/it  [16:12:13<14:54:37]  mem=34.07GB
full_cpt step  8620/12852 ( 67.1%)  nll=3.2131  ppl=24.85  grad=0.875  lr=6.48e-06  6850 tok/s  12.65s/it  [16:14:09<14:52:20]  mem=34.07GB
full_cpt step  8630/12852 ( 67.1%)  nll=3.3496  ppl=28.49  grad=0.883  lr=6.46e-06  6851 tok/s  12.65s/it  [16:16:05<14:50:04]  mem=34.07GB
full_cpt step  8640/12852 ( 67.2%)  nll=3.3939  ppl=29.78  grad=0.859  lr=6.44e-06  6853 tok/s  12.65s/it  [16:18:01<14:47:48]  mem=34.07GB
full_cpt step  8650/12852 ( 67.3%)  nll=3.1312  ppl=22.90  grad=0.855  lr=6.42e-06  6854 tok/s  12.64s/it  [16:19:57<14:45:32]  mem=34.07GB
full_cpt step  8660/12852 ( 67.4%)  nll=3.2593  ppl=26.03  grad=0.910  lr=6.41e-06  6856 tok/s  12.64s/it  [16:21:53<14:43:16]  mem=34.07GB
full_cpt step  8670/12852 ( 67.5%)  nll=3.2149  ppl=24.90  grad=0.875  lr=6.39e-06  6857 tok/s  12.64s/it  [16:23:49<14:41:01]  mem=34.07GB
full_cpt step  8680/12852 ( 67.5%)  nll=3.1961  ppl=24.44  grad=0.906  lr=6.37e-06  6858 tok/s  12.64s/it  [16:25:45<14:38:45]  mem=34.07GB
full_cpt step  8690/12852 ( 67.6%)  nll=3.2869  ppl=26.76  grad=0.871  lr=6.35e-06  6860 tok/s  12.64s/it  [16:27:41<14:36:30]  mem=34.07GB
full_cpt step  8700/12852 ( 67.7%)  nll=3.2899  ppl=26.84  grad=0.820  lr=6.33e-06  6861 tok/s  12.63s/it  [16:29:37<14:34:14]  mem=34.07GB
  am   nll=3.9697  ppl=52.97  bpb=1.3959
  ta   nll=3.9432  ppl=51.58  bpb=0.5727
  kk   nll=2.7496  ppl=15.64  bpb=0.9259
  tl   nll=2.6594  ppl=14.29  bpb=1.2618
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 8700  01:27 (87.0s)
full_cpt step  8710/12852 ( 67.8%)  nll=3.3029  ppl=27.19  grad=0.883  lr=6.31e-06  6852 tok/s  12.65s/it  [16:33:00<14:33:15]  mem=34.07GB
full_cpt step  8720/12852 ( 67.8%)  nll=3.2654  ppl=26.19  grad=0.875  lr=6.29e-06  6853 tok/s  12.65s/it  [16:34:56<14:30:59]  mem=34.07GB
full_cpt step  8730/12852 ( 67.9%)  nll=3.3671  ppl=28.99  grad=0.867  lr=6.27e-06  6854 tok/s  12.65s/it  [16:36:52<14:28:44]  mem=34.07GB
full_cpt step  8740/12852 ( 68.0%)  nll=3.2224  ppl=25.09  grad=0.875  lr=6.25e-06  6855 tok/s  12.64s/it  [16:38:48<14:26:28]  mem=34.07GB
full_cpt step  8750/12852 ( 68.1%)  nll=3.2239  ppl=25.13  grad=0.828  lr=6.23e-06  6857 tok/s  12.64s/it  [16:40:44<14:24:13]  mem=34.07GB
full_cpt step  8760/12852 ( 68.2%)  nll=3.2695  ppl=26.30  grad=0.832  lr=6.22e-06  6858 tok/s  12.64s/it  [16:42:40<14:21:57]  mem=34.07GB
full_cpt step  8770/12852 ( 68.2%)  nll=3.1655  ppl=23.70  grad=0.824  lr=6.20e-06  6859 tok/s  12.64s/it  [16:44:36<14:19:42]  mem=34.07GB
full_cpt step  8780/12852 ( 68.3%)  nll=3.2058  ppl=24.67  grad=0.824  lr=6.18e-06  6860 tok/s  12.63s/it  [16:46:32<14:17:27]  mem=34.07GB
full_cpt step  8790/12852 ( 68.4%)  nll=3.3325  ppl=28.01  grad=0.828  lr=6.16e-06  6862 tok/s  12.63s/it  [16:48:28<14:15:11]  mem=34.07GB
full_cpt step  8800/12852 ( 68.5%)  nll=3.3215  ppl=27.70  grad=0.887  lr=6.14e-06  6863 tok/s  12.63s/it  [16:50:24<14:12:56]  mem=34.07GB
  am   nll=3.9698  ppl=52.97  bpb=1.3960
  ta   nll=3.9434  ppl=51.59  bpb=0.5728
  kk   nll=2.7493  ppl=15.63  bpb=0.9258
  tl   nll=2.6594  ppl=14.29  bpb=1.2618
  en   nll=2.8175  ppl=16.73  bpb=0.8772
eval full_cpt step 8800  01:26 (86.4s)
full_cpt step  8810/12852 ( 68.5%)  nll=3.4132  ppl=30.36  grad=0.809  lr=6.12e-06  6854 tok/s  12.65s/it  [16:53:46<14:11:54]  mem=34.07GB
full_cpt step  8820/12852 ( 68.6%)  nll=3.3718  ppl=29.13  grad=0.855  lr=6.10e-06  6856 tok/s  12.64s/it  [16:55:42<14:09:39]  mem=34.07GB
full_cpt step  8830/12852 ( 68.7%)  nll=3.2657  ppl=26.20  grad=0.895  lr=6.08e-06  6857 tok/s  12.64s/it  [16:57:38<14:07:24]  mem=34.07GB
full_cpt step  8840/12852 ( 68.8%)  nll=3.2375  ppl=25.47  grad=0.914  lr=6.07e-06  6858 tok/s  12.64s/it  [16:59:34<14:05:09]  mem=34.07GB
full_cpt step  8850/12852 ( 68.9%)  nll=3.2847  ppl=26.70  grad=0.875  lr=6.05e-06  6860 tok/s  12.64s/it  [17:01:30<14:02:54]  mem=34.07GB
full_cpt step  8860/12852 ( 68.9%)  nll=3.3063  ppl=27.28  grad=0.926  lr=6.03e-06  6861 tok/s  12.63s/it  [17:03:26<14:00:39]  mem=34.07GB
full_cpt step  8870/12852 ( 69.0%)  nll=3.2991  ppl=27.09  grad=0.820  lr=6.01e-06  6862 tok/s  12.63s/it  [17:05:22<13:58:24]  mem=34.07GB
full_cpt step  8880/12852 ( 69.1%)  nll=3.3511  ppl=28.53  grad=0.836  lr=5.99e-06  6863 tok/s  12.63s/it  [17:07:17<13:56:09]  mem=34.07GB
full_cpt step  8890/12852 ( 69.2%)  nll=3.3804  ppl=29.38  grad=0.883  lr=5.97e-06  6864 tok/s  12.63s/it  [17:09:13<13:53:54]  mem=34.07GB
full_cpt step  8900/12852 ( 69.2%)  nll=3.2070  ppl=24.70  grad=0.910  lr=5.96e-06  6865 tok/s  12.63s/it  [17:11:09<13:51:40]  mem=34.07GB
  am   nll=3.9696  ppl=52.96  bpb=1.3959
  ta   nll=3.9432  ppl=51.58  bpb=0.5727
  kk   nll=2.7494  ppl=15.63  bpb=0.9258
  tl   nll=2.6591  ppl=14.28  bpb=1.2617
  en   nll=2.8173  ppl=16.73  bpb=0.8772
eval full_cpt step 8900  01:26 (86.2s)
full_cpt step  8910/12852 ( 69.3%)  nll=3.3377  ppl=28.15  grad=0.820  lr=5.94e-06  6857 tok/s  12.64s/it  [17:14:32<13:50:34]  mem=34.07GB
full_cpt step  8920/12852 ( 69.4%)  nll=3.3744  ppl=29.21  grad=0.895  lr=5.92e-06  6858 tok/s  12.64s/it  [17:16:27<13:48:19]  mem=34.07GB
full_cpt step  8930/12852 ( 69.5%)  nll=3.3003  ppl=27.12  grad=0.836  lr=5.90e-06  6859 tok/s  12.64s/it  [17:18:23<13:46:05]  mem=34.07GB
full_cpt step  8940/12852 ( 69.6%)  nll=3.2992  ppl=27.09  grad=0.891  lr=5.88e-06  6860 tok/s  12.64s/it  [17:20:19<13:43:50]  mem=34.07GB
full_cpt step  8950/12852 ( 69.6%)  nll=3.2359  ppl=25.43  grad=0.828  lr=5.86e-06  6862 tok/s  12.63s/it  [17:22:15<13:41:35]  mem=34.07GB
full_cpt step  8960/12852 ( 69.7%)  nll=3.2321  ppl=25.33  grad=0.863  lr=5.85e-06  6863 tok/s  12.63s/it  [17:24:11<13:39:21]  mem=34.07GB
full_cpt step  8970/12852 ( 69.8%)  nll=3.2445  ppl=25.65  grad=0.898  lr=5.83e-06  6864 tok/s  12.63s/it  [17:26:07<13:37:06]  mem=34.07GB
full_cpt step  8980/12852 ( 69.9%)  nll=3.2224  ppl=25.09  grad=0.855  lr=5.81e-06  6865 tok/s  12.63s/it  [17:28:03<13:34:52]  mem=34.07GB
full_cpt step  8990/12852 ( 70.0%)  nll=3.3423  ppl=28.28  grad=0.879  lr=5.79e-06  6866 tok/s  12.63s/it  [17:29:59<13:32:38]  mem=34.07GB
full_cpt step  9000/12852 ( 70.0%)  nll=3.4348  ppl=31.03  grad=0.844  lr=5.77e-06  6867 tok/s  12.62s/it  [17:31:55<13:30:24]  mem=34.07GB
  am   nll=3.9695  ppl=52.96  bpb=1.3959
  ta   nll=3.9431  ppl=51.58  bpb=0.5727
  kk   nll=2.7493  ppl=15.63  bpb=0.9258
  tl   nll=2.6594  ppl=14.29  bpb=1.2618
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 9000  01:26 (86.1s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.46s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.46s/it]
removing old checkpoint checkpoint-8000
kept last 2 resume checkpoints: checkpoint-8500 checkpoint-9000
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-9000 at step 9000 next_row=1152000

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.68s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.68s/it]
wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-9000 (weights+expanded tokenizer, no optimizer)
full_cpt step  9010/12852 ( 70.1%)  nll=3.3050  ppl=27.25  grad=0.812  lr=5.75e-06  6853 tok/s  12.65s/it  [17:36:18<13:30:02]  mem=34.07GB
full_cpt step  9020/12852 ( 70.2%)  nll=3.3260  ppl=27.83  grad=0.848  lr=5.74e-06  6854 tok/s  12.65s/it  [17:38:14<13:27:48]  mem=34.07GB
full_cpt step  9030/12852 ( 70.3%)  nll=3.3661  ppl=28.97  grad=0.855  lr=5.72e-06  6855 tok/s  12.65s/it  [17:40:10<13:25:33]  mem=34.07GB
full_cpt step  9040/12852 ( 70.3%)  nll=3.2030  ppl=24.61  grad=0.918  lr=5.70e-06  6856 tok/s  12.64s/it  [17:42:06<13:23:19]  mem=34.07GB
full_cpt step  9050/12852 ( 70.4%)  nll=3.3193  ppl=27.64  grad=0.902  lr=5.68e-06  6857 tok/s  12.64s/it  [17:44:01<13:21:04]  mem=34.07GB
full_cpt step  9060/12852 ( 70.5%)  nll=3.2559  ppl=25.94  grad=0.828  lr=5.66e-06  6858 tok/s  12.64s/it  [17:45:57<13:18:50]  mem=34.07GB
full_cpt step  9070/12852 ( 70.6%)  nll=3.3017  ppl=27.16  grad=0.875  lr=5.65e-06  6859 tok/s  12.64s/it  [17:47:53<13:16:36]  mem=34.07GB
full_cpt step  9080/12852 ( 70.7%)  nll=3.3319  ppl=27.99  grad=0.852  lr=5.63e-06  6860 tok/s  12.64s/it  [17:49:49<13:14:22]  mem=34.07GB
full_cpt step  9090/12852 ( 70.7%)  nll=3.3994  ppl=29.95  grad=0.855  lr=5.61e-06  6862 tok/s  12.63s/it  [17:51:45<13:12:08]  mem=34.07GB
full_cpt step  9100/12852 ( 70.8%)  nll=3.3619  ppl=28.84  grad=0.867  lr=5.59e-06  6863 tok/s  12.63s/it  [17:53:41<13:09:54]  mem=34.07GB
  am   nll=3.9695  ppl=52.96  bpb=1.3959
  ta   nll=3.9436  ppl=51.60  bpb=0.5728
  kk   nll=2.7493  ppl=15.63  bpb=0.9258
  tl   nll=2.6593  ppl=14.29  bpb=1.2618
  en   nll=2.8175  ppl=16.73  bpb=0.8772
eval full_cpt step 9100  01:26 (86.3s)
full_cpt step  9110/12852 ( 70.9%)  nll=3.2238  ppl=25.12  grad=0.875  lr=5.58e-06  6854 tok/s  12.65s/it  [17:57:03<13:08:43]  mem=34.07GB
full_cpt step  9120/12852 ( 71.0%)  nll=3.1622  ppl=23.62  grad=0.875  lr=5.56e-06  6855 tok/s  12.64s/it  [17:58:59<13:06:29]  mem=34.07GB
full_cpt step  9130/12852 ( 71.0%)  nll=3.3299  ppl=27.94  grad=0.852  lr=5.54e-06  6857 tok/s  12.64s/it  [18:00:55<13:04:15]  mem=34.07GB
full_cpt step  9140/12852 ( 71.1%)  nll=3.2462  ppl=25.69  grad=0.887  lr=5.52e-06  6858 tok/s  12.64s/it  [18:02:51<13:02:01]  mem=34.07GB
full_cpt step  9150/12852 ( 71.2%)  nll=3.4121  ppl=30.33  grad=0.898  lr=5.51e-06  6859 tok/s  12.64s/it  [18:04:47<12:59:47]  mem=34.07GB
full_cpt step  9160/12852 ( 71.3%)  nll=3.1818  ppl=24.09  grad=0.859  lr=5.49e-06  6860 tok/s  12.64s/it  [18:06:43<12:57:33]  mem=34.07GB
full_cpt step  9170/12852 ( 71.4%)  nll=3.3793  ppl=29.35  grad=0.879  lr=5.47e-06  6861 tok/s  12.63s/it  [18:08:39<12:55:19]  mem=34.07GB
full_cpt step  9180/12852 ( 71.4%)  nll=3.2624  ppl=26.11  grad=0.859  lr=5.45e-06  6863 tok/s  12.63s/it  [18:10:35<12:53:05]  mem=34.07GB
full_cpt step  9190/12852 ( 71.5%)  nll=3.2933  ppl=26.93  grad=0.867  lr=5.43e-06  6863 tok/s  12.63s/it  [18:12:31<12:50:52]  mem=34.07GB
full_cpt step  9200/12852 ( 71.6%)  nll=3.2531  ppl=25.87  grad=0.832  lr=5.42e-06  6864 tok/s  12.63s/it  [18:14:27<12:48:38]  mem=34.07GB
  am   nll=3.9695  ppl=52.96  bpb=1.3959
  ta   nll=3.9430  ppl=51.57  bpb=0.5727
  kk   nll=2.7496  ppl=15.64  bpb=0.9259
  tl   nll=2.6593  ppl=14.29  bpb=1.2618
  en   nll=2.8178  ppl=16.74  bpb=0.8773
eval full_cpt step 9200  01:26 (86.4s)
full_cpt step  9210/12852 ( 71.7%)  nll=3.2385  ppl=25.50  grad=0.871  lr=5.40e-06  6856 tok/s  12.64s/it  [18:17:49<12:47:25]  mem=34.07GB
full_cpt step  9220/12852 ( 71.7%)  nll=3.2284  ppl=25.24  grad=0.875  lr=5.38e-06  6857 tok/s  12.64s/it  [18:19:45<12:45:11]  mem=34.07GB
full_cpt step  9230/12852 ( 71.8%)  nll=3.1938  ppl=24.38  grad=0.840  lr=5.37e-06  6859 tok/s  12.64s/it  [18:21:41<12:42:58]  mem=34.07GB
full_cpt step  9240/12852 ( 71.9%)  nll=3.2554  ppl=25.93  grad=0.848  lr=5.35e-06  6860 tok/s  12.64s/it  [18:23:37<12:40:44]  mem=34.07GB
full_cpt step  9250/12852 ( 72.0%)  nll=3.2754  ppl=26.45  grad=0.828  lr=5.33e-06  6861 tok/s  12.63s/it  [18:25:33<12:38:30]  mem=34.07GB
full_cpt step  9260/12852 ( 72.1%)  nll=3.3610  ppl=28.82  grad=0.871  lr=5.31e-06  6862 tok/s  12.63s/it  [18:27:28<12:36:17]  mem=34.07GB
full_cpt step  9270/12852 ( 72.1%)  nll=3.3201  ppl=27.66  grad=0.832  lr=5.30e-06  6863 tok/s  12.63s/it  [18:29:24<12:34:04]  mem=34.07GB
full_cpt step  9280/12852 ( 72.2%)  nll=3.2430  ppl=25.61  grad=0.871  lr=5.28e-06  6864 tok/s  12.63s/it  [18:31:21<12:31:50]  mem=34.07GB
full_cpt step  9290/12852 ( 72.3%)  nll=3.4122  ppl=30.33  grad=0.844  lr=5.26e-06  6865 tok/s  12.63s/it  [18:33:16<12:29:37]  mem=34.07GB
full_cpt step  9300/12852 ( 72.4%)  nll=3.3032  ppl=27.20  grad=0.867  lr=5.24e-06  6867 tok/s  12.63s/it  [18:35:13<12:27:24]  mem=34.07GB
  am   nll=3.9695  ppl=52.96  bpb=1.3959
  ta   nll=3.9432  ppl=51.59  bpb=0.5727
  kk   nll=2.7491  ppl=15.63  bpb=0.9257
  tl   nll=2.6593  ppl=14.29  bpb=1.2618
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 9300  01:26 (86.3s)
full_cpt step  9310/12852 ( 72.4%)  nll=3.3678  ppl=29.01  grad=0.844  lr=5.23e-06  6859 tok/s  12.64s/it  [18:38:35<12:26:09]  mem=34.07GB
full_cpt step  9320/12852 ( 72.5%)  nll=3.2698  ppl=26.31  grad=0.844  lr=5.21e-06  6860 tok/s  12.64s/it  [18:40:31<12:23:55]  mem=34.07GB
full_cpt step  9330/12852 ( 72.6%)  nll=3.3573  ppl=28.71  grad=0.824  lr=5.19e-06  6861 tok/s  12.64s/it  [18:42:27<12:21:42]  mem=34.07GB
full_cpt step  9340/12852 ( 72.7%)  nll=3.3734  ppl=29.18  grad=0.840  lr=5.18e-06  6862 tok/s  12.63s/it  [18:44:23<12:19:29]  mem=34.07GB
full_cpt step  9350/12852 ( 72.8%)  nll=3.2741  ppl=26.42  grad=0.887  lr=5.16e-06  6863 tok/s  12.63s/it  [18:46:18<12:17:15]  mem=34.07GB
full_cpt step  9360/12852 ( 72.8%)  nll=3.3592  ppl=28.77  grad=0.883  lr=5.14e-06  6864 tok/s  12.63s/it  [18:48:14<12:15:02]  mem=34.07GB
full_cpt step  9370/12852 ( 72.9%)  nll=3.2532  ppl=25.87  grad=0.902  lr=5.13e-06  6865 tok/s  12.63s/it  [18:50:10<12:12:49]  mem=34.07GB
full_cpt step  9380/12852 ( 73.0%)  nll=3.3081  ppl=27.33  grad=0.859  lr=5.11e-06  6866 tok/s  12.63s/it  [18:52:06<12:10:36]  mem=34.07GB
full_cpt step  9390/12852 ( 73.1%)  nll=3.2580  ppl=26.00  grad=0.852  lr=5.09e-06  6867 tok/s  12.62s/it  [18:54:02<12:08:23]  mem=34.07GB
full_cpt step  9400/12852 ( 73.1%)  nll=3.1491  ppl=23.32  grad=0.863  lr=5.08e-06  6868 tok/s  12.62s/it  [18:55:58<12:06:10]  mem=34.07GB
  am   nll=3.9692  ppl=52.94  bpb=1.3958
  ta   nll=3.9430  ppl=51.57  bpb=0.5727
  kk   nll=2.7494  ppl=15.63  bpb=0.9258
  tl   nll=2.6592  ppl=14.28  bpb=1.2617
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 9400  01:26 (86.5s)
full_cpt step  9410/12852 ( 73.2%)  nll=3.2305  ppl=25.29  grad=0.875  lr=5.06e-06  6860 tok/s  12.64s/it  [18:59:20<12:04:53]  mem=34.07GB
full_cpt step  9420/12852 ( 73.3%)  nll=3.3492  ppl=28.48  grad=0.863  lr=5.04e-06  6861 tok/s  12.63s/it  [19:01:16<12:02:40]  mem=34.07GB
full_cpt step  9430/12852 ( 73.4%)  nll=3.2282  ppl=25.23  grad=0.840  lr=5.03e-06  6862 tok/s  12.63s/it  [19:03:12<12:00:27]  mem=34.07GB
full_cpt step  9440/12852 ( 73.5%)  nll=3.4221  ppl=30.63  grad=0.891  lr=5.01e-06  6864 tok/s  12.63s/it  [19:05:08<11:58:14]  mem=34.07GB
full_cpt step  9450/12852 ( 73.5%)  nll=3.2776  ppl=26.51  grad=0.863  lr=4.99e-06  6864 tok/s  12.63s/it  [19:07:04<11:56:01]  mem=34.07GB
full_cpt step  9460/12852 ( 73.6%)  nll=3.3667  ppl=28.98  grad=0.895  lr=4.98e-06  6865 tok/s  12.63s/it  [19:09:00<11:53:49]  mem=34.07GB
full_cpt step  9470/12852 ( 73.7%)  nll=3.3592  ppl=28.77  grad=0.848  lr=4.96e-06  6866 tok/s  12.62s/it  [19:10:56<11:51:36]  mem=34.07GB
full_cpt step  9480/12852 ( 73.8%)  nll=3.1893  ppl=24.27  grad=0.844  lr=4.94e-06  6867 tok/s  12.62s/it  [19:12:52<11:49:23]  mem=34.07GB
full_cpt step  9490/12852 ( 73.8%)  nll=3.3568  ppl=28.70  grad=0.867  lr=4.93e-06  6869 tok/s  12.62s/it  [19:14:48<11:47:11]  mem=34.07GB
full_cpt step  9500/12852 ( 73.9%)  nll=3.4310  ppl=30.91  grad=0.875  lr=4.91e-06  6870 tok/s  12.62s/it  [19:16:44<11:44:58]  mem=34.07GB
  am   nll=3.9689  ppl=52.93  bpb=1.3957
  ta   nll=3.9430  ppl=51.57  bpb=0.5727
  kk   nll=2.7491  ppl=15.63  bpb=0.9257
  tl   nll=2.6590  ppl=14.28  bpb=1.2617
  en   nll=2.8172  ppl=16.73  bpb=0.8772
eval full_cpt step 9500  01:26 (86.5s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.16s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.16s/it]
removing old checkpoint checkpoint-8500
kept last 2 resume checkpoints: checkpoint-9000 checkpoint-9500
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-9500 at step 9500 next_row=1216000
full_cpt step  9510/12852 ( 74.0%)  nll=3.3138  ppl=27.49  grad=0.875  lr=4.89e-06  6854 tok/s  12.65s/it  [19:21:34<11:44:32]  mem=34.07GB
full_cpt step  9520/12852 ( 74.1%)  nll=3.3470  ppl=28.42  grad=0.832  lr=4.88e-06  6855 tok/s  12.65s/it  [19:23:30<11:42:19]  mem=34.07GB
full_cpt step  9530/12852 ( 74.2%)  nll=3.3686  ppl=29.04  grad=0.852  lr=4.86e-06  6856 tok/s  12.64s/it  [19:25:26<11:40:06]  mem=34.07GB
full_cpt step  9540/12852 ( 74.2%)  nll=3.1781  ppl=24.00  grad=0.918  lr=4.85e-06  6857 tok/s  12.64s/it  [19:27:22<11:37:54]  mem=34.07GB
full_cpt step  9550/12852 ( 74.3%)  nll=3.3168  ppl=27.57  grad=0.891  lr=4.83e-06  6858 tok/s  12.64s/it  [19:29:18<11:35:41]  mem=34.07GB
full_cpt step  9560/12852 ( 74.4%)  nll=3.2975  ppl=27.05  grad=0.859  lr=4.81e-06  6859 tok/s  12.64s/it  [19:31:14<11:33:29]  mem=34.07GB
full_cpt step  9570/12852 ( 74.5%)  nll=3.3533  ppl=28.60  grad=0.867  lr=4.80e-06  6860 tok/s  12.64s/it  [19:33:10<11:31:16]  mem=34.07GB
full_cpt step  9580/12852 ( 74.5%)  nll=3.1986  ppl=24.50  grad=0.895  lr=4.78e-06  6861 tok/s  12.64s/it  [19:35:06<11:29:03]  mem=34.07GB
full_cpt step  9590/12852 ( 74.6%)  nll=3.4405  ppl=31.20  grad=0.895  lr=4.76e-06  6862 tok/s  12.63s/it  [19:37:02<11:26:51]  mem=34.07GB
full_cpt step  9600/12852 ( 74.7%)  nll=3.2519  ppl=25.84  grad=0.809  lr=4.75e-06  6863 tok/s  12.63s/it  [19:38:58<11:24:39]  mem=34.07GB
  am   nll=3.9692  ppl=52.94  bpb=1.3958
  ta   nll=3.9431  ppl=51.58  bpb=0.5727
  kk   nll=2.7493  ppl=15.63  bpb=0.9258
  tl   nll=2.6594  ppl=14.29  bpb=1.2618
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 9600  01:27 (86.7s)
full_cpt step  9610/12852 ( 74.8%)  nll=3.2055  ppl=24.67  grad=0.887  lr=4.73e-06  6856 tok/s  12.65s/it  [19:42:21<11:23:16]  mem=34.07GB
full_cpt step  9620/12852 ( 74.9%)  nll=3.3598  ppl=28.78  grad=0.832  lr=4.72e-06  6857 tok/s  12.64s/it  [19:44:17<11:21:04]  mem=34.07GB
full_cpt step  9630/12852 ( 74.9%)  nll=3.2981  ppl=27.06  grad=0.891  lr=4.70e-06  6858 tok/s  12.64s/it  [19:46:13<11:18:52]  mem=34.07GB
full_cpt step  9640/12852 ( 75.0%)  nll=3.3724  ppl=29.15  grad=0.828  lr=4.68e-06  6859 tok/s  12.64s/it  [19:48:09<11:16:39]  mem=34.07GB
full_cpt step  9650/12852 ( 75.1%)  nll=3.3147  ppl=27.51  grad=0.875  lr=4.67e-06  6860 tok/s  12.64s/it  [19:50:05<11:14:27]  mem=34.07GB
full_cpt step  9660/12852 ( 75.2%)  nll=3.2752  ppl=26.45  grad=0.859  lr=4.65e-06  6861 tok/s  12.64s/it  [19:52:01<11:12:15]  mem=34.07GB
full_cpt step  9670/12852 ( 75.2%)  nll=3.3240  ppl=27.77  grad=0.875  lr=4.64e-06  6862 tok/s  12.63s/it  [19:53:57<11:10:02]  mem=34.07GB
full_cpt step  9680/12852 ( 75.3%)  nll=3.2927  ppl=26.91  grad=0.867  lr=4.62e-06  6863 tok/s  12.63s/it  [19:55:53<11:07:50]  mem=34.07GB
full_cpt step  9690/12852 ( 75.4%)  nll=3.3300  ppl=27.94  grad=0.883  lr=4.61e-06  6864 tok/s  12.63s/it  [19:57:48<11:05:38]  mem=34.07GB
full_cpt step  9700/12852 ( 75.5%)  nll=3.2749  ppl=26.44  grad=0.871  lr=4.59e-06  6865 tok/s  12.63s/it  [19:59:44<11:03:26]  mem=34.07GB
  am   nll=3.9692  ppl=52.94  bpb=1.3958
  ta   nll=3.9434  ppl=51.59  bpb=0.5727
  kk   nll=2.7492  ppl=15.63  bpb=0.9258
  tl   nll=2.6592  ppl=14.28  bpb=1.2617
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 9700  01:26 (86.2s)
full_cpt step  9710/12852 ( 75.6%)  nll=3.2895  ppl=26.83  grad=0.809  lr=4.58e-06  6858 tok/s  12.64s/it  [20:03:06<11:02:02]  mem=34.07GB
full_cpt step  9720/12852 ( 75.6%)  nll=3.2979  ppl=27.06  grad=0.875  lr=4.56e-06  6859 tok/s  12.64s/it  [20:05:02<10:59:49]  mem=34.07GB
full_cpt step  9730/12852 ( 75.7%)  nll=3.3490  ppl=28.47  grad=0.887  lr=4.54e-06  6860 tok/s  12.64s/it  [20:06:58<10:57:37]  mem=34.07GB
full_cpt step  9740/12852 ( 75.8%)  nll=3.3658  ppl=28.96  grad=0.914  lr=4.53e-06  6861 tok/s  12.64s/it  [20:08:54<10:55:25]  mem=34.07GB
full_cpt step  9750/12852 ( 75.9%)  nll=3.4192  ppl=30.55  grad=0.859  lr=4.51e-06  6862 tok/s  12.63s/it  [20:10:50<10:53:13]  mem=34.07GB
full_cpt step  9760/12852 ( 75.9%)  nll=3.3840  ppl=29.49  grad=0.953  lr=4.50e-06  6863 tok/s  12.63s/it  [20:12:46<10:51:01]  mem=34.07GB
full_cpt step  9770/12852 ( 76.0%)  nll=3.2199  ppl=25.03  grad=0.871  lr=4.48e-06  6864 tok/s  12.63s/it  [20:14:42<10:48:49]  mem=34.07GB
full_cpt step  9780/12852 ( 76.1%)  nll=3.3488  ppl=28.47  grad=0.879  lr=4.47e-06  6865 tok/s  12.63s/it  [20:16:38<10:46:38]  mem=34.07GB
full_cpt step  9790/12852 ( 76.2%)  nll=3.2712  ppl=26.34  grad=0.898  lr=4.45e-06  6865 tok/s  12.63s/it  [20:18:34<10:44:26]  mem=34.07GB
full_cpt step  9800/12852 ( 76.3%)  nll=3.2652  ppl=26.19  grad=0.902  lr=4.44e-06  6866 tok/s  12.63s/it  [20:20:30<10:42:14]  mem=34.07GB
  am   nll=3.9691  ppl=52.94  bpb=1.3957
  ta   nll=3.9430  ppl=51.58  bpb=0.5727
  kk   nll=2.7494  ppl=15.63  bpb=0.9258
  tl   nll=2.6594  ppl=14.29  bpb=1.2618
  en   nll=2.8178  ppl=16.74  bpb=0.8773
eval full_cpt step 9800  01:27 (86.7s)
full_cpt step  9810/12852 ( 76.3%)  nll=3.2995  ppl=27.10  grad=0.914  lr=4.42e-06  6859 tok/s  12.64s/it  [20:23:52<10:40:48]  mem=34.07GB
full_cpt step  9820/12852 ( 76.4%)  nll=3.3823  ppl=29.44  grad=0.883  lr=4.41e-06  6860 tok/s  12.64s/it  [20:25:48<10:38:36]  mem=34.07GB
full_cpt step  9830/12852 ( 76.5%)  nll=3.3130  ppl=27.47  grad=0.859  lr=4.39e-06  6861 tok/s  12.64s/it  [20:27:44<10:36:24]  mem=34.07GB
full_cpt step  9840/12852 ( 76.6%)  nll=3.2611  ppl=26.08  grad=0.883  lr=4.38e-06  6862 tok/s  12.63s/it  [20:29:40<10:34:12]  mem=34.07GB
full_cpt step  9850/12852 ( 76.6%)  nll=3.3671  ppl=28.99  grad=0.855  lr=4.36e-06  6863 tok/s  12.63s/it  [20:31:36<10:32:01]  mem=34.07GB
full_cpt step  9860/12852 ( 76.7%)  nll=3.3158  ppl=27.54  grad=0.840  lr=4.35e-06  6864 tok/s  12.63s/it  [20:33:32<10:29:49]  mem=34.07GB
full_cpt step  9870/12852 ( 76.8%)  nll=3.3485  ppl=28.46  grad=0.879  lr=4.33e-06  6865 tok/s  12.63s/it  [20:35:28<10:27:38]  mem=34.07GB
full_cpt step  9880/12852 ( 76.9%)  nll=3.3031  ppl=27.20  grad=0.824  lr=4.32e-06  6866 tok/s  12.63s/it  [20:37:24<10:25:26]  mem=34.07GB
full_cpt step  9890/12852 ( 77.0%)  nll=3.4453  ppl=31.35  grad=0.863  lr=4.30e-06  6867 tok/s  12.62s/it  [20:39:20<10:23:15]  mem=34.07GB
full_cpt step  9900/12852 ( 77.0%)  nll=3.3408  ppl=28.24  grad=0.871  lr=4.29e-06  6868 tok/s  12.62s/it  [20:41:16<10:21:03]  mem=34.07GB
  am   nll=3.9693  ppl=52.95  bpb=1.3958
  ta   nll=3.9431  ppl=51.58  bpb=0.5727
  kk   nll=2.7493  ppl=15.63  bpb=0.9258
  tl   nll=2.6593  ppl=14.29  bpb=1.2618
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 9900  01:27 (86.6s)
full_cpt step  9910/12852 ( 77.1%)  nll=3.1985  ppl=24.50  grad=0.855  lr=4.27e-06  6861 tok/s  12.64s/it  [20:44:38<10:19:35]  mem=34.07GB
full_cpt step  9920/12852 ( 77.2%)  nll=3.2777  ppl=26.52  grad=0.898  lr=4.26e-06  6862 tok/s  12.63s/it  [20:46:34<10:17:23]  mem=34.07GB
full_cpt step  9930/12852 ( 77.3%)  nll=3.3016  ppl=27.16  grad=0.828  lr=4.24e-06  6863 tok/s  12.63s/it  [20:48:30<10:15:12]  mem=34.07GB
full_cpt step  9940/12852 ( 77.3%)  nll=3.3257  ppl=27.82  grad=0.941  lr=4.23e-06  6863 tok/s  12.63s/it  [20:50:26<10:13:00]  mem=34.07GB
full_cpt step  9950/12852 ( 77.4%)  nll=3.3342  ppl=28.06  grad=0.945  lr=4.21e-06  6864 tok/s  12.63s/it  [20:52:22<10:10:49]  mem=34.07GB
full_cpt step  9960/12852 ( 77.5%)  nll=3.1883  ppl=24.25  grad=0.875  lr=4.20e-06  6865 tok/s  12.63s/it  [20:54:18<10:08:38]  mem=34.07GB
full_cpt step  9970/12852 ( 77.6%)  nll=3.3295  ppl=27.92  grad=0.902  lr=4.18e-06  6866 tok/s  12.63s/it  [20:56:14<10:06:27]  mem=34.07GB
full_cpt step  9980/12852 ( 77.7%)  nll=3.2195  ppl=25.02  grad=0.840  lr=4.17e-06  6867 tok/s  12.62s/it  [20:58:10<10:04:15]  mem=34.07GB
full_cpt step  9990/12852 ( 77.7%)  nll=3.2803  ppl=26.58  grad=0.871  lr=4.16e-06  6868 tok/s  12.62s/it  [21:00:06<10:02:04]  mem=34.07GB
full_cpt step 10000/12852 ( 77.8%)  nll=3.3804  ppl=29.38  grad=0.848  lr=4.14e-06  6869 tok/s  12.62s/it  [21:02:02<9:59:53]  mem=34.07GB
  am   nll=3.9692  ppl=52.94  bpb=1.3958
  ta   nll=3.9430  ppl=51.58  bpb=0.5727
  kk   nll=2.7492  ppl=15.63  bpb=0.9258
  tl   nll=2.6595  ppl=14.29  bpb=1.2619
  en   nll=2.8175  ppl=16.73  bpb=0.8772
eval full_cpt step 10000  01:27 (86.5s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.26s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.26s/it]
removing old checkpoint checkpoint-9000
kept last 2 resume checkpoints: checkpoint-9500 checkpoint-10000
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-10000 at step 10000 next_row=1280000

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.53s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.53s/it]
wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-10000 (weights+expanded tokenizer, no optimizer)
full_cpt step 10010/12852 ( 77.9%)  nll=3.2738  ppl=26.41  grad=0.910  lr=4.13e-06  6855 tok/s  12.65s/it  [21:06:44<9:59:01]  mem=34.07GB
full_cpt step 10020/12852 ( 78.0%)  nll=3.2301  ppl=25.28  grad=0.961  lr=4.11e-06  6856 tok/s  12.64s/it  [21:08:40<9:56:49]  mem=34.07GB
full_cpt step 10030/12852 ( 78.0%)  nll=3.2540  ppl=25.89  grad=0.918  lr=4.10e-06  6857 tok/s  12.64s/it  [21:10:36<9:54:38]  mem=34.07GB
full_cpt step 10040/12852 ( 78.1%)  nll=3.3511  ppl=28.54  grad=0.812  lr=4.08e-06  6858 tok/s  12.64s/it  [21:12:32<9:52:27]  mem=34.07GB
full_cpt step 10050/12852 ( 78.2%)  nll=3.2198  ppl=25.02  grad=0.867  lr=4.07e-06  6858 tok/s  12.64s/it  [21:14:28<9:50:15]  mem=34.07GB
full_cpt step 10060/12852 ( 78.3%)  nll=3.2543  ppl=25.90  grad=0.891  lr=4.06e-06  6859 tok/s  12.64s/it  [21:16:24<9:48:04]  mem=34.07GB
full_cpt step 10070/12852 ( 78.4%)  nll=3.2844  ppl=26.69  grad=0.828  lr=4.04e-06  6860 tok/s  12.64s/it  [21:18:20<9:45:53]  mem=34.07GB
full_cpt step 10080/12852 ( 78.4%)  nll=3.2946  ppl=26.97  grad=0.840  lr=4.03e-06  6861 tok/s  12.63s/it  [21:20:16<9:43:42]  mem=34.07GB
full_cpt step 10090/12852 ( 78.5%)  nll=3.2286  ppl=25.25  grad=0.938  lr=4.01e-06  6861 tok/s  12.63s/it  [21:22:11<9:41:31]  mem=34.07GB
full_cpt step 10100/12852 ( 78.6%)  nll=3.3637  ppl=28.89  grad=0.891  lr=4.00e-06  6862 tok/s  12.63s/it  [21:24:07<9:39:20]  mem=34.07GB
  am   nll=3.9690  ppl=52.93  bpb=1.3957
  ta   nll=3.9429  ppl=51.57  bpb=0.5727
  kk   nll=2.7493  ppl=15.63  bpb=0.9258
  tl   nll=2.6592  ppl=14.28  bpb=1.2617
  en   nll=2.8175  ppl=16.73  bpb=0.8772
eval full_cpt step 10100  01:26 (86.2s)
full_cpt step 10110/12852 ( 78.7%)  nll=3.3598  ppl=28.78  grad=0.855  lr=3.98e-06  6855 tok/s  12.64s/it  [21:27:30<9:37:47]  mem=34.07GB
full_cpt step 10120/12852 ( 78.7%)  nll=3.2096  ppl=24.77  grad=0.855  lr=3.97e-06  6856 tok/s  12.64s/it  [21:29:25<9:35:36]  mem=34.07GB
full_cpt step 10130/12852 ( 78.8%)  nll=3.3657  ppl=28.95  grad=0.867  lr=3.96e-06  6857 tok/s  12.64s/it  [21:31:21<9:33:25]  mem=34.07GB
full_cpt step 10140/12852 ( 78.9%)  nll=3.2137  ppl=24.87  grad=0.898  lr=3.94e-06  6858 tok/s  12.64s/it  [21:33:17<9:31:14]  mem=34.07GB
full_cpt step 10150/12852 ( 79.0%)  nll=3.3429  ppl=28.30  grad=0.863  lr=3.93e-06  6859 tok/s  12.64s/it  [21:35:13<9:29:03]  mem=34.07GB
full_cpt step 10160/12852 ( 79.1%)  nll=3.2481  ppl=25.74  grad=0.879  lr=3.92e-06  6860 tok/s  12.63s/it  [21:37:09<9:26:52]  mem=34.07GB
full_cpt step 10170/12852 ( 79.1%)  nll=3.3288  ppl=27.90  grad=0.895  lr=3.90e-06  6861 tok/s  12.63s/it  [21:39:05<9:24:42]  mem=34.07GB
full_cpt step 10180/12852 ( 79.2%)  nll=3.4284  ppl=30.83  grad=0.879  lr=3.89e-06  6862 tok/s  12.63s/it  [21:41:01<9:22:31]  mem=34.07GB
full_cpt step 10190/12852 ( 79.3%)  nll=3.2280  ppl=25.23  grad=0.891  lr=3.87e-06  6863 tok/s  12.63s/it  [21:42:57<9:20:20]  mem=34.07GB
full_cpt step 10200/12852 ( 79.4%)  nll=3.2800  ppl=26.58  grad=0.848  lr=3.86e-06  6863 tok/s  12.63s/it  [21:44:53<9:18:09]  mem=34.07GB
  am   nll=3.9692  ppl=52.94  bpb=1.3958
  ta   nll=3.9428  ppl=51.56  bpb=0.5727
  kk   nll=2.7492  ppl=15.63  bpb=0.9258
  tl   nll=2.6593  ppl=14.29  bpb=1.2618
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 10200  01:27 (86.9s)
full_cpt step 10210/12852 ( 79.4%)  nll=3.2334  ppl=25.37  grad=0.906  lr=3.85e-06  6857 tok/s  12.64s/it  [21:48:16<9:16:35]  mem=34.07GB
full_cpt step 10220/12852 ( 79.5%)  nll=3.2003  ppl=24.54  grad=0.875  lr=3.83e-06  6857 tok/s  12.64s/it  [21:50:11<9:14:25]  mem=34.07GB
full_cpt step 10230/12852 ( 79.6%)  nll=3.3162  ppl=27.56  grad=0.895  lr=3.82e-06  6858 tok/s  12.64s/it  [21:52:07<9:12:14]  mem=34.07GB
full_cpt step 10240/12852 ( 79.7%)  nll=3.3679  ppl=29.02  grad=0.840  lr=3.81e-06  6859 tok/s  12.64s/it  [21:54:03<9:10:03]  mem=34.07GB
full_cpt step 10250/12852 ( 79.8%)  nll=3.3755  ppl=29.24  grad=0.883  lr=3.79e-06  6860 tok/s  12.63s/it  [21:55:59<9:07:52]  mem=34.07GB
full_cpt step 10260/12852 ( 79.8%)  nll=3.4084  ppl=30.22  grad=0.879  lr=3.78e-06  6861 tok/s  12.63s/it  [21:57:55<9:05:42]  mem=34.07GB
full_cpt step 10270/12852 ( 79.9%)  nll=3.2672  ppl=26.24  grad=0.875  lr=3.77e-06  6862 tok/s  12.63s/it  [21:59:51<9:03:31]  mem=34.07GB
full_cpt step 10280/12852 ( 80.0%)  nll=3.1289  ppl=22.85  grad=0.879  lr=3.75e-06  6863 tok/s  12.63s/it  [22:01:47<9:01:21]  mem=34.07GB
full_cpt step 10290/12852 ( 80.1%)  nll=3.2088  ppl=24.75  grad=0.863  lr=3.74e-06  6864 tok/s  12.63s/it  [22:03:43<8:59:10]  mem=34.07GB
full_cpt step 10300/12852 ( 80.1%)  nll=3.3692  ppl=29.06  grad=0.867  lr=3.73e-06  6865 tok/s  12.63s/it  [22:05:39<8:57:00]  mem=34.07GB
  am   nll=3.9691  ppl=52.93  bpb=1.3957
  ta   nll=3.9427  ppl=51.56  bpb=0.5726
  kk   nll=2.7497  ppl=15.64  bpb=0.9259
  tl   nll=2.6592  ppl=14.28  bpb=1.2617
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 10300  01:27 (87.1s)
full_cpt step 10310/12852 ( 80.2%)  nll=3.3257  ppl=27.82  grad=0.879  lr=3.72e-06  6858 tok/s  12.64s/it  [22:09:02<8:55:24]  mem=34.07GB
full_cpt step 10320/12852 ( 80.3%)  nll=3.2483  ppl=25.75  grad=0.840  lr=3.70e-06  6859 tok/s  12.64s/it  [22:10:58<8:53:14]  mem=34.07GB
full_cpt step 10330/12852 ( 80.4%)  nll=3.2974  ppl=27.04  grad=0.867  lr=3.69e-06  6860 tok/s  12.63s/it  [22:12:54<8:51:03]  mem=34.07GB
full_cpt step 10340/12852 ( 80.5%)  nll=3.2313  ppl=25.31  grad=0.824  lr=3.68e-06  6861 tok/s  12.63s/it  [22:14:50<8:48:53]  mem=34.07GB
full_cpt step 10350/12852 ( 80.5%)  nll=3.4022  ppl=30.03  grad=0.863  lr=3.66e-06  6861 tok/s  12.63s/it  [22:16:46<8:46:42]  mem=34.07GB
full_cpt step 10360/12852 ( 80.6%)  nll=3.3053  ppl=27.26  grad=0.859  lr=3.65e-06  6862 tok/s  12.63s/it  [22:18:42<8:44:32]  mem=34.07GB
full_cpt step 10370/12852 ( 80.7%)  nll=3.3111  ppl=27.41  grad=0.883  lr=3.64e-06  6863 tok/s  12.63s/it  [22:20:38<8:42:22]  mem=34.07GB
full_cpt step 10380/12852 ( 80.8%)  nll=3.2821  ppl=26.63  grad=0.855  lr=3.62e-06  6864 tok/s  12.63s/it  [22:22:34<8:40:12]  mem=34.07GB
full_cpt step 10390/12852 ( 80.8%)  nll=3.3771  ppl=29.29  grad=0.898  lr=3.61e-06  6865 tok/s  12.62s/it  [22:24:30<8:38:01]  mem=34.07GB
full_cpt step 10400/12852 ( 80.9%)  nll=3.2285  ppl=25.24  grad=0.832  lr=3.60e-06  6866 tok/s  12.62s/it  [22:26:26<8:35:51]  mem=34.07GB
  am   nll=3.9692  ppl=52.94  bpb=1.3958
  ta   nll=3.9424  ppl=51.54  bpb=0.5726
  kk   nll=2.7492  ppl=15.63  bpb=0.9258
  tl   nll=2.6594  ppl=14.29  bpb=1.2619
  en   nll=2.8178  ppl=16.74  bpb=0.8774
eval full_cpt step 10400  01:27 (87.0s)
full_cpt step 10410/12852 ( 81.0%)  nll=3.2760  ppl=26.47  grad=0.816  lr=3.59e-06  6860 tok/s  12.63s/it  [22:29:49<8:34:14]  mem=34.07GB
full_cpt step 10420/12852 ( 81.1%)  nll=3.4020  ppl=30.02  grad=0.840  lr=3.57e-06  6860 tok/s  12.63s/it  [22:31:45<8:32:04]  mem=34.07GB
full_cpt step 10430/12852 ( 81.2%)  nll=3.2908  ppl=26.86  grad=0.930  lr=3.56e-06  6861 tok/s  12.63s/it  [22:33:41<8:29:54]  mem=34.07GB
full_cpt step 10440/12852 ( 81.2%)  nll=3.2738  ppl=26.41  grad=0.875  lr=3.55e-06  6862 tok/s  12.63s/it  [22:35:37<8:27:43]  mem=34.07GB
full_cpt step 10450/12852 ( 81.3%)  nll=3.4129  ppl=30.35  grad=0.875  lr=3.54e-06  6863 tok/s  12.63s/it  [22:37:33<8:25:33]  mem=34.07GB
full_cpt step 10460/12852 ( 81.4%)  nll=3.2423  ppl=25.59  grad=0.848  lr=3.52e-06  6864 tok/s  12.63s/it  [22:39:29<8:23:23]  mem=34.07GB
full_cpt step 10470/12852 ( 81.5%)  nll=3.2920  ppl=26.90  grad=0.922  lr=3.51e-06  6865 tok/s  12.63s/it  [22:41:25<8:21:13]  mem=34.07GB
full_cpt step 10480/12852 ( 81.5%)  nll=3.3300  ppl=27.94  grad=0.898  lr=3.50e-06  6866 tok/s  12.62s/it  [22:43:21<8:19:03]  mem=34.07GB
full_cpt step 10490/12852 ( 81.6%)  nll=3.2706  ppl=26.33  grad=0.898  lr=3.49e-06  6867 tok/s  12.62s/it  [22:45:16<8:16:53]  mem=34.07GB
full_cpt step 10500/12852 ( 81.7%)  nll=3.2467  ppl=25.70  grad=0.855  lr=3.48e-06  6867 tok/s  12.62s/it  [22:47:12<8:14:43]  mem=34.07GB
  am   nll=3.9693  ppl=52.95  bpb=1.3958
  ta   nll=3.9426  ppl=51.55  bpb=0.5726
  kk   nll=2.7495  ppl=15.64  bpb=0.9259
  tl   nll=2.6593  ppl=14.29  bpb=1.2618
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 10500  01:27 (86.9s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.29s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.30s/it]
removing old checkpoint checkpoint-9500
kept last 2 resume checkpoints: checkpoint-10000 checkpoint-10500
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-10500 at step 10500 next_row=1344000
full_cpt step 10510/12852 ( 81.8%)  nll=3.2198  ppl=25.02  grad=0.902  lr=3.46e-06  6858 tok/s  12.64s/it  [22:51:12<8:13:18]  mem=34.07GB
full_cpt step 10520/12852 ( 81.9%)  nll=3.3195  ppl=27.65  grad=0.914  lr=3.45e-06  6859 tok/s  12.64s/it  [22:53:08<8:11:08]  mem=34.07GB
full_cpt step 10530/12852 ( 81.9%)  nll=3.3433  ppl=28.31  grad=0.895  lr=3.44e-06  6860 tok/s  12.63s/it  [22:55:04<8:08:58]  mem=34.07GB
full_cpt step 10540/12852 ( 82.0%)  nll=3.3988  ppl=29.93  grad=0.863  lr=3.43e-06  6861 tok/s  12.63s/it  [22:57:00<8:06:48]  mem=34.07GB
full_cpt step 10550/12852 ( 82.1%)  nll=3.3007  ppl=27.13  grad=0.863  lr=3.41e-06  6862 tok/s  12.63s/it  [22:58:56<8:04:38]  mem=34.07GB
full_cpt step 10560/12852 ( 82.2%)  nll=3.1874  ppl=24.23  grad=0.855  lr=3.40e-06  6862 tok/s  12.63s/it  [23:00:52<8:02:28]  mem=34.07GB
full_cpt step 10570/12852 ( 82.2%)  nll=3.4589  ppl=31.78  grad=0.871  lr=3.39e-06  6863 tok/s  12.63s/it  [23:02:48<8:00:18]  mem=34.07GB
full_cpt step 10580/12852 ( 82.3%)  nll=3.3617  ppl=28.84  grad=0.828  lr=3.38e-06  6864 tok/s  12.63s/it  [23:04:44<7:58:08]  mem=34.07GB
full_cpt step 10590/12852 ( 82.4%)  nll=3.3878  ppl=29.60  grad=0.938  lr=3.37e-06  6865 tok/s  12.63s/it  [23:06:40<7:55:58]  mem=34.07GB
full_cpt step 10600/12852 ( 82.5%)  nll=3.1713  ppl=23.84  grad=0.844  lr=3.36e-06  6866 tok/s  12.62s/it  [23:08:36<7:53:48]  mem=34.07GB
  am   nll=3.9690  ppl=52.93  bpb=1.3957
  ta   nll=3.9428  ppl=51.56  bpb=0.5727
  kk   nll=2.7491  ppl=15.63  bpb=0.9258
  tl   nll=2.6593  ppl=14.29  bpb=1.2618
  en   nll=2.8179  ppl=16.74  bpb=0.8774
eval full_cpt step 10600  01:26 (86.3s)
full_cpt step 10610/12852 ( 82.6%)  nll=3.2706  ppl=26.33  grad=0.934  lr=3.34e-06  6860 tok/s  12.64s/it  [23:11:58<7:52:08]  mem=34.07GB
full_cpt step 10620/12852 ( 82.6%)  nll=3.3724  ppl=29.15  grad=0.871  lr=3.33e-06  6861 tok/s  12.63s/it  [23:13:54<7:49:58]  mem=34.07GB
full_cpt step 10630/12852 ( 82.7%)  nll=3.3626  ppl=28.86  grad=0.875  lr=3.32e-06  6862 tok/s  12.63s/it  [23:15:50<7:47:48]  mem=34.07GB
full_cpt step 10640/12852 ( 82.8%)  nll=3.3617  ppl=28.84  grad=0.891  lr=3.31e-06  6863 tok/s  12.63s/it  [23:17:46<7:45:39]  mem=34.07GB
full_cpt step 10650/12852 ( 82.9%)  nll=3.3686  ppl=29.04  grad=0.836  lr=3.30e-06  6864 tok/s  12.63s/it  [23:19:42<7:43:29]  mem=34.07GB
full_cpt step 10660/12852 ( 82.9%)  nll=3.3623  ppl=28.86  grad=0.820  lr=3.29e-06  6865 tok/s  12.63s/it  [23:21:38<7:41:19]  mem=34.07GB
full_cpt step 10670/12852 ( 83.0%)  nll=3.3187  ppl=27.63  grad=0.945  lr=3.27e-06  6865 tok/s  12.63s/it  [23:23:34<7:39:10]  mem=34.07GB
full_cpt step 10680/12852 ( 83.1%)  nll=3.3234  ppl=27.75  grad=0.879  lr=3.26e-06  6866 tok/s  12.62s/it  [23:25:30<7:37:00]  mem=34.07GB
full_cpt step 10690/12852 ( 83.2%)  nll=3.4001  ppl=29.97  grad=0.883  lr=3.25e-06  6867 tok/s  12.62s/it  [23:27:26<7:34:50]  mem=34.07GB
full_cpt step 10700/12852 ( 83.3%)  nll=3.4171  ppl=30.48  grad=0.828  lr=3.24e-06  6868 tok/s  12.62s/it  [23:29:22<7:32:41]  mem=34.07GB
  am   nll=3.9692  ppl=52.94  bpb=1.3958
  ta   nll=3.9432  ppl=51.58  bpb=0.5727
  kk   nll=2.7491  ppl=15.63  bpb=0.9258
  tl   nll=2.6592  ppl=14.28  bpb=1.2617
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 10700  01:26 (86.3s)
full_cpt step 10710/12852 ( 83.3%)  nll=3.2121  ppl=24.83  grad=0.812  lr=3.23e-06  6862 tok/s  12.63s/it  [23:32:44<7:30:59]  mem=34.07GB
full_cpt step 10720/12852 ( 83.4%)  nll=3.3478  ppl=28.44  grad=0.855  lr=3.22e-06  6862 tok/s  12.63s/it  [23:34:40<7:28:49]  mem=34.07GB
full_cpt step 10730/12852 ( 83.5%)  nll=3.3152  ppl=27.53  grad=0.902  lr=3.21e-06  6863 tok/s  12.63s/it  [23:36:36<7:26:40]  mem=34.07GB
full_cpt step 10740/12852 ( 83.6%)  nll=3.3570  ppl=28.70  grad=0.895  lr=3.20e-06  6864 tok/s  12.63s/it  [23:38:32<7:24:30]  mem=34.07GB
full_cpt step 10750/12852 ( 83.6%)  nll=3.3583  ppl=28.74  grad=0.922  lr=3.19e-06  6865 tok/s  12.63s/it  [23:40:29<7:22:21]  mem=34.07GB
full_cpt step 10760/12852 ( 83.7%)  nll=3.1867  ppl=24.21  grad=0.867  lr=3.17e-06  6865 tok/s  12.62s/it  [23:42:25<7:20:11]  mem=34.07GB
full_cpt step 10770/12852 ( 83.8%)  nll=3.3477  ppl=28.44  grad=0.875  lr=3.16e-06  6866 tok/s  12.62s/it  [23:44:21<7:18:02]  mem=34.07GB
full_cpt step 10780/12852 ( 83.9%)  nll=3.2901  ppl=26.85  grad=0.852  lr=3.15e-06  6867 tok/s  12.62s/it  [23:46:16<7:15:53]  mem=34.07GB
full_cpt step 10790/12852 ( 84.0%)  nll=3.3143  ppl=27.50  grad=0.848  lr=3.14e-06  6867 tok/s  12.62s/it  [23:48:12<7:13:43]  mem=34.07GB
full_cpt step 10800/12852 ( 84.0%)  nll=3.3161  ppl=27.55  grad=0.879  lr=3.13e-06  6868 tok/s  12.62s/it  [23:50:08<7:11:34]  mem=34.07GB
  am   nll=3.9691  ppl=52.94  bpb=1.3957
  ta   nll=3.9430  ppl=51.57  bpb=0.5727
  kk   nll=2.7495  ppl=15.63  bpb=0.9259
  tl   nll=2.6593  ppl=14.29  bpb=1.2618
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 10800  01:27 (86.6s)
full_cpt step 10810/12852 ( 84.1%)  nll=3.3150  ppl=27.52  grad=0.820  lr=3.12e-06  6862 tok/s  12.63s/it  [23:53:31<7:09:51]  mem=34.07GB
full_cpt step 10820/12852 ( 84.2%)  nll=3.2914  ppl=26.88  grad=0.891  lr=3.11e-06  6863 tok/s  12.63s/it  [23:55:27<7:07:41]  mem=34.07GB
full_cpt step 10830/12852 ( 84.3%)  nll=3.3773  ppl=29.29  grad=0.879  lr=3.10e-06  6864 tok/s  12.63s/it  [23:57:23<7:05:32]  mem=34.07GB
full_cpt step 10840/12852 ( 84.3%)  nll=3.2342  ppl=25.39  grad=0.844  lr=3.09e-06  6865 tok/s  12.63s/it  [23:59:19<7:03:23]  mem=34.07GB
full_cpt step 10850/12852 ( 84.4%)  nll=3.2465  ppl=25.70  grad=0.910  lr=3.08e-06  6866 tok/s  12.62s/it  [24:01:15<7:01:13]  mem=34.07GB
full_cpt step 10860/12852 ( 84.5%)  nll=3.3472  ppl=28.42  grad=0.875  lr=3.07e-06  6866 tok/s  12.62s/it  [24:03:11<6:59:04]  mem=34.07GB
full_cpt step 10870/12852 ( 84.6%)  nll=3.2407  ppl=25.55  grad=0.832  lr=3.06e-06  6867 tok/s  12.62s/it  [24:05:07<6:56:55]  mem=34.07GB
full_cpt step 10880/12852 ( 84.7%)  nll=3.2950  ppl=26.98  grad=0.891  lr=3.05e-06  6868 tok/s  12.62s/it  [24:07:03<6:54:46]  mem=34.07GB
full_cpt step 10890/12852 ( 84.7%)  nll=3.2224  ppl=25.09  grad=0.922  lr=3.04e-06  6869 tok/s  12.62s/it  [24:08:59<6:52:37]  mem=34.07GB
full_cpt step 10900/12852 ( 84.8%)  nll=3.2434  ppl=25.62  grad=0.836  lr=3.03e-06  6870 tok/s  12.62s/it  [24:10:55<6:50:28]  mem=34.07GB
  am   nll=3.9691  ppl=52.94  bpb=1.3957
  ta   nll=3.9434  ppl=51.60  bpb=0.5728
  kk   nll=2.7494  ppl=15.63  bpb=0.9258
  tl   nll=2.6592  ppl=14.29  bpb=1.2618
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 10900  01:27 (86.9s)
full_cpt step 10910/12852 ( 84.9%)  nll=3.3836  ppl=29.48  grad=0.832  lr=3.01e-06  6864 tok/s  12.63s/it  [24:14:17<6:48:43]  mem=34.07GB
full_cpt step 10920/12852 ( 85.0%)  nll=3.2734  ppl=26.40  grad=0.871  lr=3.00e-06  6865 tok/s  12.63s/it  [24:16:13<6:46:34]  mem=34.07GB
full_cpt step 10930/12852 ( 85.0%)  nll=3.3069  ppl=27.30  grad=0.852  lr=2.99e-06  6865 tok/s  12.62s/it  [24:18:09<6:44:25]  mem=34.07GB
full_cpt step 10940/12852 ( 85.1%)  nll=3.4619  ppl=31.88  grad=0.891  lr=2.98e-06  6866 tok/s  12.62s/it  [24:20:05<6:42:16]  mem=34.07GB
full_cpt step 10950/12852 ( 85.2%)  nll=3.3695  ppl=29.06  grad=0.887  lr=2.97e-06  6867 tok/s  12.62s/it  [24:22:01<6:40:07]  mem=34.07GB
full_cpt step 10960/12852 ( 85.3%)  nll=3.2327  ppl=25.35  grad=0.828  lr=2.96e-06  6868 tok/s  12.62s/it  [24:23:57<6:37:58]  mem=34.07GB
full_cpt step 10970/12852 ( 85.4%)  nll=3.2898  ppl=26.84  grad=0.855  lr=2.95e-06  6869 tok/s  12.62s/it  [24:25:53<6:35:49]  mem=34.07GB
full_cpt step 10980/12852 ( 85.4%)  nll=3.2185  ppl=24.99  grad=0.871  lr=2.94e-06  6869 tok/s  12.62s/it  [24:27:49<6:33:40]  mem=34.07GB
full_cpt step 10990/12852 ( 85.5%)  nll=3.1923  ppl=24.34  grad=0.918  lr=2.93e-06  6870 tok/s  12.62s/it  [24:29:45<6:31:31]  mem=34.07GB
full_cpt step 11000/12852 ( 85.6%)  nll=3.3846  ppl=29.51  grad=0.852  lr=2.92e-06  6871 tok/s  12.61s/it  [24:31:41<6:29:22]  mem=34.07GB
  am   nll=3.9688  ppl=52.92  bpb=1.3956
  ta   nll=3.9428  ppl=51.56  bpb=0.5727
  kk   nll=2.7493  ppl=15.63  bpb=0.9258
  tl   nll=2.6596  ppl=14.29  bpb=1.2619
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 11000  01:26 (86.5s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.20s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.20s/it]
removing old checkpoint checkpoint-10000
kept last 2 resume checkpoints: checkpoint-10500 checkpoint-11000
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-11000 at step 11000 next_row=1408000

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.44s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.44s/it]
wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-11000 (weights+expanded tokenizer, no optimizer)
full_cpt step 11010/12852 ( 85.7%)  nll=3.3893  ppl=29.64  grad=0.910  lr=2.91e-06  6861 tok/s  12.63s/it  [24:35:57<6:27:50]  mem=34.07GB
full_cpt step 11020/12852 ( 85.7%)  nll=3.3378  ppl=28.16  grad=0.855  lr=2.91e-06  6862 tok/s  12.63s/it  [24:37:53<6:25:41]  mem=34.07GB
full_cpt step 11030/12852 ( 85.8%)  nll=3.3418  ppl=28.27  grad=0.875  lr=2.90e-06  6863 tok/s  12.63s/it  [24:39:49<6:23:32]  mem=34.07GB
full_cpt step 11040/12852 ( 85.9%)  nll=3.2500  ppl=25.79  grad=0.895  lr=2.89e-06  6863 tok/s  12.63s/it  [24:41:45<6:21:23]  mem=34.07GB
full_cpt step 11050/12852 ( 86.0%)  nll=3.3929  ppl=29.75  grad=0.859  lr=2.88e-06  6864 tok/s  12.63s/it  [24:43:41<6:19:14]  mem=34.07GB
full_cpt step 11060/12852 ( 86.1%)  nll=3.2196  ppl=25.02  grad=0.871  lr=2.87e-06  6865 tok/s  12.63s/it  [24:45:37<6:17:05]  mem=34.07GB
full_cpt step 11070/12852 ( 86.1%)  nll=3.3425  ppl=28.29  grad=0.914  lr=2.86e-06  6865 tok/s  12.62s/it  [24:47:33<6:14:56]  mem=34.07GB
full_cpt step 11080/12852 ( 86.2%)  nll=3.3355  ppl=28.09  grad=0.855  lr=2.85e-06  6866 tok/s  12.62s/it  [24:49:29<6:12:47]  mem=34.07GB
full_cpt step 11090/12852 ( 86.3%)  nll=3.2289  ppl=25.25  grad=0.852  lr=2.84e-06  6867 tok/s  12.62s/it  [24:51:24<6:10:39]  mem=34.07GB
full_cpt step 11100/12852 ( 86.4%)  nll=3.2911  ppl=26.87  grad=0.820  lr=2.83e-06  6867 tok/s  12.62s/it  [24:53:20<6:08:30]  mem=34.07GB
  am   nll=3.9689  ppl=52.93  bpb=1.3957
  ta   nll=3.9433  ppl=51.59  bpb=0.5727
  kk   nll=2.7496  ppl=15.64  bpb=0.9259
  tl   nll=2.6594  ppl=14.29  bpb=1.2618
  en   nll=2.8175  ppl=16.73  bpb=0.8772
eval full_cpt step 11100  01:26 (86.2s)
full_cpt step 11110/12852 ( 86.4%)  nll=3.4101  ppl=30.27  grad=0.906  lr=2.82e-06  6862 tok/s  12.63s/it  [24:56:42<6:06:42]  mem=34.07GB
full_cpt step 11120/12852 ( 86.5%)  nll=3.3549  ppl=28.64  grad=0.855  lr=2.81e-06  6862 tok/s  12.63s/it  [24:58:38<6:04:33]  mem=34.07GB
full_cpt step 11130/12852 ( 86.6%)  nll=3.3015  ppl=27.15  grad=0.898  lr=2.80e-06  6863 tok/s  12.63s/it  [25:00:34<6:02:25]  mem=34.07GB
full_cpt step 11140/12852 ( 86.7%)  nll=3.3322  ppl=28.00  grad=0.848  lr=2.79e-06  6864 tok/s  12.63s/it  [25:02:30<6:00:16]  mem=34.07GB
full_cpt step 11150/12852 ( 86.8%)  nll=3.1508  ppl=23.35  grad=0.844  lr=2.78e-06  6865 tok/s  12.62s/it  [25:04:26<5:58:07]  mem=34.07GB
full_cpt step 11160/12852 ( 86.8%)  nll=3.3436  ppl=28.32  grad=0.852  lr=2.77e-06  6865 tok/s  12.62s/it  [25:06:22<5:55:58]  mem=34.07GB
full_cpt step 11170/12852 ( 86.9%)  nll=3.2663  ppl=26.21  grad=0.887  lr=2.77e-06  6866 tok/s  12.62s/it  [25:08:18<5:53:50]  mem=34.07GB
full_cpt step 11180/12852 ( 87.0%)  nll=3.3318  ppl=27.99  grad=0.875  lr=2.76e-06  6867 tok/s  12.62s/it  [25:10:14<5:51:41]  mem=34.07GB
full_cpt step 11190/12852 ( 87.1%)  nll=3.3659  ppl=28.96  grad=0.863  lr=2.75e-06  6868 tok/s  12.62s/it  [25:12:10<5:49:33]  mem=34.07GB
full_cpt step 11200/12852 ( 87.1%)  nll=3.4417  ppl=31.24  grad=0.934  lr=2.74e-06  6869 tok/s  12.62s/it  [25:14:06<5:47:24]  mem=34.07GB
  am   nll=3.9694  ppl=52.95  bpb=1.3958
  ta   nll=3.9432  ppl=51.59  bpb=0.5727
  kk   nll=2.7494  ppl=15.63  bpb=0.9258
  tl   nll=2.6592  ppl=14.28  bpb=1.2617
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 11200  01:26 (86.1s)
full_cpt step 11210/12852 ( 87.2%)  nll=3.2610  ppl=26.08  grad=0.875  lr=2.73e-06  6863 tok/s  12.63s/it  [25:17:28<5:45:35]  mem=34.07GB
full_cpt step 11220/12852 ( 87.3%)  nll=3.2138  ppl=24.87  grad=0.883  lr=2.72e-06  6864 tok/s  12.63s/it  [25:19:24<5:43:27]  mem=34.07GB
full_cpt step 11230/12852 ( 87.4%)  nll=3.2278  ppl=25.22  grad=0.918  lr=2.71e-06  6865 tok/s  12.63s/it  [25:21:20<5:41:18]  mem=34.07GB
full_cpt step 11240/12852 ( 87.5%)  nll=3.3890  ppl=29.64  grad=0.898  lr=2.70e-06  6866 tok/s  12.62s/it  [25:23:16<5:39:10]  mem=34.07GB
full_cpt step 11250/12852 ( 87.5%)  nll=3.2271  ppl=25.21  grad=0.855  lr=2.69e-06  6867 tok/s  12.62s/it  [25:25:12<5:37:01]  mem=34.07GB
full_cpt step 11260/12852 ( 87.6%)  nll=3.2460  ppl=25.69  grad=0.852  lr=2.69e-06  6867 tok/s  12.62s/it  [25:27:08<5:34:53]  mem=34.07GB
full_cpt step 11270/12852 ( 87.7%)  nll=3.2695  ppl=26.30  grad=0.844  lr=2.68e-06  6868 tok/s  12.62s/it  [25:29:04<5:32:44]  mem=34.07GB
full_cpt step 11280/12852 ( 87.8%)  nll=3.3620  ppl=28.85  grad=0.836  lr=2.67e-06  6869 tok/s  12.62s/it  [25:31:00<5:30:36]  mem=34.07GB
full_cpt step 11290/12852 ( 87.8%)  nll=3.3034  ppl=27.21  grad=0.855  lr=2.66e-06  6870 tok/s  12.62s/it  [25:32:56<5:28:27]  mem=34.07GB
full_cpt step 11300/12852 ( 87.9%)  nll=3.2827  ppl=26.65  grad=0.828  lr=2.65e-06  6871 tok/s  12.62s/it  [25:34:52<5:26:19]  mem=34.07GB
  am   nll=3.9693  ppl=52.95  bpb=1.3958
  ta   nll=3.9428  ppl=51.56  bpb=0.5727
  kk   nll=2.7491  ppl=15.63  bpb=0.9258
  tl   nll=2.6592  ppl=14.28  bpb=1.2617
  en   nll=2.8175  ppl=16.74  bpb=0.8773
eval full_cpt step 11300  01:26 (86.1s)
full_cpt step 11310/12852 ( 88.0%)  nll=3.3193  ppl=27.64  grad=0.852  lr=2.64e-06  6865 tok/s  12.63s/it  [25:38:14<5:24:29]  mem=34.07GB
full_cpt step 11320/12852 ( 88.1%)  nll=3.3934  ppl=29.77  grad=0.879  lr=2.64e-06  6866 tok/s  12.62s/it  [25:40:10<5:22:21]  mem=34.07GB
full_cpt step 11330/12852 ( 88.2%)  nll=3.3670  ppl=28.99  grad=0.855  lr=2.63e-06  6867 tok/s  12.62s/it  [25:42:06<5:20:12]  mem=34.07GB
full_cpt step 11340/12852 ( 88.2%)  nll=3.2577  ppl=25.99  grad=0.832  lr=2.62e-06  6868 tok/s  12.62s/it  [25:44:02<5:18:04]  mem=34.07GB
full_cpt step 11350/12852 ( 88.3%)  nll=3.2386  ppl=25.50  grad=0.879  lr=2.61e-06  6868 tok/s  12.62s/it  [25:45:58<5:15:55]  mem=34.07GB
full_cpt step 11360/12852 ( 88.4%)  nll=3.1962  ppl=24.44  grad=0.891  lr=2.60e-06  6869 tok/s  12.62s/it  [25:47:54<5:13:47]  mem=34.07GB
full_cpt step 11370/12852 ( 88.5%)  nll=3.3961  ppl=29.85  grad=0.824  lr=2.60e-06  6870 tok/s  12.62s/it  [25:49:50<5:11:39]  mem=34.07GB
full_cpt step 11380/12852 ( 88.5%)  nll=3.2220  ppl=25.08  grad=0.906  lr=2.59e-06  6871 tok/s  12.62s/it  [25:51:46<5:09:31]  mem=34.07GB
full_cpt step 11390/12852 ( 88.6%)  nll=3.2972  ppl=27.04  grad=0.898  lr=2.58e-06  6871 tok/s  12.61s/it  [25:53:42<5:07:23]  mem=34.07GB
full_cpt step 11400/12852 ( 88.7%)  nll=3.3606  ppl=28.81  grad=0.879  lr=2.57e-06  6872 tok/s  12.61s/it  [25:55:38<5:05:14]  mem=34.07GB
  am   nll=3.9691  ppl=52.94  bpb=1.3957
  ta   nll=3.9429  ppl=51.57  bpb=0.5727
  kk   nll=2.7493  ppl=15.63  bpb=0.9258
  tl   nll=2.6594  ppl=14.29  bpb=1.2618
  en   nll=2.8177  ppl=16.74  bpb=0.8773
eval full_cpt step 11400  01:26 (86.3s)
full_cpt step 11410/12852 ( 88.8%)  nll=3.4128  ppl=30.35  grad=0.875  lr=2.56e-06  6867 tok/s  12.62s/it  [25:59:00<5:03:23]  mem=34.07GB
full_cpt step 11420/12852 ( 88.9%)  nll=3.3287  ppl=27.90  grad=0.840  lr=2.56e-06  6867 tok/s  12.62s/it  [26:00:56<5:01:15]  mem=34.07GB
full_cpt step 11430/12852 ( 88.9%)  nll=3.3365  ppl=28.12  grad=0.875  lr=2.55e-06  6868 tok/s  12.62s/it  [26:02:52<4:59:07]  mem=34.07GB
full_cpt step 11440/12852 ( 89.0%)  nll=3.3314  ppl=27.98  grad=0.887  lr=2.54e-06  6868 tok/s  12.62s/it  [26:04:48<4:56:59]  mem=34.07GB
full_cpt step 11450/12852 ( 89.1%)  nll=3.2164  ppl=24.94  grad=0.797  lr=2.53e-06  6869 tok/s  12.62s/it  [26:06:44<4:54:50]  mem=34.07GB
full_cpt step 11460/12852 ( 89.2%)  nll=3.4636  ppl=31.93  grad=0.910  lr=2.53e-06  6870 tok/s  12.62s/it  [26:08:40<4:52:42]  mem=34.07GB
full_cpt step 11470/12852 ( 89.2%)  nll=3.1815  ppl=24.08  grad=0.883  lr=2.52e-06  6870 tok/s  12.62s/it  [26:10:36<4:50:34]  mem=34.07GB
full_cpt step 11480/12852 ( 89.3%)  nll=3.3939  ppl=29.78  grad=1.070  lr=2.51e-06  6871 tok/s  12.61s/it  [26:12:32<4:48:26]  mem=34.07GB
full_cpt step 11490/12852 ( 89.4%)  nll=3.3764  ppl=29.27  grad=0.848  lr=2.50e-06  6872 tok/s  12.61s/it  [26:14:28<4:46:18]  mem=34.07GB
full_cpt step 11500/12852 ( 89.5%)  nll=3.4465  ppl=31.39  grad=0.848  lr=2.50e-06  6872 tok/s  12.61s/it  [26:16:24<4:44:10]  mem=34.07GB
  am   nll=3.9691  ppl=52.94  bpb=1.3957
  ta   nll=3.9430  ppl=51.58  bpb=0.5727
  kk   nll=2.7491  ppl=15.63  bpb=0.9257
  tl   nll=2.6595  ppl=14.29  bpb=1.2619
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 11500  01:26 (86.2s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.17s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.17s/it]
removing old checkpoint checkpoint-10500
kept last 2 resume checkpoints: checkpoint-11000 checkpoint-11500
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-11500 at step 11500 next_row=1472000
full_cpt step 11510/12852 ( 89.6%)  nll=3.2031  ppl=24.61  grad=0.867  lr=2.49e-06  6863 tok/s  12.63s/it  [26:20:45<4:42:28]  mem=34.07GB
full_cpt step 11520/12852 ( 89.6%)  nll=3.4097  ppl=30.26  grad=0.859  lr=2.48e-06  6863 tok/s  12.63s/it  [26:22:41<4:40:20]  mem=34.07GB
full_cpt step 11530/12852 ( 89.7%)  nll=3.3344  ppl=28.06  grad=0.852  lr=2.48e-06  6864 tok/s  12.63s/it  [26:24:37<4:38:12]  mem=34.07GB
full_cpt step 11540/12852 ( 89.8%)  nll=3.3690  ppl=29.05  grad=0.906  lr=2.47e-06  6865 tok/s  12.63s/it  [26:26:33<4:36:04]  mem=34.07GB
full_cpt step 11550/12852 ( 89.9%)  nll=3.2948  ppl=26.97  grad=0.895  lr=2.46e-06  6865 tok/s  12.62s/it  [26:28:29<4:33:56]  mem=34.07GB
full_cpt step 11560/12852 ( 89.9%)  nll=3.3474  ppl=28.43  grad=0.883  lr=2.45e-06  6866 tok/s  12.62s/it  [26:30:25<4:31:48]  mem=34.07GB
full_cpt step 11570/12852 ( 90.0%)  nll=3.2816  ppl=26.62  grad=0.938  lr=2.45e-06  6867 tok/s  12.62s/it  [26:32:21<4:29:40]  mem=34.07GB
full_cpt step 11580/12852 ( 90.1%)  nll=3.2605  ppl=26.06  grad=0.832  lr=2.44e-06  6868 tok/s  12.62s/it  [26:34:17<4:27:32]  mem=34.07GB
full_cpt step 11590/12852 ( 90.2%)  nll=3.2791  ppl=26.55  grad=0.840  lr=2.43e-06  6869 tok/s  12.62s/it  [26:36:13<4:25:24]  mem=34.07GB
full_cpt step 11600/12852 ( 90.3%)  nll=3.3715  ppl=29.12  grad=0.887  lr=2.43e-06  6870 tok/s  12.62s/it  [26:38:10<4:23:17]  mem=34.07GB
  am   nll=3.9693  ppl=52.95  bpb=1.3958
  ta   nll=3.9428  ppl=51.56  bpb=0.5727
  kk   nll=2.7495  ppl=15.63  bpb=0.9259
  tl   nll=2.6593  ppl=14.29  bpb=1.2618
  en   nll=2.8178  ppl=16.74  bpb=0.8774
eval full_cpt step 11600  01:26 (86.5s)
full_cpt step 11610/12852 ( 90.3%)  nll=3.3559  ppl=28.67  grad=0.844  lr=2.42e-06  6864 tok/s  12.63s/it  [26:41:32<4:21:23]  mem=34.07GB
full_cpt step 11620/12852 ( 90.4%)  nll=3.2079  ppl=24.73  grad=0.887  lr=2.41e-06  6865 tok/s  12.63s/it  [26:43:28<4:19:15]  mem=34.07GB
full_cpt step 11630/12852 ( 90.5%)  nll=3.3224  ppl=27.73  grad=0.836  lr=2.41e-06  6866 tok/s  12.62s/it  [26:45:24<4:17:07]  mem=34.07GB
full_cpt step 11640/12852 ( 90.6%)  nll=3.3761  ppl=29.26  grad=0.863  lr=2.40e-06  6866 tok/s  12.62s/it  [26:47:20<4:14:59]  mem=34.07GB
full_cpt step 11650/12852 ( 90.6%)  nll=3.2260  ppl=25.18  grad=0.887  lr=2.39e-06  6867 tok/s  12.62s/it  [26:49:16<4:12:51]  mem=34.07GB
full_cpt step 11660/12852 ( 90.7%)  nll=3.3849  ppl=29.52  grad=0.836  lr=2.39e-06  6868 tok/s  12.62s/it  [26:51:12<4:10:43]  mem=34.07GB
full_cpt step 11670/12852 ( 90.8%)  nll=3.3109  ppl=27.41  grad=0.855  lr=2.38e-06  6868 tok/s  12.62s/it  [26:53:08<4:08:36]  mem=34.07GB
full_cpt step 11680/12852 ( 90.9%)  nll=3.2210  ppl=25.05  grad=0.895  lr=2.37e-06  6869 tok/s  12.62s/it  [26:55:04<4:06:28]  mem=34.07GB
full_cpt step 11690/12852 ( 91.0%)  nll=3.2757  ppl=26.46  grad=0.844  lr=2.37e-06  6870 tok/s  12.62s/it  [26:57:00<4:04:20]  mem=34.07GB
full_cpt step 11700/12852 ( 91.0%)  nll=3.1475  ppl=23.28  grad=0.863  lr=2.36e-06  6871 tok/s  12.62s/it  [26:58:56<4:02:13]  mem=34.07GB
  am   nll=3.9691  ppl=52.94  bpb=1.3957
  ta   nll=3.9434  ppl=51.59  bpb=0.5727
  kk   nll=2.7491  ppl=15.63  bpb=0.9257
  tl   nll=2.6593  ppl=14.29  bpb=1.2618
  en   nll=2.8173  ppl=16.73  bpb=0.8772
eval full_cpt step 11700  01:27 (86.5s)
full_cpt step 11710/12852 ( 91.1%)  nll=3.3130  ppl=27.47  grad=0.859  lr=2.36e-06  6865 tok/s  12.62s/it  [27:02:18<4:00:18]  mem=34.07GB
full_cpt step 11720/12852 ( 91.2%)  nll=3.3500  ppl=28.50  grad=0.883  lr=2.35e-06  6866 tok/s  12.62s/it  [27:04:14<3:58:10]  mem=34.07GB
full_cpt step 11730/12852 ( 91.3%)  nll=3.2161  ppl=24.93  grad=0.844  lr=2.34e-06  6867 tok/s  12.62s/it  [27:06:10<3:56:02]  mem=34.07GB
full_cpt step 11740/12852 ( 91.3%)  nll=3.4248  ppl=30.72  grad=0.875  lr=2.34e-06  6868 tok/s  12.62s/it  [27:08:06<3:53:55]  mem=34.07GB
full_cpt step 11750/12852 ( 91.4%)  nll=3.4631  ppl=31.92  grad=0.852  lr=2.33e-06  6869 tok/s  12.62s/it  [27:10:03<3:51:47]  mem=34.07GB
full_cpt step 11760/12852 ( 91.5%)  nll=3.4155  ppl=30.43  grad=0.895  lr=2.33e-06  6869 tok/s  12.62s/it  [27:11:59<3:49:39]  mem=34.07GB
full_cpt step 11770/12852 ( 91.6%)  nll=3.4101  ppl=30.27  grad=0.852  lr=2.32e-06  6870 tok/s  12.62s/it  [27:13:55<3:47:32]  mem=34.07GB
full_cpt step 11780/12852 ( 91.7%)  nll=3.2303  ppl=25.29  grad=0.863  lr=2.31e-06  6871 tok/s  12.62s/it  [27:15:51<3:45:24]  mem=34.07GB
full_cpt step 11790/12852 ( 91.7%)  nll=3.2281  ppl=25.23  grad=0.863  lr=2.31e-06  6872 tok/s  12.61s/it  [27:17:47<3:43:17]  mem=34.07GB
full_cpt step 11800/12852 ( 91.8%)  nll=3.3236  ppl=27.76  grad=0.875  lr=2.30e-06  6873 tok/s  12.61s/it  [27:19:43<3:41:09]  mem=34.07GB
  am   nll=3.9694  ppl=52.95  bpb=1.3958
  ta   nll=3.9432  ppl=51.59  bpb=0.5727
  kk   nll=2.7493  ppl=15.63  bpb=0.9258
  tl   nll=2.6593  ppl=14.29  bpb=1.2618
  en   nll=2.8173  ppl=16.73  bpb=0.8772
eval full_cpt step 11800  01:27 (86.6s)
full_cpt step 11810/12852 ( 91.9%)  nll=3.2661  ppl=26.21  grad=0.887  lr=2.30e-06  6868 tok/s  12.62s/it  [27:23:06<3:39:13]  mem=34.07GB
full_cpt step 11820/12852 ( 92.0%)  nll=3.3232  ppl=27.75  grad=0.871  lr=2.29e-06  6868 tok/s  12.62s/it  [27:25:02<3:37:06]  mem=34.07GB
full_cpt step 11830/12852 ( 92.0%)  nll=3.3590  ppl=28.76  grad=0.918  lr=2.29e-06  6869 tok/s  12.62s/it  [27:26:58<3:34:58]  mem=34.07GB
full_cpt step 11840/12852 ( 92.1%)  nll=3.3541  ppl=28.62  grad=0.855  lr=2.28e-06  6870 tok/s  12.62s/it  [27:28:54<3:32:51]  mem=34.07GB
full_cpt step 11850/12852 ( 92.2%)  nll=3.5019  ppl=33.18  grad=0.930  lr=2.27e-06  6870 tok/s  12.62s/it  [27:30:50<3:30:43]  mem=34.07GB
full_cpt step 11860/12852 ( 92.3%)  nll=3.2952  ppl=26.98  grad=0.922  lr=2.27e-06  6871 tok/s  12.62s/it  [27:32:46<3:28:36]  mem=34.07GB
full_cpt step 11870/12852 ( 92.4%)  nll=3.3790  ppl=29.34  grad=0.836  lr=2.26e-06  6872 tok/s  12.62s/it  [27:34:42<3:26:28]  mem=34.07GB
full_cpt step 11880/12852 ( 92.4%)  nll=3.2660  ppl=26.21  grad=0.867  lr=2.26e-06  6872 tok/s  12.61s/it  [27:36:38<3:24:21]  mem=34.07GB
full_cpt step 11890/12852 ( 92.5%)  nll=3.2964  ppl=27.02  grad=0.852  lr=2.25e-06  6873 tok/s  12.61s/it  [27:38:34<3:22:13]  mem=34.07GB
full_cpt step 11900/12852 ( 92.6%)  nll=3.2287  ppl=25.25  grad=0.871  lr=2.25e-06  6874 tok/s  12.61s/it  [27:40:30<3:20:06]  mem=34.07GB
  am   nll=3.9691  ppl=52.94  bpb=1.3957
  ta   nll=3.9429  ppl=51.57  bpb=0.5727
  kk   nll=2.7491  ppl=15.63  bpb=0.9257
  tl   nll=2.6595  ppl=14.29  bpb=1.2619
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 11900  01:27 (86.5s)
full_cpt step 11910/12852 ( 92.7%)  nll=3.3141  ppl=27.50  grad=0.852  lr=2.24e-06  6868 tok/s  12.62s/it  [27:43:52<3:18:09]  mem=34.07GB
full_cpt step 11920/12852 ( 92.7%)  nll=3.3723  ppl=29.14  grad=0.867  lr=2.24e-06  6869 tok/s  12.62s/it  [27:45:48<3:16:02]  mem=34.07GB
full_cpt step 11930/12852 ( 92.8%)  nll=3.3296  ppl=27.93  grad=0.930  lr=2.23e-06  6870 tok/s  12.62s/it  [27:47:44<3:13:54]  mem=34.07GB
full_cpt step 11940/12852 ( 92.9%)  nll=3.2488  ppl=25.76  grad=0.875  lr=2.23e-06  6870 tok/s  12.62s/it  [27:49:40<3:11:47]  mem=34.07GB
full_cpt step 11950/12852 ( 93.0%)  nll=3.3212  ppl=27.69  grad=0.879  lr=2.22e-06  6871 tok/s  12.62s/it  [27:51:36<3:09:40]  mem=34.07GB
full_cpt step 11960/12852 ( 93.1%)  nll=3.3053  ppl=27.26  grad=0.867  lr=2.22e-06  6872 tok/s  12.61s/it  [27:53:33<3:07:32]  mem=34.07GB
full_cpt step 11970/12852 ( 93.1%)  nll=3.1974  ppl=24.47  grad=0.836  lr=2.21e-06  6872 tok/s  12.61s/it  [27:55:29<3:05:25]  mem=34.07GB
full_cpt step 11980/12852 ( 93.2%)  nll=3.3048  ppl=27.24  grad=0.926  lr=2.21e-06  6873 tok/s  12.61s/it  [27:57:25<3:03:18]  mem=34.07GB
full_cpt step 11990/12852 ( 93.3%)  nll=3.2790  ppl=26.55  grad=0.848  lr=2.20e-06  6874 tok/s  12.61s/it  [27:59:21<3:01:11]  mem=34.07GB
full_cpt step 12000/12852 ( 93.4%)  nll=3.2217  ppl=25.07  grad=0.879  lr=2.20e-06  6875 tok/s  12.61s/it  [28:01:17<2:59:03]  mem=34.07GB
  am   nll=3.9697  ppl=52.97  bpb=1.3959
  ta   nll=3.9435  ppl=51.60  bpb=0.5728
  kk   nll=2.7491  ppl=15.63  bpb=0.9257
  tl   nll=2.6592  ppl=14.29  bpb=1.2618
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 12000  01:27 (86.9s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.29s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.29s/it]
removing old checkpoint checkpoint-11000
kept last 2 resume checkpoints: checkpoint-11500 checkpoint-12000
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-12000 at step 12000 next_row=1536000

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.49s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.49s/it]
wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-12000 (weights+expanded tokenizer, no optimizer)
full_cpt step 12010/12852 ( 93.4%)  nll=3.4169  ppl=30.47  grad=0.871  lr=2.19e-06  6867 tok/s  12.63s/it  [28:05:28<2:57:10]  mem=34.07GB
full_cpt step 12020/12852 ( 93.5%)  nll=3.3119  ppl=27.44  grad=0.855  lr=2.19e-06  6868 tok/s  12.62s/it  [28:07:24<2:55:03]  mem=34.07GB
full_cpt step 12030/12852 ( 93.6%)  nll=3.2800  ppl=26.58  grad=0.906  lr=2.18e-06  6868 tok/s  12.62s/it  [28:09:20<2:52:56]  mem=34.07GB
full_cpt step 12040/12852 ( 93.7%)  nll=3.3163  ppl=27.56  grad=0.828  lr=2.18e-06  6869 tok/s  12.62s/it  [28:11:16<2:50:49]  mem=34.07GB
full_cpt step 12050/12852 ( 93.8%)  nll=3.3509  ppl=28.53  grad=0.875  lr=2.18e-06  6870 tok/s  12.62s/it  [28:13:12<2:48:41]  mem=34.07GB
full_cpt step 12060/12852 ( 93.8%)  nll=3.1908  ppl=24.31  grad=0.863  lr=2.17e-06  6871 tok/s  12.62s/it  [28:15:08<2:46:34]  mem=34.07GB
full_cpt step 12070/12852 ( 93.9%)  nll=3.3532  ppl=28.59  grad=0.871  lr=2.17e-06  6871 tok/s  12.62s/it  [28:17:04<2:44:27]  mem=34.07GB
full_cpt step 12080/12852 ( 94.0%)  nll=3.2121  ppl=24.83  grad=0.863  lr=2.16e-06  6872 tok/s  12.62s/it  [28:19:00<2:42:20]  mem=34.07GB
full_cpt step 12090/12852 ( 94.1%)  nll=3.2520  ppl=25.84  grad=0.859  lr=2.16e-06  6873 tok/s  12.62s/it  [28:20:56<2:40:13]  mem=34.07GB
full_cpt step 12100/12852 ( 94.1%)  nll=3.2555  ppl=25.93  grad=0.855  lr=2.15e-06  6873 tok/s  12.61s/it  [28:22:52<2:38:06]  mem=34.07GB
  am   nll=3.9691  ppl=52.93  bpb=1.3957
  ta   nll=3.9430  ppl=51.57  bpb=0.5727
  kk   nll=2.7493  ppl=15.63  bpb=0.9258
  tl   nll=2.6594  ppl=14.29  bpb=1.2618
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 12100  01:27 (87.2s)
full_cpt step 12110/12852 ( 94.2%)  nll=3.2516  ppl=25.83  grad=0.820  lr=2.15e-06  6868 tok/s  12.62s/it  [28:26:15<2:36:07]  mem=34.07GB
full_cpt step 12120/12852 ( 94.3%)  nll=3.1082  ppl=22.38  grad=0.840  lr=2.15e-06  6869 tok/s  12.62s/it  [28:28:11<2:33:59]  mem=34.07GB
full_cpt step 12130/12852 ( 94.4%)  nll=3.3477  ppl=28.44  grad=0.871  lr=2.14e-06  6870 tok/s  12.62s/it  [28:30:07<2:31:52]  mem=34.07GB
full_cpt step 12140/12852 ( 94.5%)  nll=3.2855  ppl=26.72  grad=0.836  lr=2.14e-06  6870 tok/s  12.62s/it  [28:32:04<2:29:45]  mem=34.07GB
full_cpt step 12150/12852 ( 94.5%)  nll=3.4098  ppl=30.26  grad=0.867  lr=2.13e-06  6871 tok/s  12.62s/it  [28:34:00<2:27:38]  mem=34.07GB
full_cpt step 12160/12852 ( 94.6%)  nll=3.3428  ppl=28.30  grad=0.879  lr=2.13e-06  6872 tok/s  12.62s/it  [28:35:56<2:25:31]  mem=34.07GB
full_cpt step 12170/12852 ( 94.7%)  nll=3.2810  ppl=26.60  grad=0.887  lr=2.13e-06  6872 tok/s  12.62s/it  [28:37:52<2:23:24]  mem=34.07GB
full_cpt step 12180/12852 ( 94.8%)  nll=3.3206  ppl=27.68  grad=0.820  lr=2.12e-06  6873 tok/s  12.61s/it  [28:39:48<2:21:17]  mem=34.07GB
full_cpt step 12190/12852 ( 94.8%)  nll=3.2238  ppl=25.12  grad=0.883  lr=2.12e-06  6874 tok/s  12.61s/it  [28:41:44<2:19:10]  mem=34.07GB
full_cpt step 12200/12852 ( 94.9%)  nll=3.3122  ppl=27.45  grad=0.879  lr=2.12e-06  6875 tok/s  12.61s/it  [28:43:40<2:17:03]  mem=34.07GB
  am   nll=3.9692  ppl=52.94  bpb=1.3958
  ta   nll=3.9433  ppl=51.59  bpb=0.5727
  kk   nll=2.7490  ppl=15.63  bpb=0.9257
  tl   nll=2.6592  ppl=14.28  bpb=1.2617
  en   nll=2.8175  ppl=16.73  bpb=0.8772
eval full_cpt step 12200  01:26 (86.3s)
full_cpt step 12210/12852 ( 95.0%)  nll=3.2278  ppl=25.22  grad=0.855  lr=2.11e-06  6870 tok/s  12.62s/it  [28:47:02<2:15:03]  mem=34.07GB
full_cpt step 12220/12852 ( 95.1%)  nll=3.3338  ppl=28.04  grad=0.848  lr=2.11e-06  6870 tok/s  12.62s/it  [28:48:58<2:12:56]  mem=34.07GB
full_cpt step 12230/12852 ( 95.2%)  nll=3.3224  ppl=27.73  grad=0.848  lr=2.11e-06  6871 tok/s  12.62s/it  [28:50:54<2:10:49]  mem=34.07GB
full_cpt step 12240/12852 ( 95.2%)  nll=3.4459  ppl=31.37  grad=0.879  lr=2.10e-06  6872 tok/s  12.62s/it  [28:52:50<2:08:42]  mem=34.07GB
full_cpt step 12250/12852 ( 95.3%)  nll=3.2774  ppl=26.51  grad=0.895  lr=2.10e-06  6872 tok/s  12.62s/it  [28:54:46<2:06:35]  mem=34.07GB
full_cpt step 12260/12852 ( 95.4%)  nll=3.2019  ppl=24.58  grad=0.844  lr=2.10e-06  6873 tok/s  12.62s/it  [28:56:42<2:04:28]  mem=34.07GB
full_cpt step 12270/12852 ( 95.5%)  nll=3.2508  ppl=25.81  grad=0.918  lr=2.09e-06  6873 tok/s  12.61s/it  [28:58:38<2:02:21]  mem=34.07GB
full_cpt step 12280/12852 ( 95.5%)  nll=3.2031  ppl=24.61  grad=0.820  lr=2.09e-06  6874 tok/s  12.61s/it  [29:00:34<2:00:15]  mem=34.07GB
full_cpt step 12290/12852 ( 95.6%)  nll=3.2755  ppl=26.46  grad=0.863  lr=2.09e-06  6875 tok/s  12.61s/it  [29:02:30<1:58:08]  mem=34.07GB
full_cpt step 12300/12852 ( 95.7%)  nll=3.2840  ppl=26.68  grad=0.828  lr=2.08e-06  6876 tok/s  12.61s/it  [29:04:26<1:56:01]  mem=34.07GB
  am   nll=3.9689  ppl=52.93  bpb=1.3957
  ta   nll=3.9429  ppl=51.57  bpb=0.5727
  kk   nll=2.7496  ppl=15.64  bpb=0.9259
  tl   nll=2.6594  ppl=14.29  bpb=1.2618
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 12300  01:26 (86.2s)
full_cpt step 12310/12852 ( 95.8%)  nll=3.2591  ppl=26.03  grad=0.895  lr=2.08e-06  6871 tok/s  12.62s/it  [29:07:48<1:54:00]  mem=34.07GB
full_cpt step 12320/12852 ( 95.9%)  nll=3.2440  ppl=25.63  grad=0.875  lr=2.08e-06  6871 tok/s  12.62s/it  [29:09:44<1:51:53]  mem=34.07GB
full_cpt step 12330/12852 ( 95.9%)  nll=3.3189  ppl=27.63  grad=0.828  lr=2.07e-06  6872 tok/s  12.62s/it  [29:11:40<1:49:46]  mem=34.07GB
full_cpt step 12340/12852 ( 96.0%)  nll=3.3313  ppl=27.97  grad=0.883  lr=2.07e-06  6873 tok/s  12.62s/it  [29:13:36<1:47:39]  mem=34.07GB
full_cpt step 12350/12852 ( 96.1%)  nll=3.3223  ppl=27.72  grad=0.816  lr=2.07e-06  6873 tok/s  12.61s/it  [29:15:32<1:45:33]  mem=34.07GB
full_cpt step 12360/12852 ( 96.2%)  nll=3.2524  ppl=25.85  grad=0.875  lr=2.07e-06  6874 tok/s  12.61s/it  [29:17:28<1:43:26]  mem=34.07GB
full_cpt step 12370/12852 ( 96.2%)  nll=3.3972  ppl=29.88  grad=0.824  lr=2.06e-06  6875 tok/s  12.61s/it  [29:19:25<1:41:19]  mem=34.07GB
full_cpt step 12380/12852 ( 96.3%)  nll=3.3230  ppl=27.74  grad=0.836  lr=2.06e-06  6876 tok/s  12.61s/it  [29:21:21<1:39:12]  mem=34.07GB
full_cpt step 12390/12852 ( 96.4%)  nll=3.3778  ppl=29.31  grad=0.859  lr=2.06e-06  6876 tok/s  12.61s/it  [29:23:16<1:37:06]  mem=34.07GB
full_cpt step 12400/12852 ( 96.5%)  nll=3.2428  ppl=25.61  grad=0.871  lr=2.06e-06  6877 tok/s  12.61s/it  [29:25:12<1:34:59]  mem=34.07GB
  am   nll=3.9692  ppl=52.94  bpb=1.3958
  ta   nll=3.9424  ppl=51.54  bpb=0.5726
  kk   nll=2.7493  ppl=15.63  bpb=0.9258
  tl   nll=2.6594  ppl=14.29  bpb=1.2619
  en   nll=2.8174  ppl=16.73  bpb=0.8772
eval full_cpt step 12400  01:26 (86.4s)
full_cpt step 12410/12852 ( 96.6%)  nll=3.2718  ppl=26.36  grad=0.867  lr=2.05e-06  6872 tok/s  12.62s/it  [29:28:35<1:32:57]  mem=34.07GB
full_cpt step 12420/12852 ( 96.6%)  nll=3.3622  ppl=28.85  grad=0.859  lr=2.05e-06  6873 tok/s  12.62s/it  [29:30:31<1:30:50]  mem=34.07GB
full_cpt step 12430/12852 ( 96.7%)  nll=3.4398  ppl=31.18  grad=0.867  lr=2.05e-06  6873 tok/s  12.62s/it  [29:32:27<1:28:44]  mem=34.07GB
full_cpt step 12440/12852 ( 96.8%)  nll=3.2690  ppl=26.29  grad=0.848  lr=2.05e-06  6874 tok/s  12.61s/it  [29:34:23<1:26:37]  mem=34.07GB
full_cpt step 12450/12852 ( 96.9%)  nll=3.3431  ppl=28.31  grad=0.836  lr=2.04e-06  6875 tok/s  12.61s/it  [29:36:19<1:24:30]  mem=34.07GB
full_cpt step 12460/12852 ( 96.9%)  nll=3.2528  ppl=25.86  grad=0.906  lr=2.04e-06  6876 tok/s  12.61s/it  [29:38:15<1:22:24]  mem=34.07GB
full_cpt step 12470/12852 ( 97.0%)  nll=3.2068  ppl=24.70  grad=0.867  lr=2.04e-06  6876 tok/s  12.61s/it  [29:40:11<1:20:17]  mem=34.07GB
full_cpt step 12480/12852 ( 97.1%)  nll=3.3284  ppl=27.89  grad=0.867  lr=2.04e-06  6877 tok/s  12.61s/it  [29:42:07<1:18:11]  mem=34.07GB
full_cpt step 12490/12852 ( 97.2%)  nll=3.3612  ppl=28.82  grad=0.852  lr=2.04e-06  6877 tok/s  12.61s/it  [29:44:03<1:16:04]  mem=34.07GB
full_cpt step 12500/12852 ( 97.3%)  nll=3.2447  ppl=25.65  grad=0.820  lr=2.03e-06  6878 tok/s  12.61s/it  [29:46:00<1:13:58]  mem=34.07GB
  am   nll=3.9690  ppl=52.93  bpb=1.3957
  ta   nll=3.9425  ppl=51.55  bpb=0.5726
  kk   nll=2.7495  ppl=15.63  bpb=0.9259
  tl   nll=2.6594  ppl=14.29  bpb=1.2618
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 12500  01:26 (86.4s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.64s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.64s/it]
removing old checkpoint checkpoint-11500
kept last 2 resume checkpoints: checkpoint-12000 checkpoint-12500
wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-12500 at step 12500 next_row=1600000
full_cpt step 12510/12852 ( 97.3%)  nll=3.1386  ppl=23.07  grad=0.871  lr=2.03e-06  6869 tok/s  12.62s/it  [29:50:32<1:11:57]  mem=34.07GB
full_cpt step 12520/12852 ( 97.4%)  nll=3.4173  ppl=30.49  grad=0.871  lr=2.03e-06  6869 tok/s  12.62s/it  [29:52:28<1:09:51]  mem=34.07GB
full_cpt step 12530/12852 ( 97.5%)  nll=3.2343  ppl=25.39  grad=0.898  lr=2.03e-06  6870 tok/s  12.62s/it  [29:54:25<1:07:44]  mem=34.07GB
full_cpt step 12540/12852 ( 97.6%)  nll=3.4352  ppl=31.04  grad=0.879  lr=2.03e-06  6870 tok/s  12.62s/it  [29:56:20<1:05:38]  mem=34.07GB
full_cpt step 12550/12852 ( 97.7%)  nll=3.2317  ppl=25.32  grad=0.844  lr=2.03e-06  6871 tok/s  12.62s/it  [29:58:17<1:03:31]  mem=34.07GB
full_cpt step 12560/12852 ( 97.7%)  nll=3.2916  ppl=26.89  grad=0.867  lr=2.02e-06  6872 tok/s  12.62s/it  [30:00:13<1:01:25]  mem=34.07GB
full_cpt step 12570/12852 ( 97.8%)  nll=3.3406  ppl=28.24  grad=0.871  lr=2.02e-06  6872 tok/s  12.62s/it  [30:02:09<59:18]  mem=34.07GB
full_cpt step 12580/12852 ( 97.9%)  nll=3.3618  ppl=28.84  grad=0.910  lr=2.02e-06  6873 tok/s  12.62s/it  [30:04:05<57:12]  mem=34.07GB
full_cpt step 12590/12852 ( 98.0%)  nll=3.2779  ppl=26.52  grad=0.832  lr=2.02e-06  6874 tok/s  12.61s/it  [30:06:01<55:05]  mem=34.07GB
full_cpt step 12600/12852 ( 98.0%)  nll=3.2749  ppl=26.44  grad=0.902  lr=2.02e-06  6874 tok/s  12.61s/it  [30:07:57<52:59]  mem=34.07GB
  am   nll=3.9693  ppl=52.95  bpb=1.3958
  ta   nll=3.9426  ppl=51.55  bpb=0.5726
  kk   nll=2.7492  ppl=15.63  bpb=0.9258
  tl   nll=2.6594  ppl=14.29  bpb=1.2618
  en   nll=2.8175  ppl=16.73  bpb=0.8772
eval full_cpt step 12600  01:27 (86.9s)
full_cpt step 12610/12852 ( 98.1%)  nll=3.2084  ppl=24.74  grad=0.867  lr=2.02e-06  6869 tok/s  12.62s/it  [30:11:20<50:55]  mem=34.07GB
full_cpt step 12620/12852 ( 98.2%)  nll=3.3880  ppl=29.61  grad=0.883  lr=2.01e-06  6870 tok/s  12.62s/it  [30:13:16<48:48]  mem=34.07GB
full_cpt step 12630/12852 ( 98.3%)  nll=3.4006  ppl=29.98  grad=0.875  lr=2.01e-06  6871 tok/s  12.62s/it  [30:15:12<46:42]  mem=34.07GB
full_cpt step 12640/12852 ( 98.4%)  nll=3.2647  ppl=26.17  grad=0.875  lr=2.01e-06  6871 tok/s  12.62s/it  [30:17:08<44:35]  mem=34.07GB
full_cpt step 12650/12852 ( 98.4%)  nll=3.2486  ppl=25.76  grad=0.879  lr=2.01e-06  6872 tok/s  12.62s/it  [30:19:04<42:29]  mem=34.07GB
full_cpt step 12660/12852 ( 98.5%)  nll=3.2810  ppl=26.60  grad=0.824  lr=2.01e-06  6873 tok/s  12.62s/it  [30:21:00<40:22]  mem=34.07GB
full_cpt step 12670/12852 ( 98.6%)  nll=3.2426  ppl=25.60  grad=0.832  lr=2.01e-06  6873 tok/s  12.62s/it  [30:22:56<38:16]  mem=34.07GB
full_cpt step 12680/12852 ( 98.7%)  nll=3.2754  ppl=26.46  grad=0.848  lr=2.01e-06  6874 tok/s  12.61s/it  [30:24:52<36:10]  mem=34.07GB
full_cpt step 12690/12852 ( 98.7%)  nll=3.2832  ppl=26.66  grad=0.836  lr=2.01e-06  6874 tok/s  12.61s/it  [30:26:48<34:03]  mem=34.07GB
full_cpt step 12700/12852 ( 98.8%)  nll=3.2605  ppl=26.06  grad=0.871  lr=2.01e-06  6875 tok/s  12.61s/it  [30:28:44<31:57]  mem=34.07GB
  am   nll=3.9690  ppl=52.93  bpb=1.3957
  ta   nll=3.9429  ppl=51.57  bpb=0.5727
  kk   nll=2.7490  ppl=15.63  bpb=0.9257
  tl   nll=2.6591  ppl=14.28  bpb=1.2617
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 12700  01:27 (87.3s)
full_cpt step 12710/12852 ( 98.9%)  nll=3.3232  ppl=27.75  grad=0.848  lr=2.01e-06  6870 tok/s  12.62s/it  [30:32:07<29:52]  mem=34.07GB
full_cpt step 12720/12852 ( 99.0%)  nll=3.3074  ppl=27.31  grad=0.852  lr=2.00e-06  6871 tok/s  12.62s/it  [30:34:03<27:46]  mem=34.07GB
full_cpt step 12730/12852 ( 99.1%)  nll=3.2786  ppl=26.54  grad=0.875  lr=2.00e-06  6872 tok/s  12.62s/it  [30:35:59<25:39]  mem=34.07GB
full_cpt step 12740/12852 ( 99.1%)  nll=3.3467  ppl=28.41  grad=0.871  lr=2.00e-06  6872 tok/s  12.62s/it  [30:37:55<23:33]  mem=34.07GB
full_cpt step 12750/12852 ( 99.2%)  nll=3.2962  ppl=27.01  grad=0.840  lr=2.00e-06  6873 tok/s  12.62s/it  [30:39:51<21:27]  mem=34.07GB
full_cpt step 12760/12852 ( 99.3%)  nll=3.2643  ppl=26.16  grad=0.891  lr=2.00e-06  6873 tok/s  12.62s/it  [30:41:48<19:21]  mem=34.07GB
full_cpt step 12770/12852 ( 99.4%)  nll=3.2927  ppl=26.91  grad=0.895  lr=2.00e-06  6874 tok/s  12.61s/it  [30:43:44<17:14]  mem=34.07GB
full_cpt step 12780/12852 ( 99.4%)  nll=3.2312  ppl=25.31  grad=0.883  lr=2.00e-06  6874 tok/s  12.61s/it  [30:45:40<15:08]  mem=34.07GB
full_cpt step 12790/12852 ( 99.5%)  nll=3.3785  ppl=29.33  grad=0.902  lr=2.00e-06  6875 tok/s  12.61s/it  [30:47:36<13:02]  mem=34.07GB
full_cpt step 12800/12852 ( 99.6%)  nll=3.2819  ppl=26.63  grad=0.855  lr=2.00e-06  6876 tok/s  12.61s/it  [30:49:32<10:56]  mem=34.07GB
  am   nll=3.9691  ppl=52.93  bpb=1.3957
  ta   nll=3.9432  ppl=51.58  bpb=0.5727
  kk   nll=2.7491  ppl=15.63  bpb=0.9257
  tl   nll=2.6594  ppl=14.29  bpb=1.2618
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 12800  01:28 (88.2s)
full_cpt step 12810/12852 ( 99.7%)  nll=3.2556  ppl=25.94  grad=1.375  lr=2.00e-06  6871 tok/s  12.62s/it  [30:52:56<08:50]  mem=34.07GB
full_cpt step 12820/12852 ( 99.8%)  nll=3.2670  ppl=26.23  grad=0.867  lr=2.00e-06  6872 tok/s  12.62s/it  [30:54:53<06:44]  mem=34.07GB
full_cpt step 12830/12852 ( 99.8%)  nll=3.3131  ppl=27.47  grad=0.867  lr=2.00e-06  6872 tok/s  12.62s/it  [30:56:49<04:38]  mem=34.07GB
full_cpt step 12840/12852 ( 99.9%)  nll=3.1935  ppl=24.37  grad=0.824  lr=2.00e-06  6873 tok/s  12.62s/it  [30:58:45<02:31]  mem=34.07GB
full_cpt step 12850/12852 (100.0%)  nll=3.3764  ppl=29.26  grad=0.883  lr=2.00e-06  6874 tok/s  12.61s/it  [31:00:41<00:25]  mem=34.07GB
full_cpt step 12852/12852 (100.0%)  nll=3.3621  ppl=28.85  grad=0.914  lr=2.00e-06  6874 tok/s  12.61s/it  [31:01:04<00:00]  mem=34.07GB
  am   nll=3.9691  ppl=52.94  bpb=1.3957
  ta   nll=3.9430  ppl=51.57  bpb=0.5727
  kk   nll=2.7493  ppl=15.63  bpb=0.9258
  tl   nll=2.6592  ppl=14.29  bpb=1.2618
  en   nll=2.8176  ppl=16.74  bpb=0.8773
eval full_cpt step 12852  01:28 (88.2s)

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.30s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00,  2.30s/it]
done  research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-final