Upload log_ts_d3.txt with huggingface_hub

log_ts_d3.txt

@@ -796,31 +796,56 @@ Consider using tensor.detach() first. (Triggered internally at /pytorch/torch/cs
 
  /tmp/ts_d3_m.pt             : 100%|██████████| 75.8MB / 75.8MB            
 
 
  /tmp/ts_d3_m.pt             : 100%|██████████| 75.8MB / 75.8MB            
-01:33:19 push fail (Request ID: Root=1-6aa208df-055db7d50bb669340dd3c49b;3affe704-df38-4fdf-859d-aaa3d6f8c6c0)
+
+
 
-429 Too Many Requests: you 
-01:33:21 pulled global r=10 2.853 -> 2.893
-01:33:21 push fail (Request ID: Root=1-6aa208e1-3dc474ff7526802e0a325d81;edb0b1b7-5468-4579-8633-38cddde11d28)
+
  /tmp/ts_d3_m.pt             :   4%|▍         | 3.24MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             :   4%|▍         | 3.24MB / 75.8MB            
+
+
+
  /tmp/ts_d3_m.pt             :  26%|██▋       | 19.9MB / 75.8MB            
+
+
+
  /tmp/ts_d3_m.pt             :  71%|███████   | 53.9MB / 75.8MB            
+
+
+
  /tmp/ts_d3_m.pt             :  99%|█████████▉| 75.3MB / 75.8MB            
+
+
+
  /tmp/ts_d3_m.pt             :  99%|█████████▉| 75.3MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             :  99%|█████████▉| 75.3MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             :  99%|█████████▉| 75.3MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             :  99%|█████████▉| 75.3MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             :  99%|█████████▉| 75.3MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             :  99%|█████████▉| 75.3MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             :  99%|█████████▉| 75.3MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             :  99%|█████████▉| 75.3MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             : 100%|██████████| 75.8MB / 75.8MB            
+
+
+
  /tmp/ts_d3_m.pt             : 100%|██████████| 75.8MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             : 100%|██████████| 75.8MB / 75.8MB            
+01:40:35 pushed ts_d3.pt
+01:40:35 push fail (Request ID: Root=1-6aa20a93-3880756d011f1f83030522a9;43c90878-c777-4582-b077-1788bd83d7e0)
 
 429 Too Many Requests: you 
-01:33:36 step 72100 loss=2.7853 tok=1181.3M dt=8553
-01:33:50 step 72200 loss=2.8359 tok=1182.9M dt=8567
-01:34:05 step 72300 loss=2.8192 tok=1184.6M dt=8582
-01:34:20 step 72400 loss=2.8171 tok=1186.2M dt=8596
-01:34:34 step 72500 loss=2.8041 tok=1187.8M dt=8611
-01:34:48 step 72600 loss=2.7796 tok=1189.5M dt=8625
-01:35:02 step 72700 loss=2.8303 tok=1191.1M dt=8639
-01:35:16 step 72800 loss=2.8646 tok=1192.8M dt=8653
-01:35:31 step 72900 loss=2.8436 tok=1194.4M dt=8668
-01:35:45 step 73000 loss=2.7976 tok=1196.0M dt=8682
-01:35:46 pushed log_ts_d3.txt
-01:36:00 step 73100 loss=2.7527 tok=1197.7M dt=8697
-01:36:14 step 73200 loss=2.8079 tok=1199.3M dt=8711
-01:36:28 step 73300 loss=2.7832 tok=1200.9M dt=8725
-01:36:42 step 73400 loss=2.7436 tok=1202.6M dt=8739
-01:36:57 step 73500 loss=2.7837 tok=1204.2M dt=8753
-01:37:11 step 73600 loss=2.8045 tok=1205.9M dt=8768
-01:37:25 step 73700 loss=2.7930 tok=1207.5M dt=8782
-01:37:39 step 73800 loss=2.7792 tok=1209.1M dt=8796
-01:37:54 step 73900 loss=2.7377 tok=1210.8M dt=8811
-01:38:08 step 74000 loss=2.7989 tok=1212.4M dt=8825
+01:40:50 step 75100 loss=2.7720 tok=1230.4M dt=8987
+01:41:04 step 75200 loss=2.7496 tok=1232.1M dt=9001
+01:41:18 step 75300 loss=2.7833 tok=1233.7M dt=9015
+01:41:33 step 75400 loss=2.7686 tok=1235.4M dt=9030
+01:41:48 step 75500 loss=2.8170 tok=1237.0M dt=9044
+01:42:02 step 75600 loss=2.8384 tok=1238.6M dt=9059
+01:42:16 step 75700 loss=2.8278 tok=1240.3M dt=9073
+01:42:31 step 75800 loss=2.8101 tok=1241.9M dt=9088
+01:42:46 step 75900 loss=2.7982 tok=1243.5M dt=9102
+01:43:00 step 76000 loss=2.7799 tok=1245.2M dt=9117