Upload log_ts_d3.txt with huggingface_hub

log_ts_d3.txt

@@ -761,28 +761,55 @@ Consider using tensor.detach() first. (Triggered internally at /pytorch/torch/cs
 
  /tmp/ts_d3_m.pt             : 100%|██████████| 75.8MB / 75.8MB            
 
 
  /tmp/ts_d3_m.pt             : 100%|██████████| 75.8MB / 75.8MB            
-01:25:53 pushed ts_d3.pt
-01:25:53 push fail (Request ID: Root=1-6aa20721-51367ae648df7cd853c5ba47;40aa8fc9-07ac-41a6-879b-ed78397a1744)
+
+
+
+
  /tmp/ts_d3_m.pt             :   1%|          |  645kB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             :   1%|          |  645kB / 75.8MB            
+
+
+
  /tmp/ts_d3_m.pt             :  41%|████      | 30.7MB / 75.8MB            
+
+
+
  /tmp/ts_d3_m.pt             :  88%|████████▊ | 66.4MB / 75.8MB            
+
+
+
  /tmp/ts_d3_m.pt             : 100%|█████████▉| 75.7MB / 75.8MB            
+
+
+
  /tmp/ts_d3_m.pt             : 100%|█████████▉| 75.7MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             : 100%|█████████▉| 75.7MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             : 100%|█████████▉| 75.7MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             : 100%|█████████▉| 75.7MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             : 100%|█████████▉| 75.7MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             : 100%|█████████▉| 75.7MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             : 100%|██████████| 75.8MB / 75.8MB            
+
+
+
  /tmp/ts_d3_m.pt             : 100%|██████████| 75.8MB / 75.8MB            
+
+
  /tmp/ts_d3_m.pt             : 100%|██████████| 75.8MB / 75.8MB            
+01:33:19 push fail (Request ID: Root=1-6aa208df-055db7d50bb669340dd3c49b;3affe704-df38-4fdf-859d-aaa3d6f8c6c0)
+
+429 Too Many Requests: you 
+01:33:21 pulled global r=10 2.853 -> 2.893
+01:33:21 push fail (Request ID: Root=1-6aa208e1-3dc474ff7526802e0a325d81;edb0b1b7-5468-4579-8633-38cddde11d28)
 
 429 Too Many Requests: you 
-01:26:07 step 69100 loss=2.8322 tok=1132.1M dt=8104
-01:26:21 step 69200 loss=2.7952 tok=1133.8M dt=8118
-01:26:36 step 69300 loss=2.8049 tok=1135.4M dt=8132
-01:26:50 step 69400 loss=2.7747 tok=1137.0M dt=8147
-01:27:04 step 69500 loss=2.8326 tok=1138.7M dt=8161
-01:27:19 step 69600 loss=2.8473 tok=1140.3M dt=8176
-01:27:33 step 69700 loss=2.8324 tok=1142.0M dt=8190
-01:27:48 step 69800 loss=2.8068 tok=1143.6M dt=8205
-01:28:02 step 69900 loss=2.7929 tok=1145.2M dt=8219
-01:28:17 step 70000 loss=2.8241 tok=1146.9M dt=8234
-01:28:17 pushed log_ts_d3.txt
-01:28:32 step 70100 loss=2.7958 tok=1148.5M dt=8249
-01:28:47 step 70200 loss=2.8406 tok=1150.2M dt=8263
-01:29:01 step 70300 loss=2.7979 tok=1151.8M dt=8278
-01:29:15 step 70400 loss=2.8135 tok=1153.4M dt=8292
-01:29:29 step 70500 loss=2.8440 tok=1155.1M dt=8306
-01:29:44 step 70600 loss=2.8093 tok=1156.7M dt=8321
-01:29:58 step 70700 loss=2.8335 tok=1158.3M dt=8335
-01:30:12 step 70800 loss=2.8043 tok=1160.0M dt=8349
-01:30:27 step 70900 loss=2.8321 tok=1161.6M dt=8363
-01:30:41 step 71000 loss=2.8089 tok=1163.3M dt=8378
+01:33:36 step 72100 loss=2.7853 tok=1181.3M dt=8553
+01:33:50 step 72200 loss=2.8359 tok=1182.9M dt=8567
+01:34:05 step 72300 loss=2.8192 tok=1184.6M dt=8582
+01:34:20 step 72400 loss=2.8171 tok=1186.2M dt=8596
+01:34:34 step 72500 loss=2.8041 tok=1187.8M dt=8611
+01:34:48 step 72600 loss=2.7796 tok=1189.5M dt=8625
+01:35:02 step 72700 loss=2.8303 tok=1191.1M dt=8639
+01:35:16 step 72800 loss=2.8646 tok=1192.8M dt=8653
+01:35:31 step 72900 loss=2.8436 tok=1194.4M dt=8668
+01:35:45 step 73000 loss=2.7976 tok=1196.0M dt=8682