aux0: logs

results/aux0/logs/diloco_aux0.log

@@ -30,3 +30,5 @@
 15:11:02 w0 round 23 step 23000/32000 inner 207s (207 ms/step) sync 1s used [0] lr 1.99e-04 loss [1.3587, 0.5462, 11.9215, 14.186]
 15:14:34 w0 round 24 step 24000/32000 inner 207s (207 ms/step) sync 1s used [0] lr 1.64e-04 loss [1.3378, 0.5458, 11.9032, 14.0124] | VAL top1 0.5332 value-bce 0.546 aux top1 [0.0012, 0.0]
 15:18:21 w0 round 25 step 25000/32000 inner 207s (207 ms/step) sync 1s used [0] lr 1.31e-04 loss [1.34, 0.5491, 11.7418, 14.1249]
+15:27:12 w0 commit aux0: r25 w0 failed (ConnectionError: Network error: Request error: HTTP status client error (429 Too Many Requests), domain: https://cas-server.xethub.hf.co/v1/shards), retry 1
+15:27:57 w0 round 26 step 26000/32000 inner 207s (207 ms/step) sync 1s used [0] lr 1.03e-04 loss [1.3509, 0.5496, 12.0185, 14.664] | VAL top1 0.5379 value-bce 0.5455 aux top1 [0.001, 0.0]