arpanpathak commited on
Commit
56bf4e8
·
verified ·
1 Parent(s): 17eb791

Card: training progress with live plots

Browse files
Files changed (1) hide show
  1. README.md +11 -0
README.md CHANGED
@@ -66,6 +66,17 @@ The model reads codes from the band-group tokenizer, which splits the 128 mel ba
66
 
67
  `evaluations/<run>/` holds the paired lyrics test of each lyrics run: the per-track loss with aligned, removed and mismatched lyrics.
68
 
 
 
 
 
 
 
 
 
 
 
 
69
  ## Layout
70
 
71
  ```text
 
66
 
67
  `evaluations/<run>/` holds the paired lyrics test of each lyrics run: the per-track loss with aligned, removed and mismatched lyrics.
68
 
69
+ ## Training progress
70
+
71
+ The runs on the Jetson AGX Thor publish their metrics here every two hours
72
+ (`deploy/hf/publish.sh` in the code repository): `metrics/<run>.jsonl` holds
73
+ every logged event, and `plots/<run>.png` shows, from left to right, the
74
+ training and validation loss in nats per code against optimizer step, the
75
+ latest validation loss of each codebook (coarse levels are far easier to
76
+ predict than fine ones), and the throughput in positions per second.
77
+
78
+ ![pq-large training curves](plots/pq-large.png)
79
+
80
  ## Layout
81
 
82
  ```text