Card: training progress with live plots
Browse files
README.md
CHANGED
|
@@ -66,6 +66,17 @@ The model reads codes from the band-group tokenizer, which splits the 128 mel ba
|
|
| 66 |
|
| 67 |
`evaluations/<run>/` holds the paired lyrics test of each lyrics run: the per-track loss with aligned, removed and mismatched lyrics.
|
| 68 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
## Layout
|
| 70 |
|
| 71 |
```text
|
|
|
|
| 66 |
|
| 67 |
`evaluations/<run>/` holds the paired lyrics test of each lyrics run: the per-track loss with aligned, removed and mismatched lyrics.
|
| 68 |
|
| 69 |
+
## Training progress
|
| 70 |
+
|
| 71 |
+
The runs on the Jetson AGX Thor publish their metrics here every two hours
|
| 72 |
+
(`deploy/hf/publish.sh` in the code repository): `metrics/<run>.jsonl` holds
|
| 73 |
+
every logged event, and `plots/<run>.png` shows, from left to right, the
|
| 74 |
+
training and validation loss in nats per code against optimizer step, the
|
| 75 |
+
latest validation loss of each codebook (coarse levels are far easier to
|
| 76 |
+
predict than fine ones), and the throughput in positions per second.
|
| 77 |
+
|
| 78 |
+

|
| 79 |
+
|
| 80 |
## Layout
|
| 81 |
|
| 82 |
```text
|