File size: 2,524 Bytes
4097db2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | # Theme classifiers (romance / sadness / celebration / motivational)
For each of the 15 combinations of the inputs **lyrics, vocal, instrument, mix** and each of the
5 cross-validation folds: the configuration that the validation split picked, trained on a Kaggle
T4 GPU with the exact split, standardization, seed and settings of the reported results (re-scoring
every model on its test fold reproduces the reported test macro-F1 exactly).
## Files per model: `<combination>/fold_<k>/`
- `model.pt`: PyTorch `state_dict`
- `config.json`: modalities (input order), method and settings, input sizes, model and training config,
validation / test macro-F1
- `norm.npz`: per-input mean and std of that fold's training songs; inputs must be standardized as
`(x - mean) / std`
Two entries are late fusion (`config.json` only): they average the single-modality models of the
same fold. `index.csv` lists all 75 entries.
## Inputs
- lyrics: 768-d embedding from `lyrics_encoder/theme_tuned/fold_<k>` (the same fold!), L2-normalized
- vocal / instrument / mix: 13 x 768 = 9,984 values (all hidden layers of wav2vec 2.0 on the separated
voice / MERT on the accompaniment / MERT on the full mix, layer-major)
## Loading
```python
import json, numpy as np, torch, sys
sys.path.insert(0, "code") # code/classify.py from this repo
import classify as C
d = "classification/lyrics+instrument+mix/fold_0"
info = json.load(open(f"{d}/config.json")); norm = np.load(f"{d}/norm.npz")
model = C.build_model(info["run"], info["modalities"], info["in_dims"], info["model_config"]).cpu()
model.load_state_dict(torch.load(f"{d}/model.pt", map_location="cpu")); model.eval()
xs = [torch.tensor((X[m] - norm[f"{m}_mean"]) / norm[f"{m}_std"], dtype=torch.float32) for m in info["modalities"]]
proba = C.predict_proba(model, xs) # (n_songs, 4) in the order of info["themes"]
```
## Mean test macro-F1 over the 5 folds (late-fusion folds excluded)
| combination | macro-F1 | method chosen most |
|---|---|---|
| lyrics+instrument+mix | 0.699 | xattn |
| lyrics+instrument | 0.694 | gmu |
| lyrics+vocal | 0.693 | mlp |
| lyrics+vocal+instrument | 0.692 | mlp |
| lyrics+mix | 0.691 | gmu |
| lyrics+vocal+mix | 0.690 | gmu |
| lyrics+vocal+instrument+mix | 0.690 | mlp |
| lyrics | 0.682 | mlp |
| vocal+mix | 0.513 | gmu |
| vocal+instrument+mix | 0.513 | gmu |
| vocal+instrument | 0.508 | mlp |
| instrument+mix | 0.507 | gmu |
| mix | 0.500 | mlp |
| vocal | 0.493 | xattn |
| instrument | 0.457 | xattn |
|