| # Theme classifiers (romance / sadness / celebration / motivational) |
|
|
| For each of the 15 combinations of the inputs **lyrics, vocal, instrument, mix** and each of the |
| 5 cross-validation folds: the configuration that the validation split picked, trained on a Kaggle |
| T4 GPU with the exact split, standardization, seed and settings of the reported results (re-scoring |
| every model on its test fold reproduces the reported test macro-F1 exactly). |
|
|
| ## Files per model: `<combination>/fold_<k>/` |
| - `model.pt`: PyTorch `state_dict` |
| - `config.json`: modalities (input order), method and settings, input sizes, model and training config, |
| validation / test macro-F1 |
| - `norm.npz`: per-input mean and std of that fold's training songs; inputs must be standardized as |
| `(x - mean) / std` |
|
|
| Two entries are late fusion (`config.json` only): they average the single-modality models of the |
| same fold. `index.csv` lists all 75 entries. |
|
|
| ## Inputs |
| - lyrics: 768-d embedding from `lyrics_encoder/theme_tuned/fold_<k>` (the same fold!), L2-normalized |
| - vocal / instrument / mix: 13 x 768 = 9,984 values (all hidden layers of wav2vec 2.0 on the separated |
| voice / MERT on the accompaniment / MERT on the full mix, layer-major) |
|
|
| ## Loading |
| ```python |
| import json, numpy as np, torch, sys |
| sys.path.insert(0, "code") # code/classify.py from this repo |
| import classify as C |
| d = "classification/lyrics+instrument+mix/fold_0" |
| info = json.load(open(f"{d}/config.json")); norm = np.load(f"{d}/norm.npz") |
| model = C.build_model(info["run"], info["modalities"], info["in_dims"], info["model_config"]).cpu() |
| model.load_state_dict(torch.load(f"{d}/model.pt", map_location="cpu")); model.eval() |
| xs = [torch.tensor((X[m] - norm[f"{m}_mean"]) / norm[f"{m}_std"], dtype=torch.float32) for m in info["modalities"]] |
| proba = C.predict_proba(model, xs) # (n_songs, 4) in the order of info["themes"] |
| ``` |
|
|
| ## Mean test macro-F1 over the 5 folds (late-fusion folds excluded) |
| | combination | macro-F1 | method chosen most | |
| |---|---|---| |
| | lyrics+instrument+mix | 0.699 | xattn | |
| | lyrics+instrument | 0.694 | gmu | |
| | lyrics+vocal | 0.693 | mlp | |
| | lyrics+vocal+instrument | 0.692 | mlp | |
| | lyrics+mix | 0.691 | gmu | |
| | lyrics+vocal+mix | 0.690 | gmu | |
| | lyrics+vocal+instrument+mix | 0.690 | mlp | |
| | lyrics | 0.682 | mlp | |
| | vocal+mix | 0.513 | gmu | |
| | vocal+instrument+mix | 0.513 | gmu | |
| | vocal+instrument | 0.508 | mlp | |
| | instrument+mix | 0.507 | gmu | |
| | mix | 0.500 | mlp | |
| | vocal | 0.493 | xattn | |
| | instrument | 0.457 | xattn | |
|
|
