classification/README.md

Theme classifiers (romance / sadness / celebration / motivational)

For each of the 15 combinations of the inputs lyrics, vocal, instrument, mix and each of the 5 cross-validation folds: the configuration that the validation split picked, trained on a Kaggle T4 GPU with the exact split, standardization, seed and settings of the reported results (re-scoring every model on its test fold reproduces the reported test macro-F1 exactly).

Files per model: <combination>/fold_<k>/

• model.pt: PyTorch state_dict
• config.json: modalities (input order), method and settings, input sizes, model and training config, validation / test macro-F1
• norm.npz: per-input mean and std of that fold's training songs; inputs must be standardized as (x - mean) / std

Two entries are late fusion (config.json only): they average the single-modality models of the same fold. index.csv lists all 75 entries.

Inputs

• lyrics: 768-d embedding from lyrics_encoder/theme_tuned/fold_<k> (the same fold!), L2-normalized
• vocal / instrument / mix: 13 x 768 = 9,984 values (all hidden layers of wav2vec 2.0 on the separated voice / MERT on the accompaniment / MERT on the full mix, layer-major)

Loading

import json, numpy as np, torch, sys
sys.path.insert(0, "code")                     # code/classify.py from this repo
import classify as C
d = "classification/lyrics+instrument+mix/fold_0"
info = json.load(open(f"{d}/config.json")); norm = np.load(f"{d}/norm.npz")
model = C.build_model(info["run"], info["modalities"], info["in_dims"], info["model_config"]).cpu()
model.load_state_dict(torch.load(f"{d}/model.pt", map_location="cpu")); model.eval()
xs = [torch.tensor((X[m] - norm[f"{m}_mean"]) / norm[f"{m}_std"], dtype=torch.float32) for m in info["modalities"]]
proba = C.predict_proba(model, xs)             # (n_songs, 4) in the order of info["themes"]

Mean test macro-F1 over the 5 folds (late-fusion folds excluded)

combination	macro-F1	method chosen most
lyrics+instrument+mix	0.699	xattn
lyrics+instrument	0.694	gmu
lyrics+vocal	0.693	mlp
lyrics+vocal+instrument	0.692	mlp
lyrics+mix	0.691	gmu
lyrics+vocal+mix	0.690	gmu
lyrics+vocal+instrument+mix	0.690	mlp
lyrics	0.682	mlp
vocal+mix	0.513	gmu
vocal+instrument+mix	0.513	gmu
vocal+instrument	0.508	mlp
instrument+mix	0.507	gmu
mix	0.500	mlp
vocal	0.493	xattn
instrument	0.457	xattn