# Theme classifiers (romance / sadness / celebration / motivational)

For each of the 15 combinations of the inputs **lyrics, vocal, instrument, mix** and each of the
5 cross-validation folds: the configuration that the validation split picked, trained on a Kaggle
T4 GPU with the exact split, standardization, seed and settings of the reported results (re-scoring
every model on its test fold reproduces the reported test macro-F1 exactly).

## Files per model: `<combination>/fold_<k>/`
- `model.pt`: PyTorch `state_dict`
- `config.json`: modalities (input order), method and settings, input sizes, model and training config,
  validation / test macro-F1
- `norm.npz`: per-input mean and std of that fold's training songs; inputs must be standardized as
  `(x - mean) / std`

Two entries are late fusion (`config.json` only): they average the single-modality models of the
same fold. `index.csv` lists all 75 entries.

## Inputs
- lyrics: 768-d embedding from `lyrics_encoder/theme_tuned/fold_<k>` (the same fold!), L2-normalized
- vocal / instrument / mix: 13 x 768 = 9,984 values (all hidden layers of wav2vec 2.0 on the separated
  voice / MERT on the accompaniment / MERT on the full mix, layer-major)

## Loading
```python
import json, numpy as np, torch, sys
sys.path.insert(0, "code")                     # code/classify.py from this repo
import classify as C
d = "classification/lyrics+instrument+mix/fold_0"
info = json.load(open(f"{d}/config.json")); norm = np.load(f"{d}/norm.npz")
model = C.build_model(info["run"], info["modalities"], info["in_dims"], info["model_config"]).cpu()
model.load_state_dict(torch.load(f"{d}/model.pt", map_location="cpu")); model.eval()
xs = [torch.tensor((X[m] - norm[f"{m}_mean"]) / norm[f"{m}_std"], dtype=torch.float32) for m in info["modalities"]]
proba = C.predict_proba(model, xs)             # (n_songs, 4) in the order of info["themes"]
```

## Mean test macro-F1 over the 5 folds (late-fusion folds excluded)
| combination | macro-F1 | method chosen most |
|---|---|---|
| lyrics+instrument+mix | 0.699 | xattn |
| lyrics+instrument | 0.694 | gmu |
| lyrics+vocal | 0.693 | mlp |
| lyrics+vocal+instrument | 0.692 | mlp |
| lyrics+mix | 0.691 | gmu |
| lyrics+vocal+mix | 0.690 | gmu |
| lyrics+vocal+instrument+mix | 0.690 | mlp |
| lyrics | 0.682 | mlp |
| vocal+mix | 0.513 | gmu |
| vocal+instrument+mix | 0.513 | gmu |
| vocal+instrument | 0.508 | mlp |
| instrument+mix | 0.507 | gmu |
| mix | 0.500 | mlp |
| vocal | 0.493 | xattn |
| instrument | 0.457 | xattn |