|
Download README.md from QingCheng24/seamless_morph: direct link, hf CLI and curl.
- Browser
- Download file 2.2 kB
-
https://huggingface.co/QingCheng24/seamless_morph/resolve/main/README.md
- Command line
-
hf download hf://QingCheng24/seamless_morph/README.md
-
curl -L -o README.md https://huggingface.co/QingCheng24/seamless_morph/resolve/main/README.md
2.2 kB
| license: cc-by-nc-4.0 | |
| base_model: MCG-NJU/videomae-base-finetuned-kinetics | |
| tags: | |
| - video-classification | |
| - videomae | |
| - seamless-interaction | |
| pipeline_tag: video-classification | |
| # Morph A/B Speaker Classifier (VideoMAE-Base) | |
| A VideoMAE-Base video classifier that predicts a speaker's behavioral **"Morph" | |
| label (Morph A vs. Morph B)** from a short clip of them talking in a two-person | |
| conversation. Fine-tuned on | |
| [Meta's Seamless Interaction dataset](https://github.com/facebookresearch/seamless_interaction). | |
| **Code / training pipeline:** see the accompanying GitHub repository. | |
| ## Files | |
| | File | Description | | |
| |------|-------------| | |
| | `best_acc.ckpt` | PyTorch-Lightning checkpoint (~1 GB), best `val/acc` | | |
| | `config.yaml` | exact training config for this run | | |
| | `MODEL_CARD.md` | this card | | |
| ## Model | |
| - **Base model:** `MCG-NJU/videomae-base-finetuned-kinetics` (~86 M params), loaded | |
| via `transformers.VideoMAEForVideoClassification`. The K400 400-way head is | |
| replaced with a 2-way (Morph A / Morph B) head; the backbone is fine-tuned. | |
| - **Input:** 64-frame clips @ 4 fps, short-side 224 → top-cropped to 224×224, | |
| ImageNet mean/std normalized. VideoMAE's sinusoidal temporal position | |
| embeddings scale from the pretrained 16 frames to 64 with no interpolation. | |
| ## Training | |
| - **Data:** ~2,462 per-speaker clips (1,970 train / 492 test), classes ~62% A / 38% B. | |
| - **Split:** random **per-clip** 80/20, stratified by class. | |
| - **Optimizer:** AdamW, head lr 1e-4 / backbone lr 1e-5 (×0.1), weight decay 0.05. | |
| - **Schedule:** 10-epoch warmup + cosine decay to 1e-6; early stopping on `val/acc`. | |
| - **Loss:** class-weighted cross-entropy. | |
| ## Results | |
| | Metric | Value | | |
| |--------|------:| | |
| | Val accuracy | ~99.6% | | |
| | Val F1 | ~99.5% | | |
| ## Intended use | |
| Research baseline for whether the behavioral "Morph" distinction is decodable | |
| from raw video. | |
| ## Usage | |
| ```bash | |
| huggingface-cli download QingCheng24/seamless_morph --local-dir weights | |
| python train_videomae.py --test --ckpt_path weights/best_acc.ckpt | |
| ``` | |
| ## License | |
| Released under **CC BY-NC 4.0**, inheriting the non-commercial terms of the | |
| Seamless Interaction dataset and the Morph A/B annotations it was trained on. | |