QingCheng24 commited on
Commit
89d5b2b
·
0 Parent(s):

initial commit

Browse files
Files changed (5) hide show
  1. .gitattributes +35 -0
  2. MODEL_CARD.md +68 -0
  3. README.md +68 -0
  4. best_acc.ckpt +3 -0
  5. config.yaml +38 -0
.gitattributes ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
MODEL_CARD.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-4.0
3
+ base_model: MCG-NJU/videomae-base-finetuned-kinetics
4
+ tags:
5
+ - video-classification
6
+ - videomae
7
+ - seamless-interaction
8
+ pipeline_tag: video-classification
9
+ ---
10
+
11
+ # Morph A/B Speaker Classifier (VideoMAE-Base)
12
+
13
+ A VideoMAE-Base video classifier that predicts a speaker's behavioral **"Morph"
14
+ label (Morph A vs. Morph B)** from a short clip of them talking in a two-person
15
+ conversation. Fine-tuned on
16
+ [Meta's Seamless Interaction dataset](https://github.com/facebookresearch/seamless_interaction).
17
+
18
+ **Code / training pipeline:** see the accompanying GitHub repository.
19
+
20
+ ## Files
21
+
22
+ | File | Description |
23
+ |------|-------------|
24
+ | `best_acc.ckpt` | PyTorch-Lightning checkpoint (~1 GB), best `val/acc` |
25
+ | `config.yaml` | exact training config for this run |
26
+ | `MODEL_CARD.md` | this card |
27
+
28
+ ## Model
29
+
30
+ - **Base model:** `MCG-NJU/videomae-base-finetuned-kinetics` (~86 M params), loaded
31
+ via `transformers.VideoMAEForVideoClassification`. The K400 400-way head is
32
+ replaced with a 2-way (Morph A / Morph B) head; the backbone is fine-tuned.
33
+ - **Input:** 64-frame clips @ 4 fps, short-side 224 → top-cropped to 224×224,
34
+ ImageNet mean/std normalized. VideoMAE's sinusoidal temporal position
35
+ embeddings scale from the pretrained 16 frames to 64 with no interpolation.
36
+
37
+ ## Training
38
+
39
+ - **Data:** ~2,462 per-speaker clips (1,970 train / 492 test), classes ~62% A / 38% B.
40
+ - **Split:** random **per-clip** 80/20, stratified by class.
41
+ - **Optimizer:** AdamW, head lr 1e-4 / backbone lr 1e-5 (×0.1), weight decay 0.05.
42
+ - **Schedule:** 10-epoch warmup + cosine decay to 1e-6; early stopping on `val/acc`.
43
+ - **Loss:** class-weighted cross-entropy.
44
+
45
+ ## Results
46
+
47
+ | Metric | Value |
48
+ |--------|------:|
49
+ | Val accuracy | ~99.6% |
50
+ | Val F1 | ~99.5% |
51
+
52
+ ## Intended use
53
+
54
+ Research baseline for whether the behavioral "Morph" distinction is decodable
55
+ from raw video.
56
+
57
+ ## Usage
58
+
59
+ ```bash
60
+ huggingface-cli download QingCheng24/seamless_morph --local-dir weights
61
+
62
+ python train_videomae.py --test --ckpt_path weights/best_acc.ckpt
63
+ ```
64
+
65
+ ## License
66
+
67
+ Released under **CC BY-NC 4.0**, inheriting the non-commercial terms of the
68
+ Seamless Interaction dataset and the Morph A/B annotations it was trained on.
README.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-4.0
3
+ base_model: MCG-NJU/videomae-base-finetuned-kinetics
4
+ tags:
5
+ - video-classification
6
+ - videomae
7
+ - seamless-interaction
8
+ pipeline_tag: video-classification
9
+ ---
10
+
11
+ # Morph A/B Speaker Classifier (VideoMAE-Base)
12
+
13
+ A VideoMAE-Base video classifier that predicts a speaker's behavioral **"Morph"
14
+ label (Morph A vs. Morph B)** from a short clip of them talking in a two-person
15
+ conversation. Fine-tuned on
16
+ [Meta's Seamless Interaction dataset](https://github.com/facebookresearch/seamless_interaction).
17
+
18
+ **Code / training pipeline:** see the accompanying GitHub repository.
19
+
20
+ ## Files
21
+
22
+ | File | Description |
23
+ |------|-------------|
24
+ | `best_acc.ckpt` | PyTorch-Lightning checkpoint (~1 GB), best `val/acc` |
25
+ | `config.yaml` | exact training config for this run |
26
+ | `MODEL_CARD.md` | this card |
27
+
28
+ ## Model
29
+
30
+ - **Base model:** `MCG-NJU/videomae-base-finetuned-kinetics` (~86 M params), loaded
31
+ via `transformers.VideoMAEForVideoClassification`. The K400 400-way head is
32
+ replaced with a 2-way (Morph A / Morph B) head; the backbone is fine-tuned.
33
+ - **Input:** 64-frame clips @ 4 fps, short-side 224 → top-cropped to 224×224,
34
+ ImageNet mean/std normalized. VideoMAE's sinusoidal temporal position
35
+ embeddings scale from the pretrained 16 frames to 64 with no interpolation.
36
+
37
+ ## Training
38
+
39
+ - **Data:** ~2,462 per-speaker clips (1,970 train / 492 test), classes ~62% A / 38% B.
40
+ - **Split:** random **per-clip** 80/20, stratified by class.
41
+ - **Optimizer:** AdamW, head lr 1e-4 / backbone lr 1e-5 (×0.1), weight decay 0.05.
42
+ - **Schedule:** 10-epoch warmup + cosine decay to 1e-6; early stopping on `val/acc`.
43
+ - **Loss:** class-weighted cross-entropy.
44
+
45
+ ## Results
46
+
47
+ | Metric | Value |
48
+ |--------|------:|
49
+ | Val accuracy | ~99.6% |
50
+ | Val F1 | ~99.5% |
51
+
52
+ ## Intended use
53
+
54
+ Research baseline for whether the behavioral "Morph" distinction is decodable
55
+ from raw video.
56
+
57
+ ## Usage
58
+
59
+ ```bash
60
+ huggingface-cli download QingCheng24/seamless_morph --local-dir weights
61
+
62
+ python train_videomae.py --test --ckpt_path weights/best_acc.ckpt
63
+ ```
64
+
65
+ ## License
66
+
67
+ Released under **CC BY-NC 4.0**, inheriting the non-commercial terms of the
68
+ Seamless Interaction dataset and the Morph A/B annotations it was trained on.
best_acc.ckpt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1df6dc3c0873738f23764872de0402601bfd8920eb5dd6b8a9fcf4c6e1ed90d8
3
+ size 1034986089
config.yaml ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ data:
2
+ dataloader:
3
+ batch_size: 16
4
+ eval_batch_size: 32
5
+ num_workers: 2
6
+ persistent_workers: true
7
+ pin_memory: false
8
+ prefetch_factor: 2
9
+ seed: 42
10
+ dataset:
11
+ clips_dir: /data/qing/seamless/clips/n64_fps4_crop224_short224
12
+ hflip_prob: 0.5
13
+ image_size: 224
14
+ mean:
15
+ - 0.485
16
+ - 0.456
17
+ - 0.406
18
+ num_frames: 64
19
+ overfit: false
20
+ std:
21
+ - 0.229
22
+ - 0.224
23
+ - 0.225
24
+ test_txt: /home/qing/mesh3d/gen3d/seamless_interaction/dataset/test.txt
25
+ train_txt: /home/qing/mesh3d/gen3d/seamless_interaction/dataset/train.txt
26
+ model:
27
+ hf_model_path: /data/qing/seamless/hf_cache/videomae-base-finetuned-kinetics
28
+ image_size: 224
29
+ num_frames: 64
30
+ training:
31
+ common:
32
+ end_lr: 1.0e-06
33
+ epochs: 200
34
+ gradient_checkpointing: false
35
+ lr: 0.0001
36
+ lr_backbone_mult: 0.1
37
+ lr_warmup_epochs: 10
38
+ weight_decay: 0.05