zenonlee commited on
Commit
af4cf2b
·
verified ·
1 Parent(s): e8ebfc2

Add model card and teaser image

Browse files
Files changed (3) hide show
  1. .gitattributes +1 -0
  2. README.md +73 -0
  3. teaser.jpg +3 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ teaser.jpg filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,73 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: research-only
4
+ license_link: https://github.com/zenonLEE/MDFLD#terms-of-use
5
+ library_name: pytorch
6
+ pipeline_tag: keypoint-detection
7
+ tags:
8
+ - face-landmark-detection
9
+ - facial-landmarks
10
+ - 68-landmarks
11
+ - cartoon
12
+ - caricature
13
+ - eyelid
14
+ - torchscript
15
+ ---
16
+
17
+ # MDFLD v2: multi-domain face landmarks (68 points)
18
+
19
+ Model weights for **Towards Multi-Domain Face Landmark Detection with Synthetic Data from Diffusion Model** (Yuanming Li, Gwantae Kim, Jeong-gi Kwak, Bon-hwa Ku, Hanseok Ko; ICASSP 2024).
20
+
21
+ [Paper (arXiv:2401.13191)](https://arxiv.org/abs/2401.13191) · [Code](https://github.com/zenonLEE/MDFLD) · [Project page](https://zenonlee.github.io/MDFLD/)
22
+
23
+ ![68 landmarks and eyelid curves on a photo, paintings, a caricature and 3D animation](teaser.jpg)
24
+
25
+ This is the cartoon model v2: the paper model fine-tuned with self-training on cartoon and caricature faces. On the CariFace caricature test set (1,560 caricatures) it lowers the 68-point NME from 6.70% to 5.03% (eyes 3.49% → 2.78%, jaw 10.42% → 7.75%) with the package's three-pass crop; on held-out real-face video frames it reaches 1.74% (v1: 3.03%). See the [GitHub README](https://github.com/zenonLEE/MDFLD#cartoon-model-v2) for the full table, the training data and the limitations.
26
+
27
+ ## Use
28
+
29
+ The easiest way is the `mdfld` package, which downloads this model, finds the face and adds eyelid curves:
30
+
31
+ ```bash
32
+ pip install "mdfld[sam] @ git+https://github.com/zenonLEE/MDFLD"
33
+ mdfld image face.png --out face.json --vis face_mdfld.png
34
+ ```
35
+
36
+ The file `mdfld_cartoon_v2.torchscript.pt` can also be used directly with PyTorch, without any code from the repository:
37
+
38
+ ```python
39
+ import torch
40
+ net = torch.jit.load("mdfld_cartoon_v2.torchscript.pt", map_location="cuda")
41
+ # x: (N, 3, 256, 256) float tensor of BGR face crops, values pixel / 255 * 2 - 1
42
+ out = net(x) # (N, 68, 2), x and y in [-1, 1]
43
+ pts = ((out + 1) * 256 - 1) / 2 # pixels of the 256 x 256 crop
44
+ ```
45
+
46
+ The crop is a square centred on the face box with side equal to the larger side of the box, resized to 256 × 256. For the best accuracy, run two more passes, each on a square of 1.1 × the larger side of the box of the previous points, as the package does (`mdfld/landmarks.py`). The 68 points follow the iBUG 300-W order (0-based).
47
+
48
+ ## Files
49
+
50
+ | File | Use |
51
+ |---|---|
52
+ | `mdfld_cartoon_v2.torchscript.pt` | TorchScript model, used by the `mdfld` package |
53
+
54
+ The same weights in the original checkpoint format (`mdfld_cartoon_v2.pkl`, for the STAR-based code in the repository) are in the [v2.0 release](https://github.com/zenonLEE/MDFLD/releases/tag/v2.0).
55
+
56
+ ## Terms of use
57
+
58
+ Research use only, without warranty. The model was trained on third-party data with their own terms (CariFace is for research only; several cartoon sets contain third-party artwork). See the [terms of use](https://github.com/zenonLEE/MDFLD#terms-of-use).
59
+
60
+ ## Citation
61
+
62
+ ```bibtex
63
+ @inproceedings{li2024towards,
64
+ title={Towards Multi-Domain Face Landmark Detection with Synthetic Data from Diffusion Model},
65
+ author={Li, Yuanming and Kim, Gwantae and Kwak, Jeong-gi and Ku, Bon-hwa and Ko, Hanseok},
66
+ booktitle={ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
67
+ pages={6730--6734},
68
+ year={2024},
69
+ organization={IEEE}
70
+ }
71
+ ```
72
+
73
+ Teaser images: public-domain works from Wikimedia Commons and frames of *Sintel* (© copyright Blender Foundation | durian.blender.org, CC BY 3.0).
teaser.jpg ADDED

Git LFS Details

  • SHA256: 789ac9954848b15c4694e03902ec0aa35f93141a0515aa8b190a6d560185e803
  • Pointer size: 131 Bytes
  • Size of remote file: 384 kB