Lip_Wise / notes.md
Stable-Human's picture
Upload folder using huggingface_hub
cd98081 verified
|
Raw History Blame Contribute Delete
1.32 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

:memo: TO-DO List:

PREPROCESS

  • Add directory check in inference in the beginning.
  • Make preprocessing optimal.
  • Clear ram after no_face_filter.
  • Make face coordinates reusable:
    • Saving facial coordinates as .npy file.
    • Alter code to also include eye coordinates.

IMPROVING GAN UPSCALING

  • Merge Data Pipeline with preprocessor:
    • Remove need to recrop, realign and rewarp the image.

IMPROVING WAV2LIP

  • Merge all data Pipeline:
    • Remove the need to recrop, realign, renormalizing etc.
    • Devise a way to keep frames without face in the video.
      • Understand Mels and working of wav2lip model.

OPTIONAL

  • Gradio UI
    • A tab for configuration variables.
    • A tab for Video, Audio and Output.
    • A tab for Image, Audio and output.

URGENT REQUIREMENTS

  • setup.py

    • create venv
    • install requirements inside venv
    • codeformer arch initialization
  • Documentation

FURTHER IMPROVEMENTS

  • Inference without restorer
  • Model Improvement
  • Implement no_face_filter too

FUTURE PLANS

  • Face and Audio wise Lipsync using face recognition.
  • A separate tab for TTS.

COLAB NOTEBOOK

  • Optimize Inference.
  • Implement Checks.