| # Wav2Lip browser/static prototype |
|
|
| This project is the first step toward: |
|
|
| Hugging Face static page |
| → browser video/audio upload |
| → ONNX Wav2Lip inference in browser |
| → MP4 generated locally |
|
|
| ## What this prototype proves |
|
|
| - No Python backend. |
| - No Hugging Face inference backend. |
| - `onnxruntime-web` executes the Wav2Lip ONNX model in the browser. |
| - Video/audio are processed locally. |
| - The preprocessing follows the supplied notebook's important values: |
| - 16 kHz audio |
| - 800 FFT |
| - 200 hop |
| - 80 mel bins |
| - 55–7600 Hz |
| - 16 mel columns per Wav2Lip input chunk |
| - 6-channel masked+original 96×96 face input |
| - The notebook's ONNX input names are used: |
| `mel_spectrogram`, `video_frames` |
| - The output name is `predicted_frames`. |
|
|
| ## Current limitation |
|
|
| This is deliberately a browser proof-of-concept, not yet the final one-click MP4 application. |
|
|
| Two parts still need production hardening: |
|
|
| 1. The JavaScript mel implementation should be replaced by a tested FFT/mel implementation that is numerically matched to the exact Wav2Lip/librosa preprocessing. |
| 2. The generated frames need to be encoded and muxed with the target audio. The recommended browser path is WebCodecs + an MP4 muxer, with ffmpeg.wasm as fallback. |
|
|
| ## Model |
|
|
| For the first browser test, use the `wav2lip_gan.onnx` model from the supplied notebook/model package. |
|
|
| Do NOT put a model URL in the page until you have uploaded the ONNX model to a location that permits browser CORS requests. |
|
|
| ## Face crop |
|
|
| The supplied notebook uses a fixed face box: |
|
|
| x=1740, y=294, w=998, h=998 |
|
|
| This prototype exposes those values as editable fields. |
|
|
| A production version should add automatic browser face detection/tracking. |
|
|
| ## Hugging Face Static deployment |
|
|
| Upload these files to a Hugging Face Space configured for a static site: |
|
|
| - index.html |
| - app.js |
| - README.md |
|
|
| The ONNX model can be stored separately or in the same Space using Git LFS, depending on the repository/storage limits you have. |
|
|
| ## Privacy |
|
|
| When the full pipeline is browser-only, source video and target audio do not need to be uploaded to an inference server. |
|
|
