README.md

Wav2Lip browser/static prototype

This project is the first step toward:

Hugging Face static page → browser video/audio upload → ONNX Wav2Lip inference in browser → MP4 generated locally

What this prototype proves

• No Python backend.
• No Hugging Face inference backend.
• onnxruntime-web executes the Wav2Lip ONNX model in the browser.
• Video/audio are processed locally.
• The preprocessing follows the supplied notebook's important values:
  • 16 kHz audio
  • 800 FFT
  • 200 hop
  • 80 mel bins
  • 55–7600 Hz
  • 16 mel columns per Wav2Lip input chunk
  • 6-channel masked+original 96×96 face input
• The notebook's ONNX input names are used: mel_spectrogram, video_frames
• The output name is predicted_frames.

Current limitation

This is deliberately a browser proof-of-concept, not yet the final one-click MP4 application.

Two parts still need production hardening:

1. The JavaScript mel implementation should be replaced by a tested FFT/mel implementation that is numerically matched to the exact Wav2Lip/librosa preprocessing.
2. The generated frames need to be encoded and muxed with the target audio. The recommended browser path is WebCodecs + an MP4 muxer, with ffmpeg.wasm as fallback.

Model

For the first browser test, use the wav2lip_gan.onnx model from the supplied notebook/model package.

Do NOT put a model URL in the page until you have uploaded the ONNX model to a location that permits browser CORS requests.

Face crop

The supplied notebook uses a fixed face box:

x=1740, y=294, w=998, h=998

This prototype exposes those values as editable fields.

A production version should add automatic browser face detection/tracking.

Hugging Face Static deployment

Upload these files to a Hugging Face Space configured for a static site:

• index.html
• app.js
• README.md

The ONNX model can be stored separately or in the same Space using Git LFS, depending on the repository/storage limits you have.

Privacy

When the full pipeline is browser-only, source video and target audio do not need to be uploaded to an inference server.