maxnowack/outloud-lipreading

Outloud lip reading

The visual speech recogniser of Outloud's experimental lip reading on the Mac: Auto-AVSR's visual model, converted to MLX, with an English and a German decoder. Outloud downloads these files when the user asks for them and runs them on the Mac; each user's own adapters are learned on their Mac and never leave it.

Files

File	What it is
encoder.safetensors	Auto-AVSR's visual front end and Conformer encoder (VSR, LRS3)
decoder-en.safetensors, vocabulary-en.json	Auto-AVSR's English decoder and its SentencePiece vocabulary
decoder-de.safetensors, vocabulary-de.json	A German decoder trained on Auto-AVSR's encoder from 1,852 six-second clips of German videos on Wikimedia Commons, with its vocabulary
SOURCES.md	Every Commons video the German decoder was trained on, with its author and licence

The input is 96 × 96 greyscale crops of the mouth at 25 frames a second.

Licence and attribution

Non-commercial use only.

• The encoder and the English decoder are the weights of Auto-AVSR by Pingchuan Ma, Alexandros Haliassos, Adriana Fernandez-Lopez, Honglie Chen, Stavros Petridis and Maja Pantic. Auto-AVSR's code is under the Apache License 2.0; its weights may be used non-commercially only, under the terms of the datasets they were trained on: LRS2, LRS3, VoxCeleb2 and AVSpeech.
• The German decoder was trained on the encoder above with mouth crops of videos from Wikimedia Commons published under CC BY and CC BY-SA licences. Their titles, authors and licences are listed in SOURCES.md; the videos and crops themselves are not part of this repository.

Ma et al., "Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels", ICASSP 2023.

Accuracy

Experimental. On held-out clips, the German decoder misread 77 % of the words; Outloud adapts the model to each user's lips from their own dictations, on their Mac.