ebowwa/silentvsr-models

SilentVSR Core ML models

Visual-speech (lip-reading) models for the SilentVSR macOS/iOS client. This is a public, gated release — the model card is visible to everyone, but weight downloads (base and personalization) require an approved access request.

• base/ — the general (speaker-independent) Core ML models the app bundles.
• personalization/ — an experimental, speaker-specific rank-8 LoRA personalization (the ANEtest test variant), included to demonstrate that on-device personalization works end-to-end. It is one individual's visual-speech data, which is why the repository is gated rather than open.

This is a derivative/mirror, not an independently trained model. See Provenance.

---

Provenance

This model is a clone/derivative of AD1TEYA/lip-reading-model, which is itself a re-host of the official Auto-AVSR visual-speech checkpoint. The weights are byte-for-byte identical to the upstream lineage (verified by SHA-256; see fingerprints below). Nothing here was trained from scratch by the maintainer of this repo.

Lineage:

Auto-AVSR  (Pingchuan Ma et al., github.com/mpc001/auto_avsr)
   │  vsr_trlrs2lrs3vox2avsp_base.pth   (3,291 h, ~20.3% LRS3 WER, ~250M params)
   ▼
AD1TEYA/lip-reading-model        (HF re-host; NO model card, NO license, NO eval)
   │  pytorch_model.pt
   ▼
aaahmet/silent-lip-reader-model  (HF mirror; MIT; "not trained by the re-host";
   │                              "mirrored from AD1TEYA/lip-reading-model")
   │  revision ff1e35515594eb4562c469b99d147cf6a3d2891f, pytorch_model.pt
   ▼
Open-Altergo pinned checkpoint   (byte-identical; fingerprinted 2026-07-28)
   │  SHA-256 fbf7cd70ff1c0e694b3030fb779dbb4570f04e4b841d62f9296c229e94878ddb
   ▼
SilentVSRCTC  (coremltools conversion → Core ML, cloud/modal/convert_coreml.py)
   │  base/SilentVSRCTC.mlpackage (FP32, flexible) +
   │  base/SilentVSRCTC_fixed105_fp16.mlpackage (FP16, fixed-105)
   ▼
personalization/  (speaker-specific rank-8 LoRA fine-tune on top of the base)

License: mit — following the documented aaahmet/silent-lip-reader-model card and the upstream Auto-AVSR code license. ⚠️ Caveat: AD1TEYA/lip-reading-model itself documents no license and no evaluation, and the exact upstream training split / checkpoint-averaging recipe is not fully established. Treat the MIT tagging as a best-effort inference from the documented mirror; confirm upstream terms before any public redistribution.

Snapshot note: The base weights in this repo are a snapshot of the mirror above, fingerprint-verified on 2026-07-28 against the official Google-Drive model-zoo file.

---

Repository contents

Path	Size	What it is
base/SilentVSRCTC.mlpackage	~388 MB	Base model, FP32, flexible-length (8–250 frames). The working baseline.
base/SilentVSRCTC_fixed105_fp16.mlpackage	~362 MB	Base model, FP16, fixed 105-frame window — a Neural-Engine placement candidate.
personalization/SilentVSRFrontend.mlmodelc	~354 MB	Personalized visual frontend/encoder (compiled Core ML).
personalization/SilentVSRCTCProjection.mlmodelc	~8 MB	Personalized encoder→decoder projection head.
personalization/SilentVSRCTCProjectionPersonalizedANEtest.mlmodelc	~8 MB	Pre-merged static personalized projection (ANE-targeted test variant).
personalization/adapter-a-rank8.f32le	24 KB	Rank-8 LoRA adapter A (raw float32 little-endian).
personalization/adapter-b-rank8-tiled.f32le	160 KB	Rank-8 LoRA adapter B, tiled.

Base artifacts are .mlpackage (source); personalization artifacts are compiled .mlmodelc bundles. Integrity is content-pinned by SHA-256 in the app manifest, not by filename.

---

Architecture (Auto-AVSR, visual-only, ~250M params)

Stage	Configuration
3D frontend	Conv3D 1→64, kernel 5×7×7, stride 1×2×2
Spatial trunk	ResNet-18-style [2,2,2,2], widths 64/128/256/512
Visual frontend output	[B, T, 512] per frame
Encoder projection	Linear 512→768
Encoder	Conformer, 12 blocks, 12 heads, FFN 3072, conv kernel 31
CTC head	5,049 output classes
Decoder	Transformer, 6 blocks, width 768, 12 heads, FFN 3072
Objective	0.1·CTC + 0.9·attention (label smoothing 0.1)
Modality	Video-only (no audio path)

One encoder feature vector is produced per input frame (temporal stride 1 in the frontend). T is dynamic in the source checkpoint; the fixed 105-frame window is a deployment choice, not a model constraint.

---

Core ML I/O contract

	Flexible FP32	Fixed-105 FP16
Input name	input	input
Input shape	[-1, -1, 1, 88, 88] (flex. T)	[1, 105, 1, 88, 88]
Input dtype	float32	float16
Output name	var_2595	var_2595
Output shape	[-1, -1, -1]	[-1, -1, -1]
Frames	8–250	105
Deployment target	macOS 14+ / iOS 17+	iOS 18+ / macOS 15+ (NE candidate)

• Frame rate: 25 FPS · window: 105 frames · stride: 90 frames
• Decoding: CTC prefix-beam search, beam width 20, blank token ID 0
• Vocabulary (vocab.json) SHA-256: 076b4f80e4ce1a14f18c94d6185f2688c0b1819b434301d5847b6fe345270bcd
• Artifact SHA-256: flexible-FP32 8a52b695…727f5 · fixed-105-FP16 f142eadc…e7276

---

Preprocessing contract

1. Decode video, normalize to exactly 25 FPS.
2. Face/landmark detection (MediaPipe/RetinaFace), affine stabilization.
3. Extract 96×96 RGB mouth-centered crop per frame (intermediate).
4. Random/center crop 96→88, RGB→1 grayscale channel.
5. Normalize: mean 0.421, std 0.165 (after /255).
6. Network tensor: [B, T, 1, 88, 88] float32 (frontend transposes to [B, 1, T, 88, 88] internally).

---

Vocabulary

Immutable SentencePiece unigram-5000 tokenizer that must travel with every compatible model:

ID 0       = <blank>  (CTC)
IDs 1–5047 = SentencePiece units
ID 5048    = shared <sos>/<eos>
odim       = 5049
ignore ID  = -1

Transcripts are uppercased before tokenization. Do not swap the tokenizer, unit file, ID ordering, blank ID, or EOS ID — weights will still load but decode to garbage.

---

Personalization

personalization/ is a speaker-specific adaptation of the base model for a single speaker — it encodes that speaker's mouth geometry, articulation, and capture conditions. This upload is the experimental ANEtest test variant, published (gated) to demonstrate the on-device personalization path; it is not claimed to be a production-quality personal model.

• Method: rank-8 LoRA-style fine-tune, default encoder-decoder strategy (last 2 of 12 Conformer blocks + encoder projection + Transformer decoder + CTC head trainable; visual frontend frozen).
• Runtime paths: the app applies personalization either as a pre-merged static projection (SilentVSRCTCProjectionPersonalizedANEtest.mlmodelc, coremlStatic) or by merging the rank-8 LoRA adapters at inference via Accelerate (swiftAccelerate). Adapter ID 7875BF49-…-rank8-v1, rank 8.
• Personalization artifacts retain their original internal Core ML metadata (generatedClassName, sqeakd.personalization=lora, sqeakd.personalization.rank=8); only the repository filenames were renamed to the SilentVSR* scheme.

Training/conversion references: training/ and pretraining/auto-avsr-checkpoints/ in the source project.

---

Performance

No new benchmark was produced for this snapshot. Reported numbers are upstream and informal:

• Official Auto-AVSR vsr_trlrs2lrs3vox2avsp_base.pth: ~20.3% WER on LRS3 (clean benchmark, upstream evaluation pipeline).
• aaahmet/silent-lip-reader-model card claims ~25–30% WER on clean, frontal, deliberately-articulated English — informal, no evaluation table.
• These are not expected WERs for webcam video, silent mouthing, a single personalized speaker, mobile capture, streaming windows, or out-of-domain language.
• No personalization WER has been established yet.

---

Intended use & limitations

• Intended: on-device visual-speech transcription for a cooperative, front-facing speaker in the SilentVSR client; research/personal use.
• Not intended: surveillance, covert capture, speaker identification across unknown subjects, or any deployment without consent.
• Limitations: lip-reading is inherently ambiguous (many phonemes are visually identical); accuracy drops sharply with head angle, lighting, distance, and casual/ silent speech.

---

Citation

• Auto-AVSR — Pingchuan Ma et al., "Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels." Code: https://github.com/mpc001/auto_avsr
• AD1TEYA/lip-reading-model — https://huggingface.co/AD1TEYA/lip-reading-model (re-host; undocumented)
• aaahmet/silent-lip-reader-model — Ahmet Dedeler, https://huggingface.co/aaahmet/silent-lip-reader-model (MIT mirror; documented)

---

Disclaimer

This is a derivative re-host/conversion, not an independently trained model. The base weights are byte-identical to the Auto-AVSR / AD1TEYA / aaahmet lineage. The repository is gated — vet access requests carefully, since personalization/ is speaker-specific (one individual's visual-speech data). Confirm upstream licenses before any further redistribution.