OmniVLM
One model that sees, answers, speaks β and moves its lips.
One trunk. One token space. Every modality is a head on it.
Trained from random initialisation on a single RTX PRO 6000 β no pretrained vision backbone, no pretrained TTS, no HiFi-GAN, no external forced aligner. The same weights read an image, choose the words, time the phonemes, synthesise the waveform and drive a face, so the face cannot disagree with the voice it came from.
Status and roadmap
| stage | what it does | state |
|---|---|---|
| vlm | eyes + trunk, from random weights β the only true pretraining | done, 12 000 steps (150% of plan), loss 0.062 |
| ear | audio β phonemes (CTC); also produces the alignments | running, 63% β ctc 0.0067 |
| speech | voice: phonemes, durations, mel, on the same trunk | running, 54% |
| vocoder | mel β waveform, from scratch | running, 72% β stft 0.866 |
| captions | real photographs + human captions on that trunk | running, 5% β needs its budget re-thought |
| align | forced alignment of the speech corpus | complete, 13 100 clips |
299 distinct checkpoints are archived on the Hub, across 5 live stages, from 873 000 trained steps. Every stage checkpoints to the Hub and resumes exactly; the sandbox is ephemeral by design, not by accident.
Try it
pip install torch numpy pillow scipy huggingface_hub && apt-get install -y ffmpeg
git clone https://huggingface.co/hkblockchaintechnology/VLM-all-inclusive
python scripts/make_movie.py --question "describe the image"
One image and one question in; an mp4 with the answer spoken and the mouth moving comes out β and the published samples carry their own audio track, so picture and voice stay in sync when you play them.
What it is
Text-to-image, text-to-speech and text-to-video are separate models that each re-learn language. Here there is one token space and one trunk: the same weights that read an image and answer a question also drive the speech heads and the lips. The speech stages are trained multi-task with the vision-language task, so the trunk keeps both abilities instead of being forked into per-task copies.
| part | implementation | trained here |
|---|---|---|
| eyes | ViT patch encoder | yes β on procedurally rendered scenes |
| brain | causal trunk, cross-attends the vision tokens | yes |
| text | word head: question β answer | yes |
| ear | CTC phoneme recogniser over log-mel, and it produces the alignments too | yes β on the public-domain LJSpeech corpus |
| voice | phoneme + duration heads + FastSpeech-style mel decoder, on the same trunk | yes |
| sound | Vocos-style mel β waveform | yes β multi-resolution STFT loss, no HiFi-GAN |
| lips | ARPAbet β viseme parameters β a photoreal face whose mouth is composited from a generated shape per viseme, cross-faded between | the shapes were generated once; the timing that selects them is the model's own phoneme timeline |
Everything enters and leaves through one trunk. The lips read the same phoneme timeline that produced the audio, so the face cannot disagree with the voice.
The ten viseme mouth shapes, and the mouth saying one real sentence in real time.
From scratch β and you can check it
Every weight here was initialised from random numbers and trained on this machine. There is no pretrained backbone underneath, and no borrowed network anywhere in the pipeline: the vision encoder, the trunk, the phoneme recogniser, the duration and mel heads and the vocoder are all created empty and taught from the corpora. That is a rare thing to be able to say, so it should not be taken on trust β it is a claim you can falsify in one command:
grep -rEn 'from_pretrained|torch[.]hub|(from|import) (transformers|timm|torchvision)|whisper|wav2vec|hifigan' omni/ notebook.py
# no matches
# no matches
The complete list of things this project downloads from the internet is three datasets and one dictionary β no models:
| downloaded | what it is | licence |
|---|---|---|
| LJSpeech-1.1 | 24 h of read speech, one speaker β the audio corpus | public domain |
| flickr30k | 31 783 photographs, 5 human captions each | CC-BY 4.0 |
| CMUdict | an ARPAbet pronunciation dictionary, 135 166 entries: text in, phonemes out | BSD-2 |
| β | the scene images are not downloaded: omni/scenes.py generates them |
β |
Two footnotes, because "from scratch" should not be allowed to do quiet work:
- CMUdict is a lexicon, not a model. It is a list of how English words are pronounced. A
word outside it falls back to ten hand-written letter-to-sound mappings in
omni/text.py(a -> AH,x -> K S, ...) β not a learned grapheme-to-phoneme network. - The viseme table is hand-authored, not learned.
omni/avatar.pymaps 39 ARPAbet phones onto 15 mouth shapes using numbers a person chose. What the model contributes is the timing β its own phoneme sequence and its own durations β and what the renderer does with that is arithmetic and a rasteriser.
So the honest summary is: a new architecture, new weights, trained from random initialisation on public data, with the whole pipeline published. If a future version of this repository ever loads a pretrained component, the grep above will stop returning nothing. That is the point of stating it as a check rather than as a claim.
Why an all-inclusive model β the ground for OmniVLM
Today's multimodal models are, almost always, one output modality per model. Look at what
actually ships on the Hub β sampled by pipeline tag, first 1 000 results each, via
https://huggingface.co/api/models?filter=<tag>, 17 Nov 2025:
| pipeline tag | models (sampled) | what it gives you |
|---|---|---|
image-text-to-text |
>1 000 | sees, answers in text β no sound, no face |
text-to-speech |
>1 000 | says it β no face, no visual grounding |
text-to-video / image-to-video |
>1 000 each | moves pixels β the voice is bolted on afterwards, with no shared representation |
any-to-any |
>1 000 | the all-inclusive bucket β but it stops at audio out (e.g. Qwen/Qwen2.5-Omni-7B, openbmb/MiniCPM-o-2_6); no rendered, lip-synced face |
audio-to-audio, text-to-audio |
>1 000 each | voice/audio-only stacks |
And lip sync, the piece that makes any of it look alive, is a niche of ~200 repos outside
every mainstream pipeline: lipsync 40, wav2lip 54, sadtalker 58, musetalk 51,
talking-head 15 β there is no Hub pipeline tag for it at all, and nearly all of those are
weights for a handful of upstream projects that each want a pre-existing video plus a
pre-existing audio track.
So the field's default shape is three half-measures:
- video with no sound β a diffusion video model; you add TTS later, and the mouth was never conditioned on the words it is now speaking;
- sound with no video β an excellent TTS model; no face, and no grounding in anything visual;
- TTS with no lip sync β or lips driven by a second, separately trained model that has never seen the first one's representation of language.
Each seam is a place where the modalities disagree. Bolting them together cannot fix that, because there is no shared token space in which "the word I am saying", "the sound I am making" and "the shape my mouth is in" are the same object.
That is the ground OmniVLM stands on. Here the modalities are not stitched: there is one
token space (words and ARPAbet phones in one vocabulary) and one causal trunk whose heads
are the modalities β vision tokens cross-attended for grounding, a word head, a phoneme head,
a duration head and a mel decoder on the same hidden state, with the speech stages trained
multi-task alongside the vision-language task so neither ability is forked off. The lips are
therefore driven by the model's own phoneme timeline β the same timeline that produced the
waveform β instead of by a separate lip-sync network. And unlike the any-to-any giants, the
whole thing is trained from scratch on one GPU with published checkpoints, a public held-out
metric (BLEU-4 on flickr30k val) and scripts that reproduce it.
Stated fairly: the claim is not that no multimodal model exists β Gemini/Qwen-Omni-class
systems are more capable than this, and any-to-any is a real, thriving category. The claim is
narrower and checkable: no published artefact trains eyes, language, voice, sound and lips
jointly, from scratch, and ships the whole pipeline β weights, eval, resume policy β as one
thing you can download and retrain. That is the gap this project exists to fill, and it is why
the quality caveats below are the honest frontier rather than a hedge.
What this is for
The unusual property here -- one token space, one trunk, every modality trained together -- is exactly what an agent needs, because an agent has to see, decide, speak and be seen doing it in the same moment, not in a pipeline of separate models:
| where it applies | why this shape fits |
|---|---|
| computer-use agents | the agent reads a screen, acts, and narrates what it is doing. Because the seeing and the saying come from one forward pass over one shared representation, the narration cannot drift away from the action it is describing. |
| phone / on-device agents | one small trunk plus three tiny heads replaces a separate vision model, a separate ASR, a separate TTS and a separate lip-sync stack. Fewer artefacts to ship, one thing to keep aligned, and the whole assistant can live on the device instead of making a cloud round trip per modality. |
| robotics and embodied agents | a robot that perceives, answers and speaks from one set of weights is a robot whose words match its perception. And because the mouth is driven by the same phoneme timeline that produced the audio, the face and the voice are generated together rather than dubbed over one another. |
The field is full of models that are brilliant at one channel and hand the rest to a different model with a different vocabulary, a different training run and a different clock. Every seam between them is latency, and every seam is somewhere the channels can disagree.
Emotion is a property of simultaneity, not a fourth model
Feeling is not a separate head bolted on at the end; it is what the coincidence of the channels conveys. A mouth one phoneme behind its voice reads as uncanny. A voice that says one thing while the eyes and brow say another reads as insincere. For a face to move a person, the timing has to be real -- one decision, at one instant, driving the word, the sound and the movement.
That is what training the modalities together buys: the timings that carry feeling are computed from a single hidden state rather than reconstructed afterwards by an integrator guessing at alignment. It is what lets a delivery carry not only knowledge and a machine voice, but something that reads as intention and warmth -- smooth, aligned and coherent from the inside out, whether it arrives through a desktop agent, a phone, or a robot standing in front of you.
Stated honestly: on this checkpoint the claim is architectural, not measured. These weights were trained on procedurally rendered shapes and one speaker's read speech, so what ships today is a small research artefact. The point of the table above is that the architecture has somewhere to go that a stitched pipeline cannot follow -- and the checkpoints, the eval and the resume policy are published so that someone can take it there.
Questions we expect β and honest answers
Full list: 100 rehearsed challenges with their evidence status.
The twelve that will be asked first
"Your voice is gibberish." β Yes. An independent Whisper transcribes the synthesised speech at mean WER 0.97, and each sentence runs 15 s where 2 s is expected: the decoder loops instead of stopping. The cause is teacher forcing.
"But your metrics said phone_acc 1.000." β And that is the lesson. The phoneme head is
trained to predict each phone given the true previous phone, so it never learns to run on its
own output. A teacher-forced loss is not evidence about generation. The card reports the WER
because the loss could not have told you.
"More training made it worse?" β Yes, and it is measured: VQA exact match 47.0% at step 39 002 versus 35.5% at 49 502. The speech stage fine-tunes the whole trunk on a speech objective, which erodes the vision-language ability. Serving is pinned to the better checkpoint.
"From scratch β really?" β Check it. A grep for from_pretrained, torch.hub,
transformers, timm, torchvision, whisper, wav2vec and hifigan across all 20 source
files returns nothing. Three datasets and one pronunciation dictionary are the only downloads.
"The face is generated, so the demo is not from scratch." β Correct, and stated: the portrait is SDXL-generated once and its mouth is ten generated shapes selected by the model's own phoneme timeline. Only the timing is the model's.
"You judged the face with a model, not a human." β Yes β and that model's aesthetic bar is photorealism, so it cannot judge a deliberately stylised face. A human rating sheet is in the evidence plan.
"Everything is self-referential." β It was. The WER test with Whisper is the first thing in this project that can falsify a claim, and it immediately did.
"No baseline, no variance, no ablation." β Correct on all three. Single seeds, no random-init baseline, nothing removed and re-measured.
"Your learning rate reached zero." β It did: the vocoder sat at stft 0.9130 for 30 000
steps at lr = 0.000e+00. Now floored at 5%.
"You pruned your own best checkpoints." β Twice. Recovered from the Hub, and pruning now protects every pin.
"Your backup fails silently." β It did, three ways: a private-storage ceiling that refused every commit while training continued, an upload that returned a commit URL having uploaded nothing, and a batched commit that was refused with nothing saved. All three are now detected and named.
"Thousands will attack this." β Yes. Of 100 rehearsed challenges: ~11 are answered with a measurement I would defend, ~30 with a measured number that is bad and published anyway, and ~25 are open admissions. The engineering claims are hard; the modelling claims are soft.
Training state on this machine (RTX PRO 6000 Blackwell, 100 GB)
| stage | run | progress |
|---|---|---|
| vlm | vlm-xl-main-df8703 |
step 12000 - loss 0.0622 |
| ear | ear-xl-main-53b51f |
step 291501 - |
| align | align-xl-main-56f984 |
step 130003 - |
| speech | speech-xl-v2-f93271 |
step 94003 - loss 0.4676 |
| vocoder | vocoder-xl-main-bfba0a |
step 589501 - |
Measured, and where the number comes from
| skill | result | how it was measured |
|---|---|---|
| sees and answers | exact match 47.0% | 200 questions over freshly generated scenes |
| describes unseen photos | BLEU-4 8.2 | 200 held-out flickr30k photographs with 5 human references each |
| end-to-end movie | 2/4 exactly right | image + question -> answer -> speech -> talking head -> mp4 |
| mouth vs audio | corr |
Every row is read from the run's own artifacts at publication time, so the card cannot claim more than was measured. Synthetic-scene accuracy is grounded but the world is shapes; a held-out BLEU-4 of 8 is real and low in absolute terms, not a captioning benchmark result.
Measured on this machine
| seed | question | answer | truth | exact |
|---|---|---|---|---|
| 7 | what size is the gray diamond | large | small | no |
| 18 | is there a red hexagon | no | no | yes |
| 26 | how many gray triangles are there | one | one | yes |
| 4 | describe the image | one cyan circle and one red diamond and one red square | one black star and one cyan diamond and one red square | no |
Loss per stage, read straight out of each run's log.jsonl on this machine.
Data β three datasets and one lexicon, nothing pretrained
- vision-language: procedurally rendered scenes (shapes, colours, sizes, 3x3 positions) and grounded Q/A β counting, colour, shape, location, size, captioning, comparatives, left/above/below relations, yes/no. Rendered on the GPU with a vectorised rasteriser, which is what keeps the GPU busy instead of waiting on a CPU data loader.
- speech: LJSpeech-1.1 (public domain, LibriVox, 24 h, single speaker), phonemised with CMUdict; alignments come from the model's own CTC ear, not an external aligner.
- One shared vocabulary from both corpora (8011 words) plus the 39-symbol ARPAbet phone set β the shared token space is what lets one trunk serve every modality.
Scenes are generated, so every answer is verifiable β which is what makes the exact-match numbers on this card mean something.
Resume policy β the sandbox is ephemeral by design
The training sandbox is ephemeral; only the notebook is snapshotted. So every stage
checkpoints model+optimizer+step to hkblockchaintechnology/omni-vlm-ckpt (pointer file written last, so its presence
implies the payload arrived), training runs detached so it survives a kernel restart,
batches are a pure function of the step counter so a resume continues exactly where it
stopped, and the run id is a hash of the architecture config so a changed config starts a
new run instead of corrupting an old one.
Reproduce β the exact commands
python -m omni.train --stage vlm --preset xl --steps 8000 --batch 384
python -m omni.train --stage ear --steps 4000 --batch 24
python -m omni.train --stage align
python -m omni.train --stage speech --steps 6000 --batch 24
python -m omni.train --stage vocoder --steps 4000 --batch 16
Limitations (honest list)
- The vision-language corpus is synthetic: the grounding is real, the world is shapes.
Swap
omni/scenes.pyfor a real caption/VQA corpus to grow it. - The voice is not intelligible yet, and the training metrics cannot see it. An
independent judge (
openai/whisper-small, which has never seen this project) transcribes the synthesised speech at a mean word error rate of 0.97 over five sentences, and each "sentence" runs 15 s where 2 s is expected β the decoder loops instead of stopping. The internal numbers said the opposite:phone_acc 1.000,dur_loss 0.005, fallingmel_loss. They are teacher-forced: the phoneme head is trained to predict each phone given the true previous phone, so it never learns to run on its own output, and autoregressive synthesis then drifts into repetition. Loss on a teacher-forced objective is not evidence about generation β that is the single most important thing this card can tell you. - The face is a still image, and the mouth is a set of generated shapes. A photoreal portrait carries ten mouth states, one per viseme, each produced by img2img from that same portrait at a fixed seed and composited through a feathered mask -- so the identity is pixel-identical outside the mouth and cannot drift. The model's own phoneme timeline selects and cross-fades between them. What that means honestly: the lips articulate through ten shapes rather than through a learned, continuous generator, the head does not turn, and there is no blinking. It is a puppet with the model's timing, not a learned talking-head model.
- No safety tuning, no RLHF, no moderation. This is a from-scratch research artefact.






