OmniVLM

One model that sees, answers, speaks β€” and moves its lips.

One trunk. One token space. Every modality is a head on it.

trained from scratch stages exact match checkpoints licence python

Trained from random initialisation on a single RTX PRO 6000 β€” no pretrained vision backbone, no pretrained TTS, no HiFi-GAN, no external forced aligner. The same weights read an image, choose the words, time the phonemes, synthesise the waveform and drive a face, so the face cannot disagree with the voice it came from.

the whole pipeline


Status and roadmap

roadmap

stage what it does state
vlm eyes + trunk, from random weights β€” the only true pretraining done, 12 000 steps (150% of plan), loss 0.062
ear audio β†’ phonemes (CTC); also produces the alignments running, 63% β€” ctc 0.0067
speech voice: phonemes, durations, mel, on the same trunk running, 54%
vocoder mel β†’ waveform, from scratch running, 72% β€” stft 0.866
captions real photographs + human captions on that trunk running, 5% β€” needs its budget re-thought
align forced alignment of the speech corpus complete, 13 100 clips

299 distinct checkpoints are archived on the Hub, across 5 live stages, from 873 000 trained steps. Every stage checkpoints to the Hub and resumes exactly; the sandbox is ephemeral by design, not by accident.

Try it

pip install torch numpy pillow scipy huggingface_hub && apt-get install -y ffmpeg
git clone https://huggingface.co/hkblockchaintechnology/VLM-all-inclusive
python scripts/make_movie.py --question "describe the image"

One image and one question in; an mp4 with the answer spoken and the mouth moving comes out β€” and the published samples carry their own audio track, so picture and voice stay in sync when you play them.


What it is

Text-to-image, text-to-speech and text-to-video are separate models that each re-learn language. Here there is one token space and one trunk: the same weights that read an image and answer a question also drive the speech heads and the lips. The speech stages are trained multi-task with the vision-language task, so the trunk keeps both abilities instead of being forked into per-task copies.

part implementation trained here
eyes ViT patch encoder yes β€” on procedurally rendered scenes
brain causal trunk, cross-attends the vision tokens yes
text word head: question β†’ answer yes
ear CTC phoneme recogniser over log-mel, and it produces the alignments too yes β€” on the public-domain LJSpeech corpus
voice phoneme + duration heads + FastSpeech-style mel decoder, on the same trunk yes
sound Vocos-style mel β†’ waveform yes β€” multi-resolution STFT loss, no HiFi-GAN
lips ARPAbet β†’ viseme parameters β†’ a photoreal face whose mouth is composited from a generated shape per viseme, cross-faded between the shapes were generated once; the timing that selects them is the model's own phoneme timeline

one trunk, heads are the modalities

Everything enters and leaves through one trunk. The lips read the same phoneme timeline that produced the audio, so the face cannot disagree with the voice.

the lips, on the phoneme timeline

The ten viseme mouth shapes, and the mouth saying one real sentence in real time.

From scratch β€” and you can check it

Every weight here was initialised from random numbers and trained on this machine. There is no pretrained backbone underneath, and no borrowed network anywhere in the pipeline: the vision encoder, the trunk, the phoneme recogniser, the duration and mel heads and the vocoder are all created empty and taught from the corpora. That is a rare thing to be able to say, so it should not be taken on trust β€” it is a claim you can falsify in one command:

grep -rEn 'from_pretrained|torch[.]hub|(from|import) (transformers|timm|torchvision)|whisper|wav2vec|hifigan' omni/ notebook.py
# no matches
# no matches

The complete list of things this project downloads from the internet is three datasets and one dictionary β€” no models:

downloaded what it is licence
LJSpeech-1.1 24 h of read speech, one speaker β€” the audio corpus public domain
flickr30k 31 783 photographs, 5 human captions each CC-BY 4.0
CMUdict an ARPAbet pronunciation dictionary, 135 166 entries: text in, phonemes out BSD-2
β€” the scene images are not downloaded: omni/scenes.py generates them β€”

Two footnotes, because "from scratch" should not be allowed to do quiet work:

  • CMUdict is a lexicon, not a model. It is a list of how English words are pronounced. A word outside it falls back to ten hand-written letter-to-sound mappings in omni/text.py (a -> AH, x -> K S, ...) β€” not a learned grapheme-to-phoneme network.
  • The viseme table is hand-authored, not learned. omni/avatar.py maps 39 ARPAbet phones onto 15 mouth shapes using numbers a person chose. What the model contributes is the timing β€” its own phoneme sequence and its own durations β€” and what the renderer does with that is arithmetic and a rasteriser.

So the honest summary is: a new architecture, new weights, trained from random initialisation on public data, with the whole pipeline published. If a future version of this repository ever loads a pretrained component, the grep above will stop returning nothing. That is the point of stating it as a check rather than as a claim.

Why an all-inclusive model β€” the ground for OmniVLM

Today's multimodal models are, almost always, one output modality per model. Look at what actually ships on the Hub β€” sampled by pipeline tag, first 1 000 results each, via https://huggingface.co/api/models?filter=<tag>, 17 Nov 2025:

pipeline tag models (sampled) what it gives you
image-text-to-text >1 000 sees, answers in text β€” no sound, no face
text-to-speech >1 000 says it β€” no face, no visual grounding
text-to-video / image-to-video >1 000 each moves pixels β€” the voice is bolted on afterwards, with no shared representation
any-to-any >1 000 the all-inclusive bucket β€” but it stops at audio out (e.g. Qwen/Qwen2.5-Omni-7B, openbmb/MiniCPM-o-2_6); no rendered, lip-synced face
audio-to-audio, text-to-audio >1 000 each voice/audio-only stacks

And lip sync, the piece that makes any of it look alive, is a niche of ~200 repos outside every mainstream pipeline: lipsync 40, wav2lip 54, sadtalker 58, musetalk 51, talking-head 15 β€” there is no Hub pipeline tag for it at all, and nearly all of those are weights for a handful of upstream projects that each want a pre-existing video plus a pre-existing audio track.

So the field's default shape is three half-measures:

  1. video with no sound β€” a diffusion video model; you add TTS later, and the mouth was never conditioned on the words it is now speaking;
  2. sound with no video β€” an excellent TTS model; no face, and no grounding in anything visual;
  3. TTS with no lip sync β€” or lips driven by a second, separately trained model that has never seen the first one's representation of language.

Each seam is a place where the modalities disagree. Bolting them together cannot fix that, because there is no shared token space in which "the word I am saying", "the sound I am making" and "the shape my mouth is in" are the same object.

That is the ground OmniVLM stands on. Here the modalities are not stitched: there is one token space (words and ARPAbet phones in one vocabulary) and one causal trunk whose heads are the modalities β€” vision tokens cross-attended for grounding, a word head, a phoneme head, a duration head and a mel decoder on the same hidden state, with the speech stages trained multi-task alongside the vision-language task so neither ability is forked off. The lips are therefore driven by the model's own phoneme timeline β€” the same timeline that produced the waveform β€” instead of by a separate lip-sync network. And unlike the any-to-any giants, the whole thing is trained from scratch on one GPU with published checkpoints, a public held-out metric (BLEU-4 on flickr30k val) and scripts that reproduce it.

Stated fairly: the claim is not that no multimodal model exists β€” Gemini/Qwen-Omni-class systems are more capable than this, and any-to-any is a real, thriving category. The claim is narrower and checkable: no published artefact trains eyes, language, voice, sound and lips jointly, from scratch, and ships the whole pipeline β€” weights, eval, resume policy β€” as one thing you can download and retrain. That is the gap this project exists to fill, and it is why the quality caveats below are the honest frontier rather than a hedge.

What this is for

The unusual property here -- one token space, one trunk, every modality trained together -- is exactly what an agent needs, because an agent has to see, decide, speak and be seen doing it in the same moment, not in a pipeline of separate models:

where it applies why this shape fits
computer-use agents the agent reads a screen, acts, and narrates what it is doing. Because the seeing and the saying come from one forward pass over one shared representation, the narration cannot drift away from the action it is describing.
phone / on-device agents one small trunk plus three tiny heads replaces a separate vision model, a separate ASR, a separate TTS and a separate lip-sync stack. Fewer artefacts to ship, one thing to keep aligned, and the whole assistant can live on the device instead of making a cloud round trip per modality.
robotics and embodied agents a robot that perceives, answers and speaks from one set of weights is a robot whose words match its perception. And because the mouth is driven by the same phoneme timeline that produced the audio, the face and the voice are generated together rather than dubbed over one another.

three models, three clocks β€” or one trunk, one clock

The field is full of models that are brilliant at one channel and hand the rest to a different model with a different vocabulary, a different training run and a different clock. Every seam between them is latency, and every seam is somewhere the channels can disagree.

Emotion is a property of simultaneity, not a fourth model

Feeling is not a separate head bolted on at the end; it is what the coincidence of the channels conveys. A mouth one phoneme behind its voice reads as uncanny. A voice that says one thing while the eyes and brow say another reads as insincere. For a face to move a person, the timing has to be real -- one decision, at one instant, driving the word, the sound and the movement.

That is what training the modalities together buys: the timings that carry feeling are computed from a single hidden state rather than reconstructed afterwards by an integrator guessing at alignment. It is what lets a delivery carry not only knowledge and a machine voice, but something that reads as intention and warmth -- smooth, aligned and coherent from the inside out, whether it arrives through a desktop agent, a phone, or a robot standing in front of you.

Stated honestly: on this checkpoint the claim is architectural, not measured. These weights were trained on procedurally rendered shapes and one speaker's read speech, so what ships today is a small research artefact. The point of the table above is that the architecture has somewhere to go that a stitched pipeline cannot follow -- and the checkpoints, the eval and the resume policy are published so that someone can take it there.

Questions we expect β€” and honest answers

Full list: 100 rehearsed challenges with their evidence status.

The twelve that will be asked first

"Your voice is gibberish." β€” Yes. An independent Whisper transcribes the synthesised speech at mean WER 0.97, and each sentence runs 15 s where 2 s is expected: the decoder loops instead of stopping. The cause is teacher forcing.

"But your metrics said phone_acc 1.000." β€” And that is the lesson. The phoneme head is trained to predict each phone given the true previous phone, so it never learns to run on its own output. A teacher-forced loss is not evidence about generation. The card reports the WER because the loss could not have told you.

"More training made it worse?" β€” Yes, and it is measured: VQA exact match 47.0% at step 39 002 versus 35.5% at 49 502. The speech stage fine-tunes the whole trunk on a speech objective, which erodes the vision-language ability. Serving is pinned to the better checkpoint.

"From scratch β€” really?" β€” Check it. A grep for from_pretrained, torch.hub, transformers, timm, torchvision, whisper, wav2vec and hifigan across all 20 source files returns nothing. Three datasets and one pronunciation dictionary are the only downloads.

"The face is generated, so the demo is not from scratch." β€” Correct, and stated: the portrait is SDXL-generated once and its mouth is ten generated shapes selected by the model's own phoneme timeline. Only the timing is the model's.

"You judged the face with a model, not a human." β€” Yes β€” and that model's aesthetic bar is photorealism, so it cannot judge a deliberately stylised face. A human rating sheet is in the evidence plan.

"Everything is self-referential." β€” It was. The WER test with Whisper is the first thing in this project that can falsify a claim, and it immediately did.

"No baseline, no variance, no ablation." β€” Correct on all three. Single seeds, no random-init baseline, nothing removed and re-measured.

"Your learning rate reached zero." β€” It did: the vocoder sat at stft 0.9130 for 30 000 steps at lr = 0.000e+00. Now floored at 5%.

"You pruned your own best checkpoints." β€” Twice. Recovered from the Hub, and pruning now protects every pin.

"Your backup fails silently." β€” It did, three ways: a private-storage ceiling that refused every commit while training continued, an upload that returned a commit URL having uploaded nothing, and a batched commit that was refused with nothing saved. All three are now detected and named.

"Thousands will attack this." β€” Yes. Of 100 rehearsed challenges: ~11 are answered with a measurement I would defend, ~30 with a measured number that is bad and published anyway, and ~25 are open admissions. The engineering claims are hard; the modelling claims are soft.

Training state on this machine (RTX PRO 6000 Blackwell, 100 GB)

stage run progress
vlm vlm-xl-main-df8703 step 12000 - loss 0.0622
ear ear-xl-main-53b51f step 291501 -
align align-xl-main-56f984 step 130003 -
speech speech-xl-v2-f93271 step 94003 - loss 0.4676
vocoder vocoder-xl-main-bfba0a step 589501 -

Measured, and where the number comes from

skill result how it was measured
sees and answers exact match 47.0% 200 questions over freshly generated scenes
describes unseen photos BLEU-4 8.2 200 held-out flickr30k photographs with 5 human references each
end-to-end movie 2/4 exactly right image + question -> answer -> speech -> talking head -> mp4
mouth vs audio corr

Every row is read from the run's own artifacts at publication time, so the card cannot claim more than was measured. Synthetic-scene accuracy is grounded but the world is shapes; a held-out BLEU-4 of 8 is real and low in absolute terms, not a captioning benchmark result.

Measured on this machine

seed question answer truth exact
7 what size is the gray diamond large small no
18 is there a red hexagon no no yes
26 how many gray triangles are there one one yes
4 describe the image one cyan circle and one red diamond and one red square one black star and one cyan diamond and one red square no

measured training curves

Loss per stage, read straight out of each run's log.jsonl on this machine.

Data β€” three datasets and one lexicon, nothing pretrained
  • vision-language: procedurally rendered scenes (shapes, colours, sizes, 3x3 positions) and grounded Q/A β€” counting, colour, shape, location, size, captioning, comparatives, left/above/below relations, yes/no. Rendered on the GPU with a vectorised rasteriser, which is what keeps the GPU busy instead of waiting on a CPU data loader.
  • speech: LJSpeech-1.1 (public domain, LibriVox, 24 h, single speaker), phonemised with CMUdict; alignments come from the model's own CTC ear, not an external aligner.
  • One shared vocabulary from both corpora (8011 words) plus the 39-symbol ARPAbet phone set β€” the shared token space is what lets one trunk serve every modality.

what it is trained on

Scenes are generated, so every answer is verifiable β€” which is what makes the exact-match numbers on this card mean something.

Resume policy β€” the sandbox is ephemeral by design

The training sandbox is ephemeral; only the notebook is snapshotted. So every stage checkpoints model+optimizer+step to hkblockchaintechnology/omni-vlm-ckpt (pointer file written last, so its presence implies the payload arrived), training runs detached so it survives a kernel restart, batches are a pure function of the step counter so a resume continues exactly where it stopped, and the run id is a hash of the architecture config so a changed config starts a new run instead of corrupting an old one.

Reproduce β€” the exact commands
python -m omni.train --stage vlm     --preset xl --steps 8000 --batch 384
python -m omni.train --stage ear     --steps 4000 --batch 24
python -m omni.train --stage align
python -m omni.train --stage speech  --steps 6000 --batch 24
python -m omni.train --stage vocoder --steps 4000 --batch 16

Limitations (honest list)

  • The vision-language corpus is synthetic: the grounding is real, the world is shapes. Swap omni/scenes.py for a real caption/VQA corpus to grow it.
  • The voice is not intelligible yet, and the training metrics cannot see it. An independent judge (openai/whisper-small, which has never seen this project) transcribes the synthesised speech at a mean word error rate of 0.97 over five sentences, and each "sentence" runs 15 s where 2 s is expected β€” the decoder loops instead of stopping. The internal numbers said the opposite: phone_acc 1.000, dur_loss 0.005, falling mel_loss. They are teacher-forced: the phoneme head is trained to predict each phone given the true previous phone, so it never learns to run on its own output, and autoregressive synthesis then drifts into repetition. Loss on a teacher-forced objective is not evidence about generation β€” that is the single most important thing this card can tell you.
  • The face is a still image, and the mouth is a set of generated shapes. A photoreal portrait carries ten mouth states, one per viseme, each produced by img2img from that same portrait at a fixed seed and composited through a feathered mask -- so the identity is pixel-identical outside the mouth and cannot drift. The model's own phoneme timeline selects and cross-fades between them. What that means honestly: the lips articulate through ten shapes rather than through a learned, continuous generator, the head does not turn, and there is no blinking. It is a puppet with the model's timing, not a learned talking-head model.
  • No safety tuning, no RLHF, no moderation. This is a from-scratch research artefact.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support