JoaoZaokk/glm-4-voice-bridge-GLM-4.6V-Flash-alpha

glm-4-voice → GLM-4.6V-Flash: audio bridge (alpha)

Built with glm-4. A research artifact from a model-surgery experiment: giving "ears" to zai-org/GLM-4.6V-Flash (image + text, MIT) by using the speech embeddings of zai-org/glm-4-voice-9b as a donor, without touching a single host weight. The long-term goal is one model with audio, video and image; this repository covers the first step, audio. It exists to record the method, the measured numbers and the causal controls. Nothing here is a benchmark claim.

Update 2026-09-06 — the earlier negative result was wrong, and it was wrong for a boring reason. The first version of this card said open-vocabulary Portuguese ASR "does not work yet", based on runs capped at 12,000 steps. Trained for 10 epochs over the full corpus (87,550 steps), the same 131,073-parameter bridge reaches CER 0.4250 on a speaker-disjoint held-out set, against 0.7573 for the best constant transcription of the corpus and 0.7991 for a paired control given the same budget. The plateau was undertraining, not a ceiling.

1. Headline numbers [measured]

Final bridge, 87,550 steps, 300 held-out clips with speakers never seen in training. The shuffled_map row is from its curve samples (120 clips per eval) because that run died at 40,000 steps; every other row is a 300-clip final.

arm	CER	WER	exact sentences
map (real bridge)	0.4250	0.7482	10/300
random control, same budget	0.7991	1.1685	0/300
shuffled_map control (died at 40,000 steps)	0.78–0.80	1.07–1.15	0/480
best constant transcription of the corpus	0.7573	1.0241	0/300

The random control sits at 0.7826 on its own training set, which rules out the alternative reading: it is not "learned but did not transfer", it is "did not learn".

Sentence identification, 10-way, length-matched distractors (the model scores 10 candidate transcriptions and picks the highest log-probability; distractor lengths matched to within 0.12 characters, so the duration shortcut is closed):

condition	accuracy	n
final bridge, 87,550 steps	0.9867	296/300
bridge at 12,000 steps (the old result)	0.1700	51/300
random control bridge, same 87,550 steps	0.1267	38/300
floor, same prompt with the audio rows removed	0.0933	28/300
chance	0.100	—

Choosing is far easier than generating, which is why 0.9867 coexists with CER 0.4250. The acoustic information arrives at the host essentially intact; the bottleneck is decoding it into text.

2. What it is

• Host: GLM-4.6V-Flash, entirely frozen (embedding, layers, lm_head). No new output tokens; the output vocabulary is still text only.

• Donor: glm-4-voice-9b. It provides (a) the speech tokenizer zai-org/glm-4-voice-tokenizer (Whisper encoder + VQ, 16,384 <|audio_N|> codes, 30 s window) and (b) the embedding table of those codes (ids from 152353 on).

• Bridge (the only trained module), per code c:

  E[c] = base[c] + (v[c] @ A) @ B · (α / rank), with gain = exp(gain_log) scaling the acoustic block.

  base is the map table (donor speech embeddings carried into the host input space by a ridge regression fitted on the 151,329 text tokens the two vocabularies share); v is the centered donor speech embedding; A (4096×16) and B (16×4096) start with B = 0, so at step 0 the bridge is exactly the table's zero-shot. 131,073 trainable parameters.

• Sequence: [gMASK]<sop><|user|>\n instruction \n [AUDIO] \n<|assistant|>\n<think></think> transcription <eos>. Loss falls only on the transcription and the <eos>. The empty <think></think> is not decoration: without closing the reasoning block the model "thinks" instead of answering (logit 31.5 for <think> against 16.75 for the runner-up, measured).

• Corpus: Common Voice PT, 8,755 training clips and 1,552 held-out clips, disjoint speakers.

• Paired controls in every measurement: shuffled_map (the same table rows, permuted with the same index on base and v: same information, same parameter count, donor geometry destroyed) and random (Gaussian noise with per-dimension mean and std matched). No gain is accepted without them.

3. The qualitative change is the most convincing part

At 20,000 steps the wrong hypotheses were fluent, plausible Portuguese with the right content words. At 87,550 steps they became phonetic transcription with broken word segmentation:

steps	hypothesis	reference
20k	"Um cachorro luta com uma dona em um estacionamento sujo."	"um cachorro solitário correndo em uma estrada suja"
87.5k	"Um cachorro lita com um cão em um meia estrada suja."	(same)
87.5k	"Horário de Janelas é a cidade mais louca do planeta."	"o rio de janeiro é a cidade mais louca do planeta"
87.5k	"Para istores da língua dos conuns"	"Para a História da Liga dos Comunistas"

"Horário de Janelas" for "Rio de Janeiro"; "istores da língua dos conuns" for "História da Liga dos Comunistas". The model started following the sound and failing at orthography and word boundaries. That is the signature of an acoustic front-end working without a strong language model on top — the opposite of hallucinating a likely sentence.

4. What is in this repository

file	what it is
pontes/cv-pt-map-r16-10ep-final-87550.safetensors	the final bridge, 87,550 steps (CER 0.4250, ident 296/300)
pontes/cv-pt-map-r16-10ep-40k.safetensors	same run at 40,000 steps (CER 0.4695)
pontes/cv-pt-map-r16-10ep-parcial-20k.safetensors	same run at 20,000 steps (CER 0.5277)
pontes/cv-pt-random-r16-10ep-final-87550.safetensors	the random control, full 87,550 steps
pontes/cv-pt-map-r16-12k.safetensors	the old 12,000-step bridge (reports 007/010)
scripts/g2_voice_etapa0_tables.py	builds the map, voice_centered, shuffled_map, random tables
scripts/g2_voice_etapa1_tokenize.py	tokenizes an audio manifest into glm-4-voice VQ codes
scripts/g2_voice_etapa1_asr.py	bridge training and evaluation (WER/CER, controls, watchdog, partial checkpoints)
scripts/g2_voice_etapa1_ident.py	the N-way identification test and its --sem-audio floor
relatorios/G2-voice-etapa0-008..011.md	measured reports, in Portuguese; 011 is the current one
logs/	raw logs and JSON records of every run, including the ones that died
licenses/glm-4-voice-LICENSE.txt	donor license, mandatory copy

Bridge tensors: A [4096,16] f32, B [16,4096] f32, gain_log scalar f32, with metadata condicao, rank, passos (steps), lr. The embedding tables are not here (1 GB, derived from the donor weights); the script rebuilds them from the two original models.

5. How to use (research level)

Requirements: both original checkpoints, PyTorch + Transformers with glm4v support, ffmpeg, a 24 GB GPU (the host in bf16 takes ~19.7 GiB; training peaks at 21.8 GiB reserved).

# 1. tables (once): reads the host text embedding and the donor speech embedding
python scripts/g2_voice_etapa0_tables.py
# 2. VQ codes for a {audio, text} manifest
python scripts/g2_voice_etapa1_tokenize.py --manifest my-manifest.jsonl --out manifest-codes.jsonl
# 3. evaluate the final bridge (300 held-out clips, paired with 300 training clips)
python scripts/g2_voice_etapa1_asr.py --codes manifest-codes.jsonl --condition map --rank 16 \
  --load-bridge pontes/cv-pt-map-r16-10ep-final-87550.safetensors --steps 0 --final-n 300 --out eval.json
# 4. the identification test and its floor
python scripts/g2_voice_etapa1_ident.py --codes manifest-codes.jsonl --condition map --rank 16 \
  --load-bridge pontes/cv-pt-map-r16-10ep-final-87550.safetensors \
  --n-clipes 300 --n-vias 10 --distratoras comprimento --out ident.json

The scripts expect the layout of the source repository (models/zai-org__GLM-4.6V-Flash, derived_models/etapa0-audio); adjust the paths at the top of each file.

6. History of the measurements

Hard reference on the held-out set: the best constant transcription of the corpus gives CER 0.7573. Anything that does not go below that has learned nothing useful.

date	run	question	measured result
04 Sep	etapa0-001	zero-shot: does the host read speech codes without training?	no; all three tables at chance
04 Sep	etapa0-002	closed set (digits, 32 words), held-out speaker	0.900 digits, 0.872 words; controls behind
04 Sep	etapa0-003	composition: combination and speaker never seen	0.890 exact vs factored baseline 0.060, chance 0.0125
04 Sep	etapa0-004	robustness: trained on clean audio only	0.910 at 32 kbps mp3, 0.670 at 10 dB SNR
04–05 Sep	-005, -006, -007, -009	capacity, gain, batch noise, language, unfrozen backbone	all negative at 12,000 steps — see the caveat below
05 Sep	etapa0-008	do the codes carry anything? 10-way identification	51/300 = 0.170 vs floor 0.0867, p = 0.0023
05 Sep	etapa0-010	ceiling and generalization curve at fixed cost (12k steps)	held-out never below 0.7573 → "the bridge memorizes and never transfers"
06 Sep	etapa0-011	was the bottleneck steps?	yes: CER 0.4250, identification 296/300, controls flat

Caveat that matters [judgment]. Reports 005, 007 and 009 concluded negatively using the same 12,000-step budget that report 011 has now shown to be undertraining — at 10,000 steps even the real bridge sits at CER 0.80, indistinguishable from the controls. Their negative conclusions are measured in the wrong regime and need to be redone before being treated as fact. The most valuable of them is 009 (LoRA on the host + bridge), because with identification already at 0.9867 the bottleneck is plausibly decoding, which is exactly what unfreezing the backbone would attack.

7. Limitations

• Not usable ASR. CER 0.4250 on Portuguese with an unseen speaker is far from any practical use.
• Not a ready-to-use model: it needs both original checkpoints, the rebuilt tables and the scripts.
• The generalization gap opened by the end: 0.3354 on training clips against 0.4250 held-out, and 4× more exact sentences on training. It is starting to memorize speakers, though held-out kept improving to the last step.
• Portuguese only (Common Voice PT). English was tested once, at the undertrained budget, and lost.
• No video. Image is native to the host and was neither touched nor measured after the bridge.
• The bridge is input-side only: the model reads speech codes, it does not generate speech.
• No standard benchmark; the numbers are CER/WER on a speaker-disjoint held-out split of the same corpus.

8. Licenses and attribution

• The bridge weights (A, B, gain_log), the scripts and the reports in this repository: MIT (LICENSE).
• The host zai-org/GLM-4.6V-Flash is MIT.
• The donor zai-org/glm-4-voice-9b is under the glm-4-voice License (copy in licenses/glm-4-voice-LICENSE.txt): free academic use, commercial use upon registration with Zhipu, the obligation to display "Built with glm-4" and to start the name of any model created with its materials with "glm-4". The tables this method uses derive from the donor embedding; that is why they are not redistributed here and why this repository's name starts with glm-4.
• Common Voice (Mozilla) was used only as an evaluation corpus; no audio or transcription is redistributed.