File size: 1,543 Bytes
4c83778 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | ---
license: apache-2.0
base_model: openbmb/MiniCPM-o-4_5
library_name: transformers
tags:
- multimodal
- audio
- video
- full-duplex
- gander
---
# Gander-v63-Unit8 Thinker
This directory contains the materialized Thinker component of Gander v63. It combines the
fine-tuned language model and audio projection with the frozen vision encoder, audio encoder,
and visual resampler from `openbmb/MiniCPM-o-4_5`. It does not contain the Talker speech decoder.
## Streaming contract
- 1 second input units
- up to 8 lexical text tokens in each speaking unit
- 50 S3 speech tokens in the paired Talker unit
- streaming-exact audio frontend
- vocabulary size 151749, including `<|backchannel|>` at token ID 151748
## Loading
```python
import torch
from transformers import AutoModel, AutoTokenizer
path = "/path/to/thinker"
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
model = AutoModel.from_pretrained(
path,
trust_remote_code=True,
torch_dtype=torch.bfloat16,
init_vision=True,
init_audio=True,
init_tts=False,
).eval()
```
Full-duplex streaming, task tools, task slate, and detached speech generation require the
matching Gander runtime. A Codex or other worker is not embedded in these weights.
## Training summary
The Thinker was initialized from MiniCPM-o 4.5 and trained for one epoch (8407 optimizer
steps). The LLM and audio projection were trainable; the vision encoder, audio encoder,
resampler, and TTS modules were frozen. Training used BF16 and the Unit-8/50 contract above.
|