red-kit/meeting-avatar

Meeting Avatar — a real-person avatar that attends live meetings

A production-oriented, fully pre-trained stack: a photoreal lip-synced avatar of a real person with a cloned voice and an LLM brain, served over WebRTC, ready to bridge into Zoom / Google Meet / Teams / LiveKit.

Built on OpenAvatarChat (pinned commit 8b7b3b4, Apache-2.0) with a deployment-focused Docker image, configs, and meeting-bridge documentation.

Live demo Space: red-kit/meeting-avatar-demo (slim image + Storage Bucket for weights — see below).

Pipeline

browser / meeting bridge (WebRTC)
  └─ Silero VAD → SenseVoice ASR (local)
       └─ LLM brain (any OpenAI-compatible API)
            └─ CosyVoice2-0.5B zero-shot voice clone (local, streaming)
                 └─ MuseTalk real-time lip-sync over the person's footage
                      └─ avatar video+audio back over WebRTC

Component	Model	License	Notes
Avatar (face)	MuseTalk 1.5	MIT	lip-syncs a real video of the person — identity by construction
Voice	CosyVoice2-0.5B	Apache-2.0 (code) / model terms	zero-shot cloning from 5–10 s reference audio
ASR	SenseVoiceSmall	Apache-2.0 (code) / model terms	multilingual, local
VAD	Silero VAD	MIT
Brain	any OpenAI-compatible API	yours	default: gpt-4o-mini @ api.openai.com

End-to-end speech-to-speech latency: ~0.8–2.2 s on one A10G-class GPU (upstream reports 2.2 s avg on RTX 4090). One concurrent session per GPU with the MuseTalk config; use config/meeting_avatar_lite.yaml for lighter hardware.

Verified on GPU

End-to-end boot smoke-tested on Hugging Face Jobs (NVIDIA A10G, CUDA 12.8): system deps → pinned clone → install → weights → boot → HTTP 200 + CosyVoice2 zero-shot synthesis on GPU (9 min, exit 0; handler load 22.5 s, warmup synthesis rtf 1.68, both TTS processors live). Reproducible via scripts/smoke_test.sh. Verification surfaced and fixed four real deployment bugs, all now baked into the Dockerfile: a typeguard/typing_extensions pin conflict, a pyarrow/NumPy wheel conflict, a stale upstream voice-asset path, and an upstream CosyVoice warmup-call signature bug (patched at build time).

⚠️ Runtime hardware: two host shapes get the stack OOM-killed at boot (exit 137), even though the image is identical: a10g-small (only 15 GB system RAM) and any multi-GPU host without the single-GPU pin (a 2x-a10g Space with 92 GB RAM did it — the stack is single-GPU by design; CUDA_VISIBLE_DEVICES=0 is baked into the Dockerfile). Use single-GPU hardware with ≥46 GB RAM: l40s or a10g-large, or a 32 GB+ RAM GPU box.

Deploy

Option 1 — any CUDA 12.8 host (simplest, weights baked in)

git clone https://huggingface.co/redkits/meeting-avatar && cd meeting-avatar
echo "DASHSCOPE_API_KEY=sk-your-openai-compatible-key" > .env
docker compose up --build   # then open http://localhost:7860

Requires ~24 GB VRAM and ≥46 GB system RAM (A10G-large / L40S / RTX 3090 or 4090 with 32+ GB RAM). Image is ~20 GB — all weights are baked at build time; restarts never re-download.

Option 2 — Hugging Face Space (slim image + Storage Bucket)

⚠️ Do not deploy this repo's fat Dockerfile to a Space as-is. HF builds Docker Spaces on shared builder infrastructure with a ~4 GB RAM cgroup and no GPU, regardless of attached hardware — baking ~20 GB of weights into the image OOM-kills the build (exit 137, OOMKilled, stage BUILD_ERROR). The demo Space therefore uses a slim variant of this image whose entrypoint downloads the weights once at first boot into an attached Storage Bucket.

To deploy (or duplicate the demo Space):

1. Create a Docker Space and copy the demo Space's files (Dockerfile, scripts/space-entrypoint.sh, config/, docs/).
2. Settings → Secrets → add DASHSCOPE_API_KEY (your LLM API key — the env var name is an upstream default, any OpenAI-compatible key works).
3. Settings → Storage buckets → create a private bucket and attach it at mount path /models, read-write (weights persist across restarts).
4. Settings → Hardware → attach l40s (48 GB VRAM, 62 GB RAM, $1.80/h). ⚠️ Not a10g-small (15 GB system RAM → OOM-killed at boot) and not multi-GPU flavors (single-GPU stack; the second GPU sits idle while billing — the CUDA_VISIBLE_DEVICES=0 pin is baked in).
5. First boot downloads 20 GB of weights (10–20 min, placeholder page meanwhile); restarts skip the download.

Option 3 — HF Inference Endpoints

Inference Endpoints are request/response HTTP — they do not fit a long-lived WebRTC session. Use them later for a headless render API if you outgrow the single-GPU Space; the interactive avatar itself wants a persistent container (Options 1–2).

Make it your person (no training required)

Two files, both specified in docs/footage-collection-spec.md:

• Face: 1–3 min static-camera clip of the person → set AvatarMusetalk.avatar_video_path in config/meeting_avatar.yaml.
• Voice: 5–10 s clean recording + exact transcript → set CosyVoice.ref_audio_path / ref_audio_text.

Until real footage exists, the stack runs on bundled stand-in assets. For the photoreal LoRA upgrade path (Wan-S2V person LoRA), the collection spec includes what to record for that too.

Getting into meetings

See docs/meeting-bridges.md: OBS virtual camera (zero-code, universal), Recall.ai output-media bot (production, Zoom/Meet/Teams), or LiveKit native participant (cleanest architecture). Includes the latency budget and the honest list of what nobody has solved yet.

Compliance — read before first use

Cloning a real person's face and voice requires their written consent, and many jurisdictions and meeting platforms require disclosing AI participants. Details in the collection spec. This repo ships stand-in assets only.

Repo layout

Dockerfile                     self-host image (weights baked in — NOT for HF Spaces)
docker-compose.yml             self-host launcher
config/meeting_avatar.yaml     production config (MuseTalk)
config/meeting_avatar_lite.yaml  lightweight fallback (LiteAvatar)
docs/footage-collection-spec.md  how to collect face + voice assets
docs/meeting-bridges.md        Zoom/Meet/Teams/LiveKit integration routes
scripts/smoke_test.sh          GPU end-to-end boot verification

For the HF Space deployment (slim Dockerfile + entrypoint + bucket), see red-kit/meeting-avatar-demo.

Upstream credits: OpenAvatarChat, MuseTalk, CosyVoice, Silero VAD, SenseVoice (FunAudioLLM).