| --- |
| library_name: lyra-studio |
| license: apache-2.0 |
| language: [yue, zh, en] |
| tags: [text-to-image, text-to-speech, music-generation, lip-sync, virtual-idol, cantonese, cantopop, digital-human] |
| --- |
| |
| # LYRA — an AI-native Cantopop idol |
| ### 莉拉 · 一個喺機器入面出世嘅香港女仔 |
|
|
| **LYRA is an original AI-native character**, built end-to-end on a single GPU by a *studio of |
| specialist models* cooperating like a small animation/music house — the same **methodology** |
| behind characters like *Yuri* (AI.TALK), not a copy of one. |
|
|
| She is a **Hong Kong girl** chasing her dream to become a Cantopop star, who exists to |
| **give positive energy back to the people of Hong Kong.** |
|
|
| > **The lesson we took from the reference:** the moat is not the models — everyone has the models. |
| > The moat is **forming a character**: one face, one voice, one tone, one timeline, never drifting. |
| > One bible drives every specialist. See `character/studio_core.py`. |
| |
| ## 🎬 Watch — `assets/video/lyra_showcase.mp4` (2:29) |
|
|
| | # | Song | Video | |
| |---|---|---| |
| | 1 | **《霓虹裡的我》** *Me in the Neon* — 58.7 s | `neon_me_mv.mp4` | |
| | 2 | **《香港嘅女孩》** *The Hong Kong Girl* — 84.3 s | `hongkong_girl_mv.mp4` | |
| | ★ | **Both, back to back** | `assets/video/lyra_showcase.mp4` | |
|
|
| Each is a title card → LYRA **lip-syncing her own sung vocal** → full Cantopop song. All **44.1 kHz**. |
|
|
| | Asset | Path | |
| |---|---| |
| | 🎬 Showcase / MVs | `assets/video/lyra_showcase.mp4`, `neon_me_mv.mp4`, `hongkong_girl_mv.mp4` | |
| | 🎞️ Isolated performances | `assets/video/lyra_perf_song1.mp4`, `lyra_perf_song2.mp4` | |
| | 🗣️ Talking clip | `assets/video/lyra_talk_sig.mp4` | |
| | 🎵 Songs (44.1k) | `assets/music/song1_mix.mp3`, `song2_mix.mp3` | |
| | 🎤 Stems | `song1_vocal.wav` / `song1_instrumental.wav` (+ song2) | |
| | 🎨 Renders (10) | `assets/img/*.png` | |
| | 🗣️ Cantonese voice lines | `assets/audio/*.wav` | |
|
|
| **Lyrics — 《香港嘅女孩》** |
| > 我係香港嘅女孩 追住嗰道光 / 就算跌低 都要唱落去 |
| > 我係香港嘅女孩 唔怕天黑 / 因為我哋一齊 就會有光 |
|
|
| ## 👥 The studio (one GPU, five departments) |
|
|
| | Dept | Model | Role | |
| |---|---|---| |
| | 🧠 Director | **Qwen2.5-VL-7B-Instruct** | writes lyrics, scripts, shot prompts; judges | |
| | 🎨 Concept Artist | **Qwen-Image 20B** | renders her face, outfits, scenery, title cards | |
| | 🧑🎨 Vision Critic | **Qwen2.5-VL-7B-Instruct** | scores each frame vs the bible, returns fixes | |
| | 🗣️ Voice Actor | **Qwen3-TTS 1.7B + HK-Cantonese LoRA** (`jmtl/hk-cantonese-cc0-qwen3-tts`) | true Hong Kong Cantonese | |
| | 🎵 Composer | **YuE 7B + 1B** | lyrics-to-song, dedicated **Cantonese** tag | |
| | 🎞️ Animator | **Wav2Lip** | lip-sync to her own voice | |
| | ✂️ Editor | **ffmpeg** | title card + performance + song | |
|
|
| **Self-improving loop:** the Critic's *fixes are fused back into the prompt and persisted* |
| (`character/refinements.json`) — she sharpens every run. |
|
|
| **Verified, not assumed:** her Cantonese was round-tripped through **Whisper** — intended |
| `我唔係真人，但我一樣會唱到你心入面。` returned as `我不是真人但我一样会唱到你心里面`. |
|
|
| ## 🧬 Her bible (single source of truth) |
|
|
| ```python |
| CHARACTER = { |
| "name": "LYRA", "home": "Hong Kong", "look_age": "19", |
| "primary_language": "Cantonese (yue-HK)", |
| "dream": "to become a Cantopop star", |
| "mission": "to give positive energy back to the people of Hong Kong", |
| "values": ["resilience", "warmth", "home", "never stop believing"], |
| "origin": "grew up in a Mong Kok walk-up, watching the harbour lights from the rooftop", |
| } |
| ``` |
|
|
| ## 🛠️ Reproduce |
|
|
| `scripts/setup_voice.sh` → `merge_voice.sh` → `setup_music.sh` → `yue_sing.sh` → |
| `setup_lipsync.sh` → `studio_daemon.py` → `edit_mv.sh` |
|
|
| > Traps we hit: molab exports `PYTHONPATH` (breaks every isolated venv — clear it) and sets |
| > `PYTHONSAFEPATH=1` (hides your script dir). Current `torchaudio` demands TorchCodec, so YuE's |
| > four save/load sites were patched to `soundfile`. |
|
|
| ## ⚖️ Credits & licenses |
|
|
| No model weights are shipped here — artefacts and orchestration code only. |
| - **Qwen-Image**, **Qwen2.5-VL-7B**, **Qwen3-TTS-12Hz-1.7B** — Apache-2.0 (Qwen/Alibaba) |
| - **hk-cantonese-cc0-qwen3-tts** LoRA — Apache-2.0 (`jmtl`), CC-0 Common Voice `yue` |
| - **YuE** — see upstream licence · **Wav2Lip** — research/non-commercial · **ffmpeg** — LGPL/GPL |
|
|
| **LYRA is an original fictional character.** Not Yuri, not affiliated with AI.TALK, not a likeness |
| of any real person. Released as a technical demo. |
|
|
