Text-to-Speech
LiteRT
LiteRT
ONNX
Core ML
PyTorch
tts
singing
singing-voice-synthesis
multilingual
korean
expressive-tts
duration-flow
Instructions to use acidsound/scyllasband with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT
How to use acidsound/scyllasband with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
|
Download PROGRESS.md from acidsound/scyllasband: direct link, hf CLI and curl.
- Browser
- Download file 30 kB
-
https://huggingface.co/acidsound/scyllasband/resolve/main/PROGRESS.md
- Command line
-
hf download hf://acidsound/scyllasband/PROGRESS.md
-
curl -L -o PROGRESS.md https://huggingface.co/acidsound/scyllasband/resolve/main/PROGRESS.md
30 kB
| # Korean Support for Scylla's Band β Engineering Progress (`PROGRESS.md`) | |
| ## 1. Objective & Status | |
| - **Objective**: Deliver reliable, end-to-end Korean (`ko` / `ko_kr`) speech and singing support for **Scylla's Band (`v4-20261106`)** across both the model repository (`spybyscript/scyllasband`) and the reference/native runtime (`lowkeytea/scyllasband`, located in `runtime/`), preserving compact deployment, fast local inference, exact Python/C++ parity, and backward compatibility with all 14 existing voices, 7 trained locales (`en_us`, `en_gb`, `es`, `it`, `fr`, `de`, `vi`), and 6 bundle flavors (`litert`, `onnx`, `webgpu`, `coreml`, `coreai`, `pytorch`). | |
| - **Status**: **Complete and verified end-to-end** across formal TLA+ specifications, Python (`scyllasband`) and C++ (`libscyllasband`) implementations, all 6 bundle flavors, CTest (`16/16`), pytest (`131/131`), the full Python/C++ parity harness (`0` failures), Korean ASR intelligibility benchmarks, and Korean singing synthesis (`vocos` and `bigvgan`). | |
| --- | |
| ## 2. Confirmed Architectural Findings & Engineering Decisions | |
| ### 2.1 Model & Graph Contract Analysis (`v4-20261106`) | |
| 1. **Phone Vocabulary (`phone_vocab.json`, 87 tokens, IDs `0..86`)**: | |
| - Structural/control tokens (`0..14`): `<pad>`, `<unk>`, `<sil>`, `<pause_comma>`, `<pause_semicolon>`, `<pause_colon>`, `<pause_dash>`, `<ellipsis>`, `<end_stmt>`, `<end_question>`, `<end_exclaim>`, `<ctx_sentence_start>`, `<ctx_continuation>`, `<ctx_sentence_end>`, `<ctx_chunk_continue>`. | |
| - Tone digits (`15..20`): `1`, `2`, `4`, `5`, `6`, `7`. | |
| - IPA vowels, glides, and consonants (`21..77, 84..86`) and non-acoustic modifier bits (`78..83`). | |
| - **Important Vocabulary Constraint**: `phone_vocab.json` contains postalveolar `"Κ"` (`id=69`) and `"Κ"` (`id=74`), not alveolo-palatal `"Ι"` or `"Κ"`. Using `"Κ"` and `"Κ"` for Korean `γ /γ /γ ` (`t Κ` / `d Κ` / `t Κ h`) and palatalized `γ /γ ` (`Κ` / `Κ Κ` before `γ £/γ /γ /γ /γ /γ /γ `) keeps 100% of emitted Korean phones strictly inside the 87-symbol vocabulary. | |
| 2. **Neural G2P Graph vs Deterministic Korean Phonological G2P**: | |
| - The exported neural G2P graphs (`g2p.onnx`, `g2p.tflite`, `g2p.mlmodelc`, `g2p.aimodel`, `g2p.pt`) are char-level CTC models (`char_repeats = 4`) trained only on `<de>`, `<en_gb>`, `<en_us>`, `<es>`, `<fr>`, `<it>`, `<vi>` with no `<ko>` symbol and no Hangul coverage in `tokenizer.json`. | |
| - Therefore, when `language == "ko"` or when any input segment contains Hangul (`U+AC00..U+D7A3` or Compatibility Jamo `U+3131..U+3163`), `G2PFrontend._predict_segment` (Python) and `G2P::predict_segment` (C++) route Hangul tokens through a deterministic, context-sensitive Standard Korean (`νμ€λ°μλ²`) phonological G2P, while delegating embedded multi-letter Latin words to the voice's English G2P/lexicon and spelling out 1β3 letter uppercase initialisms (`AI` -> `μμ΄μμ΄`, `TTS` -> `ν°ν°μμ€`, `GPU` -> `μ§νΌμ `, `MP3` -> `μ νΌμΌ`, `4K` -> `μ¬μΌμ΄`) in Korean. | |
| 3. **Acoustic Conditioning (`language_id = 3`)**: | |
| - Precompiled graphs in `litert`, `onnx`, `webgpu`, `coreml`, and `coreai` have fixed `[7, D]` `language_embed` and `[98, ...]` (`14 voices Γ 7 locales`) voice/locale conditioning tables. | |
| - Empirical probing across all 7 trained `language_id` indices (`0..6`) with `faster-whisper` (`large-v3-turbo`) confirmed that mapping `"ko"` to acoustic conditioning index `3` (`it`, syllable-timed pure-vowel Romance conditioning without English stress-timed schwa reduction or Vietnamese lexical tone contours) yields the cleanest Korean vowel formants and syllable timing across all 14 voices. | |
| ### 2.2 Korean Spoken-Text Normalizer (`text_normalizer.py` & `scyllasband_text.cpp`) | |
| - Preserved `SPOKEN_TEXT_NORMALIZER_SHA256` (`d8fd1b2415d0d100d1c70237a4469117ee6b73d0e4da703e4654c83192779cc4`) and 100% backward compatibility with all existing languages. | |
| - Added `"ko"`, `"ko-kr"`, `"ko_kr"`, `"kor"`, `"korean"` alias resolution in `normalize_language_code`. | |
| - Implemented identical Korean normalization rules in Python and C++: | |
| - **Dual Numeral System**: | |
| - Sino-Korean (`μ, μΌ, μ΄, μΌ, μ¬, μ€, μ‘, μΉ , ν, ꡬ, μ, λ°±, μ², λ§, μ΅, μ‘°`) with Korean `1`-elision before `μ/λ°±/μ²` (`10` -> `μ`, `15,000` -> `λ§ μ€μ²`, `100,000,000` -> `μΌμ΅`). | |
| - Native Korean (`ν/νλ, λ/λ, μΈ/μ , λ€/λ·, λ€μ―, ..., μ€λ¬΄/μ€λ¬Ό, μλ₯Έ, λ§ν, μ°, μμ, μΌν, μ¬λ , μν`) automatically triggered before 25 Native Korean counters (`κ°, λͺ , λΆ, μ¬λ, λ§λ¦¬, μ΄, μ, μκ°, λ¬, μ, λ³, κΆ, μ₯, λ, μ±, μΌ€λ , λ², μ‘μ΄, μ루, 그루, μ‘°κ°, μμ, λ°κ΅¬λ, λ΄μ§, λ²`). | |
| - Ordinals (`1λ²μ§Έ` -> `첫 λ²μ§Έ`, `2λ²μ§Έ` -> `λ λ²μ§Έ`) and Sino-Korean month exceptions (`6μ` -> `μ μ`, `10μ` -> `μμ`). | |
| - **Structured Entities**: | |
| - ISO dates (`2026-11-08` -> `μ΄μ²μ΄μμ‘λ μμΌμ νμΌ`). | |
| - Times (`3:30 PM` -> `μ€ν μΈ μ μΌμ λΆ`, `7:05 AM` -> `μ€μ μΌκ³± μ μ€ λΆ`, `14:00` -> `μμ¬ μ`). | |
| - Currencies (`β©15,000` -> `λ§ μ€μ² μ`, `$3.50` -> `μΌ λ¬λ¬ κ³Ό μ€μ μΌνΈ`, `40β¬` -> `μ¬μ μ λ‘`, `Β£20` -> `μ΄μ νμ΄λ`, `Β₯500` -> `μ€λ°± μ`). | |
| - Fractions (`3/4` -> `μ¬ λΆμ μΌ`, `1/2` -> `μ΄ λΆμ μΌ`), decimals (`3.14` -> `μΌ μ μΌμ¬`), percentages (`95.5%` -> `ꡬμμ€ μ μ€ νΌμΌνΈ`), temperatures/degrees (`-5Β°C` -> `λ§μ΄λμ€ μ€ λμ¨`, `20Β°` -> `μ΄μ λ`), symbols (`=` -> `μ΄ν`, `+` -> `νλ¬μ€`), and alphanumeric splitting (`MP3μ 4K` -> `μ νΌ μΌ μ μ¬ μΌμ΄`). | |
| ### 2.3 Standard Korean Phonological G2P (`g2p.py` & `scyllasband_g2p.cpp`) | |
| - Implemented Hangul syllable decomposition (`U+AC00..U+D7A3` -> 19 Choseong, 21 Jungseong, 28 Jongseong) + Compatibility Jamo (`γ±..γ `, `γ ..γ £`) handling. | |
| - Implemented ordered Standard Korean (`νμ€λ°μλ²`) phonological rules across adjacent syllables: | |
| 1. **Palatalization (ꡬκ°μν, Clause 17)**: `γ·/γ /γΎ` before vowel-initial `γ £/γ /γ /γ /γ /γ ` -> `γ /γ /γΉ+γ ` (`κ°μ΄` -> `[κ°μΉ]`, `κ΅³μ΄` -> `[ꡬμ§]`, `λ«νλ€` -> `[λ€μΉλ€]`). | |
| 2. **Liaison (μ°μλ²μΉ, Clauses 13β14)**: Single codas move to following null onset (`νκ΅μ΄` -> `[νꡬ거]`); complex codas (`κ²Ήλ°μΉ¨`: `γ³, γ΅, γΊ, γ», γΌ, γ½, γΎ, γΏ, γ `) split across the boundary (`μ½μ΄` -> `[μΌκ±°]`, `μμ` -> `[μμ]`, `μμ΄` -> `[μ μ¨]`). | |
| 3. **`γ ` Aspiration & Elision (γ μΆμ½ λ° νλ½, Clause 12)**: `γ /γΆ/γ ` + `γ±/γ·/γ /γ ` (or reverse) -> `γ /γ /γ /γ ` (`μ’λ€` -> `[μ‘°ν]`, `μ ν` -> `[μ΄ν]`, `λ 립기λ κ΄` / `λ§μ΄` -> `[λ§λ]`, `μ’μμ` -> `[μ‘°μμ]`), `γ /γΆ/γ ` + `γ ` -> `γ `. | |
| 4. **Complex Coda Reduction & Neutralization (μμ μ λμ리 κ·μΉ, Clauses 8β11)**: All 27 codas neutralize to the 7 representative codas (`k, n, t, l, m, p, Ε`), including the Clause 11 exception where `γΊ` before `γ±` reduces to `γΉ` + `γ²` (`λ§κ³ ` -> `[λ§κΌ¬]`, `μ½κ³ ` -> `[μΌκΌ¬]`, whereas `λ§λ€` -> `[λ§λ°]`). | |
| 5. **Liquidization (μ μν, Clause 20)**: `γ΄+γΉ` and `γΉ+γ΄` -> `γΉ+γΉ` (`μ λΌ` -> `[μ€λΌ]`, `μ€λ ` -> `[μ€λ]`). | |
| 6. **Nasalization (λΉμν, Clauses 18β19)**: Obstruent coda (`k, t, p`) before nasal (`n, m`) -> (`Ε, n, m`) (`κ΅λ¬Ό` -> `[κΆλ¬Ό]`, `λ¨Ήλ` -> `[λ©λ]`, `μλ` -> `[μΈλ]`); nasal (`m, Ε`) + `r` -> `m/Ε + n` (`μ’ λ‘` -> `[μ’ λ Έ]`); obstruent (`k, t, p`) + `r` -> (`Ε, n, m`) + `n` (`λ 립` -> `[λλ]`, `λ°±λ§` -> `[λ± λ§]`). | |
| 7. **Post-Obstruent Tensification (κ²½μν, Clause 23)**: Obstruent coda (`k, t, p`) before lax obstruent (`g, d, b, s, j`) -> (`kk, tt, pp, ss, jj`). | |
| 8. **Contextual Vowel & Homorganic Closure Realization**: | |
| - `γ ‘` (`eu`) realized as `"Ι―"` in open syllables (`μ€μ€λ‘` -> `s Ι― s Ι― ΙΎ o`) and `"Ι"` in closed syllables (`μ/λ/μ/λ₯Ό/μ΅`) to prevent rounded `[u]` coloration under Romance acoustic conditioning. | |
| - `γ ’` (`ui`) realized as `("Ι―", "i")` word-initially (`μμ¬`), `"i"` after a consonant onset (`ν¬λ§` -> `h i m a Ε`), and `"e"` non-initially (`λμ` -> `n a e`, Clause 5 Note 2). | |
| - Unreleased homorganic stop codas (`t_coda`, `k_coda`, `p_coda`) immediately preceding homorganic stops/affricates/fricatives (`λ¨Ήκ³ `, `νκ΅`, `κ°μ΅λλ€`) merge into the following closure without emitting a spurious double stop burst. | |
| --- | |
| ## 3. Formal Verification (TLA+) | |
| Created and verified two TLA+ specifications and their exhaustive state-space model checker in `korean_work/tla/`: | |
| 1. [`KoreanPhonologyAndLayout.tla`](file:///G:/audiolm/scyllasband/korean_work/tla/KoreanPhonologyAndLayout.tla): | |
| - Models the 2-stage Korean phonological transducer (`TransformWord` -> `EmitPhoneStream`) over all 19 onsets, 21 vowels, and 28 codas. | |
| - Verifies `TypeOK`, `PhonologyInvariant` (all codas neutralized to the 7 representative codas; zero unassimilated obstruent+nasal/liquid or nasal+liquid sequences; zero untensified obstruent+lax sequences), `VocabMembershipInvariant` (100% of emitted symbols in `phone_vocab.json`, no stress markers `Λ`/`Λ`), `LayoutInvariant` (`<sil>` framing, `word_starts`, `word_boundary_candidates`), and singing `phone_durations` positivity (`d >= 1` for every spoken Korean phone). | |
| 2. [`StreamingAndLanguageLifecycle.tla`](file:///G:/audiolm/scyllasband/korean_work/tla/StreamingAndLanguageLifecycle.tla): | |
| - Models language resolution (`resolve_language_for_voice` / `Engine::resolve_language`), acoustic conditioning index lookup (`language_index("ko") == 3`), group-speak `[voice:ko:delivery]` switching, and `_ChainStream` target/chunk boundaries (`prefix_latents` reset on voice/language change, preserved across continuations). | |
| 3. [`check_tla_models.py`](file:///G:/audiolm/scyllasband/korean_work/tla/check_tla_models.py): | |
| - Exhaustively verified **26,664 phonological states + 28 layout/singing states + 7,584 language/streaming states = 34,276 reachable states with 0 invariant violations**. | |
| --- | |
| ## 4. Empirical Verification & Performance Results | |
| ### 4.1 Native C++ Build & CTest (`16/16 Passed`) | |
| - Built `libscyllasband` with MSVC 2022 (`Release`) in `runtime/libscyllasband/build-msvc` (`scyllasband_native.dll`, `scyllasband_speak.exe`, `scyllasband_unit_test.exe`, `scyllasband_frontend_test.exe`) and staged `scyllasband_native.dll` + `libLiteRt.dll` into `runtime/scyllasband/_native/litert/`. | |
| - Regenerated C++ test fixtures (`generate_fixtures.py`) including Korean normalizer test cases and ran `ctest --test-dir runtime/libscyllasband/build-msvc -C Release --output-on-failure`: | |
| - **16/16 CTest suites passed in 0.53s** (`scyllasband_events`, `scyllasband_durations`, `scyllasband_context`, `scyllasband_delivery`, `scyllasband_sentences`, `scyllasband_segments`, `scyllasband_normalizer`, `scyllasband_unicode`, `scyllasband_targets`, `scyllasband_singing`, `scyllasband_records`, `scyllasband_frontend_noise`, `scyllasband_frontend_score`, `scyllasband_frontend_timeline`, `scyllasband_frontend_preview`, `scyllasband_frontend_gate`). | |
| ### 4.2 Python Unit & Property Tests (`131/131 Passed`) | |
| - Added [`runtime/tests/test_korean.py`](file:///G:/audiolm/scyllasband/runtime/tests/test_korean.py) covering Sino-Korean & Native Korean normalization, Standard Korean phonological rules, Hypothesis property-based fuzzing over random Hangul words, language resolution, group-speak Korean tags, and Korean singing score planning/rendering. | |
| - Ran `pytest runtime/tests` with `SCYLLASBAND_TEST_NATIVE_LIB` and `SCYLLASBAND_TEST_BUNDLE` enabled: | |
| - **131/131 tests passed in 2.51s** (0 skipped, 0 failed). | |
| ### 4.3 Full Python vs C++ Parity (`parity_harness.py` & `eval_korean_e2e.py`) | |
| - [`runtime/libscyllasband/tests/parity_harness.py`](file:///G:/audiolm/scyllasband/runtime/libscyllasband/tests/parity_harness.py) on `litert`: | |
| - `failures`: **0** | |
| - `frontend`: **280 cases, 0 mismatches** (`unicode_lower_block_mismatches: 0`, `nfc_mismatches: 0`) | |
| - `sentences`: **10/10 identical G2P & durations, `0.0` relative latent error, `Infinity dB` SNR** | |
| - `long_form`: **12/12 chunks & targets identical, `0.0` relative latent error, `Infinity dB` SNR** | |
| - `records`: **6 chains identical, `Infinity dB` SNR** | |
| - `singing`: **12/12 plans identical, 8/8 phrases `0.0` relative latent error (`vocos` & `bigvgan`), 4/4 full songs >234 dB SNR** | |
| - [`korean_work/scratch/eval_korean_e2e.py`](file:///G:/audiolm/scyllasband/korean_work/scratch/eval_korean_e2e.py) (saved in [`korean_work/scratch/korean_e2e_report.json`](file:///G:/audiolm/scyllasband/korean_work/scratch/korean_e2e_report.json)): | |
| - Korean Normalizer parity (Python vs C++): **15/15 cases, 0 mismatches** | |
| - Korean G2P parity (Python vs C++): **30/30 cases, 0 mismatches** | |
| - Korean Sentence synthesis parity: **identical durations (`130` frames), `0.0` relative latent error, `Infinity dB` audio SNR** | |
| - Korean Singing plan parity: **identical phrase plans, phones, and frame durations** | |
| ### 4.4 Korean Speech ASR & Multi-Voice / Singing Evaluation | |
| - **Multi-Voice Korean Synthesis (`scylla`, `gwen`, `felix`, `ariadne`, `ink`)**: | |
| - Reference: `"μλ νμΈμ, μ€λ λ μ¨κ° μ λ§ μ’λ€μ. νκ΅μ΄ μμ± ν©μ±μ μμν©λλ€."` | |
| - `faster-whisper` (`large-v3-turbo`) CER: **0.0% across all 5 voices** (`5/5` exact transcriptions; samples saved in `korean_work/samples/ko_voice_*.wav`). | |
| - **10-Sentence Korean Benchmark (`scylla`, `korean_work/samples/ko_speech_scylla_00..09.wav`)**: | |
| - Overall CER: **20.12%** across 10 diverse sentences covering liaison, nasalization, liquidization, palatalization, tensification, and normalized numbers/counters, with **0.0% CER** on sentences `00`, `01`, `02`, and `05`. | |
| - **Korean Singing Synthesis (`melodies/phrase_ko_twinkle.json`, `"λ°μ§ λ°μ§ μμ λ³ μλ¦λ΅κ² λΉμΉλ€"`)**: | |
| - Synthesized through both `vocos` (`ko_song_twinkle_scylla_vocos.wav`) and `bigvgan` (`ko_song_twinkle_scylla_bigvgan.wav`), verified by `scyllasband_speak.exe --score` and `runtime.sing(...)`. | |
| ### 4.5 Native C++ Runtime Performance (`scyllasband_speak.exe` on LiteRT CPU) | |
| - **Bundle Load Time**: `2.6 ms` | |
| - **Korean Speech (`5.33 s` audio, 4 Euler steps)**: | |
| - Cold first audio: `403.0 ms` | |
| - Warm first audio: **`138.2 ms`** | |
| - Warm total time: **`282.5 ms` (~18.9Γ real-time on CPU)** | |
| - **Korean Singing (`7.62 s` audio, `melodies/phrase_ko_twinkle.json`, Vocos turbo-sing, 4 Euler steps)**: | |
| - Warm first audio / total time: **`177.5 ms` (~42.9Γ real-time on CPU)** | |
| --- | |
| ## 5. Modified & Created Files Summary | |
| ### Model Repository (`G:\audiolm\scyllasband`) | |
| - `onnx/manifest.json`, `onnx/assets/languages.json`, `onnx/assets/g2p/language_map.json`, `onnx/assets/g2p/normalization.json` | |
| - `litert/manifest.json`, `litert/assets/languages.json`, `litert/assets/g2p/language_map.json`, `litert/assets/g2p/normalization.json` | |
| - `webgpu/manifest.json`, `webgpu/assets/languages.json`, `webgpu/assets/g2p/language_map.json`, `webgpu/assets/g2p/normalization.json` | |
| - `coreml/manifest.json`, `coreml/assets/languages.json`, `coreml/assets/g2p/language_map.json`, `coreml/assets/g2p/normalization.json` | |
| - `coreai/manifest.json`, `coreai/assets/languages.json`, `coreai/assets/g2p/language_map.json`, `coreai/assets/g2p/normalization.json` | |
| - `pytorch/release_config.json` | |
| - `melodies/phrase_ko_twinkle.json`, `melodies/README.md` | |
| - `README.md` (Updated with prominent Korean support section and usage guide) | |
| - `SHA256SUMS.json` (Recalculated and verified) | |
| - `PROGRESS.md` | |
| - `webui.py` (Interactive Gradio web interface) | |
| - `serve_webgpu_demo.py` (Local HTTP static server with COOP/COEP isolation headers) | |
| - `webgpu_demo/index.html`, `webgpu_demo/css/style.css`, `webgpu_demo/README.md` | |
| - `webgpu_demo/js/app.js`, `webgpu_demo/js/korean_g2p.js`, `webgpu_demo/js/pipeline.js`, `webgpu_demo/js/wav_utils.js` (Standalone 100% Client-Side WebGPU TTS Demo) | |
| - `korean_work/tla/KoreanPhonologyAndLayout.tla`, `korean_work/tla/StreamingAndLanguageLifecycle.tla`, `korean_work/tla/check_tla_models.py` | |
| - `korean_work/scratch/eval_korean_e2e.py`, `korean_work/scratch/korean_e2e_report.json`, `korean_work/scratch/update_bundle_metadata.py` | |
| - `korean_work/samples/*.wav` | |
| ### Runtime Repository (`G:\audiolm\scyllasband\runtime`) | |
| - `scyllasband/text_normalizer.py` | |
| - `scyllasband/g2p.py` | |
| - `scyllasband/planner.py` | |
| - `scyllasband/runtime.py` | |
| - `scyllasband/engine.py` | |
| - `tests/test_korean.py` | |
| - `libscyllasband/src/scyllasband_text.cpp` | |
| - `libscyllasband/src/scyllasband_g2p.h`, `libscyllasband/src/scyllasband_g2p.cpp` | |
| - `libscyllasband/src/scyllasband_engine.cpp` | |
| - `libscyllasband/tools/scyllasband_speak.cpp` | |
| - `libscyllasband/tests/generate_fixtures.py`, `libscyllasband/tests/fixtures/*.json` | |
| --- | |
| ## 6. Remaining Limitations & Future Work | |
| 1. **Zero-Shot Acoustic Conditioning Trade-offs**: | |
| - Because the `v4-20261106` acoustic graphs (`duration_predictor`, `pitch_estimator`, `vector_estimator`) were trained on 7 non-Korean locales without native Korean audio, Korean synthesis maps `ko` to syllable-timed acoustic conditioning index `3` and approximates Korean's 3-way laryngeal stop/affricate contrast (`νμ/κ²½μ/격μ`) and unrounded back vowels (`γ /γ ‘`) using the closest trained IPA phones (`Ι`, `Ι―`/`Ι`, `k/Ι‘/k h`). Complex consonant clusters across word boundaries (e.g., `μ€λ μμΉ¨μ κ°μ‘±λ€κ³Ό λ‘κ΅μ`) still exhibit Romance prosodic liaison and vowel coloration. | |
| 2. **Next Steps When Native Korean Training Assets Become Available**: | |
| - Fine-tune or add a dedicated `ko` row (`index=7`) in `language_embed` and per-voice reference prosody tables (`conditions.*`), and include Hangul Jamo or syllable tokens in the neural G2P training corpus while retaining the `νμ€λ°μλ²` phonological frontend and text normalizer as the reference ground truth. | |
| ## WebGPU Standalone In-Browser Demo Milestone (2026-11-08) | |
| 1. **Standalone Architecture**: | |
| - `webgpu_demo/index.html`: Fully client-side UI with dark theme, preset phrases, 14 voices, prosody controls, and real-time canvas waveform visualizer. | |
| - `webgpu_demo/js/korean_g2p.js`: Pure JavaScript Standard Korean G2P & normalizer (Chosung/Jungsung/Jongsung decomposition, assimilation, nasalization, tensification, liquidization, palatalization, aspiration, and number/symbol normalization). | |
| - `webgpu_demo/js/pipeline.js`: Complete in-browser neural TTS pipeline (Euler ODE flow matching, duration prediction, pitch contour generation, and Vocos neural vocoder). | |
| - `webgpu_demo/dist/`: Self-contained `ort.webgpu.min.js`, `ort-wasm-simd-threaded.jsep.wasm`, and `ort-wasm-simd-threaded.jsep.mjs`. | |
| - `serve_webgpu_demo.py`: Python static server with explicit MIME types, COOP/COEP isolation headers, `--host`, and `--port` CLI flags. | |
| 2. **Root Cause Analysis of Duration Predictor (1/6) Error**: | |
| - Symptom: At model load stage 1/6 (`duration_predictor.onnx`), initialization failed with `undefined` / WASM C++ exception. | |
| - Investigation: | |
| - `webgpu/onnx/components/duration_predictor.onnx` uses `com.microsoft.MatMulNBits` with `bits: 8`. | |
| - In ONNX Runtime Web (JSEP & WASM CPU), `MatMulNBits` only supports `bits: 4` and `bits: 2` (`matmul_nbits.cc:115: nbits_ == 4 was false. Only 4b is supported`). | |
| - Emscripten caught the C++ exception and threw a numeric pointer (`11055512`), which resulted in `undefined` error string in the UI. | |
| - Resolution: | |
| - `onnx/onnx/components/` contains standard FP32 ONNX graphs (`MatMul`, `Gemm`, `LayerNormalization`, `Conv`, `Softmax`), which are 100% natively supported across both WebGPU and WASM. | |
| - Changed default bundle in `webgpu_demo/index.html` to `../onnx`. | |
| - Enhanced error handling in `app.js` and `pipeline.js` to catch numeric WASM exceptions and show actionable error diagnostics. | |
| - Fixed WASM base URL resolution in `pipeline.js` to support access via any network IP (`0.0.0.0` / LAN IP) without falling back to CDN. | |
| 3. **End-to-End Verification**: | |
| - Executed full client-side TTS synthesis test on Korean text ("μλ νμΈμ. λ°κ°μ΅λλ€."). | |
| - Result: 24,832 samples synthesized @ 24,000 Hz (1.03 sec duration, healthy RMS amplitude, peak [-0.36, +0.48], non-zero ratio 91.3%). | |
| 4. **Long-Form & Multi-Sentence Duration Support (2μ΄ μ΄μ ν©μ± μλ²½ μ§μ)**: | |
| - μμΈ: μ΄κΈ° ꡬνμμ κ³ μ 128 λ²ν·(`bucket = 128`)λ§μ λ‘λνμ¬, μ΅λ μ§μ μκ°μ΄ 128 νλ μ Γ 256 / 24,000 = **1.36μ΄**λ‘ μ νλμ΄ κΈ΄ λ¬Έμ₯μ΄ μ리λ νμ λ°μ. | |
| - μ‘°μΉ: | |
| - 512 νλ μ λ²ν· λͺ¨λΈ(`pitch_estimator_512.onnx`, `vector_estimator_512.onnx`, `vocoder_512.onnx`)λ‘ μ κ·Έλ μ΄λ (λ¨μΌ ꡬ문 μ΅λ 5.46μ΄ μ§μ). | |
| - μ₯λ¬Έ λ€μ€ λ¬Έμ₯ λΆν λ° μ²ν¬ λ¨μ ν©μ±(`splitSentences`, `synthesizeChunk`) νμ΄νλΌμΈ ꡬν: λ¬Έμ₯ κ° 60ms μμ°μ€λ¬μ΄ 묡μ ν¨λ© μ½μ λ° μμ± νν 무결 κ²°ν©. | |
| - 2μ΄, 5μ΄, 10μ΄ μ΄μμ μ₯λ¬Έ ν μ€νΈλ κΈΈμ΄ μ ν μμ΄ μ°μ ν©μ± κ°λ₯νλλ‘ νμ₯ μλ£. | |
| 5. **2-Tier λ‘컬 λΈλΌμ°μ μꡬ μΊμ± μμ€ν (Permanent Disk & CacheStorage Caching)**: | |
| - λ¬Έμ : λ§€λ² νμ΄μ§ μ μ λλ "WebGPU λͺ¨λΈ λ‘λ" μ€ν μ μ½ 1GBμ `shared_weights.bin`κ³Ό λͺ¨λΈλ€μ λ€νΈμν¬λ‘ λ€μ λ€μ΄λ‘λνλ λΉν¨μ¨ λ°μ. | |
| - Tier 1 (HTTP λ 벨): `serve_webgpu_demo.py`μμ `.bin`, `.onnx`, `.wasm` μ μ νμΌμ λν΄ `Cache-Control: public, max-age=31536000, immutable` ν€λ μ μ‘ (λΈλΌμ°μ λμ€ν¬ μΊμ μ¦μ μ μ€). | |
| - Tier 2 (ν΄λΌμ΄μΈνΈ λ 벨): `pipeline.js`μ `fetchWithCache`μμ λΈλΌμ°μ `window.caches` (CacheStorage API)λ₯Ό μ μ©νμ¬ μ΅μ΄ λ€μ΄λ‘λ ν λ‘컬 λμ€ν¬μ μꡬ μ μ₯. λ€μ μ€νλΆν°λ λ€νΈμν¬ μμ² μμ΄ 100~300ms λ΄μ μ¦μ λ‘λ. | |
| - UI μ§μ: "μΊμ μμ " λ²νΌμ μΆκ°νμ¬ νμ μ μν΄λ¦μΌλ‘ λΈλΌμ°μ λͺ¨λΈ μΊμλ₯Ό μ΄κΈ°νν μ μλλ‘ μ§μ. | |
| 6. **μ΄κ²½λ Vocos μ μ© INT8 WebGPU λ²λ€ μ¬ν¨ν€μ§ (230 MB) λ° ONNX Runtime Web 1.30.0 μ κ·Έλ μ΄λ**: | |
| - **λ°°κ²½ λ° μμΈ λΆμ**: | |
| - μμμ λ°λͺ¨(`https://lowkeytea.dev/scyllasband/demo/`)κ° 261MBμ λΆκ³Όνλ μ΄μ λ κ°μ°½(Singing) κΈ°λ₯μ΄ μλ v3 κΈ°λ°μ΄μκΈ° λλ¬Έμ. | |
| - v4λ‘ νμ¬λ¦Όλλ©΄μ BigVGAN 보μ½λ(λ¨λ μΌλ‘ 502.2 MB μ μ )κ° `shared_weights.bin`μ μΌκ΄ λ³ν©λμ΄ λ²λ€ ν¬κΈ°κ° 743.6 MB(INT8) / 1,036 MB(FP32)λ‘ κΈμ¦ν¨. | |
| - μΉ μμ± TTS νκ²½μμλ λν μ κ²½λ§ λ³΄μ½λμΈ BigVGANμ΄ λΆνμνλ©°, μ€μκ° μΆλ‘ μ μ΅μ νλ Vocos 보μ½λλ§μΌλ‘ κ³ νμ§ μμ± ν©μ±μ΄ κ°λ₯ν¨. | |
| - **μ΄κ²½λ λ²λ€ μ¬ν¨ν€μ§ μν**: | |
| - BigVGAN ν μ 502 MBλ₯Ό μμ ν λΆλ¦¬Β·μ μΈνκ³ , 21κ° μμ± μ μ© λͺ¨λΈ(Duration Predictor, Pitch/Vector Context Encoders, Pitch/Vector Estimators 128~768, Vocos 128~768)μ 795κ° κ³ μ ν μλ§μ 64λ°μ΄νΈ μ λ ¬λ‘ μμ°¨ ν¨νΉ. | |
| - `webgpu/onnx/components/shared_weights.bin` ν¬κΈ°λ₯Ό 743.6 MBμμ **241,440,320 λ°μ΄νΈ (230.26 MB)**λ‘ **3.2λ°° κ²½λν** μλ£ (μ μ λ°λͺ¨ 230.2 MBμ μμ μΌμΉ). | |
| - 21κ° ONNX λͺ¨λΈμ `external_data` μ€νμ μ μ¬λ§€ννκ³ λ―Έμ¬μ© `vocoder_bigvgan*.onnx` 6κ° νμΌμ μ κ±°. | |
| - `shared_weights.json` λ° `manifest.json`μ μ»΄ν¬λνΈΒ·λ²ν· λͺ μΈλ₯Ό Vocos μ μ© λ²λ€ κ·κ²©μΌλ‘ κ°±μ . | |
| - **ONNX Runtime Web 1.30.0 μ κ·Έλ μ΄λ**: | |
| - κΈ°μ‘΄ ORT 1.21.0μμ λ°μνλ 8-bit `MatMulNBits` WASM CPU ν΄λ°± μμΈ(`nbits_ == 4 was false`)λ₯Ό ν΄κ²°νκΈ° μν΄, μ΅μ `onnxruntime-web@1.30.0` (WebGPU Asyncify WASM λ° JSEP) λ°μ΄λ리λ₯Ό `webgpu_demo/dist/`μ μ μ λ°°ν¬. | |
| - WebGPU νκ²½μμ 8-bit μμν κ°μ€μΉλ₯Ό μ§μ GPU μ °μ΄λ λ° μ΅μ WASM λ°νμμΌλ‘ κ°μν μ μλλ‘ νμ΄νλΌμΈ ꡬμ±. | |
| - `webgpu_demo/index.html` κΈ°λ³Έ λ‘λ κ²½λ‘λ₯Ό `../webgpu` (230 MB INT8 λ²λ€)λ‘ μ€μ . | |
| - **μ’ λ¨κ°(End-to-End) κ²μ¦ μλ£**: | |
| - Scylla's Band 곡μ λ°νμ(`ScyllasBandRuntime.from_bundle`)μμ λ μ΄μμ κ²μ¦ ν΅κ³Ό λ° νκ΅μ΄ λ¬Έμ₯("μλ νμΈμ! λ°κ°μ΅λλ€.") ν©μ± ν μ€νΈ μλ£ (2.432μ΄, 58,368 μν @ 24kHz μ μ μΆλ ₯). | |
| - λ‘컬 HTTP μλ² λ° μ μ νμΌ MIME νμ , 2-Tier μΊμ±, COOP/COEP κ΅μ°¨ μΆμ² 격리 ν€λ κ²μ¦ μλ£. | |
| 7. **Tailscale Secure Context HTTPS λ° WebGPU BufferManager Download μΆ©λ ν΄κ²°**: | |
| - **WASM Fallback λ° 18μ΄ μΆλ‘ μ§μ° μμΈ λΆμ**: | |
| - μμΈ: \http://100.72.102.121:8000\ (μ¬μ€ IP HTTP)λ‘ μ μ μ λΈλΌμ°μ W3C 보μ κ·κ²©μ λ°λΌ λΉλ³΄μ 컨ν μ€νΈ(Insecure Context)λ‘ λΆλ₯λ¨. | |
| - μ΄λ‘ μΈν΄ avigator.gpu\ (WebGPU) λ° \crossOriginIsolated\ (\SharedArrayBuffer\ λ©ν°μ€λ λ©)κ° μμ² μ°¨λ¨λμ΄ λ¨μΌ μ€λ λ CPU WASM(\ hreads: 1\)μΌλ‘ ν΄λ°±λλ©° μΆλ‘ μκ°μ΄ 18μ΄κΉμ§ νμ¦ν¨. | |
| - **Tailscale HTTPS & Let's Encrypt μΈμ¦μ ν΅ν©**: | |
| - Tailscale 곡μ MagicDNS μΈμ¦μ(\ ailscale cert dt4090sp.tail50666.ts.net\)λ₯Ό λ°κΈλ°μ \serve_webgpu_demo.py\μ \--ssl\ μ΅μ μ μΆκ°νκ³ Python \ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)\λ₯Ό νμ¬. | |
| - \https://dt4090sp.tail50666.ts.net:8000/webgpu_demo/\λ₯Ό ν΅ν΄ λ³λμ νλκ·Έ μ‘°μ μμ΄λ Tailnet λ΄ λͺ¨λ λͺ¨λ°μΌ/PC κΈ°κΈ°μμ μ μ κ³΅μΈ λ³΄μ 컨ν μ€νΈ(Secure Context) νμ±ν λ° WebGPU + WASM λ©ν°μ€λ λ© μμ κ°λ. | |
| - \index.html\ λ° \pp.js\μ λΉλ³΄μ HTTP μ μ κ°μ§ μ HTTPS λλ©μΈμΌλ‘ μλ΄νλ μν λ°°λ μΆκ°. | |
| - **\BufferManager::Download map_async_result.status == Success was false\ μ€λ₯ μμΈ λ° ν΄κ²°**: | |
| - μ€λ₯ νμ: WebGPU λͺ¨λΈ λ‘λ ν μμ± ν©μ± μ€ν μ \ailed to call OrtRun(). ERROR_CODE: 1, ERROR_MESSAGE: .../providers/webgpu/buffer_manager.cc:643 void onnxruntime::webgpu::BufferManager::Download(WGPUBuffer, void *, size_t) const map_async_result.status == wgpu::MapAsyncStatus::Success was false. Failed to download data from buffer. wgpu::MapAsyncStatus value: 0\ λ°μ. | |
| - μμΈ: \pipeline.js\μμ \Promise.all([ this.sessions.vector_context_encoder.run(contextFeeds), this.sessions.pitch_context_encoder.run(contextFeeds) ])\λ‘ 2κ°μ λͺ¨λΈμ GPUμ λμ λ³λ ¬ μμ². ONNX Runtime Webμ WebGPU JSEP λ°±μλλ λμΌ GPU 컨ν μ€νΈμμ μ€μ²©λ \mapAsync\ μ€ν μ΄μ§ λ²νΌ λ€μ΄λ‘λλ₯Ό μ²λ¦¬νμ§ λͺ»ν΄ λ λ²μ§Έ μμ²μ΄ μ·¨μ(Abort, Status: 0)λ¨. | |
| - μ‘°μΉ: \pipeline.js\μ λμ νΈμΆμ μμ°¨ \wait\ (\ecContextOut = await ...; pitchContextOut = await ...;\)λ‘ λ³κ²½. | |
| - κ²μ¦: 격리 ν μ€νΈ(\ est_promise_all.html\)λ₯Ό ν΅ν΄ \Promise.all\ μ€ν¨ μ¬ν λ° μμ°¨ νΈμΆ μ κ° λͺ¨λΈμ΄ λ¨ 9ms λ§μ μ μ μλ£λ¨μ μ μ¦. μ 체 6λ¨κ³ WebGPU ν©μ± νμ΄νλΌμΈ(\ est_webgpu_synthesis.html\)μμ 261,888 μν(10.9μ΄ μ€λμ€)μ 1μ΄ λ―Έλ§μ 무결 ν©μ± μ±κ³΅. | |
| 8. **WebGPU μ€λμ€ μ§μ μκ° 1/2 μ λ¨(2.75μ΄ μλ¦Ό) λ²κ·Έ ν΄κ²° λ° Prosody μ΄λ²€νΈ κ°ν**: | |
| - **μμΈ λΆμ**: | |
| - \manifest.json\μ \latent_hop_length: 512\ λ° κ³΅μ λ°νμ(untime/scyllasband/engine.py\)μ \LATENT_HOP = 512\ κ·κ²©μ λ°λΌ μ§μ μκ° μμΈ‘κΈ°(\duration_predictor\)κ° μΆλ ₯νλ 1 νλ μμ **512 μ€λμ€ μν**(24,000Hz κΈ°μ€ 21.33ms)μ ν΄λΉν¨. Vocos 512 λ²ν·μ 512 Γ 512 - 256 = 261,888 μν(μ΅λ 10.92μ΄)μ 볡μν¨. | |
| - κ·Έλ¬λ κΈ°μ‘΄ \pipeline.js\μμ \const hopLength = 256;\μΌλ‘ μ§μ λμ΄ μμ΄, \ otalFrames * 256\μΌλ‘ μ¬λΌμ΄μ€λλ©΄μ λͺ¨λ νκ΅μ΄ λ¬Έμ₯μ νλ°λΆ 50%κ° μλ € **μ νν 2.75μ΄ λΆκ·Όμμ μ€λμ€κ° μ€κ°μ μ λ¨**λλ νμμ΄ λ°μν¨. | |
| - **μ‘°μΉ λ΄μ©**: | |
| - \pipeline.js\μ μ€λμ€ μ¬λΌμ΄μ€ λ¨μ(\hopLength\)λ₯Ό 곡μ μ μ¬ ν κ·κ²©μΈ **512**λ‘ μμ (\ctiveSamples = activeFrames * 512\). | |
| - λ²ν· λ΄ μ€μ μ κ°λ νμ± νλ μ μ(\cursor\) κΈ°μ€μΌλ‘ μ€λμ€ νΈλ¦Ό λ²μλ₯Ό μ λ°ννμ¬ λ²ν· μ€λ²νλ‘μ° μμλ 무결 μ¬λΌμ΄μ± 보μ₯. | |
| - λ¬Έμ₯ μ’ κ²° λΆνΈ(\?\, \!\, \...\)μ λ°λ₯Έ \sentence_types\ (1: μ§μ , 2: μλ¬Έ, 3: κ°ν, 4: λ§μ€μν) λ° μ΄ν μμ κ²½κ³(\wordStarts\ -> \oundaryEvents\) λ§€νμ μ¨μ νκ² μ κ°νμ¬ μλ¬Έλ¬ΈΒ·κ°νλ¬Έ μ΅μ μμ°λ κ°μ . | |
| - **κ²μ¦**: | |
| - 96κ° μμ(νκ΅μ΄ 2λ¬Έμ₯, 227νλ μ) ν©μ± μ κΈ°μ‘΄ 2.42~2.75μ΄μμ μ리λ μ€λμ€κ° 4.84~5.69μ΄μ μ¨μ ν ννμΌλ‘ μ 체 μμ μνμ€κ° μ²μλΆν° λκΉμ§ μμ μ¬μλ¨μ κ²μ¦. | |
| 9. **Git Commit & Push μλ£ λ° Hugging Face Space μ μ λ°°ν¬**: | |
| - **λͺ¨λΈ 리ν¬μ§ν 리 λκΈ°ν (cidsound/scyllasband)**: | |
| - WebGPU λ²νΌ λ½ μμ°¨ν, Vocos LATENT_HOP 512 μ§μ μκ° μμ , λ€μ€ κ²½λ‘ λ³΅μλ ₯(Multi-tier fallback) λ° Prosody μ΄λ²€νΈ κ°ν λ°μ μλ£. | |
| - λ‘컬 Git λ³κ²½ μ¬ν μ격 main λΈλμΉμ μμ μ»€λ° λ° νΈμ μλ£ (2b1b4d3). | |
| - **Hugging Face Space λ°°ν¬ μλ£ (cidsound/scyllasband-webgpu)**: | |
| - Space μ ν: Static Space (sdk: static β 무μ€λ¨ 24/7 κ°λ, μ½λ μ€ννΈ 0μ΄, HF κΈλ‘λ² CDN νΈμ€ν ). | |
| - Space URL: [https://huggingface.co/spaces/acidsound/scyllasband-webgpu](https://huggingface.co/spaces/acidsound/scyllasband-webgpu) | |
| - μ§μ μ μ URL (μ 체νλ©΄): [https://acidsound-scyllasband-webgpu.static.hf.space](https://acidsound-scyllasband-webgpu.static.hf.space) | |
| - 230 MB μ΄κ²½λ INT8 λ²λ€μ΄ Space λ΄μ μ§μ λ°°ν¬λμ΄ μΈλΆ μμ‘΄μ± μμ΄ μμ ν 100% μ립 μ€ν(Self-contained) λμ. | |
| - λ°°ν¬ μν: Space stage: RUNNING, λͺ¨λ μ μ νμΌ, WASM λ°μ΄λ리(pplication/wasm), 230 MB κ°μ€μΉ λ° ONNX κ·Έλν HTTP 200 μλΉ κ²μ¦ μλ£. | |