scyllasband / PROGRESS.md
acidsound's picture
docs: update PROGRESS.md with Hugging Face Space deployment milestone
c35977c
|
Raw History Blame Contribute Delete
30 kB
# Korean Support for Scylla's Band β€” Engineering Progress (`PROGRESS.md`)
## 1. Objective & Status
- **Objective**: Deliver reliable, end-to-end Korean (`ko` / `ko_kr`) speech and singing support for **Scylla's Band (`v4-20261106`)** across both the model repository (`spybyscript/scyllasband`) and the reference/native runtime (`lowkeytea/scyllasband`, located in `runtime/`), preserving compact deployment, fast local inference, exact Python/C++ parity, and backward compatibility with all 14 existing voices, 7 trained locales (`en_us`, `en_gb`, `es`, `it`, `fr`, `de`, `vi`), and 6 bundle flavors (`litert`, `onnx`, `webgpu`, `coreml`, `coreai`, `pytorch`).
- **Status**: **Complete and verified end-to-end** across formal TLA+ specifications, Python (`scyllasband`) and C++ (`libscyllasband`) implementations, all 6 bundle flavors, CTest (`16/16`), pytest (`131/131`), the full Python/C++ parity harness (`0` failures), Korean ASR intelligibility benchmarks, and Korean singing synthesis (`vocos` and `bigvgan`).
---
## 2. Confirmed Architectural Findings & Engineering Decisions
### 2.1 Model & Graph Contract Analysis (`v4-20261106`)
1. **Phone Vocabulary (`phone_vocab.json`, 87 tokens, IDs `0..86`)**:
- Structural/control tokens (`0..14`): `<pad>`, `<unk>`, `<sil>`, `<pause_comma>`, `<pause_semicolon>`, `<pause_colon>`, `<pause_dash>`, `<ellipsis>`, `<end_stmt>`, `<end_question>`, `<end_exclaim>`, `<ctx_sentence_start>`, `<ctx_continuation>`, `<ctx_sentence_end>`, `<ctx_chunk_continue>`.
- Tone digits (`15..20`): `1`, `2`, `4`, `5`, `6`, `7`.
- IPA vowels, glides, and consonants (`21..77, 84..86`) and non-acoustic modifier bits (`78..83`).
- **Important Vocabulary Constraint**: `phone_vocab.json` contains postalveolar `"Κƒ"` (`id=69`) and `"Κ’"` (`id=74`), not alveolo-palatal `"Ι•"` or `"Κ‘"`. Using `"Κƒ"` and `"Κ’"` for Korean `γ…ˆ/γ…‰/γ…Š` (`t Κƒ` / `d Κ’` / `t Κƒ h`) and palatalized `γ……/γ…†` (`Κƒ` / `Κƒ Κƒ` before `γ…£/γ…‘/γ…•/γ…›/γ… /γ…–/γ…Ÿ`) keeps 100% of emitted Korean phones strictly inside the 87-symbol vocabulary.
2. **Neural G2P Graph vs Deterministic Korean Phonological G2P**:
- The exported neural G2P graphs (`g2p.onnx`, `g2p.tflite`, `g2p.mlmodelc`, `g2p.aimodel`, `g2p.pt`) are char-level CTC models (`char_repeats = 4`) trained only on `<de>`, `<en_gb>`, `<en_us>`, `<es>`, `<fr>`, `<it>`, `<vi>` with no `<ko>` symbol and no Hangul coverage in `tokenizer.json`.
- Therefore, when `language == "ko"` or when any input segment contains Hangul (`U+AC00..U+D7A3` or Compatibility Jamo `U+3131..U+3163`), `G2PFrontend._predict_segment` (Python) and `G2P::predict_segment` (C++) route Hangul tokens through a deterministic, context-sensitive Standard Korean (`ν‘œμ€€λ°œμŒλ²•`) phonological G2P, while delegating embedded multi-letter Latin words to the voice's English G2P/lexicon and spelling out 1–3 letter uppercase initialisms (`AI` -> `에이아이`, `TTS` -> `ν‹°ν‹°μ—μŠ€`, `GPU` -> `μ§€ν”Όμœ `, `MP3` -> `μ— ν”Όμ‚Ό`, `4K` -> `사케이`) in Korean.
3. **Acoustic Conditioning (`language_id = 3`)**:
- Precompiled graphs in `litert`, `onnx`, `webgpu`, `coreml`, and `coreai` have fixed `[7, D]` `language_embed` and `[98, ...]` (`14 voices Γ— 7 locales`) voice/locale conditioning tables.
- Empirical probing across all 7 trained `language_id` indices (`0..6`) with `faster-whisper` (`large-v3-turbo`) confirmed that mapping `"ko"` to acoustic conditioning index `3` (`it`, syllable-timed pure-vowel Romance conditioning without English stress-timed schwa reduction or Vietnamese lexical tone contours) yields the cleanest Korean vowel formants and syllable timing across all 14 voices.
### 2.2 Korean Spoken-Text Normalizer (`text_normalizer.py` & `scyllasband_text.cpp`)
- Preserved `SPOKEN_TEXT_NORMALIZER_SHA256` (`d8fd1b2415d0d100d1c70237a4469117ee6b73d0e4da703e4654c83192779cc4`) and 100% backward compatibility with all existing languages.
- Added `"ko"`, `"ko-kr"`, `"ko_kr"`, `"kor"`, `"korean"` alias resolution in `normalize_language_code`.
- Implemented identical Korean normalization rules in Python and C++:
- **Dual Numeral System**:
- Sino-Korean (`영, 일, 이, μ‚Ό, 사, 였, 윑, μΉ , νŒ”, ꡬ, μ‹­, λ°±, 천, 만, μ–΅, μ‘°`) with Korean `1`-elision before `μ‹­/λ°±/천` (`10` -> `μ‹­`, `15,000` -> `만 였천`, `100,000,000` -> `일얡`).
- Native Korean (`ν•œ/ν•˜λ‚˜, 두/λ‘˜, μ„Έ/μ…‹, λ„€/λ„·, λ‹€μ„―, ..., 슀무/슀물, μ„œλ₯Έ, λ§ˆν”, μ‰°, 예순, 일흔, μ—¬λ“ , 아흔`) automatically triggered before 25 Native Korean counters (`개, λͺ…, λΆ„, μ‚¬λžŒ, 마리, μ‚΄, μ‹œ, μ‹œκ°„, 달, μž”, 병, ꢌ, μž₯, λŒ€, 채, 케레, 벌, 솑이, 자루, 그루, 쑰각, μƒμž, λ°”κ΅¬λ‹ˆ, 봉지, 번`).
- Ordinals (`1번째` -> `첫 번째`, `2번째` -> `두 번째`) and Sino-Korean month exceptions (`6μ›”` -> `μœ μ›”`, `10μ›”` -> `μ‹œμ›”`).
- **Structured Entities**:
- ISO dates (`2026-11-08` -> `μ΄μ²œμ΄μ‹­μœ‘λ…„ 십일월 νŒ”μΌ`).
- Times (`3:30 PM` -> `μ˜€ν›„ μ„Έ μ‹œ μ‚Όμ‹­ λΆ„`, `7:05 AM` -> `μ˜€μ „ 일곱 μ‹œ 였 λΆ„`, `14:00` -> `십사 μ‹œ`).
- Currencies (`β‚©15,000` -> `만 였천 원`, `$3.50` -> `μ‚Ό λ‹¬λŸ¬ κ³Ό μ˜€μ‹­ μ„ΌνŠΈ`, `40€` -> `사십 유둜`, `Β£20` -> `이십 νŒŒμš΄λ“œ`, `Β₯500` -> `였백 μ—”`).
- Fractions (`3/4` -> `사 λΆ„μ˜ μ‚Ό`, `1/2` -> `이 λΆ„μ˜ 일`), decimals (`3.14` -> `μ‚Ό 점 일사`), percentages (`95.5%` -> `κ΅¬μ‹­μ˜€ 점 였 νΌμ„ΌνŠΈ`), temperatures/degrees (`-5Β°C` -> `λ§ˆμ΄λ„ˆμŠ€ 였 도씨`, `20Β°` -> `이십 도`), symbols (`=` -> `이퀄`, `+` -> `ν”ŒλŸ¬μŠ€`), and alphanumeric splitting (`MP3와 4K` -> `μ—  ν”Ό μ‚Ό 와 사 케이`).
### 2.3 Standard Korean Phonological G2P (`g2p.py` & `scyllasband_g2p.cpp`)
- Implemented Hangul syllable decomposition (`U+AC00..U+D7A3` -> 19 Choseong, 21 Jungseong, 28 Jongseong) + Compatibility Jamo (`γ„±..γ…Ž`, `ㅏ..γ…£`) handling.
- Implemented ordered Standard Korean (`ν‘œμ€€λ°œμŒλ²•`) phonological rules across adjacent syllables:
1. **Palatalization (κ΅¬κ°œμŒν™”, Clause 17)**: `γ„·/γ…Œ/γ„Ύ` before vowel-initial `γ…£/γ…‘/γ…•/γ…›/γ… /γ…–` -> `γ…ˆ/γ…Š/γ„Ή+γ…Š` (`같이` -> `[κ°€μΉ˜]`, `ꡳ이` -> `[ꡬ지]`, `λ‹«νžˆλ‹€` -> `[λ‹€μΉ˜λ‹€]`).
2. **Liaison (μ—°μŒλ²•μΉ™, Clauses 13–14)**: Single codas move to following null onset (`ν•œκ΅­μ–΄` -> `[ν•œκ΅¬κ±°]`); complex codas (`κ²Ήλ°›μΉ¨`: `γ„³, γ„΅, γ„Ί, γ„», γ„Ό, γ„½, γ„Ύ, γ„Ώ, γ…„`) split across the boundary (`읽어` -> `[일거]`, `앉아` -> `[μ•ˆμž]`, `μ—†μ–΄` -> `[업써]`).
3. **`γ…Ž` Aspiration & Elision (γ…Ž μΆ•μ•½ 및 νƒˆλ½, Clause 12)**: `γ…Ž/γ„Ά/γ…€` + `γ„±/γ„·/γ…‚/γ…ˆ` (or reverse) -> `γ…‹/γ…Œ/ㅍ/γ…Š` (`μ’‹λ‹€` -> `[쑰타]`, `μž…ν•™` -> `[이팍]`, `독립기념관` / `많이` -> `[λ§ˆλ‹ˆ]`, `μ’‹μ•„μš”` -> `[μ‘°μ•„μš”]`), `γ…Ž/γ„Ά/γ…€` + `γ……` -> `γ…†`.
4. **Complex Coda Reduction & Neutralization (음절의 λμ†Œλ¦¬ κ·œμΉ™, Clauses 8–11)**: All 27 codas neutralize to the 7 representative codas (`k, n, t, l, m, p, Ε‹`), including the Clause 11 exception where `γ„Ί` before `γ„±` reduces to `γ„Ή` + `γ„²` (`λ§‘κ³ ` -> `[말꼬]`, `읽고` -> `[일꼬]`, whereas `λ§‘λ‹€` -> `[막따]`).
5. **Liquidization (μœ μŒν™”, Clause 20)**: `γ„΄+γ„Ή` and `γ„Ή+γ„΄` -> `γ„Ή+γ„Ή` (`신라` -> `[싀라]`, `μ„€λ‚ ` -> `[μ„€λž„]`).
6. **Nasalization (λΉ„μŒν™”, Clauses 18–19)**: Obstruent coda (`k, t, p`) before nasal (`n, m`) -> (`Ε‹, n, m`) (`κ΅­λ¬Ό` -> `[ꢁ물]`, `λ¨ΉλŠ”` -> `[λ©λŠ”]`, `μžˆλŠ”` -> `[μΈλŠ”]`); nasal (`m, Ε‹`) + `r` -> `m/Ε‹ + n` (`μ’…λ‘œ` -> `[μ’…λ…Έ]`); obstruent (`k, t, p`) + `r` -> (`Ε‹, n, m`) + `n` (`독립` -> `[동닙]`, `백만` -> `[λ±…λ§Œ]`).
7. **Post-Obstruent Tensification (κ²½μŒν™”, Clause 23)**: Obstruent coda (`k, t, p`) before lax obstruent (`g, d, b, s, j`) -> (`kk, tt, pp, ss, jj`).
8. **Contextual Vowel & Homorganic Closure Realization**:
- `γ…‘` (`eu`) realized as `"Ι―"` in open syllables (`슀슀둜` -> `s Ι― s Ι― ΙΎ o`) and `"Ι™"` in closed syllables (`은/λŠ”/을/λ₯Ό/슡`) to prevent rounded `[u]` coloration under Romance acoustic conditioning.
- `γ…’` (`ui`) realized as `("Ι―", "i")` word-initially (`μ˜μ‚¬`), `"i"` after a consonant onset (`희망` -> `h i m a Ε‹`), and `"e"` non-initially (`λ‚˜μ˜` -> `n a e`, Clause 5 Note 2).
- Unreleased homorganic stop codas (`t_coda`, `k_coda`, `p_coda`) immediately preceding homorganic stops/affricates/fricatives (`λ¨Ήκ³ `, `학ꡐ`, `κ°”μŠ΅λ‹ˆλ‹€`) merge into the following closure without emitting a spurious double stop burst.
---
## 3. Formal Verification (TLA+)
Created and verified two TLA+ specifications and their exhaustive state-space model checker in `korean_work/tla/`:
1. [`KoreanPhonologyAndLayout.tla`](file:///G:/audiolm/scyllasband/korean_work/tla/KoreanPhonologyAndLayout.tla):
- Models the 2-stage Korean phonological transducer (`TransformWord` -> `EmitPhoneStream`) over all 19 onsets, 21 vowels, and 28 codas.
- Verifies `TypeOK`, `PhonologyInvariant` (all codas neutralized to the 7 representative codas; zero unassimilated obstruent+nasal/liquid or nasal+liquid sequences; zero untensified obstruent+lax sequences), `VocabMembershipInvariant` (100% of emitted symbols in `phone_vocab.json`, no stress markers `ˈ`/`ˌ`), `LayoutInvariant` (`<sil>` framing, `word_starts`, `word_boundary_candidates`), and singing `phone_durations` positivity (`d >= 1` for every spoken Korean phone).
2. [`StreamingAndLanguageLifecycle.tla`](file:///G:/audiolm/scyllasband/korean_work/tla/StreamingAndLanguageLifecycle.tla):
- Models language resolution (`resolve_language_for_voice` / `Engine::resolve_language`), acoustic conditioning index lookup (`language_index("ko") == 3`), group-speak `[voice:ko:delivery]` switching, and `_ChainStream` target/chunk boundaries (`prefix_latents` reset on voice/language change, preserved across continuations).
3. [`check_tla_models.py`](file:///G:/audiolm/scyllasband/korean_work/tla/check_tla_models.py):
- Exhaustively verified **26,664 phonological states + 28 layout/singing states + 7,584 language/streaming states = 34,276 reachable states with 0 invariant violations**.
---
## 4. Empirical Verification & Performance Results
### 4.1 Native C++ Build & CTest (`16/16 Passed`)
- Built `libscyllasband` with MSVC 2022 (`Release`) in `runtime/libscyllasband/build-msvc` (`scyllasband_native.dll`, `scyllasband_speak.exe`, `scyllasband_unit_test.exe`, `scyllasband_frontend_test.exe`) and staged `scyllasband_native.dll` + `libLiteRt.dll` into `runtime/scyllasband/_native/litert/`.
- Regenerated C++ test fixtures (`generate_fixtures.py`) including Korean normalizer test cases and ran `ctest --test-dir runtime/libscyllasband/build-msvc -C Release --output-on-failure`:
- **16/16 CTest suites passed in 0.53s** (`scyllasband_events`, `scyllasband_durations`, `scyllasband_context`, `scyllasband_delivery`, `scyllasband_sentences`, `scyllasband_segments`, `scyllasband_normalizer`, `scyllasband_unicode`, `scyllasband_targets`, `scyllasband_singing`, `scyllasband_records`, `scyllasband_frontend_noise`, `scyllasband_frontend_score`, `scyllasband_frontend_timeline`, `scyllasband_frontend_preview`, `scyllasband_frontend_gate`).
### 4.2 Python Unit & Property Tests (`131/131 Passed`)
- Added [`runtime/tests/test_korean.py`](file:///G:/audiolm/scyllasband/runtime/tests/test_korean.py) covering Sino-Korean & Native Korean normalization, Standard Korean phonological rules, Hypothesis property-based fuzzing over random Hangul words, language resolution, group-speak Korean tags, and Korean singing score planning/rendering.
- Ran `pytest runtime/tests` with `SCYLLASBAND_TEST_NATIVE_LIB` and `SCYLLASBAND_TEST_BUNDLE` enabled:
- **131/131 tests passed in 2.51s** (0 skipped, 0 failed).
### 4.3 Full Python vs C++ Parity (`parity_harness.py` & `eval_korean_e2e.py`)
- [`runtime/libscyllasband/tests/parity_harness.py`](file:///G:/audiolm/scyllasband/runtime/libscyllasband/tests/parity_harness.py) on `litert`:
- `failures`: **0**
- `frontend`: **280 cases, 0 mismatches** (`unicode_lower_block_mismatches: 0`, `nfc_mismatches: 0`)
- `sentences`: **10/10 identical G2P & durations, `0.0` relative latent error, `Infinity dB` SNR**
- `long_form`: **12/12 chunks & targets identical, `0.0` relative latent error, `Infinity dB` SNR**
- `records`: **6 chains identical, `Infinity dB` SNR**
- `singing`: **12/12 plans identical, 8/8 phrases `0.0` relative latent error (`vocos` & `bigvgan`), 4/4 full songs >234 dB SNR**
- [`korean_work/scratch/eval_korean_e2e.py`](file:///G:/audiolm/scyllasband/korean_work/scratch/eval_korean_e2e.py) (saved in [`korean_work/scratch/korean_e2e_report.json`](file:///G:/audiolm/scyllasband/korean_work/scratch/korean_e2e_report.json)):
- Korean Normalizer parity (Python vs C++): **15/15 cases, 0 mismatches**
- Korean G2P parity (Python vs C++): **30/30 cases, 0 mismatches**
- Korean Sentence synthesis parity: **identical durations (`130` frames), `0.0` relative latent error, `Infinity dB` audio SNR**
- Korean Singing plan parity: **identical phrase plans, phones, and frame durations**
### 4.4 Korean Speech ASR & Multi-Voice / Singing Evaluation
- **Multi-Voice Korean Synthesis (`scylla`, `gwen`, `felix`, `ariadne`, `ink`)**:
- Reference: `"μ•ˆλ…•ν•˜μ„Έμš”, 였늘 날씨가 정말 μ’‹λ„€μš”. ν•œκ΅­μ–΄ μŒμ„± 합성을 μ‹œμž‘ν•©λ‹ˆλ‹€."`
- `faster-whisper` (`large-v3-turbo`) CER: **0.0% across all 5 voices** (`5/5` exact transcriptions; samples saved in `korean_work/samples/ko_voice_*.wav`).
- **10-Sentence Korean Benchmark (`scylla`, `korean_work/samples/ko_speech_scylla_00..09.wav`)**:
- Overall CER: **20.12%** across 10 diverse sentences covering liaison, nasalization, liquidization, palatalization, tensification, and normalized numbers/counters, with **0.0% CER** on sentences `00`, `01`, `02`, and `05`.
- **Korean Singing Synthesis (`melodies/phrase_ko_twinkle.json`, `"반짝 반짝 μž‘μ€ 별 μ•„λ¦„λ‹΅κ²Œ λΉ„μΉ˜λ„€"`)**:
- Synthesized through both `vocos` (`ko_song_twinkle_scylla_vocos.wav`) and `bigvgan` (`ko_song_twinkle_scylla_bigvgan.wav`), verified by `scyllasband_speak.exe --score` and `runtime.sing(...)`.
### 4.5 Native C++ Runtime Performance (`scyllasband_speak.exe` on LiteRT CPU)
- **Bundle Load Time**: `2.6 ms`
- **Korean Speech (`5.33 s` audio, 4 Euler steps)**:
- Cold first audio: `403.0 ms`
- Warm first audio: **`138.2 ms`**
- Warm total time: **`282.5 ms` (~18.9Γ— real-time on CPU)**
- **Korean Singing (`7.62 s` audio, `melodies/phrase_ko_twinkle.json`, Vocos turbo-sing, 4 Euler steps)**:
- Warm first audio / total time: **`177.5 ms` (~42.9Γ— real-time on CPU)**
---
## 5. Modified & Created Files Summary
### Model Repository (`G:\audiolm\scyllasband`)
- `onnx/manifest.json`, `onnx/assets/languages.json`, `onnx/assets/g2p/language_map.json`, `onnx/assets/g2p/normalization.json`
- `litert/manifest.json`, `litert/assets/languages.json`, `litert/assets/g2p/language_map.json`, `litert/assets/g2p/normalization.json`
- `webgpu/manifest.json`, `webgpu/assets/languages.json`, `webgpu/assets/g2p/language_map.json`, `webgpu/assets/g2p/normalization.json`
- `coreml/manifest.json`, `coreml/assets/languages.json`, `coreml/assets/g2p/language_map.json`, `coreml/assets/g2p/normalization.json`
- `coreai/manifest.json`, `coreai/assets/languages.json`, `coreai/assets/g2p/language_map.json`, `coreai/assets/g2p/normalization.json`
- `pytorch/release_config.json`
- `melodies/phrase_ko_twinkle.json`, `melodies/README.md`
- `README.md` (Updated with prominent Korean support section and usage guide)
- `SHA256SUMS.json` (Recalculated and verified)
- `PROGRESS.md`
- `webui.py` (Interactive Gradio web interface)
- `serve_webgpu_demo.py` (Local HTTP static server with COOP/COEP isolation headers)
- `webgpu_demo/index.html`, `webgpu_demo/css/style.css`, `webgpu_demo/README.md`
- `webgpu_demo/js/app.js`, `webgpu_demo/js/korean_g2p.js`, `webgpu_demo/js/pipeline.js`, `webgpu_demo/js/wav_utils.js` (Standalone 100% Client-Side WebGPU TTS Demo)
- `korean_work/tla/KoreanPhonologyAndLayout.tla`, `korean_work/tla/StreamingAndLanguageLifecycle.tla`, `korean_work/tla/check_tla_models.py`
- `korean_work/scratch/eval_korean_e2e.py`, `korean_work/scratch/korean_e2e_report.json`, `korean_work/scratch/update_bundle_metadata.py`
- `korean_work/samples/*.wav`
### Runtime Repository (`G:\audiolm\scyllasband\runtime`)
- `scyllasband/text_normalizer.py`
- `scyllasband/g2p.py`
- `scyllasband/planner.py`
- `scyllasband/runtime.py`
- `scyllasband/engine.py`
- `tests/test_korean.py`
- `libscyllasband/src/scyllasband_text.cpp`
- `libscyllasband/src/scyllasband_g2p.h`, `libscyllasband/src/scyllasband_g2p.cpp`
- `libscyllasband/src/scyllasband_engine.cpp`
- `libscyllasband/tools/scyllasband_speak.cpp`
- `libscyllasband/tests/generate_fixtures.py`, `libscyllasband/tests/fixtures/*.json`
---
## 6. Remaining Limitations & Future Work
1. **Zero-Shot Acoustic Conditioning Trade-offs**:
- Because the `v4-20261106` acoustic graphs (`duration_predictor`, `pitch_estimator`, `vector_estimator`) were trained on 7 non-Korean locales without native Korean audio, Korean synthesis maps `ko` to syllable-timed acoustic conditioning index `3` and approximates Korean's 3-way laryngeal stop/affricate contrast (`ν‰μŒ/경음/격음`) and unrounded back vowels (`γ…“/γ…‘`) using the closest trained IPA phones (`Ι”`, `Ι―`/`Ι™`, `k/Ι‘/k h`). Complex consonant clusters across word boundaries (e.g., `μ„€λ‚  아침에 κ°€μ‘±λ“€κ³Ό 떑ꡭ을`) still exhibit Romance prosodic liaison and vowel coloration.
2. **Next Steps When Native Korean Training Assets Become Available**:
- Fine-tune or add a dedicated `ko` row (`index=7`) in `language_embed` and per-voice reference prosody tables (`conditions.*`), and include Hangul Jamo or syllable tokens in the neural G2P training corpus while retaining the `ν‘œμ€€λ°œμŒλ²•` phonological frontend and text normalizer as the reference ground truth.
## WebGPU Standalone In-Browser Demo Milestone (2026-11-08)
1. **Standalone Architecture**:
- `webgpu_demo/index.html`: Fully client-side UI with dark theme, preset phrases, 14 voices, prosody controls, and real-time canvas waveform visualizer.
- `webgpu_demo/js/korean_g2p.js`: Pure JavaScript Standard Korean G2P & normalizer (Chosung/Jungsung/Jongsung decomposition, assimilation, nasalization, tensification, liquidization, palatalization, aspiration, and number/symbol normalization).
- `webgpu_demo/js/pipeline.js`: Complete in-browser neural TTS pipeline (Euler ODE flow matching, duration prediction, pitch contour generation, and Vocos neural vocoder).
- `webgpu_demo/dist/`: Self-contained `ort.webgpu.min.js`, `ort-wasm-simd-threaded.jsep.wasm`, and `ort-wasm-simd-threaded.jsep.mjs`.
- `serve_webgpu_demo.py`: Python static server with explicit MIME types, COOP/COEP isolation headers, `--host`, and `--port` CLI flags.
2. **Root Cause Analysis of Duration Predictor (1/6) Error**:
- Symptom: At model load stage 1/6 (`duration_predictor.onnx`), initialization failed with `undefined` / WASM C++ exception.
- Investigation:
- `webgpu/onnx/components/duration_predictor.onnx` uses `com.microsoft.MatMulNBits` with `bits: 8`.
- In ONNX Runtime Web (JSEP & WASM CPU), `MatMulNBits` only supports `bits: 4` and `bits: 2` (`matmul_nbits.cc:115: nbits_ == 4 was false. Only 4b is supported`).
- Emscripten caught the C++ exception and threw a numeric pointer (`11055512`), which resulted in `undefined` error string in the UI.
- Resolution:
- `onnx/onnx/components/` contains standard FP32 ONNX graphs (`MatMul`, `Gemm`, `LayerNormalization`, `Conv`, `Softmax`), which are 100% natively supported across both WebGPU and WASM.
- Changed default bundle in `webgpu_demo/index.html` to `../onnx`.
- Enhanced error handling in `app.js` and `pipeline.js` to catch numeric WASM exceptions and show actionable error diagnostics.
- Fixed WASM base URL resolution in `pipeline.js` to support access via any network IP (`0.0.0.0` / LAN IP) without falling back to CDN.
3. **End-to-End Verification**:
- Executed full client-side TTS synthesis test on Korean text ("μ•ˆλ…•ν•˜μ„Έμš”. λ°˜κ°‘μŠ΅λ‹ˆλ‹€.").
- Result: 24,832 samples synthesized @ 24,000 Hz (1.03 sec duration, healthy RMS amplitude, peak [-0.36, +0.48], non-zero ratio 91.3%).
4. **Long-Form & Multi-Sentence Duration Support (2초 이상 ν•©μ„± μ™„λ²½ 지원)**:
- 원인: 초기 κ΅¬ν˜„μ—μ„œ κ³ μ • 128 버킷(`bucket = 128`)λ§Œμ„ λ‘œλ“œν•˜μ—¬, μ΅œλŒ€ 지속 μ‹œκ°„μ΄ 128 ν”„λ ˆμž„ Γ— 256 / 24,000 = **1.36초**둜 μ œν•œλ˜μ–΄ κΈ΄ λ¬Έμž₯이 μž˜λ¦¬λŠ” ν˜„μƒ λ°œμƒ.
- 쑰치:
- 512 ν”„λ ˆμž„ 버킷 λͺ¨λΈ(`pitch_estimator_512.onnx`, `vector_estimator_512.onnx`, `vocoder_512.onnx`)둜 μ—…κ·Έλ ˆμ΄λ“œ (단일 ꡬ문 μ΅œλŒ€ 5.46초 지원).
- μž₯λ¬Έ 닀쀑 λ¬Έμž₯ λΆ„ν•  및 청크 λ‹¨μœ„ ν•©μ„±(`splitSentences`, `synthesizeChunk`) νŒŒμ΄ν”„λΌμΈ κ΅¬ν˜„: λ¬Έμž₯ κ°„ 60ms μžμ—°μŠ€λŸ¬μš΄ 묡음 νŒ¨λ”© μ‚½μž… 및 μŒμ„± νŒŒν˜• 무결 κ²°ν•©.
- 2초, 5초, 10초 μ΄μƒμ˜ μž₯λ¬Έ ν…μŠ€νŠΈλ„ 길이 μ œν•œ 없이 연속 ν•©μ„± κ°€λŠ₯ν•˜λ„λ‘ ν™•μž₯ μ™„λ£Œ.
5. **2-Tier 둜컬 λΈŒλΌμš°μ € 영ꡬ 캐싱 μ‹œμŠ€ν…œ (Permanent Disk & CacheStorage Caching)**:
- 문제: 맀번 νŽ˜μ΄μ§€ 접속 λ˜λŠ” "WebGPU λͺ¨λΈ λ‘œλ“œ" μ‹€ν–‰ μ‹œ μ•½ 1GB의 `shared_weights.bin`κ³Ό λͺ¨λΈλ“€μ„ λ„€νŠΈμ›Œν¬λ‘œ λ‹€μ‹œ λ‹€μš΄λ‘œλ“œν•˜λ˜ λΉ„νš¨μœ¨ λ°œμƒ.
- Tier 1 (HTTP 레벨): `serve_webgpu_demo.py`μ—μ„œ `.bin`, `.onnx`, `.wasm` 정적 νŒŒμΌμ— λŒ€ν•΄ `Cache-Control: public, max-age=31536000, immutable` 헀더 전솑 (λΈŒλΌμš°μ € λ””μŠ€ν¬ μΊμ‹œ μ¦‰μ‹œ 적쀑).
- Tier 2 (ν΄λΌμ΄μ–ΈνŠΈ 레벨): `pipeline.js`의 `fetchWithCache`μ—μ„œ λΈŒλΌμš°μ € `window.caches` (CacheStorage API)λ₯Ό μ μš©ν•˜μ—¬ 졜초 λ‹€μš΄λ‘œλ“œ ν›„ 둜컬 λ””μŠ€ν¬μ— 영ꡬ μ €μž₯. λ‹€μŒ μ‹€ν–‰λΆ€ν„°λŠ” λ„€νŠΈμ›Œν¬ μš”μ²­ 없이 100~300ms 내에 μ¦‰μ‹œ λ‘œλ“œ.
- UI 지원: "μΊμ‹œ μ‚­μ œ" λ²„νŠΌμ„ μΆ”κ°€ν•˜μ—¬ ν•„μš” μ‹œ μ›ν΄λ¦­μœΌλ‘œ λΈŒλΌμš°μ € λͺ¨λΈ μΊμ‹œλ₯Ό μ΄ˆκΈ°ν™”ν•  수 μžˆλ„λ‘ 지원.
6. **μ΄ˆκ²½λŸ‰ Vocos μ „μš© INT8 WebGPU λ²ˆλ“€ μž¬νŒ¨ν‚€μ§• (230 MB) 및 ONNX Runtime Web 1.30.0 μ—…κ·Έλ ˆμ΄λ“œ**:
- **λ°°κ²½ 및 원인 뢄석**:
- μ›μž‘μž 데λͺ¨(`https://lowkeytea.dev/scyllasband/demo/`)κ°€ 261MB에 λΆˆκ³Όν–ˆλ˜ μ΄μœ λŠ” κ°€μ°½(Singing) κΈ°λŠ₯이 μ—†λŠ” v3 κΈ°λ°˜μ΄μ—ˆκΈ° λ•Œλ¬Έμž„.
- v4둜 νŒμ˜¬λ¦Όλ˜λ©΄μ„œ BigVGAN 보코더(λ‹¨λ…μœΌλ‘œ 502.2 MB 점유)κ°€ `shared_weights.bin`에 일괄 λ³‘ν•©λ˜μ–΄ λ²ˆλ“€ 크기가 743.6 MB(INT8) / 1,036 MB(FP32)둜 급증함.
- μ›Ή μŒμ„± TTS ν™˜κ²½μ—μ„œλŠ” λŒ€ν˜• 신경망 보코더인 BigVGAN이 λΆˆν•„μš”ν•˜λ©°, μ‹€μ‹œκ°„ 좔둠에 μ΅œμ ν™”λœ Vocos λ³΄μ½”λ”λ§ŒμœΌλ‘œ κ³ ν’ˆμ§ˆ μŒμ„± 합성이 κ°€λŠ₯함.
- **μ΄ˆκ²½λŸ‰ λ²ˆλ“€ μž¬νŒ¨ν‚€μ§• μˆ˜ν–‰**:
- BigVGAN ν…μ„œ 502 MBλ₯Ό μ™„μ „νžˆ λΆ„λ¦¬Β·μ œμ™Έν•˜κ³ , 21개 μŒμ„± μ „μš© λͺ¨λΈ(Duration Predictor, Pitch/Vector Context Encoders, Pitch/Vector Estimators 128~768, Vocos 128~768)의 795개 고유 ν…μ„œλ§Œμ„ 64λ°”μ΄νŠΈ μ •λ ¬λ‘œ 순차 νŒ¨ν‚Ή.
- `webgpu/onnx/components/shared_weights.bin` 크기λ₯Ό 743.6 MBμ—μ„œ **241,440,320 λ°”μ΄νŠΈ (230.26 MB)**둜 **3.2λ°° κ²½λŸ‰ν™”** μ™„λ£Œ (μ €μž 데λͺ¨ 230.2 MB와 μ™„μ „ 일치).
- 21개 ONNX λͺ¨λΈμ˜ `external_data` μ˜€ν”„μ…‹μ„ μž¬λ§€ν•‘ν•˜κ³  λ―Έμ‚¬μš© `vocoder_bigvgan*.onnx` 6개 νŒŒμΌμ„ 제거.
- `shared_weights.json` 및 `manifest.json`의 μ»΄ν¬λ„ŒνŠΈΒ·λ²„ν‚· λͺ…μ„Έλ₯Ό Vocos μ „μš© λ²ˆλ“€ 규격으둜 κ°±μ‹ .
- **ONNX Runtime Web 1.30.0 μ—…κ·Έλ ˆμ΄λ“œ**:
- κΈ°μ‘΄ ORT 1.21.0μ—μ„œ λ°œμƒν•˜λ˜ 8-bit `MatMulNBits` WASM CPU 폴백 μ˜ˆμ™Έ(`nbits_ == 4 was false`)λ₯Ό ν•΄κ²°ν•˜κΈ° μœ„ν•΄, μ΅œμ‹  `onnxruntime-web@1.30.0` (WebGPU Asyncify WASM 및 JSEP) λ°”μ΄λ„ˆλ¦¬λ₯Ό `webgpu_demo/dist/`에 정식 배포.
- WebGPU ν™˜κ²½μ—μ„œ 8-bit μ–‘μžν™” κ°€μ€‘μΉ˜λ₯Ό 직접 GPU 셰이더 및 μ΅œμ‹  WASM λŸ°νƒ€μž„μœΌλ‘œ 가속할 수 μžˆλ„λ‘ νŒŒμ΄ν”„λΌμΈ ꡬ성.
- `webgpu_demo/index.html` κΈ°λ³Έ λ‘œλ“œ 경둜λ₯Ό `../webgpu` (230 MB INT8 λ²ˆλ“€)둜 μ„€μ •.
- **쒅단간(End-to-End) 검증 μ™„λ£Œ**:
- Scylla's Band 곡식 λŸ°νƒ€μž„(`ScyllasBandRuntime.from_bundle`)μ—μ„œ λ ˆμ΄μ•„μ›ƒ 검증 톡과 및 ν•œκ΅­μ–΄ λ¬Έμž₯("μ•ˆλ…•ν•˜μ„Έμš”! λ°˜κ°‘μŠ΅λ‹ˆλ‹€.") ν•©μ„± ν…ŒμŠ€νŠΈ μ™„λ£Œ (2.432초, 58,368 μƒ˜ν”Œ @ 24kHz 정상 좜λ ₯).
- 둜컬 HTTP μ„œλ²„ 및 정적 파일 MIME νƒ€μž…, 2-Tier 캐싱, COOP/COEP ꡐ차 좜처 격리 헀더 검증 μ™„λ£Œ.
7. **Tailscale Secure Context HTTPS 및 WebGPU BufferManager Download 좩돌 ν•΄κ²°**:
- **WASM Fallback 및 18초 μΆ”λ‘  μ§€μ—° 원인 뢄석**:
- 원인: \http://100.72.102.121:8000\ (사섀 IP HTTP)둜 접속 μ‹œ λΈŒλΌμš°μ € W3C λ³΄μ•ˆ κ·œκ²©μ— 따라 λΉ„λ³΄μ•ˆ μ»¨ν…μŠ€νŠΈ(Insecure Context)둜 λΆ„λ₯˜λ¨.
- 이둜 인해 avigator.gpu\ (WebGPU) 및 \crossOriginIsolated\ (\SharedArrayBuffer\ λ©€ν‹°μŠ€λ ˆλ”©)κ°€ μ›μ²œ μ°¨λ‹¨λ˜μ–΄ 단일 μŠ€λ ˆλ“œ CPU WASM(\ hreads: 1\)으둜 폴백되며 μΆ”λ‘  μ‹œκ°„μ΄ 18μ΄ˆκΉŒμ§€ 폭증함.
- **Tailscale HTTPS & Let's Encrypt μΈμ¦μ„œ 톡합**:
- Tailscale 곡식 MagicDNS μΈμ¦μ„œ(\ ailscale cert dt4090sp.tail50666.ts.net\)λ₯Ό λ°œκΈ‰λ°›μ•„ \serve_webgpu_demo.py\에 \--ssl\ μ˜΅μ…˜μ„ μΆ”κ°€ν•˜κ³  Python \ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)\λ₯Ό νƒ‘μž¬.
- \https://dt4090sp.tail50666.ts.net:8000/webgpu_demo/\λ₯Ό 톡해 λ³„λ„μ˜ ν”Œλž˜κ·Έ μ‘°μ • 없이도 Tailnet λ‚΄ λͺ¨λ“  λͺ¨λ°”일/PC κΈ°κΈ°μ—μ„œ 정식 곡인 λ³΄μ•ˆ μ»¨ν…μŠ€νŠΈ(Secure Context) ν™œμ„±ν™” 및 WebGPU + WASM λ©€ν‹°μŠ€λ ˆλ”© μ™„μ „ 가동.
- \index.html\ 및 \pp.js\에 λΉ„λ³΄μ•ˆ HTTP 접속 감지 μ‹œ HTTPS λ„λ©”μΈμœΌλ‘œ μ•ˆλ‚΄ν•˜λŠ” μƒνƒœ λ°°λ„ˆ μΆ”κ°€.
- **\BufferManager::Download map_async_result.status == Success was false\ 였λ₯˜ 원인 및 ν•΄κ²°**:
- 였λ₯˜ ν˜„μƒ: WebGPU λͺ¨λΈ λ‘œλ“œ ν›„ μŒμ„± ν•©μ„± μ‹€ν–‰ μ‹œ \ ailed to call OrtRun(). ERROR_CODE: 1, ERROR_MESSAGE: .../providers/webgpu/buffer_manager.cc:643 void onnxruntime::webgpu::BufferManager::Download(WGPUBuffer, void *, size_t) const map_async_result.status == wgpu::MapAsyncStatus::Success was false. Failed to download data from buffer. wgpu::MapAsyncStatus value: 0\ λ°œμƒ.
- 원인: \pipeline.js\μ—μ„œ \Promise.all([ this.sessions.vector_context_encoder.run(contextFeeds), this.sessions.pitch_context_encoder.run(contextFeeds) ])\둜 2개의 λͺ¨λΈμ„ GPU에 λ™μ‹œ 병렬 μš”μ²­. ONNX Runtime Web의 WebGPU JSEP λ°±μ—”λ“œλŠ” 동일 GPU μ»¨ν…μŠ€νŠΈμ—μ„œ μ€‘μ²©λœ \mapAsync\ μŠ€ν…Œμ΄μ§• 버퍼 λ‹€μš΄λ‘œλ“œλ₯Ό μ²˜λ¦¬ν•˜μ§€ λͺ»ν•΄ 두 번째 μš”μ²­μ΄ μ·¨μ†Œ(Abort, Status: 0)됨.
- 쑰치: \pipeline.js\의 λ™μ‹œ ν˜ΈμΆœμ„ 순차 \wait\ (\ ecContextOut = await ...; pitchContextOut = await ...;\)둜 λ³€κ²½.
- 검증: 격리 ν…ŒμŠ€νŠΈ(\ est_promise_all.html\)λ₯Ό 톡해 \Promise.all\ μ‹€νŒ¨ μž¬ν˜„ 및 순차 호좜 μ‹œ 각 λͺ¨λΈμ΄ 단 9ms λ§Œμ— 정상 μ™„λ£Œλ¨μ„ μž…μ¦. 전체 6단계 WebGPU ν•©μ„± νŒŒμ΄ν”„λΌμΈ(\ est_webgpu_synthesis.html\)μ—μ„œ 261,888 μƒ˜ν”Œ(10.9초 μ˜€λ””μ˜€)을 1초 λ―Έλ§Œμ— 무결 ν•©μ„± 성곡.
8. **WebGPU μ˜€λ””μ˜€ 지속 μ‹œκ°„ 1/2 μ ˆλ‹¨(2.75초 잘림) 버그 ν•΄κ²° 및 Prosody 이벀트 κ°•ν™”**:
- **원인 뢄석**:
- \manifest.json\의 \latent_hop_length: 512\ 및 곡식 λŸ°νƒ€μž„(untime/scyllasband/engine.py\)의 \LATENT_HOP = 512\ κ·œκ²©μ— 따라 지속 μ‹œκ°„ 예츑기(\duration_predictor\)κ°€ 좜λ ₯ν•˜λŠ” 1 ν”„λ ˆμž„μ€ **512 μ˜€λ””μ˜€ μƒ˜ν”Œ**(24,000Hz κΈ°μ€€ 21.33ms)에 해당함. Vocos 512 버킷은 512 Γ— 512 - 256 = 261,888 μƒ˜ν”Œ(μ΅œλŒ€ 10.92초)을 볡원함.
- κ·ΈλŸ¬λ‚˜ κΈ°μ‘΄ \pipeline.js\μ—μ„œ \const hopLength = 256;\으둜 μ§€μ •λ˜μ–΄ μžˆμ–΄, \ otalFrames * 256\으둜 μŠ¬λΌμ΄μŠ€λ˜λ©΄μ„œ λͺ¨λ“  ν•œκ΅­μ–΄ λ¬Έμž₯의 ν›„λ°˜λΆ€ 50%κ°€ 잘렀 **μ •ν™•νžˆ 2.75초 λΆ€κ·Όμ—μ„œ μ˜€λ””μ˜€κ°€ 쀑간에 μ ˆλ‹¨**λ˜λŠ” ν˜„μƒμ΄ λ°œμƒν•¨.
- **쑰치 λ‚΄μš©**:
- \pipeline.js\의 μ˜€λ””μ˜€ 슬라이슀 λ‹¨μœ„(\hopLength\)λ₯Ό 곡식 잠재 홉 규격인 **512**둜 μˆ˜μ • (\ctiveSamples = activeFrames * 512\).
- 버킷 λ‚΄ μ‹€μ œ μ „κ°œλœ ν™œμ„± ν”„λ ˆμž„ 수(\cursor\) κΈ°μ€€μœΌλ‘œ μ˜€λ””μ˜€ 트림 λ²”μœ„λ₯Ό μ •λ°€ν™”ν•˜μ—¬ 버킷 μ˜€λ²„ν”Œλ‘œμš° μ‹œμ—λ„ 무결 μŠ¬λΌμ΄μ‹± 보μž₯.
- λ¬Έμž₯ μ’…κ²° λΆ€ν˜Έ(\?\, \!\, \...\)에 λ”°λ₯Έ \sentence_types\ (1: μ§„μˆ , 2: 의문, 3: 감탄, 4: λ§μ€„μž„ν‘œ) 및 μ–΄νœ˜ μ‹œμž‘ 경계(\wordStarts\ -> \oundaryEvents\) 맀핑을 μ˜¨μ „ν•˜κ²Œ μ „κ°œν•˜μ—¬ μ˜λ¬Έλ¬ΈΒ·κ°νƒ„λ¬Έ μ–΅μ–‘ μžμ—°λ„ κ°œμ„ .
- **검증**:
- 96개 μŒμ†Œ(ν•œκ΅­μ–΄ 2λ¬Έμž₯, 227ν”„λ ˆμž„) ν•©μ„± μ‹œ κΈ°μ‘΄ 2.42~2.75μ΄ˆμ—μ„œ 잘리던 μ˜€λ””μ˜€κ°€ 4.84~5.69초의 μ˜¨μ „ν•œ νŒŒν˜•μœΌλ‘œ 전체 μŒμ†Œ μ‹œν€€μŠ€κ°€ μ²˜μŒλΆ€ν„° λκΉŒμ§€ μ™„μ „ μž¬μƒλ¨μ„ 검증.
9. **Git Commit & Push μ™„λ£Œ 및 Hugging Face Space 정식 배포**:
- **λͺ¨λΈ 리포지토리 동기화 (cidsound/scyllasband)**:
- WebGPU 버퍼 락 μˆœμ°¨ν™”, Vocos LATENT_HOP 512 지속 μ‹œκ°„ μˆ˜μ •, 닀쀑 경둜 볡원λ ₯(Multi-tier fallback) 및 Prosody 이벀트 κ°•ν™” 반영 μ™„λ£Œ.
- 둜컬 Git λ³€κ²½ 사항 원격 main λΈŒλžœμΉ˜μ— μ™„μ „ 컀밋 및 ν‘Έμ‹œ μ™„λ£Œ (2b1b4d3).
- **Hugging Face Space 배포 μ™„λ£Œ (cidsound/scyllasband-webgpu)**:
- Space μœ ν˜•: Static Space (sdk: static β€” 무쀑단 24/7 가동, μ½œλ“œ μŠ€νƒ€νŠΈ 0초, HF κΈ€λ‘œλ²Œ CDN ν˜ΈμŠ€νŒ…).
- Space URL: [https://huggingface.co/spaces/acidsound/scyllasband-webgpu](https://huggingface.co/spaces/acidsound/scyllasband-webgpu)
- 직접 접속 URL (전체화면): [https://acidsound-scyllasband-webgpu.static.hf.space](https://acidsound-scyllasband-webgpu.static.hf.space)
- 230 MB μ΄ˆκ²½λŸ‰ INT8 λ²ˆλ“€μ΄ Space 내에 직접 λ°°ν¬λ˜μ–΄ μ™ΈλΆ€ μ˜μ‘΄μ„± 없이 μ™„μ „ν•œ 100% 자립 μ‹€ν–‰(Self-contained) λ™μž‘.
- 배포 μƒνƒœ: Space stage: RUNNING, λͺ¨λ“  정적 파일, WASM λ°”μ΄λ„ˆλ¦¬(pplication/wasm), 230 MB κ°€μ€‘μΉ˜ 및 ONNX κ·Έλž˜ν”„ HTTP 200 μ„œλΉ™ 검증 μ™„λ£Œ.