# Korean Support for Scylla's Band — Engineering Progress (`PROGRESS.md`) ## 1. Objective & Status - **Objective**: Deliver reliable, end-to-end Korean (`ko` / `ko_kr`) speech and singing support for **Scylla's Band (`v4-20261106`)** across both the model repository (`spybyscript/scyllasband`) and the reference/native runtime (`lowkeytea/scyllasband`, located in `runtime/`), preserving compact deployment, fast local inference, exact Python/C++ parity, and backward compatibility with all 14 existing voices, 7 trained locales (`en_us`, `en_gb`, `es`, `it`, `fr`, `de`, `vi`), and 6 bundle flavors (`litert`, `onnx`, `webgpu`, `coreml`, `coreai`, `pytorch`). - **Status**: **Complete and verified end-to-end** across formal TLA+ specifications, Python (`scyllasband`) and C++ (`libscyllasband`) implementations, all 6 bundle flavors, CTest (`16/16`), pytest (`131/131`), the full Python/C++ parity harness (`0` failures), Korean ASR intelligibility benchmarks, and Korean singing synthesis (`vocos` and `bigvgan`). --- ## 2. Confirmed Architectural Findings & Engineering Decisions ### 2.1 Model & Graph Contract Analysis (`v4-20261106`) 1. **Phone Vocabulary (`phone_vocab.json`, 87 tokens, IDs `0..86`)**: - Structural/control tokens (`0..14`): ``, ``, ``, ``, ``, ``, ``, ``, ``, ``, ``, ``, ``, ``, ``. - Tone digits (`15..20`): `1`, `2`, `4`, `5`, `6`, `7`. - IPA vowels, glides, and consonants (`21..77, 84..86`) and non-acoustic modifier bits (`78..83`). - **Important Vocabulary Constraint**: `phone_vocab.json` contains postalveolar `"ʃ"` (`id=69`) and `"ʒ"` (`id=74`), not alveolo-palatal `"ɕ"` or `"ʑ"`. Using `"ʃ"` and `"ʒ"` for Korean `ㅈ/ㅉ/ㅊ` (`t ʃ` / `d ʒ` / `t ʃ h`) and palatalized `ㅅ/ㅆ` (`ʃ` / `ʃ ʃ` before `ㅣ/ㅑ/ㅕ/ㅛ/ㅠ/ㅖ/ㅟ`) keeps 100% of emitted Korean phones strictly inside the 87-symbol vocabulary. 2. **Neural G2P Graph vs Deterministic Korean Phonological G2P**: - The exported neural G2P graphs (`g2p.onnx`, `g2p.tflite`, `g2p.mlmodelc`, `g2p.aimodel`, `g2p.pt`) are char-level CTC models (`char_repeats = 4`) trained only on ``, ``, ``, ``, ``, ``, `` with no `` symbol and no Hangul coverage in `tokenizer.json`. - Therefore, when `language == "ko"` or when any input segment contains Hangul (`U+AC00..U+D7A3` or Compatibility Jamo `U+3131..U+3163`), `G2PFrontend._predict_segment` (Python) and `G2P::predict_segment` (C++) route Hangul tokens through a deterministic, context-sensitive Standard Korean (`표준발음법`) phonological G2P, while delegating embedded multi-letter Latin words to the voice's English G2P/lexicon and spelling out 1–3 letter uppercase initialisms (`AI` -> `에이아이`, `TTS` -> `티티에스`, `GPU` -> `지피유`, `MP3` -> `엠피삼`, `4K` -> `사케이`) in Korean. 3. **Acoustic Conditioning (`language_id = 3`)**: - Precompiled graphs in `litert`, `onnx`, `webgpu`, `coreml`, and `coreai` have fixed `[7, D]` `language_embed` and `[98, ...]` (`14 voices × 7 locales`) voice/locale conditioning tables. - Empirical probing across all 7 trained `language_id` indices (`0..6`) with `faster-whisper` (`large-v3-turbo`) confirmed that mapping `"ko"` to acoustic conditioning index `3` (`it`, syllable-timed pure-vowel Romance conditioning without English stress-timed schwa reduction or Vietnamese lexical tone contours) yields the cleanest Korean vowel formants and syllable timing across all 14 voices. ### 2.2 Korean Spoken-Text Normalizer (`text_normalizer.py` & `scyllasband_text.cpp`) - Preserved `SPOKEN_TEXT_NORMALIZER_SHA256` (`d8fd1b2415d0d100d1c70237a4469117ee6b73d0e4da703e4654c83192779cc4`) and 100% backward compatibility with all existing languages. - Added `"ko"`, `"ko-kr"`, `"ko_kr"`, `"kor"`, `"korean"` alias resolution in `normalize_language_code`. - Implemented identical Korean normalization rules in Python and C++: - **Dual Numeral System**: - Sino-Korean (`영, 일, 이, 삼, 사, 오, 육, 칠, 팔, 구, 십, 백, 천, 만, 억, 조`) with Korean `1`-elision before `십/백/천` (`10` -> `십`, `15,000` -> `만 오천`, `100,000,000` -> `일억`). - Native Korean (`한/하나, 두/둘, 세/셋, 네/넷, 다섯, ..., 스무/스물, 서른, 마흔, 쉰, 예순, 일흔, 여든, 아흔`) automatically triggered before 25 Native Korean counters (`개, 명, 분, 사람, 마리, 살, 시, 시간, 달, 잔, 병, 권, 장, 대, 채, 켤레, 벌, 송이, 자루, 그루, 조각, 상자, 바구니, 봉지, 번`). - Ordinals (`1번째` -> `첫 번째`, `2번째` -> `두 번째`) and Sino-Korean month exceptions (`6월` -> `유월`, `10월` -> `시월`). - **Structured Entities**: - ISO dates (`2026-11-08` -> `이천이십육년 십일월 팔일`). - Times (`3:30 PM` -> `오후 세 시 삼십 분`, `7:05 AM` -> `오전 일곱 시 오 분`, `14:00` -> `십사 시`). - Currencies (`₩15,000` -> `만 오천 원`, `$3.50` -> `삼 달러 과 오십 센트`, `40€` -> `사십 유로`, `£20` -> `이십 파운드`, `¥500` -> `오백 엔`). - Fractions (`3/4` -> `사 분의 삼`, `1/2` -> `이 분의 일`), decimals (`3.14` -> `삼 점 일사`), percentages (`95.5%` -> `구십오 점 오 퍼센트`), temperatures/degrees (`-5°C` -> `마이너스 오 도씨`, `20°` -> `이십 도`), symbols (`=` -> `이퀄`, `+` -> `플러스`), and alphanumeric splitting (`MP3와 4K` -> `엠 피 삼 와 사 케이`). ### 2.3 Standard Korean Phonological G2P (`g2p.py` & `scyllasband_g2p.cpp`) - Implemented Hangul syllable decomposition (`U+AC00..U+D7A3` -> 19 Choseong, 21 Jungseong, 28 Jongseong) + Compatibility Jamo (`ㄱ..ㅎ`, `ㅏ..ㅣ`) handling. - Implemented ordered Standard Korean (`표준발음법`) phonological rules across adjacent syllables: 1. **Palatalization (구개음화, Clause 17)**: `ㄷ/ㅌ/ㄾ` before vowel-initial `ㅣ/ㅑ/ㅕ/ㅛ/ㅠ/ㅖ` -> `ㅈ/ㅊ/ㄹ+ㅊ` (`같이` -> `[가치]`, `굳이` -> `[구지]`, `닫히다` -> `[다치다]`). 2. **Liaison (연음법칙, Clauses 13–14)**: Single codas move to following null onset (`한국어` -> `[한구거]`); complex codas (`겹받침`: `ㄳ, ㄵ, ㄺ, ㄻ, ㄼ, ㄽ, ㄾ, ㄿ, ㅄ`) split across the boundary (`읽어` -> `[일거]`, `앉아` -> `[안자]`, `없어` -> `[업써]`). 3. **`ㅎ` Aspiration & Elision (ㅎ 축약 및 탈락, Clause 12)**: `ㅎ/ㄶ/ㅀ` + `ㄱ/ㄷ/ㅂ/ㅈ` (or reverse) -> `ㅋ/ㅌ/ㅍ/ㅊ` (`좋다` -> `[조타]`, `입학` -> `[이팍]`, `독립기념관` / `많이` -> `[마니]`, `좋아요` -> `[조아요]`), `ㅎ/ㄶ/ㅀ` + `ㅅ` -> `ㅆ`. 4. **Complex Coda Reduction & Neutralization (음절의 끝소리 규칙, Clauses 8–11)**: All 27 codas neutralize to the 7 representative codas (`k, n, t, l, m, p, ŋ`), including the Clause 11 exception where `ㄺ` before `ㄱ` reduces to `ㄹ` + `ㄲ` (`맑고` -> `[말꼬]`, `읽고` -> `[일꼬]`, whereas `맑다` -> `[막따]`). 5. **Liquidization (유음화, Clause 20)**: `ㄴ+ㄹ` and `ㄹ+ㄴ` -> `ㄹ+ㄹ` (`신라` -> `[실라]`, `설날` -> `[설랄]`). 6. **Nasalization (비음화, Clauses 18–19)**: Obstruent coda (`k, t, p`) before nasal (`n, m`) -> (`ŋ, n, m`) (`국물` -> `[궁물]`, `먹는` -> `[멍는]`, `있는` -> `[인는]`); nasal (`m, ŋ`) + `r` -> `m/ŋ + n` (`종로` -> `[종노]`); obstruent (`k, t, p`) + `r` -> (`ŋ, n, m`) + `n` (`독립` -> `[동닙]`, `백만` -> `[뱅만]`). 7. **Post-Obstruent Tensification (경음화, Clause 23)**: Obstruent coda (`k, t, p`) before lax obstruent (`g, d, b, s, j`) -> (`kk, tt, pp, ss, jj`). 8. **Contextual Vowel & Homorganic Closure Realization**: - `ㅡ` (`eu`) realized as `"ɯ"` in open syllables (`스스로` -> `s ɯ s ɯ ɾ o`) and `"ə"` in closed syllables (`은/는/을/를/습`) to prevent rounded `[u]` coloration under Romance acoustic conditioning. - `ㅢ` (`ui`) realized as `("ɯ", "i")` word-initially (`의사`), `"i"` after a consonant onset (`희망` -> `h i m a ŋ`), and `"e"` non-initially (`나의` -> `n a e`, Clause 5 Note 2). - Unreleased homorganic stop codas (`t_coda`, `k_coda`, `p_coda`) immediately preceding homorganic stops/affricates/fricatives (`먹고`, `학교`, `갔습니다`) merge into the following closure without emitting a spurious double stop burst. --- ## 3. Formal Verification (TLA+) Created and verified two TLA+ specifications and their exhaustive state-space model checker in `korean_work/tla/`: 1. [`KoreanPhonologyAndLayout.tla`](file:///G:/audiolm/scyllasband/korean_work/tla/KoreanPhonologyAndLayout.tla): - Models the 2-stage Korean phonological transducer (`TransformWord` -> `EmitPhoneStream`) over all 19 onsets, 21 vowels, and 28 codas. - Verifies `TypeOK`, `PhonologyInvariant` (all codas neutralized to the 7 representative codas; zero unassimilated obstruent+nasal/liquid or nasal+liquid sequences; zero untensified obstruent+lax sequences), `VocabMembershipInvariant` (100% of emitted symbols in `phone_vocab.json`, no stress markers `ˈ`/`ˌ`), `LayoutInvariant` (`` framing, `word_starts`, `word_boundary_candidates`), and singing `phone_durations` positivity (`d >= 1` for every spoken Korean phone). 2. [`StreamingAndLanguageLifecycle.tla`](file:///G:/audiolm/scyllasband/korean_work/tla/StreamingAndLanguageLifecycle.tla): - Models language resolution (`resolve_language_for_voice` / `Engine::resolve_language`), acoustic conditioning index lookup (`language_index("ko") == 3`), group-speak `[voice:ko:delivery]` switching, and `_ChainStream` target/chunk boundaries (`prefix_latents` reset on voice/language change, preserved across continuations). 3. [`check_tla_models.py`](file:///G:/audiolm/scyllasband/korean_work/tla/check_tla_models.py): - Exhaustively verified **26,664 phonological states + 28 layout/singing states + 7,584 language/streaming states = 34,276 reachable states with 0 invariant violations**. --- ## 4. Empirical Verification & Performance Results ### 4.1 Native C++ Build & CTest (`16/16 Passed`) - Built `libscyllasband` with MSVC 2022 (`Release`) in `runtime/libscyllasband/build-msvc` (`scyllasband_native.dll`, `scyllasband_speak.exe`, `scyllasband_unit_test.exe`, `scyllasband_frontend_test.exe`) and staged `scyllasband_native.dll` + `libLiteRt.dll` into `runtime/scyllasband/_native/litert/`. - Regenerated C++ test fixtures (`generate_fixtures.py`) including Korean normalizer test cases and ran `ctest --test-dir runtime/libscyllasband/build-msvc -C Release --output-on-failure`: - **16/16 CTest suites passed in 0.53s** (`scyllasband_events`, `scyllasband_durations`, `scyllasband_context`, `scyllasband_delivery`, `scyllasband_sentences`, `scyllasband_segments`, `scyllasband_normalizer`, `scyllasband_unicode`, `scyllasband_targets`, `scyllasband_singing`, `scyllasband_records`, `scyllasband_frontend_noise`, `scyllasband_frontend_score`, `scyllasband_frontend_timeline`, `scyllasband_frontend_preview`, `scyllasband_frontend_gate`). ### 4.2 Python Unit & Property Tests (`131/131 Passed`) - Added [`runtime/tests/test_korean.py`](file:///G:/audiolm/scyllasband/runtime/tests/test_korean.py) covering Sino-Korean & Native Korean normalization, Standard Korean phonological rules, Hypothesis property-based fuzzing over random Hangul words, language resolution, group-speak Korean tags, and Korean singing score planning/rendering. - Ran `pytest runtime/tests` with `SCYLLASBAND_TEST_NATIVE_LIB` and `SCYLLASBAND_TEST_BUNDLE` enabled: - **131/131 tests passed in 2.51s** (0 skipped, 0 failed). ### 4.3 Full Python vs C++ Parity (`parity_harness.py` & `eval_korean_e2e.py`) - [`runtime/libscyllasband/tests/parity_harness.py`](file:///G:/audiolm/scyllasband/runtime/libscyllasband/tests/parity_harness.py) on `litert`: - `failures`: **0** - `frontend`: **280 cases, 0 mismatches** (`unicode_lower_block_mismatches: 0`, `nfc_mismatches: 0`) - `sentences`: **10/10 identical G2P & durations, `0.0` relative latent error, `Infinity dB` SNR** - `long_form`: **12/12 chunks & targets identical, `0.0` relative latent error, `Infinity dB` SNR** - `records`: **6 chains identical, `Infinity dB` SNR** - `singing`: **12/12 plans identical, 8/8 phrases `0.0` relative latent error (`vocos` & `bigvgan`), 4/4 full songs >234 dB SNR** - [`korean_work/scratch/eval_korean_e2e.py`](file:///G:/audiolm/scyllasband/korean_work/scratch/eval_korean_e2e.py) (saved in [`korean_work/scratch/korean_e2e_report.json`](file:///G:/audiolm/scyllasband/korean_work/scratch/korean_e2e_report.json)): - Korean Normalizer parity (Python vs C++): **15/15 cases, 0 mismatches** - Korean G2P parity (Python vs C++): **30/30 cases, 0 mismatches** - Korean Sentence synthesis parity: **identical durations (`130` frames), `0.0` relative latent error, `Infinity dB` audio SNR** - Korean Singing plan parity: **identical phrase plans, phones, and frame durations** ### 4.4 Korean Speech ASR & Multi-Voice / Singing Evaluation - **Multi-Voice Korean Synthesis (`scylla`, `gwen`, `felix`, `ariadne`, `ink`)**: - Reference: `"안녕하세요, 오늘 날씨가 정말 좋네요. 한국어 음성 합성을 시작합니다."` - `faster-whisper` (`large-v3-turbo`) CER: **0.0% across all 5 voices** (`5/5` exact transcriptions; samples saved in `korean_work/samples/ko_voice_*.wav`). - **10-Sentence Korean Benchmark (`scylla`, `korean_work/samples/ko_speech_scylla_00..09.wav`)**: - Overall CER: **20.12%** across 10 diverse sentences covering liaison, nasalization, liquidization, palatalization, tensification, and normalized numbers/counters, with **0.0% CER** on sentences `00`, `01`, `02`, and `05`. - **Korean Singing Synthesis (`melodies/phrase_ko_twinkle.json`, `"반짝 반짝 작은 별 아름답게 비치네"`)**: - Synthesized through both `vocos` (`ko_song_twinkle_scylla_vocos.wav`) and `bigvgan` (`ko_song_twinkle_scylla_bigvgan.wav`), verified by `scyllasband_speak.exe --score` and `runtime.sing(...)`. ### 4.5 Native C++ Runtime Performance (`scyllasband_speak.exe` on LiteRT CPU) - **Bundle Load Time**: `2.6 ms` - **Korean Speech (`5.33 s` audio, 4 Euler steps)**: - Cold first audio: `403.0 ms` - Warm first audio: **`138.2 ms`** - Warm total time: **`282.5 ms` (~18.9× real-time on CPU)** - **Korean Singing (`7.62 s` audio, `melodies/phrase_ko_twinkle.json`, Vocos turbo-sing, 4 Euler steps)**: - Warm first audio / total time: **`177.5 ms` (~42.9× real-time on CPU)** --- ## 5. Modified & Created Files Summary ### Model Repository (`G:\audiolm\scyllasband`) - `onnx/manifest.json`, `onnx/assets/languages.json`, `onnx/assets/g2p/language_map.json`, `onnx/assets/g2p/normalization.json` - `litert/manifest.json`, `litert/assets/languages.json`, `litert/assets/g2p/language_map.json`, `litert/assets/g2p/normalization.json` - `webgpu/manifest.json`, `webgpu/assets/languages.json`, `webgpu/assets/g2p/language_map.json`, `webgpu/assets/g2p/normalization.json` - `coreml/manifest.json`, `coreml/assets/languages.json`, `coreml/assets/g2p/language_map.json`, `coreml/assets/g2p/normalization.json` - `coreai/manifest.json`, `coreai/assets/languages.json`, `coreai/assets/g2p/language_map.json`, `coreai/assets/g2p/normalization.json` - `pytorch/release_config.json` - `melodies/phrase_ko_twinkle.json`, `melodies/README.md` - `README.md` (Updated with prominent Korean support section and usage guide) - `SHA256SUMS.json` (Recalculated and verified) - `PROGRESS.md` - `webui.py` (Interactive Gradio web interface) - `serve_webgpu_demo.py` (Local HTTP static server with COOP/COEP isolation headers) - `webgpu_demo/index.html`, `webgpu_demo/css/style.css`, `webgpu_demo/README.md` - `webgpu_demo/js/app.js`, `webgpu_demo/js/korean_g2p.js`, `webgpu_demo/js/pipeline.js`, `webgpu_demo/js/wav_utils.js` (Standalone 100% Client-Side WebGPU TTS Demo) - `korean_work/tla/KoreanPhonologyAndLayout.tla`, `korean_work/tla/StreamingAndLanguageLifecycle.tla`, `korean_work/tla/check_tla_models.py` - `korean_work/scratch/eval_korean_e2e.py`, `korean_work/scratch/korean_e2e_report.json`, `korean_work/scratch/update_bundle_metadata.py` - `korean_work/samples/*.wav` ### Runtime Repository (`G:\audiolm\scyllasband\runtime`) - `scyllasband/text_normalizer.py` - `scyllasband/g2p.py` - `scyllasband/planner.py` - `scyllasband/runtime.py` - `scyllasband/engine.py` - `tests/test_korean.py` - `libscyllasband/src/scyllasband_text.cpp` - `libscyllasband/src/scyllasband_g2p.h`, `libscyllasband/src/scyllasband_g2p.cpp` - `libscyllasband/src/scyllasband_engine.cpp` - `libscyllasband/tools/scyllasband_speak.cpp` - `libscyllasband/tests/generate_fixtures.py`, `libscyllasband/tests/fixtures/*.json` --- ## 6. Remaining Limitations & Future Work 1. **Zero-Shot Acoustic Conditioning Trade-offs**: - Because the `v4-20261106` acoustic graphs (`duration_predictor`, `pitch_estimator`, `vector_estimator`) were trained on 7 non-Korean locales without native Korean audio, Korean synthesis maps `ko` to syllable-timed acoustic conditioning index `3` and approximates Korean's 3-way laryngeal stop/affricate contrast (`평음/경음/격음`) and unrounded back vowels (`ㅓ/ㅡ`) using the closest trained IPA phones (`ɔ`, `ɯ`/`ə`, `k/ɡ/k h`). Complex consonant clusters across word boundaries (e.g., `설날 아침에 가족들과 떡국을`) still exhibit Romance prosodic liaison and vowel coloration. 2. **Next Steps When Native Korean Training Assets Become Available**: - Fine-tune or add a dedicated `ko` row (`index=7`) in `language_embed` and per-voice reference prosody tables (`conditions.*`), and include Hangul Jamo or syllable tokens in the neural G2P training corpus while retaining the `표준발음법` phonological frontend and text normalizer as the reference ground truth. ## WebGPU Standalone In-Browser Demo Milestone (2026-11-08) 1. **Standalone Architecture**: - `webgpu_demo/index.html`: Fully client-side UI with dark theme, preset phrases, 14 voices, prosody controls, and real-time canvas waveform visualizer. - `webgpu_demo/js/korean_g2p.js`: Pure JavaScript Standard Korean G2P & normalizer (Chosung/Jungsung/Jongsung decomposition, assimilation, nasalization, tensification, liquidization, palatalization, aspiration, and number/symbol normalization). - `webgpu_demo/js/pipeline.js`: Complete in-browser neural TTS pipeline (Euler ODE flow matching, duration prediction, pitch contour generation, and Vocos neural vocoder). - `webgpu_demo/dist/`: Self-contained `ort.webgpu.min.js`, `ort-wasm-simd-threaded.jsep.wasm`, and `ort-wasm-simd-threaded.jsep.mjs`. - `serve_webgpu_demo.py`: Python static server with explicit MIME types, COOP/COEP isolation headers, `--host`, and `--port` CLI flags. 2. **Root Cause Analysis of Duration Predictor (1/6) Error**: - Symptom: At model load stage 1/6 (`duration_predictor.onnx`), initialization failed with `undefined` / WASM C++ exception. - Investigation: - `webgpu/onnx/components/duration_predictor.onnx` uses `com.microsoft.MatMulNBits` with `bits: 8`. - In ONNX Runtime Web (JSEP & WASM CPU), `MatMulNBits` only supports `bits: 4` and `bits: 2` (`matmul_nbits.cc:115: nbits_ == 4 was false. Only 4b is supported`). - Emscripten caught the C++ exception and threw a numeric pointer (`11055512`), which resulted in `undefined` error string in the UI. - Resolution: - `onnx/onnx/components/` contains standard FP32 ONNX graphs (`MatMul`, `Gemm`, `LayerNormalization`, `Conv`, `Softmax`), which are 100% natively supported across both WebGPU and WASM. - Changed default bundle in `webgpu_demo/index.html` to `../onnx`. - Enhanced error handling in `app.js` and `pipeline.js` to catch numeric WASM exceptions and show actionable error diagnostics. - Fixed WASM base URL resolution in `pipeline.js` to support access via any network IP (`0.0.0.0` / LAN IP) without falling back to CDN. 3. **End-to-End Verification**: - Executed full client-side TTS synthesis test on Korean text ("안녕하세요. 반갑습니다."). - Result: 24,832 samples synthesized @ 24,000 Hz (1.03 sec duration, healthy RMS amplitude, peak [-0.36, +0.48], non-zero ratio 91.3%). 4. **Long-Form & Multi-Sentence Duration Support (2초 이상 합성 완벽 지원)**: - 원인: 초기 구현에서 고정 128 버킷(`bucket = 128`)만을 로드하여, 최대 지속 시간이 128 프레임 × 256 / 24,000 = **1.36초**로 제한되어 긴 문장이 잘리는 현상 발생. - 조치: - 512 프레임 버킷 모델(`pitch_estimator_512.onnx`, `vector_estimator_512.onnx`, `vocoder_512.onnx`)로 업그레이드 (단일 구문 최대 5.46초 지원). - 장문 다중 문장 분할 및 청크 단위 합성(`splitSentences`, `synthesizeChunk`) 파이프라인 구현: 문장 간 60ms 자연스러운 묵음 패딩 삽입 및 음성 파형 무결 결합. - 2초, 5초, 10초 이상의 장문 텍스트도 길이 제한 없이 연속 합성 가능하도록 확장 완료. 5. **2-Tier 로컬 브라우저 영구 캐싱 시스템 (Permanent Disk & CacheStorage Caching)**: - 문제: 매번 페이지 접속 또는 "WebGPU 모델 로드" 실행 시 약 1GB의 `shared_weights.bin`과 모델들을 네트워크로 다시 다운로드하던 비효율 발생. - Tier 1 (HTTP 레벨): `serve_webgpu_demo.py`에서 `.bin`, `.onnx`, `.wasm` 정적 파일에 대해 `Cache-Control: public, max-age=31536000, immutable` 헤더 전송 (브라우저 디스크 캐시 즉시 적중). - Tier 2 (클라이언트 레벨): `pipeline.js`의 `fetchWithCache`에서 브라우저 `window.caches` (CacheStorage API)를 적용하여 최초 다운로드 후 로컬 디스크에 영구 저장. 다음 실행부터는 네트워크 요청 없이 100~300ms 내에 즉시 로드. - UI 지원: "캐시 삭제" 버튼을 추가하여 필요 시 원클릭으로 브라우저 모델 캐시를 초기화할 수 있도록 지원. 6. **초경량 Vocos 전용 INT8 WebGPU 번들 재패키징 (230 MB) 및 ONNX Runtime Web 1.30.0 업그레이드**: - **배경 및 원인 분석**: - 원작자 데모(`https://lowkeytea.dev/scyllasband/demo/`)가 261MB에 불과했던 이유는 가창(Singing) 기능이 없는 v3 기반이었기 때문임. - v4로 판올림되면서 BigVGAN 보코더(단독으로 502.2 MB 점유)가 `shared_weights.bin`에 일괄 병합되어 번들 크기가 743.6 MB(INT8) / 1,036 MB(FP32)로 급증함. - 웹 음성 TTS 환경에서는 대형 신경망 보코더인 BigVGAN이 불필요하며, 실시간 추론에 최적화된 Vocos 보코더만으로 고품질 음성 합성이 가능함. - **초경량 번들 재패키징 수행**: - BigVGAN 텐서 502 MB를 완전히 분리·제외하고, 21개 음성 전용 모델(Duration Predictor, Pitch/Vector Context Encoders, Pitch/Vector Estimators 128~768, Vocos 128~768)의 795개 고유 텐서만을 64바이트 정렬로 순차 패킹. - `webgpu/onnx/components/shared_weights.bin` 크기를 743.6 MB에서 **241,440,320 바이트 (230.26 MB)**로 **3.2배 경량화** 완료 (저자 데모 230.2 MB와 완전 일치). - 21개 ONNX 모델의 `external_data` 오프셋을 재매핑하고 미사용 `vocoder_bigvgan*.onnx` 6개 파일을 제거. - `shared_weights.json` 및 `manifest.json`의 컴포넌트·버킷 명세를 Vocos 전용 번들 규격으로 갱신. - **ONNX Runtime Web 1.30.0 업그레이드**: - 기존 ORT 1.21.0에서 발생하던 8-bit `MatMulNBits` WASM CPU 폴백 예외(`nbits_ == 4 was false`)를 해결하기 위해, 최신 `onnxruntime-web@1.30.0` (WebGPU Asyncify WASM 및 JSEP) 바이너리를 `webgpu_demo/dist/`에 정식 배포. - WebGPU 환경에서 8-bit 양자화 가중치를 직접 GPU 셰이더 및 최신 WASM 런타임으로 가속할 수 있도록 파이프라인 구성. - `webgpu_demo/index.html` 기본 로드 경로를 `../webgpu` (230 MB INT8 번들)로 설정. - **종단간(End-to-End) 검증 완료**: - Scylla's Band 공식 런타임(`ScyllasBandRuntime.from_bundle`)에서 레이아웃 검증 통과 및 한국어 문장("안녕하세요! 반갑습니다.") 합성 테스트 완료 (2.432초, 58,368 샘플 @ 24kHz 정상 출력). - 로컬 HTTP 서버 및 정적 파일 MIME 타입, 2-Tier 캐싱, COOP/COEP 교차 출처 격리 헤더 검증 완료. 7. **Tailscale Secure Context HTTPS 및 WebGPU BufferManager Download 충돌 해결**: - **WASM Fallback 및 18초 추론 지연 원인 분석**: - 원인: \http://100.72.102.121:8000\ (사설 IP HTTP)로 접속 시 브라우저 W3C 보안 규격에 따라 비보안 컨텍스트(Insecure Context)로 분류됨. - 이로 인해 avigator.gpu\ (WebGPU) 및 \crossOriginIsolated\ (\SharedArrayBuffer\ 멀티스레딩)가 원천 차단되어 단일 스레드 CPU WASM(\ hreads: 1\)으로 폴백되며 추론 시간이 18초까지 폭증함. - **Tailscale HTTPS & Let's Encrypt 인증서 통합**: - Tailscale 공식 MagicDNS 인증서(\ ailscale cert dt4090sp.tail50666.ts.net\)를 발급받아 \serve_webgpu_demo.py\에 \--ssl\ 옵션을 추가하고 Python \ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)\를 탑재. - \https://dt4090sp.tail50666.ts.net:8000/webgpu_demo/\를 통해 별도의 플래그 조정 없이도 Tailnet 내 모든 모바일/PC 기기에서 정식 공인 보안 컨텍스트(Secure Context) 활성화 및 WebGPU + WASM 멀티스레딩 완전 가동. - \index.html\ 및 \pp.js\에 비보안 HTTP 접속 감지 시 HTTPS 도메인으로 안내하는 상태 배너 추가. - **\BufferManager::Download map_async_result.status == Success was false\ 오류 원인 및 해결**: - 오류 현상: WebGPU 모델 로드 후 음성 합성 실행 시 \ ailed to call OrtRun(). ERROR_CODE: 1, ERROR_MESSAGE: .../providers/webgpu/buffer_manager.cc:643 void onnxruntime::webgpu::BufferManager::Download(WGPUBuffer, void *, size_t) const map_async_result.status == wgpu::MapAsyncStatus::Success was false. Failed to download data from buffer. wgpu::MapAsyncStatus value: 0\ 발생. - 원인: \pipeline.js\에서 \Promise.all([ this.sessions.vector_context_encoder.run(contextFeeds), this.sessions.pitch_context_encoder.run(contextFeeds) ])\로 2개의 모델을 GPU에 동시 병렬 요청. ONNX Runtime Web의 WebGPU JSEP 백엔드는 동일 GPU 컨텍스트에서 중첩된 \mapAsync\ 스테이징 버퍼 다운로드를 처리하지 못해 두 번째 요청이 취소(Abort, Status: 0)됨. - 조치: \pipeline.js\의 동시 호출을 순차 \wait\ (\ ecContextOut = await ...; pitchContextOut = await ...;\)로 변경. - 검증: 격리 테스트(\ est_promise_all.html\)를 통해 \Promise.all\ 실패 재현 및 순차 호출 시 각 모델이 단 9ms 만에 정상 완료됨을 입증. 전체 6단계 WebGPU 합성 파이프라인(\ est_webgpu_synthesis.html\)에서 261,888 샘플(10.9초 오디오)을 1초 미만에 무결 합성 성공. 8. **WebGPU 오디오 지속 시간 1/2 절단(2.75초 잘림) 버그 해결 및 Prosody 이벤트 강화**: - **원인 분석**: - \manifest.json\의 \latent_hop_length: 512\ 및 공식 런타임(untime/scyllasband/engine.py\)의 \LATENT_HOP = 512\ 규격에 따라 지속 시간 예측기(\duration_predictor\)가 출력하는 1 프레임은 **512 오디오 샘플**(24,000Hz 기준 21.33ms)에 해당함. Vocos 512 버킷은 512 × 512 - 256 = 261,888 샘플(최대 10.92초)을 복원함. - 그러나 기존 \pipeline.js\에서 \const hopLength = 256;\으로 지정되어 있어, \ otalFrames * 256\으로 슬라이스되면서 모든 한국어 문장의 후반부 50%가 잘려 **정확히 2.75초 부근에서 오디오가 중간에 절단**되는 현상이 발생함. - **조치 내용**: - \pipeline.js\의 오디오 슬라이스 단위(\hopLength\)를 공식 잠재 홉 규격인 **512**로 수정 (\ctiveSamples = activeFrames * 512\). - 버킷 내 실제 전개된 활성 프레임 수(\cursor\) 기준으로 오디오 트림 범위를 정밀화하여 버킷 오버플로우 시에도 무결 슬라이싱 보장. - 문장 종결 부호(\?\, \!\, \...\)에 따른 \sentence_types\ (1: 진술, 2: 의문, 3: 감탄, 4: 말줄임표) 및 어휘 시작 경계(\wordStarts\ -> \oundaryEvents\) 매핑을 온전하게 전개하여 의문문·감탄문 억양 자연도 개선. - **검증**: - 96개 음소(한국어 2문장, 227프레임) 합성 시 기존 2.42~2.75초에서 잘리던 오디오가 4.84~5.69초의 온전한 파형으로 전체 음소 시퀀스가 처음부터 끝까지 완전 재생됨을 검증. 9. **Git Commit & Push 완료 및 Hugging Face Space 정식 배포**: - **모델 리포지토리 동기화 (cidsound/scyllasband)**: - WebGPU 버퍼 락 순차화, Vocos LATENT_HOP 512 지속 시간 수정, 다중 경로 복원력(Multi-tier fallback) 및 Prosody 이벤트 강화 반영 완료. - 로컬 Git 변경 사항 원격 main 브랜치에 완전 커밋 및 푸시 완료 (2b1b4d3). - **Hugging Face Space 배포 완료 (cidsound/scyllasband-webgpu)**: - Space 유형: Static Space (sdk: static — 무중단 24/7 가동, 콜드 스타트 0초, HF 글로벌 CDN 호스팅). - Space URL: [https://huggingface.co/spaces/acidsound/scyllasband-webgpu](https://huggingface.co/spaces/acidsound/scyllasband-webgpu) - 직접 접속 URL (전체화면): [https://acidsound-scyllasband-webgpu.static.hf.space](https://acidsound-scyllasband-webgpu.static.hf.space) - 230 MB 초경량 INT8 번들이 Space 내에 직접 배포되어 외부 의존성 없이 완전한 100% 자립 실행(Self-contained) 동작. - 배포 상태: Space stage: RUNNING, 모든 정적 파일, WASM 바이너리(pplication/wasm), 230 MB 가중치 및 ONNX 그래프 HTTP 200 서빙 검증 완료.