| --- |
| license: apache-2.0 |
| pipeline_tag: text-to-speech |
| language: |
| - zh |
| tags: |
| - vits |
| - text-to-speech |
| - tts |
| - csmsc |
| - paddlespeech |
| - chinese |
| - robotics |
| - IB-Robot |
| - ascend |
| - edge-deployment |
| --- |
| |
| # Model Card for VITS-CSMSC (IB-Robot, hybrid CPU + Ascend 310P) |
|
|
| PaddleSpeech VITS-CSMSC — single-speaker Mandarin text-to-speech, 22.05 kHz — |
| packaged for the [IB-Robot](https://atomgit.com/openeuler/IB_Robot) framework |
| as a **hybrid deployment**: the text encoder and duration predictor run as ONNX |
| on CPU, and the flow decoder (which contains the vocoder and dominates compute) |
| runs as an `.om` on Ascend 310P1. |
|
|
| This is the speech-output stage of the robot pipeline. Accepts arbitrary |
| Chinese text; the voice is the fixed CSMSC speaker. |
|
|
| ## Deployments |
|
|
| | deployment | backend | flow artifact | frames | notes | |
| |---|---|---|---:|---| |
| | `ascend_310p` | ONNX CPU + Ascend ACL | `vits_flow_decoder_dynamic_fp16.om` | variable | **recommended**, any utterance length | |
| | `torch_cpu` | ONNX Runtime CPU | `artifacts/onnx/vits_flow_decoder.onnx` | variable | full CPU reference path | |
|
|
| Two fixed-shape flow decoders also ship in the repo without their own |
| deployment entry, matching the convention across this organisation that one |
| hardware target carries one deployment: |
|
|
| | file | frames | audio | notes | |
| |---|---:|---:|---| |
| | `vits_flow_decoder_t639_fp16.om` | 639 | 7.42 s | ~20 % lower flow latency than dynamic, 44 % of its size | |
| | `vits_flow_decoder_t681_fp16.om` | 681 | 7.91 s | second fixed-shape build | |
|
|
| Both are declared in `bundle.files` and load by pointing an ACL session at the |
| file directly. They accept **exactly** their frame count, so an utterance has |
| to be padded or trimmed to fit; the dynamic build takes any length, which is |
| why it is the one wired to `ascend_310p`. Accuracy for all three is in |
| [Validation](#validation) — they are within 1e-5 cosine of each other. |
|
|
| The `ascend_310p` row is a **composite deployment** in the manifest: there is no |
| single `runtime_profile`, because there is no single backend. `role_identities` |
| and `role_runtime_profiles` name the backend per role — `onnx` on CPU for |
| `text_encoder` and `duration`, `ascend`/`acl` for `flow_decoder`. `torch_cpu` |
| runs all three roles on ONNX Runtime and so keeps one shared profile. |
|
|
| The two host-side steps between duration and flow (`duration_expand_path`, |
| `sample_z_p`) are listed in each deployment's `metadata.host_stages`; they are |
| not model roles and have no artifact. |
|
|
| Shared contract: `tensor_model/vits/synthesize` |
| (`tts.phone_ids [1,-1] -> tts.audio [1,-1]`), preprocessing |
| `paddlespeech-zh-frontend-jieba-pypinyin-phoneid-v1`, output |
| `mono-float32-pcm-22050hz-v1`, logical revision |
| `paddlespeech-vits-csmsc@onnx-export-v1`. |
|
|
| `ascend_310p` uses the dynamic-shape flow decoder, so no padding or trimming |
| is needed. For latency-sensitive use with a known utterance length, load one of |
| the two fixed-shape builds above directly. |
|
|
| ## Pipeline |
|
|
| ```text |
| Chinese text |
| → PaddleSpeech frontend / pypinyin → phone ids |
| → text encoder (ONNX, CPU) → x, m_p, logs_p (192-d), x_mask |
| → duration predictor (ONNX, CPU) → logw |
| → host: alignment expansion + z_p (see below) |
| → flow decoder (.om, Ascend 310P1) → waveform |
| → 22.05 kHz mono wav |
| ``` |
|
|
| ### Host stage (between duration and flow) |
|
|
| This is not a thin glue layer — it is part of the model's semantics and must be |
| reproduced exactly. See `assets/reference_impl/run_vits_hybrid_310p.py`: |
|
|
| ```python |
| dur = ceil(exp(logw) * float32(x_mask)) # per-phone frame counts |
| path = generate_path(dur, ...) # [1, T_text, T_frames] alignment |
| m = (path.transpose @ m_p.transpose).transpose # expand to frames |
| logs = (path.transpose @ logs_p.transpose).transpose |
| y_mask = ones([1, 1, T_frames]) |
| z_p = m + randn_like(m) * exp(logs) * 0.667 # noise_scale 0.667 |
| ``` |
|
|
| The reference uses **seed 1234** for reproducibility; the sampling of `z_p` is |
| genuinely stochastic, so runs with different seeds produce different (equally |
| valid) audio, and bit-exact comparison against a reference requires pinning it. |
|
|
| ## Tensor contract |
|
|
| ```text |
| text encoder text int64 [1, -1] phone ids |
| text_lengths int64 [1] |
| → x float32 [1, 192, -1] outputs are positional and |
| m_p float32 [1, 192, -1] come back in this order |
| logs_p float32 [1, 192, -1] |
| x_mask bool [1, 1, -1] |
| |
| duration x float32 [1, 192, -1] |
| x_mask bool [1, 1, -1] bool, not float |
| → logw float32 [1, 1, -1] |
| |
| flow decoder z_p float32 [1, 192, T] T = 639 / 681 / -1 |
| y_mask float32 [1, 1, T] |
| → wav float32 [1, T*256] 256 samples per frame |
| ``` |
|
|
| Sample rate 22050 Hz, hop 256 → one frame ≈ 11.61 ms. |
|
|
| ## Repository Structure |
|
|
| - `inference_manifest.json` — deployment routing (schema v3) |
| - `assets/adapter.json` — adapter identity (`vits` / `synthesize`) |
| - `assets/phone_id_map.txt` — 268-entry phone→id map used by the frontend |
| - `assets/snapshot_iter_150000.pdz` — the upstream Paddle training checkpoint |
| (1.1 GB), byte-identical to the one in `vits_csmsc_ckpt_1.4.0.zip` |
| - `assets/default.yaml` — that checkpoint's own training config |
| - `assets/reference_impl/run_vits_hybrid_310p.py` — the hybrid runner, and the |
| normative source for the host stage above |
| - `assets/reference_impl/synthesize_vits_310p.py` — text → wav CLI |
| - `artifacts/onnx/*.onnx` — the three ONNX graphs (CPU stages + CPU reference) |
| - `artifacts/ascend_310p/*.om` — three flow decoder builds |
|
|
| ## Source Model |
|
|
| Upstream is **PaddleSpeech VITS-CSMSC**, released checkpoint |
| **`vits_csmsc_ckpt_1.4.0`**. The manifest's logical revision is |
| `paddlespeech-vits-csmsc@onnx-export-v1` — it names the ONNX export this bundle |
| runs, not the upstream release tag, which is `1.4.0`. |
| |
| ```bash |
| wget https://paddlespeech.cdn.bcebos.com/Parakeet/released_models/vits/\ |
| vits_csmsc_ckpt_1.4.0.zip |
| # 1,027,863,302 bytes |
| # sha256 994340036777b24f3bf02967bf37f570d40e5514b54ff0001c5b4732921ae064 |
| ``` |
| |
| The archive holds exactly three files, and **all three are shipped here |
| byte-for-byte** (sha256 verified against a fresh download and extraction): |
| |
| | shipped as | in archive | bytes | sha256 | |
| |---|---|---:|---| |
| | `assets/snapshot_iter_150000.pdz` | `snapshot_iter_150000.pdz` | 1,116,898,380 | `ef0faad8186cdd1aa278b61e0b2a3cb7dee9216d1b84f1bc760ec887bcd935c7` | |
| | `assets/default.yaml` | `default.yaml` | 7,835 | `7eebdd044a663f2485b6abd54f6179e90da64c0c7c109778295f9a23be6dcbfc` | |
| | `assets/phone_id_map.txt` | `phone_id_map.txt` | 2,128 | `ca00d60618a00a2dc1e42002faaa9024c9e5057e04b1e834325950111cee59c3` | |
| |
| `phone_id_map.txt` pins the phone inventory, and therefore the text frontend |
| contract, to this exact release. `default.yaml` is the checkpoint's own training |
| config and independently confirms the numbers this bundle's tensor contract |
| depends on: `fs: 22050`, `n_shift: 256` (the 256 samples per frame above) and |
| `hidden_channels: 192` (the 192-d `z_p`). |
| |
| Shipping the checkpoint follows the convention in the rest of this |
| organisation, where the upstream weights live under `assets/`. Note that it is |
| **not** the artifact any deployment loads — every deployment runs the ONNX |
| graphs below. It is there so the bundle carries its own provenance. |
|
|
| The three ONNX graphs were exported from that checkpoint in separate passes, |
| which is why their opsets differ (12 / 13 / 11): |
|
|
| | file | bytes | opset | sha256 | |
| |---|---:|---:|---| |
| | `vits_text_encoder_clean.onnx` | 55,329,745 | 12 | `15c5fb01538ce38c4748c9bc5892fc0af9bee7c65b228cd116651237a3f72536` | |
| | `vits_duration.onnx` | 2,644,862 | 13 | `f75fd325801a62976ecea0f52ba03a384197f574fedf0486598e5c0fc840c89c` | |
| | `vits_flow_decoder.onnx` | 86,596,889 | 11 | `3cb9dfe036b843abce71b867626f23d60745c0f0f35f29787ecec7fbfaf0f7fa` | |
| | `vits_flow_decoder_t639_fp16.om` | 91,708,818 | — | `b7da515895536b7baf75314b908344164efb916ffc8cc2246c6dac0e4c4cf91b` | |
| | `vits_flow_decoder_t681_fp16.om` | 91,706,422 | — | `df192baf48cefb732f52945cd411d3c88bbd310cac784dc51e11a8a7ed0d2b61` | |
| | `vits_flow_decoder_dynamic_fp16.om` | 210,986,822 | — | `c8b26160a0bc7e4de987424fab228bcc8dfd3fb053bd910737443999412c40d8` | |
|
|
| Shipping the checkpoint does **not** make the ONNX byte-reproducible from it: |
| the export script is not part of this bundle, and PaddleSpeech static-model |
| export behaviour changed with PaddlePaddle ≥ 2.5. What *is* established is the |
| accuracy of the export — cosine `0.9999999999998159` against the Paddle |
| reference, below — plus the exact checkpoint and config it came from. |
|
|
| ## Validation |
|
|
| All measured on a real Ascend 310P1, against ONNX Runtime CPU fp32. |
|
|
| **ONNX export vs Paddle reference** (text encoder + duration) — the export |
| itself is lossless: |
|
|
| ```text |
| cosine 0.9999999999998159 |
| max_abs 4.291534423828125e-06 |
| ``` |
|
|
| **Flow decoder OM vs ONNX**, on the flow subgraph alone: |
|
|
| | build | cosine | max_abs | latency mean | p95 | subgraph RTF | |
| |---|---:|---:|---:|---:|---:| |
| | t639 | 0.9999842557218173 | 0.01094835 | 96.17 ms | 96.38 ms | 0.01296 | |
| | t681 | 0.9999692545131995 | 0.01450025 | 102.72 ms | — | 0.01299 | |
| |
| Those are the conversion-time numbers. They have since been **independently |
| reproduced on the shipped artifacts** — same board, same `artifacts/onnx/ |
| vits_flow_decoder.onnx` reference, but with `z_p` built through the real |
| encoder → duration → host-alignment path rather than synthetic tensors |
| (`_tools/board/verify_vits_flow_om_vs_onnx.py`): |
|
|
| | build | frames | cosine | max_abs | rel_l2 | latency mean | p95 | subgraph RTF | |
| |---|---:|---:|---:|---:|---:|---:|---:| |
| | t639 | 639 | 0.99998847 | 0.008860 | 0.00480 | 95.89 ms | 96.00 ms | 0.01292 | |
| | t681 | 681 | 0.99998923 | 0.008666 | 0.00464 | 102.47 ms | 102.71 ms | 0.01296 | |
| | dynamic | 399 | 0.99998542 | 0.006929 | 0.00540 | 77.32 ms | 77.90 ms | 0.01669 | |
|
|
| Latency matches the conversion report to within 0.3 ms on every build, and |
| accuracy comes out slightly *better* on realistic `z_p` than on the synthetic |
| input used at conversion time. The dynamic build's higher RTF at 399 frames is |
| the fixed per-call overhead spread over less audio, not a slower kernel — at |
| 639 frames it lands within ~40 ms of the fixed build (see below). |
|
|
| **End-to-end hybrid pipeline**, 20 runs, 7.42 s of output audio. Measured with |
| the fixed **t639** build, which is what the reference runner loads |
| (`assets/reference_impl/run_vits_hybrid_310p.py:15`); the same pipeline on the |
| `ascend_310p` dynamic build is the `baseline` row two tables down, 367.63 ms: |
|
|
| ```text |
| cosine_vs_onnx 0.9999853236698978 |
| max_abs 0.008778512477874756 |
| latency mean 327.38 ms p50 323.81 ms p95 355.89 ms |
| RTF 0.0441 |
| ``` |
|
|
| Per-stage breakdown (mean, ms): |
|
|
| | stage | backend | ms | share | |
| |---|---|---:|---:| |
| | text encoder | ONNX CPU | 48.71 | 15% | |
| | duration | ONNX CPU | 25.38 | 8% | |
| | host alignment + z_p | numpy CPU | 156.64 | 48% | |
| | flow decoder | Ascend OM | 96.53 | 29% | |
| |
| The host alignment stage is now the single largest cost — larger than the NPU |
| work it feeds. It is pure numpy (`generate_path` plus two matmuls) and is the |
| obvious next optimisation target; moving it onto the NPU or vectorising it |
| would cut end-to-end latency by roughly a third. |
|
|
| **Dynamic build, variable text length:** |
|
|
| | sample | phones | frames | audio | latency mean | RTF | |
| |---|---:|---:|---:|---:|---:| |
| | baseline | 101 | 639 | 7.4188 s | 367.63 ms | 0.04955 | |
| | new_text | 144 | 915 | 10.6231 s | 414.97 ms | 0.03906 | |
| |
| Latency grows sub-linearly with length, so RTF improves on longer utterances. |
| The dynamic build costs ~40 ms over the fixed t639 build at the same length. |
| |
| ## Usage |
| |
| ```python |
| from inference_manifest import load_inference_manifest |
|
|
| validated = load_inference_manifest("models/vits-csmsc", "ascend_310p") |
| ``` |
| |
| The hybrid path needs both an ONNX Runtime CPU session (encoder, duration) and |
| an ACL session (flow), plus the host stage between them — see |
| `assets/reference_impl/run_vits_hybrid_310p.py`, which is the normative |
| implementation. |
| |
| ## Known Limitations |
| |
| - **Not a pure-NPU TTS.** Text encoder and duration remain on CPU; only the |
| flow decoder is compiled to OM. |
| - **Chinese only.** English and other languages are not a validation target. |
| - **Fixed speaker.** Single-speaker CSMSC voice, no speaker conditioning. |
| - **The two extra fixed-shape OMs require exact frame counts** (639 / 681). |
| The `ascend_310p` deployment uses the dynamic build and has no such limit; |
| the fixed builds are for callers who know their length and want the ~40 ms. |
| - Output is stochastic (`noise_scale = 0.667`) unless the RNG seed is pinned. |
|
|
| ## Environment |
|
|
| Validated on: Ascend 310P1 with CANN and `ais_bench`, Python 3.11.6, |
| ONNX Runtime 1.23.2. |
|
|
| ## License |
|
|
| Code and packaging: Apache-2.0. Model weights follow the upstream |
| [PaddleSpeech](https://github.com/PaddlePaddle/PaddleSpeech) license |
| (Apache-2.0); the CSMSC corpus is subject to its own terms. |
|
|
| ## Citation |
|
|
| @inproceedings{vits2021, |
| title = {Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech}, |
| author = {Kim, Jaehyeon and Kong, Jungil and Son, Juhee}, |
| booktitle = {arXiv preprint arXiv:2106.06103}, |
| year = {2021} |
| } |
| @article{paddlespeech2022, |
| title = {PaddleSpeech: An Easy-to-Use All-in-One Speech Toolkit}, |
| author = {Zhang, Hui and Yuan, Tian and Chen, Junkun and Li, Xintong and Zheng, Renjie and Huang, Yuxin and Chen, Xiaojie and Gong, Enlei and Chen, Zeyu and Hu, Xiaoguang and Yu, Dianhai and Ma, Yanjun and Huang, Liang}, |
| journal = {arXiv preprint arXiv:2205.12007}, |
| year = {2022} |
| } |
| @software{ib_robot, |
| title = {IB-Robot: Intelligence Boom Robot}, |
| url = {https://atomgit.com/openeuler/IB_Robot}, |
| license = {Apache-2.0} |
| } |
| |
