Hugging Face's logo

hdae
/
karume-sbv2-jvnv

Text-to-Speech
karume
webgpu
japanese
Model card Files
xet
 Community

Style-Bert-VITS2 JVNV β€” Karume

What is this

A Japanese text-to-speech distribution: Style-Bert-VITS2 JP-Extra voices converted into the WebGPU inference runtime Karume's container format (a .krm part sequence whose first part carries the graph and model descriptors). Runs as-is in the browser and in Deno.

  • The acoustic chain is shipped as fused graphs: text_encoder (a Japanese DeBERTa), front (phoneme encoder + duration predictors) and voice (flow + HiFi-GAN decoder).
  • Style and speaker are looked up at run time from the shipped tables β€” the names in the tables below index the rows of style_vectors / speaker_embeddings.
  • Not readable by Style-Bert-VITS2 (it's a different container with an embedded graph); the reader is a pipeline that implements sbv2/1.
  • Exporter used for the conversion: karume/0.13.0. The distribution manifest is karume.json (karume/5).

Base weights and attribution

Converted into the container format β€” the original checkpoints are not distributed here.

  • Voices: jvnv-F1-jp/ / jvnv-F2-jp/ / jvnv-M1-jp/ / jvnv-M2-jp/ of litagin/style_bert_vits2_jvnv
  • Architecture: Style-Bert-VITS2 JP-Extra (version: 2.0-JP-Extra)
  • Terms: CC BY-SA 4.0, inherited from the JVNV corpus the voices were trained on β€” the source repository states the corpus license carries over to the models. Redistribution here keeps it: credit the authors, name the source and the license URL, state that the weights were converted to another format and quantized to i8 (i4 for the group-quantized weights), license any derivative work under CC BY-SA 4.0 as well, and impose no further restrictions. There is no NonCommercial and no NoDerivatives clause β€” commercial use and modification are both allowed.
  • Changes made here: conversion into the Karume container format and quantization of the weights β€” the voices in f16 / i8 / mixed, the text encoder in i8 / mixed, where the mixed form stores the group-quantizable weights as i4 and everything else as i8 (the quant table below says which storage each quant selects). No retraining, no fine-tuning β€” the voices are the source checkpoints in a different storage form.
  • Text encoder: ku-nlp/deberta-v2-large-japanese-char-wwm, licensed cc-by-sa-4.0 (as of retrieval). It is redistributed here in the container format as the text_encoder component β€” the same license as the voices, so the share-alike term is consistent across everything in this repository.
  • Training data: the JVNV corpus, licensed CC BY-SA 4.0. Detai Xin, Junfeng Jiang, Shinnosuke Takamichi, Yuki Saito, Akiko Aizawa, Hiroshi Saruwatari, JVNV: A Corpus of Japanese Emotional Speech with Verbal Content and Nonverbal Expressions, arXiv:2310.06072. Corpus page: https://sites.google.com/site/shinnosuketakamichi/research-topics/jvnv_corpus
  • Training implementation: Style-Bert-VITS2 (AGPL-3.0). Karume is a separate implementation and does not contain the original implementation.

Models

Model Pipeline Quants Default quant
F1 (default) sbv2/1 f16+bert8 / i8 / i8-a8 / i8+bert4 / i4 i8+bert4
F2 sbv2/1 f16+bert8 / i8 / i8-a8 / i8+bert4 / i4 i8+bert4
M1 sbv2/1 f16+bert8 / i8 / i8-a8 / i8+bert4 / i4 i8+bert4
M2 sbv2/1 f16+bert8 / i8 / i8-a8 / i8+bert4 / i4 i8+bert4

model selects one of these; omitted, it is F1. quant defaults to that model's own default quant.

Usage 

import { analyzeWithWords } from "jsr:@hdae/yomi";
import { getDictionary } from "jsr:@hdae/yomi/loader";
import { encodeWav } from "jsr:@karume/models";
import { Pipeline, toUtterance } from "jsr:@karume/models/sbv2";

await using pipeline = await Pipeline.fromPretrained({
  repo: "hdae/karume-sbv2-jvnv",
  // Pin a commit for reproducible builds β€” without it you track `main`, and a future
  // repo update (renamed files, new manifest format) may break your app.
  // Copy the full hash from this repo's "Files and versions" tab:
  // revision: "<full commit sha>",
}, {
  // model: "F1", // default β€” available: F1 / F2 / M1 / M2
  // quant: "i8+bert4", // default β€” available: f16+bert8 / i4 / i8 / i8+bert4 / i8-a8
});

// Text analysis (readings and accents) is the caller's job:
const utterance = toUtterance(analyzeWithWords(await getDictionary(), "γ“γ‚“γ«γ‘γ―γ€γ“γ‚Œγ―γƒ†γ‚Ήγƒˆγ§γ™γ€‚"));

const audio = await pipeline.generate(utterance, {
  // Voice β€” the names below come from the tables further down; every model in
  // this repository brings its own set:
  // style: "Neutral", // default β€” available: Angry / Disgust / Fear / Happy / Neutral / Sad / Surprise
  // styleWeight: 1.0, // default β€” 0 = the average style, 1 = the named one
  // speaker: "jvnv-F1-jp", // default β€” available: jvnv-F1-jp

  // Delivery:
  // lengthScale: 1.0, // default β€” larger is slower
  // sdpRatio: 0.2, // default β€” 1 = stochastic duration, 0 = deterministic
  // noiseScale: 0.6, // default β€” sampling noise on z_p
  // noiseScaleW: 0.8, // default β€” sampling noise inside the stochastic predictor

  seed: 42, // same seed + same knobs β†’ same waveform
});
await Deno.writeFile("sbv2.wav", encodeWav(audio.data, audio.sampleRate));

generate() returns { sampleRate, data }, where data is an f32 mono waveform β€” exactly what encodeWav takes. Weights are fetched once and cached (verified against karume.json's size / sha256).

generate() takes an already-analyzed utterance: text analysis (readings and accents) belongs to the caller, and the package fetches no dictionary of its own. The snippet uses @hdae/yomi, whose dictionary (about 19 MB) is fetched on the first getDictionary() and read from the Cache API afterwards; to fix a reading, put yomi's overlay dictionary in front of the analysis. The SBV2 page of the Karume Playground is the worked version.

Model: F1

Quants

Quant What it is Download Weights Compute
f16+bert8 Highest fidelity (f16 synthesis) β€” front and voice in f16 storage, with the int8 text encoder β€” the largest download, and the closest match to the source checkpoint's audio. 414 MiB (295 MiB shared) text_encoder = i8 / front = f16 / voice = f16 β€”
i8 Half size (int8) β€” Every component stored as int8 and computed in f32 β€” roughly half the f16 download, with the execution path left unchanged. 356 MiB (295 MiB shared) text_encoder = i8 / front = i8 / voice = i8 β€”
i8-a8 int8 with int8 linear activations β€” The int8 weights with per-token int8 activations in the linear layers β€” faster on GPUs with dp4a, same download as the plain int8 seat. 356 MiB (295 MiB shared) text_encoder = i8 / front = i8 / voice = i8 linearCompute = a8
i8+bert4 (default) Balanced (int8 + int4 text encoder) β€” int8 synthesis with the text encoder in GPTQ-calibrated int4 (group-32): a markedly smaller download that stays audibly level with f32. 248 MiB (187 MiB shared) text_encoder = i4 / front = i8 / voice = i8 β€”
i4 Smallest (int4) β€” Text encoder, front and voice all in int4 group-32 β€” the smallest download and the fastest warm start, with slightly lower tension than f32. 226 MiB (187 MiB shared) text_encoder = i4 / front = i4 / voice = i4 β€”

If no quant is given, it runs as i8+bert4 (this model's recommended default). In a quant name, bert is the text_encoder component. Per-file size and sha256 live in karume.json β€” verify against that at the fetch layer. Dtype labels use the runtime's storage dtype vocabulary (f16 / bf16 / i8 / i4 / i2), not the fp16 spelling common elsewhere in the ecosystem. Weights ship as Karume container files (.krm), split into numbered parts; a part is fetched and verified on its own. A path under shared/ is one this model shares byte for byte with another model in this repository (it is fetched and cached once).

How the stored weights were rounded:

  • f16+bert8 β€” front / voice in f16, each weight rounded to the nearest f16 value; text_encoder in i8, rounded per output channel (plain RTN).
  • i8 β€” every weight in i8, rounded per output channel (plain RTN).
  • i8-a8 β€” the same stored weights as i8. Nothing more is rounded at export time: the linear layers quantize their input activations to int8 per token at run time.
  • i8+bert4 (default) β€” text_encoder in i4 group-32: its linear layers rounded with GPTQ calibration (a 48-sentence Japanese corpus), its embedding table plain RTN. front / voice stay i8.
  • i4 β€” the same text encoder as i8+bert4, plus front / voice linear and conv1d weights in i4 group-32 with plain RTN. Smallest download and fastest warm synthesis; the output is very close to i8+bert4, with slightly lower tension than the source checkpoint's unquantized f32 reference.

Styles

Style ID
Neutral 0
Angry 1
Disgust 2
Fear 3
Happy 4
Sad 5
Surprise 6

style takes one of these names β€” the ID is the row it selects in F1/styles/style_vectors.safetensors. styleWeight blends between the average style (0) and the named one (1).

Speakers

Speaker ID
jvnv-F1-jp 0

speaker takes one of these names β€” the ID is the row it selects in F1/speakers/speaker_embeddings.safetensors.

Defaults

Any knob not passed to generate() is filled in from the manifest's defaults.

  • speaker: jvnv-F1-jp
  • style: Neutral
  • styleWeight: 1.0
  • sdpRatio: 0.2
  • noiseScale: 0.6
  • noiseScaleW: 0.8
  • lengthScale: 1.0

seed is the one knob the manifest does not carry β€” it defaults to 0, and the same seed with the same knobs gives the same waveform.

Model: F2

Quants

Quant What it is Download Weights Compute
f16+bert8 Highest fidelity (f16 synthesis) β€” front and voice in f16 storage, with the int8 text encoder β€” the largest download, and the closest match to the source checkpoint's audio. 414 MiB (295 MiB shared) text_encoder = i8 / front = f16 / voice = f16 β€”
i8 Half size (int8) β€” Every component stored as int8 and computed in f32 β€” roughly half the f16 download, with the execution path left unchanged. 356 MiB (295 MiB shared) text_encoder = i8 / front = i8 / voice = i8 β€”
i8-a8 int8 with int8 linear activations β€” The int8 weights with per-token int8 activations in the linear layers β€” faster on GPUs with dp4a, same download as the plain int8 seat. 356 MiB (295 MiB shared) text_encoder = i8 / front = i8 / voice = i8 linearCompute = a8
i8+bert4 (default) Balanced (int8 + int4 text encoder) β€” int8 synthesis with the text encoder in GPTQ-calibrated int4 (group-32): a markedly smaller download that stays audibly level with f32. 248 MiB (187 MiB shared) text_encoder = i4 / front = i8 / voice = i8 β€”
i4 Smallest (int4) β€” Text encoder, front and voice all in int4 group-32 β€” the smallest download and the fastest warm start, with slightly lower tension than f32. 226 MiB (187 MiB shared) text_encoder = i4 / front = i4 / voice = i4 β€”

If no quant is given, it runs as i8+bert4 (this model's recommended default). In a quant name, bert is the text_encoder component. Per-file size and sha256 live in karume.json β€” verify against that at the fetch layer. Dtype labels use the runtime's storage dtype vocabulary (f16 / bf16 / i8 / i4 / i2), not the fp16 spelling common elsewhere in the ecosystem. Weights ship as Karume container files (.krm), split into numbered parts; a part is fetched and verified on its own. A path under shared/ is one this model shares byte for byte with another model in this repository (it is fetched and cached once).

How the stored weights were rounded:

  • f16+bert8 β€” front / voice in f16, each weight rounded to the nearest f16 value; text_encoder in i8, rounded per output channel (plain RTN).
  • i8 β€” every weight in i8, rounded per output channel (plain RTN).
  • i8-a8 β€” the same stored weights as i8. Nothing more is rounded at export time: the linear layers quantize their input activations to int8 per token at run time.
  • i8+bert4 (default) β€” text_encoder in i4 group-32: its linear layers rounded with GPTQ calibration (a 48-sentence Japanese corpus), its embedding table plain RTN. front / voice stay i8.
  • i4 β€” the same text encoder as i8+bert4, plus front / voice linear and conv1d weights in i4 group-32 with plain RTN. Smallest download and fastest warm synthesis; the output is very close to i8+bert4, with slightly lower tension than the source checkpoint's unquantized f32 reference.

Styles

Style ID
Neutral 0
Angry 1
Disgust 2
Fear 3
Happy 4
Sad 5
Surprise 6

style takes one of these names β€” the ID is the row it selects in F2/styles/style_vectors.safetensors. styleWeight blends between the average style (0) and the named one (1).

Speakers

Speaker ID
jvnv-F2-jp 0

speaker takes one of these names β€” the ID is the row it selects in F2/speakers/speaker_embeddings.safetensors.

Defaults

Any knob not passed to generate() is filled in from the manifest's defaults.

  • speaker: jvnv-F2-jp
  • style: Neutral
  • styleWeight: 1.0
  • sdpRatio: 0.2
  • noiseScale: 0.6
  • noiseScaleW: 0.8
  • lengthScale: 1.0

seed is the one knob the manifest does not carry β€” it defaults to 0, and the same seed with the same knobs gives the same waveform.

Model: M1

Quants

Quant What it is Download Weights Compute
f16+bert8 Highest fidelity (f16 synthesis) β€” front and voice in f16 storage, with the int8 text encoder β€” the largest download, and the closest match to the source checkpoint's audio. 414 MiB (295 MiB shared) text_encoder = i8 / front = f16 / voice = f16 β€”
i8 Half size (int8) β€” Every component stored as int8 and computed in f32 β€” roughly half the f16 download, with the execution path left unchanged. 356 MiB (295 MiB shared) text_encoder = i8 / front = i8 / voice = i8 β€”
i8-a8 int8 with int8 linear activations β€” The int8 weights with per-token int8 activations in the linear layers β€” faster on GPUs with dp4a, same download as the plain int8 seat. 356 MiB (295 MiB shared) text_encoder = i8 / front = i8 / voice = i8 linearCompute = a8
i8+bert4 (default) Balanced (int8 + int4 text encoder) β€” int8 synthesis with the text encoder in GPTQ-calibrated int4 (group-32): a markedly smaller download that stays audibly level with f32. 248 MiB (187 MiB shared) text_encoder = i4 / front = i8 / voice = i8 β€”
i4 Smallest (int4) β€” Text encoder, front and voice all in int4 group-32 β€” the smallest download and the fastest warm start, with slightly lower tension than f32. 226 MiB (187 MiB shared) text_encoder = i4 / front = i4 / voice = i4 β€”

If no quant is given, it runs as i8+bert4 (this model's recommended default). In a quant name, bert is the text_encoder component. Per-file size and sha256 live in karume.json β€” verify against that at the fetch layer. Dtype labels use the runtime's storage dtype vocabulary (f16 / bf16 / i8 / i4 / i2), not the fp16 spelling common elsewhere in the ecosystem. Weights ship as Karume container files (.krm), split into numbered parts; a part is fetched and verified on its own. A path under shared/ is one this model shares byte for byte with another model in this repository (it is fetched and cached once).

How the stored weights were rounded:

  • f16+bert8 β€” front / voice in f16, each weight rounded to the nearest f16 value; text_encoder in i8, rounded per output channel (plain RTN).
  • i8 β€” every weight in i8, rounded per output channel (plain RTN).
  • i8-a8 β€” the same stored weights as i8. Nothing more is rounded at export time: the linear layers quantize their input activations to int8 per token at run time.
  • i8+bert4 (default) β€” text_encoder in i4 group-32: its linear layers rounded with GPTQ calibration (a 48-sentence Japanese corpus), its embedding table plain RTN. front / voice stay i8.
  • i4 β€” the same text encoder as i8+bert4, plus front / voice linear and conv1d weights in i4 group-32 with plain RTN. Smallest download and fastest warm synthesis; the output is very close to i8+bert4, with slightly lower tension than the source checkpoint's unquantized f32 reference.

Styles

Style ID
Neutral 0
Angry 1
Disgust 2
Fear 3
Happy 4
Sad 5
Surprise 6

style takes one of these names β€” the ID is the row it selects in M1/styles/style_vectors.safetensors. styleWeight blends between the average style (0) and the named one (1).

Speakers

Speaker ID
jvnv-M1-jp 0

speaker takes one of these names β€” the ID is the row it selects in M1/speakers/speaker_embeddings.safetensors.

Defaults

Any knob not passed to generate() is filled in from the manifest's defaults.

  • speaker: jvnv-M1-jp
  • style: Neutral
  • styleWeight: 1.0
  • sdpRatio: 0.2
  • noiseScale: 0.6
  • noiseScaleW: 0.8
  • lengthScale: 1.0

seed is the one knob the manifest does not carry β€” it defaults to 0, and the same seed with the same knobs gives the same waveform.

Model: M2

Quants

Quant What it is Download Weights Compute
f16+bert8 Highest fidelity (f16 synthesis) β€” front and voice in f16 storage, with the int8 text encoder β€” the largest download, and the closest match to the source checkpoint's audio. 414 MiB (295 MiB shared) text_encoder = i8 / front = f16 / voice = f16 β€”
i8 Half size (int8) β€” Every component stored as int8 and computed in f32 β€” roughly half the f16 download, with the execution path left unchanged. 356 MiB (295 MiB shared) text_encoder = i8 / front = i8 / voice = i8 β€”
i8-a8 int8 with int8 linear activations β€” The int8 weights with per-token int8 activations in the linear layers β€” faster on GPUs with dp4a, same download as the plain int8 seat. 356 MiB (295 MiB shared) text_encoder = i8 / front = i8 / voice = i8 linearCompute = a8
i8+bert4 (default) Balanced (int8 + int4 text encoder) β€” int8 synthesis with the text encoder in GPTQ-calibrated int4 (group-32): a markedly smaller download that stays audibly level with f32. 248 MiB (187 MiB shared) text_encoder = i4 / front = i8 / voice = i8 β€”
i4 Smallest (int4) β€” Text encoder, front and voice all in int4 group-32 β€” the smallest download and the fastest warm start, with slightly lower tension than f32. 226 MiB (187 MiB shared) text_encoder = i4 / front = i4 / voice = i4 β€”

If no quant is given, it runs as i8+bert4 (this model's recommended default). In a quant name, bert is the text_encoder component. Per-file size and sha256 live in karume.json β€” verify against that at the fetch layer. Dtype labels use the runtime's storage dtype vocabulary (f16 / bf16 / i8 / i4 / i2), not the fp16 spelling common elsewhere in the ecosystem. Weights ship as Karume container files (.krm), split into numbered parts; a part is fetched and verified on its own. A path under shared/ is one this model shares byte for byte with another model in this repository (it is fetched and cached once).

How the stored weights were rounded:

  • f16+bert8 β€” front / voice in f16, each weight rounded to the nearest f16 value; text_encoder in i8, rounded per output channel (plain RTN).
  • i8 β€” every weight in i8, rounded per output channel (plain RTN).
  • i8-a8 β€” the same stored weights as i8. Nothing more is rounded at export time: the linear layers quantize their input activations to int8 per token at run time.
  • i8+bert4 (default) β€” text_encoder in i4 group-32: its linear layers rounded with GPTQ calibration (a 48-sentence Japanese corpus), its embedding table plain RTN. front / voice stay i8.
  • i4 β€” the same text encoder as i8+bert4, plus front / voice linear and conv1d weights in i4 group-32 with plain RTN. Smallest download and fastest warm synthesis; the output is very close to i8+bert4, with slightly lower tension than the source checkpoint's unquantized f32 reference.

Styles

Style ID
Neutral 0
Angry 1
Disgust 2
Fear 3
Happy 4
Sad 5
Surprise 6

style takes one of these names β€” the ID is the row it selects in M2/styles/style_vectors.safetensors. styleWeight blends between the average style (0) and the named one (1).

Speakers

Speaker ID
jvnv-M2-jp 0

speaker takes one of these names β€” the ID is the row it selects in M2/speakers/speaker_embeddings.safetensors.

Defaults

Any knob not passed to generate() is filled in from the manifest's defaults.

  • speaker: jvnv-M2-jp
  • style: Neutral
  • styleWeight: 1.0
  • sdpRatio: 0.2
  • noiseScale: 0.6
  • noiseScaleW: 0.8
  • lengthScale: 1.0

seed is the one knob the manifest does not carry β€” it defaults to 0, and the same seed with the same knobs gives the same waveform.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
Text-to-Speech
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for hdae/karume-sbv2-jvnv

Base model

ku-nlp/deberta-v2-large-japanese-char-wwm
Quantized
(4)
this model

Paper for hdae/karume-sbv2-jvnv