Hugging Face's logo

hdae
/
karume-sbv2-jvnv

Text-to-Speech
karume
webgpu
japanese
Model card Files
xet
 Community

Style-Bert-VITS2 JVNV β€” Karume

What is this

A Japanese text-to-speech distribution: Style-Bert-VITS2 JP-Extra voices converted into the WebGPU inference runtime Karume's container format (a .krm part sequence whose first part carries the graph and model descriptors). Runs as-is in the browser and in Deno.

  • The acoustic chain is shipped as fused graphs: text_encoder (a Japanese DeBERTa), front (phoneme encoder + duration predictors) and voice (flow + HiFi-GAN decoder).
  • Style and speaker are looked up at run time from the shipped tables β€” the names in the tables below index the rows of style_vectors / speaker_embeddings.
  • Not readable by Style-Bert-VITS2 (it's a different container with an embedded graph); the reader is a pipeline that implements sbv2/1.
  • Exporter used for the conversion: karume/0.13.0. The distribution manifest is karume.json (karume/5).

Base weights and attribution

Converted into the container format β€” the original checkpoints are not distributed here.

  • Voices: jvnv-F1-jp/ / jvnv-F2-jp/ / jvnv-M1-jp/ / jvnv-M2-jp/ of litagin/style_bert_vits2_jvnv
  • Architecture: Style-Bert-VITS2 JP-Extra (version: 2.0-JP-Extra)
  • Terms: CC BY-SA 4.0, inherited from the JVNV corpus the voices were trained on β€” the source repository states the corpus license carries over to the models. Redistribution here keeps it: credit the authors, name the source and the license URL, state that the weights were converted to another format and quantized to i8 (i4 for the group-quantized weights), license any derivative work under CC BY-SA 4.0 as well, and impose no further restrictions. There is no NonCommercial and no NoDerivatives clause β€” commercial use and modification are both allowed.
  • Changes made here: conversion into the Karume container format and quantization of the weights β€” the voices in f16 / i8 / mixed, the text encoder in i8 / mixed, where the mixed form stores the group-quantizable weights as i4 and everything else as i8 (the quant table below says which storage each quant selects). No retraining, no fine-tuning β€” the voices are the source checkpoints in a different storage form.
  • Text encoder: ku-nlp/deberta-v2-large-japanese-char-wwm, licensed cc-by-sa-4.0 (as of retrieval). It is redistributed here in the container format as the text_encoder component β€” the same license as the voices, so the share-alike term is consistent across everything in this repository.
  • Training data: the JVNV corpus, licensed CC BY-SA 4.0. Detai Xin, Junfeng Jiang, Shinnosuke Takamichi, Yuki Saito, Akiko Aizawa, Hiroshi Saruwatari, JVNV: A Corpus of Japanese Emotional Speech with Verbal Content and Nonverbal Expressions, arXiv:2310.06072. Corpus page: https://sites.google.com/site/shinnosuketakamichi/research-topics/jvnv_corpus
  • Training implementation: Style-Bert-VITS2 (AGPL-3.0). Karume's runtime contains none of that code: it is an independent implementation that reads these weights from its own container format, so the AGPL terms govern the training implementation, not the runtime that plays these files.

Models

Model Pipeline Quants Default quant
F1 (default) sbv2/1 f16+bert8 / i8 / i8-a8 / i8+bert4 / i4 i8+bert4
F2 sbv2/1 f16+bert8 / i8 / i8-a8 / i8+bert4 / i4 i8+bert4
M1 sbv2/1 f16+bert8 / i8 / i8-a8 / i8+bert4 / i4 i8+bert4
M2 sbv2/1 f16+bert8 / i8 / i8-a8 / i8+bert4 / i4 i8+bert4

model selects one of these; omitted, it is F1. quant defaults to that model's own default quant.

Usage 

import { analyzeWithWords } from "jsr:@hdae/yomi";
import { getDictionary } from "jsr:@hdae/yomi/loader";
import { encodeWav, Sbv2Pipeline, toSbv2Utterance } from "jsr:@karume/models";

await using pipeline = await Sbv2Pipeline.fromPretrained({
  repo: "hdae/karume-sbv2-jvnv",
  // Pin a commit for reproducible builds β€” without it you track `main`, and a future
  // repo update (renamed files, new manifest format) may break your app.
  // Copy the full hash from this repo's "Files and versions" tab:
  // revision: "<full commit sha>",
}, {
  // model: "F1", // default β€” available: F1 / F2 / M1 / M2
  // quant: "i8+bert4", // default β€” available: f16+bert8 / i4 / i8 / i8+bert4 / i8-a8
});

// Text analysis (readings and accents) is the caller's job:
const utterance = toSbv2Utterance(analyzeWithWords(await getDictionary(), "γ“γ‚“γ«γ‘γ―γ€γ“γ‚Œγ―γƒ†γ‚Ήγƒˆγ§γ™γ€‚"));

const audio = await pipeline.generate(utterance, {
  // Voice β€” the names below come from the tables further down; every model in
  // this repository brings its own set:
  // style: "Neutral", // default β€” available: Angry / Disgust / Fear / Happy / Neutral / Sad / Surprise
  // styleWeight: 1.0, // default β€” 0 = the average style, 1 = the named one
  // speaker: "jvnv-F1-jp", // default β€” available: jvnv-F1-jp

  // Delivery:
  // lengthScale: 1.0, // default β€” larger is slower
  // sdpRatio: 0.2, // default β€” 1 = stochastic duration, 0 = deterministic
  // noiseScale: 0.6, // default β€” sampling noise on z_p
  // noiseScaleW: 0.8, // default β€” sampling noise inside the stochastic predictor

  seed: 42, // same seed + same knobs β†’ same waveform
});
await Deno.writeFile("sbv2.wav", encodeWav(audio.data, audio.sampleRate));

generate() returns { sampleRate, data }, where data is an f32 mono waveform β€” exactly what encodeWav takes. Weights are fetched once and cached (verified against karume.json's size / sha256).

generate() takes an already-analyzed utterance: text analysis (readings and accents) belongs to the caller, and the package fetches no dictionary of its own. The snippet uses @hdae/yomi, whose dictionary (about 19 MB) is fetched on the first getDictionary() and read from the Cache API afterwards; to fix a reading, put yomi's overlay dictionary in front of the analysis. The SBV2 example is the worked version.

Model: F1

Quants

Quant What it is Download Weights Compute
f16+bert8 Highest fidelity (f16 synthesis) β€” front and voice in f16 storage, with the int8 text encoder β€” the largest download, and the closest match to the source checkpoint's audio. 414 MiB (295 MiB shared) text_encoder = i8 / front = f16 / voice = f16 β€”
i8 Half size (int8) β€” Every component stored as int8 and computed in f32 β€” roughly half the f16 download, with the execution path left unchanged. 356 MiB (295 MiB shared) text_encoder = i8 / front = i8 / voice = i8 β€”
i8-a8 int8 with int8 linear activations β€” The int8 weights with per-token int8 activations in the linear layers β€” faster on GPUs with dp4a, same download as the plain int8 seat. 356 MiB (295 MiB shared) text_encoder = i8 / front = i8 / voice = i8 linearCompute = a8
i8+bert4 (default) Balanced (int8 + int4 text encoder) β€” int8 synthesis with the text encoder in GPTQ-calibrated int4 (group-32): a markedly smaller download that stays audibly level with f32. 248 MiB (187 MiB shared) text_encoder = i4 / front = i8 / voice = i8 β€”
i4 Smallest (int4) β€” Text encoder, front and voice all in int4 group-32 β€” the smallest download and the fastest warm start, with slightly lower tension than f32. 226 MiB (187 MiB shared) text_encoder = i4 / front = i4 / voice = i4 β€”

If no quant is given, it runs as i8+bert4 (this model's recommended default). In a quant name, bert is the text_encoder component. Per-file size and sha256 live in karume.json β€” verify against that at the fetch layer. Dtype labels use the runtime's storage dtype vocabulary (f16 / i8 / i4 / i2), not the fp16 spelling common elsewhere in the ecosystem. Weights ship as Karume container files (.krm), split into numbered parts; a part is fetched and verified on its own. A path under shared/ is one this model shares byte for byte with another model in this repository (it is fetched and cached once).

How the stored weights were rounded:

  • f16+bert8 β€” front / voice in f16, each weight rounded to the nearest f16 value; text_encoder in i8, rounded per output channel (plain RTN).
  • i8 β€” every weight in i8, rounded per output channel (plain RTN).
  • i8-a8 β€” the same stored weights as i8. Nothing more is rounded at export time: the linear layers quantize their input activations to int8 per token at run time.
  • i8+bert4 (default) β€” text_encoder in i4 group-32: its linear layers rounded with GPTQ calibration (a 48-sentence Japanese corpus), its embedding table plain RTN. front / voice stay i8.
  • i4 β€” the same text encoder as i8+bert4, plus front / voice linear and conv1d weights in i4 group-32 with plain RTN. Smallest download and fastest warm synthesis; the output is very close to i8+bert4, with slightly lower tension than the source checkpoint's unquantized f32 reference.

Styles

Style ID
Neutral 0
Angry 1
Disgust 2
Fear 3
Happy 4
Sad 5
Surprise 6

style takes one of these names β€” the ID is the row it selects in F1/styles/style_vectors.safetensors. styleWeight blends between the average style (0) and the named one (1).

Speakers

Speaker ID
jvnv-F1-jp 0

speaker takes one of these names β€” the ID is the row it selects in F1/speakers/speaker_embeddings.safetensors.

Defaults

Any knob not passed to generate() is filled in from the manifest's defaults.

  • speaker: jvnv-F1-jp
  • style: Neutral
  • styleWeight: 1.0
  • sdpRatio: 0.2
  • noiseScale: 0.6
  • noiseScaleW: 0.8
  • lengthScale: 1.0

seed is the one knob the manifest does not carry β€” it defaults to 0, and the same seed with the same knobs gives the same waveform.

Model: F2

Quants

Quant What it is Download Weights Compute
f16+bert8 Highest fidelity (f16 synthesis) β€” front and voice in f16 storage, with the int8 text encoder β€” the largest download, and the closest match to the source checkpoint's audio. 414 MiB (295 MiB shared) text_encoder = i8 / front = f16 / voice = f16 β€”
i8 Half size (int8) β€” Every component stored as int8 and computed in f32 β€” roughly half the f16 download, with the execution path left unchanged. 356 MiB (295 MiB shared) text_encoder = i8 / front = i8 / voice = i8 β€”
i8-a8 int8 with int8 linear activations β€” The int8 weights with per-token int8 activations in the linear layers β€” faster on GPUs with dp4a, same download as the plain int8 seat. 356 MiB (295 MiB shared) text_encoder = i8 / front = i8 / voice = i8 linearCompute = a8
i8+bert4 (default) Balanced (int8 + int4 text encoder) β€” int8 synthesis with the text encoder in GPTQ-calibrated int4 (group-32): a markedly smaller download that stays audibly level with f32. 248 MiB (187 MiB shared) text_encoder = i4 / front = i8 / voice = i8 β€”
i4 Smallest (int4) β€” Text encoder, front and voice all in int4 group-32 β€” the smallest download and the fastest warm start, with slightly lower tension than f32. 226 MiB (187 MiB shared) text_encoder = i4 / front = i4 / voice = i4 β€”

If no quant is given, it runs as i8+bert4 (this model's recommended default). In a quant name, bert is the text_encoder component. Per-file size and sha256 live in karume.json β€” verify against that at the fetch layer. Dtype labels use the runtime's storage dtype vocabulary (f16 / i8 / i4 / i2), not the fp16 spelling common elsewhere in the ecosystem. Weights ship as Karume container files (.krm), split into numbered parts; a part is fetched and verified on its own. A path under shared/ is one this model shares byte for byte with another model in this repository (it is fetched and cached once).

How the stored weights were rounded:

  • f16+bert8 β€” front / voice in f16, each weight rounded to the nearest f16 value; text_encoder in i8, rounded per output channel (plain RTN).
  • i8 β€” every weight in i8, rounded per output channel (plain RTN).
  • i8-a8 β€” the same stored weights as i8. Nothing more is rounded at export time: the linear layers quantize their input activations to int8 per token at run time.
  • i8+bert4 (default) β€” text_encoder in i4 group-32: its linear layers rounded with GPTQ calibration (a 48-sentence Japanese corpus), its embedding table plain RTN. front / voice stay i8.
  • i4 β€” the same text encoder as i8+bert4, plus front / voice linear and conv1d weights in i4 group-32 with plain RTN. Smallest download and fastest warm synthesis; the output is very close to i8+bert4, with slightly lower tension than the source checkpoint's unquantized f32 reference.

Styles

Style ID
Neutral 0
Angry 1
Disgust 2
Fear 3
Happy 4
Sad 5
Surprise 6

style takes one of these names β€” the ID is the row it selects in F2/styles/style_vectors.safetensors. styleWeight blends between the average style (0) and the named one (1).

Speakers

Speaker ID
jvnv-F2-jp 0

speaker takes one of these names β€” the ID is the row it selects in F2/speakers/speaker_embeddings.safetensors.

Defaults

Any knob not passed to generate() is filled in from the manifest's defaults.

  • speaker: jvnv-F2-jp
  • style: Neutral
  • styleWeight: 1.0
  • sdpRatio: 0.2
  • noiseScale: 0.6
  • noiseScaleW: 0.8
  • lengthScale: 1.0

seed is the one knob the manifest does not carry β€” it defaults to 0, and the same seed with the same knobs gives the same waveform.

Model: M1

Quants

Quant What it is Download Weights Compute
f16+bert8 Highest fidelity (f16 synthesis) β€” front and voice in f16 storage, with the int8 text encoder β€” the largest download, and the closest match to the source checkpoint's audio. 414 MiB (295 MiB shared) text_encoder = i8 / front = f16 / voice = f16 β€”
i8 Half size (int8) β€” Every component stored as int8 and computed in f32 β€” roughly half the f16 download, with the execution path left unchanged. 356 MiB (295 MiB shared) text_encoder = i8 / front = i8 / voice = i8 β€”
i8-a8 int8 with int8 linear activations β€” The int8 weights with per-token int8 activations in the linear layers β€” faster on GPUs with dp4a, same download as the plain int8 seat. 356 MiB (295 MiB shared) text_encoder = i8 / front = i8 / voice = i8 linearCompute = a8
i8+bert4 (default) Balanced (int8 + int4 text encoder) β€” int8 synthesis with the text encoder in GPTQ-calibrated int4 (group-32): a markedly smaller download that stays audibly level with f32. 248 MiB (187 MiB shared) text_encoder = i4 / front = i8 / voice = i8 β€”
i4 Smallest (int4) β€” Text encoder, front and voice all in int4 group-32 β€” the smallest download and the fastest warm start, with slightly lower tension than f32. 226 MiB (187 MiB shared) text_encoder = i4 / front = i4 / voice = i4 β€”

If no quant is given, it runs as i8+bert4 (this model's recommended default). In a quant name, bert is the text_encoder component. Per-file size and sha256 live in karume.json β€” verify against that at the fetch layer. Dtype labels use the runtime's storage dtype vocabulary (f16 / i8 / i4 / i2), not the fp16 spelling common elsewhere in the ecosystem. Weights ship as Karume container files (.krm), split into numbered parts; a part is fetched and verified on its own. A path under shared/ is one this model shares byte for byte with another model in this repository (it is fetched and cached once).

How the stored weights were rounded:

  • f16+bert8 β€” front / voice in f16, each weight rounded to the nearest f16 value; text_encoder in i8, rounded per output channel (plain RTN).
  • i8 β€” every weight in i8, rounded per output channel (plain RTN).
  • i8-a8 β€” the same stored weights as i8. Nothing more is rounded at export time: the linear layers quantize their input activations to int8 per token at run time.
  • i8+bert4 (default) β€” text_encoder in i4 group-32: its linear layers rounded with GPTQ calibration (a 48-sentence Japanese corpus), its embedding table plain RTN. front / voice stay i8.
  • i4 β€” the same text encoder as i8+bert4, plus front / voice linear and conv1d weights in i4 group-32 with plain RTN. Smallest download and fastest warm synthesis; the output is very close to i8+bert4, with slightly lower tension than the source checkpoint's unquantized f32 reference.

Styles

Style ID
Neutral 0
Angry 1
Disgust 2
Fear 3
Happy 4
Sad 5
Surprise 6

style takes one of these names β€” the ID is the row it selects in M1/styles/style_vectors.safetensors. styleWeight blends between the average style (0) and the named one (1).

Speakers

Speaker ID
jvnv-M1-jp 0

speaker takes one of these names β€” the ID is the row it selects in M1/speakers/speaker_embeddings.safetensors.

Defaults

Any knob not passed to generate() is filled in from the manifest's defaults.

  • speaker: jvnv-M1-jp
  • style: Neutral
  • styleWeight: 1.0
  • sdpRatio: 0.2
  • noiseScale: 0.6
  • noiseScaleW: 0.8
  • lengthScale: 1.0

seed is the one knob the manifest does not carry β€” it defaults to 0, and the same seed with the same knobs gives the same waveform.

Model: M2

Quants

Quant What it is Download Weights Compute
f16+bert8 Highest fidelity (f16 synthesis) β€” front and voice in f16 storage, with the int8 text encoder β€” the largest download, and the closest match to the source checkpoint's audio. 414 MiB (295 MiB shared) text_encoder = i8 / front = f16 / voice = f16 β€”
i8 Half size (int8) β€” Every component stored as int8 and computed in f32 β€” roughly half the f16 download, with the execution path left unchanged. 356 MiB (295 MiB shared) text_encoder = i8 / front = i8 / voice = i8 β€”
i8-a8 int8 with int8 linear activations β€” The int8 weights with per-token int8 activations in the linear layers β€” faster on GPUs with dp4a, same download as the plain int8 seat. 356 MiB (295 MiB shared) text_encoder = i8 / front = i8 / voice = i8 linearCompute = a8
i8+bert4 (default) Balanced (int8 + int4 text encoder) β€” int8 synthesis with the text encoder in GPTQ-calibrated int4 (group-32): a markedly smaller download that stays audibly level with f32. 248 MiB (187 MiB shared) text_encoder = i4 / front = i8 / voice = i8 β€”
i4 Smallest (int4) β€” Text encoder, front and voice all in int4 group-32 β€” the smallest download and the fastest warm start, with slightly lower tension than f32. 226 MiB (187 MiB shared) text_encoder = i4 / front = i4 / voice = i4 β€”

If no quant is given, it runs as i8+bert4 (this model's recommended default). In a quant name, bert is the text_encoder component. Per-file size and sha256 live in karume.json β€” verify against that at the fetch layer. Dtype labels use the runtime's storage dtype vocabulary (f16 / i8 / i4 / i2), not the fp16 spelling common elsewhere in the ecosystem. Weights ship as Karume container files (.krm), split into numbered parts; a part is fetched and verified on its own. A path under shared/ is one this model shares byte for byte with another model in this repository (it is fetched and cached once).

How the stored weights were rounded:

  • f16+bert8 β€” front / voice in f16, each weight rounded to the nearest f16 value; text_encoder in i8, rounded per output channel (plain RTN).
  • i8 β€” every weight in i8, rounded per output channel (plain RTN).
  • i8-a8 β€” the same stored weights as i8. Nothing more is rounded at export time: the linear layers quantize their input activations to int8 per token at run time.
  • i8+bert4 (default) β€” text_encoder in i4 group-32: its linear layers rounded with GPTQ calibration (a 48-sentence Japanese corpus), its embedding table plain RTN. front / voice stay i8.
  • i4 β€” the same text encoder as i8+bert4, plus front / voice linear and conv1d weights in i4 group-32 with plain RTN. Smallest download and fastest warm synthesis; the output is very close to i8+bert4, with slightly lower tension than the source checkpoint's unquantized f32 reference.

Styles

Style ID
Neutral 0
Angry 1
Disgust 2
Fear 3
Happy 4
Sad 5
Surprise 6

style takes one of these names β€” the ID is the row it selects in M2/styles/style_vectors.safetensors. styleWeight blends between the average style (0) and the named one (1).

Speakers

Speaker ID
jvnv-M2-jp 0

speaker takes one of these names β€” the ID is the row it selects in M2/speakers/speaker_embeddings.safetensors.

Defaults

Any knob not passed to generate() is filled in from the manifest's defaults.

  • speaker: jvnv-M2-jp
  • style: Neutral
  • styleWeight: 1.0
  • sdpRatio: 0.2
  • noiseScale: 0.6
  • noiseScaleW: 0.8
  • lengthScale: 1.0

seed is the one knob the manifest does not carry β€” it defaults to 0, and the same seed with the same knobs gives the same waveform.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
Text-to-Speech
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for hdae/karume-sbv2-jvnv

Base model

ku-nlp/deberta-v2-large-japanese-char-wwm
Quantized
(4)
this model

Paper for hdae/karume-sbv2-jvnv