Hugging Face's logo

Transformers documentation

Gradium Phonon TTS

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.20.2).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

This model was contributed to Hugging Face Transformers on 2026-10-20.

Gradium Phonon TTS

Overview

Gradium Phonon TTS is Gradium’s family of on-device text-to-speech models, trained with the recipe of Pocket TTS: a causal transformer reads a voice prompt and the text, then emits one continuous 32-dimensional audio latent per step at 12.5 Hz through a one-step flow head, and an end-of-speech head decides when the utterance is over. The latents are decoded to 24 kHz audio by a Pocket TTS audio tokenizer held by the processor, Pocket TTS’s PocketTTSProcessor.

It differs from Pocket TTS in its voice conditioning: the voice prompt is made of 512-dimensional latents of a separate speaker codec, and a learnt speaker vector is added to every audio latent position. The speaker codec is not released, so voices come as precomputed latents (each checkpoint’s default-voice.safetensors, and the catalog of gradium/phonon-voices) and the audio tokenizer is a decoder only: voices cannot be cloned from a recording.

CheckpointParametersLanguagesLicense
gradium/phonon-31m-en31MEnglishApache 2.0
gradium/phonon-40m-en40MEnglishCC BY-NC 4.0
gradium/phonon-40m-fr40MFrenchCC BY-NC 4.0
gradium/phonon-40m-de40MGermanCC BY-NC 4.0
gradium/phonon-40m-es40MSpanishCC BY-NC 4.0
gradium/phonon-40m-pt40MPortugueseCC BY-NC 4.0
gradium/phonon-40m-multilingual40MEnglish, French, German, Spanish, PortugueseCC BY-NC 4.0

They all load the same way: switching checkpoints means changing the repo id.

This model was contributed by Eustache Le Bihan. The original code is gradium-ai/xn-ptts.

Usage tips

  • Prepare inputs with apply_chat_template(): one user message per text, an empty {"type": "audio"} item where the voice goes, the voice latents in processor_kwargs={"audio_prompt_latents": [...]}, and add_generation_prompt=True.
  • The tokenizer does not spell out numbers, amounts, symbols or addresses, see Text normalization.
  • generate returns audio_latents and audio_latents_mask; decode them with PocketTTSProcessor.decode().
  • Generation samples as Gradium’s runtime does: temperature 0.3 (the variance of the flow’s starting noise), and it stops shortly after the end-of-speech logit crosses eos_threshold. do_sample=False starts the flow from zero noise.
  • The model loads in bf16, the dtype its weights are stored in; pass dtype=torch.float32 on CPU to match Gradium’s runtime there.
  • Split long text with PocketTTSProcessor.split_text(), one conversation per chunk.

Usage

Voice

A voice is a tensor of latents (num_frames, 512). Each checkpoint ships one (default-voice.safetensors), and gradium/phonon-voices has about a hundred more, in every language, each under its voice ID (voices.jsonl lists their names, languages and descriptions). Any voice works with any checkpoint.

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoProcessor, GradiumPhononTTSForConditionalGeneration

model_id = "gradium/phonon-31m-en"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = GradiumPhononTTSForConditionalGeneration.from_pretrained(model_id, device_map="auto")


def load_voice(voice_id):
    """A voice of gradium/phonon-voices, as latents of shape (num_frames, 512)."""
    path = hf_hub_download("gradium/phonon-voices", f"voices/{voice_id}.safetensors")
    return load_file(path)["speaker_wavs"][0].T.float()


# Easton (en-us); the checkpoint's own voice: hf_hub_download(model_id, "default-voice.safetensors")
voice = load_voice("LMDIYE0Oa377Qh6Z")
# an empty `audio` item marks the voice
conversation = [
    {"role": "user", "content": [{"type": "audio"}, {"type": "text", "text": "Hello from Phonon."}]}
]
inputs = processor.apply_chat_template(
    conversation,
    tokenize=True,
    return_dict=True,
    add_generation_prompt=True,
    processor_kwargs={"audio_prompt_latents": [voice]},
).to(model.device)

outputs = model.generate(**inputs)
audio = processor.decode(outputs.audio_latents, outputs.audio_latents_mask)
processor.save_audio(audio, "phonon.wav")

Batches and long text

Batch several texts, each with its own voice, with long text split into chunks:

text = (
    "Phonon is a small text-to-speech model that runs on your device. "
    "Long texts are split into sentences, and each chunk is one conversation of the batch."
)
chunks = processor.split_text(text)
texts = chunks + ["And this one is said by another voice."]
voices = [load_voice("LMDIYE0Oa377Qh6Z")] * len(chunks) + [load_voice("NbpkqMVS3CJeq2j8")]
conversations = [
    [{"role": "user", "content": [{"type": "audio"}, {"type": "text", "text": text}]}] for text in texts
]
inputs = processor.apply_chat_template(
    conversations,
    tokenize=True,
    return_dict=True,
    add_generation_prompt=True,
    processor_kwargs={"audio_prompt_latents": voices},
).to(model.device)

outputs = model.generate(**inputs)
audio = processor.decode(outputs.audio_latents, outputs.audio_latents_mask)
processor.save_audio([torch.cat(audio[: len(chunks)]), audio[-1]], ["long.wav", "other.wav"])

Text normalization

The tokenizer normalizes what does not depend on the language, as Gradium’s runtime does:

TextTokenizer
“Hello” — it’s here…"Hello" it's here.
hello (world); see you 🚀Hello, world, see you.

It does not spell out numbers, amounts, symbols or addresses, which depend on the language:

TextTokenizerGradium’s runtime
It costs $1,234.56.It costs $1,234.56.It costs 1 thousand 234 dollars and 56 cents.
Write to jean@gradium.aiWrite to jean@gradium.ai.Write to jean at gradium dot A-I.
It is 21°CIt is 21°C.It is 21 degrees Celsius.

For those, use the text normalization of Gradium’s runtime, ptts, or write them out.

GradiumPhononTTSConfig

class transformers.GradiumPhononTTSConfig

< >

( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Noneflow_head_config: dict | transformers.configuration_utils.PreTrainedConfig | None = Nonevocab_size: int = 4001hidden_size: int = 768num_hidden_layers: int = 6num_attention_heads: int = 12num_key_value_heads: int | None = Nonehead_dim: int | None = Noneintermediate_size: int = 3072hidden_act: str = 'gelu'layer_norm_eps: float = 1e-05attention_bias: bool = Falseattention_dropout: float = 0.0max_position_embeddings: int = 8192rope_parameters: transformers.modeling_rope_utils.RopeParameters | dict | None = Nonelatent_dim: int = 32pad_token_id: int = 3bos_token_id: int | None = Noneeos_token_id: int | None = Noneaudio_prompt_token_id: int = 5audio_latent_token_id: int = 6audio_bos_token_id: int = 4frame_rate: float = 12.5initializer_range: float = 0.02use_cache: bool = Trueeos_loss_weight: float = 0.1flow_loss_num_samples: int = 4lsd_distill_probability: float = 0.25lsd_diagonal_weight: float = 0.75lsd_time_logit_mean: float = 0.4lsd_time_logit_std: float = 1.0latent_stats_ema_steps: int = 1000latent_stats_ema_decay: float = 0.999audio_prompt_dim: int = 512num_speakers: int = 1num_speakers_vocab_size: int = 32num_speakers_embedding_dim: int = 16duration_delta: float = 0.0duration_delta_embedding_dim: int = 128duration_delta_scale: float = 1000.0duration_delta_max_period: float = 10000.0 )

Parameters

  • flow_head_config (Union[GradiumPhononTTSFlowHeadConfig, dict], optional) — The config of the flow head that samples each latent from the backbone hidden state.
  • vocab_size (int, optional, defaults to 4001) — Vocabulary size of the model. Defines the number of different tokens that can be represented by the input_ids.
  • hidden_size (int, optional, defaults to 768) — Dimension of the hidden representations.
  • num_hidden_layers (int, optional, defaults to 6) — Number of hidden layers in the Transformer decoder.
  • num_attention_heads (int, optional, defaults to 12) — Number of attention heads for each attention layer in the Transformer decoder.
  • num_key_value_heads (int, optional) — This is the number of key_value heads that should be used to implement Grouped Query Attention. If num_key_value_heads=num_attention_heads, the model will use Multi Head Attention (MHA), if num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out this paper. If it is not specified, will default to num_attention_heads.
  • head_dim (int, optional) — The attention head dimension. If None, it will default to hidden_size // num_attention_heads
  • intermediate_size (int, optional, defaults to 3072) — Dimension of the MLP representations.
  • hidden_act (str, optional, defaults to gelu) — The non-linear activation function (function or string) in the decoder. For example, "gelu", "relu", "silu", etc.
  • layer_norm_eps (float, optional, defaults to 1e-05) — Epsilon of the layer normalizations of the backbone.
  • attention_bias (bool, optional, defaults to False) — Whether to use a bias in the query, key, value and output projection layers during self-attention.
  • attention_dropout (float, optional, defaults to 0.0) — The dropout ratio for the attention probabilities.
  • max_position_embeddings (int, optional, defaults to 8192) — The maximum sequence length that this model might ever be used with.
  • rope_parameters (Union[~modeling_rope_utils.RopeParameters, dict], optional) — Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for rope_theta and optionally parameters used for scaling in case you want to use RoPE with longer max_position_embeddings.
  • latent_dim (int, optional, defaults to 32) — Dimension of the audio latents the backbone takes as input (audio_latents); equals the flow head’s and the audio tokenizer’s latent_dim.
  • pad_token_id (int, optional, defaults to 3) — Token id used for padding in the vocabulary.
  • bos_token_id (int, optional) — Token id used for beginning-of-stream in the vocabulary.
  • eos_token_id (int, optional) — Token id used for end-of-stream in the vocabulary.
  • audio_prompt_token_id (int, optional, defaults to 5) — Placeholder token id of the voice-prompt frames in input_ids (<|audio_prompt|>).
  • audio_latent_token_id (int, optional, defaults to 6) — Placeholder token id of the audio latent positions in input_ids (<|audio_latent|>).
  • audio_bos_token_id (int, optional, defaults to 4) — Token id of the first audio position, closing the prompt: it holds the learnt BOS latent, from which the first frame is generated (<|audio_bos|>).
  • frame_rate (float, optional, defaults to 12.5) — Number of audio latents per second of audio: generate derives its default frame budget from it.
  • initializer_range (float, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
  • use_cache (bool, optional, defaults to True) — Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if config.is_decoder=True or when the model is a decoder-only generative model.
  • eos_loss_weight (float, optional, defaults to 0.1) — Weight of the end-of-speech loss in the training loss.
  • flow_loss_num_samples (int, optional, defaults to 4) — Number of noise draws per target frame in the flow loss.
  • lsd_distill_probability (float, optional, defaults to 0.25) — Probability, per training step, of computing the self-distillation term of the LSD loss (reweighted to keep the expected loss); 1 computes it at every step.
  • lsd_diagonal_weight (float, optional, defaults to 0.75) — Weight of the diagonal (instantaneous flow) term of the LSD loss; the self-distillation term weighs the rest.
  • lsd_time_logit_mean (float, optional, defaults to 0.4) — Mean of the logit-normal distribution of the flow times drawn by the LSD loss.
  • lsd_time_logit_std (float, optional, defaults to 1.0) — Standard deviation of the logit-normal distribution of the flow times drawn by the LSD loss.
  • latent_stats_ema_steps (int, optional, defaults to 1000) — Number of first training steps during which latent_mean and latent_std follow an exponential moving average of the target latents; 0 keeps them frozen.
  • latent_stats_ema_decay (float, optional, defaults to 0.999) — Decay of that moving average.
  • audio_prompt_dim (int, optional, defaults to 512) — Dimension of the voice-prompt latents (audio_prompt_latents), the unquantized latents of Gradium’s speaker codec (a Mimi encoder, not the audio tokenizer’s), which the backbone projects to hidden_size.
  • num_speakers (int, optional, defaults to 1) — Value of the num_speakers condition: the row of its lookup table added to every audio latent position.
  • num_speakers_vocab_size (int, optional, defaults to 32) — Number of rows of the num_speakers lookup table.
  • num_speakers_embedding_dim (int, optional, defaults to 16) — Width of the num_speakers lookup table, projected to hidden_size.
  • duration_delta (float, optional, defaults to 0.0) — Value of the duration_delta condition, embedded sinusoidally and added to every audio latent position.
  • duration_delta_embedding_dim (int, optional, defaults to 128) — Size of the sinusoidal embedding of duration_delta, projected to hidden_size.
  • duration_delta_scale (float, optional, defaults to 1000.0) — Factor applied to duration_delta before its sinusoidal embedding.
  • duration_delta_max_period (float, optional, defaults to 10000.0) — Maximum period of the sinusoidal embedding of duration_delta.

This is the configuration class to store the configuration of a GradiumPhononTTSModel. It is used to instantiate a Gradium Phonon Tts model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the eustlb/gradium-phonon-tts

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

GradiumPhononTTSFlowHeadConfig

class transformers.GradiumPhononTTSFlowHeadConfig

< >

( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Nonehidden_size: int = 512num_hidden_layers: int = 6latent_dim: int = 32condition_dim: int = 768frequency_embedding_size: int = 256hidden_act: str = 'silu'layer_norm_eps: float = 1e-06rms_norm_eps: float = 1e-05max_period: float = 10000.0flow_type: typing.Literal['lsd', 'drifting'] = 'lsd'loss_weighting_hidden_size: int = 32loss_weighting_num_layers: int = 3 )

Parameters

  • hidden_size (int, optional, defaults to 512) — Dimension of the hidden representations.
  • num_hidden_layers (int, optional, defaults to 6) — Number of hidden layers in the Transformer decoder.
  • latent_dim (int, optional, defaults to 32) — Dimension of the continuous audio latents the head predicts (the latent dimension of the audio tokenizer).
  • condition_dim (int, optional, defaults to 768) — Dimension of the backbone hidden state the head is conditioned on (the backbone hidden_size).
  • frequency_embedding_size (int, optional, defaults to 256) — Size of the sinusoidal embedding of the flow times.
  • hidden_act (str, optional, defaults to silu) — The non-linear activation function (function or string) in the decoder. For example, "gelu", "relu", "silu", etc.
  • layer_norm_eps (float, optional, defaults to 1e-06) — Epsilon of the layer normalizations inside the residual blocks and the final layer.
  • rms_norm_eps (float, optional, defaults to 1e-05) — Epsilon of the normalization closing the time embeddings.
  • max_period (float, optional, defaults to 10000.0) — Maximum period of the sinusoidal time embedding, which the checkpoint stores as freqs.
  • flow_type (str, optional, defaults to "lsd") — Objective the head was trained with, which sets how it is conditioned and sampled. "lsd" (Lagrangian self-distillation): conditioned on the start and end times of a flow step, sampled by Euler steps from Gaussian noise. "drifting": no time conditioning, the head maps noise to the latent in one call.
  • loss_weighting_hidden_size (int, optional, defaults to 32) — Width of the MLP that learns the uncertainty weighting of the LSD loss terms from their flow times (training only).
  • loss_weighting_num_layers (int, optional, defaults to 3) — Number of hidden layers of that MLP.

This is the configuration class to store the configuration of a GradiumPhononTTSModel. It is used to instantiate a Gradium Phonon Tts model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the eustlb/gradium-phonon-tts

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

GradiumPhononTTSModel

class transformers.GradiumPhononTTSModel

< >

( config: GradiumPhononTTSConfig )

Parameters

  • config (GradiumPhononTTSConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The Gradium Phonon TTS backbone: a causal transformer over one stream made of the voice prompt (speaker-codec latents), the text tokens and the audio latents, returning the normalised hidden states.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< >

( input_ids: typing.Optional[torch.LongTensor] = Noneaudio_prompt_latents: typing.Optional[torch.FloatTensor] = Noneaudio_prompt_latents_mask: typing.Optional[torch.Tensor] = Noneaudio_latents: typing.Optional[torch.FloatTensor] = Noneaudio_latents_mask: typing.Optional[torch.Tensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneposition_ids: typing.Optional[torch.LongTensor] = Nonepast_key_values: transformers.cache_utils.Cache | None = Noneinputs_embeds: typing.Optional[torch.FloatTensor] = Noneuse_cache: bool | None = None**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → BaseModelOutputWithPast or tuple(torch.FloatTensor)

Parameters

  • input_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

    Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

    What are input IDs?

  • audio_prompt_latents (torch.FloatTensor of shape (batch_size, num_prompt_frames, latent_dim), optional) — Voice-prompt latents of the audio tokenizer (encoded by the processor, or a checkpoint’s voices/), right-padded to the longest voice of the batch, scattered into the audio_prompt_token_id positions.
  • audio_prompt_latents_mask (torch.Tensor of shape (batch_size, num_prompt_frames), optional) — Mask of the voice frames (1) as opposed to padding (0) in audio_prompt_latents.
  • audio_latents (torch.FloatTensor of shape (batch_size, num_latents, latent_dim), optional) — Normalised audio latents, scattered into the audio_latent_token_id positions. The audio_bos_token_id position before them, which closes the prompt and from which the first frame is generated, always gets the learnt BOS latent.
  • audio_latents_mask (torch.Tensor of shape (batch_size, num_latents), optional) — Mask of each row’s latents (1) as opposed to padding (0) in audio_latents, when the rows hold different numbers of latent positions (training).
  • attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:

    • 1 for tokens that are not masked,
    • 0 for tokens that are masked.

    What are attention masks?

  • position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.n_positions - 1].

    What are position IDs?

  • past_key_values (~cache_utils.Cache, optional) — Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the past_key_values returned by the model at a previous stage of decoding, when use_cache=True or config.use_cache=True.

    Only Cache instance is allowed as input, see our kv cache guide. If no past_key_values are passed, DynamicCache will be initialized by default.

    The model will output the same cache format that is fed as input.

    If past_key_values are used, the user is expected to input only unprocessed input_ids (those that don’t have their past key value states given to this model) of shape (batch_size, unprocessed_length) instead of all input_ids of shape (batch_size, sequence_length).

  • inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
  • use_cache (bool, optional) — If set to True, past_key_values key value states are returned and can be used to speed up decoding (see past_key_values).

Returns

BaseModelOutputWithPast or tuple(torch.FloatTensor)

A BaseModelOutputWithPast or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (GradiumPhononTTSConfig) and inputs.

The GradiumPhononTTSModel forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

  • last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.

    If past_key_values is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.

  • past_key_values (Cache, optional, returned when use_cache=True is passed or when config.use_cache=True) — It is a Cache instance. For more details, see our kv cache guide.

    Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if config.is_encoder_decoder=True in the cross-attention blocks) that can be used (see past_key_values input) to speed up sequential decoding.

  • hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

    Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

  • attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

    Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

GradiumPhononTTSForConditionalGeneration

class transformers.GradiumPhononTTSForConditionalGeneration

< >

( config: GradiumPhononTTSConfig )

Parameters

  • config (GradiumPhononTTSConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The Gradium Phonon TTS text-to-speech model: the backbone, the flow head that samples one audio latent per step and the end-of-speech head. It generates audio tokenizer latents, which the processor decodes into 24 kHz audio.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< >

( input_ids: typing.Optional[torch.LongTensor] = Noneaudio_prompt_latents: typing.Optional[torch.FloatTensor] = Noneaudio_prompt_latents_mask: typing.Optional[torch.Tensor] = Noneaudio_latents: typing.Optional[torch.FloatTensor] = Nonelabels: typing.Optional[torch.FloatTensor] = Nonelabels_mask: typing.Optional[torch.Tensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneposition_ids: typing.Optional[torch.LongTensor] = Nonepast_key_values: transformers.cache_utils.Cache | None = Noneinputs_embeds: typing.Optional[torch.FloatTensor] = Noneuse_cache: bool | None = None**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → GradiumPhononTTSOutputWithPast or tuple(torch.FloatTensor)

Parameters

  • input_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

    Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

    What are input IDs?

  • audio_prompt_latents (torch.FloatTensor of shape (batch_size, num_prompt_frames, latent_dim), optional) — Voice-prompt latents of the audio tokenizer, right-padded, scattered into the audio_prompt_token_id positions.
  • audio_prompt_latents_mask (torch.Tensor of shape (batch_size, num_prompt_frames), optional) — Mask of the voice frames (1) as opposed to padding (0) in audio_prompt_latents.
  • audio_latents (torch.FloatTensor of shape (batch_size, num_latents, latent_dim), optional) — Normalised audio latents, scattered into the audio_latent_token_id positions.
  • labels (torch.FloatTensor of shape (batch_size, num_frames, latent_dim), optional) — Target latents of the audio tokenizer (training), right-padded. Each row’s prompt ends with the audio_bos_token_id position followed by one audio_latent_token_id position per frame: the model feeds the normalised targets (teacher forcing), applies the flow loss to the outputs of the BOS and all but the last frame, which predict the frames, and the end-of-speech loss to all of them, the last one being the end of speech.
  • labels_mask (torch.Tensor of shape (batch_size, num_frames), optional) — Mask of each row’s target frames (1) as opposed to padding (0) in labels.
  • attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:

    • 1 for tokens that are not masked,
    • 0 for tokens that are masked.

    What are attention masks?

  • position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.n_positions - 1].

    What are position IDs?

  • past_key_values (~cache_utils.Cache, optional) — Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the past_key_values returned by the model at a previous stage of decoding, when use_cache=True or config.use_cache=True.

    Only Cache instance is allowed as input, see our kv cache guide. If no past_key_values are passed, DynamicCache will be initialized by default.

    The model will output the same cache format that is fed as input.

    If past_key_values are used, the user is expected to input only unprocessed input_ids (those that don’t have their past key value states given to this model) of shape (batch_size, unprocessed_length) instead of all input_ids of shape (batch_size, sequence_length).

  • inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
  • use_cache (bool, optional) — If set to True, past_key_values key value states are returned and can be used to speed up decoding (see past_key_values).

Returns

GradiumPhononTTSOutputWithPast or tuple(torch.FloatTensor)

A GradiumPhononTTSOutputWithPast or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (GradiumPhononTTSConfig) and inputs.

The GradiumPhononTTSForConditionalGeneration forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

  • loss (torch.FloatTensor of shape (), optional, returned when labels is passed) — Training loss: the LSD flow loss of the target latents plus config.eos_loss_weight times the end-of-speech loss.

  • logits (torch.FloatTensor of shape (batch_size, sequence_length, 1)) — End-of-speech logits, one per position.

  • past_key_values (Cache, optional, returned when use_cache=True is passed or when config.use_cache=True) — It is a Cache instance. For more details, see our kv cache guide.

    Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see past_key_values input) to speed up sequential decoding.

  • hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

    Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

  • attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

    Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

  • last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Normalised hidden states of the backbone, which condition the flow head.

Example:

>>> from transformers import AutoProcessor, GradiumPhononTTSForConditionalGeneration

>>> model = GradiumPhononTTSForConditionalGeneration.from_pretrained("eustlb/gradium-phonon-tts")

>>> processor = AutoProcessor.from_pretrained("eustlb/gradium-phonon-tts")
>>> inputs = processor(text="Hello, my dog is cute", return_tensors="pt")

>>> # generate speech
>>> speech = model(inputs["input_ids"])

generate

< >

( inputs: typing.Optional[torch.Tensor] = None**kwargs )

Parameters

  • inputs (torch.LongTensor of shape (batch_size, sequence_length), optional) — The input_ids prepared by PocketTTSProcessor. Can also be passed as input_ids=.
  • kwargs (dict[str, Any], optional) — attention_mask, the voice prompt (audio_prompt_latents and audio_prompt_latents_mask, or the cache of a voice prefix as past_key_values, the shipped voices, with prompts of the text only), frames_after_eos from the processor, the generation parameters (do_sample, temperature, max_new_tokens, …), and the Pocket TTS-specific num_flow_steps, eos_threshold and min_frames_before_eos, which default to the generation config.

Generates speech. This method overrides generate() to default max_new_tokens to the original runtime’s frame budget, ceil((num_text_tokens / 3 + 2) * frame_rate) for the longest text of the batch. Everything else goes through the regular generation hooks.

Update on GitHub

←GLM-ASR GraniteSpeech→