Liquid-Codec-ONNX

ONNX graphs of LiquidAI/Liquid-Codec, the compact 24 kHz speech codec behind LFM3-Audio and LFM3-Omni, for ONNX Runtime on CPU or GPU, without PyTorch or transformers:

• encoder.onnx: 24 kHz speech -> 12 codes per 80 ms frame (12.5 Hz)
• decoder.onnx: codes -> 24 kHz speech
• decoder_step.onnx: the streaming decoder, one chunk of frames per call, with its state passed in and out as tensors
• decoder_fp16.onnx, decoder_step_fp16.onnx: FP16 versions of both decoders, half the size

The FP32 graphs reproduce the original model's encode and decode to within float32 rounding, and the FP16 decoders stay within 0.02 dB log-mel of the original. All graphs run faster than the transformers implementation on both CPU and GPU (see Benchmark).

Files

File	Size	Input	Output
encoder.onnx	148 MiB	input_values float32 [batch, num_samples], 24 kHz mono	audio_codes int64 [batch, 12, num_frames]
decoder.onnx	145 MiB	audio_codes int64 [batch, 12, num_frames]	audio_values float32 [batch, 1920 * num_frames]
decoder_step.onnx	145 MiB	audio_codes int64 [batch, 12, k] + state	audio_values float32 [batch, 1920 * k], final_values [batch, 480] + next state
decoder_fp16.onnx	76 MiB	same as decoder.onnx	same as decoder.onnx
decoder_step_fp16.onnx	76 MiB	same as decoder_step.onnx	same as decoder_step.onnx
example.py		runnable demo: offline and streaming (--fp16 for the FP16 decoders)

All graphs use ONNX opset 18 (ONNX Runtime 1.14 or newer; tested with 1.31.1). Batch and length axes are dynamic. Each graph's ONNX metadata records its source model, revision, license and a description of the changes.

Usage

pip install onnxruntime numpy soundfile huggingface_hub

Encode and decode

import numpy as np
import onnxruntime as ort
import soundfile as sf
from huggingface_hub import snapshot_download

path = snapshot_download("Aalto-Speech-Synthesis/Liquid-Codec-ONNX")
encoder = ort.InferenceSession(f"{path}/encoder.onnx", providers=["CPUExecutionProvider"])
decoder = ort.InferenceSession(f"{path}/decoder.onnx", providers=["CPUExecutionProvider"])

audio, rate = sf.read("speech.wav", dtype="float32")    # mono, 24 kHz (resample first if not)
wave = audio[None]                                      # [1, num_samples]

codes = encoder.run(None, {"input_values": wave})[0]    # int64 [1, 12, ceil(num_samples / 1920)]
decoded = decoder.run(None, {"audio_codes": codes})[0]  # float32 [1, 1920 * num_frames]
decoded = decoded[:, : wave.shape[1]]                   # trim the tail padding
sf.write("reconstruction.wav", decoded[0], 24000)

Streaming decode

The streaming decoder keeps no state inside the graph. Every call takes a chunk of codes plus the current state, and returns the new audio plus the next state. A new stream starts from all-zero state:

step = ort.InferenceSession(f"{path}/decoder_step.onnx", providers=["CPUExecutionProvider"])

# All-zero state, with shapes read from the graph (batch of 1).
state = {
    item.name: np.zeros([1 if isinstance(d, str) else d for d in item.shape],
                        dtype=np.int64 if item.type == "tensor(int64)" else np.float32)
    for item in step.get_inputs()[1:]
}
names = list(state)

pieces = []
for start in range(codes.shape[-1]):                    # one 80 ms frame per step
    chunk = np.ascontiguousarray(codes[..., start:start + 1])
    audio, final, *next_state = step.run(None, {"audio_codes": chunk, **state})
    state = dict(zip(names, next_state))
    pieces.append(audio)                                # 1920 new samples, ready to play

stream = np.concatenate(pieces + [final], axis=1)[:, 480:]
# stream == decoder.run(None, {"audio_codes": codes})[0], up to float rounding

Two details make the stitched stream equal to offline decoding:

• Pre-roll: the first 480 samples (20 ms) of a stream are zeros, the algorithmic delay that offline decoding trims. Drop them, or simply play them as silence.
• End of stream: every step also returns final_values, the 480 samples that end the stream if this chunk is the last one. Append them after the last chunk.

A chunk may hold any number of frames, from step to step. Streams in one batch share the chunk sizes and the position counter.

FP16 decoders

decoder_fp16.onnx and decoder_step_fp16.onnx are drop-in replacements for decoder.onnx and decoder_step.onnx. They have the same inputs and outputs, including FP32 audio and FP32 stream state, so the code above works unchanged with only the file name swapped:

decoder = ort.InferenceSession(f"{path}/decoder_fp16.onnx", providers=["CPUExecutionProvider"])
step = ort.InferenceSession(f"{path}/decoder_step_fp16.onnx", providers=["CPUExecutionProvider"])

What runs in FP16, and what doesn't:

• FP16: the learned layers, meaning the dequantizer projection, the LFM3 layers and the output projection.
• FP32: the inverse STFT and overlap-add, as in the original model.
• Encoder: FP32 only, because its quantized codes are sensitive to rounding.

GPU

pip install "onnxruntime-gpu"

providers = [("CUDAExecutionProvider", {"use_tf32": "0"}), "CPUExecutionProvider"]
encoder = ort.InferenceSession(f"{path}/encoder.onnx", providers=providers)

ONNX Runtime's CUDA provider uses TF32 for FP32 matrix products by default. use_tf32=0 keeps full FP32 accuracy; the benchmark below uses it.

example.py puts these pieces together (resampling with soxr, offline and streaming decoding, CPU or --cuda):

python example.py speech.wav

Interface details

Encoder. Equals LiquidCodecModel.encode(input_values).audio_codes. It accepts any length of 1 sample or more and zero-pads the tail to a whole frame of 1920 samples, so num_frames = ceil(num_samples / 1920). Codes are in [0, 2016).

Decoder. Equals LiquidCodecModel.decode(audio_codes).audio_values without a streaming cache. It returns 1920 * num_frames samples; trim them to the input length, as LiquidCodecModel.forward does. Unlike transformers, the graph does not check that codes lie in [0, 2016), and out-of-range codes give undefined output.

Streaming decoder. Equals LiquidCodecModel.decode(..., streaming_cache, finalize), with the cache turned into tensors:

Input	Shape	Meaning	Output
audio_codes int64	[batch, 12, k]	the next k >= 1 frames	audio_values float32 [batch, 1920 k]
			final_values float32 [batch, 480]
position int64	[1]	frames decoded so far	next_position
key_cache float32	[3, batch, 2, 29, 64]	keys of the last 29 STFT frames, per attention layer	next_key_cache
value_cache float32	[3, batch, 2, 29, 64]	values of the same frames	next_value_cache
conv_state float32	[5, batch, 512, 6]	last 6 inputs of each short-convolution layer	next_conv_state
overlap_state float32	[batch, 960]	overlap-add samples that later frames still add to	next_overlap_state

The state never grows: about 150 KB per stream, however long the stream runs.

Benchmark against transformers

Measured on an Intel Core i5-14400 (16 threads) and an NVIDIA GeForce RTX 4070 SUPER, with torch 2.15.0+cu134, transformers 5.22.2 and onnxruntime-gpu 1.31.1.

The reference is the transformers implementation in FP32 on CPU. Every row is checked against it on every clip:

• Codes: at least 95% of the encoder's codes must equal the reference's.
• Log-mel distance: the decoded audio's 80-bin log-mel spectrogram must stay within 0.5 dB mean absolute difference and 2 dB at the 95th percentile. This is checked for the decoder on the reference codes, and for the round trip on the implementation's own codes.

TF32 is off in both PyTorch and ONNX Runtime, so every FP32 row is true FP32.

Accuracy

Implementation	Device	Code agreement (median / min)	Decoder log-mel MAE	Round-trip log-mel MAE
transformers, FP32 (reference)	CPU	100.00% / 100.00%	0.000 dB	0.000 dB
transformers, FP32	CUDA	99.85% / 99.75%	0.000 dB	0.017 dB
transformers, BF16 (native weights)	CUDA	95.72% / 94.58%	0.069 dB	0.330 dB
ONNX (this repo)	CPU	99.93% / 99.83%	0.000 dB	0.009 dB
ONNX (this repo)	CUDA	99.93% / 99.83%	0.000 dB	0.009 dB
ONNX, FP16 decoder	CPU	99.93% / 99.83%	0.004 dB	0.013 dB
ONNX, FP16 decoder	CUDA	99.93% / 99.83%	0.010 dB	0.018 dB

The ONNX round trip stays within 0.01 dB of the reference, and within 0.02 dB with the FP16 decoder (whose rows use the FP32 encoder). Native BF16 weights flip 4-5% of codes.

Offline speed

Median over the seven clips of each clip's median latency (50 runs after 10 warmups). RTF is processing time divided by audio duration; lower is faster. Latency includes host-device copies and excludes model loading.

Implementation	Device	Encode	Decode	Encode RTF	Decode RTF
transformers, FP32 (reference)	CPU	41.02 ms	147.63 ms	0.0056	0.0197
ONNX (this repo)	CPU	23.32 ms	98.71 ms	0.0030	0.0130
transformers, FP32	CUDA	3.76 ms	5.16 ms	0.0005	0.0007
transformers, BF16 (native weights)	CUDA	4.32 ms	4.38 ms	0.0006	0.0006
ONNX (this repo)	CUDA	2.64 ms	3.92 ms	0.0004	0.0005
ONNX, FP16 decoder	CPU		98.31 ms		0.0134
ONNX, FP16 decoder	CUDA		2.77 ms		0.0004

ONNX Runtime is faster than transformers in FP32: 1.76x (encode) and 1.50x (decode) on CPU, 1.42x and 1.32x on CUDA. It is also faster than transformers with BF16 weights, while matching FP32 accuracy. On CUDA the FP16 decoder is a further 1.41x faster than the FP32 one (2.77 vs 3.92 ms). On CPU, ONNX Runtime has few FP16 kernels, so the FP16 decoder runs at the same speed as the FP32 one, at half the size.

Streaming speed

Each clip's reference codes are decoded as a stream of fixed-size chunks, three times after a warmup stream:

• Step latency: pooled over every step of every clip.
• First step: the time to first audio.
• Headroom: chunk duration divided by the 95th-percentile step latency. Above 1x, the decoder keeps up with playback.
• Errors: compare the stitched stream with transformers' offline decoding.

Implementation	Device	Chunk	Step latency (median / p95)	First step	Headroom	Log-mel MAE	Max abs error
transformers LiquidCodecStreamingCache	CPU	1 frame (80 ms)	9.14 / 12.12 ms	9.42 ms	7x	0.0000 dB	2.1e-06
transformers LiquidCodecStreamingCache	CPU	4 frames (320 ms)	21.94 / 25.39 ms	22.12 ms	13x	0.0000 dB	1.8e-06
ONNX decoder_step.onnx	CPU	1 frame (80 ms)	4.18 / 4.58 ms	4.19 ms	17x	0.0001 dB	3.1e-06
ONNX decoder_step.onnx	CPU	4 frames (320 ms)	6.19 / 6.86 ms	6.34 ms	47x	0.0000 dB	3.6e-06
transformers LiquidCodecStreamingCache	CUDA	1 frame (80 ms)	3.80 / 3.86 ms	3.86 ms	21x	0.0000 dB	4.5e-06
transformers LiquidCodecStreamingCache	CUDA	4 frames (320 ms)	3.86 / 4.02 ms	3.84 ms	80x	0.0000 dB	3.8e-06
ONNX decoder_step.onnx	CUDA	1 frame (80 ms)	1.34 / 1.44 ms	1.39 ms	56x	0.0000 dB	5.6e-06
ONNX decoder_step.onnx	CUDA	4 frames (320 ms)	1.47 / 2.00 ms	2.12 ms	160x	0.0000 dB	2.6e-06
ONNX decoder_step_fp16.onnx	CPU	1 frame (80 ms)	4.59 / 4.68 ms	4.65 ms	17x	0.0042 dB	7.4e-03
ONNX decoder_step_fp16.onnx	CPU	4 frames (320 ms)	6.30 / 7.10 ms	6.50 ms	45x	0.0042 dB	7.5e-03
ONNX decoder_step_fp16.onnx	CUDA	1 frame (80 ms)	1.34 / 1.82 ms	1.44 ms	44x	0.0099 dB	7.2e-03
ONNX decoder_step_fp16.onnx	CUDA	4 frames (320 ms)	1.45 / 2.10 ms	2.01 ms	153x	0.0097 dB	7.1e-03

decoder_step.onnx on CUDA decodes an 80 ms chunk in 1.34 ms, 2.8x faster than transformers' streaming cache on the same GPU. On CPU it is 2.2x faster. The FP16 step is no faster than the FP32 one: a streaming step is so small that launching its operators, not arithmetic, dominates. It is half the size, and stays within 0.01 dB of the reference.

Changes from the original model

The original PyTorch checkpoint (revision b5abfac4) was converted to ONNX opset 18 in FP32, with these numerically equivalent rewrites:

• the complex torch.stft as a strided Conv1d with a fixed windowed DFT basis;
• torch.polar + irfft as cos/sin and one matrix product with an inverse-DFT basis;
• F.fold overlap-add as four shifted additions;
• the attention masks built explicitly, with eager attention;
• the x6 nearest-neighbour upsampling as a repeat.

For decoder_step.onnx, the state held by LiquidCodecStreamingCache became explicit inputs and outputs.

decoder_fp16.onnx and decoder_step_fp16.onnx are the same graphs with the learned layers converted to FP16:

• FP16: the dequantizer projection, the LFM3 layers' matrix products and convolutions, and the output projection.
• FP32: the RMSNorm variance, softmax and rotary tables, the inverse STFT and overlap-add, and the inputs, outputs and stream state.