File size: 3,626 Bytes
a0283e3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 | ---
license: apache-2.0
tags:
- audio
- heartcodec
- encoder
---
# HeartCodec OSS Encoder
Encoder-only weights for HeartCodec audio tokenization. Pair this checkpoint
with [HeartCodec-oss-20260123](https://huggingface.co/HeartMuLa/HeartCodec-oss-20260123),
which supplies the shared RVQ quantizer and audio decoder. Together they support
audio → eight token streams at 12.5 Hz → 48 kHz stereo reconstruction.
The matching code and reconstruction examples are in
[HeartMuLa/heartlib](https://github.com/HeartMuLa/heartlib).
We hope this contribution will be useful to the music research community.
## Installation and reconstruction
Use Python 3.10, a CUDA-compatible PyTorch installation, and `ffmpeg` for MP3
output. Install `heartlib` in a separate environment.
```bash
git clone https://github.com/HeartMuLa/heartlib.git
cd heartlib
pip install -e .
hf download HeartMuLa/HeartCodec-oss-encoder --local-dir ./ckpt/HeartCodec-oss-encoder
hf download HeartMuLa/HeartCodec-oss-20260123 --local-dir ./ckpt/HeartCodec-oss-20260123
python examples/run_music_reconstruction.py \
--encoder_path ./ckpt/HeartCodec-oss-encoder \
--decoder_path ./ckpt/HeartCodec-oss-20260123 \
--input_path ./assets/reference.mp3 \
--save_path ./reconstructed.mp3
```
Replace `--input_path` to use your own authorized audio. MP3 is encoded directly from float32 output at
320 kbps, without an intermediate PCM WAV. WAV output is available by selecting
a `.wav` output path. Output is trimmed to the input duration.
## Python API
```python
import torch
from heartlib.heartcodec.modeling_heartcodec import HeartCodec
codec = HeartCodec.from_encoder_decoder_pretrained(
"HeartMuLa/HeartCodec-oss-20260123",
"HeartMuLa/HeartCodec-oss-encoder",
dtype=torch.float32,
).to("cuda").eval()
# waveform: float32 tensor [samples], [1, samples], or [2, samples]
# sample_rate: positive integer sampling rate of the input waveform
# tokens = codec.tokenize(waveform, sample_rate, batch_size=1)
# reconstructed = codec.detokenize(tokens, num_steps=10, guidance_scale=1.25)
```
Do not pass this encoder-only repository to `HeartCodec.from_pretrained()`.
The existing decoder-only loading API remains supported. Encoder and decoder
weights are strictly checked by the split-checkpoint loader.
## Files
- `encoder.safetensors`: 1,006 encoder tensors, approximately 2.15 GB;
original `encoder.*` names, shapes, dtypes and values are preserved.
- `encoder_config.json`: feature extractor and query encoder configuration.
- `SHA256SUMS`: weight-file SHA-256 checksum.
- `LICENSE`: the source checkpoint's Apache-2.0 license.
No decoder or RVQ weights are included here. The shared quantizer belongs to
the paired decoder checkpoint; loading two independently trained quantizers
is not supported.
## Validation
On an NVIDIA B300, the split encoder and published decoder completed a
239.293-second stereo audio round trip using FP32, batch size 1, seed 42,
10 decoder steps and guidance scale 1.25. Tokens had shape `[8, 2992]`.
Output was finite, non-silent, 48 kHz stereo and matched the input duration.
PyTorch peak allocated memory was approximately 12.65 GiB; peak reserved
memory was approximately 15.41 GiB. These are measurements for one input and
configuration, not minimum-memory requirements or an audio-quality benchmark.
All exported encoder tensor bytes match the source checkpoint. The paired
decoder's 818 tensors match the source decoder numerically; eight RVQ
initialization flags differ only in stored shape (scalar versus `[1]`).
## License
Apache-2.0, as declared by the source checkpoint. See `LICENSE`.
|