File size: 3,626 Bytes
b2b7452 9d0a8a5 b2b7452 430ca55 b2b7452 9d0a8a5 b2b7452 9d0a8a5 430ca55 b2b7452 9d0a8a5 430ca55 9d0a8a5 430ca55 9d0a8a5 430ca55 9d0a8a5 b2b7452 9d0a8a5 b2b7452 9d0a8a5 b2b7452 9d0a8a5 b2b7452 9d0a8a5 b2b7452 9d0a8a5 b2b7452 9d0a8a5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 | ---
license: apache-2.0
tags:
- audio
- heartcodec
- encoder
---
# HeartCodec OSS Encoder
Encoder-only weights for HeartCodec audio tokenization. Pair this checkpoint
with [HeartCodec-oss-20260123](https://huggingface.co/HeartMuLa/HeartCodec-oss-20260123),
which supplies the shared RVQ quantizer and audio decoder. Together they support
audio → eight token streams at 12.5 Hz → 48 kHz stereo reconstruction.
The matching code and reconstruction examples are in
[HeartMuLa/heartlib](https://github.com/HeartMuLa/heartlib).
We hope this contribution will be useful to the music research community.
## Installation and reconstruction
Use Python 3.10, a CUDA-compatible PyTorch installation, and `ffmpeg` for MP3
output. Install `heartlib` in a separate environment.
```bash
git clone https://github.com/HeartMuLa/heartlib.git
cd heartlib
pip install -e .
hf download HeartMuLa/HeartCodec-oss-encoder --local-dir ./ckpt/HeartCodec-oss-encoder
hf download HeartMuLa/HeartCodec-oss-20260123 --local-dir ./ckpt/HeartCodec-oss-20260123
python examples/run_music_reconstruction.py \
--encoder_path ./ckpt/HeartCodec-oss-encoder \
--decoder_path ./ckpt/HeartCodec-oss-20260123 \
--input_path ./assets/reference.mp3 \
--save_path ./reconstructed.mp3
```
Replace `--input_path` to use your own authorized audio. MP3 is encoded directly from float32 output at
320 kbps, without an intermediate PCM WAV. WAV output is available by selecting
a `.wav` output path. Output is trimmed to the input duration.
## Python API
```python
import torch
from heartlib.heartcodec.modeling_heartcodec import HeartCodec
codec = HeartCodec.from_encoder_decoder_pretrained(
"HeartMuLa/HeartCodec-oss-20260123",
"HeartMuLa/HeartCodec-oss-encoder",
dtype=torch.float32,
).to("cuda").eval()
# waveform: float32 tensor [samples], [1, samples], or [2, samples]
# sample_rate: positive integer sampling rate of the input waveform
# tokens = codec.tokenize(waveform, sample_rate, batch_size=1)
# reconstructed = codec.detokenize(tokens, num_steps=10, guidance_scale=1.25)
```
Do not pass this encoder-only repository to `HeartCodec.from_pretrained()`.
The existing decoder-only loading API remains supported. Encoder and decoder
weights are strictly checked by the split-checkpoint loader.
## Files
- `encoder.safetensors`: 1,006 encoder tensors, approximately 2.15 GB;
original `encoder.*` names, shapes, dtypes and values are preserved.
- `encoder_config.json`: feature extractor and query encoder configuration.
- `SHA256SUMS`: weight-file SHA-256 checksum.
- `LICENSE`: the source checkpoint's Apache-2.0 license.
No decoder or RVQ weights are included here. The shared quantizer belongs to
the paired decoder checkpoint; loading two independently trained quantizers
is not supported.
## Validation
On an NVIDIA B300, the split encoder and published decoder completed a
239.293-second stereo audio round trip using FP32, batch size 1, seed 42,
10 decoder steps and guidance scale 1.25. Tokens had shape `[8, 2992]`.
Output was finite, non-silent, 48 kHz stereo and matched the input duration.
PyTorch peak allocated memory was approximately 12.65 GiB; peak reserved
memory was approximately 15.41 GiB. These are measurements for one input and
configuration, not minimum-memory requirements or an audio-quality benchmark.
All exported encoder tensor bytes match the source checkpoint. The paired
decoder's 818 tensors match the source decoder numerically; eight RVQ
initialization flags differ only in stored shape (scalar versus `[1]`).
## License
Apache-2.0, as declared by the source checkpoint. See `LICENSE`.
|