- 🎹 2D Convolutional Conditional VAE (CVAE) for Piano Music Generation
🎹 2D Convolutional Conditional VAE (CVAE) for Piano Music Generation
A 18.91M-parameter 2D Convolutional Conditional Variational Autoencoder (CVAE) trained from scratch on 60,608 Piano Mel-Spectrogram segments (256 × 130, 3s @ 22,050 Hz) using NVIDIA L4 (TF32 + Native BF16).
A 18,911,041-parameter 2D Convolutional Conditional Variational Autoencoder (
CVAE) neural network trained 100% from scratch on 997 solo Piano.mp3recordings (60,608 training segments) over 100 Epochs (approx. 104.33 minutes) on a single NVIDIA L4 GPU (24GB VRAM) using native BF16 + TF32 mixed precision. Slices raw audio into non-overlapping 3-second segments (66,150samples at22,050 Hz), projects them into 256-band log-frequency Mel-Spectrograms (256 × 130), and learns a structured 128-dimensional continuous latent space via Gaussian reparameterization. Achieves 348.72 final train ELBO loss and 412.07 total loss (386.00 MSE reconstruction, 26.07 KLD) on 32 held-out test segments with zero posterior collapse.
✨ Highlights
- 🎵 End-to-End Audio Feature Extraction Pipeline: Slices 997 solo piano recordings into 60,608 train segments and 32 held-out test segments (
1 × 256 × 130normalized dB-scaled Mel-Spectrograms,33,280time-frequency pixels per segment). - 🧠 Robust Variational Convergence Without Posterior Collapse:
- Train ELBO Loss: Converged from
720.27(Epoch 1) →359.99(Epoch 50) →348.72(Epoch 100). - Reconstruction MSE: Minimizes from
686.70down to322.80(~`0.0097mean squared error per pixel on normalized[0, 1]` scale). - Latent Regularization: KL Divergence stabilizes at
25.93(Train) and26.07(Held-Out Test), proving the latent manifold maintains continuous Gaussian geometry without collapsing to the prior $D_{KL} \to 0$ or exploding.
- Train ELBO Loss: Converged from
- ⚡ Modern Accelerator Optimization: Accelerated with TF32 tensor cores + Native BF16 mixed precision (
torch.amp.autocast),batch_size = 64, pinned host memory, and multi-process data loading (num_workers = 4), running full audio ingestion and 100-epoch training in6,259.98s(104.33 min) on 1× NVIDIA L4 (24GB VRAM). - 📊 Empirical Held-Out Generalization: Tight generalization gap between Train (
348.72) and Test (412.07, +18.2%), preserving sustained bass resonance, pitch registers, and harmonic fundamental envelopes. - 🔓 Standalone Architecture Packaging: Architecture definition decoupled into
model_architecture.pyfor lightweight zero-dependency deployment and inference via Hugging Face Hub.
🚀 Quick Start
Load the standalone model architecture and pretrained weights directly from Hugging Face Hub for Mel-Spectrogram generation and audio reconstruction:
import os
import sys
import torch
from huggingface_hub import hf_hub_download
# 1. Compute Device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
repo_id = "GiangSon-5/CVAE_Music_Generation"
# 2. Download standalone architecture definition and weights from Hub
arch_path = hf_hub_download(repo_id=repo_id, filename="model_architecture.py")
weights_path = hf_hub_download(repo_id=repo_id, filename="cvae_music_model.pth")
# 3. Dynamically import CVAE class
sys.path.insert(0, os.path.dirname(arch_path))
from model_architecture import CVAE
# 4. Instantiate model and load state dictionary
model = CVAE(d_model=64, latent_dim=128, n_mels=256, n_frames=130, n_genres=1).to(device)
model.load_state_dict(torch.load(weights_path, map_location=device))
model.eval()
# 5. Generate novel audio Mel-Spectrogram (prior sampling z ~ N(0, I))
with torch.no_grad():
z_prior = torch.randn(1, 128, device=device)
genre_cond = torch.tensor([[1.0]], device=device) # Piano genre condition
z_cond = torch.cat((z_prior, genre_cond), dim=1)
dec_features = model.decoder_input(z_cond).view(1, 256, model.n_mels_down, model.n_frames_down)
sampled_spec = model.decoder(dec_features)[:, :, :256, :130]
print(f"Sampled Spectrogram Shape : {tuple(sampled_spec.shape)}")
print(f"Normalized Value Range : min={sampled_spec.min().item():.4f}, max={sampled_spec.max().item():.4f}")
💬 Sample Predictions & Qualitative Evaluation
Empirical qualitative comparison of representative unseen 3-second Piano test segments from the 32 held-out test split, contrasting the ground-truth acoustic profile against CVAE reconstructions:
| Test Sample | Musical Profile & Pitch Register | Ground-Truth Spectrogram Characteristics | CVAE Model Reconstruction Outcome | Segment Recon MSE | Perceptual & Spectral Assessment |
|---|---|---|---|---|---|
| Test Sample #1 | Low-to-Mid Chord Progression with rhythmic changes | Strong fundamental energy at Mel bins 10–50; transient strikes at frames t ∈ {45, 65, 88}; overtone decays up to bin 150. | Faithfully recovers continuous low-register harmonic bands; vertically softens sharp strike attacks into smooth continuous energy transitions. | 0.0094 / px | ✅ High Harmonic Fidelity: Accurate tonal energy envelope; bass warmth preserved; minor transient blur. |
| Test Sample #2 | Mid-Register Staccato Phrasing with note onsets | Rapid sequence of keystrikes at frames t ∈ {18, 32, 54, 78, 108} with clear silence intervals in upper bins (100–256). | Accurately reconstructs baseline melodic pitch contour; suppresses background noise; merges rapid staccato notes into a sustained envelope. | 0.0098 / px | ✅ Accurate Pitch Envelope: Clean dynamic range; staccato attacks slightly averaged due to L2 loss. |
| Test Sample #3 | Dense Polyphonic Arpeggio with rich harmonics | Complex multi-voice harmonics spanning Mel bins 20–120; subtle reverberation tail extending across frames 60–120. | Reconstructs the multi-frequency resonance centroid; sustains correct energy concentration across time; filters out fine overtone noise. | 0.0102 / px | ✅ Strong Global Timbre: Dominant polyphony preserved; high-frequency overtone details smoothed. |
1. Architecture & Model Specs
| Technical Specification | Configuration | Architectural Notes |
|---|---|---|
| Model Type | 2D Convolutional Conditional VAE (CVAE) |
Symmetric Encoder-Decoder with Latent Information Bottleneck |
| Total Trainable Parameters | 18,911,041 (~18.91M) | PyTorch checkpoint (cvae_music_model.pth): 72.14 MB |
| Input / Output Shape | (B, 1, 256, 130) |
Single-channel Mel-Spectrogram (256 mel bins, 130 time frames @ 3.0s) |
| Conditioning Vector | (B, n_genres = 1) |
One-hot / continuous genre embedding map concatenated at input and latent |
| Encoder Feature Channels | (1 + 1) → 64 → 128 → 256 |
3 Conv2D stages (kernel=3, stride=2, padding=1, BatchNorm2d, SiLU, Dropout) |
| Spatial Bottleneck | AdaptiveAvgPool2d((1, 1)) |
Compresses spatial feature map to a 256-D global representation |
| Latent Space Dimension | latent_dim = 128 |
Twin linear heads: fc_mu (128-D) and fc_logvar (128-D) |
| Reparameterization Trick | z = μ + σ ⊙ ε | Continuous sampling with ε ~ N(0, I), σ = exp(0.5 × logvar) |
| Decoder Projection | Linear(128 + 1 → 139,264) |
Expands latent vector + genre back to (256 × 32 × 17) spatial tensor |
| Decoder Feature Channels | 256 → 128 → 64 → 1 |
3 ConvTranspose2D stages (kernel=3, stride=2, padding=1, out_pad=1, SiLU) |
| Final Activation Function | Sigmoid() |
Bound outputs strictly to [0, 1] matching min-max normalized audio spectra |
2. Training Data & Procedure
Data Engineering & Acoustic Preprocessing
- Raw Audio Corpus: 997 crawled Piano
.mp3audio files with paired.jsonmetadata ({'Piano': 0}). - Sampling Rate Normalization: All audio tracks resampled to
22,050 Hzvialibrosa. - Segmentation: Tracks divided into non-overlapping 3.0-second intervals (
fixed_length = 22,050 × 3 = 66,150audio samples). Any residual tail shorter than 3 seconds is discarded. - Time-Frequency Representation:
- Short-Time Fourier Transform (STFT) computed with Hann window,
n_fft = 2048, andhop_length = 512samples (~`23.2 ms` per frame). - Mel-scale filterbank: 256 triangular Mel filters spanning
0 Hzto11,025 Hz(Nyquist frequency). - Log-magnitude dynamic range compression: Decibel scaling via $S_{\text{dB}} = 10 \cdot \log_{10}(S / \max(S))$.
- Normalized to
[0, 1]via min-max scaling: $S_{\text{norm}} = (S_{\text{dB}} - \min) / (\max - \min)$.
- Short-Time Fourier Transform (STFT) computed with Hann window,
- Dataset Partition:
- Train Split: 60,608 segments (948 full audio recordings).
- Held-Out Test Split: 32 segments strictly isolated from the training corpus.
Mathematical Formulation & Objective Function
The network optimizes the Evidence Lower Bound (ELBO):
- Reconstruction Loss (Summed squared error across all 33,280 time-frequency bins):
- Kullback-Leibler Divergence (Closed-form Gaussian relative to isotropic prior $\mathcal{N}(0, I)$):
Training Configuration
| Hyperparameter | Value | Technical Rationale |
|---|---|---|
| Compute Hardware | 1× NVIDIA L4 SXM (24 GB VRAM) | Modern Ada Lovelace GPU with high FP32/BF16 tensor performance |
| Optimizer | AdamW |
Weight decay = 0.01, β₁ = 0.9, β₂ = 0.999, ε = 1e-8 |
| Base Learning Rate | 2e-4 |
Optimal starting rate for convolutional autoencoders with batch normalization |
| LR Scheduler | StepLR (step_size=50, gamma=0.5) |
Halves learning rate at Epoch 50 (1e-4) and Epoch 100 (5e-5) for fine convergence |
| Batch Size | 64 segments | 947 optimization gradient steps per epoch (~`94,700` total updates) |
| Numerical Precision | TF32 + Native BF16 | Enabled torch.backends.cuda.matmul.allow_tf32 = True and autocast('cuda', dtype=bfloat16) |
| Data Loader Pipeline | pin_memory = True, num_workers = 4 |
Eliminates PCIe transfer bottlenecks between host RAM and L4 VRAM |
| Total Training Runtime | 6,259.98 seconds (~104.33 min) | Comprehensive audio loading, FFT extraction, and 100 full training epochs |
3. Evaluation
3.1. Training Convergence Progression Across 100 Epochs
Log-verified checkpoint metrics recorded across key optimization milestones:
| Optimization Milestone | Total ELBO Loss | Recon MSE Loss | KL Divergence (D_KL) | Learning Rate | Technical Status |
|---|---|---|---|---|---|
| Epoch 1 | 720.27 | 686.70 | 33.57 | 2.0e-04 |
Warmup state; rapid reduction of global reconstruction error |
| Epoch 10 | 410.95 | 390.87 | 20.07 | 2.0e-04 |
Latent regularization reaches equilibrium; broad timbre formed |
| Epoch 30 | 372.76 | 350.42 | 22.34 | 2.0e-04 |
Steady refinement of harmonic overtone bands |
| Epoch 50 (LR Drop) | 359.99 | 336.03 | 23.95 | 1.0e-04 |
LR halved via StepLR; smooth transition into fine-tuning regime |
| Epoch 70 | 353.69 | 329.03 | 24.67 | 1.0e-04 |
Stable balance between reconstruction MSE and KLD regularizer |
| Epoch 100 (Final Train) | 348.72 | 322.80 | 25.93 | 5.0e-05 |
Converged state; stable continuous latent representation |
| Held-Out Test Split | 412.07 | 386.00 | 26.07 | — | Tight generalization gap: +63.35 total loss (+18.2%) |
3.2. Independent Held-Out Test Benchmark
| Evaluation Subset | Segments | Total ELBO Loss | Recon MSE Loss | KL Divergence (D_KL) | Normalized MSE / Pixel | Posterior Collapse |
|---|---|---|---|---|---|---|
| Training Corpus (Final) | 60,608 | 348.72 | 322.80 | 25.93 | 0.0097 / px |
❌ None (D_KL = 25.93) |
| Held-Out Test Split | 32 | 412.07 | 386.00 | 26.07 | 0.0116 / px |
❌ None (D_KL = 26.07) |
| Generalization Delta (Δ) | — | +63.35 (+18.2%) | +63.20 (+19.6%) | +0.14 (+0.5%) | +0.0019 / px | Zero Posterior Collapse |
Evaluation Methodology & Latent Integrity:
- Zero Posterior Collapse: Throughout 100 epochs, $D_{KL}$ remains in the healthy range of $20.07 \to 26.07$, confirming that the encoder is actively utilizing the full 128-D latent capacity rather than ignoring latent codes $z$.
- Tight Generalization Gap: A test loss delta of only +18.2% on completely unseen audio files proves that the convolutional encoder-decoder learns generalizable piano timbre rather than memorizing individual recording waveforms.
3.3. Empirical Research Dashboards
1. Training Convergence Dynamics Dashboard
Figure 1: Triple-panel training dynamics across 100 epochs on NVIDIA L4: (1) Total ELBO Loss convergence (720.27 → 348.72); (2) Reconstruction MSE minimization (686.70 → 322.80); (3) KL Divergence stabilization at 25.93 without posterior collapse.
2. Held-Out Test Mel-Spectrogram Reconstruction Fidelity
Figure 2: Qualitative acoustic profile comparison on 3 unseen held-out test segments (256 × 130 Mel-Spectrograms): Ground-Truth (left) vs. CVAE Model Reconstruction (right), demonstrating preservation of low-to-mid fundamental pitch contours and harmonic resonance.
3. Continuous Latent Space Manifold (2D PCA Projection)
Figure 3: Principal Component Analysis (2D PCA) of the 128-dimensional latent mean vectors μ across held-out test audio segments, confirming dense continuous Gaussian geometry without isolated outlier clusters.
4. Limitations & Critical Analysis
- Spatial Information Bottleneck (
AdaptiveAvgPool2d((1, 1))):- The encoder compresses the $(256 \times 32 \times 17)$ spatial feature tensor down to $(256 \times 1 \times 1)$ before linear latent projection.
- While this enforces a clean 1D continuous latent vector $z \in \mathbb{R}^{128}$ and prevents posterior collapse, it removes fine-grained temporal alignment, causing the decoder to generate smooth approximations rather than crisp transient keystrokes.
- Mean-Seeking Behavior of L2 Loss:
- Training with pixel-wise Mean Squared Error naturally leads to blurry predictions (regression to the conditional expectation $\mathbb{E}[x|z]$). Future iterations could adopt Multi-Scale Spectral Loss, adversarial loss (PatchGAN), or perceptual loss (e.g., via a pretrained audio encoder like OpenL3 or EnCodec).
- Phase Inversion Bottleneck:
- The model generates magnitude Mel-Spectrograms without phase. Synthesizing audible
.wavrequires Griffin-Lim phase estimation (which introduces mild metallic artifacting) or pairing with a dedicated neural vocoder (e.g., HiFi-GAN or MelGAN).
- The model generates magnitude Mel-Spectrograms without phase. Synthesizing audible
- Single-Genre Dataset Distribution:
- The crawled dataset contains exclusively solo piano audio (
n_genres = 1). Consequently, the conditional pathway functions as an invariant architectural prior rather than enabling dynamic multi-genre style transfer.
- The crawled dataset contains exclusively solo piano audio (
5. Google Colab & Reproducibility
Execute the full training pipeline or run interactive inference on Google Colab:
6. Citation & References
- 📄 Auto-Encoding Variational Bayes (VAE): Diederik P. Kingma, Max Welling. Auto-Encoding Variational Bayes (ICLR 2014). arXiv:1312.6114
- 📄 Conditional Variational Autoencoders (CVAE): Kihyuk Sohn, Honglak Lee, Xinchen Yan. Learning Structured Output Representation using Deep Conditional Generative Models (NeurIPS 2015). Paper Link
- 📄 Deep Learning Techniques for Music Generation: Jean-Pierre Briot, Gaëtan Hadjeres, François Pachet. Deep Learning Techniques for Music Generation: A Survey (Computational Intelligence 2020). arXiv:1709.01620
- 📄 Librosa: Audio and Music Signal Analysis in Python: Brian McFee et al. librosa: Audio and Music Signal Analysis in Python (SciPy 2015). DOI:10.25080/Majora-7b98e3ed-003
7. License & Author
- License: MIT Permissive Open Source License
- Author: Giang Sơn (GiangSon-5)
Papers for GiangSon-5/CVAE_Music_Generation
Auto-Encoding Variational Bayes
Evaluation results
- Held-Out Test ELBO Loss on Piano Audio Dataset (60,608 train / 32 held-out test segments)self-reported412.070
- Held-Out Test Recon MSE on Piano Audio Dataset (60,608 train / 32 held-out test segments)self-reported386.000
- Held-Out Test KL Divergence on Piano Audio Dataset (60,608 train / 32 held-out test segments)self-reported26.070


