Music_Generator / README.md
hephaestusonfire's picture
Update README.md
b3d9dcd verified
|
Raw History Blame Contribute Delete
9.46 kB
---
title: Classical Music Generator
emoji: 🎼
colorFrom: purple
colorTo: blue
sdk: gradio
sdk_version: "4.44.1"
python_version: '3.8'
app_file: app.py
pinned: false
---
# Classical Music Generator
A Transformer-based neural network system for generating multi-instrument classical music using a grid-based representation. This project uses PyTorch and provides an interactive Gradio web interface with interactive music composition controls.
## Overview
Classical Music Generator is an AI-powered music composition tool that generates original classical music arrangements across four instrument families: Strings, Woodwinds, Brass, and Percussion. The system uses a Transformer encoder with per-instrument prediction heads to generate music sequences quantized on a fixed 8-step-per-bar grid at 120 BPM.
### Key Features
- **Multi-instrument composition**: Generates music for Strings, Woodwinds, Brass, and Percussion simultaneously
- **Grid-based representation**: Music is represented as a quantized 8-step-per-bar grid for precise control
- **Pitch control**: Melodic instruments limited to one octave (C4–B4, 12 pitches), percussion mapped to standard drum sounds
- **Transformer architecture**: Uses attention-based Transformer encoder with positional encoding for sequential music generation
- **Interactive web UI**: Gradio-based interface with real-time audio synthesis and visual feedback
- **Music theory constraints**: Built-in support for scale enforcement and note limiting to ensure musical coherence
- **MIDI support**: Full MIDI input/output for compatibility with DAWs and other music software
## Architecture
### Model Details
The core model is a **MusicTransformer** with the following components:
- **Input layer**: Projects 39-dimensional multi-hot grid (12 strings + 12 woodwinds + 12 brass + 3 percussion pitches) to the model embedding
- **Positional encoding**: Sinusoidal positional encoding for temporal sequence information
- **Transformer encoder**:
- Hidden size: 256 dimensions
- Layers: 3 transformer encoder layers
- Attention heads: 4
- Feed-forward dimension: 512
- Dropout: 0.1
- **Output heads**: 4 separate linear heads for per-instrument pitch prediction
- Strings head: 12 pitches
- Woodwinds head: 12 pitches
- Brass head: 12 pitches
- Percussion head: 3 pitches (Kick, Snare, Hi-hat)
### Grid-Based Representation
Music is represented as a sequence of time steps with the following quantization:
- **Tempo**: 120 BPM
- **Time signature**: 4/4
- **Steps per bar**: 8
- **Seconds per step**: 0.125 (125 ms)
- **Input context**: 3 bars (24 steps) of conditioning
- **Output**: Generation continues from seeded context
Each time step is represented as a 39-dimensional binary vector:
```
[strings (12) | woodwinds (12) | brass (12) | percussion (3)]
```
## Installation
### Prerequisites
- Python 3.8 or higher
- PyTorch 2.0+
- CUDA 11.8+ (optional, for GPU acceleration)
### Setup
1. Clone the repository:
```bash
git clone <repository-url>
cd ClassicalMusicGenerator
```
2. Install dependencies:
```bash
pip install -r requirements.txt
```
3. Install system dependencies for audio synthesis:
```bash
# macOS
brew install fluidsynth
# Ubuntu/Debian
sudo apt-get install fluidsynth
# Fedora
sudo dnf install fluidsynth
```
## Running the Project
### 1. Interactive Web Interface
Launch the Gradio-based interactive UI with visual grid controls:
```bash
python app.py
```
Then open your browser to `http://localhost:7860` (or the URL displayed in the terminal).
**Features**:
- Interactive 8-step grid editor for each instrument family
- Real-time audio preview via Web Audio synthesis
- Generate longer compositions with customizable length
- Visual feedback showing active notes and pitch controls
- Color-coded instrument families for easy identification
### 2. Training the Model
To train on your own MIDI dataset:
```bash
python train.py \
--dataset-path data/training_data.npz \
--num-epochs 20 \
--batch-size 32 \
--learning-rate 1e-3
```
**Options**:
- `--dataset-path`: Path to preprocessed dataset (NPZ or PT format)
- `--num-epochs`: Number of training epochs
- `--batch-size`: Batch size for training
- `--learning-rate`: Adam optimizer learning rate
- `--val-split`: Validation split ratio (default: 0.1)
- `--device`: Device to use ('cuda' or 'cpu', auto-detected by default)
- `--save-dir`: Directory to save model checkpoints (default: 'models')
**Resume training**:
The training script automatically resumes from checkpoints if they exist:
```bash
python train.py --dataset-path data/training_data.npz --extra-epochs 10
```
### 3. Data Preprocessing
Prepare MIDI files for training:
```bash
python data_preprocessing.py \
--midi-dir path/to/midi/files \
--output-path data/training_data.npz
```
This script:
- Loads MIDI files from a directory
- Maps instruments to 4 families (Strings, Woodwinds, Brass, Percussion)
- Quantizes notes to the fixed grid (120 BPM, 8 steps/bar)
- Creates input-output pairs for training
- Saves as NumPy NPZ format
### 4. Inference Scripts
For programmatic music generation:
```python
from inference import load_model, generate_midi_and_audio
import numpy as np
# Load pre-trained model
model = load_model("models/transformer_music_best.pt")
# Create seed grids (3 bars of context)
strings_seed = np.zeros((3, 8, 12), dtype=np.float32)
woodwinds_seed = np.zeros((3, 8, 12), dtype=np.float32)
brass_seed = np.zeros((3, 8, 12), dtype=np.float32)
perc_seed = np.zeros((3, 8, 3), dtype=np.float32)
# Generate 1 minute of music
midi_path, wav_path = generate_midi_and_audio(
strings_seed=strings_seed,
woodwinds_seed=woodwinds_seed,
brass_seed=brass_seed,
perc_seed=perc_seed,
target_minutes=1.0,
device="cpu"
)
```
## Project Structure
```
ClassicalMusicGenerator/
β”œβ”€β”€ app.py # Gradio web interface with grid controls
β”œβ”€β”€ config.py # Global configuration (grid, model, tempo)
β”œβ”€β”€ model.py # MusicTransformer architecture
β”œβ”€β”€ train.py # Training loop with checkpoint support
β”œβ”€β”€ inference.py # Model loading and generation utilities
β”œβ”€β”€ data_preprocessing.py # MIDI processing and dataset creation
β”œβ”€β”€ classicalmusicgenerator.py # Colab notebook (reference)
β”œβ”€β”€ test_grad.py # Gradient checking utilities
β”œβ”€β”€ requirements.txt # Python dependencies
β”œβ”€β”€ models/
β”‚ └── transformer_music_best.pt # Pre-trained model weights
└── utils/
β”œβ”€β”€ midi_utils.py # MIDI conversion utilities
β”œβ”€β”€ sampling_utils.py # Generation constraints and sampling
└── __pycache__/
```
## Configuration
Edit [config.py](config.py) to customize:
### Grid Parameters
```python
STEPS_PER_BAR = 8 # Time steps per 4/4 bar
INPUT_BARS = 3 # Context bars for conditioning
TEMPO_BPM = 120 # Tempo in beats per minute
```
### Model Hyperparameters
```python
D_MODEL = 256 # Hidden dimension
N_LAYERS = 3 # Transformer layers
N_HEADS = 4 # Attention heads
FFN_DIM = 512 # Feed-forward dimension
DROPOUT = 0.1 # Dropout rate
```
### Generation Constraints
```python
BAR_SIMILARITY_THRESHOLD = 0.9 # Avoid repetitive sections
MAX_RESAMPLE_ATTEMPTS_PER_BAR = 5 # Resampling attempts per bar
MIN_ACTIVE_FAMILIES_PER_BAR = 2 # Minimum instrument families per bar
```
## Dataset Format
The preprocessing creates training data where:
- **X**: Input sequences of shape `[N, INPUT_STEPS, INPUT_FEATURES]` (context)
- **Y**: Target sequences of shape `[N, INPUT_STEPS, INPUT_FEATURES]` (next step prediction)
Where:
- `N` = number of sequences
- `INPUT_STEPS` = 24 (3 bars Γ— 8 steps)
- `INPUT_FEATURES` = 39 (12+12+12+3 pitches)
Saved as NPZ or PyTorch PT format with keys `"X"` and `"Y"`.
## Training Tips
- **Dataset size**: Aim for at least 10,000-50,000 sequences for good results
- **Learning rate**: Start with 1e-3, reduce by 0.5Γ— if loss plateaus
- **Batch size**: 32 or 64 depending on GPU memory
- **Validation split**: 10% is reasonable; increase to 15-20% for smaller datasets
- **Checkpointing**: Models are auto-saved at each epoch, best model tracked separately
## Troubleshooting
### CUDA Out of Memory
Reduce batch size or use CPU:
```bash
python train.py --dataset-path data/training_data.npz --batch-size 16 --device cpu
```
### Missing Audio Dependencies
Ensure fluidsynth is installed (see Installation section above).
### Gradio Connection Issues
If the web interface won't open, check the terminal output for the correct localhost address, typically `http://localhost:7860`.
## References
- Transformer architecture: "[Attention Is All You Need](https://arxiv.org/abs/1706.03762)" (Vaswani et al., 2017)
- Music representation: Grid-based quantization inspired by [MusicVAE](https://magenta.tensorflow.org/music-vae) and [Jukebox](https://openai.com/blog/jukebox/)
- Implementation uses [PyTorch](https://pytorch.org/), [Gradio](https://gradio.app/), and [pretty_midi](https://github.com/craffel/pretty_midi)
## Contributing
Contributions are welcome! Please:
1. Fork the repository
2. Create a feature branch
3. Commit your changes
4. Submit a pull request