--- title: Classical Music Generator emoji: 🎼 colorFrom: purple colorTo: blue sdk: gradio sdk_version: "4.44.1" python_version: '3.8' app_file: app.py pinned: false --- # Classical Music Generator A Transformer-based neural network system for generating multi-instrument classical music using a grid-based representation. This project uses PyTorch and provides an interactive Gradio web interface with interactive music composition controls. ## Overview Classical Music Generator is an AI-powered music composition tool that generates original classical music arrangements across four instrument families: Strings, Woodwinds, Brass, and Percussion. The system uses a Transformer encoder with per-instrument prediction heads to generate music sequences quantized on a fixed 8-step-per-bar grid at 120 BPM. ### Key Features - **Multi-instrument composition**: Generates music for Strings, Woodwinds, Brass, and Percussion simultaneously - **Grid-based representation**: Music is represented as a quantized 8-step-per-bar grid for precise control - **Pitch control**: Melodic instruments limited to one octave (C4–B4, 12 pitches), percussion mapped to standard drum sounds - **Transformer architecture**: Uses attention-based Transformer encoder with positional encoding for sequential music generation - **Interactive web UI**: Gradio-based interface with real-time audio synthesis and visual feedback - **Music theory constraints**: Built-in support for scale enforcement and note limiting to ensure musical coherence - **MIDI support**: Full MIDI input/output for compatibility with DAWs and other music software ## Architecture ### Model Details The core model is a **MusicTransformer** with the following components: - **Input layer**: Projects 39-dimensional multi-hot grid (12 strings + 12 woodwinds + 12 brass + 3 percussion pitches) to the model embedding - **Positional encoding**: Sinusoidal positional encoding for temporal sequence information - **Transformer encoder**: - Hidden size: 256 dimensions - Layers: 3 transformer encoder layers - Attention heads: 4 - Feed-forward dimension: 512 - Dropout: 0.1 - **Output heads**: 4 separate linear heads for per-instrument pitch prediction - Strings head: 12 pitches - Woodwinds head: 12 pitches - Brass head: 12 pitches - Percussion head: 3 pitches (Kick, Snare, Hi-hat) ### Grid-Based Representation Music is represented as a sequence of time steps with the following quantization: - **Tempo**: 120 BPM - **Time signature**: 4/4 - **Steps per bar**: 8 - **Seconds per step**: 0.125 (125 ms) - **Input context**: 3 bars (24 steps) of conditioning - **Output**: Generation continues from seeded context Each time step is represented as a 39-dimensional binary vector: ``` [strings (12) | woodwinds (12) | brass (12) | percussion (3)] ``` ## Installation ### Prerequisites - Python 3.8 or higher - PyTorch 2.0+ - CUDA 11.8+ (optional, for GPU acceleration) ### Setup 1. Clone the repository: ```bash git clone cd ClassicalMusicGenerator ``` 2. Install dependencies: ```bash pip install -r requirements.txt ``` 3. Install system dependencies for audio synthesis: ```bash # macOS brew install fluidsynth # Ubuntu/Debian sudo apt-get install fluidsynth # Fedora sudo dnf install fluidsynth ``` ## Running the Project ### 1. Interactive Web Interface Launch the Gradio-based interactive UI with visual grid controls: ```bash python app.py ``` Then open your browser to `http://localhost:7860` (or the URL displayed in the terminal). **Features**: - Interactive 8-step grid editor for each instrument family - Real-time audio preview via Web Audio synthesis - Generate longer compositions with customizable length - Visual feedback showing active notes and pitch controls - Color-coded instrument families for easy identification ### 2. Training the Model To train on your own MIDI dataset: ```bash python train.py \ --dataset-path data/training_data.npz \ --num-epochs 20 \ --batch-size 32 \ --learning-rate 1e-3 ``` **Options**: - `--dataset-path`: Path to preprocessed dataset (NPZ or PT format) - `--num-epochs`: Number of training epochs - `--batch-size`: Batch size for training - `--learning-rate`: Adam optimizer learning rate - `--val-split`: Validation split ratio (default: 0.1) - `--device`: Device to use ('cuda' or 'cpu', auto-detected by default) - `--save-dir`: Directory to save model checkpoints (default: 'models') **Resume training**: The training script automatically resumes from checkpoints if they exist: ```bash python train.py --dataset-path data/training_data.npz --extra-epochs 10 ``` ### 3. Data Preprocessing Prepare MIDI files for training: ```bash python data_preprocessing.py \ --midi-dir path/to/midi/files \ --output-path data/training_data.npz ``` This script: - Loads MIDI files from a directory - Maps instruments to 4 families (Strings, Woodwinds, Brass, Percussion) - Quantizes notes to the fixed grid (120 BPM, 8 steps/bar) - Creates input-output pairs for training - Saves as NumPy NPZ format ### 4. Inference Scripts For programmatic music generation: ```python from inference import load_model, generate_midi_and_audio import numpy as np # Load pre-trained model model = load_model("models/transformer_music_best.pt") # Create seed grids (3 bars of context) strings_seed = np.zeros((3, 8, 12), dtype=np.float32) woodwinds_seed = np.zeros((3, 8, 12), dtype=np.float32) brass_seed = np.zeros((3, 8, 12), dtype=np.float32) perc_seed = np.zeros((3, 8, 3), dtype=np.float32) # Generate 1 minute of music midi_path, wav_path = generate_midi_and_audio( strings_seed=strings_seed, woodwinds_seed=woodwinds_seed, brass_seed=brass_seed, perc_seed=perc_seed, target_minutes=1.0, device="cpu" ) ``` ## Project Structure ``` ClassicalMusicGenerator/ ├── app.py # Gradio web interface with grid controls ├── config.py # Global configuration (grid, model, tempo) ├── model.py # MusicTransformer architecture ├── train.py # Training loop with checkpoint support ├── inference.py # Model loading and generation utilities ├── data_preprocessing.py # MIDI processing and dataset creation ├── classicalmusicgenerator.py # Colab notebook (reference) ├── test_grad.py # Gradient checking utilities ├── requirements.txt # Python dependencies ├── models/ │ └── transformer_music_best.pt # Pre-trained model weights └── utils/ ├── midi_utils.py # MIDI conversion utilities ├── sampling_utils.py # Generation constraints and sampling └── __pycache__/ ``` ## Configuration Edit [config.py](config.py) to customize: ### Grid Parameters ```python STEPS_PER_BAR = 8 # Time steps per 4/4 bar INPUT_BARS = 3 # Context bars for conditioning TEMPO_BPM = 120 # Tempo in beats per minute ``` ### Model Hyperparameters ```python D_MODEL = 256 # Hidden dimension N_LAYERS = 3 # Transformer layers N_HEADS = 4 # Attention heads FFN_DIM = 512 # Feed-forward dimension DROPOUT = 0.1 # Dropout rate ``` ### Generation Constraints ```python BAR_SIMILARITY_THRESHOLD = 0.9 # Avoid repetitive sections MAX_RESAMPLE_ATTEMPTS_PER_BAR = 5 # Resampling attempts per bar MIN_ACTIVE_FAMILIES_PER_BAR = 2 # Minimum instrument families per bar ``` ## Dataset Format The preprocessing creates training data where: - **X**: Input sequences of shape `[N, INPUT_STEPS, INPUT_FEATURES]` (context) - **Y**: Target sequences of shape `[N, INPUT_STEPS, INPUT_FEATURES]` (next step prediction) Where: - `N` = number of sequences - `INPUT_STEPS` = 24 (3 bars × 8 steps) - `INPUT_FEATURES` = 39 (12+12+12+3 pitches) Saved as NPZ or PyTorch PT format with keys `"X"` and `"Y"`. ## Training Tips - **Dataset size**: Aim for at least 10,000-50,000 sequences for good results - **Learning rate**: Start with 1e-3, reduce by 0.5× if loss plateaus - **Batch size**: 32 or 64 depending on GPU memory - **Validation split**: 10% is reasonable; increase to 15-20% for smaller datasets - **Checkpointing**: Models are auto-saved at each epoch, best model tracked separately ## Troubleshooting ### CUDA Out of Memory Reduce batch size or use CPU: ```bash python train.py --dataset-path data/training_data.npz --batch-size 16 --device cpu ``` ### Missing Audio Dependencies Ensure fluidsynth is installed (see Installation section above). ### Gradio Connection Issues If the web interface won't open, check the terminal output for the correct localhost address, typically `http://localhost:7860`. ## References - Transformer architecture: "[Attention Is All You Need](https://arxiv.org/abs/1706.03762)" (Vaswani et al., 2017) - Music representation: Grid-based quantization inspired by [MusicVAE](https://magenta.tensorflow.org/music-vae) and [Jukebox](https://openai.com/blog/jukebox/) - Implementation uses [PyTorch](https://pytorch.org/), [Gradio](https://gradio.app/), and [pretty_midi](https://github.com/craffel/pretty_midi) ## Contributing Contributions are welcome! Please: 1. Fork the repository 2. Create a feature branch 3. Commit your changes 4. Submit a pull request