File size: 9,459 Bytes
f4b8912
 
 
 
 
 
b3d9dcd
69ba7ff
f4b8912
 
 
 
9c7e38e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
69ba7ff
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
---
title: Classical Music Generator
emoji: 🎼
colorFrom: purple
colorTo: blue
sdk: gradio
sdk_version: "4.44.1"
python_version: '3.8'
app_file: app.py
pinned: false
---

# Classical Music Generator

A Transformer-based neural network system for generating multi-instrument classical music using a grid-based representation. This project uses PyTorch and provides an interactive Gradio web interface with interactive music composition controls.

## Overview

Classical Music Generator is an AI-powered music composition tool that generates original classical music arrangements across four instrument families: Strings, Woodwinds, Brass, and Percussion. The system uses a Transformer encoder with per-instrument prediction heads to generate music sequences quantized on a fixed 8-step-per-bar grid at 120 BPM.

### Key Features

- **Multi-instrument composition**: Generates music for Strings, Woodwinds, Brass, and Percussion simultaneously
- **Grid-based representation**: Music is represented as a quantized 8-step-per-bar grid for precise control
- **Pitch control**: Melodic instruments limited to one octave (C4–B4, 12 pitches), percussion mapped to standard drum sounds
- **Transformer architecture**: Uses attention-based Transformer encoder with positional encoding for sequential music generation
- **Interactive web UI**: Gradio-based interface with real-time audio synthesis and visual feedback
- **Music theory constraints**: Built-in support for scale enforcement and note limiting to ensure musical coherence
- **MIDI support**: Full MIDI input/output for compatibility with DAWs and other music software

## Architecture

### Model Details

The core model is a **MusicTransformer** with the following components:

- **Input layer**: Projects 39-dimensional multi-hot grid (12 strings + 12 woodwinds + 12 brass + 3 percussion pitches) to the model embedding
- **Positional encoding**: Sinusoidal positional encoding for temporal sequence information
- **Transformer encoder**: 
  - Hidden size: 256 dimensions
  - Layers: 3 transformer encoder layers
  - Attention heads: 4
  - Feed-forward dimension: 512
  - Dropout: 0.1
- **Output heads**: 4 separate linear heads for per-instrument pitch prediction
  - Strings head: 12 pitches
  - Woodwinds head: 12 pitches
  - Brass head: 12 pitches
  - Percussion head: 3 pitches (Kick, Snare, Hi-hat)

### Grid-Based Representation

Music is represented as a sequence of time steps with the following quantization:

- **Tempo**: 120 BPM
- **Time signature**: 4/4
- **Steps per bar**: 8
- **Seconds per step**: 0.125 (125 ms)
- **Input context**: 3 bars (24 steps) of conditioning
- **Output**: Generation continues from seeded context

Each time step is represented as a 39-dimensional binary vector:
```
[strings (12) | woodwinds (12) | brass (12) | percussion (3)]
```

## Installation

### Prerequisites

- Python 3.8 or higher
- PyTorch 2.0+
- CUDA 11.8+ (optional, for GPU acceleration)

### Setup

1. Clone the repository:
```bash
git clone <repository-url>
cd ClassicalMusicGenerator
```

2. Install dependencies:
```bash
pip install -r requirements.txt
```

3. Install system dependencies for audio synthesis:
```bash
# macOS
brew install fluidsynth

# Ubuntu/Debian
sudo apt-get install fluidsynth

# Fedora
sudo dnf install fluidsynth
```

## Running the Project

### 1. Interactive Web Interface

Launch the Gradio-based interactive UI with visual grid controls:

```bash
python app.py
```

Then open your browser to `http://localhost:7860` (or the URL displayed in the terminal).

**Features**:
- Interactive 8-step grid editor for each instrument family
- Real-time audio preview via Web Audio synthesis
- Generate longer compositions with customizable length
- Visual feedback showing active notes and pitch controls
- Color-coded instrument families for easy identification

### 2. Training the Model

To train on your own MIDI dataset:

```bash
python train.py \
    --dataset-path data/training_data.npz \
    --num-epochs 20 \
    --batch-size 32 \
    --learning-rate 1e-3
```

**Options**:
- `--dataset-path`: Path to preprocessed dataset (NPZ or PT format)
- `--num-epochs`: Number of training epochs
- `--batch-size`: Batch size for training
- `--learning-rate`: Adam optimizer learning rate
- `--val-split`: Validation split ratio (default: 0.1)
- `--device`: Device to use ('cuda' or 'cpu', auto-detected by default)
- `--save-dir`: Directory to save model checkpoints (default: 'models')

**Resume training**:
The training script automatically resumes from checkpoints if they exist:
```bash
python train.py --dataset-path data/training_data.npz --extra-epochs 10
```

### 3. Data Preprocessing

Prepare MIDI files for training:

```bash
python data_preprocessing.py \
    --midi-dir path/to/midi/files \
    --output-path data/training_data.npz
```

This script:
- Loads MIDI files from a directory
- Maps instruments to 4 families (Strings, Woodwinds, Brass, Percussion)
- Quantizes notes to the fixed grid (120 BPM, 8 steps/bar)
- Creates input-output pairs for training
- Saves as NumPy NPZ format

### 4. Inference Scripts

For programmatic music generation:

```python
from inference import load_model, generate_midi_and_audio
import numpy as np

# Load pre-trained model
model = load_model("models/transformer_music_best.pt")

# Create seed grids (3 bars of context)
strings_seed = np.zeros((3, 8, 12), dtype=np.float32)
woodwinds_seed = np.zeros((3, 8, 12), dtype=np.float32)
brass_seed = np.zeros((3, 8, 12), dtype=np.float32)
perc_seed = np.zeros((3, 8, 3), dtype=np.float32)

# Generate 1 minute of music
midi_path, wav_path = generate_midi_and_audio(
    strings_seed=strings_seed,
    woodwinds_seed=woodwinds_seed,
    brass_seed=brass_seed,
    perc_seed=perc_seed,
    target_minutes=1.0,
    device="cpu"
)
```

## Project Structure

```
ClassicalMusicGenerator/
β”œβ”€β”€ app.py                      # Gradio web interface with grid controls
β”œβ”€β”€ config.py                   # Global configuration (grid, model, tempo)
β”œβ”€β”€ model.py                    # MusicTransformer architecture
β”œβ”€β”€ train.py                    # Training loop with checkpoint support
β”œβ”€β”€ inference.py                # Model loading and generation utilities
β”œβ”€β”€ data_preprocessing.py        # MIDI processing and dataset creation
β”œβ”€β”€ classicalmusicgenerator.py   # Colab notebook (reference)
β”œβ”€β”€ test_grad.py                # Gradient checking utilities
β”œβ”€β”€ requirements.txt            # Python dependencies
β”œβ”€β”€ models/
β”‚   └── transformer_music_best.pt    # Pre-trained model weights
└── utils/
    β”œβ”€β”€ midi_utils.py           # MIDI conversion utilities
    β”œβ”€β”€ sampling_utils.py       # Generation constraints and sampling
    └── __pycache__/
```

## Configuration

Edit [config.py](config.py) to customize:

### Grid Parameters
```python
STEPS_PER_BAR = 8              # Time steps per 4/4 bar
INPUT_BARS = 3                 # Context bars for conditioning
TEMPO_BPM = 120                # Tempo in beats per minute
```

### Model Hyperparameters
```python
D_MODEL = 256                  # Hidden dimension
N_LAYERS = 3                   # Transformer layers
N_HEADS = 4                    # Attention heads
FFN_DIM = 512                  # Feed-forward dimension
DROPOUT = 0.1                  # Dropout rate
```

### Generation Constraints
```python
BAR_SIMILARITY_THRESHOLD = 0.9          # Avoid repetitive sections
MAX_RESAMPLE_ATTEMPTS_PER_BAR = 5      # Resampling attempts per bar
MIN_ACTIVE_FAMILIES_PER_BAR = 2        # Minimum instrument families per bar
```

## Dataset Format

The preprocessing creates training data where:
- **X**: Input sequences of shape `[N, INPUT_STEPS, INPUT_FEATURES]` (context)
- **Y**: Target sequences of shape `[N, INPUT_STEPS, INPUT_FEATURES]` (next step prediction)

Where:
- `N` = number of sequences
- `INPUT_STEPS` = 24 (3 bars Γ— 8 steps)
- `INPUT_FEATURES` = 39 (12+12+12+3 pitches)

Saved as NPZ or PyTorch PT format with keys `"X"` and `"Y"`.

## Training Tips

- **Dataset size**: Aim for at least 10,000-50,000 sequences for good results
- **Learning rate**: Start with 1e-3, reduce by 0.5Γ— if loss plateaus
- **Batch size**: 32 or 64 depending on GPU memory
- **Validation split**: 10% is reasonable; increase to 15-20% for smaller datasets
- **Checkpointing**: Models are auto-saved at each epoch, best model tracked separately

## Troubleshooting

### CUDA Out of Memory
Reduce batch size or use CPU:
```bash
python train.py --dataset-path data/training_data.npz --batch-size 16 --device cpu
```

### Missing Audio Dependencies
Ensure fluidsynth is installed (see Installation section above).

### Gradio Connection Issues
If the web interface won't open, check the terminal output for the correct localhost address, typically `http://localhost:7860`.


## References

- Transformer architecture: "[Attention Is All You Need](https://arxiv.org/abs/1706.03762)" (Vaswani et al., 2017)
- Music representation: Grid-based quantization inspired by [MusicVAE](https://magenta.tensorflow.org/music-vae) and [Jukebox](https://openai.com/blog/jukebox/)
- Implementation uses [PyTorch](https://pytorch.org/), [Gradio](https://gradio.app/), and [pretty_midi](https://github.com/craffel/pretty_midi)

## Contributing

Contributions are welcome! Please:
1. Fork the repository
2. Create a feature branch
3. Commit your changes
4. Submit a pull request