Spaces:
Sleeping
Sleeping
Upload 11 files
Browse files- README.md +282 -15
- app.py +375 -0
- config.py +58 -0
- data_preprocessing.py +387 -0
- inference.py +485 -0
- model.py +163 -0
- models/transformer_music_best.pt +3 -0
- requirements.txt +19 -0
- train.py +215 -0
- utils/midi_utils.py +242 -0
- utils/sampling_utils.py +307 -0
README.md
CHANGED
|
@@ -1,15 +1,282 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
-
|
| 14 |
-
|
| 15 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Classical Music Generator
|
| 2 |
+
|
| 3 |
+
A Transformer-based neural network system for generating multi-instrument classical music using a grid-based representation. This project uses PyTorch and provides an interactive Gradio web interface with interactive music composition controls.
|
| 4 |
+
|
| 5 |
+
## Overview
|
| 6 |
+
|
| 7 |
+
Classical Music Generator is an AI-powered music composition tool that generates original classical music arrangements across four instrument families: Strings, Woodwinds, Brass, and Percussion. The system uses a Transformer encoder with per-instrument prediction heads to generate music sequences quantized on a fixed 8-step-per-bar grid at 120 BPM.
|
| 8 |
+
|
| 9 |
+
### Key Features
|
| 10 |
+
|
| 11 |
+
- **Multi-instrument composition**: Generates music for Strings, Woodwinds, Brass, and Percussion simultaneously
|
| 12 |
+
- **Grid-based representation**: Music is represented as a quantized 8-step-per-bar grid for precise control
|
| 13 |
+
- **Pitch control**: Melodic instruments limited to one octave (C4–B4, 12 pitches), percussion mapped to standard drum sounds
|
| 14 |
+
- **Transformer architecture**: Uses attention-based Transformer encoder with positional encoding for sequential music generation
|
| 15 |
+
- **Interactive web UI**: Gradio-based interface with real-time audio synthesis and visual feedback
|
| 16 |
+
- **Music theory constraints**: Built-in support for scale enforcement and note limiting to ensure musical coherence
|
| 17 |
+
- **MIDI support**: Full MIDI input/output for compatibility with DAWs and other music software
|
| 18 |
+
|
| 19 |
+
## Architecture
|
| 20 |
+
|
| 21 |
+
### Model Details
|
| 22 |
+
|
| 23 |
+
The core model is a **MusicTransformer** with the following components:
|
| 24 |
+
|
| 25 |
+
- **Input layer**: Projects 39-dimensional multi-hot grid (12 strings + 12 woodwinds + 12 brass + 3 percussion pitches) to the model embedding
|
| 26 |
+
- **Positional encoding**: Sinusoidal positional encoding for temporal sequence information
|
| 27 |
+
- **Transformer encoder**:
|
| 28 |
+
- Hidden size: 256 dimensions
|
| 29 |
+
- Layers: 3 transformer encoder layers
|
| 30 |
+
- Attention heads: 4
|
| 31 |
+
- Feed-forward dimension: 512
|
| 32 |
+
- Dropout: 0.1
|
| 33 |
+
- **Output heads**: 4 separate linear heads for per-instrument pitch prediction
|
| 34 |
+
- Strings head: 12 pitches
|
| 35 |
+
- Woodwinds head: 12 pitches
|
| 36 |
+
- Brass head: 12 pitches
|
| 37 |
+
- Percussion head: 3 pitches (Kick, Snare, Hi-hat)
|
| 38 |
+
|
| 39 |
+
### Grid-Based Representation
|
| 40 |
+
|
| 41 |
+
Music is represented as a sequence of time steps with the following quantization:
|
| 42 |
+
|
| 43 |
+
- **Tempo**: 120 BPM
|
| 44 |
+
- **Time signature**: 4/4
|
| 45 |
+
- **Steps per bar**: 8
|
| 46 |
+
- **Seconds per step**: 0.125 (125 ms)
|
| 47 |
+
- **Input context**: 3 bars (24 steps) of conditioning
|
| 48 |
+
- **Output**: Generation continues from seeded context
|
| 49 |
+
|
| 50 |
+
Each time step is represented as a 39-dimensional binary vector:
|
| 51 |
+
```
|
| 52 |
+
[strings (12) | woodwinds (12) | brass (12) | percussion (3)]
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
## Installation
|
| 56 |
+
|
| 57 |
+
### Prerequisites
|
| 58 |
+
|
| 59 |
+
- Python 3.8 or higher
|
| 60 |
+
- PyTorch 2.0+
|
| 61 |
+
- CUDA 11.8+ (optional, for GPU acceleration)
|
| 62 |
+
|
| 63 |
+
### Setup
|
| 64 |
+
|
| 65 |
+
1. Clone the repository:
|
| 66 |
+
```bash
|
| 67 |
+
git clone <repository-url>
|
| 68 |
+
cd ClassicalMusicGenerator
|
| 69 |
+
```
|
| 70 |
+
|
| 71 |
+
2. Install dependencies:
|
| 72 |
+
```bash
|
| 73 |
+
pip install -r requirements.txt
|
| 74 |
+
```
|
| 75 |
+
|
| 76 |
+
3. Install system dependencies for audio synthesis:
|
| 77 |
+
```bash
|
| 78 |
+
# macOS
|
| 79 |
+
brew install fluidsynth
|
| 80 |
+
|
| 81 |
+
# Ubuntu/Debian
|
| 82 |
+
sudo apt-get install fluidsynth
|
| 83 |
+
|
| 84 |
+
# Fedora
|
| 85 |
+
sudo dnf install fluidsynth
|
| 86 |
+
```
|
| 87 |
+
|
| 88 |
+
## Running the Project
|
| 89 |
+
|
| 90 |
+
### 1. Interactive Web Interface
|
| 91 |
+
|
| 92 |
+
Launch the Gradio-based interactive UI with visual grid controls:
|
| 93 |
+
|
| 94 |
+
```bash
|
| 95 |
+
python app.py
|
| 96 |
+
```
|
| 97 |
+
|
| 98 |
+
Then open your browser to `http://localhost:7860` (or the URL displayed in the terminal).
|
| 99 |
+
|
| 100 |
+
**Features**:
|
| 101 |
+
- Interactive 8-step grid editor for each instrument family
|
| 102 |
+
- Real-time audio preview via Web Audio synthesis
|
| 103 |
+
- Generate longer compositions with customizable length
|
| 104 |
+
- Visual feedback showing active notes and pitch controls
|
| 105 |
+
- Color-coded instrument families for easy identification
|
| 106 |
+
|
| 107 |
+
### 2. Training the Model
|
| 108 |
+
|
| 109 |
+
To train on your own MIDI dataset:
|
| 110 |
+
|
| 111 |
+
```bash
|
| 112 |
+
python train.py \
|
| 113 |
+
--dataset-path data/training_data.npz \
|
| 114 |
+
--num-epochs 20 \
|
| 115 |
+
--batch-size 32 \
|
| 116 |
+
--learning-rate 1e-3
|
| 117 |
+
```
|
| 118 |
+
|
| 119 |
+
**Options**:
|
| 120 |
+
- `--dataset-path`: Path to preprocessed dataset (NPZ or PT format)
|
| 121 |
+
- `--num-epochs`: Number of training epochs
|
| 122 |
+
- `--batch-size`: Batch size for training
|
| 123 |
+
- `--learning-rate`: Adam optimizer learning rate
|
| 124 |
+
- `--val-split`: Validation split ratio (default: 0.1)
|
| 125 |
+
- `--device`: Device to use ('cuda' or 'cpu', auto-detected by default)
|
| 126 |
+
- `--save-dir`: Directory to save model checkpoints (default: 'models')
|
| 127 |
+
|
| 128 |
+
**Resume training**:
|
| 129 |
+
The training script automatically resumes from checkpoints if they exist:
|
| 130 |
+
```bash
|
| 131 |
+
python train.py --dataset-path data/training_data.npz --extra-epochs 10
|
| 132 |
+
```
|
| 133 |
+
|
| 134 |
+
### 3. Data Preprocessing
|
| 135 |
+
|
| 136 |
+
Prepare MIDI files for training:
|
| 137 |
+
|
| 138 |
+
```bash
|
| 139 |
+
python data_preprocessing.py \
|
| 140 |
+
--midi-dir path/to/midi/files \
|
| 141 |
+
--output-path data/training_data.npz
|
| 142 |
+
```
|
| 143 |
+
|
| 144 |
+
This script:
|
| 145 |
+
- Loads MIDI files from a directory
|
| 146 |
+
- Maps instruments to 4 families (Strings, Woodwinds, Brass, Percussion)
|
| 147 |
+
- Quantizes notes to the fixed grid (120 BPM, 8 steps/bar)
|
| 148 |
+
- Creates input-output pairs for training
|
| 149 |
+
- Saves as NumPy NPZ format
|
| 150 |
+
|
| 151 |
+
### 4. Inference Scripts
|
| 152 |
+
|
| 153 |
+
For programmatic music generation:
|
| 154 |
+
|
| 155 |
+
```python
|
| 156 |
+
from inference import load_model, generate_midi_and_audio
|
| 157 |
+
import numpy as np
|
| 158 |
+
|
| 159 |
+
# Load pre-trained model
|
| 160 |
+
model = load_model("models/transformer_music_best.pt")
|
| 161 |
+
|
| 162 |
+
# Create seed grids (3 bars of context)
|
| 163 |
+
strings_seed = np.zeros((3, 8, 12), dtype=np.float32)
|
| 164 |
+
woodwinds_seed = np.zeros((3, 8, 12), dtype=np.float32)
|
| 165 |
+
brass_seed = np.zeros((3, 8, 12), dtype=np.float32)
|
| 166 |
+
perc_seed = np.zeros((3, 8, 3), dtype=np.float32)
|
| 167 |
+
|
| 168 |
+
# Generate 1 minute of music
|
| 169 |
+
midi_path, wav_path = generate_midi_and_audio(
|
| 170 |
+
strings_seed=strings_seed,
|
| 171 |
+
woodwinds_seed=woodwinds_seed,
|
| 172 |
+
brass_seed=brass_seed,
|
| 173 |
+
perc_seed=perc_seed,
|
| 174 |
+
target_minutes=1.0,
|
| 175 |
+
device="cpu"
|
| 176 |
+
)
|
| 177 |
+
```
|
| 178 |
+
|
| 179 |
+
## Project Structure
|
| 180 |
+
|
| 181 |
+
```
|
| 182 |
+
ClassicalMusicGenerator/
|
| 183 |
+
├── app.py # Gradio web interface with grid controls
|
| 184 |
+
├── config.py # Global configuration (grid, model, tempo)
|
| 185 |
+
├── model.py # MusicTransformer architecture
|
| 186 |
+
├── train.py # Training loop with checkpoint support
|
| 187 |
+
├── inference.py # Model loading and generation utilities
|
| 188 |
+
├── data_preprocessing.py # MIDI processing and dataset creation
|
| 189 |
+
├── classicalmusicgenerator.py # Colab notebook (reference)
|
| 190 |
+
├── test_grad.py # Gradient checking utilities
|
| 191 |
+
├── requirements.txt # Python dependencies
|
| 192 |
+
├── models/
|
| 193 |
+
│ └── transformer_music_best.pt # Pre-trained model weights
|
| 194 |
+
└── utils/
|
| 195 |
+
├── midi_utils.py # MIDI conversion utilities
|
| 196 |
+
├── sampling_utils.py # Generation constraints and sampling
|
| 197 |
+
└── __pycache__/
|
| 198 |
+
```
|
| 199 |
+
|
| 200 |
+
## Configuration
|
| 201 |
+
|
| 202 |
+
Edit [config.py](config.py) to customize:
|
| 203 |
+
|
| 204 |
+
### Grid Parameters
|
| 205 |
+
```python
|
| 206 |
+
STEPS_PER_BAR = 8 # Time steps per 4/4 bar
|
| 207 |
+
INPUT_BARS = 3 # Context bars for conditioning
|
| 208 |
+
TEMPO_BPM = 120 # Tempo in beats per minute
|
| 209 |
+
```
|
| 210 |
+
|
| 211 |
+
### Model Hyperparameters
|
| 212 |
+
```python
|
| 213 |
+
D_MODEL = 256 # Hidden dimension
|
| 214 |
+
N_LAYERS = 3 # Transformer layers
|
| 215 |
+
N_HEADS = 4 # Attention heads
|
| 216 |
+
FFN_DIM = 512 # Feed-forward dimension
|
| 217 |
+
DROPOUT = 0.1 # Dropout rate
|
| 218 |
+
```
|
| 219 |
+
|
| 220 |
+
### Generation Constraints
|
| 221 |
+
```python
|
| 222 |
+
BAR_SIMILARITY_THRESHOLD = 0.9 # Avoid repetitive sections
|
| 223 |
+
MAX_RESAMPLE_ATTEMPTS_PER_BAR = 5 # Resampling attempts per bar
|
| 224 |
+
MIN_ACTIVE_FAMILIES_PER_BAR = 2 # Minimum instrument families per bar
|
| 225 |
+
```
|
| 226 |
+
|
| 227 |
+
## Dataset Format
|
| 228 |
+
|
| 229 |
+
The preprocessing creates training data where:
|
| 230 |
+
- **X**: Input sequences of shape `[N, INPUT_STEPS, INPUT_FEATURES]` (context)
|
| 231 |
+
- **Y**: Target sequences of shape `[N, INPUT_STEPS, INPUT_FEATURES]` (next step prediction)
|
| 232 |
+
|
| 233 |
+
Where:
|
| 234 |
+
- `N` = number of sequences
|
| 235 |
+
- `INPUT_STEPS` = 24 (3 bars × 8 steps)
|
| 236 |
+
- `INPUT_FEATURES` = 39 (12+12+12+3 pitches)
|
| 237 |
+
|
| 238 |
+
Saved as NPZ or PyTorch PT format with keys `"X"` and `"Y"`.
|
| 239 |
+
|
| 240 |
+
## Training Tips
|
| 241 |
+
|
| 242 |
+
- **Dataset size**: Aim for at least 10,000-50,000 sequences for good results
|
| 243 |
+
- **Learning rate**: Start with 1e-3, reduce by 0.5× if loss plateaus
|
| 244 |
+
- **Batch size**: 32 or 64 depending on GPU memory
|
| 245 |
+
- **Validation split**: 10% is reasonable; increase to 15-20% for smaller datasets
|
| 246 |
+
- **Checkpointing**: Models are auto-saved at each epoch, best model tracked separately
|
| 247 |
+
|
| 248 |
+
## Troubleshooting
|
| 249 |
+
|
| 250 |
+
### CUDA Out of Memory
|
| 251 |
+
Reduce batch size or use CPU:
|
| 252 |
+
```bash
|
| 253 |
+
python train.py --dataset-path data/training_data.npz --batch-size 16 --device cpu
|
| 254 |
+
```
|
| 255 |
+
|
| 256 |
+
### Missing Audio Dependencies
|
| 257 |
+
Ensure fluidsynth is installed (see Installation section above).
|
| 258 |
+
|
| 259 |
+
### Gradio Connection Issues
|
| 260 |
+
If the web interface won't open, check the terminal output for the correct localhost address, typically `http://localhost:7860`.
|
| 261 |
+
|
| 262 |
+
## License
|
| 263 |
+
|
| 264 |
+
[Specify your license here]
|
| 265 |
+
|
| 266 |
+
## References
|
| 267 |
+
|
| 268 |
+
- Transformer architecture: "[Attention Is All You Need](https://arxiv.org/abs/1706.03762)" (Vaswani et al., 2017)
|
| 269 |
+
- Music representation: Grid-based quantization inspired by [MusicVAE](https://magenta.tensorflow.org/music-vae) and [Jukebox](https://openai.com/blog/jukebox/)
|
| 270 |
+
- Implementation uses [PyTorch](https://pytorch.org/), [Gradio](https://gradio.app/), and [pretty_midi](https://github.com/craffel/pretty_midi)
|
| 271 |
+
|
| 272 |
+
## Contributing
|
| 273 |
+
|
| 274 |
+
Contributions are welcome! Please:
|
| 275 |
+
1. Fork the repository
|
| 276 |
+
2. Create a feature branch
|
| 277 |
+
3. Commit your changes
|
| 278 |
+
4. Submit a pull request
|
| 279 |
+
|
| 280 |
+
## Contact
|
| 281 |
+
|
| 282 |
+
[Add contact information if desired]
|
app.py
ADDED
|
@@ -0,0 +1,375 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Gradio app for classical music generation with a grid-based Transformer.
|
| 3 |
+
Completely native, optimized Gradio 6.0 engine with variable frequency pitch
|
| 4 |
+
synthesis and an active coordinate control surface HUD display.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
import os
|
| 8 |
+
import tempfile
|
| 9 |
+
from typing import Tuple
|
| 10 |
+
import gradio as gr
|
| 11 |
+
import numpy as np
|
| 12 |
+
|
| 13 |
+
import config
|
| 14 |
+
from inference import generate_midi_and_audio
|
| 15 |
+
|
| 16 |
+
# ==============================================================================
|
| 17 |
+
# UI Constants & Color Mappings
|
| 18 |
+
# ==============================================================================
|
| 19 |
+
_MELODIC_NOTE_NAMES = ["C4","C#4","D4","D#4","E4","F4","F#4","G4","G#4","A4","A#4","B4"]
|
| 20 |
+
_PERC_NOTE_NAMES = ["Kick", "Snare", "Hi-hat"]
|
| 21 |
+
|
| 22 |
+
STRINGS_COLOR = "#63b3ed"
|
| 23 |
+
WOODWINDS_COLOR = "#6ed29b"
|
| 24 |
+
BRASS_COLOR = "#fbbf24"
|
| 25 |
+
PERC_COLOR = "#fc814a"
|
| 26 |
+
|
| 27 |
+
# CSS Esthetic styling system overrides to restore the sleek studio grid interface layout
|
| 28 |
+
CSS_ESTHETIC = """
|
| 29 |
+
.instrument-container {
|
| 30 |
+
background-color: #151b26 !important;
|
| 31 |
+
border: 1px solid rgba(255, 255, 255, 0.05) !important;
|
| 32 |
+
padding: 14px !important;
|
| 33 |
+
border-radius: 8px !important;
|
| 34 |
+
}
|
| 35 |
+
.instrument-margin {
|
| 36 |
+
margin-top: 15px !important;
|
| 37 |
+
}
|
| 38 |
+
.instrument-margin-button {
|
| 39 |
+
margin-top: 5px !important;
|
| 40 |
+
}
|
| 41 |
+
.grid-label {
|
| 42 |
+
font-family: monospace !important;
|
| 43 |
+
font-size: 11px !important;
|
| 44 |
+
font-weight: bold !important;
|
| 45 |
+
color: #c9d1e8 !important;
|
| 46 |
+
text-align: right;
|
| 47 |
+
align-self: center;
|
| 48 |
+
}
|
| 49 |
+
.matrix-cell {
|
| 50 |
+
min-width: 20px !important;
|
| 51 |
+
height: 26px !important;
|
| 52 |
+
padding: 0 !important;
|
| 53 |
+
margin: 0 !important;
|
| 54 |
+
border-radius: 2px !important;
|
| 55 |
+
font-size: 0px !important;
|
| 56 |
+
cursor: pointer !important;
|
| 57 |
+
}
|
| 58 |
+
.cell-off {
|
| 59 |
+
background-color: #111827 !important;
|
| 60 |
+
border: 1px solid rgba(255, 255, 255, 0.02) !important;
|
| 61 |
+
}
|
| 62 |
+
.cell-strings-on { background-color: #63b3ed !important; box-shadow: inset 0 0 5px #fff !important; }
|
| 63 |
+
.cell-brass-on { background-color: #fbbf24 !important; box-shadow: inset 0 0 4px #fff !important; }
|
| 64 |
+
.cell-woodwinds-on { background-color: #6ed29b !important; box-shadow: inset 0 0 4px #fff !important; }
|
| 65 |
+
.cell-perc-on { background-color: #fc814a !important; box-shadow: inset 0 0 4px #fff !important; }
|
| 66 |
+
|
| 67 |
+
/* Timeline Bar Separations */
|
| 68 |
+
.bar-line-left {
|
| 69 |
+
border-left: 2px solid rgba(255, 255, 255, 0.25) !important;
|
| 70 |
+
}
|
| 71 |
+
.action-bar-layout {
|
| 72 |
+
margin-top: 10px;
|
| 73 |
+
}
|
| 74 |
+
.hud-container {
|
| 75 |
+
background-color: #0f141c !important;
|
| 76 |
+
border: 1px solid rgba(99, 179, 237, 0.2) !important;
|
| 77 |
+
padding: 12px !important;
|
| 78 |
+
border-radius: 6px !important;
|
| 79 |
+
}
|
| 80 |
+
"""
|
| 81 |
+
|
| 82 |
+
# Dynamic Web Audio JS script to synthesize accurate note pitches
|
| 83 |
+
JS_AUDIO_ENGINE = """
|
| 84 |
+
(row, is_perc) => {
|
| 85 |
+
try {
|
| 86 |
+
window.AudioContext = window.AudioContext || window.webkitAudioContext;
|
| 87 |
+
let ctx = window._gradioAudioCtx || new AudioContext();
|
| 88 |
+
window._gradioAudioCtx = ctx;
|
| 89 |
+
if (ctx.state === 'suspended') ctx.resume();
|
| 90 |
+
|
| 91 |
+
let now = ctx.currentTime;
|
| 92 |
+
let gainNode = ctx.createGain();
|
| 93 |
+
gainNode.connect(ctx.destination);
|
| 94 |
+
|
| 95 |
+
if (is_perc === 1) {
|
| 96 |
+
let bufferSize = Math.floor(ctx.sampleRate * 0.12);
|
| 97 |
+
let buffer = ctx.createBuffer(1, bufferSize, ctx.sampleRate);
|
| 98 |
+
let data = buffer.getChannelData(0);
|
| 99 |
+
for (let i = 0; i < bufferSize; i++) data[i] = Math.random() * 2 - 1;
|
| 100 |
+
|
| 101 |
+
let source = ctx.createBufferSource(); source.buffer = buffer;
|
| 102 |
+
let filter = ctx.createBiquadFilter(); filter.type = 'bandpass';
|
| 103 |
+
|
| 104 |
+
if (row === 0) { filter.frequency.setValueAtTime(55, now); filter.Q.setValueAtTime(5, now); gainNode.gain.setValueAtTime(0.8, now); gainNode.gain.exponentialRampToValueAtTime(0.001, now + 0.1); }
|
| 105 |
+
else if (row === 1) { filter.frequency.setValueAtTime(900, now); filter.Q.setValueAtTime(2, now); gainNode.gain.setValueAtTime(0.4, now); gainNode.gain.exponentialRampToValueAtTime(0.001, now + 0.12); }
|
| 106 |
+
else { filter.frequency.setValueAtTime(8000, now); filter.Q.setValueAtTime(8, now); gainNode.gain.setValueAtTime(0.2, now); gainNode.gain.exponentialRampToValueAtTime(0.001, now + 0.04); }
|
| 107 |
+
|
| 108 |
+
source.connect(filter); filter.connect(gainNode);
|
| 109 |
+
source.start(now); source.stop(now + 0.12);
|
| 110 |
+
} else {
|
| 111 |
+
let midiNote = 60 + row;
|
| 112 |
+
let freq = 440 * Math.pow(2, (midiNote - 69) / 12);
|
| 113 |
+
|
| 114 |
+
let osc = ctx.createOscillator();
|
| 115 |
+
osc.type = 'triangle';
|
| 116 |
+
osc.frequency.setValueAtTime(freq, now);
|
| 117 |
+
|
| 118 |
+
gainNode.gain.setValueAtTime(0.25, now);
|
| 119 |
+
gainNode.gain.exponentialRampToValueAtTime(0.001, now + 0.3);
|
| 120 |
+
|
| 121 |
+
osc.connect(gainNode);
|
| 122 |
+
osc.start(now); osc.stop(now + 0.3);
|
| 123 |
+
}
|
| 124 |
+
} catch(e) { console.error(e); }
|
| 125 |
+
return [];
|
| 126 |
+
}
|
| 127 |
+
"""
|
| 128 |
+
|
| 129 |
+
# ==============================================================================
|
| 130 |
+
# Pipeline Execution Adapter
|
| 131 |
+
# ==============================================================================
|
| 132 |
+
|
| 133 |
+
def generate(strings, woodwinds, brass, perc, length, temp, tk, lead):
|
| 134 |
+
"""Callback parsing python state matrices smoothly straight into your core backend."""
|
| 135 |
+
tmpdir = tempfile.mkdtemp()
|
| 136 |
+
midi_path = os.path.join(tmpdir, "generated.mid")
|
| 137 |
+
audio_path = os.path.join(tmpdir, "generated.wav")
|
| 138 |
+
|
| 139 |
+
def to_backend_layout(flat_grid, r_dims):
|
| 140 |
+
arr = np.array(flat_grid, dtype=np.float32).reshape(r_dims, config.INPUT_STEPS)
|
| 141 |
+
reshaped = arr.reshape(r_dims, config.INPUT_BARS, config.STEPS_PER_BAR)
|
| 142 |
+
return np.transpose(reshaped, (1, 2, 0))
|
| 143 |
+
|
| 144 |
+
s_seed = to_backend_layout(strings, config.PITCHES_MELODIC)
|
| 145 |
+
w_seed = to_backend_layout(woodwinds, config.PITCHES_MELODIC)
|
| 146 |
+
b_seed = to_backend_layout(brass, config.PITCHES_MELODIC)
|
| 147 |
+
p_seed = to_backend_layout(perc, config.PITCHES_PERC)
|
| 148 |
+
|
| 149 |
+
midi_path, audio_path = generate_midi_and_audio(
|
| 150 |
+
strings_seed=s_seed, woodwinds_seed=w_seed, brass_seed=b_seed, perc_seed=p_seed,
|
| 151 |
+
target_minutes=length, temperature=temp, top_k=int(tk),
|
| 152 |
+
midi_out_path=midi_path, audio_out_path=audio_path, device="cpu", primary_family=lead
|
| 153 |
+
)
|
| 154 |
+
return midi_path, (audio_path if os.path.exists(audio_path) else None)
|
| 155 |
+
|
| 156 |
+
# ==============================================================================
|
| 157 |
+
# Interactive Runtime State Logic
|
| 158 |
+
# ==============================================================================
|
| 159 |
+
|
| 160 |
+
def toggle_cell_state(flat_grid, r, c, max_rows, family):
|
| 161 |
+
"""Toggles state values natively in pure python, updates button styles, and tracks HUD info."""
|
| 162 |
+
idx = (r * config.INPUT_STEPS) + c
|
| 163 |
+
current_value = flat_grid[idx]
|
| 164 |
+
new_value = 1 if current_value == 0 else 0
|
| 165 |
+
flat_grid[idx] = new_value
|
| 166 |
+
|
| 167 |
+
cls = "cell-off" if new_value == 0 else f"cell-{family}-on"
|
| 168 |
+
if c > 0 and c % config.STEPS_PER_BAR == 0:
|
| 169 |
+
cls += " bar-line-left"
|
| 170 |
+
|
| 171 |
+
bar_num = (c // config.STEPS_PER_BAR) + 1
|
| 172 |
+
step_num = (c % config.STEPS_PER_BAR) + 1
|
| 173 |
+
note_name = _PERC_NOTE_NAMES[r] if family == "perc" else _MELODIC_NOTE_NAMES[r]
|
| 174 |
+
status = "🔴 ACTIVATED" if new_value == 1 else "⚪ DEACTIVATED"
|
| 175 |
+
|
| 176 |
+
hud_info = f"### 🎛️ Selection HUD\n* **Family:** {family.upper()}\n* **Note / Track:** `{note_name}`\n* **Location:** Bar {bar_num}, Step {step_num} (Global Index {c})\n* **Status:** {status}"
|
| 177 |
+
|
| 178 |
+
return flat_grid, gr.update(elem_classes=["matrix-cell", cls]), hud_info
|
| 179 |
+
|
| 180 |
+
def clear_family_grid(max_rows):
|
| 181 |
+
"""Clears a whole family block instantly."""
|
| 182 |
+
blank = [0] * (max_rows * config.INPUT_STEPS)
|
| 183 |
+
updates = []
|
| 184 |
+
for r in range(max_rows):
|
| 185 |
+
for c in range(config.INPUT_STEPS):
|
| 186 |
+
cls = "cell-off"
|
| 187 |
+
if c > 0 and c % config.STEPS_PER_BAR == 0:
|
| 188 |
+
cls += " bar-line-left"
|
| 189 |
+
updates.append(gr.update(elem_classes=["matrix-cell", cls]))
|
| 190 |
+
return [blank] + updates
|
| 191 |
+
|
| 192 |
+
# ==============================================================================
|
| 193 |
+
# UI Composition Window
|
| 194 |
+
# ==============================================================================
|
| 195 |
+
|
| 196 |
+
with gr.Blocks(title="Classical Music Grid Generator") as demo:
|
| 197 |
+
|
| 198 |
+
gr.HTML(f"<script>{JS_AUDIO_ENGINE}</script>")
|
| 199 |
+
|
| 200 |
+
gr.HTML("""
|
| 201 |
+
<div style="text-align:center; padding:15px 0;">
|
| 202 |
+
<h1 style="font-size:2.2rem; font-weight:800; background:linear-gradient(135deg, #63b3ed 0%, #9b5de5 50%, #fc814a 100%); -webkit-background-clip:text; -webkit-text-fill-color:transparent; margin:0 0 5px 0;">
|
| 203 |
+
🎼 Classical Music Grid Generator
|
| 204 |
+
</h1>
|
| 205 |
+
<p style="color:#8892aa; font-size:14px; max-width:650px; margin:0 auto;">
|
| 206 |
+
Click native studio blocks to build arrangement structures. Notes play instant tone reviews as you paint them.
|
| 207 |
+
</p>
|
| 208 |
+
</div>
|
| 209 |
+
""")
|
| 210 |
+
|
| 211 |
+
with gr.Accordion("📖 Setup Blueprint & Manual Guide", open=True):
|
| 212 |
+
gr.HTML("""
|
| 213 |
+
<div style="display:grid; grid-template-columns:repeat(auto-fit,minmax(220px,1fr)); gap:12px; padding:6px 0;">
|
| 214 |
+
<div style="background:rgba(99,179,237,0.06); border:1px solid rgba(99,179,237,0.2); border-radius:8px; padding:12px;">
|
| 215 |
+
<strong style="color:#63b3ed;">1. Draw Framework Across 3 Bars</strong>
|
| 216 |
+
<p style="color:#8892aa; font-size:12px; margin:4px 0 0; line-height:1.4;">
|
| 217 |
+
Click empty slots to activate note triggers across all three bars. Click an active colorful note to erase it from the timeline instantly.
|
| 218 |
+
</p>
|
| 219 |
+
</div>
|
| 220 |
+
<div style="background:rgba(110,210,155,0.06); border:1px solid rgba(110,210,155,0.2); border-radius:8px; padding:12px;">
|
| 221 |
+
<strong style="color:#6ed29b;">2. Coordinate Inspection HUD</strong>
|
| 222 |
+
<p style="color:#8892aa; font-size:12px; margin:4px 0 0; line-height:1.4;">
|
| 223 |
+
Look at the live <strong>Active Selection HUD</strong> panel on the right after any click to inspect precisely which note and step index value was changed.
|
| 224 |
+
</p>
|
| 225 |
+
</div>
|
| 226 |
+
<div style="background:rgba(252,129,74,0.06); border:1px solid rgba(252,129,74,0.2); border-radius:8px; padding:12px;">
|
| 227 |
+
<strong style="color:#fc814a;">3. Compile & Listen</strong>
|
| 228 |
+
<p style="color:#8892aa; font-size:12px; margin:4px 0 0; line-height:1.4;">
|
| 229 |
+
Adjust composition parameter sliders, then click generate. Use the inline media layout player widget to listen to your full classical orchestration piece.
|
| 230 |
+
</p>
|
| 231 |
+
</div>
|
| 232 |
+
</div>
|
| 233 |
+
""")
|
| 234 |
+
|
| 235 |
+
strings_state = gr.State([0] * (config.PITCHES_MELODIC * config.INPUT_STEPS))
|
| 236 |
+
woodwinds_state = gr.State([0] * (config.PITCHES_MELODIC * config.INPUT_STEPS))
|
| 237 |
+
brass_state = gr.State([0] * (config.PITCHES_MELODIC * config.INPUT_STEPS))
|
| 238 |
+
perc_state = gr.State([0] * (config.PITCHES_PERC * config.INPUT_STEPS))
|
| 239 |
+
|
| 240 |
+
with gr.Row():
|
| 241 |
+
# --- LEFT PANEL COLUMN: INSTRUMENT MATRIX SLOTS ---
|
| 242 |
+
with gr.Column(scale=4):
|
| 243 |
+
|
| 244 |
+
# --- STRINGS DECK ---
|
| 245 |
+
with gr.Column(elem_classes="instrument-container"):
|
| 246 |
+
gr.HTML("<div style='color:#63b3ed; font-weight:bold; font-size:13px; margin-bottom:6px;'>🎻 Strings Family (All 3 Bars)</div>")
|
| 247 |
+
s_buttons = []
|
| 248 |
+
for r in reversed(range(config.PITCHES_MELODIC)):
|
| 249 |
+
with gr.Row():
|
| 250 |
+
gr.HTML(f"<div class='grid-label'>{_MELODIC_NOTE_NAMES[r]}</div>", scale=1)
|
| 251 |
+
for c in range(config.INPUT_STEPS):
|
| 252 |
+
cls = "cell-off"
|
| 253 |
+
if c > 0 and c % config.STEPS_PER_BAR == 0:
|
| 254 |
+
cls += " bar-line-left"
|
| 255 |
+
btn = gr.Button("", elem_classes=["matrix-cell", cls], min_width=15, scale=2)
|
| 256 |
+
s_buttons.append((btn, r, c))
|
| 257 |
+
s_clear = gr.Button("🗑️ Clear Strings", size="sm", elem_classes=["instrument-margin-button"])
|
| 258 |
+
|
| 259 |
+
# --- WOODWINDS DECK ---
|
| 260 |
+
with gr.Column(elem_classes="instrument-container instrument-margin"):
|
| 261 |
+
gr.HTML("<div style='color:#6ed29b; font-weight:bold; font-size:13px; margin-bottom:6px;'>🪵 Woodwinds Family (All 3 Bars)</div>")
|
| 262 |
+
w_buttons = []
|
| 263 |
+
for r in reversed(range(config.PITCHES_MELODIC)):
|
| 264 |
+
with gr.Row():
|
| 265 |
+
gr.HTML(f"<div class='grid-label'>{_MELODIC_NOTE_NAMES[r]}</div>", scale=1)
|
| 266 |
+
for c in range(config.INPUT_STEPS):
|
| 267 |
+
cls = "cell-off"
|
| 268 |
+
if c > 0 and c % config.STEPS_PER_BAR == 0:
|
| 269 |
+
cls += " bar-line-left"
|
| 270 |
+
btn = gr.Button("", elem_classes=["matrix-cell", cls], min_width=15, scale=2)
|
| 271 |
+
w_buttons.append((btn, r, c))
|
| 272 |
+
w_clear = gr.Button("🗑️ Clear Woodwinds", size="sm", elem_classes=["instrument-margin-button"])
|
| 273 |
+
|
| 274 |
+
# --- BRASS DECK ---
|
| 275 |
+
with gr.Column(elem_classes="instrument-container instrument-margin"):
|
| 276 |
+
gr.HTML("<div style='color:#fbbf24; font-weight:bold; font-size:13px; margin-bottom:6px;'>🎺 Brass Family (All 3 Bars)</div>")
|
| 277 |
+
b_buttons = []
|
| 278 |
+
for r in reversed(range(config.PITCHES_MELODIC)):
|
| 279 |
+
with gr.Row():
|
| 280 |
+
gr.HTML(f"<div class='grid-label'>{_MELODIC_NOTE_NAMES[r]}</div>", scale=1)
|
| 281 |
+
for c in range(config.INPUT_STEPS):
|
| 282 |
+
cls = "cell-off"
|
| 283 |
+
if c > 0 and c % config.STEPS_PER_BAR == 0:
|
| 284 |
+
cls += " bar-line-left"
|
| 285 |
+
btn = gr.Button("", elem_classes=["matrix-cell", cls], min_width=15, scale=2)
|
| 286 |
+
b_buttons.append((btn, r, c))
|
| 287 |
+
b_clear = gr.Button("🗑️ Clear Brass", size="sm", elem_classes=["instrument-margin-button"])
|
| 288 |
+
|
| 289 |
+
# --- PERCUSSION DECK ---
|
| 290 |
+
with gr.Column(elem_classes="instrument-container instrument-margin"):
|
| 291 |
+
gr.HTML("<div style='color:#fc814a; font-weight:bold; font-size:13px; margin-bottom:6px;'>🥁 Percussion Grid (All 3 Bars)</div>")
|
| 292 |
+
p_buttons = []
|
| 293 |
+
for r in reversed(range(config.PITCHES_PERC)):
|
| 294 |
+
with gr.Row():
|
| 295 |
+
gr.HTML(f"<div class='grid-label'>{_PERC_NOTE_NAMES[r]}</div>", scale=1)
|
| 296 |
+
for c in range(config.INPUT_STEPS):
|
| 297 |
+
cls = "cell-off"
|
| 298 |
+
if c > 0 and c % config.STEPS_PER_BAR == 0:
|
| 299 |
+
cls += " bar-line-left"
|
| 300 |
+
btn = gr.Button("", elem_classes=["matrix-cell", cls], min_width=15, scale=2)
|
| 301 |
+
p_buttons.append((btn, r, c))
|
| 302 |
+
p_clear = gr.Button("🗑️ Clear Percussion", size="sm", elem_classes=["instrument-margin-button"])
|
| 303 |
+
|
| 304 |
+
# --- RIGHT PANEL COLUMN: ACTIVE SELECTION INSPECTION HUD ---
|
| 305 |
+
with gr.Column(scale=1):
|
| 306 |
+
with gr.Column(elem_classes="hud-container"):
|
| 307 |
+
hud_display = gr.Markdown("### 🎛️ Selection HUD\n*Click any block in the orchestration grids to inspect active coordinate parameters.*")
|
| 308 |
+
|
| 309 |
+
# ==============================================================================
|
| 310 |
+
# Core Event Bindings Engine Loops
|
| 311 |
+
# ==============================================================================
|
| 312 |
+
|
| 313 |
+
# Strings Action Pipeline Mapping
|
| 314 |
+
for btn, r, c in s_buttons:
|
| 315 |
+
btn.click(
|
| 316 |
+
fn=lambda grid, row=r, col=c: toggle_cell_state(grid, row, col, config.PITCHES_MELODIC, "strings"),
|
| 317 |
+
inputs=[strings_state], outputs=[strings_state, btn, hud_display]
|
| 318 |
+
).then(fn=None, inputs=[gr.State(r), gr.State(0)], outputs=None, js="window.jsAudioEngine.playNote")
|
| 319 |
+
s_clear.click(fn=lambda: clear_family_grid(config.PITCHES_MELODIC), inputs=None, outputs=[strings_state] + [b for b,_,_ in s_buttons])
|
| 320 |
+
|
| 321 |
+
# Woodwinds Action Pipeline Mapping
|
| 322 |
+
for btn, r, c in w_buttons:
|
| 323 |
+
btn.click(
|
| 324 |
+
fn=lambda grid, row=r, col=c: toggle_cell_state(grid, row, col, config.PITCHES_MELODIC, "woodwinds"),
|
| 325 |
+
inputs=[woodwinds_state], outputs=[woodwinds_state, btn, hud_display]
|
| 326 |
+
).then(fn=None, inputs=[gr.State(r), gr.State(0)], outputs=None, js="window.jsAudioEngine.playNote")
|
| 327 |
+
w_clear.click(fn=lambda: clear_family_grid(config.PITCHES_MELODIC), inputs=None, outputs=[woodwinds_state] + [b for b,_,_ in w_buttons])
|
| 328 |
+
|
| 329 |
+
# Brass Action Pipeline Mapping
|
| 330 |
+
for btn, r, c in b_buttons:
|
| 331 |
+
btn.click(
|
| 332 |
+
fn=lambda grid, row=r, col=c: toggle_cell_state(grid, row, col, config.PITCHES_MELODIC, "brass"),
|
| 333 |
+
inputs=[brass_state], outputs=[brass_state, btn, hud_display]
|
| 334 |
+
).then(fn=None, inputs=[gr.State(r), gr.State(0)], outputs=None, js="window.jsAudioEngine.playNote")
|
| 335 |
+
b_clear.click(fn=lambda: clear_family_grid(config.PITCHES_MELODIC), inputs=None, outputs=[brass_state] + [b for b,_,_ in b_buttons])
|
| 336 |
+
|
| 337 |
+
# Percussion Action Pipeline Mapping
|
| 338 |
+
for btn, r, c in p_buttons:
|
| 339 |
+
btn.click(
|
| 340 |
+
fn=lambda grid, row=r, col=c: toggle_cell_state(grid, row, col, config.PITCHES_PERC, "perc"),
|
| 341 |
+
inputs=[perc_state], outputs=[perc_state, btn, hud_display]
|
| 342 |
+
).then(fn=None, inputs=[gr.State(r), gr.State(1)], outputs=None, js="window.jsAudioEngine.playNote")
|
| 343 |
+
p_clear.click(fn=lambda: clear_family_grid(config.PITCHES_PERC), inputs=None, outputs=[perc_state] + [b for b,_,_ in p_buttons])
|
| 344 |
+
|
| 345 |
+
# ==============================================================================
|
| 346 |
+
# Global Parameters & Output Management
|
| 347 |
+
# ==============================================================================
|
| 348 |
+
gr.HTML("<div style='margin-top:20px; border-bottom:1px solid rgba(255,255,255,0.08);'></div>")
|
| 349 |
+
|
| 350 |
+
with gr.Row():
|
| 351 |
+
with gr.Column(scale=3):
|
| 352 |
+
with gr.Row():
|
| 353 |
+
target_minutes = gr.Slider(minimum=1.0, maximum=5.0, value=2.0, step=0.5, label="🕐 Target Length (Minutes)")
|
| 354 |
+
temperature = gr.Slider(minimum=0.7, maximum=1.3, value=1.0, step=0.05, label="🌡️ Creativity Temperature")
|
| 355 |
+
with gr.Row():
|
| 356 |
+
top_k = gr.Slider(minimum=2, maximum=12, value=6, step=1, label="🎲 Top-K Distribution Filter")
|
| 357 |
+
primary_family = gr.Dropdown(choices=["strings", "woodwinds", "brass"], value="strings", label="🎼 Primary Orchestration Lead")
|
| 358 |
+
|
| 359 |
+
generate_button = gr.Button("🎵 Execute Transformer Music Generation Pipeline", variant="primary", size="lg")
|
| 360 |
+
|
| 361 |
+
with gr.Column(scale=2):
|
| 362 |
+
audio_output = gr.Audio(label="🔊 Fluidsynth Generated Audio Player (Listen Live)", type="filepath")
|
| 363 |
+
midi_output = gr.File(label="⬇ Download Output Standard MIDI File (.mid)")
|
| 364 |
+
|
| 365 |
+
generate_button.click(
|
| 366 |
+
fn=generate,
|
| 367 |
+
inputs=[
|
| 368 |
+
strings_state, woodwinds_state, brass_state, perc_state,
|
| 369 |
+
target_minutes, temperature, top_k, primary_family
|
| 370 |
+
],
|
| 371 |
+
outputs=[midi_output, audio_output]
|
| 372 |
+
)
|
| 373 |
+
|
| 374 |
+
if __name__ == "__main__":
|
| 375 |
+
demo.launch(css=CSS_ESTHETIC)
|
config.py
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Global configuration for the classical music grid Transformer.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
# =========================
|
| 6 |
+
# Grid and tempo constants
|
| 7 |
+
# =========================
|
| 8 |
+
|
| 9 |
+
# Time grid
|
| 10 |
+
STEPS_PER_BAR: int = 8
|
| 11 |
+
INPUT_BARS: int = 3
|
| 12 |
+
INPUT_STEPS: int = INPUT_BARS * STEPS_PER_BAR # 24
|
| 13 |
+
|
| 14 |
+
# Pitch grid
|
| 15 |
+
PITCHES_MELODIC: int = 12 # one octave: 12 pitch classes
|
| 16 |
+
PITCHES_PERC: int = 3 # e.g., Kick, Snare, Hi-hat
|
| 17 |
+
|
| 18 |
+
# Instrument families (for clarity / future use)
|
| 19 |
+
N_FAMILIES: int = 4 # Strings, Woodwinds, Brass, Percussion
|
| 20 |
+
|
| 21 |
+
# Tempo and meter
|
| 22 |
+
TEMPO_BPM: int = 120
|
| 23 |
+
BEATS_PER_BAR: int = 4 # assuming 4/4 time
|
| 24 |
+
|
| 25 |
+
# Derived timing values
|
| 26 |
+
SECONDS_PER_BEAT: float = 60.0 / TEMPO_BPM
|
| 27 |
+
SECONDS_PER_BAR: float = BEATS_PER_BAR * SECONDS_PER_BEAT
|
| 28 |
+
SECONDS_PER_STEP: float = SECONDS_PER_BAR / STEPS_PER_BAR
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
# =========================
|
| 32 |
+
# Model hyperparameters
|
| 33 |
+
# =========================
|
| 34 |
+
|
| 35 |
+
# Transformer dimensions
|
| 36 |
+
D_MODEL: int = 256 # hidden size
|
| 37 |
+
N_LAYERS: int = 3 # number of Transformer encoder layers (2–4 is reasonable)
|
| 38 |
+
N_HEADS: int = 4 # attention heads
|
| 39 |
+
FFN_DIM: int = 512 # feed-forward inner dimension
|
| 40 |
+
DROPOUT: float = 0.1 # dropout rate
|
| 41 |
+
|
| 42 |
+
# Input feature size per time step:
|
| 43 |
+
# Strings (12) + Woodwinds (12) + Brass (12) + Percussion (3) = 39
|
| 44 |
+
INPUT_FEATURES: int = (3 * PITCHES_MELODIC) + PITCHES_PERC # 39
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
# =========================
|
| 48 |
+
# Sampling and constraints
|
| 49 |
+
# =========================
|
| 50 |
+
|
| 51 |
+
# Bar-level similarity threshold (0–1); above this, bar is considered "too similar"
|
| 52 |
+
BAR_SIMILARITY_THRESHOLD: float = 0.9
|
| 53 |
+
|
| 54 |
+
# Maximum number of resampling attempts per bar before accepting anyway
|
| 55 |
+
MAX_RESAMPLE_ATTEMPTS_PER_BAR: int = 5
|
| 56 |
+
|
| 57 |
+
# Minimum number of instrument families that must be active (have ≥1 note) per bar
|
| 58 |
+
MIN_ACTIVE_FAMILIES_PER_BAR: int = 2
|
data_preprocessing.py
ADDED
|
@@ -0,0 +1,387 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Data preprocessing for grid-based multi-instrument music generation.
|
| 3 |
+
|
| 4 |
+
- Loads MIDI files from a directory (e.g., MusicNet).
|
| 5 |
+
- Maps raw MIDI instruments to 4 families: Strings, Woodwinds, Brass, Percussion.
|
| 6 |
+
- Quantizes notes to a fixed grid at 120 BPM, with 8 steps per bar.
|
| 7 |
+
- Limits melodic pitches to one octave (C4–B4) and maps to indices 0–11.
|
| 8 |
+
- Maps percussion to Kick/Snare/Hi-hat rows.
|
| 9 |
+
- Builds sequences of length SEQ_LEN (in steps), with each time step a 39-dim vector:
|
| 10 |
+
[12 strings, 12 woodwinds, 12 brass, 3 perc]
|
| 11 |
+
- Saves X (inputs) and Y (targets) arrays to disk.
|
| 12 |
+
"""
|
| 13 |
+
|
| 14 |
+
from typing import List, Tuple, Dict
|
| 15 |
+
|
| 16 |
+
import glob
|
| 17 |
+
import os
|
| 18 |
+
|
| 19 |
+
import numpy as np
|
| 20 |
+
import pretty_midi
|
| 21 |
+
|
| 22 |
+
import config
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
# One octave: C4 (60) to B4 (71)
|
| 26 |
+
MELODIC_BASE_PITCH: int = 60
|
| 27 |
+
MELODIC_MAX_PITCH: int = MELODIC_BASE_PITCH + config.PITCHES_MELODIC - 1 # 71
|
| 28 |
+
|
| 29 |
+
# Percussion pitches
|
| 30 |
+
PERC_KICK_PITCH: int = 36 # Bass Drum 1
|
| 31 |
+
PERC_SNARE_PITCH: int = 38 # Acoustic Snare
|
| 32 |
+
PERC_HIHAT_PITCH: int = 42 # Closed Hi-Hat
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def load_midi_paths_from_dir(midi_root: str, pattern: str = "**/*.mid*") -> List[str]:
|
| 36 |
+
"""
|
| 37 |
+
Recursively collect MIDI file paths from a root directory.
|
| 38 |
+
|
| 39 |
+
Args:
|
| 40 |
+
midi_root: Root directory containing MIDI files.
|
| 41 |
+
pattern: Glob pattern relative to root.
|
| 42 |
+
|
| 43 |
+
Returns:
|
| 44 |
+
List of file paths.
|
| 45 |
+
"""
|
| 46 |
+
search_path = os.path.join(midi_root, pattern)
|
| 47 |
+
filepaths = glob.glob(search_path, recursive=True)
|
| 48 |
+
return filepaths
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
# ============
|
| 52 |
+
# Instrument family mapping
|
| 53 |
+
# ============
|
| 54 |
+
|
| 55 |
+
def get_family_for_program(program: int, is_drum: bool) -> str:
|
| 56 |
+
"""
|
| 57 |
+
Map a MIDI program number to one of the four families.
|
| 58 |
+
|
| 59 |
+
Args:
|
| 60 |
+
program: MIDI program number (0–127).
|
| 61 |
+
is_drum: Whether this is a drum channel.
|
| 62 |
+
|
| 63 |
+
Returns:
|
| 64 |
+
One of: "strings", "woodwinds", "brass", "perc".
|
| 65 |
+
"""
|
| 66 |
+
if is_drum:
|
| 67 |
+
return "perc"
|
| 68 |
+
|
| 69 |
+
# Very simple heuristic based on General MIDI program ranges.
|
| 70 |
+
# You can refine this mapping later as needed.
|
| 71 |
+
|
| 72 |
+
# Strings: Violins, Violas, Celli, etc. (40–47)
|
| 73 |
+
if 40 <= program <= 47:
|
| 74 |
+
return "strings"
|
| 75 |
+
|
| 76 |
+
# Woodwinds/Flutes/Clarinets: Flute (73), Clarinet (71), etc.
|
| 77 |
+
if 64 <= program <= 79:
|
| 78 |
+
return "woodwinds"
|
| 79 |
+
|
| 80 |
+
# Brass: Trumpet, Trombone, etc. (56–63)
|
| 81 |
+
if 56 <= program <= 63:
|
| 82 |
+
return "brass"
|
| 83 |
+
|
| 84 |
+
# Fallback: treat others as strings
|
| 85 |
+
return "strings"
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
# ============
|
| 89 |
+
# Quantization and grid building
|
| 90 |
+
# ============
|
| 91 |
+
|
| 92 |
+
def time_to_step_index(t: float) -> int:
|
| 93 |
+
"""
|
| 94 |
+
Convert time (seconds) to global step index on the quantized grid.
|
| 95 |
+
|
| 96 |
+
Using config.SECONDS_PER_STEP.
|
| 97 |
+
|
| 98 |
+
Args:
|
| 99 |
+
t: Time in seconds.
|
| 100 |
+
|
| 101 |
+
Returns:
|
| 102 |
+
Global step index (int).
|
| 103 |
+
"""
|
| 104 |
+
return int(np.floor(t / config.SECONDS_PER_STEP))
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def clip_pitch_to_melodic_range(pitch: int) -> int:
|
| 108 |
+
"""
|
| 109 |
+
Map MIDI pitch to [MELODIC_BASE_PITCH, MELODIC_MAX_PITCH] if possible.
|
| 110 |
+
|
| 111 |
+
Returns:
|
| 112 |
+
-1 if outside range (should be ignored), or
|
| 113 |
+
pitch_idx in [0, PITCHES_MELODIC-1] otherwise.
|
| 114 |
+
"""
|
| 115 |
+
if pitch < MELODIC_BASE_PITCH or pitch > MELODIC_MAX_PITCH:
|
| 116 |
+
return -1
|
| 117 |
+
return pitch - MELODIC_BASE_PITCH
|
| 118 |
+
|
| 119 |
+
|
| 120 |
+
def instrument_to_family_grids(
|
| 121 |
+
pm: pretty_midi.PrettyMIDI,
|
| 122 |
+
max_bars: int = None,
|
| 123 |
+
) -> Dict[str, np.ndarray]:
|
| 124 |
+
"""
|
| 125 |
+
Convert a PrettyMIDI object into per-family grids.
|
| 126 |
+
|
| 127 |
+
Grids have shape:
|
| 128 |
+
strings: [bars, STEPS_PER_BAR, 12]
|
| 129 |
+
woodwinds: [bars, STEPS_PER_BAR, 12]
|
| 130 |
+
brass: [bars, STEPS_PER_BAR, 12]
|
| 131 |
+
perc: [bars, STEPS_PER_BAR, 3]
|
| 132 |
+
|
| 133 |
+
Args:
|
| 134 |
+
pm: PrettyMIDI object.
|
| 135 |
+
max_bars: Optional cap on number of bars.
|
| 136 |
+
|
| 137 |
+
Returns:
|
| 138 |
+
Dict of family → grid.
|
| 139 |
+
"""
|
| 140 |
+
total_seconds = pm.get_end_time()
|
| 141 |
+
bar_duration = config.SECONDS_PER_BAR
|
| 142 |
+
total_bars = int(np.ceil(total_seconds / bar_duration))
|
| 143 |
+
|
| 144 |
+
if max_bars is not None:
|
| 145 |
+
total_bars = min(total_bars, max_bars)
|
| 146 |
+
|
| 147 |
+
T = total_bars * config.STEPS_PER_BAR
|
| 148 |
+
|
| 149 |
+
strings = np.zeros((total_bars, config.STEPS_PER_BAR, config.PITCHES_MELODIC), dtype=np.float32)
|
| 150 |
+
woodwinds = np.zeros_like(strings)
|
| 151 |
+
brass = np.zeros_like(strings)
|
| 152 |
+
perc = np.zeros((total_bars, config.STEPS_PER_BAR, config.PITCHES_PERC), dtype=np.float32)
|
| 153 |
+
|
| 154 |
+
# Helper to convert time to (bar_idx, step_idx)
|
| 155 |
+
def t_to_bar_step(t: float) -> Tuple[int, int]:
|
| 156 |
+
step_idx_global = time_to_step_index(t)
|
| 157 |
+
bar_idx = step_idx_global // config.STEPS_PER_BAR
|
| 158 |
+
step_idx = step_idx_global % config.STEPS_PER_BAR
|
| 159 |
+
if bar_idx >= total_bars:
|
| 160 |
+
bar_idx = total_bars - 1
|
| 161 |
+
return bar_idx, step_idx
|
| 162 |
+
|
| 163 |
+
for inst in pm.instruments:
|
| 164 |
+
is_drum = inst.is_drum
|
| 165 |
+
family = get_family_for_program(inst.program, is_drum=is_drum)
|
| 166 |
+
|
| 167 |
+
if family == "perc":
|
| 168 |
+
for note in inst.notes:
|
| 169 |
+
b, s = t_to_bar_step(note.start)
|
| 170 |
+
if note.pitch == PERC_KICK_PITCH:
|
| 171 |
+
k = 0
|
| 172 |
+
elif note.pitch == PERC_SNARE_PITCH:
|
| 173 |
+
k = 1
|
| 174 |
+
elif note.pitch == PERC_HIHAT_PITCH:
|
| 175 |
+
k = 2
|
| 176 |
+
else:
|
| 177 |
+
continue
|
| 178 |
+
perc[b, s, k] = 1.0
|
| 179 |
+
else:
|
| 180 |
+
# Select grid
|
| 181 |
+
if family == "strings":
|
| 182 |
+
grid = strings
|
| 183 |
+
elif family == "woodwinds":
|
| 184 |
+
grid = woodwinds
|
| 185 |
+
elif family == "brass":
|
| 186 |
+
grid = brass
|
| 187 |
+
else:
|
| 188 |
+
grid = strings # fallback
|
| 189 |
+
|
| 190 |
+
for note in inst.notes:
|
| 191 |
+
b, s = t_to_bar_step(note.start)
|
| 192 |
+
pitch_idx = clip_pitch_to_melodic_range(note.pitch)
|
| 193 |
+
if pitch_idx == -1:
|
| 194 |
+
continue
|
| 195 |
+
grid[b, s, pitch_idx] = 1.0
|
| 196 |
+
|
| 197 |
+
return {
|
| 198 |
+
"strings": strings,
|
| 199 |
+
"woodwinds": woodwinds,
|
| 200 |
+
"brass": brass,
|
| 201 |
+
"perc": perc,
|
| 202 |
+
}
|
| 203 |
+
|
| 204 |
+
|
| 205 |
+
def grids_to_feature_sequences(
|
| 206 |
+
family_grids: Dict[str, np.ndarray],
|
| 207 |
+
) -> np.ndarray:
|
| 208 |
+
"""
|
| 209 |
+
Convert per-family grids into a [T, 39] feature sequence.
|
| 210 |
+
|
| 211 |
+
Families:
|
| 212 |
+
strings: [bars, STEPS_PER_BAR, 12]
|
| 213 |
+
woodwinds: [bars, STEPS_PER_BAR, 12]
|
| 214 |
+
brass: [bars, STEPS_PER_BAR, 12]
|
| 215 |
+
perc: [bars, STEPS_PER_BAR, 3]
|
| 216 |
+
|
| 217 |
+
Returns:
|
| 218 |
+
seq: [T, 39], where T = bars * STEPS_PER_BAR.
|
| 219 |
+
"""
|
| 220 |
+
strings = family_grids["strings"]
|
| 221 |
+
woodwinds = family_grids["woodwinds"]
|
| 222 |
+
brass = family_grids["brass"]
|
| 223 |
+
perc = family_grids["perc"]
|
| 224 |
+
|
| 225 |
+
bars, steps_per_bar, n_strings = strings.shape
|
| 226 |
+
T = bars * steps_per_bar
|
| 227 |
+
|
| 228 |
+
# Reshape to [T, pitches]
|
| 229 |
+
strings_flat = strings.reshape(T, n_strings)
|
| 230 |
+
woodwinds_flat = woodwinds.reshape(T, woodwinds.shape[-1])
|
| 231 |
+
brass_flat = brass.reshape(T, brass.shape[-1])
|
| 232 |
+
perc_flat = perc.reshape(T, perc.shape[-1])
|
| 233 |
+
|
| 234 |
+
# Concatenate along feature dimension: [T, 12+12+12+3] = [T, 39]
|
| 235 |
+
seq = np.concatenate(
|
| 236 |
+
[strings_flat, woodwinds_flat, brass_flat, perc_flat],
|
| 237 |
+
axis=-1,
|
| 238 |
+
)
|
| 239 |
+
|
| 240 |
+
return seq
|
| 241 |
+
|
| 242 |
+
|
| 243 |
+
def build_sequences_for_file(
|
| 244 |
+
midi_path: str,
|
| 245 |
+
seq_len_steps: int,
|
| 246 |
+
max_bars: int = None,
|
| 247 |
+
) -> Tuple[np.ndarray, np.ndarray]:
|
| 248 |
+
"""
|
| 249 |
+
Build (X, Y) sequences for a single MIDI file.
|
| 250 |
+
|
| 251 |
+
- Convert MIDI → family grids → [T, 39] sequence.
|
| 252 |
+
- Build sliding window sequences of length seq_len_steps:
|
| 253 |
+
X: first seq_len_steps-1 steps
|
| 254 |
+
Y: next step (for next-step prediction)
|
| 255 |
+
For convenience, we pack them as:
|
| 256 |
+
X: [N, seq_len_steps, 39]
|
| 257 |
+
Y: [N, seq_len_steps, 39] (shifted by 1).
|
| 258 |
+
|
| 259 |
+
Args:
|
| 260 |
+
midi_path: Path to MIDI file.
|
| 261 |
+
seq_len_steps: Sequence length in grid steps (e.g., 64, 128).
|
| 262 |
+
max_bars: Optional max bars to consider from this file.
|
| 263 |
+
|
| 264 |
+
Returns:
|
| 265 |
+
X: [N, seq_len_steps, 39]
|
| 266 |
+
Y: [N, seq_len_steps, 39]
|
| 267 |
+
"""
|
| 268 |
+
try:
|
| 269 |
+
pm = pretty_midi.PrettyMIDI(midi_path)
|
| 270 |
+
except Exception:
|
| 271 |
+
# Skip unreadable files
|
| 272 |
+
return np.empty((0, seq_len_steps, config.INPUT_FEATURES), dtype=np.float32), \
|
| 273 |
+
np.empty((0, seq_len_steps, config.INPUT_FEATURES), dtype=np.float32)
|
| 274 |
+
|
| 275 |
+
family_grids = instrument_to_family_grids(pm, max_bars=max_bars)
|
| 276 |
+
seq = grids_to_feature_sequences(family_grids) # [T, 39]
|
| 277 |
+
|
| 278 |
+
T = seq.shape[0]
|
| 279 |
+
if T <= seq_len_steps:
|
| 280 |
+
return np.empty((0, seq_len_steps, config.INPUT_FEATURES), dtype=np.float32), \
|
| 281 |
+
np.empty((0, seq_len_steps, config.INPUT_FEATURES), dtype=np.float32)
|
| 282 |
+
|
| 283 |
+
# Sliding window
|
| 284 |
+
X_list = []
|
| 285 |
+
Y_list = []
|
| 286 |
+
# We'll create sequences of length seq_len_steps,
|
| 287 |
+
# with Y being the same sequence shifted by 1 step.
|
| 288 |
+
for start in range(0, T - seq_len_steps):
|
| 289 |
+
end = start + seq_len_steps
|
| 290 |
+
seq_window = seq[start:end] # [seq_len_steps, 39]
|
| 291 |
+
seq_window_next = seq[start + 1:end + 1] # shifted by 1 step
|
| 292 |
+
|
| 293 |
+
X_list.append(seq_window)
|
| 294 |
+
Y_list.append(seq_window_next)
|
| 295 |
+
|
| 296 |
+
X = np.stack(X_list, axis=0) # [N, seq_len_steps, 39]
|
| 297 |
+
Y = np.stack(Y_list, axis=0) # [N, seq_len_steps, 39]
|
| 298 |
+
|
| 299 |
+
return X.astype(np.float32), Y.astype(np.float32)
|
| 300 |
+
|
| 301 |
+
|
| 302 |
+
def build_dataset_from_dir(
|
| 303 |
+
midi_root: str,
|
| 304 |
+
seq_len_bars: int,
|
| 305 |
+
max_files: int = None,
|
| 306 |
+
max_bars_per_file: int = None,
|
| 307 |
+
pattern: str = "**/*.mid*",
|
| 308 |
+
) -> Tuple[np.ndarray, np.ndarray]:
|
| 309 |
+
"""
|
| 310 |
+
Build a full dataset (X, Y) from all MIDI files in a directory.
|
| 311 |
+
|
| 312 |
+
Args:
|
| 313 |
+
midi_root: Root directory containing MIDI files (e.g., MusicNet midis).
|
| 314 |
+
seq_len_bars: Sequence length in bars.
|
| 315 |
+
max_files: Optional max number of files to use.
|
| 316 |
+
max_bars_per_file: Optional cap on bars per file for preprocessing.
|
| 317 |
+
pattern: Glob pattern for finding midis under midi_root.
|
| 318 |
+
|
| 319 |
+
Returns:
|
| 320 |
+
X_all: [N_total, seq_len_steps, 39]
|
| 321 |
+
Y_all: [N_total, seq_len_steps, 39]
|
| 322 |
+
"""
|
| 323 |
+
midi_paths = load_midi_paths_from_dir(midi_root, pattern=pattern)
|
| 324 |
+
|
| 325 |
+
if max_files is not None:
|
| 326 |
+
midi_paths = midi_paths[:max_files]
|
| 327 |
+
|
| 328 |
+
seq_len_steps = seq_len_bars * config.STEPS_PER_BAR
|
| 329 |
+
|
| 330 |
+
X_all_list = []
|
| 331 |
+
Y_all_list = []
|
| 332 |
+
|
| 333 |
+
for path in midi_paths:
|
| 334 |
+
X, Y = build_sequences_for_file(
|
| 335 |
+
midi_path=path,
|
| 336 |
+
seq_len_steps=seq_len_steps,
|
| 337 |
+
max_bars=max_bars_per_file,
|
| 338 |
+
)
|
| 339 |
+
if X.shape[0] == 0:
|
| 340 |
+
continue
|
| 341 |
+
X_all_list.append(X)
|
| 342 |
+
Y_all_list.append(Y)
|
| 343 |
+
|
| 344 |
+
if not X_all_list:
|
| 345 |
+
# No data
|
| 346 |
+
return np.empty((0, seq_len_steps, config.INPUT_FEATURES), dtype=np.float32), \
|
| 347 |
+
np.empty((0, seq_len_steps, config.INPUT_FEATURES), dtype=np.float32)
|
| 348 |
+
|
| 349 |
+
X_all = np.concatenate(X_all_list, axis=0)
|
| 350 |
+
Y_all = np.concatenate(Y_all_list, axis=0)
|
| 351 |
+
|
| 352 |
+
return X_all, Y_all
|
| 353 |
+
|
| 354 |
+
|
| 355 |
+
def save_dataset_npz(
|
| 356 |
+
X: np.ndarray,
|
| 357 |
+
Y: np.ndarray,
|
| 358 |
+
out_path: str,
|
| 359 |
+
) -> None:
|
| 360 |
+
"""
|
| 361 |
+
Save (X, Y) arrays to a .npz file.
|
| 362 |
+
|
| 363 |
+
Args:
|
| 364 |
+
X: [N, seq_len_steps, 39]
|
| 365 |
+
Y: [N, seq_len_steps, 39]
|
| 366 |
+
out_path: path to .npz file.
|
| 367 |
+
"""
|
| 368 |
+
np.savez_compressed(out_path, X=X, Y=Y)
|
| 369 |
+
|
| 370 |
+
|
| 371 |
+
def save_dataset_pt(
|
| 372 |
+
X: np.ndarray,
|
| 373 |
+
Y: np.ndarray,
|
| 374 |
+
out_path: str,
|
| 375 |
+
) -> None:
|
| 376 |
+
"""
|
| 377 |
+
Save (X, Y) arrays to a .pt file for PyTorch.
|
| 378 |
+
|
| 379 |
+
Args:
|
| 380 |
+
X: [N, seq_len_steps, 39]
|
| 381 |
+
Y: [N, seq_len_steps, 39]
|
| 382 |
+
out_path: path to .pt file.
|
| 383 |
+
"""
|
| 384 |
+
import torch
|
| 385 |
+
|
| 386 |
+
data = {"X": torch.from_numpy(X), "Y": torch.from_numpy(Y)}
|
| 387 |
+
torch.save(data, out_path)
|
inference.py
ADDED
|
@@ -0,0 +1,485 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Inference utilities: load trained model and generate music from seed grids.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
from typing import Dict, Tuple
|
| 6 |
+
|
| 7 |
+
import math
|
| 8 |
+
import os
|
| 9 |
+
|
| 10 |
+
import numpy as np
|
| 11 |
+
import torch
|
| 12 |
+
|
| 13 |
+
import config
|
| 14 |
+
from model import build_model
|
| 15 |
+
from utils import sampling_utils, midi_utils
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
# =========================
|
| 19 |
+
# Model loading
|
| 20 |
+
# =========================
|
| 21 |
+
|
| 22 |
+
_model: torch.nn.Module = None
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def load_model(
|
| 26 |
+
weights_path: str = "models/transformer_music_best.pt",
|
| 27 |
+
device: str = "cpu",
|
| 28 |
+
) -> torch.nn.Module:
|
| 29 |
+
"""
|
| 30 |
+
Load the trained MusicTransformer model for inference.
|
| 31 |
+
|
| 32 |
+
Args:
|
| 33 |
+
weights_path: Path to the .pt weights file.
|
| 34 |
+
device: Device to map the model to ("cpu" for HF Spaces).
|
| 35 |
+
|
| 36 |
+
Returns:
|
| 37 |
+
model: Loaded MusicTransformer in eval mode.
|
| 38 |
+
"""
|
| 39 |
+
global _model
|
| 40 |
+
|
| 41 |
+
if _model is not None:
|
| 42 |
+
return _model
|
| 43 |
+
|
| 44 |
+
model = build_model(config)
|
| 45 |
+
state_dict = torch.load(weights_path, map_location=device)
|
| 46 |
+
model.load_state_dict(state_dict)
|
| 47 |
+
model.to(device)
|
| 48 |
+
model.eval()
|
| 49 |
+
|
| 50 |
+
_model = model
|
| 51 |
+
return _model
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
# =========================
|
| 55 |
+
# Seed conversion
|
| 56 |
+
# =========================
|
| 57 |
+
|
| 58 |
+
def seed_grids_to_sequence(
|
| 59 |
+
strings_seed: np.ndarray,
|
| 60 |
+
woodwinds_seed: np.ndarray,
|
| 61 |
+
brass_seed: np.ndarray,
|
| 62 |
+
perc_seed: np.ndarray,
|
| 63 |
+
) -> np.ndarray:
|
| 64 |
+
"""
|
| 65 |
+
Convert 4 seed grids into a [T_seed, 39] feature sequence.
|
| 66 |
+
|
| 67 |
+
Seed shapes:
|
| 68 |
+
strings_seed: [INPUT_BARS, STEPS_PER_BAR, 12]
|
| 69 |
+
woodwinds_seed: [INPUT_BARS, STEPS_PER_BAR, 12]
|
| 70 |
+
brass_seed: [INPUT_BARS, STEPS_PER_BAR, 12]
|
| 71 |
+
perc_seed: [INPUT_BARS, STEPS_PER_BAR, 3]
|
| 72 |
+
|
| 73 |
+
Returns:
|
| 74 |
+
seq: [T_seed, 39] float32
|
| 75 |
+
"""
|
| 76 |
+
# Basic checks
|
| 77 |
+
bars, steps, p_s = strings_seed.shape
|
| 78 |
+
assert bars == config.INPUT_BARS
|
| 79 |
+
assert steps == config.STEPS_PER_BAR
|
| 80 |
+
assert p_s == config.PITCHES_MELODIC
|
| 81 |
+
|
| 82 |
+
# Flatten bars × steps → T
|
| 83 |
+
T_seed = bars * steps
|
| 84 |
+
|
| 85 |
+
strings_flat = strings_seed.reshape(T_seed, config.PITCHES_MELODIC)
|
| 86 |
+
woodwinds_flat = woodwinds_seed.reshape(T_seed, config.PITCHES_MELODIC)
|
| 87 |
+
brass_flat = brass_seed.reshape(T_seed, config.PITCHES_MELODIC)
|
| 88 |
+
perc_flat = perc_seed.reshape(T_seed, config.PITCHES_PERC)
|
| 89 |
+
|
| 90 |
+
seq = np.concatenate(
|
| 91 |
+
[strings_flat, woodwinds_flat, brass_flat, perc_flat],
|
| 92 |
+
axis=-1,
|
| 93 |
+
).astype(np.float32) # [T_seed, 39]
|
| 94 |
+
|
| 95 |
+
return seq
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
def sequence_to_grids(
|
| 99 |
+
seq: np.ndarray,
|
| 100 |
+
) -> Dict[str, np.ndarray]:
|
| 101 |
+
"""
|
| 102 |
+
Convert a [T, 39] sequence back into per-family grids.
|
| 103 |
+
|
| 104 |
+
Returns:
|
| 105 |
+
{
|
| 106 |
+
"strings": [bars, STEPS_PER_BAR, 12],
|
| 107 |
+
"woodwinds": [bars, STEPS_PER_BAR, 12],
|
| 108 |
+
"brass": [bars, STEPS_PER_BAR, 12],
|
| 109 |
+
"perc": [bars, STEPS_PER_BAR, 3],
|
| 110 |
+
}
|
| 111 |
+
"""
|
| 112 |
+
T = seq.shape[0]
|
| 113 |
+
bars = T // config.STEPS_PER_BAR
|
| 114 |
+
seq = seq[: bars * config.STEPS_PER_BAR] # trim to full bars if needed
|
| 115 |
+
|
| 116 |
+
strings = seq[:, 0:12]
|
| 117 |
+
woodwinds = seq[:, 12:24]
|
| 118 |
+
brass = seq[:, 24:36]
|
| 119 |
+
perc = seq[:, 36:39]
|
| 120 |
+
|
| 121 |
+
strings_grid = strings.reshape(bars, config.STEPS_PER_BAR, config.PITCHES_MELODIC)
|
| 122 |
+
woodwinds_grid = woodwinds.reshape(bars, config.STEPS_PER_BAR, config.PITCHES_MELODIC)
|
| 123 |
+
brass_grid = brass.reshape(bars, config.STEPS_PER_BAR, config.PITCHES_MELODIC)
|
| 124 |
+
perc_grid = perc.reshape(bars, config.STEPS_PER_BAR, config.PITCHES_PERC)
|
| 125 |
+
|
| 126 |
+
return {
|
| 127 |
+
"strings": strings_grid,
|
| 128 |
+
"woodwinds": woodwinds_grid,
|
| 129 |
+
"brass": brass_grid,
|
| 130 |
+
"perc": perc_grid,
|
| 131 |
+
}
|
| 132 |
+
|
| 133 |
+
|
| 134 |
+
# =========================
|
| 135 |
+
# Generation
|
| 136 |
+
# =========================
|
| 137 |
+
|
| 138 |
+
def compute_target_bars(target_minutes: float) -> int:
|
| 139 |
+
"""
|
| 140 |
+
Compute number of bars needed to approximate target_minutes,
|
| 141 |
+
given fixed tempo and meter.
|
| 142 |
+
|
| 143 |
+
Returns:
|
| 144 |
+
target_bars: int
|
| 145 |
+
"""
|
| 146 |
+
total_seconds = target_minutes * 60.0
|
| 147 |
+
seconds_per_bar = config.SECONDS_PER_BAR
|
| 148 |
+
target_bars = math.ceil(total_seconds / seconds_per_bar)
|
| 149 |
+
return target_bars
|
| 150 |
+
|
| 151 |
+
|
| 152 |
+
def generate_from_seed(
|
| 153 |
+
strings_seed: np.ndarray,
|
| 154 |
+
woodwinds_seed: np.ndarray,
|
| 155 |
+
brass_seed: np.ndarray,
|
| 156 |
+
perc_seed: np.ndarray,
|
| 157 |
+
target_minutes: float,
|
| 158 |
+
temperature: float = 1.0,
|
| 159 |
+
top_k: int = 0,
|
| 160 |
+
device: str = "cpu",
|
| 161 |
+
) -> Dict[str, np.ndarray]:
|
| 162 |
+
"""
|
| 163 |
+
Generate full grids from seed grids using the trained model.
|
| 164 |
+
|
| 165 |
+
Args:
|
| 166 |
+
strings_seed, woodwinds_seed, brass_seed, perc_seed:
|
| 167 |
+
Seed grids from UI, shapes:
|
| 168 |
+
[INPUT_BARS, STEPS_PER_BAR, 12] for melodic families,
|
| 169 |
+
[INPUT_BARS, STEPS_PER_BAR, 3] for perc.
|
| 170 |
+
target_minutes: Desired approximate length in minutes.
|
| 171 |
+
temperature: Sampling temperature.
|
| 172 |
+
top_k: top-k filter.
|
| 173 |
+
device: "cpu" or "cuda".
|
| 174 |
+
|
| 175 |
+
Returns:
|
| 176 |
+
full_grids dict:
|
| 177 |
+
{
|
| 178 |
+
"strings": [total_bars, STEPS_PER_BAR, 12],
|
| 179 |
+
"woodwinds": [total_bars, STEPS_PER_BAR, 12],
|
| 180 |
+
"brass": [total_bars, STEPS_PER_BAR, 12],
|
| 181 |
+
"perc": [total_bars, STEPS_PER_BAR, 3],
|
| 182 |
+
}
|
| 183 |
+
This includes seed bars + generated bars.
|
| 184 |
+
"""
|
| 185 |
+
model = load_model(device=device) # global cached
|
| 186 |
+
|
| 187 |
+
# Convert seeds to [T_seed, 39]
|
| 188 |
+
seed_seq = seed_grids_to_sequence(
|
| 189 |
+
strings_seed=strings_seed,
|
| 190 |
+
woodwinds_seed=woodwinds_seed,
|
| 191 |
+
brass_seed=brass_seed,
|
| 192 |
+
perc_seed=perc_seed,
|
| 193 |
+
) # [T_seed, 39]
|
| 194 |
+
|
| 195 |
+
T_seed = seed_seq.shape[0]
|
| 196 |
+
seed_bars = T_seed // config.STEPS_PER_BAR
|
| 197 |
+
|
| 198 |
+
target_bars = compute_target_bars(target_minutes)
|
| 199 |
+
# Ensure target_bars >= seed_bars
|
| 200 |
+
if target_bars <= seed_bars:
|
| 201 |
+
target_bars = seed_bars + 1
|
| 202 |
+
|
| 203 |
+
total_bars_to_generate = target_bars - seed_bars
|
| 204 |
+
|
| 205 |
+
# Start with seed sequence as context
|
| 206 |
+
seq = seed_seq.copy() # [T_cur, 39]
|
| 207 |
+
|
| 208 |
+
# Keep track of bar dicts (including seed bars)
|
| 209 |
+
all_bars: Dict[str, list] = {
|
| 210 |
+
"strings": [],
|
| 211 |
+
"woodwinds": [],
|
| 212 |
+
"brass": [],
|
| 213 |
+
"perc": [],
|
| 214 |
+
}
|
| 215 |
+
|
| 216 |
+
# First, add the seed bars to all_bars
|
| 217 |
+
seed_grids = sequence_to_grids(seed_seq)
|
| 218 |
+
for key in all_bars.keys():
|
| 219 |
+
for b in range(seed_grids[key].shape[0]):
|
| 220 |
+
all_bars[key].append(seed_grids[key][b])
|
| 221 |
+
|
| 222 |
+
# Previous generated bar for similarity check (full dict)
|
| 223 |
+
prev_bar_dict = {
|
| 224 |
+
"strings": all_bars["strings"][-1],
|
| 225 |
+
"woodwinds": all_bars["woodwinds"][-1],
|
| 226 |
+
"brass": all_bars["brass"][-1],
|
| 227 |
+
"perc": all_bars["perc"][-1],
|
| 228 |
+
}
|
| 229 |
+
|
| 230 |
+
for bar_idx in range(total_bars_to_generate):
|
| 231 |
+
# Collect logits for one bar
|
| 232 |
+
bar_step_logits = {
|
| 233 |
+
"strings": [],
|
| 234 |
+
"woodwinds": [],
|
| 235 |
+
"brass": [],
|
| 236 |
+
"perc": [],
|
| 237 |
+
}
|
| 238 |
+
|
| 239 |
+
for step_in_bar in range(config.STEPS_PER_BAR):
|
| 240 |
+
seq_tensor = torch.from_numpy(seq).unsqueeze(0).to(device) # [1, T_cur, 39]
|
| 241 |
+
|
| 242 |
+
with torch.no_grad():
|
| 243 |
+
outputs = model(seq_tensor) # dict of [1, T_cur, pitches]
|
| 244 |
+
|
| 245 |
+
# Take logits for the last time step
|
| 246 |
+
strings_logits = outputs["strings"][:, -1, :] # [1, 12]
|
| 247 |
+
woodwinds_logits = outputs["woodwinds"][:, -1, :]
|
| 248 |
+
brass_logits = outputs["brass"][:, -1, :]
|
| 249 |
+
perc_logits = outputs["perc"][:, -1, :]
|
| 250 |
+
|
| 251 |
+
bar_step_logits["strings"].append(strings_logits)
|
| 252 |
+
bar_step_logits["woodwinds"].append(woodwinds_logits)
|
| 253 |
+
bar_step_logits["brass"].append(brass_logits)
|
| 254 |
+
bar_step_logits["perc"].append(perc_logits)
|
| 255 |
+
|
| 256 |
+
# Append placeholder next-step feature
|
| 257 |
+
next_step_placeholder = np.zeros((1, config.INPUT_FEATURES), dtype=np.float32)
|
| 258 |
+
seq = np.concatenate([seq, next_step_placeholder], axis=0) # [T_cur+1, 39]
|
| 259 |
+
|
| 260 |
+
# Stack logits to [1, T, P] per family
|
| 261 |
+
logits_dict = {
|
| 262 |
+
"strings": torch.stack(bar_step_logits["strings"], dim=1), # [1, T, 12]
|
| 263 |
+
"woodwinds": torch.stack(bar_step_logits["woodwinds"], dim=1),
|
| 264 |
+
"brass": torch.stack(bar_step_logits["brass"], dim=1),
|
| 265 |
+
"perc": torch.stack(bar_step_logits["perc"], dim=1), # [1, T, 3]
|
| 266 |
+
}
|
| 267 |
+
|
| 268 |
+
# Sample a bar with constraints
|
| 269 |
+
bar_dict = sampling_utils.resample_bar_if_needed(
|
| 270 |
+
prev_bar=prev_bar_dict,
|
| 271 |
+
logits_dict=logits_dict,
|
| 272 |
+
temperature=temperature,
|
| 273 |
+
top_k=top_k,
|
| 274 |
+
# using defaults inside resample for min_notes_per_bar, max_notes_per_step
|
| 275 |
+
)
|
| 276 |
+
|
| 277 |
+
# Append this bar
|
| 278 |
+
for key in all_bars.keys():
|
| 279 |
+
all_bars[key].append(bar_dict[key])
|
| 280 |
+
|
| 281 |
+
# Update prev_bar_dict
|
| 282 |
+
prev_bar_dict = bar_dict
|
| 283 |
+
|
| 284 |
+
# Update the last STEPS_PER_BAR placeholders in seq with actual features
|
| 285 |
+
T_cur = seq.shape[0]
|
| 286 |
+
start_idx = T_cur - config.STEPS_PER_BAR
|
| 287 |
+
end_idx = T_cur
|
| 288 |
+
|
| 289 |
+
strings_bar = bar_dict["strings"] # [T, 12]
|
| 290 |
+
woodwinds_bar = bar_dict["woodwinds"]
|
| 291 |
+
brass_bar = bar_dict["brass"]
|
| 292 |
+
perc_bar = bar_dict["perc"]
|
| 293 |
+
|
| 294 |
+
bar_features = np.concatenate(
|
| 295 |
+
[strings_bar, woodwinds_bar, brass_bar, perc_bar],
|
| 296 |
+
axis=-1,
|
| 297 |
+
).astype(np.float32) # [T, 39]
|
| 298 |
+
|
| 299 |
+
seq[start_idx:end_idx, :] = bar_features
|
| 300 |
+
|
| 301 |
+
# Convert all_bars lists to numpy grids
|
| 302 |
+
strings_grid = np.stack(all_bars["strings"], axis=0) # [bars, STEPS_PER_BAR, 12]
|
| 303 |
+
woodwinds_grid = np.stack(all_bars["woodwinds"], axis=0)
|
| 304 |
+
brass_grid = np.stack(all_bars["brass"], axis=0)
|
| 305 |
+
perc_grid = np.stack(all_bars["perc"], axis=0)
|
| 306 |
+
|
| 307 |
+
return {
|
| 308 |
+
"strings": strings_grid,
|
| 309 |
+
"woodwinds": woodwinds_grid,
|
| 310 |
+
"brass": brass_grid,
|
| 311 |
+
"perc": perc_grid,
|
| 312 |
+
}
|
| 313 |
+
|
| 314 |
+
|
| 315 |
+
# =========================
|
| 316 |
+
# Orchestration from primary family
|
| 317 |
+
# =========================
|
| 318 |
+
|
| 319 |
+
def orchestrate_from_primary(
|
| 320 |
+
full_grids: Dict[str, np.ndarray],
|
| 321 |
+
primary_family: str,
|
| 322 |
+
) -> Dict[str, np.ndarray]:
|
| 323 |
+
"""
|
| 324 |
+
Build other families from a chosen primary melodic family using simple rules.
|
| 325 |
+
|
| 326 |
+
Args:
|
| 327 |
+
full_grids: raw model output:
|
| 328 |
+
"strings": [B, T, 12]
|
| 329 |
+
"woodwinds": [B, T, 12]
|
| 330 |
+
"brass": [B, T, 12]
|
| 331 |
+
"perc": [B, T, 3]
|
| 332 |
+
primary_family: one of "strings", "woodwinds", "brass".
|
| 333 |
+
|
| 334 |
+
Returns:
|
| 335 |
+
new_grids: dict with same keys, but non-primary families derived from primary.
|
| 336 |
+
"""
|
| 337 |
+
grids = {k: v.copy() for k, v in full_grids.items()}
|
| 338 |
+
|
| 339 |
+
if primary_family not in ["strings", "woodwinds", "brass"]:
|
| 340 |
+
return grids
|
| 341 |
+
|
| 342 |
+
primary = grids[primary_family] # [B, T, 12]
|
| 343 |
+
B, T, P = primary.shape
|
| 344 |
+
|
| 345 |
+
# Reset melodic families to zero
|
| 346 |
+
for fam in ["strings", "woodwinds", "brass"]:
|
| 347 |
+
grids[fam] = np.zeros_like(primary)
|
| 348 |
+
|
| 349 |
+
strong_steps = [0, 2, 4, 6]
|
| 350 |
+
|
| 351 |
+
def thinned_copy(src: np.ndarray) -> np.ndarray:
|
| 352 |
+
dst = np.zeros_like(src)
|
| 353 |
+
dst[:, strong_steps, :] = src[:, strong_steps, :]
|
| 354 |
+
return dst
|
| 355 |
+
|
| 356 |
+
def root_pattern(src: np.ndarray) -> np.ndarray:
|
| 357 |
+
dst = np.zeros_like(src)
|
| 358 |
+
keep_steps = [0, 4]
|
| 359 |
+
dst[:, keep_steps, :] = src[:, keep_steps, :]
|
| 360 |
+
return dst
|
| 361 |
+
|
| 362 |
+
if primary_family == "strings":
|
| 363 |
+
grids["strings"] = primary
|
| 364 |
+
grids["woodwinds"] = thinned_copy(primary)
|
| 365 |
+
grids["brass"] = root_pattern(primary)
|
| 366 |
+
elif primary_family == "woodwinds":
|
| 367 |
+
grids["woodwinds"] = primary
|
| 368 |
+
grids["strings"] = thinned_copy(primary)
|
| 369 |
+
grids["brass"] = root_pattern(primary)
|
| 370 |
+
elif primary_family == "brass":
|
| 371 |
+
grids["brass"] = primary
|
| 372 |
+
grids["strings"] = thinned_copy(primary)
|
| 373 |
+
grids["woodwinds"] = root_pattern(primary)
|
| 374 |
+
|
| 375 |
+
# Percussion: simple fixed pattern
|
| 376 |
+
perc = np.zeros_like(grids["perc"])
|
| 377 |
+
for b in range(B):
|
| 378 |
+
for t in range(T):
|
| 379 |
+
if t in [0, 4]:
|
| 380 |
+
perc[b, t, 0] = 1.0 # kick-like
|
| 381 |
+
if t in [2, 6]:
|
| 382 |
+
perc[b, t, 1] = 1.0 # snare-like
|
| 383 |
+
grids["perc"] = perc
|
| 384 |
+
|
| 385 |
+
return grids
|
| 386 |
+
|
| 387 |
+
|
| 388 |
+
# =========================
|
| 389 |
+
# MIDI / audio helpers
|
| 390 |
+
# =========================
|
| 391 |
+
|
| 392 |
+
def generate_midi_file(
|
| 393 |
+
strings_seed: np.ndarray,
|
| 394 |
+
woodwinds_seed: np.ndarray,
|
| 395 |
+
brass_seed: np.ndarray,
|
| 396 |
+
perc_seed: np.ndarray,
|
| 397 |
+
target_minutes: float,
|
| 398 |
+
temperature: float,
|
| 399 |
+
top_k: int,
|
| 400 |
+
midi_out_path: str = "output.mid",
|
| 401 |
+
device: str = "cpu",
|
| 402 |
+
primary_family: str = "strings",
|
| 403 |
+
) -> str:
|
| 404 |
+
"""
|
| 405 |
+
High-level helper: generate music from seed grids and write to a MIDI file.
|
| 406 |
+
|
| 407 |
+
Returns:
|
| 408 |
+
Path to the written .mid file.
|
| 409 |
+
"""
|
| 410 |
+
full_grids = generate_from_seed(
|
| 411 |
+
strings_seed=strings_seed,
|
| 412 |
+
woodwinds_seed=woodwinds_seed,
|
| 413 |
+
brass_seed=brass_seed,
|
| 414 |
+
perc_seed=perc_seed,
|
| 415 |
+
target_minutes=target_minutes,
|
| 416 |
+
temperature=temperature,
|
| 417 |
+
top_k=top_k,
|
| 418 |
+
device=device,
|
| 419 |
+
)
|
| 420 |
+
|
| 421 |
+
# Rebuild orchestration from chosen primary family
|
| 422 |
+
orchestrated = orchestrate_from_primary(full_grids, primary_family=primary_family)
|
| 423 |
+
|
| 424 |
+
pm = midi_utils.create_pretty_midi_from_grids(
|
| 425 |
+
strings_grid=orchestrated["strings"],
|
| 426 |
+
woodwinds_grid=orchestrated["woodwinds"],
|
| 427 |
+
brass_grid=orchestrated["brass"],
|
| 428 |
+
perc_grid=orchestrated["perc"],
|
| 429 |
+
)
|
| 430 |
+
|
| 431 |
+
os.makedirs(os.path.dirname(midi_out_path) or ".", exist_ok=True)
|
| 432 |
+
return midi_utils.save_midi(pm, midi_out_path)
|
| 433 |
+
|
| 434 |
+
|
| 435 |
+
def generate_midi_and_audio(
|
| 436 |
+
strings_seed: np.ndarray,
|
| 437 |
+
woodwinds_seed: np.ndarray,
|
| 438 |
+
brass_seed: np.ndarray,
|
| 439 |
+
perc_seed: np.ndarray,
|
| 440 |
+
target_minutes: float,
|
| 441 |
+
temperature: float,
|
| 442 |
+
top_k: int,
|
| 443 |
+
midi_out_path: str = "output.mid",
|
| 444 |
+
audio_out_path: str = "output.wav",
|
| 445 |
+
device: str = "cpu",
|
| 446 |
+
primary_family: str = "strings",
|
| 447 |
+
) -> Tuple[str, str]:
|
| 448 |
+
"""
|
| 449 |
+
Generate MIDI and render audio via Fluidsynth.
|
| 450 |
+
|
| 451 |
+
Returns:
|
| 452 |
+
(midi_path, audio_path)
|
| 453 |
+
"""
|
| 454 |
+
midi_path = generate_midi_file(
|
| 455 |
+
strings_seed=strings_seed,
|
| 456 |
+
woodwinds_seed=woodwinds_seed,
|
| 457 |
+
brass_seed=brass_seed,
|
| 458 |
+
perc_seed=perc_seed,
|
| 459 |
+
target_minutes=target_minutes,
|
| 460 |
+
temperature=temperature,
|
| 461 |
+
top_k=top_k,
|
| 462 |
+
midi_out_path=midi_out_path,
|
| 463 |
+
device=device,
|
| 464 |
+
primary_family=primary_family,
|
| 465 |
+
)
|
| 466 |
+
|
| 467 |
+
# Render audio (requires fluidsynth and soundfont)
|
| 468 |
+
try:
|
| 469 |
+
import fluidsynth
|
| 470 |
+
import pretty_midi
|
| 471 |
+
from scipy.io import wavfile
|
| 472 |
+
|
| 473 |
+
fs = fluidsynth.Synth()
|
| 474 |
+
sfid = fs.sfload("soundfont.sf2")
|
| 475 |
+
fs.program_select(0, sfid, 0, 0)
|
| 476 |
+
|
| 477 |
+
pm = pretty_midi.PrettyMIDI(midi_path)
|
| 478 |
+
audio = pm.fluidsynth(fs=16000)
|
| 479 |
+
wavfile.write(audio_out_path, 16000, audio.astype(np.float32))
|
| 480 |
+
fs.delete()
|
| 481 |
+
except Exception as e:
|
| 482 |
+
print(f"Audio rendering failed: {e}")
|
| 483 |
+
audio_out_path = ""
|
| 484 |
+
|
| 485 |
+
return midi_path, audio_out_path
|
model.py
ADDED
|
@@ -0,0 +1,163 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Transformer model for multi-instrument grid-based music generation.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
from typing import Dict
|
| 6 |
+
|
| 7 |
+
import torch
|
| 8 |
+
import torch.nn as nn
|
| 9 |
+
import torch.nn.functional as F
|
| 10 |
+
|
| 11 |
+
import config
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
class PositionalEncoding(nn.Module):
|
| 15 |
+
"""
|
| 16 |
+
Standard sinusoidal positional encoding for Transformer inputs.
|
| 17 |
+
Expects input of shape [batch, seq_len, d_model].
|
| 18 |
+
"""
|
| 19 |
+
|
| 20 |
+
def __init__(self, d_model: int, max_len: int = 2048):
|
| 21 |
+
super().__init__()
|
| 22 |
+
self.d_model = d_model
|
| 23 |
+
|
| 24 |
+
pe = torch.zeros(max_len, d_model) # [max_len, d_model]
|
| 25 |
+
position = torch.arange(0, max_len, dtype=torch.float32).unsqueeze(1)
|
| 26 |
+
div_term = torch.exp(
|
| 27 |
+
torch.arange(0, d_model, 2, dtype=torch.float32)
|
| 28 |
+
* (-torch.log(torch.tensor(10000.0)) / d_model)
|
| 29 |
+
)
|
| 30 |
+
|
| 31 |
+
pe[:, 0::2] = torch.sin(position * div_term) # even indices
|
| 32 |
+
pe[:, 1::2] = torch.cos(position * div_term) # odd indices
|
| 33 |
+
|
| 34 |
+
pe = pe.unsqueeze(0) # [1, max_len, d_model]
|
| 35 |
+
self.register_buffer("pe", pe)
|
| 36 |
+
|
| 37 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 38 |
+
"""
|
| 39 |
+
Add positional encoding to input.
|
| 40 |
+
|
| 41 |
+
Args:
|
| 42 |
+
x: [batch, seq_len, d_model]
|
| 43 |
+
|
| 44 |
+
Returns:
|
| 45 |
+
[batch, seq_len, d_model]
|
| 46 |
+
"""
|
| 47 |
+
seq_len = x.size(1)
|
| 48 |
+
# pe[:, :seq_len] has shape [1, seq_len, d_model]
|
| 49 |
+
return x + self.pe[:, :seq_len]
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
class MusicTransformer(nn.Module):
|
| 53 |
+
"""
|
| 54 |
+
Transformer encoder with per-instrument output heads.
|
| 55 |
+
|
| 56 |
+
Input:
|
| 57 |
+
x: [batch, seq_len, INPUT_FEATURES] (multi-hot grid)
|
| 58 |
+
|
| 59 |
+
Output:
|
| 60 |
+
dict of logits:
|
| 61 |
+
{
|
| 62 |
+
"strings": [batch, seq_len, 12],
|
| 63 |
+
"woodwinds":[batch, seq_len, 12],
|
| 64 |
+
"brass": [batch, seq_len, 12],
|
| 65 |
+
"perc": [batch, seq_len, 3],
|
| 66 |
+
}
|
| 67 |
+
"""
|
| 68 |
+
|
| 69 |
+
def __init__(
|
| 70 |
+
self,
|
| 71 |
+
input_dim: int = config.INPUT_FEATURES,
|
| 72 |
+
d_model: int = config.D_MODEL,
|
| 73 |
+
n_layers: int = config.N_LAYERS,
|
| 74 |
+
n_heads: int = config.N_HEADS,
|
| 75 |
+
ffn_dim: int = config.FFN_DIM,
|
| 76 |
+
dropout: float = config.DROPOUT,
|
| 77 |
+
):
|
| 78 |
+
super().__init__()
|
| 79 |
+
|
| 80 |
+
self.input_dim = input_dim
|
| 81 |
+
self.d_model = d_model
|
| 82 |
+
|
| 83 |
+
# Project 39-dim multi-hot input to d_model
|
| 84 |
+
self.input_proj = nn.Linear(input_dim, d_model)
|
| 85 |
+
|
| 86 |
+
# Positional encoding
|
| 87 |
+
self.pos_encoding = PositionalEncoding(d_model=d_model)
|
| 88 |
+
|
| 89 |
+
# Transformer encoder
|
| 90 |
+
encoder_layer = nn.TransformerEncoderLayer(
|
| 91 |
+
d_model=d_model,
|
| 92 |
+
nhead=n_heads,
|
| 93 |
+
dim_feedforward=ffn_dim,
|
| 94 |
+
dropout=dropout,
|
| 95 |
+
batch_first=True, # [batch, seq, d_model]
|
| 96 |
+
)
|
| 97 |
+
self.transformer_encoder = nn.TransformerEncoder(
|
| 98 |
+
encoder_layer,
|
| 99 |
+
num_layers=n_layers,
|
| 100 |
+
)
|
| 101 |
+
|
| 102 |
+
# Per-instrument heads
|
| 103 |
+
self.strings_head = nn.Linear(d_model, config.PITCHES_MELODIC) # 12
|
| 104 |
+
self.woodwinds_head = nn.Linear(d_model, config.PITCHES_MELODIC) # 12
|
| 105 |
+
self.brass_head = nn.Linear(d_model, config.PITCHES_MELODIC) # 12
|
| 106 |
+
self.perc_head = nn.Linear(d_model, config.PITCHES_PERC) # 3
|
| 107 |
+
|
| 108 |
+
def forward(self, x: torch.Tensor) -> Dict[str, torch.Tensor]:
|
| 109 |
+
"""
|
| 110 |
+
Forward pass.
|
| 111 |
+
|
| 112 |
+
Args:
|
| 113 |
+
x: Tensor of shape [batch, seq_len, INPUT_FEATURES]
|
| 114 |
+
|
| 115 |
+
Returns:
|
| 116 |
+
Dict of logits, each [batch, seq_len, pitches]:
|
| 117 |
+
{
|
| 118 |
+
"strings": logits_S,
|
| 119 |
+
"woodwinds": logits_W,
|
| 120 |
+
"brass": logits_B,
|
| 121 |
+
"perc": logits_P,
|
| 122 |
+
}
|
| 123 |
+
"""
|
| 124 |
+
# Project input features to d_model
|
| 125 |
+
h = self.input_proj(x) # [batch, seq_len, d_model]
|
| 126 |
+
|
| 127 |
+
# Add positional encoding
|
| 128 |
+
h = self.pos_encoding(h) # [batch, seq_len, d_model]
|
| 129 |
+
|
| 130 |
+
# Transformer encoder
|
| 131 |
+
# (no src_key_padding_mask here; you can add later if needed)
|
| 132 |
+
h = self.transformer_encoder(h) # [batch, seq_len, d_model]
|
| 133 |
+
|
| 134 |
+
# Per-instrument logits
|
| 135 |
+
logits_strings = self.strings_head(h) # [batch, seq_len, 12]
|
| 136 |
+
logits_woodwinds = self.woodwinds_head(h)
|
| 137 |
+
logits_brass = self.brass_head(h)
|
| 138 |
+
logits_perc = self.perc_head(h) # [batch, seq_len, 3]
|
| 139 |
+
|
| 140 |
+
return {
|
| 141 |
+
"strings": logits_strings,
|
| 142 |
+
"woodwinds": logits_woodwinds,
|
| 143 |
+
"brass": logits_brass,
|
| 144 |
+
"perc": logits_perc,
|
| 145 |
+
}
|
| 146 |
+
|
| 147 |
+
|
| 148 |
+
def build_model(cfg=config) -> MusicTransformer:
|
| 149 |
+
"""
|
| 150 |
+
Convenience function to build a MusicTransformer from config values.
|
| 151 |
+
|
| 152 |
+
Returns:
|
| 153 |
+
MusicTransformer instance with parameters from config.py.
|
| 154 |
+
"""
|
| 155 |
+
model = MusicTransformer(
|
| 156 |
+
input_dim=cfg.INPUT_FEATURES,
|
| 157 |
+
d_model=cfg.D_MODEL,
|
| 158 |
+
n_layers=cfg.N_LAYERS,
|
| 159 |
+
n_heads=cfg.N_HEADS,
|
| 160 |
+
ffn_dim=cfg.FFN_DIM,
|
| 161 |
+
dropout=cfg.DROPOUT,
|
| 162 |
+
)
|
| 163 |
+
return model
|
models/transformer_music_best.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4dd4e457a4a7c11a0a0628e8c746cf07d370d8eba0eed7236b65fb2f966e101f
|
| 3 |
+
size 8522289
|
requirements.txt
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Core ML
|
| 2 |
+
torch>=2.0.0
|
| 3 |
+
|
| 4 |
+
# Web UI
|
| 5 |
+
gradio>=4.0.0
|
| 6 |
+
|
| 7 |
+
# MIDI and audio
|
| 8 |
+
pretty_midi>=0.2.9
|
| 9 |
+
pyfluidsynth>=1.3.0
|
| 10 |
+
mido>=1.2.10
|
| 11 |
+
|
| 12 |
+
# Numerics
|
| 13 |
+
numpy>=1.24.0
|
| 14 |
+
scipy>=1.10.0
|
| 15 |
+
|
| 16 |
+
# Utilities (optional but common)
|
| 17 |
+
tqdm>=4.65.0
|
| 18 |
+
|
| 19 |
+
fluidsynth
|
train.py
ADDED
|
@@ -0,0 +1,215 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import os
|
| 2 |
+
from typing import Tuple
|
| 3 |
+
|
| 4 |
+
import numpy as np
|
| 5 |
+
import torch
|
| 6 |
+
import torch.nn as nn
|
| 7 |
+
from torch.utils.data import DataLoader, TensorDataset, random_split
|
| 8 |
+
|
| 9 |
+
import config
|
| 10 |
+
from model import build_model
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
def load_dataset_npz(path: str) -> Tuple[torch.Tensor, torch.Tensor]:
|
| 14 |
+
data = np.load(path)
|
| 15 |
+
X = data["X"].astype(np.float32)
|
| 16 |
+
Y = data["Y"].astype(np.float32)
|
| 17 |
+
return torch.from_numpy(X), torch.from_numpy(Y)
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def load_dataset_pt(path: str) -> Tuple[torch.Tensor, torch.Tensor]:
|
| 21 |
+
data = torch.load(path)
|
| 22 |
+
return data["X"].float(), data["Y"].float()
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def train(
|
| 26 |
+
dataset_path: str,
|
| 27 |
+
use_npz: bool = True,
|
| 28 |
+
batch_size: int = 32,
|
| 29 |
+
num_epochs: int = 20,
|
| 30 |
+
val_split: float = 0.1,
|
| 31 |
+
learning_rate: float = 1e-3,
|
| 32 |
+
device: str = None,
|
| 33 |
+
save_dir: str = "models",
|
| 34 |
+
best_name: str = "transformer_music_best.pt",
|
| 35 |
+
last_name: str = "transformer_music_last.pt",
|
| 36 |
+
checkpoint_name: str = "transformer_music_checkpoint.pt",
|
| 37 |
+
extra_epochs: int = None,
|
| 38 |
+
) -> None:
|
| 39 |
+
"""
|
| 40 |
+
Train MusicTransformer with resume & best-model saving.
|
| 41 |
+
|
| 42 |
+
If extra_epochs is not None, the function will:
|
| 43 |
+
- load checkpoint (if any) and read checkpoint["epoch"] as last_epoch,
|
| 44 |
+
- start from last_epoch + 1,
|
| 45 |
+
- run for exactly extra_epochs epochs (end_epoch = last_epoch + extra_epochs).
|
| 46 |
+
|
| 47 |
+
If extra_epochs is None, num_epochs is treated as a global target:
|
| 48 |
+
- start from start_epoch (1 or checkpoint_epoch+1),
|
| 49 |
+
- run until epoch == num_epochs.
|
| 50 |
+
"""
|
| 51 |
+
|
| 52 |
+
if device is None:
|
| 53 |
+
device = "cuda" if torch.cuda.is_available() else "cpu"
|
| 54 |
+
|
| 55 |
+
# Load dataset
|
| 56 |
+
if use_npz:
|
| 57 |
+
X, Y = load_dataset_npz(dataset_path)
|
| 58 |
+
else:
|
| 59 |
+
X, Y = load_dataset_pt(dataset_path)
|
| 60 |
+
|
| 61 |
+
print(f"Loaded dataset: X {X.shape}, Y {Y.shape}")
|
| 62 |
+
|
| 63 |
+
dataset = TensorDataset(X, Y)
|
| 64 |
+
|
| 65 |
+
# Train/val split
|
| 66 |
+
n_total = len(dataset)
|
| 67 |
+
n_val = int(n_total * val_split)
|
| 68 |
+
n_train = n_total - n_val
|
| 69 |
+
|
| 70 |
+
train_ds, val_ds = random_split(dataset, [n_train, n_val])
|
| 71 |
+
|
| 72 |
+
train_loader = DataLoader(train_ds, batch_size=batch_size, shuffle=True)
|
| 73 |
+
val_loader = DataLoader(val_ds, batch_size=batch_size, shuffle=False)
|
| 74 |
+
|
| 75 |
+
# Build model, optimizer, loss
|
| 76 |
+
model = build_model(config).to(device)
|
| 77 |
+
bce = nn.BCEWithLogitsLoss()
|
| 78 |
+
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
|
| 79 |
+
|
| 80 |
+
os.makedirs(save_dir, exist_ok=True)
|
| 81 |
+
best_model_path = os.path.join(save_dir, best_name)
|
| 82 |
+
last_model_path = os.path.join(save_dir, last_name)
|
| 83 |
+
checkpoint_path = os.path.join(save_dir, checkpoint_name)
|
| 84 |
+
|
| 85 |
+
# ===== Resume from checkpoint if present =====
|
| 86 |
+
start_epoch = 1
|
| 87 |
+
best_val_loss = float("inf")
|
| 88 |
+
|
| 89 |
+
if os.path.exists(checkpoint_path):
|
| 90 |
+
print(f"Found checkpoint at {checkpoint_path}, loading...")
|
| 91 |
+
checkpoint = torch.load(checkpoint_path, map_location=device)
|
| 92 |
+
model.load_state_dict(checkpoint["model_state_dict"])
|
| 93 |
+
optimizer.load_state_dict(checkpoint["optimizer_state_dict"])
|
| 94 |
+
start_epoch = checkpoint["epoch"] + 1
|
| 95 |
+
best_val_loss = checkpoint["best_val_loss"]
|
| 96 |
+
print(f"Resuming from epoch {start_epoch}, best_val_loss={best_val_loss:.4f}")
|
| 97 |
+
|
| 98 |
+
# Determine end_epoch based on extra_epochs / num_epochs
|
| 99 |
+
if extra_epochs is not None:
|
| 100 |
+
# Run exactly extra_epochs from the current checkpoint point
|
| 101 |
+
end_epoch = start_epoch + extra_epochs - 1
|
| 102 |
+
else:
|
| 103 |
+
# Run until global num_epochs
|
| 104 |
+
end_epoch = num_epochs
|
| 105 |
+
|
| 106 |
+
if start_epoch > end_epoch:
|
| 107 |
+
print(f"No epochs to run (start_epoch={start_epoch}, end_epoch={end_epoch}).")
|
| 108 |
+
print("Training complete.")
|
| 109 |
+
print(f"Best model: {best_model_path}")
|
| 110 |
+
print(f"Last model: {last_model_path}")
|
| 111 |
+
return
|
| 112 |
+
|
| 113 |
+
# ===== Training loop =====
|
| 114 |
+
for epoch in range(start_epoch, end_epoch + 1):
|
| 115 |
+
model.train()
|
| 116 |
+
train_loss_sum = 0.0
|
| 117 |
+
train_batches = 0
|
| 118 |
+
|
| 119 |
+
for X_batch, Y_batch in train_loader:
|
| 120 |
+
X_batch = X_batch.to(device) # [B, T, 39]
|
| 121 |
+
Y_batch = Y_batch.to(device) # [B, T, 39]
|
| 122 |
+
|
| 123 |
+
optimizer.zero_grad()
|
| 124 |
+
|
| 125 |
+
outputs = model(X_batch)
|
| 126 |
+
|
| 127 |
+
# Slice per-instrument targets
|
| 128 |
+
strings_target = Y_batch[:, :, 0:12]
|
| 129 |
+
woodwinds_target = Y_batch[:, :, 12:24]
|
| 130 |
+
brass_target = Y_batch[:, :, 24:36]
|
| 131 |
+
perc_target = Y_batch[:, :, 36:39]
|
| 132 |
+
|
| 133 |
+
loss_strings = bce(outputs["strings"], strings_target)
|
| 134 |
+
loss_wood = bce(outputs["woodwinds"], woodwinds_target)
|
| 135 |
+
loss_brass = bce(outputs["brass"], brass_target)
|
| 136 |
+
loss_perc = bce(outputs["perc"], perc_target)
|
| 137 |
+
|
| 138 |
+
loss = loss_strings + loss_wood + loss_brass + loss_perc
|
| 139 |
+
|
| 140 |
+
loss.backward()
|
| 141 |
+
optimizer.step()
|
| 142 |
+
|
| 143 |
+
train_loss_sum += loss.item()
|
| 144 |
+
train_batches += 1
|
| 145 |
+
|
| 146 |
+
avg_train_loss = train_loss_sum / max(train_batches, 1)
|
| 147 |
+
|
| 148 |
+
# Validation
|
| 149 |
+
model.eval()
|
| 150 |
+
val_loss_sum = 0.0
|
| 151 |
+
val_batches = 0
|
| 152 |
+
with torch.no_grad():
|
| 153 |
+
for X_batch, Y_batch in val_loader:
|
| 154 |
+
X_batch = X_batch.to(device)
|
| 155 |
+
Y_batch = Y_batch.to(device)
|
| 156 |
+
|
| 157 |
+
outputs = model(X_batch)
|
| 158 |
+
|
| 159 |
+
strings_target = Y_batch[:, :, 0:12]
|
| 160 |
+
woodwinds_target = Y_batch[:, :, 12:24]
|
| 161 |
+
brass_target = Y_batch[:, :, 24:36]
|
| 162 |
+
perc_target = Y_batch[:, :, 36:39]
|
| 163 |
+
|
| 164 |
+
loss_strings = bce(outputs["strings"], strings_target)
|
| 165 |
+
loss_wood = bce(outputs["woodwinds"], woodwinds_target)
|
| 166 |
+
loss_brass = bce(outputs["brass"], brass_target)
|
| 167 |
+
loss_perc = bce(outputs["perc"], perc_target)
|
| 168 |
+
|
| 169 |
+
loss = loss_strings + loss_wood + loss_perc + loss_brass
|
| 170 |
+
|
| 171 |
+
val_loss_sum += loss.item()
|
| 172 |
+
val_batches += 1
|
| 173 |
+
|
| 174 |
+
avg_val_loss = val_loss_sum / max(val_batches, 1)
|
| 175 |
+
|
| 176 |
+
print(
|
| 177 |
+
f"Epoch {epoch:03d} | "
|
| 178 |
+
f"Train loss: {avg_train_loss:.4f} | "
|
| 179 |
+
f"Val loss: {avg_val_loss:.4f}"
|
| 180 |
+
)
|
| 181 |
+
|
| 182 |
+
# ===== Save checkpoint (for resume) =====
|
| 183 |
+
checkpoint = {
|
| 184 |
+
"epoch": epoch,
|
| 185 |
+
"model_state_dict": model.state_dict(),
|
| 186 |
+
"optimizer_state_dict": optimizer.state_dict(),
|
| 187 |
+
"best_val_loss": best_val_loss,
|
| 188 |
+
}
|
| 189 |
+
torch.save(checkpoint, checkpoint_path)
|
| 190 |
+
print(f" ↳ Checkpoint saved to {checkpoint_path}")
|
| 191 |
+
|
| 192 |
+
# ===== Best model logic =====
|
| 193 |
+
if avg_val_loss < best_val_loss:
|
| 194 |
+
best_val_loss = avg_val_loss
|
| 195 |
+
torch.save(model.state_dict(), best_model_path)
|
| 196 |
+
print(f" ↳ New best model saved to {best_model_path}")
|
| 197 |
+
|
| 198 |
+
# Always save "last" model as well (optional)
|
| 199 |
+
torch.save(model.state_dict(), last_model_path)
|
| 200 |
+
|
| 201 |
+
print("Training complete.")
|
| 202 |
+
print(f"Best model: {best_model_path}")
|
| 203 |
+
print(f"Last model: {last_model_path}")
|
| 204 |
+
|
| 205 |
+
|
| 206 |
+
if __name__ == "__main__":
|
| 207 |
+
dataset_path = "musicnet_grid_dataset.npz"
|
| 208 |
+
train(
|
| 209 |
+
dataset_path=dataset_path,
|
| 210 |
+
use_npz=True,
|
| 211 |
+
batch_size=32,
|
| 212 |
+
num_epochs=20,
|
| 213 |
+
val_split=0.1,
|
| 214 |
+
learning_rate=1e-3,
|
| 215 |
+
)
|
utils/midi_utils.py
ADDED
|
@@ -0,0 +1,242 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
MIDI utilities: convert between grid representation and PrettyMIDI.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
from typing import Dict, Tuple, Optional
|
| 6 |
+
|
| 7 |
+
import numpy as np
|
| 8 |
+
import pretty_midi
|
| 9 |
+
|
| 10 |
+
import config
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
# Fixed MIDI pitches for one octave, center around middle range.
|
| 14 |
+
# You can adjust these if you want a different register.
|
| 15 |
+
# Example: C4 (60) to B4 (71) → 12 pitches.
|
| 16 |
+
MELODIC_BASE_PITCH: int = 60 # MIDI note for C4
|
| 17 |
+
|
| 18 |
+
# Percussion: General MIDI standard channel 9 (index 9), with typical drum notes.
|
| 19 |
+
PERCUSSION_CHANNEL: int = 9
|
| 20 |
+
PERC_KICK_PITCH: int = 36 # Bass Drum 1
|
| 21 |
+
PERC_SNARE_PITCH: int = 38 # Acoustic Snare
|
| 22 |
+
PERC_HIHAT_PITCH: int = 42 # Closed Hi-Hat
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def create_pretty_midi_from_grids(
|
| 26 |
+
strings_grid: np.ndarray,
|
| 27 |
+
woodwinds_grid: np.ndarray,
|
| 28 |
+
brass_grid: np.ndarray,
|
| 29 |
+
perc_grid: np.ndarray,
|
| 30 |
+
velocity: int = 80,
|
| 31 |
+
) -> pretty_midi.PrettyMIDI:
|
| 32 |
+
"""
|
| 33 |
+
Convert per-family grids to a PrettyMIDI object.
|
| 34 |
+
|
| 35 |
+
Args:
|
| 36 |
+
strings_grid: [bars, STEPS_PER_BAR, 12] binary array.
|
| 37 |
+
woodwinds_grid: [bars, STEPS_PER_BAR, 12] binary array.
|
| 38 |
+
brass_grid: [bars, STEPS_PER_BAR, 12] binary array.
|
| 39 |
+
perc_grid: [bars, STEPS_PER_BAR, 3] binary array (Kick, Snare, Hi-hat).
|
| 40 |
+
velocity: MIDI velocity for melodic notes.
|
| 41 |
+
|
| 42 |
+
Returns:
|
| 43 |
+
pm: PrettyMIDI object with 4 instrument tracks.
|
| 44 |
+
"""
|
| 45 |
+
# Basic checks
|
| 46 |
+
bars, steps_per_bar, n_pitch_strings = strings_grid.shape
|
| 47 |
+
assert steps_per_bar == config.STEPS_PER_BAR
|
| 48 |
+
assert n_pitch_strings == config.PITCHES_MELODIC
|
| 49 |
+
|
| 50 |
+
# Percussion shape check
|
| 51 |
+
bars_p, steps_per_bar_p, n_perc = perc_grid.shape
|
| 52 |
+
assert bars_p == bars
|
| 53 |
+
assert steps_per_bar_p == config.STEPS_PER_BAR
|
| 54 |
+
assert n_perc == config.PITCHES_PERC
|
| 55 |
+
|
| 56 |
+
pm = pretty_midi.PrettyMIDI()
|
| 57 |
+
|
| 58 |
+
# Create instrument tracks
|
| 59 |
+
strings_inst = pretty_midi.Instrument(
|
| 60 |
+
program=pretty_midi.instrument_name_to_program("Violin")
|
| 61 |
+
)
|
| 62 |
+
woodwinds_inst = pretty_midi.Instrument(
|
| 63 |
+
program=pretty_midi.instrument_name_to_program("Clarinet")
|
| 64 |
+
)
|
| 65 |
+
brass_inst = pretty_midi.Instrument(
|
| 66 |
+
program=pretty_midi.instrument_name_to_program("Trumpet")
|
| 67 |
+
)
|
| 68 |
+
# Percussion uses channel 9; PrettyMIDI uses program but channel is set internally.
|
| 69 |
+
perc_inst = pretty_midi.Instrument(program=0, is_drum=True)
|
| 70 |
+
|
| 71 |
+
# Duration per step
|
| 72 |
+
step_duration = config.SECONDS_PER_STEP
|
| 73 |
+
|
| 74 |
+
# Helper to convert (bar_idx, step_idx, pitch_idx) to start/end times
|
| 75 |
+
def cell_times(bar_idx: int, step_idx: int) -> Tuple[float, float]:
|
| 76 |
+
start = (bar_idx * config.STEPS_PER_BAR + step_idx) * step_duration
|
| 77 |
+
end = start + step_duration # simple: one-step duration
|
| 78 |
+
return start, end
|
| 79 |
+
|
| 80 |
+
# Melodic families
|
| 81 |
+
def add_melodic_notes(grid: np.ndarray, instrument: pretty_midi.Instrument) -> None:
|
| 82 |
+
"""
|
| 83 |
+
Add notes to a melodic instrument from a [bars, steps, pitches] grid.
|
| 84 |
+
Each active cell becomes a note of one-step duration.
|
| 85 |
+
"""
|
| 86 |
+
for b in range(bars):
|
| 87 |
+
for s in range(config.STEPS_PER_BAR):
|
| 88 |
+
for p in range(config.PITCHES_MELODIC):
|
| 89 |
+
if grid[b, s, p] > 0.5: # treat >0.5 as "on"
|
| 90 |
+
start, end = cell_times(b, s)
|
| 91 |
+
midi_pitch = MELODIC_BASE_PITCH + p
|
| 92 |
+
note = pretty_midi.Note(
|
| 93 |
+
velocity=velocity,
|
| 94 |
+
pitch=midi_pitch,
|
| 95 |
+
start=start,
|
| 96 |
+
end=end,
|
| 97 |
+
)
|
| 98 |
+
instrument.notes.append(note)
|
| 99 |
+
|
| 100 |
+
add_melodic_notes(strings_grid, strings_inst)
|
| 101 |
+
add_melodic_notes(woodwinds_grid, woodwinds_inst)
|
| 102 |
+
add_melodic_notes(brass_grid, brass_inst)
|
| 103 |
+
|
| 104 |
+
# Percussion
|
| 105 |
+
# Map indices 0,1,2 to Kick, Snare, Hi-hat.
|
| 106 |
+
PERC_PITCHES = [PERC_KICK_PITCH, PERC_SNARE_PITCH, PERC_HIHAT_PITCH]
|
| 107 |
+
|
| 108 |
+
for b in range(bars):
|
| 109 |
+
for s in range(config.STEPS_PER_BAR):
|
| 110 |
+
for k in range(config.PITCHES_PERC):
|
| 111 |
+
if perc_grid[b, s, k] > 0.5:
|
| 112 |
+
start, end = cell_times(b, s)
|
| 113 |
+
pitch = PERC_PITCHES[k]
|
| 114 |
+
note = pretty_midi.Note(
|
| 115 |
+
velocity=velocity,
|
| 116 |
+
pitch=pitch,
|
| 117 |
+
start=start,
|
| 118 |
+
end=end,
|
| 119 |
+
)
|
| 120 |
+
perc_inst.notes.append(note)
|
| 121 |
+
|
| 122 |
+
# Append instruments to the PrettyMIDI object
|
| 123 |
+
pm.instruments.append(strings_inst)
|
| 124 |
+
pm.instruments.append(woodwinds_inst)
|
| 125 |
+
pm.instruments.append(brass_inst)
|
| 126 |
+
pm.instruments.append(perc_inst)
|
| 127 |
+
|
| 128 |
+
return pm
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
def save_midi(
|
| 132 |
+
pm: pretty_midi.PrettyMIDI,
|
| 133 |
+
out_path: str,
|
| 134 |
+
) -> str:
|
| 135 |
+
"""
|
| 136 |
+
Save PrettyMIDI object to a .mid file.
|
| 137 |
+
|
| 138 |
+
Args:
|
| 139 |
+
pm: PrettyMIDI object.
|
| 140 |
+
out_path: Path to write .mid file.
|
| 141 |
+
|
| 142 |
+
Returns:
|
| 143 |
+
out_path: Same path, for convenience.
|
| 144 |
+
"""
|
| 145 |
+
pm.write(out_path)
|
| 146 |
+
return out_path
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
# =========================
|
| 150 |
+
# Optional: MIDI → grid for debugging
|
| 151 |
+
# =========================
|
| 152 |
+
|
| 153 |
+
def midi_to_grids(
|
| 154 |
+
midi_path: str,
|
| 155 |
+
max_bars: Optional[int] = None,
|
| 156 |
+
) -> Dict[str, np.ndarray]:
|
| 157 |
+
"""
|
| 158 |
+
Very simple debugging helper: convert a MIDI file back into rough grids.
|
| 159 |
+
Assumes the file broadly follows the same tempo/grid assumptions.
|
| 160 |
+
|
| 161 |
+
This is intentionally approximate and mainly for inspection, not training.
|
| 162 |
+
|
| 163 |
+
Args:
|
| 164 |
+
midi_path: Path to a MIDI file.
|
| 165 |
+
max_bars: Optional max number of bars to decode.
|
| 166 |
+
|
| 167 |
+
Returns:
|
| 168 |
+
{
|
| 169 |
+
"strings": [bars, STEPS_PER_BAR, 12],
|
| 170 |
+
"woodwinds": [bars, STEPS_PER_BAR, 12],
|
| 171 |
+
"brass": [bars, STEPS_PER_BAR, 12],
|
| 172 |
+
"perc": [bars, STEPS_PER_BAR, 3],
|
| 173 |
+
}
|
| 174 |
+
"""
|
| 175 |
+
pm = pretty_midi.PrettyMIDI(midi_path)
|
| 176 |
+
|
| 177 |
+
# Estimate total length in bars
|
| 178 |
+
total_seconds = pm.get_end_time()
|
| 179 |
+
bar_duration = config.SECONDS_PER_BAR
|
| 180 |
+
total_bars = int(np.ceil(total_seconds / bar_duration))
|
| 181 |
+
|
| 182 |
+
if max_bars is not None:
|
| 183 |
+
total_bars = min(total_bars, max_bars)
|
| 184 |
+
|
| 185 |
+
# Initialize empty grids
|
| 186 |
+
strings_grid = np.zeros((total_bars, config.STEPS_PER_BAR, config.PITCHES_MELODIC), dtype=np.float32)
|
| 187 |
+
woodwinds_grid = np.zeros_like(strings_grid)
|
| 188 |
+
brass_grid = np.zeros_like(strings_grid)
|
| 189 |
+
perc_grid = np.zeros((total_bars, config.STEPS_PER_BAR, config.PITCHES_PERC), dtype=np.float32)
|
| 190 |
+
|
| 191 |
+
def time_to_indices(t: float) -> Tuple[int, int]:
|
| 192 |
+
"""
|
| 193 |
+
Convert time in seconds to (bar_idx, step_idx), clamped to grid.
|
| 194 |
+
"""
|
| 195 |
+
step_idx_global = int(np.floor(t / config.SECONDS_PER_STEP))
|
| 196 |
+
bar_idx = step_idx_global // config.STEPS_PER_BAR
|
| 197 |
+
step_idx = step_idx_global % config.STEPS_PER_BAR
|
| 198 |
+
bar_idx = min(bar_idx, total_bars - 1)
|
| 199 |
+
return bar_idx, step_idx
|
| 200 |
+
|
| 201 |
+
# Very naive family routing: you may refine this mapping later.
|
| 202 |
+
for inst in pm.instruments:
|
| 203 |
+
# Percussion
|
| 204 |
+
if inst.is_drum:
|
| 205 |
+
for note in inst.notes:
|
| 206 |
+
b, s = time_to_indices(note.start)
|
| 207 |
+
if note.pitch == PERC_KICK_PITCH:
|
| 208 |
+
k = 0
|
| 209 |
+
elif note.pitch == PERC_SNARE_PITCH:
|
| 210 |
+
k = 1
|
| 211 |
+
elif note.pitch == PERC_HIHAT_PITCH:
|
| 212 |
+
k = 2
|
| 213 |
+
else:
|
| 214 |
+
continue
|
| 215 |
+
perc_grid[b, s, k] = 1.0
|
| 216 |
+
else:
|
| 217 |
+
# Choose family by program range (very rough heuristic).
|
| 218 |
+
program = inst.program
|
| 219 |
+
# Strings (e.g., Violin, Viola, Cello)
|
| 220 |
+
if program in range(40, 49):
|
| 221 |
+
target_grid = strings_grid
|
| 222 |
+
# Woodwinds
|
| 223 |
+
elif program in range(72, 81) or program in range(64, 73):
|
| 224 |
+
target_grid = woodwinds_grid
|
| 225 |
+
# Brass
|
| 226 |
+
elif program in range(56, 65):
|
| 227 |
+
target_grid = brass_grid
|
| 228 |
+
else:
|
| 229 |
+
target_grid = strings_grid # fallback
|
| 230 |
+
|
| 231 |
+
for note in inst.notes:
|
| 232 |
+
b, s = time_to_indices(note.start)
|
| 233 |
+
pitch_idx = note.pitch - MELODIC_BASE_PITCH
|
| 234 |
+
if 0 <= pitch_idx < config.PITCHES_MELODIC:
|
| 235 |
+
target_grid[b, s, pitch_idx] = 1.0
|
| 236 |
+
|
| 237 |
+
return {
|
| 238 |
+
"strings": strings_grid,
|
| 239 |
+
"woodwinds": woodwinds_grid,
|
| 240 |
+
"brass": brass_grid,
|
| 241 |
+
"perc": perc_grid,
|
| 242 |
+
}
|
utils/sampling_utils.py
ADDED
|
@@ -0,0 +1,307 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Sampling and bar-level constraints for music generation.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
from typing import Dict
|
| 6 |
+
|
| 7 |
+
import numpy as np
|
| 8 |
+
import torch
|
| 9 |
+
import torch.nn.functional as F # noqa: F401
|
| 10 |
+
|
| 11 |
+
import config
|
| 12 |
+
|
| 13 |
+
# =========================
|
| 14 |
+
# Simple music-theory rules
|
| 15 |
+
# =========================
|
| 16 |
+
|
| 17 |
+
# C major scale degrees within one octave (0–11):
|
| 18 |
+
# C, D, E, F, G, A, B
|
| 19 |
+
C_MAJOR_INDICES = [0, 2, 4, 5, 7, 9, 11]
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def enforce_scale(
|
| 23 |
+
bar: Dict[str, np.ndarray],
|
| 24 |
+
allowed_indices=C_MAJOR_INDICES,
|
| 25 |
+
) -> Dict[str, np.ndarray]:
|
| 26 |
+
"""
|
| 27 |
+
Zero out melodic pitches not in the allowed scale for each melodic family.
|
| 28 |
+
|
| 29 |
+
Args:
|
| 30 |
+
bar: dict with keys "strings", "woodwinds", "brass", "perc".
|
| 31 |
+
allowed_indices: list of pitch indices (0–11) allowed in the scale.
|
| 32 |
+
|
| 33 |
+
Returns:
|
| 34 |
+
bar with non-scale pitches zeroed for melodic families.
|
| 35 |
+
"""
|
| 36 |
+
mask = np.zeros((config.PITCHES_MELODIC,), dtype=np.float32)
|
| 37 |
+
mask[allowed_indices] = 1.0
|
| 38 |
+
|
| 39 |
+
for key in ["strings", "woodwinds", "brass"]:
|
| 40 |
+
grid = bar[key] # [T, 12]
|
| 41 |
+
bar[key] = grid * mask # broadcast over time
|
| 42 |
+
|
| 43 |
+
return bar
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def limit_notes_per_step(
|
| 47 |
+
bar: Dict[str, np.ndarray],
|
| 48 |
+
max_notes_per_step: int = 4,
|
| 49 |
+
) -> Dict[str, np.ndarray]:
|
| 50 |
+
"""
|
| 51 |
+
For each melodic family and time step, restrict to at most max_notes_per_step notes.
|
| 52 |
+
|
| 53 |
+
This reduces dense vertical clusters and produces simpler chords.
|
| 54 |
+
"""
|
| 55 |
+
for key in ["strings", "woodwinds", "brass"]:
|
| 56 |
+
grid = bar[key] # [T, pitches]
|
| 57 |
+
T, P = grid.shape
|
| 58 |
+
for t in range(T):
|
| 59 |
+
on_indices = np.where(grid[t] > 0.5)[0]
|
| 60 |
+
if len(on_indices) > max_notes_per_step:
|
| 61 |
+
# Keep the first few; zero out the rest
|
| 62 |
+
to_keep = on_indices[:max_notes_per_step]
|
| 63 |
+
mask = np.zeros_like(grid[t])
|
| 64 |
+
mask[to_keep] = 1.0
|
| 65 |
+
grid[t] = mask
|
| 66 |
+
bar[key] = grid
|
| 67 |
+
return bar
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def count_total_notes(
|
| 71 |
+
bar: Dict[str, np.ndarray],
|
| 72 |
+
) -> int:
|
| 73 |
+
"""
|
| 74 |
+
Count total number of "on" notes across all instrument families in a bar.
|
| 75 |
+
"""
|
| 76 |
+
total = 0
|
| 77 |
+
for key in ["strings", "woodwinds", "brass", "perc"]:
|
| 78 |
+
total += np.count_nonzero(bar[key] > 0.5)
|
| 79 |
+
return int(total)
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def emphasize_strong_beats(
|
| 83 |
+
bar: Dict[str, np.ndarray],
|
| 84 |
+
) -> Dict[str, np.ndarray]:
|
| 85 |
+
"""
|
| 86 |
+
Encourage notes on strong beats (0, 2, 4, 6) for melodic families.
|
| 87 |
+
If a strong beat has no note in a family, randomly turn one on.
|
| 88 |
+
"""
|
| 89 |
+
strong_steps = [0, 2, 4, 6]
|
| 90 |
+
for key in ["strings", "woodwinds", "brass"]:
|
| 91 |
+
grid = bar[key] # [T, 12]
|
| 92 |
+
T, P = grid.shape
|
| 93 |
+
for t in strong_steps:
|
| 94 |
+
if t >= T:
|
| 95 |
+
continue
|
| 96 |
+
if np.sum(grid[t] > 0.5) == 0:
|
| 97 |
+
p = np.random.randint(0, P)
|
| 98 |
+
grid[t, p] = 1.0
|
| 99 |
+
bar[key] = grid
|
| 100 |
+
return bar
|
| 101 |
+
|
| 102 |
+
|
| 103 |
+
# =========================
|
| 104 |
+
# Logit → binary grid sampling
|
| 105 |
+
# =========================
|
| 106 |
+
|
| 107 |
+
def sample_binary_from_logits(
|
| 108 |
+
logits: torch.Tensor,
|
| 109 |
+
temperature: float = 1.0,
|
| 110 |
+
top_k: int = 0,
|
| 111 |
+
) -> torch.Tensor:
|
| 112 |
+
"""
|
| 113 |
+
Sample binary activations from logits for one instrument.
|
| 114 |
+
|
| 115 |
+
Args:
|
| 116 |
+
logits: [T, P] tensor (time steps, pitches).
|
| 117 |
+
temperature: Temperature for scaling; >1.0 → more random.
|
| 118 |
+
top_k: If >0, keep only top_k logits per time step before sampling.
|
| 119 |
+
|
| 120 |
+
Returns:
|
| 121 |
+
activations: [T, P] tensor of 0/1 (float).
|
| 122 |
+
"""
|
| 123 |
+
if temperature <= 0:
|
| 124 |
+
raise ValueError("temperature must be > 0")
|
| 125 |
+
|
| 126 |
+
# Scale by temperature
|
| 127 |
+
scaled_logits = logits / temperature
|
| 128 |
+
|
| 129 |
+
# Optional top-k per time step
|
| 130 |
+
if top_k > 0:
|
| 131 |
+
# Clamp k so it never exceeds number of pitches for this instrument
|
| 132 |
+
num_pitches = scaled_logits.shape[-1]
|
| 133 |
+
k = min(top_k, num_pitches)
|
| 134 |
+
|
| 135 |
+
values, indices = torch.topk(scaled_logits, k=k, dim=-1)
|
| 136 |
+
mask = torch.full_like(scaled_logits, float("-inf"))
|
| 137 |
+
mask.scatter_(-1, indices, values)
|
| 138 |
+
scaled_logits = mask
|
| 139 |
+
|
| 140 |
+
# Convert to probabilities
|
| 141 |
+
probs = torch.sigmoid(scaled_logits) # [T, P]
|
| 142 |
+
|
| 143 |
+
# Bernoulli sampling
|
| 144 |
+
bernoulli_samples = torch.bernoulli(probs)
|
| 145 |
+
|
| 146 |
+
return bernoulli_samples
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
def sample_bar_from_logits_dict(
|
| 150 |
+
logits_dict: Dict[str, torch.Tensor],
|
| 151 |
+
temperature: float = 1.0,
|
| 152 |
+
top_k: int = 0,
|
| 153 |
+
) -> Dict[str, np.ndarray]:
|
| 154 |
+
"""
|
| 155 |
+
Sample a full bar (all instruments) from per-instrument logits.
|
| 156 |
+
|
| 157 |
+
Args:
|
| 158 |
+
logits_dict:
|
| 159 |
+
"strings": [T, 12],
|
| 160 |
+
"woodwinds": [T, 12],
|
| 161 |
+
"brass": [T, 12],
|
| 162 |
+
"perc": [T, 3],
|
| 163 |
+
where T should be config.STEPS_PER_BAR.
|
| 164 |
+
temperature: Temperature for sampling.
|
| 165 |
+
top_k: top-k filter per instrument per time step.
|
| 166 |
+
|
| 167 |
+
Returns:
|
| 168 |
+
Dict of numpy arrays with shape [STEPS_PER_BAR, pitches] and 0/1 values.
|
| 169 |
+
"""
|
| 170 |
+
bar_grids: Dict[str, np.ndarray] = {}
|
| 171 |
+
|
| 172 |
+
for name, logits in logits_dict.items():
|
| 173 |
+
# Expect [T, P]
|
| 174 |
+
if logits.dim() == 3:
|
| 175 |
+
logits = logits.squeeze(0) # [T, P]
|
| 176 |
+
assert logits.shape[0] == config.STEPS_PER_BAR, f"{name}: wrong T"
|
| 177 |
+
|
| 178 |
+
samples = sample_binary_from_logits(
|
| 179 |
+
logits=logits,
|
| 180 |
+
temperature=temperature,
|
| 181 |
+
top_k=top_k,
|
| 182 |
+
) # [T, P]
|
| 183 |
+
|
| 184 |
+
bar_grids[name] = samples.cpu().numpy().astype(np.float32)
|
| 185 |
+
|
| 186 |
+
return bar_grids
|
| 187 |
+
|
| 188 |
+
|
| 189 |
+
# =========================
|
| 190 |
+
# Bar similarity and resampling
|
| 191 |
+
# =========================
|
| 192 |
+
|
| 193 |
+
def bar_similarity(
|
| 194 |
+
bar1: Dict[str, np.ndarray],
|
| 195 |
+
bar2: Dict[str, np.ndarray],
|
| 196 |
+
) -> float:
|
| 197 |
+
"""
|
| 198 |
+
Compute normalized similarity between two bars.
|
| 199 |
+
|
| 200 |
+
Both bars are dicts:
|
| 201 |
+
"strings": [STEPS_PER_BAR, 12],
|
| 202 |
+
"woodwinds": [STEPS_PER_BAR, 12],
|
| 203 |
+
"brass": [STEPS_PER_BAR, 12],
|
| 204 |
+
"perc": [STEPS_PER_BAR, 3],
|
| 205 |
+
|
| 206 |
+
Similarity is defined as:
|
| 207 |
+
1 - (Hamming distance / total_cells)
|
| 208 |
+
"""
|
| 209 |
+
flat1_list = []
|
| 210 |
+
flat2_list = []
|
| 211 |
+
for key in ["strings", "woodwinds", "brass", "perc"]:
|
| 212 |
+
g1 = bar1[key].astype(np.float32)
|
| 213 |
+
g2 = bar2[key].astype(np.float32)
|
| 214 |
+
flat1_list.append(g1.reshape(-1))
|
| 215 |
+
flat2_list.append(g2.reshape(-1))
|
| 216 |
+
|
| 217 |
+
flat1 = np.concatenate(flat1_list)
|
| 218 |
+
flat2 = np.concatenate(flat2_list)
|
| 219 |
+
|
| 220 |
+
total_cells = flat1.size
|
| 221 |
+
if total_cells == 0:
|
| 222 |
+
return 0.0
|
| 223 |
+
|
| 224 |
+
diff = np.abs(flat1 - flat2)
|
| 225 |
+
hamming = np.count_nonzero(diff)
|
| 226 |
+
|
| 227 |
+
similarity = 1.0 - (hamming / float(total_cells))
|
| 228 |
+
return float(similarity)
|
| 229 |
+
|
| 230 |
+
|
| 231 |
+
def count_active_families(
|
| 232 |
+
bar: Dict[str, np.ndarray],
|
| 233 |
+
) -> int:
|
| 234 |
+
"""
|
| 235 |
+
Count how many instrument families have at least one active note in the bar.
|
| 236 |
+
"""
|
| 237 |
+
active_count = 0
|
| 238 |
+
for key in ["strings", "woodwinds", "brass", "perc"]:
|
| 239 |
+
grid = bar[key]
|
| 240 |
+
if np.any(grid > 0.5):
|
| 241 |
+
active_count += 1
|
| 242 |
+
return active_count
|
| 243 |
+
|
| 244 |
+
|
| 245 |
+
def resample_bar_if_needed(
|
| 246 |
+
prev_bar: Dict[str, np.ndarray],
|
| 247 |
+
logits_dict: Dict[str, torch.Tensor],
|
| 248 |
+
temperature: float,
|
| 249 |
+
top_k: int,
|
| 250 |
+
similarity_threshold: float = config.BAR_SIMILARITY_THRESHOLD,
|
| 251 |
+
max_attempts: int = config.MAX_RESAMPLE_ATTEMPTS_PER_BAR,
|
| 252 |
+
min_active_families: int = config.MIN_ACTIVE_FAMILIES_PER_BAR,
|
| 253 |
+
min_notes_per_bar: int = 2,
|
| 254 |
+
max_notes_per_step: int = 4,
|
| 255 |
+
) -> Dict[str, np.ndarray]:
|
| 256 |
+
"""
|
| 257 |
+
Sample a bar from logits, enforcing:
|
| 258 |
+
- Not too similar to prev_bar (if prev_bar is provided).
|
| 259 |
+
- At least min_active_families have notes.
|
| 260 |
+
- At least min_notes_per_bar total notes.
|
| 261 |
+
- At most max_notes_per_step notes per step per melodic family.
|
| 262 |
+
"""
|
| 263 |
+
attempt = 0
|
| 264 |
+
best_bar = None
|
| 265 |
+
best_similarity = -1.0
|
| 266 |
+
|
| 267 |
+
while attempt < max_attempts:
|
| 268 |
+
attempt += 1
|
| 269 |
+
|
| 270 |
+
# Sample raw bar
|
| 271 |
+
candidate_bar = sample_bar_from_logits_dict(
|
| 272 |
+
logits_dict=logits_dict,
|
| 273 |
+
temperature=temperature,
|
| 274 |
+
top_k=top_k,
|
| 275 |
+
)
|
| 276 |
+
|
| 277 |
+
# Apply theory rules
|
| 278 |
+
candidate_bar = enforce_scale(candidate_bar)
|
| 279 |
+
candidate_bar = emphasize_strong_beats(candidate_bar)
|
| 280 |
+
candidate_bar = limit_notes_per_step(candidate_bar, max_notes_per_step=max_notes_per_step)
|
| 281 |
+
|
| 282 |
+
# Activity and density
|
| 283 |
+
active = count_active_families(candidate_bar)
|
| 284 |
+
total_notes = count_total_notes(candidate_bar)
|
| 285 |
+
|
| 286 |
+
# If no prev_bar, only activity + density constraints
|
| 287 |
+
if prev_bar is None:
|
| 288 |
+
if active >= min_active_families and total_notes >= min_notes_per_bar:
|
| 289 |
+
return candidate_bar
|
| 290 |
+
if best_bar is None:
|
| 291 |
+
best_bar = candidate_bar
|
| 292 |
+
continue
|
| 293 |
+
|
| 294 |
+
# Compare to previous bar
|
| 295 |
+
sim = bar_similarity(candidate_bar, prev_bar)
|
| 296 |
+
|
| 297 |
+
# Track best candidate for fallback
|
| 298 |
+
if sim > best_similarity:
|
| 299 |
+
best_similarity = sim
|
| 300 |
+
best_bar = candidate_bar
|
| 301 |
+
|
| 302 |
+
# Check constraints: not too similar, enough active families, enough notes
|
| 303 |
+
if (sim <= similarity_threshold) and (active >= min_active_families) and (total_notes >= min_notes_per_bar):
|
| 304 |
+
return candidate_bar
|
| 305 |
+
|
| 306 |
+
# Fallback
|
| 307 |
+
return best_bar
|