--- license: other license_name: mit-with-attribution-requirement license_link: LICENSE tags: - music - midi - cc1 - onnx - safetensors --- # instrument-aire-models ## 1. Model introduction **AIRE** is a lightweight model family for **generating expressive MIDI control curves** from musical notes, designed for practical local inference in a web browser. It combines note-derived frame features with musical conditioning to predict a time-aligned expression curve while preserving the original notes. The current A02b release generates **CC1 only**, including movement within sustained notes; it does not generate notes or predict note velocity. CC1's audible effect depends on the receiving instrument and its controller mapping. | Model | Scope | Files | | --- | --- | --- | | S03 | Shared orchestral strings: violin, viola, cello, double bass | [safetensors](models/aire-strings-a02b-s03/model.safetensors) · [ONNX](models/aire-strings-a02b-s03/model.onnx) | | B01 | Brass family | [safetensors](models/aire-brass-a02b-b01/model.safetensors) · [ONNX](models/aire-brass-a02b-b01/model.onnx) | | W01 | Woodwind family | [safetensors](models/aire-woodwind-a02b-w01/model.safetensors) · [ONNX](models/aire-woodwind-a02b-w01/model.onnx) | | U01 | One model conditioned on string, woodwind or brass | [safetensors](models/aire-unified-a02b-u01/model.safetensors) · [ONNX](models/aire-unified-a02b-u01/model.onnx) | ## 2. Architecture, inputs and outputs ![A02b model architecture](assets/a02b-architecture.png) [Vector architecture diagram](assets/a02b-architecture.svg). A02b uses a width-96 backbone with two kernel-5 convolutions, four bidirectional encoder blocks, four attention heads and a 384-wide feed-forward layer. Frame embeddings and projections are **added**, together with a broadcast global control projection. Global conditioning combines mood (16-D), track role (8-D), instrument family (8-D) and six numerical controls through a 38→64→64 MLP. Independent FiLM modules condition attention and feed-forward paths. All encoder layers share a learned signed relative-attention bias table. A LayerNorm, linear 96→1 and sigmoid produce one normalized CC1 value per frame. The note-level input uses quarter-note beats, MIDI pitch 0–127 and sounding velocity 1–127. Tempo and meter come from the input MIDI; family, mood, track role and CC bounds are supplied controls. Notes are encoded at **16 frames per beat**, with a **one-beat tail** after the final note. Output events contain `beat`, continuous `value`, and half-up rounded integer `cc1`. Decode normalized predictions as `cc_min + y * (cc_max - cc_min)`. ### Actual four-bar strings example This example uses **C–Am–F–G (I–vi–IV–V)** in 4/4 at 120 BPM, with each triad sustained for one bar. `pad` is the **track role**; the mood is `peaceful`. The family is `string`, bounds are [0,127], and all original note velocities remain 64. | Bar | Beats | Chord | MIDI pitches | | --- | --- | --- | --- | | 1 | 0–4 | C | 60, 64, 67 | | 2 | 4–8 | Am | 57, 60, 64 | | 3 | 8–12 | F | 53, 57, 60 | | 4 | 12–16 | G | 55, 59, 62 | ![Actual S03 CC1 inference for the four-bar example](assets/s03-four-bars-cc1.png) The plot is the **actual S03 inference output**, not a hand-drawn target or smoothed curve. There are **272 frames**: 16 beats of notes plus one beat of tail. [Input MIDI](examples/four-bars-input.mid) · [MIDI with generated CC1](examples/four-bars-s03-cc1.mid) · [complete input JSON](examples/four-bars-request.json) · [complete inference output](examples/four-bars-s03-output.json). Input excerpt: ```json { "tempo_bpm": 120, "time_signature": [4, 4], "bar_origin_beats": 0, "controls": { "instrument": "string", "mood": "peaceful", "track_role": "pad", "cc_min": 0, "cc_max": 127, "use_velocity": false }, "notes": [ {"pitch": 60, "start": 0, "end": 4, "velocity": 64}, {"pitch": 64, "start": 0, "end": 4, "velocity": 64}, {"pitch": 67, "start": 0, "end": 4, "velocity": 64} ] } ``` The nine notes of the remaining three bars are omitted above; use the linked complete input for reproduction. First six output frames: ```json [ { "beat": 0.0, "value": 37.85149, "cc1": 38 }, { "beat": 0.0625, "value": 42.028435, "cc1": 42 }, { "beat": 0.125, "value": 40.26553, "cc1": 40 }, { "beat": 0.1875, "value": 40.282303, "cc1": 40 }, { "beat": 0.25, "value": 41.185257, "cc1": 41 }, { "beat": 0.3125, "value": 42.31284, "cc1": 42 } ] ``` … remaining 266 frames omitted. Continuous values above are displayed to six decimal places; the full output retains the computed values. The example was run with the released S03 safetensors weights on CPU; ONNX in Python, JavaScript in Node, and browser WASM reproduce the same MIDI events for this example. ### Low-level tensor interface The scripts handle this encoding. Direct ONNX callers must supply all ten tensors, with batch size 1 and dynamic frame count `T`: | Input | Type | Shape | | --- | --- | --- | | `lead_pitch` | int64 | [1,T] | | `pitch_roll`, `onset_roll`, `offset_roll` | float32 | [1,T,128] each | | `frame_numeric` | float32 | [1,T,25] | | `mood_id`, `role_id`, `instrument_id` | int64 | [1] each | | `ctrl_numeric` | float32 | [1,6] | | `mask` | bool | [1,T] | Output: `normalized_cc1`, float32 [1,T]. Lead pitch 128 represents rest. The mask is a valid prefix followed by right padding; rests and the tail are valid frames. Preprocessing is implemented in [Python](inference/encoder.py) and [JavaScript](inference/encoder.mjs), including the exact feature order and effective-velocity policy. ## 3. Relationship to MID-FiLD Both approaches generate fine-level CC1 expression conditioned on supplied notes and musical metadata, including instrument, mood, track role and expression bounds. The training material comes from MID-FiLD, whose dataset and task formulation made this project possible. | Aspect | MID-FiLD paper's demonstration model | AIRE-A02b release | | --- | --- | --- | | Representation | Tokenized note events and CC1 position/value events | Fixed frame grid with pitch rolls and numerical note features | | Architecture | Transformer encoder with an autoregressive decoder | Convolutions and a bidirectional encoder with FiLM | | Generation | Autoregressive CC1 token sequence | Whole-passage continuous CC1 predictions in one forward pass | | Instrument scope | Original dataset includes a broader instrument inventory | Selected orchestral strings, brass and woodwind; family-level conditioning | | Deployment goal | Demonstrate learning expressive dynamics from the dataset | Small FP32 models with ONNX and browser inference support | This is an independent architecture trained from fresh weights, rather than a reproduction of the paper's baseline or a fine-tune of its checkpoint. We do not claim to outperform MID-FiLD's original model; the evaluation protocols are not directly comparable. See the [original paper](https://ojs.aaai.org/index.php/AAAI/article/view/27774) for its model and tokenization details. ## 4. Usage and limits Download this repository's scripts and the model files you need. Run the commands below from the repository root. Scripts accept a **full model-file path**, an **input MIDI path** and a **new output MIDI path**. They replace CC1 on the note channel while preserving the original notes, velocities, tracks, programs, tempo, meter and other events at their absolute ticks. End-of-track may extend to the one-beat tail. Existing output files are not overwritten. ### Python Python 3.12 is the tested version. Install dependencies in your own environment: ```bash python -m pip install -r requirements.txt python inference/infer.py \ --model "/absolute/path/models/aire-strings-a02b-s03/model.safetensors" \ --input "/absolute/path/input.mid" \ --output "/absolute/path/output-cc1.mid" \ --mood peaceful --role pad ``` Use `model.onnx` in the same command for native ONNX inference. [The complete runnable Python script](inference/infer.py) includes preprocessing, inference and MIDI serialization. Programmatic example, saved as `run.py` in the repository root: ```python from pathlib import Path import sys sys.path.insert(0, str(Path(__file__).resolve().parent / "inference")) from infer import process_midi model_path, input_midi, output_midi = sys.argv[1:4] process_midi(model_path, input_midi, output_midi, model_id="S03", mood="peaceful", role="pad") ``` ### JavaScript / web Install the pinned ONNX Runtime Web dependency, then run the Node command-line example. It uses the same WASM runtime and 2,048-frame cap as the browser helper: ```bash npm install node inference/infer.mjs \ --model "/absolute/path/models/aire-strings-a02b-s03/model.onnx" \ --input "/absolute/path/input.mid" \ --output "/absolute/path/output-cc1.mid" \ --mood peaceful --role pad ``` [The complete runnable JavaScript script](inference/infer.mjs) accepts those paths. Programmatic example, saved as `run.mjs` in the repository root: ```js import * as ort from "onnxruntime-web"; import {readFile, writeFile} from "node:fs/promises"; import {inferMidi} from "./inference/infer-core.mjs"; const [modelPath, inputMidi, outputMidi] = process.argv.slice(2); const result = await inferMidi( ort, new Uint8Array(await readFile(modelPath)), new Uint8Array(await readFile(inputMidi)), {modelId: "S03", mood: "peaceful", role: "pad"} ); await writeFile(outputMidi, result.midi, {flag: "wx"}); ``` Browser applications can import `inferMidi` through a bundler and supply ONNX Runtime Web, model bytes and MIDI bytes from `fetch()` or a file picker. The returned `Uint8Array` can be downloaded as a MIDI Blob. Browsers cannot read arbitrary filesystem paths; absolute paths apply to the Node CLI. Serve the application over HTTP(S), keep runtime WASM files available, and run inference in a Worker to keep the UI responsive. The helper defaults to single-thread WASM; browser WebGPU support for the model graphs was validated separately, and application-level fallback must be handled by the caller. Both CLIs default to `peaceful` / `pad` / [0,127]. Use `--cc-min` and `--cc-max` for explicit bounds, `--disable-velocity` to disable velocity conditioning, and `--json-output` for the decoded frame events. If the model path does not identify S03/B01/W01/U01, pass `--model-id`. **U01 additionally requires `--family string`, `--family woodwind` or `--family brass`.** The same option values apply to Python and JavaScript. ### Limits - **Python/native cap: 4,096 frames. JavaScript/web cap: 2,048 frames.** For a nonempty request, `T = ceil(16 × (last_note_end_beats + 1))`; leading silence and the one-beat tail count toward the cap. No cropping, chunking or streaming is implemented. Native S03 ONNX was exercised at 4,096 frames; the public browser helper was exercised at 2,048 frames and rejects 2,049. - The MIDI examples support PPQ format 0/1, exactly one note-bearing track/channel, polyphonic chords, constant tempo and constant meter. They reject SMPTE timing, tempo/meter changes, unmatched or unclosed notes and overlapping identical pitches. Multi-voice inference is not implemented in these helpers. - CC1 output is voice-level, not per note or per chord tone. Original note velocity is preserved. Uniform input velocities disable velocity conditioning automatically. Mood, role and bounds are explicit controls; meaningful independent musical control has not been established. - These are experimental models. W01's full-range validation error exceeded its midpoint baseline; U01 did not improve the matched full-range errors of the separate family models. Final-test and perceptual quality remain unverified. A reference CC1 curve is not a unique correct musical answer. ## 5. Category order All categorical IDs are **zero based in the exact order listed**. Category lookup is included in [the inference helper configuration](inference/categories.json). Do not reuse one model's mood IDs for another model. ### S03 Family IDs: `string`. Mood IDs: `dreamy, groovy, hopeful, inspiring, magical, peaceful, relaxing, romantic, sad, tense, uplifting, happy, scary`. ### B01 Family IDs: `brass`. Mood IDs: `dreamy, groovy, hopeful, inspiring, magical, peaceful, relaxing, romantic, sad, tense, uplifting, bouncy, happy, mysterious, scary`. ### W01 Family IDs: `woodwind`. Mood IDs: `dreamy, groovy, hopeful, inspiring, magical, peaceful, relaxing, romantic, sad, tense, uplifting, funny, mysterious, scary, tragicomic`. ### U01 Family IDs: `string, woodwind, brass`. Mood IDs: `dreamy, groovy, hopeful, inspiring, magical, peaceful, relaxing, romantic, sad, tense, uplifting, happy, scary, funny, mysterious, tragicomic, bouncy`. Track-role IDs, shared by all models: `main_melody, pad, riff, sub_melody, accompaniment, bass`. ## About AIRE **AIRE** explores compact neural models that turn MIDI notes into continuous expression, with an emphasis on preserving the supplied performance and making inference accessible on the web. A02b uses an independently designed convolutional and bidirectional encoder architecture with FiLM conditioning, trained from fresh weights to predict the full CC1 curve in one forward pass. The project is **heavily inspired by [MID-FiLD: MIDI Dataset for Fine-Level Dynamics](https://ojs.aaai.org/index.php/AAAI/article/view/27774)** (AAAI 2024), and the released models are trained on filtered subsets of its dataset, available in the [official MID-FiLD GitHub repository](https://github.com/pozalabs/MID-FiLD). ## 6. License and required credit Author: **xiaohan-tian**. Organization: **KGAudioLab**. URL: https://huggingface.co/KGAudioLab. The models and accompanying code/documentation use **MIT + Attribution Requirement**; see [LICENSE](LICENSE). The extra mandatory credit condition makes this a custom license based on MIT, rather than unmodified MIT. Commercial use is permitted subject to those terms. If you use these models in a project, fine-tune or further train them, or retrain using these weights or adapting these models' architecture or implementation, credit **xiaohan-tian / KGAudioLab** and include https://huggingface.co/KGAudioLab in your project documentation or credits. Suggested wording: > Uses or is derived from instrument-aire models by xiaohan-tian / KGAudioLab (https://huggingface.co/KGAudioLab). Please also acknowledge the original MID-FiLD paper and dataset when describing the origin of the training material. Its [upstream repository](https://github.com/pozalabs/MID-FiLD) provides the original dataset license and attribution.