Encoder and DX7 catalogs of @audio /neural-timbre

2dd88e1 verified 16 days ago

8.38 kB

---
license: mit
library_name: onnx
pipeline_tag: feature-extraction
tags:
- audio
- timbre
- synthesizer
- preset-retrieval
- yamaha-dx7
- contrastive-learning
- onnx
---
# DX7 timbre: encoder and catalogs

A small encoder that maps one note to a 128-dimensional unit vector, and two catalogs of Yamaha DX7 voices, one vector
per voice, that name the sound by cosine. These are the files
[@audio/neural-timbre](https://github.com/audiojs/neural/tree/main/packages/neural-timbre)
([npm](https://www.npmjs.com/package/@audio/neural-timbre)) reads: each package version pins a commit of this
repository and checks every file against its SHA-256.

| File | Size | SHA-256 |
|---|---|---|
| `encoder.int8.onnx` | 1.87 MB (1,871,901 bytes) | `aaa9fae0337325fc9ae0f15c45f711073255c09264a31a9779da635ec21b8484` |
| `catalogs/dx7-factory.json` | 0.21 MB (209,548 bytes) | `7ab9d48c1b90a4ec04af3aeb63737352d63f959db8b1459d6b238cea365a0900` |
| `catalogs/dx7-all.json` | 7.9 MB (7,894,349 bytes) | `0dd0fd646e8aef7e417151e8ae76b5c2bec7fde3cb531e01a3bc4e5b964e3671` |

`SHA256SUMS` lists the same.

```js
import { createMatcher } from '@audio/neural-timbre'
let matcher = await createMatcher()                           // dx7-factory; createMatcher('dx7-all') for every voice
let [best] = await matcher.match(note, { sampleRate: 44100 })  // one note
best.name       // 'E.PIANO 1'
best.cartridge  // 'ROM1A'
best.score      // cosine, -1 to 1
matcher.free()
```

## Encoder

`encoder.int8.onnx`: four convolution stages (32, 64, 128, 256 channels), a 1×1 convolution to 96 channels, the mean
and maximum over time, two linear layers to a unit vector; 1.85M parameters, weights rounded to int8 per output row.

| | name | shape | |
|---|---|---|---|
| input | `features` | [n, 1, 96, 128] | constant-Q power spectrogram of one note from its onset: 96 bins 12 an octave from 41.2 Hz, 128 frames 512 samples apart at 44.1 kHz, dB below the loudest cell with an 80 dB floor, scaled to [−1, 1] (the package's `features.js`) |
| output | `embedding` | [n, 128] | the unit vector a catalog ranks against |
| output | `algorithm` | [n, 32] | the voice's DX7 algorithm, read linearly from the vector (weak: 37.1% right on held-out queries) |

Trained once with `scripts/train.py` of the package on 26,206 DX7 voices, eight rendered views each (any note 33–99,
velocity 20–127, held or staccato, two thirds through an augmentation chain: chorus, EQ, reverb, another voice or drums
underneath, noise, compression, MP3 or Opus): supervised contrastive loss (Khosla et al. 2020) and a margin softmax
against one proxy per voice (CosFace, Wang et al. 2018). The renders come from
[@audio/synth-dx7](https://github.com/audiojs/synth/tree/main/packages/synth-dx7), bit-exact to Google's Music
Synthesizer for Android engine. ONNX output matches PyTorch within 2·10⁻⁶ in Node and in Chromium on WebGPU. The recipe
is [gpu-font](https://github.com/dy/gpu-font)'s, applied to sound.

## Catalogs

Each voice rendered at MIDI 36, 48, 60, 72 and 84, velocities 64 and 112, held 1 s; the unit mean of the ten embeddings,
stored with one scale per row as 4-bit integers. Each catalog carries the encoder's and the features' SHA-256; the
package refuses a catalog bound to another encoder.

| Catalog | Voices | From |
|---|---:|---|
| `dx7-factory` | 988 | Yamaha's own cartridges: ROM1A–ROM4B (223 distinct sounds) and VRC-101 to VRC-112 (765) |
| `dx7-all` | 29,840 | every audible voice of every source below |

An entry holds a voice's `name`, its `cartridge` (ROM1A for Yamaha's, else the file's path in its archive), `index`
(its number in that file), `source`, `algorithm` (its DX7 algorithm number, 1–32) and `id` (the first 16 hex digits of a
SHA-1 of the parameters that shape its sound, so voices that render the same samples share it). **Names and vectors only**: no
sysex and no other voice parameter. To play a voice, load its cartridge from its source and take the voice by number.

## Benchmark

From the package's README, frozen before training: 2,925 voices from 247 cartridges never seen in training or
checkpoint selection, one query each at a note and velocity no catalog reference uses.

On 2,925 voices from cartridges it never saw, one clean note names the voice 75.0% of the time and puts it in the top
five 92.6%; through the augmentation chain, 56.4% and 79.0%. Those figures use 8-bit catalog vectors; the 4-bit
vectors hosted here:

| | held-out clean top-1 | top-5 | augmented top-1 | top-5 | all voices clean top-1 | top-5 |
|---|---:|---:|---:|---:|---:|---:|
| int8 encoder, int8 catalog | 75.0% | 92.6% | 56.4% | 79.0% | 45.0% | 75.8% |
| **int8 encoder, int4 catalog: these files** | **74.0%** | **92.1%** | **56.4%** | **79.0%** | **44.3%** | **75.1%** |

"All voices": the right voice among all 29,840, the training voices included. Per condition (reverb, EQ, compression,
chorus, codecs, noise, another voice underneath, drums) and the method:
[the package's README, Benchmark](https://github.com/audiojs/neural/tree/main/packages/neural-timbre#benchmark).

## Intended use

**Use when:** naming which preset of a known synth plays a recorded note (a DX7 factory sound, a cartridge voice), as
the seed of a rebuild in the original synth; indexing your own presets into a catalog and searching them by sound.

**Not for:** chords, layered or heavily processed parts without separating them first (another voice under the note at
0 dB: 28.6% top-1); synths no catalog covers (the closest preset comes back with a lower score, not a refusal: there is
no calibrated threshold); real-time use.

## Sources

The voices the encoder trained on and the catalogs name come from these sources:

| `source` | Sounds in `dx7-all` | What, from where | Terms stated |
|---|---:|---|---|
| `yamaha-factory` | 223 | Yamaha's ROM1A–ROM4B cartridges, from [yamahablackboxes.com](https://yamahablackboxes.com/collection/yamaha-dx7-synthesizer/patches/), byte-identical to rohandrape.net's copies | none; voice data © Yamaha Corporation, 1983 |
| `yamaha-vrc` | 765 | Yamaha's VRC-101 to VRC-112 cartridges, sides A and B, from the same site | none; © Yamaha and the named programmers |
| `greymatter` | 56 | Grey Matter Response E! card disks 2, 5 and 7, from the same site | none; © Grey Matter Response |
| `alltheweb` | 28,571 | [DX7 All The Web](https://bobbyblues.recup.ch/yamaha_dx7/dx7_patches.html): 13,192 sysex files gathered from the web by Bobby Blues | "public domain material" gleaned from the Net, per the collector, who removes material at its owners' request; it includes commercial banks |
| `library` | 225 | BlackWinny's [Yamaha-DX7-patch-library](https://github.com/visualizersdotnl/Yamaha-DX7-patch-library) v1.0 (2015), commit 6b17188 | CC0-1.0, marked by its uploader, who does not own every bank |

No source licenses all its voices for this use. The catalogs follow gpu-font's policy for DaFont and Adobe Fonts: names
and vectors only, published by the maintainer's decision, and a source comes out at its owner's request.

**Removal**: if you own voices listed here and want them out, open a discussion on this repository or an issue at
[github.com/audiojs/neural](https://github.com/audiojs/neural/issues), naming the source or cartridges. Their entries
are removed from the catalogs and a new revision published.

## Licence

The encoder weights (`encoder.int8.onnx`): MIT, Copyright (c) Dmitry Iv, as
[@audio/neural](https://github.com/audiojs/neural/blob/main/LICENSE).

The catalogs: the vectors are this encoder's output on renders of the voices above. The voice names, cartridge names and
file paths are the ones their authors and collectors gave; they are listed to identify the voices and stay theirs. No
rights over them are claimed or granted here.

## References

J. C. Brown, "Calculation of a constant Q spectral transform," *JASA* 89(1), 1991. · J. C. Brown, M. S. Puckette, "An
efficient algorithm for the calculation of a constant Q transform," *JASA* 92(5), 1992. · P. Khosla et al.,
"Supervised contrastive learning," *NeurIPS* 2020. · H. Wang et al., "CosFace: large margin cosine loss for deep face
recognition," *CVPR* 2018. · R. Levien, [Music Synthesizer for Android](https://github.com/google/music-synthesizer-for-android)
(the DX7 engine, Apache-2.0). · [gpu-font](https://github.com/dy/gpu-font) (the recipe).

	---
	license: mit
	library_name: onnx
	pipeline_tag: feature-extraction
	tags:
	- audio
	- timbre
	- synthesizer
	- preset-retrieval
	- yamaha-dx7
	- contrastive-learning
	- onnx
	---

	# DX7 timbre: encoder and catalogs

	A small encoder that maps one note to a 128-dimensional unit vector, and two catalogs of Yamaha DX7 voices, one vector
	per voice, that name the sound by cosine. These are the files
	[@audio/neural-timbre](https://github.com/audiojs/neural/tree/main/packages/neural-timbre)
	([npm](https://www.npmjs.com/package/@audio/neural-timbre)) reads: each package version pins a commit of this
	repository and checks every file against its SHA-256.

	\| File \| Size \| SHA-256 \|
	\|---\|---\|---\|
	\| `encoder.int8.onnx` \| 1.87 MB (1,871,901 bytes) \| `aaa9fae0337325fc9ae0f15c45f711073255c09264a31a9779da635ec21b8484` \|
	\| `catalogs/dx7-factory.json` \| 0.21 MB (209,548 bytes) \| `7ab9d48c1b90a4ec04af3aeb63737352d63f959db8b1459d6b238cea365a0900` \|
	\| `catalogs/dx7-all.json` \| 7.9 MB (7,894,349 bytes) \| `0dd0fd646e8aef7e417151e8ae76b5c2bec7fde3cb531e01a3bc4e5b964e3671` \|

	`SHA256SUMS` lists the same.

	```js
	import { createMatcher } from '@audio/neural-timbre'

	let matcher = await createMatcher() // dx7-factory; createMatcher('dx7-all') for every voice
	let [best] = await matcher.match(note, { sampleRate: 44100 }) // one note
	best.name // 'E.PIANO 1'
	best.cartridge // 'ROM1A'
	best.score // cosine, -1 to 1
	matcher.free()
	```

	## Encoder

	`encoder.int8.onnx`: four convolution stages (32, 64, 128, 256 channels), a 1×1 convolution to 96 channels, the mean
	and maximum over time, two linear layers to a unit vector; 1.85M parameters, weights rounded to int8 per output row.

	\| \| name \| shape \| \|
	\|---\|---\|---\|---\|
	\| input \| `features` \| [n, 1, 96, 128] \| constant-Q power spectrogram of one note from its onset: 96 bins 12 an octave from 41.2 Hz, 128 frames 512 samples apart at 44.1 kHz, dB below the loudest cell with an 80 dB floor, scaled to [−1, 1] (the package's `features.js`) \|
	\| output \| `embedding` \| [n, 128] \| the unit vector a catalog ranks against \|
	\| output \| `algorithm` \| [n, 32] \| the voice's DX7 algorithm, read linearly from the vector (weak: 37.1% right on held-out queries) \|

	Trained once with `scripts/train.py` of the package on 26,206 DX7 voices, eight rendered views each (any note 33–99,
	velocity 20–127, held or staccato, two thirds through an augmentation chain: chorus, EQ, reverb, another voice or drums
	underneath, noise, compression, MP3 or Opus): supervised contrastive loss (Khosla et al. 2020) and a margin softmax
	against one proxy per voice (CosFace, Wang et al. 2018). The renders come from
	[@audio/synth-dx7](https://github.com/audiojs/synth/tree/main/packages/synth-dx7), bit-exact to Google's Music
	Synthesizer for Android engine. ONNX output matches PyTorch within 2·10⁻⁶ in Node and in Chromium on WebGPU. The recipe
	is [gpu-font](https://github.com/dy/gpu-font)'s, applied to sound.

	## Catalogs

	Each voice rendered at MIDI 36, 48, 60, 72 and 84, velocities 64 and 112, held 1 s; the unit mean of the ten embeddings,
	stored with one scale per row as 4-bit integers. Each catalog carries the encoder's and the features' SHA-256; the
	package refuses a catalog bound to another encoder.

	\| Catalog \| Voices \| From \|
	\|---\|---:\|---\|
	\| `dx7-factory` \| 988 \| Yamaha's own cartridges: ROM1A–ROM4B (223 distinct sounds) and VRC-101 to VRC-112 (765) \|
	\| `dx7-all` \| 29,840 \| every audible voice of every source below \|

	An entry holds a voice's `name`, its `cartridge` (ROM1A for Yamaha's, else the file's path in its archive), `index`
	(its number in that file), `source`, `algorithm` (its DX7 algorithm number, 1–32) and `id` (the first 16 hex digits of a
	SHA-1 of the parameters that shape its sound, so voices that render the same samples share it). Names and vectors only: no
	sysex and no other voice parameter. To play a voice, load its cartridge from its source and take the voice by number.

	## Benchmark

	From the package's README, frozen before training: 2,925 voices from 247 cartridges never seen in training or
	checkpoint selection, one query each at a note and velocity no catalog reference uses.

	On 2,925 voices from cartridges it never saw, one clean note names the voice 75.0% of the time and puts it in the top
	five 92.6%; through the augmentation chain, 56.4% and 79.0%. Those figures use 8-bit catalog vectors; the 4-bit
	vectors hosted here:

	\| \| held-out clean top-1 \| top-5 \| augmented top-1 \| top-5 \| all voices clean top-1 \| top-5 \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| int8 encoder, int8 catalog \| 75.0% \| 92.6% \| 56.4% \| 79.0% \| 45.0% \| 75.8% \|
	\| int8 encoder, int4 catalog: these files \| 74.0% \| 92.1% \| 56.4% \| 79.0% \| 44.3% \| 75.1% \|

	"All voices": the right voice among all 29,840, the training voices included. Per condition (reverb, EQ, compression,
	chorus, codecs, noise, another voice underneath, drums) and the method:
	[the package's README, Benchmark](https://github.com/audiojs/neural/tree/main/packages/neural-timbre#benchmark).

	## Intended use

	Use when: naming which preset of a known synth plays a recorded note (a DX7 factory sound, a cartridge voice), as
	the seed of a rebuild in the original synth; indexing your own presets into a catalog and searching them by sound.

	Not for: chords, layered or heavily processed parts without separating them first (another voice under the note at
	0 dB: 28.6% top-1); synths no catalog covers (the closest preset comes back with a lower score, not a refusal: there is
	no calibrated threshold); real-time use.

	## Sources

	The voices the encoder trained on and the catalogs name come from these sources:

	\| `source` \| Sounds in `dx7-all` \| What, from where \| Terms stated \|
	\|---\|---:\|---\|---\|
	\| `yamaha-factory` \| 223 \| Yamaha's ROM1A–ROM4B cartridges, from [yamahablackboxes.com](https://yamahablackboxes.com/collection/yamaha-dx7-synthesizer/patches/), byte-identical to rohandrape.net's copies \| none; voice data © Yamaha Corporation, 1983 \|
	\| `yamaha-vrc` \| 765 \| Yamaha's VRC-101 to VRC-112 cartridges, sides A and B, from the same site \| none; © Yamaha and the named programmers \|
	\| `greymatter` \| 56 \| Grey Matter Response E! card disks 2, 5 and 7, from the same site \| none; © Grey Matter Response \|
	\| `alltheweb` \| 28,571 \| [DX7 All The Web](https://bobbyblues.recup.ch/yamaha_dx7/dx7_patches.html): 13,192 sysex files gathered from the web by Bobby Blues \| "public domain material" gleaned from the Net, per the collector, who removes material at its owners' request; it includes commercial banks \|
	\| `library` \| 225 \| BlackWinny's [Yamaha-DX7-patch-library](https://github.com/visualizersdotnl/Yamaha-DX7-patch-library) v1.0 (2015), commit 6b17188 \| CC0-1.0, marked by its uploader, who does not own every bank \|

	No source licenses all its voices for this use. The catalogs follow gpu-font's policy for DaFont and Adobe Fonts: names
	and vectors only, published by the maintainer's decision, and a source comes out at its owner's request.

	Removal: if you own voices listed here and want them out, open a discussion on this repository or an issue at
	[github.com/audiojs/neural](https://github.com/audiojs/neural/issues), naming the source or cartridges. Their entries
	are removed from the catalogs and a new revision published.

	## Licence

	The encoder weights (`encoder.int8.onnx`): MIT, Copyright (c) Dmitry Iv, as
	[@audio/neural](https://github.com/audiojs/neural/blob/main/LICENSE).

	The catalogs: the vectors are this encoder's output on renders of the voices above. The voice names, cartridge names and
	file paths are the ones their authors and collectors gave; they are listed to identify the voices and stay theirs. No
	rights over them are claimed or granted here.

	## References

	J. C. Brown, "Calculation of a constant Q spectral transform," JASA 89(1), 1991. · J. C. Brown, M. S. Puckette, "An
	efficient algorithm for the calculation of a constant Q transform," JASA 92(5), 1992. · P. Khosla et al.,
	"Supervised contrastive learning," NeurIPS 2020. · H. Wang et al., "CosFace: large margin cosine loss for deep face
	recognition," CVPR 2018. · R. Levien, [Music Synthesizer for Android](https://github.com/google/music-synthesizer-for-android)
	(the DX7 engine, Apache-2.0). · [gpu-font](https://github.com/dy/gpu-font) (the recipe).