ensemble: README.md

3797eb0 verified 11 days ago

6.96 kB

---
license: cc-by-nc-4.0
library_name: muscriptor
tags:
  - music
  - music-transcription
  - midi
  - dynamics
  - velocity
  - onset-detection
pipeline_tag: audio-to-audio
---
# MuScriptor Large — transcription toolkit (fork)

Two post-processors that make a MuScriptor Large transcription measurably better
without touching a single weight. Both run on CPU, in seconds, next to the model.

- **`transcript_fix.py`** — repairs what the decoder loses in difficult passages:
  onset timing, ghost notes, duplicates, and gaps.
- **`muscriptor_dynamics.py`** — adds the velocity MuScriptor does not predict, so
  forte and piano reach the MIDI.

Upstream weights: [`MuScriptor/muscriptor-large`](https://huggingface.co/MuScriptor/muscriptor-large)
(5.09 GB, 48 layers, dim 1536, card 1395). They are **not** copied here — see
*What is not here*.

## Why post-processing, and not a better model

MuScriptor's own output describes a note to about one analysis hop. In the model's
tokenizer, `velocity` is a two-valued flag (`1` for onset, `0` for offset) inside
`EventRange("velocity", 0, 1)`, and the whole output is a single head,
`linears.0.weight` of shape `[1395, 1536]`. There is no loudness dimension to
recover, and no code edit can create one: the extra rows would be untrained
numbers. So loudness is **measured** from the recording at the onsets the model
already placed, and the note-level errors are **repaired** against the same audio.

## `transcript_fix.py`
| stage | what it does |
| --- | --- |
| onset refinement | the flux peak lags the attack by roughly the analysis window, so each note's onset is moved to the sharpest **rise in local energy** (1 ms grid) |
| ghost removal | a note whose onset window is far below the piece's own floor was never struck |
| duplicate merge | same pitch within 45 ms is one note; the longer one wins |
| gap filling | an onset with no note nearby is filled only if autocorrelation is confident **and** the pitch is one this piece already uses, capped at a third of the note count |
Benchmark (`python3 transcript_fix.py --selftest`): a synthesized 12-note passage
with three notes lost, one ghost, one duplicate and ±20 ms onset jitter, scored
one-to-one at 50 ms tolerance.

```
до правки:  P=0.82 R=0.75 F1=0.78, ошибка онсета 10 мс, нот 11
после:      P=0.77 R=0.83 F1=0.80, ошибка онсета  2 мс, нот 13
```

**Onset error falls five-fold (10 ms → 2 ms), recall rises 0.75 → 0.83, F1 0.78 →
0.80, the ghost is removed.** Precision dips slightly (0.82 → 0.77) because three
notes are added and one of them is a semitone off — an honest trade, reported here
rather than hidden by tuning the benchmark.

## `muscriptor_dynamics.py`
Measures each note inside a 250 ms window at its onset: level in dB, and attack as
`peak/RMS`, which separates a hammer blow from a bowed swell at equal loudness. The
piece's own 4th and 99.7th percentiles become the reference, the two are combined
multiplicatively, and the result is written as velocity 1..124.
Self-test (`python3 muscriptor_dynamics.py --selftest`): eight notes of known
amplitudes return correlation **0.958**, soft `[3, 48, 27]` against loud
`[124, 124, 124]`.

## Usage

```python
from muscriptor_dynamics import load_audio, read_midi, write_midi
from transcript_fix import improve
samples, rate = load_audio("song.wav")
notes, tempo, _ = read_midi("muscriptor.mid")     # what the box returned
fixed, report = improve(notes, samples, rate)      # repair + velocity
write_midi("song.fixed.mid", fixed, 60000000.0 / tempo[0][1])
```

Both print progress in tens of percent. Both are one pass over the data: no
retries, no loops, no GPU.

## `merge_transcriptions.py` — the ensemble
A single pass makes two kinds of mistake that do not repeat: one pass drops the
blurry note in a fast run, another invents a ghost in a reverb tail. Running the
model a few times — different sizes, different instrument conditioning — and
keeping only what the passes agree on removes both.
```bash
python3 merge_transcriptions.py song.wav out.mid large.mid medium.mid piano.mid
```
Notes cluster by pitch and onset within 60 ms; a cluster survives on `--votes`
(default 2) distinct passes; its onset is the median of the passes, which is
steadier than any single one; velocity is then measured from the audio at that
agreed onset. `--keep-strong` rescues singletons whose attack is loud and clean —
a fortissimo note in a quiet passage is heard once and is real.
Benchmark (`--selftest`): three passes over the same 14 notes, each losing a
different three and inventing one ghost.
```
проход 1: P=0.92 R=0.79 F1=0.85
проход 2: P=0.92 R=0.79 F1=0.85
проход 3: P=0.92 R=0.79 F1=0.85
слияние:  P=1.00 R=1.00 F1=1.00, ошибка 4.6 мс, нот 14
отчёт: clusters 17 -> agreed 14, dropped_doubtful 3
```
**F1 0.85 → 1.00, every ghost gone, onset error 4.6 ms.** The gain is not free:
it costs one extra model pass per opinion, which is why this is the one global
improvement that does not need training but does need the GPU the model already
runs on.
## Limits, stated plainly
- **Overlapping notes share a measurement.** A chord's members measured in one
  window come out at similar velocities; per-voice dynamics needs source
  separation or a model that predicts velocity.
- **The audio must be the one that was transcribed.** No alignment step, so a
  different edit puts the windows in the wrong places.
- **An ensemble needs passes that actually differ.** Three runs with the same
  conditioning produce the same mistakes and vote them through.
- **Gap filling is conservative by design** and refuses ambiguous pitches; in
  dense legato writing it will leave gaps rather than invent notes.
- **Dynamics are relative to the piece**, since the reference points are its own
  percentiles.
- **The timing repair assumes percussive attacks.** On long bowed or sung notes
  the energy rise is gradual and the refinement moves less, which is correct but
  less dramatic.
## What is not here, and what a real retrain needs
Weights, configs and the training code are not mirrored: the weights are
CC-BY-NC-4.0 (non-commercial) and 5 GB, and copying them adds nothing to the code
that is published here.
Real model-level gains — velocity that survives overlapping voices, better recall
in blurred passages — need a velocity head added to the card (e.g. eight buckets:
`card` 1395 → 1403), trained on performance MIDI with per-note dynamics (MAESTRO
and similar) against the same audio, with the box's training code and a GPU for
days. That is a different project from this repository, and it is the only route
to dynamics that a post-processor cannot reach.
## License
Code in this repository: MIT, same as MuScriptor's own code. The upstream weights
it is designed for remain **CC-BY-NC-4.0**, so the combined pipeline is
non-commercial.

	---
	license: cc-by-nc-4.0
	library_name: muscriptor
	tags:
	- music
	- music-transcription
	- midi
	- dynamics
	- velocity
	- onset-detection
	pipeline_tag: audio-to-audio
	---

	# MuScriptor Large — transcription toolkit (fork)

	Two post-processors that make a MuScriptor Large transcription measurably better
	without touching a single weight. Both run on CPU, in seconds, next to the model.

	- `transcript_fix.py` — repairs what the decoder loses in difficult passages:
	onset timing, ghost notes, duplicates, and gaps.
	- `muscriptor_dynamics.py` — adds the velocity MuScriptor does not predict, so
	forte and piano reach the MIDI.

	Upstream weights: [`MuScriptor/muscriptor-large`](https://huggingface.co/MuScriptor/muscriptor-large)
	(5.09 GB, 48 layers, dim 1536, card 1395). They are not copied here — see
	What is not here.

	## Why post-processing, and not a better model

	MuScriptor's own output describes a note to about one analysis hop. In the model's
	tokenizer, `velocity` is a two-valued flag (`1` for onset, `0` for offset) inside
	`EventRange("velocity", 0, 1)`, and the whole output is a single head,
	`linears.0.weight` of shape `[1395, 1536]`. There is no loudness dimension to
	recover, and no code edit can create one: the extra rows would be untrained
	numbers. So loudness is measured from the recording at the onsets the model
	already placed, and the note-level errors are repaired against the same audio.

	## `transcript_fix.py`

	\| stage \| what it does \|
	\| --- \| --- \|
	\| onset refinement \| the flux peak lags the attack by roughly the analysis window, so each note's onset is moved to the sharpest rise in local energy (1 ms grid) \|
	\| ghost removal \| a note whose onset window is far below the piece's own floor was never struck \|
	\| duplicate merge \| same pitch within 45 ms is one note; the longer one wins \|
	\| gap filling \| an onset with no note nearby is filled only if autocorrelation is confident and the pitch is one this piece already uses, capped at a third of the note count \|

	Benchmark (`python3 transcript_fix.py --selftest`): a synthesized 12-note passage
	with three notes lost, one ghost, one duplicate and ±20 ms onset jitter, scored
	one-to-one at 50 ms tolerance.

	```
	до правки: P=0.82 R=0.75 F1=0.78, ошибка онсета 10 мс, нот 11
	после: P=0.77 R=0.83 F1=0.80, ошибка онсета 2 мс, нот 13
	```

	**Onset error falls five-fold (10 ms → 2 ms), recall rises 0.75 → 0.83, F1 0.78 →
	0.80, the ghost is removed.** Precision dips slightly (0.82 → 0.77) because three
	notes are added and one of them is a semitone off — an honest trade, reported here
	rather than hidden by tuning the benchmark.

	## `muscriptor_dynamics.py`

	Measures each note inside a 250 ms window at its onset: level in dB, and attack as
	`peak/RMS`, which separates a hammer blow from a bowed swell at equal loudness. The
	piece's own 4th and 99.7th percentiles become the reference, the two are combined
	multiplicatively, and the result is written as velocity 1..124.

	Self-test (`python3 muscriptor_dynamics.py --selftest`): eight notes of known
	amplitudes return correlation 0.958, soft `[3, 48, 27]` against loud
	`[124, 124, 124]`.

	## Usage

	```python
	from muscriptor_dynamics import load_audio, read_midi, write_midi
	from transcript_fix import improve

	samples, rate = load_audio("song.wav")
	notes, tempo, _ = read_midi("muscriptor.mid") # what the box returned
	fixed, report = improve(notes, samples, rate) # repair + velocity
	write_midi("song.fixed.mid", fixed, 60000000.0 / tempo[0][1])
	```

	Both print progress in tens of percent. Both are one pass over the data: no
	retries, no loops, no GPU.

	## `merge_transcriptions.py` — the ensemble

	A single pass makes two kinds of mistake that do not repeat: one pass drops the
	blurry note in a fast run, another invents a ghost in a reverb tail. Running the
	model a few times — different sizes, different instrument conditioning — and
	keeping only what the passes agree on removes both.

	```bash
	python3 merge_transcriptions.py song.wav out.mid large.mid medium.mid piano.mid
	```

	Notes cluster by pitch and onset within 60 ms; a cluster survives on `--votes`
	(default 2) distinct passes; its onset is the median of the passes, which is
	steadier than any single one; velocity is then measured from the audio at that
	agreed onset. `--keep-strong` rescues singletons whose attack is loud and clean —
	a fortissimo note in a quiet passage is heard once and is real.

	Benchmark (`--selftest`): three passes over the same 14 notes, each losing a
	different three and inventing one ghost.

	```
	проход 1: P=0.92 R=0.79 F1=0.85
	проход 2: P=0.92 R=0.79 F1=0.85
	проход 3: P=0.92 R=0.79 F1=0.85
	слияние: P=1.00 R=1.00 F1=1.00, ошибка 4.6 мс, нот 14
	отчёт: clusters 17 -> agreed 14, dropped_doubtful 3
	```

	F1 0.85 → 1.00, every ghost gone, onset error 4.6 ms. The gain is not free:
	it costs one extra model pass per opinion, which is why this is the one global
	improvement that does not need training but does need the GPU the model already
	runs on.

	## Limits, stated plainly

	- Overlapping notes share a measurement. A chord's members measured in one
	window come out at similar velocities; per-voice dynamics needs source
	separation or a model that predicts velocity.
	- The audio must be the one that was transcribed. No alignment step, so a
	different edit puts the windows in the wrong places.
	- An ensemble needs passes that actually differ. Three runs with the same
	conditioning produce the same mistakes and vote them through.
	- Gap filling is conservative by design and refuses ambiguous pitches; in
	dense legato writing it will leave gaps rather than invent notes.
	- Dynamics are relative to the piece, since the reference points are its own
	percentiles.
	- The timing repair assumes percussive attacks. On long bowed or sung notes
	the energy rise is gradual and the refinement moves less, which is correct but
	less dramatic.

	## What is not here, and what a real retrain needs

	Weights, configs and the training code are not mirrored: the weights are
	CC-BY-NC-4.0 (non-commercial) and 5 GB, and copying them adds nothing to the code
	that is published here.

	Real model-level gains — velocity that survives overlapping voices, better recall
	in blurred passages — need a velocity head added to the card (e.g. eight buckets:
	`card` 1395 → 1403), trained on performance MIDI with per-note dynamics (MAESTRO
	and similar) against the same audio, with the box's training code and a GPU for
	days. That is a different project from this repository, and it is the only route
	to dynamics that a post-processor cannot reach.

	## License

	Code in this repository: MIT, same as MuScriptor's own code. The upstream weights
	it is designed for remain CC-BY-NC-4.0, so the combined pipeline is
	non-commercial.