vdeturckheim/skywave-segmenter-small

Skywave Segmenter — small

Is a person speaking over the music right now?

Nothing in a radio station's metadata answers that. It says which record is playing, never that a presenter has started talking over it — and no public dataset labels it either, which is why this exists.

159,058 parameters. 192 KB as CoreML at int8. On an M5 Max it takes 3.1 ms to label a forty-second window end to end, features included: about 13,000× faster than the audio arrives.

What it does

Log-mel in, two classes out — music and speech — at 12.5 verdicts a second, for audio of any length. It is fully convolutional over time, so the same weights handle a three-second clip and a three-minute one.

"Speech" here means a human talking over or between records, which is a narrower question than voice activity. A sung vocal is music. That distinction is the whole difficulty: the two things that most resemble a presenter are a rapper and a singer.

Files

SkywaveSegmenter.mlpackage	CoreML, int8, what the app runs
skywave-segmenter.safetensors	the trained weights, MLX layout
skywave-segmenter-norm.safetensors	per-bin mean and standard deviation
skywave-segmenter.json	the model card the runtime reads: classes, stride, front end

The normalisation is folded into the CoreML graph, so a caller feeds raw log-mel and the model normalises internally. The safetensors weights are unnormalised and expect it applied.

The front end is part of the model

16 kHz mono · 400-sample window · 320-sample hop · 64 mel bands · 50–7600 Hz
log(energy + 1e-6) · 50 frames per second

These are not suggestions. The model learned on frames computed exactly this way, and frames computed any other way make it degrade quietly — which looks like a mediocre model rather than a mismatch. The reference implementation is SkywaveCore, which is checked against committed fixtures to 1.5e-04 on every build.

One detail that has cost real accuracy: decode to mono the way ffmpeg -ac 1 does, which divides a stereo pair by √2 rather than by 2 because it normalises to preserve power. The model reads absolute level, and averaging the channels instead is three decibels quieter — measured to be the difference between finding an announcement and reporting silence.

Results

Held out from training (four-hour chunks h20 and h12), scored end to end — through the smoothing and the span merging, counting announcements a listener would notice rather than frames:

Announcements found	33 of 33
Sung vocals wrongly flagged	1 of 15
Ordinary music wrongly flagged	0 of 40

The last row is the one that matters: a false positive pulls a listener off a lossless track mid-song, and it does not happen.

Do not quote a per-frame number for this model. They disagree with the end-to-end result badly enough to invert a ranking. A 4-bit quantization of this network agrees with its own float32 self on 81% of individual verdicts while losing six spans of eight — it reports 1.36 s of speech where there is 7.28 s. A model can look four fifths right and detect almost nothing, because the frames it loses are the ones carrying the announcements.

Size, and why this one

Twelve architectures were trained, 41k to 2.5M parameters. None of them can be told apart: every one lands at 32–33 of 33. With 33 announcements to be right about, the benchmark cannot separate an architecture from a seed — three seeds of one architecture gave 32–33 found and 1–3 sung false positives.

So small was chosen on what does not swing: it is small, and its sung false positives were 1 of 15 in all three seeds where the 41k variant gave 1 to 3. A tiny variant at 41,610 parameters measured equal and is published separately for anything where 41 KB matters more than margin.

int8 is free here — identical spans to float16, verified through the app's own inference path. Below eight bits nothing survives.

Training data, and its limits

28 hours of FIP (Radio France), labelled by an earlier transcribe-and-reason pipeline whose verdicts this model distils. The audio is not published and the derived corpus is private: it is a broadcaster's material, not ours.

This is a distillation of a teacher, not ground truth. It inherits that pipeline's judgement, including its mistakes — three of eighteen "hard negatives" in the original labelling were real station dressing rejected for the wrong reason, because FIP puns on its own name and the transcriber wrote "flippe" and "Philippe".

It knows one station. Trained on French-language presenters over predominantly anglophone music. On a station where the presenter and the songs share a language, the harder case, it has not been measured. The held-out split is also thin in one class: h12 contains no presenter frames at all, only music and station idents.

It cannot report a station ident. The corpus carries three classes — music, host, ident — and this model collapses the last two, so a jingle and a presenter both come back as speech. A three-class variant trained directly reaches only 25% ident recall: idents are 0.48% of frames, too thin for the class weighting used.

Using it

Swift, via skywave-model:

let segmenter = try Segmenter()            // reads the card beside the model
let spans = try segmenter.detail(samples)  // mono 16 kHz Float
for span in spans where span.label != "music" {
    print(span.start, span.end, span.confidence)
}

Verdicts become final 2.8 s after the audio: the receptive field is ±2.16 s, measured by perturbing one input frame and watching which outputs move, plus half the 1.25 s smoothing vote. Nothing beyond that window changes a verdict, so there is no reason to hand it more than about twelve seconds at a time.