SLM-SS

Paper: arXiv:2610.27785

Code, installation, inference and evaluation: Herobrinelth/SLM-SS-Inference.

Model weights and matching configurations for the l_s and l_l SLM-SS autoregressive speech separators. The inference implementation uses a WavLM encoder, an autoregressive audio-token decoder and the single-codebook WavTokenizer codec.

Variant	Encoder	AR layers	AR hidden size	Checkpoint
l_s	WavLM Large	10	512	SLM-SS/checkpoints/l_s.ckpt
l_l	WavLM Large	12	1024	SLM-SS/checkpoints/l_l.ckpt

Both checkpoints contain model tensors and configuration, including fine-tuned WavLM parameters. They contain no optimizer or learning-rate scheduler state.

From the root of the cloned GitHub repository:

hf download Herobrinelth/SLM-SS \
  --include "SLM-SS/checkpoints/*.ckpt" "SLM-SS/configs/*.yaml" \
  --local-dir .

Third-party models

This model repository distributes only the two SLM-SS checkpoints and their YAML configurations. Obtain the codec and evaluation models directly from their respective upstream projects:

• WavTokenizer: novateur/WavTokenizer-large-speech-75token, wavtokenizer_large_speech_320_v2.ckpt.
• UTMOS22: the official sarulab-speech/UTMOS-demo Space provides epoch=3-step=7459.ckpt and wav2vec_small.pt.
• Fairseq wav2vec 2.0: the self-supervised backbone used by UTMOS.
• DNSMOS: DNSMOS/DNSMOS/sig_bak_ovr.onnx.
• ESPnet / OWSM: espnet/owsm_v3.1_ebf for WER evaluation.
• LPS: facebook/wav2vec2-lv-60-espeak-cv-ft, loaded through Transformers.
• SBS: utter-project/mHuBERT-147, using the precision calculation adapted from DiscreteSpeechMetrics.

Follow the GitHub README's Third-party model downloads section for exact download locations and the local checkpoint-conversion commands. Upstream WavTokenizer and UTMOS checkpoints need conversion before use with this repository's tensor-only loaders. The conversion preserves the downloaded originals and refuses to overwrite existing files.

Third-party models remain subject to their respective upstream terms. Their weights are not redistributed in this repository.

Inference and evaluation

Create the slmss Conda environment using the GitHub repository's environment.yml, download the SLM-SS checkpoints above, and prepare WavTokenizer as described in the GitHub README. Run from the source repository root:

conda activate slmss
python SLM-SS/inference.py \
  --model l_s --manifest examples/2mix/manifest.jsonl \
  --output-dir outputs/l_s

Use --model l_l and a different output directory for the larger variant. The example includes a two-speaker mixture, both GT waveforms and transcripts.

The evaluator runs WER, UTMOS, DNSMOS, PESQ, LPS and SBS by default, writing all six metrics into one summary.json and one samples.jsonl in the chosen output directory. Use --metrics to select a subset. Prepare the WER/MOS assets using the GitHub README and cache the LPS/SBS models, then evaluate:

hf download facebook/wav2vec2-lv-60-espeak-cv-ft --include "*.json" "model.safetensors"
hf download utter-project/mHuBERT-147 --include "*.json" "model.safetensors"

python SLM-SS/evaluate.py \
  --manifest examples/2mix/manifest.jsonl \
  --results outputs/l_s/results.jsonl \
  --metrics wer utmos dnsmos pesq lps sbs --offline \
  --output-dir outputs/l_s_metrics

• WER: OWSM espnet/owsm_v3.1_ebf, beam size 5 and Whisper basic text normalization. Report corpus-pooled WER and per-stream mean/std.
• UTMOS: the baseline model with domain 0, judge 288 and output 2*x+3.
• DNSMOS: the mean of calibrated SIG, BAK and OVRL scores.
• PESQ: 16 kHz wideband PESQ on each pair's common audio length.
• LPS: 1 - phoneme_edit_distance / reference_phoneme_count, using the wav2vec2 phoneme recognizer. Higher is better; negative values are valid.
• SBS: SpeechBERTScore precision, using mHuBERT-147 hidden_states[8]; this is not recall or F1.

WER uses reference transcripts; PESQ, LPS and SBS use GT audio. WER, PESQ, LPS and SBS each optimize their speaker assignments independently. MOS scores use the PESQ-matched streams by default. LPS and SBS resample to 16 kHz with soxr and preserve individual audio lengths. The shared reports include per-stream scores, aggregate mean/std, matching details and excluded-stream counts. Failed or truncated predictions are reported as errors.

Use --device cpu for CPU evaluation. --lps-model and --sbs-model accept local model directories as well as Hugging Face model IDs. Offline evaluation requires all selected model files to be cached or available locally.

All six metrics use the dependencies in environment.yml; no separate phonemizer/eSpeak or full discrete-speech-metrics installation is needed. See the GitHub README for the WER/MOS/PESQ definitions and reference-matching policies.