CLAP fine-tuned on scraping sounds (hear-you-scrape)

Sound encoders of CLAP fine-tuned to say if a scraped surface is wood, metal, textile or polymer. They come from the project hear-you-scrape and are shared for further research.

Read this first. These models are good on recording setups they were trained on, and worse than the original CLAP on a setup they have never seen. That is the main result of the project, and the reason we share them: to let others study why.

The files

Each file holds only the sound side of CLAP (audio_model and audio_projection). The text side is the original one.

File Trained on
clap_without_<dataset>.safetensors (6 files) the five other public datasets; <dataset> was never seen
clap_only_<dataset>.safetensors (6 files) that dataset alone
clap_all_six_datasets.safetensors the six public datasets
clap_all_six_datasets_and_our_phone_and_synthetic.safetensors the six datasets, a few phone recordings of ours and 2000 synthetic scrapes

The six datasets: LMT108, LMT69, TUM building materials, SENS3, Cluster, Greatest Hits (see the repository for links). LMT108 and LMT69 share their surfaces, so when one is left out the other is left out too.

How to use

import torch
from safetensors.torch import load_file
from transformers import ClapModel, ClapProcessor

model = ClapModel.from_pretrained("laion/clap-htsat-unfused")
model.load_state_dict(load_file("clap_all_six_datasets.safetensors"), strict=False)  # replaces the sound encoder
processor = ClapProcessor.from_pretrained("laion/clap-htsat-unfused")

Then compare a sound with sentences such as "the sound of scraping wood", as in the notebook 4_new_setup.ipynb. The models were trained on windows of 0.5 s at 16 kHz, all at the same loudness, resampled to 48 kHz.

Training

Recipe of 6_fine_tuning.ipynb: whole sound encoder, text side frozen, 1200 steps of 12 windows, learning rate 2e-5, one bias per training dataset.

Results

Balanced accuracy on four families (chance 0.25), on the dataset that was left out of the training:

Left out Original CLAP clap_without_<dataset>
LMT108 0.53 0.36
LMT69 0.65 0.58
TUM 0.34 0.28
SENS3 0.32 0.12
Cluster 0.24 0.18
Greatest Hits 0.48 0.34
mean 0.43 0.31

On new surfaces of datasets seen in training, fine-tuning goes from about 0.45 to 0.70.

Licence and data

The base model is under Apache-2.0. The training datasets have their own terms: SENS3 is under CC BY-NC 4.0, others state no licence or allow research use. We therefore share these weights under CC BY-NC 4.0, for research. No recording is included. If you use them, please cite the datasets as well.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for lchappuis/hear-you-scrape-clap

Finetuned
(3)
this model