Instructions to use lchappuis/hear-you-scrape-clap with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use lchappuis/hear-you-scrape-clap with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("lchappuis/hear-you-scrape-clap", device_map="auto") - Notebooks
- Google Colab
- Kaggle
CLAP fine-tuned on scraping sounds (hear-you-scrape)
Sound encoders of CLAP fine-tuned to say if a scraped surface is wood, metal, textile or polymer. They come from the project hear-you-scrape and are shared for further research.
Read this first. These models are good on recording setups they were trained on, and worse than the original CLAP on a setup they have never seen. That is the main result of the project, and the reason we share them: to let others study why.
The files
Each file holds only the sound side of CLAP (audio_model and audio_projection). The text side is the original one.
| File | Trained on |
|---|---|
clap_without_<dataset>.safetensors (6 files) |
the five other public datasets; <dataset> was never seen |
clap_only_<dataset>.safetensors (6 files) |
that dataset alone |
clap_all_six_datasets.safetensors |
the six public datasets |
clap_all_six_datasets_and_our_phone_and_synthetic.safetensors |
the six datasets, a few phone recordings of ours and 2000 synthetic scrapes |
The six datasets: LMT108, LMT69, TUM building materials, SENS3, Cluster, Greatest Hits (see the repository for links). LMT108 and LMT69 share their surfaces, so when one is left out the other is left out too.
How to use
import torch
from safetensors.torch import load_file
from transformers import ClapModel, ClapProcessor
model = ClapModel.from_pretrained("laion/clap-htsat-unfused")
model.load_state_dict(load_file("clap_all_six_datasets.safetensors"), strict=False) # replaces the sound encoder
processor = ClapProcessor.from_pretrained("laion/clap-htsat-unfused")
Then compare a sound with sentences such as "the sound of scraping wood", as in the notebook
4_new_setup.ipynb.
The models were trained on windows of 0.5 s at 16 kHz, all at the same loudness, resampled to 48 kHz.
Training
Recipe of 6_fine_tuning.ipynb:
whole sound encoder, text side frozen, 1200 steps of 12 windows, learning rate 2e-5, one bias per training dataset.
Results
Balanced accuracy on four families (chance 0.25), on the dataset that was left out of the training:
| Left out | Original CLAP | clap_without_<dataset> |
|---|---|---|
| LMT108 | 0.53 | 0.36 |
| LMT69 | 0.65 | 0.58 |
| TUM | 0.34 | 0.28 |
| SENS3 | 0.32 | 0.12 |
| Cluster | 0.24 | 0.18 |
| Greatest Hits | 0.48 | 0.34 |
| mean | 0.43 | 0.31 |
On new surfaces of datasets seen in training, fine-tuning goes from about 0.45 to 0.70.
Licence and data
The base model is under Apache-2.0. The training datasets have their own terms: SENS3 is under CC BY-NC 4.0, others state no licence or allow research use. We therefore share these weights under CC BY-NC 4.0, for research. No recording is included. If you use them, please cite the datasets as well.
Model tree for lchappuis/hear-you-scrape-clap
Base model
laion/clap-htsat-unfused