| --- |
| library_name: transformers |
| tags: |
| - audio |
| - spatial |
| - speech |
| license: mit |
| datasets: |
| - agkphysics/AudioSet |
| metrics: |
| - accuracy |
| pipeline_tag: feature-extraction |
| --- |
| |
| # Model Card for Model ID |
|
|
| AudioSphere is trained on AudioSet with joint spatial-acoustic reconstruction loss. |
| AudioSphere supports ambisonics localization tasks unlike other models in the literature. Therefore, this model can be used for understanding spatial scenes as well as classifying mono sounds. |
|
|
|
|
| ## Model Details |
|
|
| AudioSphere earns spatial audio representation by reconstructing multi-channel masked spectrogram |
| patches. First, a patch extractor consisting of a single convolutional layer with 2D convolutional |
| filters divides each multi-channel spectrogram into n non-overlapping patches. Nonmasked patch embeddings are input to the encoder, for which we selected the 12-layer ViT-Base |
| (ViT-B) Transformer (Dosovitskiy et al., 2021) similar to Huang et al. (2022); Yadav et al. (2024). |
| The encoder outputs patch representations that can be further used for fine-tuning. |
|
|
| ### Model Description |
|
|
| AudioSphere is a self-supervised, multi-channel masked auto-encoder model that efficiently learns acoustic-spatial representations |
| from simulated real-world sound scenes. To train AudioSphere, we developed a custom pipeline which |
| makes use of the Soundspace 2.0 platform (Chen et al., 2022a) to simulate high-quality real-world |
| sound scenes from AudioSet (Gemmeke et al., 2017), and of WHAMR! (Maciejewski et al., 2020) |
| for adding background noise. |
|
|
|
|
| - **Developed by:** Goksenin Yuksel, goksenin.yuksel@ru.nl |
| - **Model type:** Transformers, Audio Foundation Models |
| - **Language(s) (NLP):** AudioSphere supports all languages, but mainly English. |
| - **License:** MIT |
|
|
| ### Model Sources |
|
|
| - **Repository:** https://github.com/labhamlet/AudioSphere |
| - **Paper:** https://arxiv.org/abs/2506.00934 |
|
|
| ## Uses |
|
|
| AudioSphere can be used as a powerful feature extractor for downstream tasks such as environmental sound classification, speech recognition, speaker counting, sound localization. |
| Later, training a linear head on top of these extracted features would yield a fine-tuned audio scene analysis model. |
|
|
|
|
| ## How to Get Started with the Model |
|
|
|
|
| AudioSphere have three strategies to choose from; "raw", "mean" or "cls". We advise to use "raw" strategy as this prooduces embeddings over the time frames, and is more robust to silent parts. |
|
|
| Raw : Break audio clips into non-overlapping 2 second chunks, concatenating the features in time and finally taking a mean over the time axis to generate a fixed vector representation independent of the input audio duration |
| Mean : Break audio clips into non-overlapping 2 second chunks, and taking a mean over all the patches to generate a fixed vector representation independent of the input audio duration |
| Cls : Break audio clips into non-overlapping 2 second chunks, and return the "[CLS]" token representation. |
| ~~~python |
| from transformers import AutoModel, AutoFeatureExtractor |
| |
| model = AutoModel.from_pretrained("labhamlet/AudioSphere", trust_remote_code=True) |
| extractor = AutoFeatureExtractor.from_pretrained("labhamlet/AudioSphere", trust_remote_code=True) |
| |
| audio = torch.zeros([1,4,320000]) |
| extracted = extractor(audio, return_tensors="pt") |
| log_mel = extracted['input_values'] |
| print(model(log_mel, strategy = "raw").shape) |
| ~~~ |
|
|
| ### Training Data |
|
|
| The 85,000 naturalistic scenes were split into a train set of 70,000 scenes (corresponding to 70 Matterport3D houses), and a test set of 15,000 scenes (15 |
| Matterport3D houses) for down-stream evaluation (see Section 3.4). We used the 70,000 naturalistic |
| scenes in the train set to generate naturalistic scenes for all audio clips in the unbalanced training |
| set of AudioSet (10-second sound tracks of 1.74 million YouTube videos (Gemmeke et al., 2017)). |
| Specifically, during training we randomly paired an AudioSet clip with a noise sound clip from the |
| WHAMR! background noise database (Maciejewski et al., 2020). WHAMR! noise clips longer than |
| 10 s were trimmed to 10 s duration and a linear fade-in/fade-out of 200 ms was applied to every noise |
| clip prior to mixing of the sound scene. |
|
|
| #### Preprocessing |
|
|
| Firstly, RMS Normalization was applied to audio clips to get all of them in the same loudness levels. Later, instance normalization was applied to the convolved scenes |
|
|
|
|
| ## Evaluation |
|
|
| We evaluate AudioSphere and other state-of-the-art models on the HEAR benchmark |
| task suite, which presents a wide range of tasks to evaluate the downstream performance of audio |
| representation models (Turian et al., 2022). We additionally evaluated performance on real-worldsound scenes using the newly released RealSELD benchmark. |
|
|
| #### Testing Data |
|
|
| **HEAR**: The HEAR benchmark aims to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios. |
|
|
| ### Results |
|
|
|
|
| | Model | DCASE | FSD50K | LC | ESC-50 | CD | VL | SC-5 | NS | BO | Mri-S | Mri-T | s(m) | |
| |-------|-------|--------|-------|--------|-------|-------|------|-----|-------|-------|-------|------| |
| | HEAR-Naive | 8.8 | 13.2 | 43.5 ± 1.6 | 28.6 ± 3.1 | 38.0 ± 2.3 | 14.8 ± 3.0 | 13.3 | 87.6 | 98.7 ± 1.9 | 94.1 ± 0.5 | 87.6 ± 6.4 | 0.0 | |
| | Wav2Vec 2.0 | 23.5 | 29.4 | 69.9 ± 2.1 | 46.4 ± 1.8 | 57.3 ± 1.1 | 34.9 ± 2.4 | 85.3 | 17.4 | 81.4 ± 4.8 | 90.7 ± 0.8 | 77.0 ± 0.9 | 31.5 | |
| | HuBERT | 78.3 | 32.8 | 63.3 ± 1.2 | 58.6 ± 2.8 | 71.2 ± 1.2 | 65.2 ± 2.9 | 94.0 | 19.8 | 93.2 ± 5.9 | 94.6 ± 0.4 | 85.0 ± 2.5 | 44.7 | |
| | WavLM | 27.0 | 25.7 | 61.3 ± 2.3 | 49.5 ± 3.8 | 64.3 ± 1.3 | 60.1 ± 3.2 | 93.8 | 18.2 | 84.3 ± 6.3 | 88.8 ± 1.0 | 76.8 ± 0.5 | 36.8 | |
| | MAE | – | 33.4 | 62.3 ± 1.1 | 72.9 ± 2.1 | 60.8 ± 1.8 | 21.3 ± 5.8 | 66.6 | 63.6 | 94.5 ± 5.6 | 94.8 ± 0.6 | 85.1 ± 10.4 | 32.7 | |
| | SSAST* | – | 21.4 | 57.8 ± 3.3 | 58.3 ± 2.6 | 48.0 ± 2.1 | 15.4 ± 2.6 | 22.0 | 64.2 | 95.8 ± 4.3 | 90.2 ± 5.9 | 89.1 ± 8.0 | 15.8 | |
| | BEATs | – | 54.1 | 77.8 ± 1.2 | 85.8 ± 2.9 | 66.9 ± 2.5 | 39.7 ± 4.3 | 86.9 | 68.6 | 94.1 ± 3.5 | 95.5 ± 0.4 | 96.6 ± 0.5 | 61.4 | |
| | MW-MAE | 94.2 | 51.8 | 80.3 ± 1.9 | 82.2 ± 3.2 | 74.4 ± 1.5 | 45.5 ± 1.7 | 91.6 | 69.4 | 95.8 ± 4.3 | 97.5 ± 0.4 | 97.6 ± 0.6 | 71.0 | |
| | SSAM | 87.3 | 53.5 | 75.5 ± 1.4 | 82.9 ± 3.6 | 70.2 ± 0.4 | 56.4 ± 5.2 | 89.3 | 72.6 | 93.2 ± 3.5 | 97.8 ± 0.5 | 96.9 ± 0.5 | 71.2 | |
| | **AudioSphere** | 94.3 | 53.0 | 79.4 ± 1.5 | 85.9 ± 1.5 | 71.9 ± 1.9 | 53.7 ± 1.2 | 89.6 | 73.8 | 94.9 ± 4.9 | 97.6 ± 0.5 | 98.5 ± 0.4 | 73.4 | |
|
|
|
|
| ## Model Card Contact |
|
|
| Goksenin Yuksel; goksenin.yuksel@ru.nl |
