Update README.md

README.md

@@ -14,49 +14,47 @@ pipeline_tag: feature-extraction
 
 # Model Card for Model ID
 
-GRAM (General Purpose Audio Representation Model) is trained on AudioSet with newly proposed naturalistic training methadology.
-GRAMs utilize MWMAE (Multi-window multi-head attention), and RIR augmentations to achieve state-of-the-art results on downstream tasks such as FSD50K, ESC50, VL even when conditions are very adverse. What is different about GRAMs is the spatial reasoning capabilities, and the robustness to noise and reverberation.
-GRAMs natively support binaural localization, and ambisonics localization tasks unlike other models in the literature. Therefore, GRAMs can be used for understanding spatial scenes as well as classifying sounds, or recognizing speech in very noisy and reverberant environments.
+AudioSphere is trained on AudioSet with joint spatial-acoustic reconstruction loss.
+AudioSphere supports ambisonics localization tasks unlike other models in the literature. Therefore, this model can be used for understanding spatial scenes as well as classifying mono sounds.
 
 
 ## Model Details
 
-The GRAM learns spatial audio representation by reconstructing multi-channel masked spectrogram
+AudioSphere earns spatial audio representation by reconstructing multi-channel masked spectrogram
 patches. First, a patch extractor consisting of a single convolutional layer with 2D convolutional
 filters divides each multi-channel spectrogram into n non-overlapping patches. Nonmasked patch embeddings are input to the encoder, for which we selected the 12-layer ViT-Base
 (ViT-B) Transformer (Dosovitskiy et al., 2021) similar to Huang et al. (2022); Yadav et al. (2024).
 The encoder outputs patch representations that can be further used for fine-tuning.
+
 ### Model Description
 
-GRAM is a self-supervised, multi-channel masked auto-encoder model that efficiently learns spatial general-purpose audio representations
-from simulated real-world sound scenes. To train GRAM, we developed a custom pipeline which
+AudioSphere is a self-supervised, multi-channel masked auto-encoder model that efficiently learns acoustic-spatial representations
+from simulated real-world sound scenes. To train AudioSphere, we developed a custom pipeline which
 makes use of the Soundspace 2.0 platform (Chen et al., 2022a) to simulate high-quality real-world
 sound scenes from AudioSet (Gemmeke et al., 2017), and of WHAMR! (Maciejewski et al., 2020)
-for adding background noise. We present two versions of GRAM to ensure flexible application across audio formats:
-GRAM-Binaural for two-channel audio clips, and GRAM-Ambisonics for four-channel audio clips in
-the first-order Ambisonics format.
+for adding background noise. 
 
 
 - **Developed by:** Goksenin Yuksel, goksenin.yuksel@ru.nl
 - **Model type:** Transformers, Audio Foundation Models
-- **Language(s) (NLP):** GRAMs support all languages, but mainly English.
+- **Language(s) (NLP):** AudioSphere supports all languages, but mainly English.
 - **License:** MIT
 
 ### Model Sources
 
-- **Repository:** https://github.com/labhamlet/GRAM-T
+- **Repository:** https://github.com/labhamlet/AudioSphere
 - **Paper:** https://arxiv.org/abs/2506.00934
 
 ## Uses
 
-GRAMs can be used as a powerful feature extractor for downstream tasks such as enviromental sound classification, speech recognition, speaker counting, sound localization. 
+AudioSphere can be used as a powerful feature extractor for downstream tasks such as environmental sound classification, speech recognition, speaker counting, sound localization. 
 Later, training a linear head on top of these extracted features would yield a fine-tuned audio scene analysis model.
 
 
 ## How to Get Started with the Model
 
 
-GRAMs have three strategies to choose from; "raw", "mean" or "cls". We advise to use "raw" strategy as this prooduces embeddings over the time frames, and is more robust to silent parts.
+AudioSphere have three strategies to choose from; "raw", "mean" or "cls". We advise to use "raw" strategy as this prooduces embeddings over the time frames, and is more robust to silent parts.
 
 Raw : Break audio clips into non-overlapping 2 second chunks, concatenating the features in time and finally taking a mean over the time axis to generate a fixed vector representation independent of the input audio duration
 Mean : Break audio clips into non-overlapping 2 second chunks, and taking a mean over all the patches to generate a fixed vector representation independent of the input audio duration
@@ -64,8 +62,8 @@ Cls : Break audio clips into non-overlapping 2 second chunks, and return the "[C
 ~~~python
 from transformers import AutoModel, AutoFeatureExtractor
 
-model = AutoModel.from_pretrained("labhamlet/gramt-ambisonics", trust_remote_code=True)
-extractor = AutoFeatureExtractor.from_pretrained("labhamlet/gramt-ambisonics", trust_remote_code=True)
+model = AutoModel.from_pretrained("labhamlet/AudioSphere", trust_remote_code=True)
+extractor = AutoFeatureExtractor.from_pretrained("labhamlet/AudioSphere", trust_remote_code=True)
 
 audio = torch.zeros([1,4,320000])
 extracted = extractor(audio, return_tensors="pt")
@@ -73,9 +71,6 @@ log_mel = extracted['input_values']
 print(model(log_mel, strategy = "raw").shape)
 ~~~
 
-## Training Details
-
-
 ### Training Data
 
 The 85,000 naturalistic scenes were split into a train set of 70,000 scenes (corresponding to 70 Matterport3D houses), and a test set of 15,000 scenes (15
@@ -86,70 +81,21 @@ Specifically, during training we randomly paired an AudioSet clip with a noise s
 WHAMR! background noise database (Maciejewski et al., 2020). WHAMR! noise clips longer than
 10 s were trimmed to 10 s duration and a linear fade-in/fade-out of 200 ms was applied to every noise
 clip prior to mixing of the sound scene.
-To create a naturalistic sound scene, we then convolved the AudioSet clip either with BRIR(s, r, θ)
-for GRAM-Binaural, or with a ARIR(s, r, θ) for GRAM-Ambisonics, to obtain T. Similarly, we
-convolved the WHAMR! noise clip with the to obtain naturalistic scenes.
-
-
-### Training Procedure
-
-We transformed the channels of each sound scene (i.e., the waveforms) into logscale mel spectrograms using 128 mel filters in the frequency range of 50-16000 Hz with a 25 ms
-Hanning window and 10 ms hop length, resulting in spectrograms of dimension 1024 × 128. For
-GRAM-Ambisonic, we extracted normalized active Intensity Vectors (IVs) from the spectrograms
-as additional input features encoding spatial information. We concatenated mel
-spectrograms and intensity vectors, resulting in input x = [xmel, IV s] for each naturalistic scene
-generated from an AudioSet clip. In-batch sampling: As the online mixing of naturalistic acoustic scenes is computationally expensive
-due to multiple long convolutions, we used a random in-batch sampling procedure to increase the
-effective batch size in a computationally efficient manner. We randomly sampled 16 partially
-overlapping segments of 2 seconds to create 16 samples of dimension 200 × 128. This increases the
-original batch size of 96 to an effective batch size of 1536. 
-
-For pre-training, we divided the binaural spectrogram into 2×8×16,ambisonics spectrograms into 7×8×16 patches. We used an adapted version
-of the mask-based framework of MW-MAE (Yadav et al., 2024), randomly selecting a subset of
-n patches M1, . . . , Mn for i = 1, . . . , n for masking (masking ratio = 0.8) and replacing their
-embedding with a learnable mask token. Finally, we added fixed sinusoidal positional embeddings to
-all embedded patches.
-
-
-We trained all GRAMs for 500 K steps on an H100 92 GB GPU machine
-with 16 CPU cores. We used the AdamW optimizer (Loshchilov & Hutter, 2017) with weight decay
-rate of 0.01, gradient clipping, and a cosine learning rate scheduler with 10 K steps warm-up. The
-initial learning rate was set to 0.0002, and decayed to 0. We optimize the mean squared error (MSE)
-loss function between the predicted masked patches and their corresponding input spectrogram
-patches.
 
 #### Preprocessing
 
 Firstly, RMS Normalization was applied to audio clips to get all of them in the same loudness levels. Later, instance normalization was applied to the convolved scenes
 
 
-#### Training Hyperparameters
-
-- **Training regime:**: GRAMs were trained with mixed precision, torch.compile and flash attention.
-
 ## Evaluation
 
-We evaluate GRAM and other state-of-the-art models on the HEAR benchmark
+We evaluate AudioSphere and other state-of-the-art models on the HEAR benchmark
 task suite, which presents a wide range of tasks to evaluate the downstream performance of audio
-representation models (Turian et al., 2022). We additionally evaluated performance on simulated real-worldsound scenes using Nat-HEAR.
-### Testing Data, Factors & Metrics
+representation models (Turian et al., 2022). We additionally evaluated performance on real-worldsound scenes using the newly released RealSELD benchmark.
 
 #### Testing Data
 
-**HEAR**:  The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios. 
-HEAR evaluates audio representations using a benchmark suite across a variety of domains, including speech, environmental sound, and music. 
-HEAR was launched as a NeurIPS 2021 shared challenge. It still remains an open question whether one single general-purpose audio representation can perform as holistically as the human ear.
-
-**NatHEAR**: Orovides a naturalistic version of all selected datasets in the HEAR benchmark suite in
-two audio formats: a two-channel, binaural format and a four-channel, first-order Ambisonics
-format. Weincluded sound localization tasks for two different domains which we generated using
-HEAR benchmark datasets: A speech localization task based on SC-5, and an environmental sound
-localization task based on ESC-50. The localization tasks are modeled as a multi-output regression
-task in which model outputs represent the estimated 3D Cartesian coordinates [x, y, z] on the unit
-sphere (Adavanne et al., 2018). Finally, to assess the transferability of GRAM to real-world sound
-scenes, we evaluate also on the sound event detection and localization tasks in TUT Sound Events
-2018 REAL (Adavanne et al., 2019)
-
+**HEAR**:  The HEAR benchmark aims to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios. 
 
 ### Results
 
@@ -165,18 +111,8 @@ scenes, we evaluate also on the sound event detection and localization tasks in
 | BEATs | – | 54.1 | 77.8 ± 1.2 | 85.8 ± 2.9 | 66.9 ± 2.5 | 39.7 ± 4.3 | 86.9 | 68.6 | 94.1 ± 3.5 | 95.5 ± 0.4 | 96.6 ± 0.5 | 61.4 |
 | MW-MAE | 94.2 | 51.8 | 80.3 ± 1.9 | 82.2 ± 3.2 | 74.4 ± 1.5 | 45.5 ± 1.7 | 91.6 | 69.4 | 95.8 ± 4.3 | 97.5 ± 0.4 | 97.6 ± 0.6 | 71.0 |
 | SSAM | 87.3 | 53.5 | 75.5 ± 1.4 | 82.9 ± 3.6 | 70.2 ± 0.4 | 56.4 ± 5.2 | 89.3 | 72.6 | 93.2 ± 3.5 | 97.8 ± 0.5 | 96.9 ± 0.5 | 71.2 |
-| **GRAM-Binaural** | 95.6 | 56.1 | 81.0 ± 1.1 | 86.7 ± 2.4 | 75.0 ± 1.4 | 53.2 ± 3.0 | 92.5 | 77.0 | 94.9 ± 3.2 | 97.3 ± 0.3 | 98.1 ± 0.2 | 74.6 |
-| **GRAM-Ambisonics** | 94.3 | 53.0 | 79.4 ± 1.5 | 85.9 ± 1.5 | 71.9 ± 1.9 | 53.7 ± 1.2 | 89.6 | 73.8 | 94.9 ± 4.9 | 97.6 ± 0.5 | 98.5 ± 0.4 | 73.4 |
-| **GRAM-Mono** | 95.3 | 56.8 | 81.3 ± 1.8 | 87.5 ± 2.3 | 75.1 ± 0.6 | 57.3 ± 3.4 | 93.5 | 75.8 | 95.8 ± 3.7 | 97.4 ± 0.3 | 98.0 ± 0.2 | 76.1 |
-
-#### Summary
-
-We present a General-purpose, Real-world Audio representation Model (GRAM), which learns spatial
-audio representations using a multi-channel masked auto-encoder approach. GRAM demonstrates
-remarkable performance in naturalistic sound scenes as well as clean sound scenes, surpassing all
-state-of-the-art self-supervised spectrogram-based audio foundation models while requiring only a
-fraction of the training data. Moreover, GRAM is the first audio foundation model that is available
-in both a two-channel, binaural format and a four-channel, first-order ambisonics format
+| **AudioSphere** | 94.3 | 53.0 | 79.4 ± 1.5 | 85.9 ± 1.5 | 71.9 ± 1.9 | 53.7 ± 1.2 | 89.6 | 73.8 | 94.9 ± 4.9 | 97.6 ± 0.5 | 98.5 ± 0.4 | 73.4 |
+
 
 ## Model Card Contact