Update README.md

README.md

@@ -62,9 +62,7 @@ GRAMs have three strategies to choose from; "raw", "mean" or "cls". We advise to
 Raw : Break audio clips into non-overlapping 2 second chunks, concatenating the features in time and finally taking a mean over the time axis to generate a fixed vector representation independent of the input audio duration
 Mean : Break audio clips into non-overlapping 2 second chunks, and taking a mean over all the patches to generate a fixed vector representation independent of the input audio duration
 Cls : Break audio clips into non-overlapping 2 second chunks, and return the "[CLS]" token representation.
-'''
-python
-
+~~~python
 from transformers import AutoModel, AutoFeatureExtractor
 
 model = AutoModel.from_pretrained("labhamlet/gramt-ambisnonics", trust_remote_code=True)
@@ -74,7 +72,7 @@ audio = torch.zeros([1,4,320000])
 extracted = extractor(audio, return_tensors="pt")
 log_mel = extracted['input_values']
 print(model(log_mel, strategy = "raw").shape)
-'''
+~~~
 
 ## Training Details
 
@@ -139,11 +137,11 @@ representation models (Turian et al., 2022). We additionally evaluated performan
 
 #### Testing Data
 
-HEAR:  The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios. 
+**HEAR**:  The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios. 
 HEAR evaluates audio representations using a benchmark suite across a variety of domains, including speech, environmental sound, and music. 
 HEAR was launched as a NeurIPS 2021 shared challenge. It still remains an open question whether one single general-purpose audio representation can perform as holistically as the human ear.
 
-NatHEAR: Orovides a naturalistic version of all selected datasets in the HEAR benchmark suite in
+**NatHEAR**: Orovides a naturalistic version of all selected datasets in the HEAR benchmark suite in
 two audio formats: a two-channel, binaural format and a four-channel, first-order Ambisonics
 format. Weincluded sound localization tasks for two different domains which we generated using
 HEAR benchmark datasets: A speech localization task based on SC-5, and an environmental sound
@@ -156,19 +154,21 @@ scenes, we evaluate also on the sound event detection and localization tasks in
 
 ### Results
 
-Model DCASE FSD50K LC ESC-50 CD VL SC-5 NS BO Mri-S Mri-T s(m)
-HEAR-Naive 8.8 13.2 43.5 ± 1.6 28.6 ± 3.1 38.0 ± 2.3 14.8 ± 3.0 13.3 87.6 98.7 ± 1.9 94.1 ± 0.5 87.6 ± 6.4 0.0
-Wav2Vec 2.0 23.5 29.4 69.9 ± 2.1 46.4 ± 1.8 57.3 ± 1.1 34.9 ± 2.4 85.3 17.4 81.4 ± 4.8 90.7 ± 0.8 77.0 ± 0.9 31.5
-HuBERT 78.3 32.8 63.3 ± 1.2 58.6 ± 2.8 71.2 ± 1.2 65.2 ± 2.9 94.0 19.8 93.2 ± 5.9 94.6 ± 0.4 85.0 ± 2.5 44.7
-WavLM 27.0 25.7 61.3 ± 2.3 49.5 ± 3.8 64.3 ± 1.3 60.1 ± 3.2 93.8 18.2 84.3 ± 6.3 88.8 ± 1.0 76.8 ± 0.5 36.8
-MAE – 33.4 62.3 ± 1.1 72.9 ± 2.1 60.8 ± 1.8 21.3 ± 5.8 66.6 63.6 94.5 ± 5.6 94.8 ± 0.6 85.1 ± 10.4 32.7
-SSAST* – 21.4 57.8 ± 3.3 58.3 ± 2.6 48.0 ± 2.1 15.4 ± 2.6 22.0 64.2 95.8 ± 4.3 90.2 ± 5.9 89.1 ± 8.0 15.8
-BEATs – 54.1 77.8 ± 1.2 85.8 ± 2.9 66.9 ± 2.5 39.7 ± 4.3 86.9 68.6 94.1 ± 3.5 95.5 ± 0.4 96.6 ± 0.5 61.4
-MW-MAE 94.2 51.8 80.3 ± 1.9 82.2 ± 3.2 74.4 ± 1.5 45.5 ± 1.7 91.6 69.4 95.8 ± 4.3 97.5 ± 0.4 97.6 ± 0.6 71.0
-SSAM 87.3 53.5 75.5 ± 1.4 82.9 ± 3.6 70.2 ± 0.4 56.4 ± 5.2 89.3 72.6 93.2 ± 3.5 97.8 ± 0.5 96.9 ± 0.5 71.2
-GRAM-Binaural 95.6 56.1 81.0 ± 1.1 86.7 ± 2.4 75.0 ± 1.4 53.2 ± 3.0 92.5 77.0 94.9 ± 3.2 97.3 ± 0.3 98.1 ± 0.2 74.6
-GRAM-Ambisonics 94.3 53.0 79.4 ± 1.5 85.9 ± 1.5 71.9 ± 1.9 53.7 ± 1.2 89.6 73.8 94.9 ± 4.9 97.6 ± 0.5 98.5 ± 0.4 73.4
-GRAM-Clean 95.3 56.8 81.3 ± 1.8 87.5 ± 2.3 75.1 ± 0.6 57.3 ± 3.4 93.5 75.8 95.8 ± 3.7 97.4 ± 0.3 98.0 ± 0.2 76.1
+
+| Model | DCASE | FSD50K | LC | ESC-50 | CD | VL | SC-5 | NS | BO | Mri-S | Mri-T | s(m) |
+|-------|-------|--------|-------|--------|-------|-------|------|-----|-------|-------|-------|------|
+| HEAR-Naive | 8.8 | 13.2 | 43.5 ± 1.6 | 28.6 ± 3.1 | 38.0 ± 2.3 | 14.8 ± 3.0 | 13.3 | 87.6 | 98.7 ± 1.9 | 94.1 ± 0.5 | 87.6 ± 6.4 | 0.0 |
+| Wav2Vec 2.0 | 23.5 | 29.4 | 69.9 ± 2.1 | 46.4 ± 1.8 | 57.3 ± 1.1 | 34.9 ± 2.4 | 85.3 | 17.4 | 81.4 ± 4.8 | 90.7 ± 0.8 | 77.0 ± 0.9 | 31.5 |
+| HuBERT | 78.3 | 32.8 | 63.3 ± 1.2 | 58.6 ± 2.8 | 71.2 ± 1.2 | 65.2 ± 2.9 | 94.0 | 19.8 | 93.2 ± 5.9 | 94.6 ± 0.4 | 85.0 ± 2.5 | 44.7 |
+| WavLM | 27.0 | 25.7 | 61.3 ± 2.3 | 49.5 ± 3.8 | 64.3 ± 1.3 | 60.1 ± 3.2 | 93.8 | 18.2 | 84.3 ± 6.3 | 88.8 ± 1.0 | 76.8 ± 0.5 | 36.8 |
+| MAE | – | 33.4 | 62.3 ± 1.1 | 72.9 ± 2.1 | 60.8 ± 1.8 | 21.3 ± 5.8 | 66.6 | 63.6 | 94.5 ± 5.6 | 94.8 ± 0.6 | 85.1 ± 10.4 | 32.7 |
+| SSAST* | – | 21.4 | 57.8 ± 3.3 | 58.3 ± 2.6 | 48.0 ± 2.1 | 15.4 ± 2.6 | 22.0 | 64.2 | 95.8 ± 4.3 | 90.2 ± 5.9 | 89.1 ± 8.0 | 15.8 |
+| BEATs | – | 54.1 | 77.8 ± 1.2 | 85.8 ± 2.9 | 66.9 ± 2.5 | 39.7 ± 4.3 | 86.9 | 68.6 | 94.1 ± 3.5 | 95.5 ± 0.4 | 96.6 ± 0.5 | 61.4 |
+| MW-MAE | 94.2 | 51.8 | 80.3 ± 1.9 | 82.2 ± 3.2 | 74.4 ± 1.5 | 45.5 ± 1.7 | 91.6 | 69.4 | 95.8 ± 4.3 | 97.5 ± 0.4 | 97.6 ± 0.6 | 71.0 |
+| SSAM | 87.3 | 53.5 | 75.5 ± 1.4 | 82.9 ± 3.6 | 70.2 ± 0.4 | 56.4 ± 5.2 | 89.3 | 72.6 | 93.2 ± 3.5 | 97.8 ± 0.5 | 96.9 ± 0.5 | 71.2 |
+| **GRAM-Binaural** | 95.6 | 56.1 | 81.0 ± 1.1 | 86.7 ± 2.4 | 75.0 ± 1.4 | 53.2 ± 3.0 | 92.5 | 77.0 | 94.9 ± 3.2 | 97.3 ± 0.3 | 98.1 ± 0.2 | 74.6 |
+| **GRAM-Ambisonics** | 94.3 | 53.0 | 79.4 ± 1.5 | 85.9 ± 1.5 | 71.9 ± 1.9 | 53.7 ± 1.2 | 89.6 | 73.8 | 94.9 ± 4.9 | 97.6 ± 0.5 | 98.5 ± 0.4 | 73.4 |
+| **GRAM-Mono** | 95.3 | 56.8 | 81.3 ± 1.8 | 87.5 ± 2.3 | 75.1 ± 0.6 | 57.3 ± 3.4 | 93.5 | 75.8 | 95.8 ± 3.7 | 97.4 ± 0.3 | 98.0 ± 0.2 | 76.1 |
 
 #### Summary