README.md

---
library_name: transformers
tags:
- audio
- spatial-audio
- ambisonics
- self-supervised
- sound-event-localization
license: mit
pipeline_tag: feature-extraction
---

# Bearings

Bearings is a self-supervised encoder for first-order Ambisonic (FOA) audio. It learns the spatial attributes of a sound field, direction and diffuseness, by reconstructing masked intensity-vector and diffuseness targets from a 7-channel log-mel + active-intensity representation. 
The encoder needs its tokens fused with a content encoder such as [GRAM-T](https://huggingface.co/labhamlet/gramt-mono) to solve sound event localisation and detection (SELD).

- **Repository:** https://github.com/root-goksenin/Embisonics
- **Developed by:** Goksenin Yuksel, goksenin.yuksel@ru.nl
- **License:** MIT


## Usage

With transformers (no clone needed, `pip install transformers torchaudio`):

```python
import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("labhamlet/bearings", trust_remote_code=True).eval()
wav = torch.randn(1, 4, int(48000 * 5.3))     # (batch, [W, Y, Z, X], samples), any sample rate

with torch.no_grad():
    out = model(wav, sampling_rate=48000)
out.last_hidden_state.shape   # (1, 600, 384): B x (T * F) x E, T = 75 time steps, F = 8 mel bands
out.timestamps.shape          # (600,) seconds, one per token, window centres: 0.04, ..., 5.96
out.n_valid_time              # tensor([67]): time steps that end before the zero padding
```

`strategy="frames"` returns (B, T, 8 * 384) per-frame vectors, `strategy="mean"` and `strategy="cls"` return one (B, 384) vector per clip. `AutoFeatureExtractor.from_pretrained("labhamlet/bearings", trust_remote_code=True)` does the resampling and padding separately if you want to cache waveforms.

With the research code, clone the repository and install `requirements.txt`, then:

```python
import torch
from src.hub import load_bearings

model = load_bearings("labhamlet/bearings", "bearings-base.ckpt")
wav = torch.randn(1, 4, int(48000 * 5.3))     # (batch, [W, Y, Z, X], samples), any sample rate

out = model.encode_waveform(wav, sr=48000)
out["embeddings"].shape    # (1, 600, 384): B x (T * F) x E, T = 75 time steps, F = 8 mel bands
out["timestamps"].shape    # (600,) seconds, one per token, window centres: 0.04, ..., 5.96
out["n_valid_time"]        # 67: time steps that end before the zero padding
```

`encode_waveform` resamples to 32 kHz, applies per-clip W-RMS normalisation, zero-pads to a whole number of 2 s windows, encodes each window and concatenates along time. The AutoModel forward does exactly the same and the two paths give identical embeddings.

**The token axis stacks the T x F grid time-major:** token `i` is time step `i // 8` and mel band `i % 8`. The 8 bands of a time step are adjacent and share a timestamp. Each time step covers 80 ms. `embeddings.view(B, n_time, n_freq, E)` restores the grid and `.reshape(B, n_time, 8 * 384)` gives the per-frame vectors the SELD probes consume.

## Evaluation

Used as a frozen spatial stream next to a frozen content encoder in SELD probes on TAU-NIGENS 2021 and STARSS23 (DCASE 2023 metrics: ER20, F20, LE, LR). Scripts and task ids are in `seld-dcase2023-gram/` of the repository.