Document 2M X-ARES and HEAR results and link the BEST-RQ-2 paper

a5c922f verified 8 days ago

12.5 kB

---
library_name: transformers
license: mit
tags:
  - audio
  - feature-extraction
  - custom_code
  - self-supervised-learning
  - audio-embeddings
  - audio-jepa
  - jepa
  - audioset
  - arxiv:2606.30700
---
# Audio-JEPA-base

[![BEST-RQ-2 paper: arXiv 2606.30700](https://img.shields.io/badge/arXiv-2606.30700-b31b1b.svg)](https://arxiv.org/abs/2606.30700)

> **Recommended: [BEST-RQ-2.2-base-2M](https://huggingface.co/ltuncay/BEST-RQ-2.2-base-2M)** has the highest reported overall X-ARES scores in this comparison (2M training steps, `overlap50` evaluation).

Audio-JEPA-base is a self-supervised audio encoder trained on AudioSet for
**200,000 optimizer steps**. It produces **768-dimensional** clip and frame
embeddings from mono **16 kHz** waveforms, and supports downstream fine-tuning.

Audio-JEPA learns by predicting the latent representations of masked spectrogram
patches from an exponential-moving-average teacher. This export contains the
**student encoder**, matching the research model's inference path; the pretraining
predictor and teacher are omitted.

**This is an updated training run, not an exact reproduction of the ICME 2025 model.**

| Version | Audio sample rate | Training steps |
| --- | --- | --- |
| Original ICME 2025 model ([ltuncay/Audio-JEPA](https://huggingface.co/ltuncay/Audio-JEPA)) | 32 kHz | 100,000 |
| This release, run `jp6l70l6` | 16 kHz | 200,000 |

The author reports better X-ARES results for this newer checkpoint. The Audio-JEPA
scores below are for this release and should not be presented as the original
ICME results. The original checkpoint remains available in its own repository.

This repository contains the trained encoder, its preprocessing configuration,
and the custom Transformers implementation. No installation of the research
repository is needed.

## X-ARES results

Scores on [X-ARES](https://arxiv.org/abs/2505.16369) (0–100, higher is better).
Audio-JEPA, BEST-RQ (Conformer), BEST-RQ (ViT), and the original BEST-RQ-2
variants below use the **same AudioSet split for 200,000 steps**.
**BEST-RQ-2.2 (2M)** uses **2,000,000 steps**, with batch size 128 per GPU
on two GPUs (effective batch 256). The pretrained baselines are shown for comparison.

| Model | Speech | Music | Environment | Global Mean | Mean of Means | Hugging Face model |
| --- | ---: | ---: | ---: | ---: | ---: | --- |
| data2vec | 50.62 | 23.24 | 15.41 | 37.83 | 29.76 | [data2vec-audio-base](https://huggingface.co/facebook/data2vec-audio-base) |
| wav2vec 2.0 | 41.79 | 34.94 | 29.52 | 37.84 | 35.42 | [wav2vec2-large-100k-voxpopuli](https://huggingface.co/facebook/wav2vec2-large-100k-voxpopuli) |
| Whisper | 49.19 | 38.67 | 28.61 | 42.75 | 38.82 | [whisper-base](https://huggingface.co/openai/whisper-base) |
| Audio-JEPA | 29.64 | 44.27 | 25.61 | 31.18 | 33.17 | [Audio-JEPA-base](https://huggingface.co/ltuncay/Audio-JEPA-base) |
| BEST-RQ (Conformer) | 40.43 | 35.58 | 30.81 | 37.43 | 35.60 | Separate codebase (recipe unavailable) |
| BEST-RQ (ViT) | 32.87 | 41.62 | 34.50 | 34.88 | 36.33 | [BEST-RQ-ViT](https://huggingface.co/ltuncay/BEST-RQ-ViT) |
| BEST-RQ-2 (Interspeech 2026) | 38.49 | 54.40 | 46.39 | 43.21 | 46.43 | [BEST-RQ-2](https://huggingface.co/ltuncay/BEST-RQ-2) |
| BEST-RQ-2.1 | 52.60 | 62.23 | 53.38 | 54.59 | 56.07 | [BEST-RQ-2.1-base](https://huggingface.co/ltuncay/BEST-RQ-2.1-base) |
| BEST-RQ-2.2 | 53.78 | 63.90 | 55.71 | 56.11 | 57.80 | [BEST-RQ-2.2-base](https://huggingface.co/ltuncay/BEST-RQ-2.2-base) |
| BEST-RQ-2.2 (2M) | 58.62 | 65.74 | 61.80 | 58.77 | 62.05 | [BEST-RQ-2.2-base-2M](https://huggingface.co/ltuncay/BEST-RQ-2.2-base-2M) |
| BEST-RQ-2.2 (2M, overlap50) | **60.87** | **65.78** | **65.90** | **61.13** | **64.18** | [BEST-RQ-2.2-base-2M](https://huggingface.co/ltuncay/BEST-RQ-2.2-base-2M) |

Global Mean averages all benchmark task scores. Mean of Means gives equal
weight to the Speech, Music, and Environment category means.

The two 2M rows report the same training checkpoint with different extraction
settings: [original evaluation](https://wandb.ai/tuncay-ludovic/X-ARES/runs/cw0rrt7o)
and [50% overlapping windows](https://wandb.ai/tuncay-ludovic/X-ARES/runs/a7a3fy9z).
Their scores use the logged **Combined** metric: equal weight for the linear-probe
and k-NN aggregates, excluding unavailable tasks separately for each method.
The earlier rows retain previously reported scores. Extraction and evaluation
settings differ across runs, so this is not a controlled training-duration ablation.

Audio-JEPA scores were supplied by the author for [run `jp6l70l6`](https://wandb.ai/tuncay-ludovic/audio%20embeddings/runs/jp6l70l6). The other 200k-model and pretrained-baseline scores are reported in the [project README](https://github.com/LudovicTuncay/audio-embeddings/blob/bb88bf790b1dcf8251c6b38e7a4766534adf33d3/README.md). These are reported research results, not a new benchmark run of the Transformers exports.

The Audio-JEPA row refers to the newer **16 kHz, 200,000-step** run, not the original ICME model (**32 kHz, 100,000 steps**). The author reports better results for this newer checkpoint.

## HEAR results

Scores on [HEAR](https://arxiv.org/abs/2203.03022) (0–100, higher is better),
using `overlap50_two_phase` extraction. All five project models cover all 19 tasks.
BEST-RQ-2.2 (2M) was trained for 2,000,000 steps; the other rows use 200,000 steps.

| Model | Speech | Music | Environment | Global Mean | Mean of Means | Coverage |
| --- | ---: | ---: | ---: | ---: | ---: | --- |
| Audio-JEPA | 36.53 | 74.20 | 51.85 | 56.42 | 54.19 | 19/19 |
| BEST-RQ-2 | 50.74 | 79.46 | 69.21 | 67.69 | 66.47 | 19/19 |
| BEST-RQ-2.1 | 64.74 | 86.47 | 70.83 | 75.50 | 74.02 | 19/19 |
| BEST-RQ-2.2 | 65.76 | 85.13 | 76.19 | 76.66 | 75.69 | 19/19 |
| BEST-RQ-2.2 (2M) | 70.18 | 85.38 | 81.82 | 79.64 | 79.13 | 19/19 |

The 2M scores come from [this W&B evaluation](https://wandb.ai/tuncay-ludovic/HEAR/runs/hear-v2-aa56c55deb6099f6); its checkpoint hash
matches the released 2M checkpoint. The earlier rows retain the reported
2026-09-19 W&B comparison. [Score sources and task-level 2M results](https://github.com/LudovicTuncay/audio-embeddings/blob/master/docs/data/best_rq_2_2_2m_results.json).
These are arithmetic means of primary task scores, with equal task weighting for
Global Mean and equal domain weighting for Mean of Means; they are not the
challenge's standardized ranking. These evaluations precede the Transformers export.

## Model and training

| Property | Value |
| --- | --- |
| Architecture | 12-layer Transformer, 768 dimensions, 12 attention heads |
| Input frontend | 128-bin mel spectrogram with a linear patch projection |
| Patch shape | 16 mel bins by 16 time frames |
| Transformer | Sinusoidal positional embeddings, LayerNorm, GELU MLP |
| Training data | AudioSet |
| Training objective | MSE prediction of EMA teacher representations |
| Teacher EMA decay | 0.996 → 1.0 |
| Masking ratio | 40–60% |
| Saved training step | 200,000, verified from checkpoint metadata |
| Exported weight dtype | float32 |
| Extraction policy | `overlap50_two_phase` |

## Training recipe

The matching experiment is
[`configs/experiment/audio_jepa/audioset/default.yaml`](https://github.com/LudovicTuncay/audio-embeddings/blob/bb88bf790b1dcf8251c6b38e7a4766534adf33d3/configs/experiment/audio_jepa/audioset/default.yaml).
The run used the earlier name `audio_jepa/baseline`. Its saved model settings,
optimizer, EMA schedule, data selection, batch size, seed and 200,000-step budget
match the current recipe. Newly explicit settings retain the defaults: white
masking noise, compilation disabled, and automatic RoPE mode (unused with the
saved sinusoidal positional embeddings). See `recipe_match.json` for the comparison.

In the research repository, reproduce the configuration with:

```bash
uv run src/train.py experiment=audio_jepa/audioset/default
uv run slurm_scripts/submit.py audio_jepa/audioset/default --gpus 1 --time 20:00:00 --dry-run
```

The Slurm command previews submission; remove `--dry-run` to submit on a configured
cluster. Dataset paths and worker counts depend on your environment. The recipe
matches the saved hyperparameters; exact numerical reproduction also depends on
the training-code version and runtime.

## Load the model

Install the runtime dependencies in your Python environment:

```bash
pip install "torch>=2.9.1" "torchaudio>=2.9.1" "timm>=0.9" "einops>=0.7" "transformers>=4.57,<6"
```

Use matching PyTorch and torchaudio versions. GPU installations may require the
appropriate PyTorch build for your CUDA version.

```python
import torch
from transformers import AutoFeatureExtractor, AutoModel
model_name = "ltuncay/Audio-JEPA-base"
processor = AutoFeatureExtractor.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True).eval()
audio = torch.zeros(processor.sampling_rate)  # Replace with real mono audio.
inputs = processor(audio, sampling_rate=processor.sampling_rate, return_tensors="pt")
with torch.inference_mode():
    outputs = model(**inputs)
clip_embeddings = outputs.pooler_output
frame_embeddings = outputs.last_hidden_state
output_dim = model.config.encoder_kwargs["embed_dim"]
```

Resample to `processor.sampling_rate` and downmix stereo before preprocessing.
The extractor performs padding only; spectrogram/convolution features are computed
inside the model. Pass the returned sample attention mask for variable durations.
Outputs include a frame attention mask and timestamps in milliseconds (-1 for padding).
`pooler_output` uses the saved HEAR extraction preset, including phase-balanced pooling.
Frame features average frequency patches; they are not the raw frequency-time ViT grid.

## Fine-tuning

For fine-tuning call `model.train()`, attach a task head and optimize its parameters
alongside the model. Save with `model.save_pretrained(path)` and
`processor.save_pretrained(path)`. This encoder export excludes pretraining predictors,
quantizers, teachers and optimizer state. Continue self-supervised research training
with the original Lightning code and checkpoints.

## Reproducibility and provenance

`revision` is optional; pin both loaders to the same full commit hash for reproducibility.
Custom Python code is included in this repository and requires `trust_remote_code=True`.
The weights use safetensors. See `export_manifest.json` for source and validation details.
Source run: [`jp6l70l6`](https://wandb.ai/tuncay-ludovic/audio%20embeddings/runs/jp6l70l6)
(access may require permission). The export used `checkpoints/last.ckpt` and the matching saved Hydra configuration.
The Lightning checkpoint was converted to safetensors before encoder extraction.
The manifest records the converted checkpoint and configuration hashes.

The exported model matched the pre-export encoder exactly on the exporter's test
batch. Loading with `AutoFeatureExtractor` and `AutoModel` was also checked in a
fresh Python process on unequal-duration inputs. These are integration checks,
not a downstream benchmark evaluation of this release.

## Intended uses and limitations

Use these embeddings as features for audio research, classification, retrieval,
or downstream fine-tuning. This is an encoder: it does not generate transcripts,
audio, or class labels without a downstream model.

AudioSet training does not establish performance on every language, acoustic
domain, demographic group, or downstream task. Evaluate the model on your target
data. No new downstream benchmark scores are claimed for this export.

The feature extractor pads waveforms and checks the sample rate; it does not
resample or downmix. Unequal-length clips are processed individually after padding
is removed. The output embeddings follow the saved windowing policy rather than
exposing the raw spectrogram token grid.

Browse the [BEST-RQ-2 family collection](https://huggingface.co/collections/ltuncay/best-rq-2-family-6aad761396dfe31da48ff91a).

## Source and citation

Research code: [audio-embeddings](https://github.com/LudovicTuncay/audio-embeddings).
If you use Audio-JEPA, cite:

```bibtex
@inproceedings{tuncay2025audio,
  title={Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning},
  author={Tuncay, Ludovic and Labb{\'e}, Etienne and Benetos, Emmanouil and Pellegrini, Thomas},
  booktitle={ICME 2025},
  year={2025}
}
```

## License

The model weights and bundled implementation are released under the MIT license.
See `LICENSE` and `CODE_LICENSE`.

	---
	library_name: transformers
	license: mit
	tags:
	- audio
	- feature-extraction
	- custom_code
	- self-supervised-learning
	- audio-embeddings
	- audio-jepa
	- jepa
	- audioset
	- arxiv:2606.30700
	---
	# Audio-JEPA-base

	[![BEST-RQ-2 paper: arXiv 2606.30700](https://img.shields.io/badge/arXiv-2606.30700-b31b1b.svg)](https://arxiv.org/abs/2606.30700)

	> Recommended: [BEST-RQ-2.2-base-2M](https://huggingface.co/ltuncay/BEST-RQ-2.2-base-2M) has the highest reported overall X-ARES scores in this comparison (2M training steps, `overlap50` evaluation).

	Audio-JEPA-base is a self-supervised audio encoder trained on AudioSet for
	200,000 optimizer steps. It produces 768-dimensional clip and frame
	embeddings from mono 16 kHz waveforms, and supports downstream fine-tuning.

	Audio-JEPA learns by predicting the latent representations of masked spectrogram
	patches from an exponential-moving-average teacher. This export contains the
	student encoder, matching the research model's inference path; the pretraining
	predictor and teacher are omitted.

	This is an updated training run, not an exact reproduction of the ICME 2025 model.

	\| Version \| Audio sample rate \| Training steps \|
	\| --- \| --- \| --- \|
	\| Original ICME 2025 model ([ltuncay/Audio-JEPA](https://huggingface.co/ltuncay/Audio-JEPA)) \| 32 kHz \| 100,000 \|
	\| This release, run `jp6l70l6` \| 16 kHz \| 200,000 \|

	The author reports better X-ARES results for this newer checkpoint. The Audio-JEPA
	scores below are for this release and should not be presented as the original
	ICME results. The original checkpoint remains available in its own repository.

	This repository contains the trained encoder, its preprocessing configuration,
	and the custom Transformers implementation. No installation of the research
	repository is needed.

	## X-ARES results

	Scores on [X-ARES](https://arxiv.org/abs/2505.16369) (0–100, higher is better).
	Audio-JEPA, BEST-RQ (Conformer), BEST-RQ (ViT), and the original BEST-RQ-2
	variants below use the same AudioSet split for 200,000 steps.
	BEST-RQ-2.2 (2M) uses 2,000,000 steps, with batch size 128 per GPU
	on two GPUs (effective batch 256). The pretrained baselines are shown for comparison.

	\| Model \| Speech \| Music \| Environment \| Global Mean \| Mean of Means \| Hugging Face model \|
	\| --- \| ---: \| ---: \| ---: \| ---: \| ---: \| --- \|
	\| data2vec \| 50.62 \| 23.24 \| 15.41 \| 37.83 \| 29.76 \| [data2vec-audio-base](https://huggingface.co/facebook/data2vec-audio-base) \|
	\| wav2vec 2.0 \| 41.79 \| 34.94 \| 29.52 \| 37.84 \| 35.42 \| [wav2vec2-large-100k-voxpopuli](https://huggingface.co/facebook/wav2vec2-large-100k-voxpopuli) \|
	\| Whisper \| 49.19 \| 38.67 \| 28.61 \| 42.75 \| 38.82 \| [whisper-base](https://huggingface.co/openai/whisper-base) \|
	\| Audio-JEPA \| 29.64 \| 44.27 \| 25.61 \| 31.18 \| 33.17 \| [Audio-JEPA-base](https://huggingface.co/ltuncay/Audio-JEPA-base) \|
	\| BEST-RQ (Conformer) \| 40.43 \| 35.58 \| 30.81 \| 37.43 \| 35.60 \| Separate codebase (recipe unavailable) \|
	\| BEST-RQ (ViT) \| 32.87 \| 41.62 \| 34.50 \| 34.88 \| 36.33 \| [BEST-RQ-ViT](https://huggingface.co/ltuncay/BEST-RQ-ViT) \|
	\| BEST-RQ-2 (Interspeech 2026) \| 38.49 \| 54.40 \| 46.39 \| 43.21 \| 46.43 \| [BEST-RQ-2](https://huggingface.co/ltuncay/BEST-RQ-2) \|
	\| BEST-RQ-2.1 \| 52.60 \| 62.23 \| 53.38 \| 54.59 \| 56.07 \| [BEST-RQ-2.1-base](https://huggingface.co/ltuncay/BEST-RQ-2.1-base) \|
	\| BEST-RQ-2.2 \| 53.78 \| 63.90 \| 55.71 \| 56.11 \| 57.80 \| [BEST-RQ-2.2-base](https://huggingface.co/ltuncay/BEST-RQ-2.2-base) \|
	\| BEST-RQ-2.2 (2M) \| 58.62 \| 65.74 \| 61.80 \| 58.77 \| 62.05 \| [BEST-RQ-2.2-base-2M](https://huggingface.co/ltuncay/BEST-RQ-2.2-base-2M) \|
	\| BEST-RQ-2.2 (2M, overlap50) \| 60.87 \| 65.78 \| 65.90 \| 61.13 \| 64.18 \| [BEST-RQ-2.2-base-2M](https://huggingface.co/ltuncay/BEST-RQ-2.2-base-2M) \|

	Global Mean averages all benchmark task scores. Mean of Means gives equal
	weight to the Speech, Music, and Environment category means.

	The two 2M rows report the same training checkpoint with different extraction
	settings: [original evaluation](https://wandb.ai/tuncay-ludovic/X-ARES/runs/cw0rrt7o)
	and [50% overlapping windows](https://wandb.ai/tuncay-ludovic/X-ARES/runs/a7a3fy9z).
	Their scores use the logged Combined metric: equal weight for the linear-probe
	and k-NN aggregates, excluding unavailable tasks separately for each method.
	The earlier rows retain previously reported scores. Extraction and evaluation
	settings differ across runs, so this is not a controlled training-duration ablation.

	Audio-JEPA scores were supplied by the author for [run `jp6l70l6`](https://wandb.ai/tuncay-ludovic/audio%20embeddings/runs/jp6l70l6). The other 200k-model and pretrained-baseline scores are reported in the [project README](https://github.com/LudovicTuncay/audio-embeddings/blob/bb88bf790b1dcf8251c6b38e7a4766534adf33d3/README.md). These are reported research results, not a new benchmark run of the Transformers exports.

	The Audio-JEPA row refers to the newer 16 kHz, 200,000-step run, not the original ICME model (32 kHz, 100,000 steps). The author reports better results for this newer checkpoint.

	## HEAR results

	Scores on [HEAR](https://arxiv.org/abs/2203.03022) (0–100, higher is better),
	using `overlap50_two_phase` extraction. All five project models cover all 19 tasks.
	BEST-RQ-2.2 (2M) was trained for 2,000,000 steps; the other rows use 200,000 steps.

	\| Model \| Speech \| Music \| Environment \| Global Mean \| Mean of Means \| Coverage \|
	\| --- \| ---: \| ---: \| ---: \| ---: \| ---: \| --- \|
	\| Audio-JEPA \| 36.53 \| 74.20 \| 51.85 \| 56.42 \| 54.19 \| 19/19 \|
	\| BEST-RQ-2 \| 50.74 \| 79.46 \| 69.21 \| 67.69 \| 66.47 \| 19/19 \|
	\| BEST-RQ-2.1 \| 64.74 \| 86.47 \| 70.83 \| 75.50 \| 74.02 \| 19/19 \|
	\| BEST-RQ-2.2 \| 65.76 \| 85.13 \| 76.19 \| 76.66 \| 75.69 \| 19/19 \|
	\| BEST-RQ-2.2 (2M) \| 70.18 \| 85.38 \| 81.82 \| 79.64 \| 79.13 \| 19/19 \|

	The 2M scores come from [this W&B evaluation](https://wandb.ai/tuncay-ludovic/HEAR/runs/hear-v2-aa56c55deb6099f6); its checkpoint hash
	matches the released 2M checkpoint. The earlier rows retain the reported
	2026-09-19 W&B comparison. [Score sources and task-level 2M results](https://github.com/LudovicTuncay/audio-embeddings/blob/master/docs/data/best_rq_2_2_2m_results.json).
	These are arithmetic means of primary task scores, with equal task weighting for
	Global Mean and equal domain weighting for Mean of Means; they are not the
	challenge's standardized ranking. These evaluations precede the Transformers export.

	## Model and training

	\| Property \| Value \|
	\| --- \| --- \|
	\| Architecture \| 12-layer Transformer, 768 dimensions, 12 attention heads \|
	\| Input frontend \| 128-bin mel spectrogram with a linear patch projection \|
	\| Patch shape \| 16 mel bins by 16 time frames \|
	\| Transformer \| Sinusoidal positional embeddings, LayerNorm, GELU MLP \|
	\| Training data \| AudioSet \|
	\| Training objective \| MSE prediction of EMA teacher representations \|
	\| Teacher EMA decay \| 0.996 → 1.0 \|
	\| Masking ratio \| 40–60% \|
	\| Saved training step \| 200,000, verified from checkpoint metadata \|
	\| Exported weight dtype \| float32 \|
	\| Extraction policy \| `overlap50_two_phase` \|

	## Training recipe

	The matching experiment is
	[`configs/experiment/audio_jepa/audioset/default.yaml`](https://github.com/LudovicTuncay/audio-embeddings/blob/bb88bf790b1dcf8251c6b38e7a4766534adf33d3/configs/experiment/audio_jepa/audioset/default.yaml).
	The run used the earlier name `audio_jepa/baseline`. Its saved model settings,
	optimizer, EMA schedule, data selection, batch size, seed and 200,000-step budget
	match the current recipe. Newly explicit settings retain the defaults: white
	masking noise, compilation disabled, and automatic RoPE mode (unused with the
	saved sinusoidal positional embeddings). See `recipe_match.json` for the comparison.

	In the research repository, reproduce the configuration with:

	```bash
	uv run src/train.py experiment=audio_jepa/audioset/default
	uv run slurm_scripts/submit.py audio_jepa/audioset/default --gpus 1 --time 20:00:00 --dry-run
	```

	The Slurm command previews submission; remove `--dry-run` to submit on a configured
	cluster. Dataset paths and worker counts depend on your environment. The recipe
	matches the saved hyperparameters; exact numerical reproduction also depends on
	the training-code version and runtime.

	## Load the model

	Install the runtime dependencies in your Python environment:

	```bash
	pip install "torch>=2.9.1" "torchaudio>=2.9.1" "timm>=0.9" "einops>=0.7" "transformers>=4.57,<6"
	```

	Use matching PyTorch and torchaudio versions. GPU installations may require the
	appropriate PyTorch build for your CUDA version.

	```python
	import torch
	from transformers import AutoFeatureExtractor, AutoModel

	model_name = "ltuncay/Audio-JEPA-base"
	processor = AutoFeatureExtractor.from_pretrained(model_name, trust_remote_code=True)
	model = AutoModel.from_pretrained(model_name, trust_remote_code=True).eval()
	audio = torch.zeros(processor.sampling_rate) # Replace with real mono audio.
	inputs = processor(audio, sampling_rate=processor.sampling_rate, return_tensors="pt")
	with torch.inference_mode():
	outputs = model(**inputs)
	clip_embeddings = outputs.pooler_output
	frame_embeddings = outputs.last_hidden_state
	output_dim = model.config.encoder_kwargs["embed_dim"]
	```

	Resample to `processor.sampling_rate` and downmix stereo before preprocessing.
	The extractor performs padding only; spectrogram/convolution features are computed
	inside the model. Pass the returned sample attention mask for variable durations.
	Outputs include a frame attention mask and timestamps in milliseconds (-1 for padding).
	`pooler_output` uses the saved HEAR extraction preset, including phase-balanced pooling.
	Frame features average frequency patches; they are not the raw frequency-time ViT grid.

	## Fine-tuning

	For fine-tuning call `model.train()`, attach a task head and optimize its parameters
	alongside the model. Save with `model.save_pretrained(path)` and
	`processor.save_pretrained(path)`. This encoder export excludes pretraining predictors,
	quantizers, teachers and optimizer state. Continue self-supervised research training
	with the original Lightning code and checkpoints.

	## Reproducibility and provenance

	`revision` is optional; pin both loaders to the same full commit hash for reproducibility.
	Custom Python code is included in this repository and requires `trust_remote_code=True`.
	The weights use safetensors. See `export_manifest.json` for source and validation details.
	Source run: [`jp6l70l6`](https://wandb.ai/tuncay-ludovic/audio%20embeddings/runs/jp6l70l6)
	(access may require permission). The export used `checkpoints/last.ckpt` and the matching saved Hydra configuration.
	The Lightning checkpoint was converted to safetensors before encoder extraction.
	The manifest records the converted checkpoint and configuration hashes.

	The exported model matched the pre-export encoder exactly on the exporter's test
	batch. Loading with `AutoFeatureExtractor` and `AutoModel` was also checked in a
	fresh Python process on unequal-duration inputs. These are integration checks,
	not a downstream benchmark evaluation of this release.

	## Intended uses and limitations

	Use these embeddings as features for audio research, classification, retrieval,
	or downstream fine-tuning. This is an encoder: it does not generate transcripts,
	audio, or class labels without a downstream model.

	AudioSet training does not establish performance on every language, acoustic
	domain, demographic group, or downstream task. Evaluate the model on your target
	data. No new downstream benchmark scores are claimed for this export.

	The feature extractor pads waveforms and checks the sample rate; it does not
	resample or downmix. Unequal-length clips are processed individually after padding
	is removed. The output embeddings follow the saved windowing policy rather than
	exposing the raw spectrogram token grid.

	Browse the [BEST-RQ-2 family collection](https://huggingface.co/collections/ltuncay/best-rq-2-family-6aad761396dfe31da48ff91a).

	## Source and citation

	Research code: [audio-embeddings](https://github.com/LudovicTuncay/audio-embeddings).
	If you use Audio-JEPA, cite:

	```bibtex
	@inproceedings{tuncay2025audio,
	title={Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning},
	author={Tuncay, Ludovic and Labb{\'e}, Etienne and Benetos, Emmanouil and Pellegrini, Thomas},
	booktitle={ICME 2025},
	year={2025}
	}
	```

	## License

	The model weights and bundled implementation are released under the MIT license.
	See `LICENSE` and `CODE_LICENSE`.