README.md

---
pipeline_tag: automatic-speech-recognition
language:
- am
- ha
- sw
- hi
- mr
- ta
- ur
- en
- es
- fr
- id
- crs
tags:
- slam-asr
- tiny-aya
- whisper
- speech-recognition
---

# LisTAya: SLAM-ASR projectors for twelve languages on the Tiny Aya decoders

All checkpoints are listed in the collection [https://huggingface.co/collections/ERISLab/listaya-slam-asr-projectors-on-tiny-aya-rocling-2026-6ab031414095f3df8805ed7b](https://huggingface.co/collections/ERISLab/listaya-slam-asr-projectors-on-tiny-aya-rocling-2026-6ab031414095f3df8805ed7b). The transcripts of their test-set evaluations are the dataset [ERISLab/LisTAya-transcripts](https://huggingface.co/datasets/ERISLab/LisTAya-transcripts).

LisTAya (Listening Tiny Aya) is a family of speech-recognition models built with the SLAM-ASR recipe: a frozen Whisper-medium speech encoder, a frozen large-language-model decoder, and a trainable linear projector between them. Only the projector is trained. The checkpoints in this collection are the models behind the paper *Does Regional Decoder Specialization Help Low-Resource ASR Based on the SLAM-ASR Framework?* (ROCLING 2026). Each checkpoint is one (language, decoder) cell of that study.

## Decoders

Four Tiny Aya variants (3.35B parameters, one shared base model and tokenizer, differing only in the regional weighting of their post-training data): Global, Earth, Fire and Water. Two controls: Tiny Aya Base, the model before instruction tuning, and Qwen3-4B, a decoder from another family.

## Languages and training data

Every projector is trained on the WorldSpeech train split of one language (formal speech from parliamentary proceedings, broadcasts and audiobooks, clips under 30 seconds): Amharic (`am_et`), Hausa (`ha_ng`), Swahili (`sw_ke`), Hindi (`hi_in`), Marathi (`mr_in`), Tamil (`ta_in`), Urdu (`ur_pk`), English (`en_us`), Spanish (`es_mx`), French (`fr_ca`), Indonesian (`id_id`), Kreol Seselwa (`crs_sc`). The region is the Tiny Aya region each language is assigned to in the study; Kreol Seselwa is supported by neither the encoder nor any decoder and has no region. Its data come from the `ERISLab/WorldSpeech` config `crs_sc`; every other language from `disco-eth/WorldSpeech`.

## Training recipe

AdamW at learning rate 1.5e-3, batch size 8 with gradient accumulation 64 (effective batch 512), 1,000 steps, no weight decay, one NVIDIA H200, validation every 10 steps. Checkpoints are saved every 50 steps and the trainer keeps the one with the lowest validation loss on the FLEURS validation split, with two exceptions: Hausa is validated on the WorldSpeech test split, and Kreol Seselwa, which FLEURS does not cover, on a held-out 2% split of its WorldSpeech data.

## What each repository holds

Repositories are named `q2a_openai_whisper-medium_<decoder>_ws-<language>-<step>`. Each cell has two: the checkpoint the trainer kept as best on validation loss (the one the paper evaluates) and the final checkpoint at step 1000. Every repository holds the full trained model (encoder, projector and decoder weights, processor and tokenizer files) plus `trainer_state.json` with the run's validation curve.

## Load a checkpoint and transcribe

Every checkpoint is a `Qwen2AudioForConditionalGeneration` model and loads with the standard classes of `transformers` 4.57.5. The prompt below is the one the projectors were trained with; the audio is a 16 kHz mono array. The weights are stored in bfloat16; the paper's evaluations load them in float32, as below. Loading in bfloat16 halves the memory and changed 3 of 40 English validation transcripts in a check.

```python
import torch
from datasets import load_dataset
from transformers import AutoProcessor, Qwen2AudioForConditionalGeneration

repo = "ERISLab/q2a_openai_whisper-medium_CohereLabs_tiny-aya-global_ws-en_us-500"
processor = AutoProcessor.from_pretrained(repo)
model = Qwen2AudioForConditionalGeneration.from_pretrained(repo, dtype=torch.float32).to("cuda")

prompt = "<|audio_bos|><|AUDIO|><|audio_eos|>Transcribe the input speech: "
sample = next(iter(load_dataset("google/fleurs", "en_us", split="validation", streaming=True)))
audio = sample["audio"]["array"]  # 16 kHz mono float array
inputs = processor(text=prompt, audio=audio, sampling_rate=16000, return_tensors="pt").to("cuda")
with torch.no_grad():
    generated = model.generate(**inputs, max_new_tokens=256, do_sample=False)
hypothesis = processor.batch_decode(generated[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]
```

On this clip the English Global checkpoint writes "when you call someone who is 1000s of miles away you are using a satellite"; the reference is "when you call someone who is thousands of miles away you are using a satellite".

## Validation CER of the kept checkpoint

Character error rate (%) of each kept checkpoint on its validation split, read from its `trainer_state.json`. This is the score of the released model, not the minimum of the run's validation curve, which the paper reports as best CER.

| Language | Region | Global | Earth | Fire | Water | Base | Qwen3-4B |
|---|---|---|---|---|---|---|---|
| Amharic (`am_et`) | Earth | 35.1 | 36.3 | 31.0 | 28.6 | 47.3 | 39.5 |
| Hausa (`ha_ng`) | Earth | 37.1 | 37.2 | 36.9 | 38.7 | 36.6 | 34.1 |
| Swahili (`sw_ke`) | Earth | 15.6 | 17.2 | 22.9 | 19.7 | 19.2 | 14.7 |
| Hindi (`hi_in`) | Fire | 15.7 | 18.8 | 18.5 | 16.4 | 12.2 | 15.5 |
| Marathi (`mr_in`) | Fire | 14.2 | 14.4 | 14.7 | 19.4 | 14.3 | 15.4 |
| Tamil (`ta_in`) | Fire | 84.5 | 72.9 | 47.3 | 80.6 | 45.7 | 53.4 |
| Urdu (`ur_pk`) | Fire | 33.1 | 21.8 | 26.2 | 25.5 | 25.3 | 48.4 |
| English (`en_us`) | Water | 10.2 | 7.2 | 4.4 | 14.8 | 5.5 | 7.2 |
| Spanish (`es_mx`) | Water | 3.5 | 5.6 | 3.6 | 3.5 | 3.1 | 3.8 |
| French (`fr_ca`) | Water | 8.6 | 7.4 | 8.5 | 7.1 | 11.1 | 7.2 |
| Indonesian (`id_id`) | Water | 6.1 | 9.1 | 5.9 | 6.1 | 6.1 | 7.4 |
| Kreol Seselwa (`crs_sc`) | none | 23.1 | 18.9 | 23.5 | 21.4 | 24.6 | 18.6 |

## Ablations and superseded runs

- **Tamil at learning rate 1.5e-2** (`ws-ta_in-<step>-lr0.015`): the four Tiny Aya regional variants retrained on Tamil at ten times the study's learning rate, best and final checkpoints. The best rate tried for Tamil (kept-checkpoint validation CER 49.2 Global, 51.6 Earth, 38.0 Fire, 49.6 Water); the same rate made Amharic worse on every variant, so the study keeps one rate throughout and these runs stay out of its tables.
- **Spanish, first attempt on `es_es`** (`ws-es_es-<step>`): the four regional variants trained on the `es_es` WorldSpeech config before the study switched to `es_mx`. One checkpoint each, stopped between step 350 and step 850; CER was not logged for these runs, so they are not comparable with the grid.

## Requirements

The versions the checkpoints were trained and evaluated with:

- `transformers==4.57.5` (the checkpoints' `config.json` records this version)
- `torch==2.9.1`
- `datasets==4.5.0` with `torchcodec==0.9.1`, only for loading audio as in the example

## Citation

```bibtex
@inproceedings{rios-etal-2026-regional,
    title = "Does Regional Decoder Specialization Help Low-Resource {ASR} Based on the {SLAM}-{ASR} Framework?",
    author = "Rios, Edwin Arkel  and
      Zaruma, Jocelyn  and
      Ewoorkar, Girish  and
      Sourabh, Sneh  and
      Mack, Julian  and
      Juan, Hung-Hui  and
      Huang, Stephen  and
      Lai, Bo-Cheng",
    booktitle = "Proceedings of the 38th Conference on Computational Linguistics and Speech Processing (ROCLING 2026)",
    year = "2026",
    publisher = "Association for Computational Linguistics"
}
```