--- pipeline_tag: automatic-speech-recognition language: - am - ha - sw - hi - mr - ta - ur - en - es - fr - id - crs tags: - slam-asr - tiny-aya - whisper - speech-recognition --- # LisTAya: SLAM-ASR projectors for twelve languages on the Tiny Aya decoders All checkpoints are listed in the collection [https://huggingface.co/collections/ERISLab/listaya-slam-asr-projectors-on-tiny-aya-rocling-2026-6ab031414095f3df8805ed7b](https://huggingface.co/collections/ERISLab/listaya-slam-asr-projectors-on-tiny-aya-rocling-2026-6ab031414095f3df8805ed7b). The transcripts of their test-set evaluations are the dataset [ERISLab/LisTAya-transcripts](https://huggingface.co/datasets/ERISLab/LisTAya-transcripts). LisTAya (Listening Tiny Aya) is a family of speech-recognition models built with the SLAM-ASR recipe: a frozen Whisper-medium speech encoder, a frozen large-language-model decoder, and a trainable linear projector between them. Only the projector is trained. The checkpoints in this collection are the models behind the paper *Does Regional Decoder Specialization Help Low-Resource ASR Based on the SLAM-ASR Framework?* (ROCLING 2026). Each checkpoint is one (language, decoder) cell of that study. ## Decoders Four Tiny Aya variants (3.35B parameters, one shared base model and tokenizer, differing only in the regional weighting of their post-training data): Global, Earth, Fire and Water. Two controls: Tiny Aya Base, the model before instruction tuning, and Qwen3-4B, a decoder from another family. ## Languages and training data Every projector is trained on the WorldSpeech train split of one language (formal speech from parliamentary proceedings, broadcasts and audiobooks, clips under 30 seconds): Amharic (`am_et`), Hausa (`ha_ng`), Swahili (`sw_ke`), Hindi (`hi_in`), Marathi (`mr_in`), Tamil (`ta_in`), Urdu (`ur_pk`), English (`en_us`), Spanish (`es_mx`), French (`fr_ca`), Indonesian (`id_id`), Kreol Seselwa (`crs_sc`). The region is the Tiny Aya region each language is assigned to in the study; Kreol Seselwa is supported by neither the encoder nor any decoder and has no region. Its data come from the `ERISLab/WorldSpeech` config `crs_sc`; every other language from `disco-eth/WorldSpeech`. ## Training recipe AdamW at learning rate 1.5e-3, batch size 8 with gradient accumulation 64 (effective batch 512), 1,000 steps, no weight decay, one NVIDIA H200, validation every 10 steps. Checkpoints are saved every 50 steps and the trainer keeps the one with the lowest validation loss on the FLEURS validation split, with two exceptions: Hausa is validated on the WorldSpeech test split, and Kreol Seselwa, which FLEURS does not cover, on a held-out 2% split of its WorldSpeech data. ## What each repository holds Repositories are named `q2a_openai_whisper-medium__ws--`. Each cell has two: the checkpoint the trainer kept as best on validation loss (the one the paper evaluates) and the final checkpoint at step 1000. Every repository holds the full trained model (encoder, projector and decoder weights, processor and tokenizer files) plus `trainer_state.json` with the run's validation curve. ## Load a checkpoint and transcribe Every checkpoint is a `Qwen2AudioForConditionalGeneration` model and loads with the standard classes of `transformers` 4.57.5. The prompt below is the one the projectors were trained with; the audio is a 16 kHz mono array. The weights are stored in bfloat16; the paper's evaluations load them in float32, as below. Loading in bfloat16 halves the memory and changed 3 of 40 English validation transcripts in a check. ```python import torch from datasets import load_dataset from transformers import AutoProcessor, Qwen2AudioForConditionalGeneration repo = "ERISLab/q2a_openai_whisper-medium_CohereLabs_tiny-aya-global_ws-en_us-500" processor = AutoProcessor.from_pretrained(repo) model = Qwen2AudioForConditionalGeneration.from_pretrained(repo, dtype=torch.float32).to("cuda") prompt = "<|audio_bos|><|AUDIO|><|audio_eos|>Transcribe the input speech: " sample = next(iter(load_dataset("google/fleurs", "en_us", split="validation", streaming=True))) audio = sample["audio"]["array"] # 16 kHz mono float array inputs = processor(text=prompt, audio=audio, sampling_rate=16000, return_tensors="pt").to("cuda") with torch.no_grad(): generated = model.generate(**inputs, max_new_tokens=256, do_sample=False) hypothesis = processor.batch_decode(generated[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0] ``` On this clip the English Global checkpoint writes "when you call someone who is 1000s of miles away you are using a satellite"; the reference is "when you call someone who is thousands of miles away you are using a satellite". ## Validation CER of the kept checkpoint Character error rate (%) of each kept checkpoint on its validation split, read from its `trainer_state.json`. This is the score of the released model, not the minimum of the run's validation curve, which the paper reports as best CER. | Language | Region | Global | Earth | Fire | Water | Base | Qwen3-4B | |---|---|---|---|---|---|---|---| | Amharic (`am_et`) | Earth | 35.1 | 36.3 | 31.0 | 28.6 | 47.3 | 39.5 | | Hausa (`ha_ng`) | Earth | 37.1 | 37.2 | 36.9 | 38.7 | 36.6 | 34.1 | | Swahili (`sw_ke`) | Earth | 15.6 | 17.2 | 22.9 | 19.7 | 19.2 | 14.7 | | Hindi (`hi_in`) | Fire | 15.7 | 18.8 | 18.5 | 16.4 | 12.2 | 15.5 | | Marathi (`mr_in`) | Fire | 14.2 | 14.4 | 14.7 | 19.4 | 14.3 | 15.4 | | Tamil (`ta_in`) | Fire | 84.5 | 72.9 | 47.3 | 80.6 | 45.7 | 53.4 | | Urdu (`ur_pk`) | Fire | 33.1 | 21.8 | 26.2 | 25.5 | 25.3 | 48.4 | | English (`en_us`) | Water | 10.2 | 7.2 | 4.4 | 14.8 | 5.5 | 7.2 | | Spanish (`es_mx`) | Water | 3.5 | 5.6 | 3.6 | 3.5 | 3.1 | 3.8 | | French (`fr_ca`) | Water | 8.6 | 7.4 | 8.5 | 7.1 | 11.1 | 7.2 | | Indonesian (`id_id`) | Water | 6.1 | 9.1 | 5.9 | 6.1 | 6.1 | 7.4 | | Kreol Seselwa (`crs_sc`) | none | 23.1 | 18.9 | 23.5 | 21.4 | 24.6 | 18.6 | ## Ablations and superseded runs - **Tamil at learning rate 1.5e-2** (`ws-ta_in--lr0.015`): the four Tiny Aya regional variants retrained on Tamil at ten times the study's learning rate, best and final checkpoints. The best rate tried for Tamil (kept-checkpoint validation CER 49.2 Global, 51.6 Earth, 38.0 Fire, 49.6 Water); the same rate made Amharic worse on every variant, so the study keeps one rate throughout and these runs stay out of its tables. - **Spanish, first attempt on `es_es`** (`ws-es_es-`): the four regional variants trained on the `es_es` WorldSpeech config before the study switched to `es_mx`. One checkpoint each, stopped between step 350 and step 850; CER was not logged for these runs, so they are not comparable with the grid. ## Requirements The versions the checkpoints were trained and evaluated with: - `transformers==4.57.5` (the checkpoints' `config.json` records this version) - `torch==2.9.1` - `datasets==4.5.0` with `torchcodec==0.9.1`, only for loading audio as in the example ## Citation ```bibtex @inproceedings{rios-etal-2026-regional, title = "Does Regional Decoder Specialization Help Low-Resource {ASR} Based on the {SLAM}-{ASR} Framework?", author = "Rios, Edwin Arkel and Zaruma, Jocelyn and Ewoorkar, Girish and Sourabh, Sneh and Mack, Julian and Juan, Hung-Hui and Huang, Stephen and Lai, Bo-Cheng", booktitle = "Proceedings of the 38th Conference on Computational Linguistics and Speech Processing (ROCLING 2026)", year = "2026", publisher = "Association for Computational Linguistics" } ```