387 kB
6 files
Updated about 1 month ago
Name
Size
README.md10.9 kB
xet
eval_outputs.csv214 kB
xet
ft-blindspot.ipynb115 kB
xet
region_metrics.csv1.31 kB
xet
top_substitutions.json9.2 kB
xet
wer_by_region.png37 kB
xet
README.md

Blind Spot: Regional Bangla Dialect Disparities in Speech Recognition

Research question and motivation

Can a strong open-weight automatic speech recognition (ASR) model transcribe regional Bangla speech equally well across regions, or does a single aggregate “Bangla” score hide meaningful differences?

I grew up in Rangpur, so regional speech is not an abstract NLP category to me. I have heard how people can use familiar local words and pronunciations that do not always match formal, written Bangla. While building VoicePandita, a voice-first learning tool aimed at making study help easier to access for learners outside well-served settings, I kept coming back to a practical question: what happens when a student asks a question in the Bangla they actually speak, but the speech model expects something closer to standard Bangla? If the system mishears the question, the rest of the learning experience can fail before the tutor even starts answering.

That is why I wanted to look beyond one overall Bangla score and compare transcription errors across regions, including Rangpur. This small experiment is an initial check, not a claim that I have already explained every regional difference.

Hypothesis: ASR error rates differ across regional varieties. Some errors may involve dialect vocabulary being replaced with other words, but substitution counts alone do not prove that the model systematically “standardizes” dialect speech; this needs manual, context-aware review.

Model

  • Model: openai/whisper-large-v3
  • Task: Bengali speech recognition / transcription
  • Approximate parameter count: 1.55 billion (within the fellowship's 0.6B–6B requirement)
  • Inference: Hugging Face Transformers ASR pipeline, Bengali transcription mode, audio resampled to 16 kHz.

Dataset and sampling

  • Dataset: RegSpeech12, loaded from the Kaggle dataset path mdrezuwanhassan/regspeech12.
  • Sampling: 20 utterances per region, selected from the test metadata with random_state=0.
  • Total evaluated: 240 utterances across 12 regions.
  • Regions: Tangail, Comilla, Narail, Habiganj, Narsingdi, Barishal, Sylhet, Rangpur, Chittagong, Sandwip, Noakhali, and Kishoreganj.
  • Reproducibility note: This is a balanced, small exploratory sample—not a complete evaluation of every speaker or dialect. Results may depend on speaker, recording quality, transcript conventions, and the particular sampled utterances.

Method

  1. Load the test metadata and associate each audio filename with its region.
  2. Sample up to 20 utterances per region using a fixed random seed.
  3. Transcribe each sample with Whisper large-v3, explicitly setting the task to Bengali transcription.
  4. Normalize punctuation and whitespace in references and predictions.
  5. Compute corpus-level word error rate (WER) and character error rate (CER) per region using jiwer.
  6. Estimate a 95% percentile bootstrap interval for each region's corpus WER by resampling utterance pairs with replacement (2,000 replicates; seed 42).
  7. Count aligned word substitutions by region for qualitative error inspection.

WER is calculated as (substitutions + deletions + insertions) / number of reference words. It can exceed 1.0 when insertions are numerous. The bootstrap intervals quantify sampling uncertainty under this resampling procedure; they do not account for dataset bias or establish population-wide differences.

Results

Regional results from the current evaluation:

Region n WER CER 95% bootstrap CI for WER WER gap vs. best
Tangail 20 0.810 0.571 [0.771, 0.846] 0.000
Comilla 20 0.863 0.684 [0.830, 0.896] 0.053
Narail 20 0.902 0.675 [0.860, 0.935] 0.092
Habiganj 20 0.923 0.648 [0.898, 0.947] 0.113
Narsingdi 20 0.925 0.715 [0.897, 0.951] 0.115
Barishal 20 0.931 0.742 [0.899, 0.953] 0.121
Sylhet 20 0.942 0.717 [0.920, 0.963] 0.132
Rangpur 20 0.946 0.726 [0.925, 0.962] 0.136
Chittagong 20 0.966 0.723 [0.948, 0.982] 0.156
Sandwip 20 0.975 0.727 [0.948, 1.012] 0.165
Noakhali 20 0.996 0.730 [0.954, 1.061] 0.186
Kishoreganj 20 1.002 0.757 [0.954, 1.076] 0.192

Observations

  • The lowest WER in this sample was Tangail: 0.810.
  • The highest WER in this sample was Kishoreganj: 1.002.
  • The absolute difference between these sample WERs was 0.192 (about 19.2 percentage points).
  • The worst-region WER is slightly above 1.0. This is valid for WER because insertion errors can make the numerator larger than the number of reference words.
  • These results are descriptive of this 240-utterance sample. With only 20 utterances per region, they should be treated as preliminary and not as definitive evidence of population-level dialect unfairness.
  • Repetition loops: Using a simple heuristic (an output of 8+ words where fewer than 30% of the words are unique), I flagged repetitive outputs in eval_outputs.csv. Looping was most common in Kishoreganj (30% of its 20 outputs) and Rangpur (25%), followed by Noakhali and Sylhet (15% each) and Barishal (10%). Chittagong, Comilla, Habiganj, Narail and Narsingdi had 5% each, and Sandwip and Tangail had none. Repeated insertions raise WER (including above 1.0), so part of the regional gap may reflect model hallucination on this audio, not only dialect vocabulary. I did not recompute WER without looped outputs. Rangpur, my home region, was among the most affected.

Candidate word substitutions for manual review

The alignment script reports frequent reference-to-hypothesis substitutions. Examples from the saved output are listed below; they are candidates for human inspection, not verified cases of dialect normalization. Some alignments can be misleading without listening to the audio and reading the full sentence.

  • Kishoreganj: এইলো->এলো (count 6); গিয়া->এলে (count 5); অর্থ->আতা (count 4)
  • Noakhali: ওয়া->কুন্ন (count 2); বেইচ্ছে->বেছে (count 2); নাই->কানি (count 2)
  • Rangpur: মাজের->তারে (count 4); আইসে->আইছে (count 3); কয়->কাই (count 3)
  • Barishal: এট্টু->এতু (count 3); বানাইলো->বেনাইলে (count 2); কীসের->আহন (count 2)
  • Sylhet: তো->এখনা (count 4); আইয়েল্টস->আয়ের (count 2); ওত্তো->করতে (count 2)

Blind spot

Common aggregate ASR reporting can hide uneven performance between regional varieties. A model can appear capable at the language level while still making many errors on particular speakers or regions. For voice-first education, such failures can affect whether a learner's question is recognized correctly.

This evaluation does not establish why the error rates differ. Dialect vocabulary and phonology are possible explanations, but recording conditions, speaker variation, transcription conventions, and sample composition may also contribute. A stronger follow-up study should manually categorize errors and use more speakers and utterances per region.

Proposed path forward

  1. Dialect-balanced data curation: collect consented speech from speakers across regions, with dialect-faithful transcripts reviewed by native speakers. Where useful, add a separate Standard Bangla paraphrase/gloss rather than replacing the original transcript.
  2. Quality and metadata controls: document speaker counts, recording conditions, duration, and transcript conventions; split evaluation by speaker to reduce leakage.
  3. Parameter-efficient fine-tuning: test LoRA or another PEFT method with region-balanced sampling. Compare against the unmodified model and report both average and worst-region WER.
  4. Dialect-aware evaluation: report per-region WER/CER and confidence intervals, plus manually reviewed error categories. Test on a held-out set from speakers not seen during fine-tuning.
  5. Separate transcription from normalization: if a standard-register transcript is desired, make it a distinct optional transformation so users can preserve the words they actually said.

These are proposed next steps, not interventions evaluated in the current experiment.

Artifacts

  • ft-blindspot.ipynb — notebook for sampling, inference, metrics, error analysis, and plotting.
  • region_metrics.csv — regional WER/CER and bootstrap intervals.
  • eval_outputs.csv — sampled utterances, reference transcripts and model outputs (file paths are from the Kaggle environment).
  • top_substitutions.json — aligned substitution candidates per region.
  • wer_by_region.png — regional WER chart with bootstrap intervals.

Limitations and responsible use

  • Only 20 utterances per region were evaluated, so these results are exploratory and may change with a larger, speaker-balanced sample.
  • Results may be sensitive to sample selection, speaker distribution, recording quality, and reference-transcript conventions.
  • Script and number-format mismatch: I checked the 240 model outputs and found that 42 outputs (17.5%) contained at least one Latin-script character, such as in “software” or “coffee”, while 25 outputs (10.4%) contained at least one digit, such as “16” where the reference used “ষোলো”. These percentages are calculated over output utterances, not individual words or total errors. Such mismatches can contribute to WER, but they are not necessarily dialect-recognition errors. The current results use the original normalized text without converting digits to Bangla words or standardizing Latin-script loanwords. A follow-up evaluation should report raw WER alongside a consistently normalized version.
  • Automatic word alignment can produce misleading substitution pairs; examples require listening-based review. A substitution list alone does not prove that the model systematically normalizes dialect speech.
  • WER/CER alone do not measure semantic preservation, dialect respect, or downstream learning impact.
  • Do not infer that a region or its speakers are inherently less intelligible; this experiment measures one model on a limited sample.
  • Data: RegSpeech12 (Kaggle: mdrezuwanhassan/regspeech12); please follow its licence.

Reproducibility

The accompanying notebook contains the experiment code and records the sampling seed, model identifier, and bootstrap procedure. Run it in an environment with access to the RegSpeech12 files and a suitable GPU. Package versions and exact hardware should be recorded when rerunning the experiment.

Total size
387 kB
Files
6
Last updated
Oct 15
Pre-warmed CDN
US EU US EU

Contributors