Premiere publication

7df18eb verified 3 days ago

6.51 kB

---
language:
- mg
license: cc-by-sa-4.0
library_name: transformers
pipeline_tag: token-classification
base_model: Davlan/xlm-roberta-base-finetuned-malagasy
datasets:
- badrex/malagasy-speech-full
tags:
- punctuation-restoration
- truecasing
- malagasy
- asr-postprocessing
- xlm-roberta
---
<p align="center"><a href="#results"><img src="https://img.shields.io/badge/formatted%20WER%20on%20ASR%20output-18.5%20%25%20vs%2021.7%20%25-2F7D55?style=flat-square&labelColor=1E2A5E" alt="formatted WER on ASR output: 18.5 % against 21.7 %"></a> <a href="#results"><img src="https://img.shields.io/badge/period%20F1-79-2F7D55?style=flat-square&labelColor=1E2A5E" alt="period F1: 79"></a> <a href="#what-it-does"><img src="https://img.shields.io/badge/words-never%20changed-E9C46A?style=flat-square&labelColor=1E2A5E" alt="words: never changed"></a> <a href="#license"><img src="https://img.shields.io/badge/license-CC--BY--SA--4.0-8A93A6?style=flat-square&labelColor=1E2A5E" alt="license: CC-BY-SA-4.0"></a> <a href="https://huggingface.co/spaces/Flo976/malagasy-asr-demo"><img src="https://img.shields.io/badge/demo-try%20it%20in%20the%20browser-E9C46A?style=flat-square&labelColor=1E2A5E" alt="demo: try it in the browser"></a></p>

# Malagasy casing and punctuation restoration for speech recognition

Speech recognition models such as [Flo976/w2v-bert-malagasy](https://huggingface.co/Flo976/w2v-bert-malagasy) output lowercase text without punctuation. This model puts back **capital letters, commas, periods, question and exclamation marks and colons, without ever changing a word**, and restores accented letters that ASR normalization strips (`tan na` → `tanàna`).

```
in:  vehivavy fohy volo iray ary miloko mainty manamboatra volo manoloana ny fitaratra
out: Vehivavy fohy volo iray ary miloko mainty, manamboatra volo manoloana ny fitaratra.
```

**Résumé en français.** Remet les majuscules et la ponctuation dans la sortie d'un modèle de reconnaissance vocale malgache, sans jamais changer les mots. Sur la vraie sortie de l'ASR, le texte final mis en forme compte 18,5 % d'erreurs contre 21,7 % pour une mise en forme minimale (majuscule initiale et point final). Les points sont bien placés (F1 79), les virgules une fois sur deux environ (F1 52).

## What it does

For every word it predicts a case (lower, Capitalized, UPPER, mixed) and the mark that follows it: none, `,` `.` `?` `!` `:` or `;`. Since it classifies words instead of generating text, it cannot invent, drop or rewrite words: every call asserts that `normalize(output) == normalize(input)`, where `normalize` is the lowercase, punctuation-free form used to score the ASR. This assertion held on every test output and in a fuzz test.

## Results

Test split of [badrex/malagasy-speech-full](https://huggingface.co/datasets/badrex/malagasy-speech-full), 3,101 clips, speakers never seen in training. *Formatted WER* compares the final text to the original transcript **with** its casing and punctuation, without any normalization: it measures what a reader actually sees.

| Input | System | Formatted WER | F1 macro `, . ? ! :` | F1 comma | F1 period | Case accuracy |
| --- | --- | --- | --- | --- | --- | --- |
| real ASR output (10.6 % WER) | capitalize first word + final period | 21.73 % | 13.1 | 0 | 65.6 | 96.0 % |
| real ASR output (10.6 % WER) | **this model** | **18.47 %** | **45.2** | 51.9 | 79.0 | 97.1 % |
| reference words | capitalize first word + final period | 13.09 % | 13.3 | 0 | 66.3 | 96.1 % |
| reference words | **this model** | **9.18 %** | **51.2** | 52.7 | 79.8 | 97.3 % |

On real ASR output the gain over the trivial baseline is **−3.26 points** of formatted WER (paired bootstrap, 95 % CI [−3.51, −3.02]). With perfect labels, the floor would be 2.4 %: quotes, brackets and some accents are out of scope.

## Usage

```python
import os, sys
from huggingface_hub import snapshot_download
path = snapshot_download("Flo976/malagasy-punctuation")
sys.path.insert(0, path)
from punctuate import Restorer
restorer = Restorer(path)  # CPU by default; Restorer(path, device="cuda") on GPU
print(restorer.restore("tena mamy tokoa ity voankazo ity ary tsy mifidy"))
```

Requirements: `torch`, `transformers`, `sentencepiece`, `numpy`. Long texts are processed in sliding windows of 150 words. On two CPU threads, a 32-word utterance takes about 80 ms.

The [demo](https://huggingface.co/spaces/Flo976/malagasy-asr-demo) also capitalizes the word after `.`, `?` and `!`, as 99.6 % of sentence starts are in the validation transcripts. That rule is not included in the measurements above.

## Training

| | |
| --- | --- |
| Base model | [Davlan/xlm-roberta-base-finetuned-malagasy](https://huggingface.co/Davlan/xlm-roberta-base-finetuned-malagasy), XLM-R base adapted to Malagasy |
| Head | one token-classification head, 28 classes (4 cases × 7 marks), read on the first sub-token of each word |
| Data | 0.93 M words of raw train-split transcripts of badrex/malagasy-speech-full + 1.5 M words of filtered Malagasy Wikipedia sentences |
| Robustness | half of the training inputs receive synthetic CTC-style errors (substituted letters, merged or split words) |
| Accent table | built from the training text, restores letters that normalization strips; same no-word-change guarantee |
| Selection | on the validation split only; the test was used once |
| Hardware | one RTX 5070 Ti, 13 minutes |

## Limitations

- **Commas remain hard** (F1 about 52): transcribers place them inconsistently.
- **Question and exclamation marks are rare** in the test (20 and 35 cases): their scores are very uncertain. Semicolons are never predicted.
- **Domain**: trained on read image descriptions and Wikipedia. Dialogue and questions, as in a voice assistant, are less well punctuated.
- **Size**: 1.1 GB in fp32. Dynamic int8 quantization breaks it (comma F1 from 54 to 19): do not quantize.
- Pauses in the audio are not used. Adding them helped a separate classifier, but they are not integrated in this model.

## License

**CC-BY-SA-4.0**, because the model is trained on Malagasy Wikipedia (CC BY-SA). The base model, Davlan/xlm-roberta-base-finetuned-malagasy, declares no license on its card; XLM-R itself is MIT. The speech dataset, badrex/malagasy-speech-full, declares no license either. Please credit Davlan (David Adelani) for the base model.

Made for [Milo Voice](https://github.com/Flo976/milo) by [Flo976](https://huggingface.co/Flo976) — Florent Didelot / [Sooatek](https://sooatek.com).

	---
	language:
	- mg
	license: cc-by-sa-4.0
	library_name: transformers
	pipeline_tag: token-classification
	base_model: Davlan/xlm-roberta-base-finetuned-malagasy
	datasets:
	- badrex/malagasy-speech-full
	tags:
	- punctuation-restoration
	- truecasing
	- malagasy
	- asr-postprocessing
	- xlm-roberta
	---

	<p align="center"><a href="#results"><img src="https://img.shields.io/badge/formatted%20WER%20on%20ASR%20output-18.5%20%25%20vs%2021.7%20%25-2F7D55?style=flat-square&labelColor=1E2A5E" alt="formatted WER on ASR output: 18.5 % against 21.7 %"></a> <a href="#results"><img src="https://img.shields.io/badge/period%20F1-79-2F7D55?style=flat-square&labelColor=1E2A5E" alt="period F1: 79"></a> <a href="#what-it-does"><img src="https://img.shields.io/badge/words-never%20changed-E9C46A?style=flat-square&labelColor=1E2A5E" alt="words: never changed"></a> <a href="#license"><img src="https://img.shields.io/badge/license-CC--BY--SA--4.0-8A93A6?style=flat-square&labelColor=1E2A5E" alt="license: CC-BY-SA-4.0"></a> <a href="https://huggingface.co/spaces/Flo976/malagasy-asr-demo"><img src="https://img.shields.io/badge/demo-try%20it%20in%20the%20browser-E9C46A?style=flat-square&labelColor=1E2A5E" alt="demo: try it in the browser"></a></p>

	# Malagasy casing and punctuation restoration for speech recognition

	Speech recognition models such as [Flo976/w2v-bert-malagasy](https://huggingface.co/Flo976/w2v-bert-malagasy) output lowercase text without punctuation. This model puts back capital letters, commas, periods, question and exclamation marks and colons, without ever changing a word, and restores accented letters that ASR normalization strips (`tan na` → `tanàna`).

	```
	in: vehivavy fohy volo iray ary miloko mainty manamboatra volo manoloana ny fitaratra
	out: Vehivavy fohy volo iray ary miloko mainty, manamboatra volo manoloana ny fitaratra.
	```

	Résumé en français. Remet les majuscules et la ponctuation dans la sortie d'un modèle de reconnaissance vocale malgache, sans jamais changer les mots. Sur la vraie sortie de l'ASR, le texte final mis en forme compte 18,5 % d'erreurs contre 21,7 % pour une mise en forme minimale (majuscule initiale et point final). Les points sont bien placés (F1 79), les virgules une fois sur deux environ (F1 52).

	## What it does

	For every word it predicts a case (lower, Capitalized, UPPER, mixed) and the mark that follows it: none, `,` `.` `?` `!` `:` or `;`. Since it classifies words instead of generating text, it cannot invent, drop or rewrite words: every call asserts that `normalize(output) == normalize(input)`, where `normalize` is the lowercase, punctuation-free form used to score the ASR. This assertion held on every test output and in a fuzz test.

	## Results

	Test split of [badrex/malagasy-speech-full](https://huggingface.co/datasets/badrex/malagasy-speech-full), 3,101 clips, speakers never seen in training. Formatted WER compares the final text to the original transcript with its casing and punctuation, without any normalization: it measures what a reader actually sees.

	\| Input \| System \| Formatted WER \| F1 macro `, . ? ! :` \| F1 comma \| F1 period \| Case accuracy \|
	\| --- \| --- \| --- \| --- \| --- \| --- \| --- \|
	\| real ASR output (10.6 % WER) \| capitalize first word + final period \| 21.73 % \| 13.1 \| 0 \| 65.6 \| 96.0 % \|
	\| real ASR output (10.6 % WER) \| this model \| 18.47 % \| 45.2 \| 51.9 \| 79.0 \| 97.1 % \|
	\| reference words \| capitalize first word + final period \| 13.09 % \| 13.3 \| 0 \| 66.3 \| 96.1 % \|
	\| reference words \| this model \| 9.18 % \| 51.2 \| 52.7 \| 79.8 \| 97.3 % \|

	On real ASR output the gain over the trivial baseline is −3.26 points of formatted WER (paired bootstrap, 95 % CI [−3.51, −3.02]). With perfect labels, the floor would be 2.4 %: quotes, brackets and some accents are out of scope.

	## Usage

	```python
	import os, sys
	from huggingface_hub import snapshot_download

	path = snapshot_download("Flo976/malagasy-punctuation")
	sys.path.insert(0, path)
	from punctuate import Restorer

	restorer = Restorer(path) # CPU by default; Restorer(path, device="cuda") on GPU
	print(restorer.restore("tena mamy tokoa ity voankazo ity ary tsy mifidy"))
	```

	Requirements: `torch`, `transformers`, `sentencepiece`, `numpy`. Long texts are processed in sliding windows of 150 words. On two CPU threads, a 32-word utterance takes about 80 ms.

	The [demo](https://huggingface.co/spaces/Flo976/malagasy-asr-demo) also capitalizes the word after `.`, `?` and `!`, as 99.6 % of sentence starts are in the validation transcripts. That rule is not included in the measurements above.

	## Training

	\| \| \|
	\| --- \| --- \|
	\| Base model \| [Davlan/xlm-roberta-base-finetuned-malagasy](https://huggingface.co/Davlan/xlm-roberta-base-finetuned-malagasy), XLM-R base adapted to Malagasy \|
	\| Head \| one token-classification head, 28 classes (4 cases × 7 marks), read on the first sub-token of each word \|
	\| Data \| 0.93 M words of raw train-split transcripts of badrex/malagasy-speech-full + 1.5 M words of filtered Malagasy Wikipedia sentences \|
	\| Robustness \| half of the training inputs receive synthetic CTC-style errors (substituted letters, merged or split words) \|
	\| Accent table \| built from the training text, restores letters that normalization strips; same no-word-change guarantee \|
	\| Selection \| on the validation split only; the test was used once \|
	\| Hardware \| one RTX 5070 Ti, 13 minutes \|

	## Limitations

	- Commas remain hard (F1 about 52): transcribers place them inconsistently.
	- Question and exclamation marks are rare in the test (20 and 35 cases): their scores are very uncertain. Semicolons are never predicted.
	- Domain: trained on read image descriptions and Wikipedia. Dialogue and questions, as in a voice assistant, are less well punctuated.
	- Size: 1.1 GB in fp32. Dynamic int8 quantization breaks it (comma F1 from 54 to 19): do not quantize.
	- Pauses in the audio are not used. Adding them helped a separate classifier, but they are not integrated in this model.

	## License

	CC-BY-SA-4.0, because the model is trained on Malagasy Wikipedia (CC BY-SA). The base model, Davlan/xlm-roberta-base-finetuned-malagasy, declares no license on its card; XLM-R itself is MIT. The speech dataset, badrex/malagasy-speech-full, declares no license either. Please credit Davlan (David Adelani) for the base model.

	Made for [Milo Voice](https://github.com/Flo976/milo) by [Flo976](https://huggingface.co/Flo976) — Florent Didelot / [Sooatek](https://sooatek.com).