Initial release

7ea8e26 verified 13 days ago

10.9 kB

---
language:
- fi
license: cc-by-nc-4.0
library_name: transformers
pipeline_tag: token-classification
base_model: Finnish-NLP/electra-base-generator-finnish
base_model_relation: finetune
tags:
- punctuation-restoration
- sentence-boundary-detection
- truecasing
- speech-recognition
- streaming
- finnish
metrics:
- f1
---
# Mateli-punct: punctuation and casing for Finnish speech recognition

This model adds punctuation (`,` `.` `?` `!`) and capitals to lowercase, unpunctuated Finnish text from a speech recogniser. It was made for the output of [Mateli](https://huggingface.co/samuli/mateli), a streaming Finnish recogniser, and can use the pauses between words from the recogniser's word times. Words are never changed, added or removed. The model is small (30 M parameters) and works word by word on a live stream as well as on whole texts. An int8 ONNX export for phones and CPUs is in [samuli/mateli-punct-onnx](https://huggingface.co/samuli/mateli-punct-onnx).

The approach is inspired by [syvai/danish-pnc](https://huggingface.co/syvai/danish-pnc). It uses token classification with one label per word and an ELECTRA encoder, and it is evaluated with the same metrics.

## Examples

`python punctuate.py "<text>"`:

| input | output |
|---|---|
| mä luulen et se on ihan hyvä juttu mut en mä tiiä mitä mieltä sä oot | Mä luulen, et se on ihan hyvä juttu. Mut en mä tiiä, mitä mieltä sä oot. |
| huomenna mennään mökille jos sää on hyvä otatko sä kalavehkeet mukaan | Huomenna mennään mökille. Jos sää on hyvä, otatko sä kalavehkeet mukaan? |
| no niin aloitetaan kokous onko kaikki paikalla hyvä ensimmäisenä asialistalla on budjetti | No niin, Aloitetaan kokous. Onko kaikki paikalla? Hyvä. Ensimmäisenä asialistalla on budjetti. |
| suomen pääkaupunki on helsinki ja siellä asuu noin kuusisataatuhatta ihmistä | Suomen pääkaupunki on Helsinki, ja siellä asuu noin kuusisataatuhatta ihmistä. |
| kiitos paljon tästä keskustelusta nähdään taas ensi viikolla | Kiitos paljon tästä keskustelusta. Nähdään taas ensi viikolla. |

Mistakes happen, as with *No niin, Aloitetaan*, which should be *No niin, aloitetaan*.

Together with the recogniser, on two public clips (CC0, from the [Mateli examples](https://huggingface.co/samuli/mateli/tree/main/examples)). The recognition errors (*repo*, *öö*, *tiedonainnosta* and others) come from the recogniser. The punctuation model only adds punctuation and capitals, using the pauses.

```python
import sys
sys.path[:0] = ["mateli", "mateli-punct"]
from mateli import Mateli, load_audio
from punctuate import Punctuator
asr, punct = Mateli.from_pretrained("mateli"), Punctuator("mateli-punct")
for name in ["voxpopuli_1", "voxpopuli_2"]:
    words = asr.transcribe(load_audio(f"mateli/examples/{name}.wav")).words
    pauses = [b.start - a.end for a, b in zip(words, words[1:])] + [None]   # silence after each word
    print(punct.punctuate([w.text for w in words], pauses))
```

```
Repo. Äänestin tämän mietinnön puolesta. Öö vaikka komission tiedonainnosta on kulunut jo yli kymmeneen vuotta, eivät kaikki matkustajat edelleenkään tiedä oikeuksistaan.
Boiko toiminen vaaleissa ja mielenosoitus ovat eri asioita, mutta erittäin huolestuttavia ovat tiedot siitä, että useampi kuin joka kymmenen säänestyspaikka ei voinut avata oviaan opposition saannon takia.
```

A live, word-by-word example is on the [ONNX model card](https://huggingface.co/samuli/mateli-punct-onnx#examples).

## How to use

```
pip install torch transformers tokenizers
hf download samuli/mateli-punct --local-dir mateli-punct
python mateli-punct/punctuate.py "mä luulen et se on ihan hyvä juttu mut en mä tiiä mitä mieltä sä oot"
```

From Python (run next to the downloaded `mateli-punct` directory):

```python
import sys
sys.path.insert(0, "mateli-punct")
from punctuate import Punctuator
p = Punctuator("mateli-punct")
print(p.punctuate("mä luulen et se on ihan hyvä juttu mut en mä tiiä mitä mieltä sä oot"))
print(p.predict("onko kaikki valmista".split()))
```

```
Mä luulen, et se on ihan hyvä juttu. Mut en mä tiiä, mitä mieltä sä oot.
['O|U', 'O', '?']
```

`punctuate.py` (about 220 lines) has three entry points:

- `Punctuator.punctuate(text_or_words, pauses=None)` punctuates whole texts of any length, in overlapping windows of 150 words.
- `Punctuator.predict(words, pauses)` returns the raw labels.
- `Punctuator.stream()` is for live use. `push(word, pause_before)` returns the text committed so far, and `flush(pause_after)` commits the rest. A word is committed once three more words are known.

`pauses[i]` is the silence after word *i* in seconds, `None` if unknown. The model loads with `AutoModelForTokenClassification`, but `punctuate.py` builds the input ids itself because of the pause markers. The code is meant to be read and adapted.

### Input format

`punctuate.py` builds the model input itself. This is how, for porting it to another language. As an example, take six words from the recogniser and the silence after each word:

| word | onko | kaikki | paikalla | hyvä | aloitetaan | kalavehkeet |
|---|---|---|---|---|---|---|
| pause after (s) | 0.05 | 0.20 | 0.90 | 0.35 | 0.10 | unknown |

The model gets this token sequence (at most 512 tokens per call):

```
[CLS] [unused0] onko kaikki [unused1] paikalla [unused4] hyvä [unused2] aloitetaan kala ##veh ##keet [SEP]
```

- `[unused0]` right after `[CLS]` tells the model that pauses are given. Without pauses, leave it out and add no pause markers. The model works both ways.
- Each word is split into pieces with the WordPiece tokenizer in `tokenizer.json` (cased, no accent stripping). Most words are a single piece, and rarer ones are split, like *kala ##veh ##keet*.
- After the last piece of a word comes a pause marker for the silence that follows it. Silences shorter than 0.14 s and unknown ones get no marker. The bucket edges are in `config.json` (`pause_markers`).

| silence after the word | marker |
|---|---|
| under 0.14 s, or unknown | none |
| 0.14 to 0.30 s | `[unused1]` |
| 0.30 to 0.50 s | `[unused2]` |
| 0.50 to 0.78 s | `[unused3]` |
| 0.78 to 1.18 s | `[unused4]` |
| 1.18 s or more | `[unused5]` |

The label of a word is read from its first piece (`kala` for *kalavehkeet*), and the outputs on the other pieces and on the markers are ignored. Each of the 15 labels (`id2label` in `config.json`) combines the punctuation after the word (`O` for none, `,` `.` `?` `!`) with its casing (nothing for lowercase, `|U` for a capital first letter, `|A` for all capitals):

| word | onko | kaikki | paikalla | hyvä | aloitetaan | kalavehkeet |
|---|---|---|---|---|---|---|
| label | `O\|U` | `O` | `?` | `.\|U` | `O\|U` | `.` |

Applied to the words, the labels give:

```
Onko kaikki paikalla? Hyvä. Aloitetaan kalavehkeet.
```

`punctuate.py` also capitalises the first word and every word after `.` `?` `!`, in case the model misses one.

### Files

| file | what |
|---|---|
| `model.safetensors` | weights, fp32 (115 MiB) |
| `config.json` | ELECTRA config with `id2label` and `pause_markers` |
| `tokenizer.json`, `tokenizer_config.json` | WordPiece tokenizer, 26,020 pieces |
| `punctuate.py` | inference: whole texts, raw labels, live streaming |

## Model

ELECTRA, 12 layers, hidden size 256, 4 attention heads, 30 M parameters (20 M of them in the embeddings). Initialised from [Finnish-NLP/electra-base-generator-finnish](https://huggingface.co/Finnish-NLP/electra-base-generator-finnish) (Apache-2.0) with the vocabulary pruned to the 26,000 pieces the training text uses (plus the marker tokens), then trained by distillation from a larger BERT-based punctuation model fine-tuned for the same task, together with the gold labels.

## Training data

Written Finnish with punctuation, and transcribed speech. For the speech, the input is the output of the Mateli recogniser, the punctuation and casing labels were transferred from human transcripts of the same audio by word alignment, and the pauses come from the recogniser's word times. Written text was lowercased and stripped of punctuation for the input, with simulated recognition noise. Windows were 16 to 160 words starting at random positions, so the model also handles text that does not begin at a sentence start.

## Evaluation

The test sets are held-out speech transcribed by the Mateli recogniser, with labels transferred from human transcripts of the same audio. The podcast set has shows not seen in training. The metrics follow syvai/danish-pnc. Each is an F1 score per word, for a sentence boundary (`.` `?` `!` interchangeably), comma, question mark and casing (capitalised or all caps vs lowercase).

| set | input | mode | boundary F1 | boundary P | boundary R | comma F1 | question F1 | casing F1 |
|---|---|---|---:|---:|---:|---:|---:|---:|
| conversational podcasts | words | whole text | 0.666 | 0.638 | 0.696 | 0.700 | 0.594 | 0.702 |
| conversational podcasts | words + pauses | whole text | 0.680 | 0.653 | 0.710 | 0.702 | 0.614 | 0.714 |
| conversational podcasts | words + pauses | live | 0.666 | 0.644 | 0.691 | 0.699 | 0.591 | 0.705 |
| TV programmes | words | whole text | 0.709 | 0.743 | 0.678 | 0.582 | 0.552 | 0.739 |
| TV programmes | words + pauses | whole text | 0.756 | 0.780 | 0.733 | 0.598 | 0.607 | 0.772 |
| TV programmes | words + pauses | live | 0.753 | 0.780 | 0.727 | 0.600 | 0.602 | 0.769 |
| long conversations | words | whole text | 0.683 | 0.668 | 0.698 | 0.697 | 0.669 | 0.714 |
| long conversations | words | live | 0.657 | 0.644 | 0.672 | 0.692 | 0.659 | 0.700 |

*Live* is the streaming policy of `Punctuator.stream()` (three words of lookahead, 64 words of context, a model call every five words). On the podcast set a word is committed on average 2.4 s after it is spoken (90th percentile 3.4 s).

The scores are well below what the same kind of model reaches on written text, for two reasons. The input has recognition errors, and the reference labels come from human transcribers, who disagree among themselves on spontaneous speech, for example on whether *ja*, *mut* or *nii* starts a new sentence and where commas go in long spoken sentences.

## Limitations

- Trained for speech recognition output in Finnish. It also punctuates written text that has lost its punctuation, but that was not the target.
- Sentence boundaries in disfluent conversation and question marks are the weakest points.
- Casing is limited to a capital first letter or all capitals (no *iPhone*-style mixed case).
- The pauses must come from the same kind of word timing as Mateli's (end of a word's last character to the start of the next word's first character, in 40 ms steps).

## License

The model weights and the code in this repository are licensed under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). The base model Finnish-NLP/electra-base-generator-finnish is Apache-2.0.

	---
	language:
	- fi
	license: cc-by-nc-4.0
	library_name: transformers
	pipeline_tag: token-classification
	base_model: Finnish-NLP/electra-base-generator-finnish
	base_model_relation: finetune
	tags:
	- punctuation-restoration
	- sentence-boundary-detection
	- truecasing
	- speech-recognition
	- streaming
	- finnish
	metrics:
	- f1
	---

	# Mateli-punct: punctuation and casing for Finnish speech recognition

	This model adds punctuation (`,` `.` `?` `!`) and capitals to lowercase, unpunctuated Finnish text from a speech recogniser. It was made for the output of [Mateli](https://huggingface.co/samuli/mateli), a streaming Finnish recogniser, and can use the pauses between words from the recogniser's word times. Words are never changed, added or removed. The model is small (30 M parameters) and works word by word on a live stream as well as on whole texts. An int8 ONNX export for phones and CPUs is in [samuli/mateli-punct-onnx](https://huggingface.co/samuli/mateli-punct-onnx).

	The approach is inspired by [syvai/danish-pnc](https://huggingface.co/syvai/danish-pnc). It uses token classification with one label per word and an ELECTRA encoder, and it is evaluated with the same metrics.

	## Examples

	`python punctuate.py "<text>"`:

	\| input \| output \|
	\|---\|---\|
	\| mä luulen et se on ihan hyvä juttu mut en mä tiiä mitä mieltä sä oot \| Mä luulen, et se on ihan hyvä juttu. Mut en mä tiiä, mitä mieltä sä oot. \|
	\| huomenna mennään mökille jos sää on hyvä otatko sä kalavehkeet mukaan \| Huomenna mennään mökille. Jos sää on hyvä, otatko sä kalavehkeet mukaan? \|
	\| no niin aloitetaan kokous onko kaikki paikalla hyvä ensimmäisenä asialistalla on budjetti \| No niin, Aloitetaan kokous. Onko kaikki paikalla? Hyvä. Ensimmäisenä asialistalla on budjetti. \|
	\| suomen pääkaupunki on helsinki ja siellä asuu noin kuusisataatuhatta ihmistä \| Suomen pääkaupunki on Helsinki, ja siellä asuu noin kuusisataatuhatta ihmistä. \|
	\| kiitos paljon tästä keskustelusta nähdään taas ensi viikolla \| Kiitos paljon tästä keskustelusta. Nähdään taas ensi viikolla. \|

	Mistakes happen, as with No niin, Aloitetaan, which should be No niin, aloitetaan.

	Together with the recogniser, on two public clips (CC0, from the [Mateli examples](https://huggingface.co/samuli/mateli/tree/main/examples)). The recognition errors (repo, öö, tiedonainnosta and others) come from the recogniser. The punctuation model only adds punctuation and capitals, using the pauses.

	```python
	import sys
	sys.path[:0] = ["mateli", "mateli-punct"]
	from mateli import Mateli, load_audio
	from punctuate import Punctuator

	asr, punct = Mateli.from_pretrained("mateli"), Punctuator("mateli-punct")
	for name in ["voxpopuli_1", "voxpopuli_2"]:
	words = asr.transcribe(load_audio(f"mateli/examples/{name}.wav")).words
	pauses = [b.start - a.end for a, b in zip(words, words[1:])] + [None] # silence after each word
	print(punct.punctuate([w.text for w in words], pauses))
	```

	```
	Repo. Äänestin tämän mietinnön puolesta. Öö vaikka komission tiedonainnosta on kulunut jo yli kymmeneen vuotta, eivät kaikki matkustajat edelleenkään tiedä oikeuksistaan.
	Boiko toiminen vaaleissa ja mielenosoitus ovat eri asioita, mutta erittäin huolestuttavia ovat tiedot siitä, että useampi kuin joka kymmenen säänestyspaikka ei voinut avata oviaan opposition saannon takia.
	```

	A live, word-by-word example is on the [ONNX model card](https://huggingface.co/samuli/mateli-punct-onnx#examples).

	## How to use

	```
	pip install torch transformers tokenizers
	hf download samuli/mateli-punct --local-dir mateli-punct
	python mateli-punct/punctuate.py "mä luulen et se on ihan hyvä juttu mut en mä tiiä mitä mieltä sä oot"
	```

	From Python (run next to the downloaded `mateli-punct` directory):

	```python
	import sys
	sys.path.insert(0, "mateli-punct")
	from punctuate import Punctuator

	p = Punctuator("mateli-punct")
	print(p.punctuate("mä luulen et se on ihan hyvä juttu mut en mä tiiä mitä mieltä sä oot"))
	print(p.predict("onko kaikki valmista".split()))
	```

	```
	Mä luulen, et se on ihan hyvä juttu. Mut en mä tiiä, mitä mieltä sä oot.
	['O\|U', 'O', '?']
	```

	`punctuate.py` (about 220 lines) has three entry points:

	- `Punctuator.punctuate(text_or_words, pauses=None)` punctuates whole texts of any length, in overlapping windows of 150 words.
	- `Punctuator.predict(words, pauses)` returns the raw labels.
	- `Punctuator.stream()` is for live use. `push(word, pause_before)` returns the text committed so far, and `flush(pause_after)` commits the rest. A word is committed once three more words are known.

	`pauses[i]` is the silence after word i in seconds, `None` if unknown. The model loads with `AutoModelForTokenClassification`, but `punctuate.py` builds the input ids itself because of the pause markers. The code is meant to be read and adapted.

	### Input format

	`punctuate.py` builds the model input itself. This is how, for porting it to another language. As an example, take six words from the recogniser and the silence after each word:

	\| word \| onko \| kaikki \| paikalla \| hyvä \| aloitetaan \| kalavehkeet \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| pause after (s) \| 0.05 \| 0.20 \| 0.90 \| 0.35 \| 0.10 \| unknown \|

	The model gets this token sequence (at most 512 tokens per call):

	```
	[CLS] [unused0] onko kaikki [unused1] paikalla [unused4] hyvä [unused2] aloitetaan kala ##veh ##keet [SEP]
	```

	- `[unused0]` right after `[CLS]` tells the model that pauses are given. Without pauses, leave it out and add no pause markers. The model works both ways.
	- Each word is split into pieces with the WordPiece tokenizer in `tokenizer.json` (cased, no accent stripping). Most words are a single piece, and rarer ones are split, like kala ##veh ##keet.
	- After the last piece of a word comes a pause marker for the silence that follows it. Silences shorter than 0.14 s and unknown ones get no marker. The bucket edges are in `config.json` (`pause_markers`).

	\| silence after the word \| marker \|
	\|---\|---\|
	\| under 0.14 s, or unknown \| none \|
	\| 0.14 to 0.30 s \| `[unused1]` \|
	\| 0.30 to 0.50 s \| `[unused2]` \|
	\| 0.50 to 0.78 s \| `[unused3]` \|
	\| 0.78 to 1.18 s \| `[unused4]` \|
	\| 1.18 s or more \| `[unused5]` \|

	The label of a word is read from its first piece (`kala` for kalavehkeet), and the outputs on the other pieces and on the markers are ignored. Each of the 15 labels (`id2label` in `config.json`) combines the punctuation after the word (`O` for none, `,` `.` `?` `!`) with its casing (nothing for lowercase, `\|U` for a capital first letter, `\|A` for all capitals):

	\| word \| onko \| kaikki \| paikalla \| hyvä \| aloitetaan \| kalavehkeet \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| label \| `O\\|U` \| `O` \| `?` \| `.\\|U` \| `O\\|U` \| `.` \|

	Applied to the words, the labels give:

	```
	Onko kaikki paikalla? Hyvä. Aloitetaan kalavehkeet.
	```

	`punctuate.py` also capitalises the first word and every word after `.` `?` `!`, in case the model misses one.

	### Files

	\| file \| what \|
	\|---\|---\|
	\| `model.safetensors` \| weights, fp32 (115 MiB) \|
	\| `config.json` \| ELECTRA config with `id2label` and `pause_markers` \|
	\| `tokenizer.json`, `tokenizer_config.json` \| WordPiece tokenizer, 26,020 pieces \|
	\| `punctuate.py` \| inference: whole texts, raw labels, live streaming \|

	## Model

	ELECTRA, 12 layers, hidden size 256, 4 attention heads, 30 M parameters (20 M of them in the embeddings). Initialised from [Finnish-NLP/electra-base-generator-finnish](https://huggingface.co/Finnish-NLP/electra-base-generator-finnish) (Apache-2.0) with the vocabulary pruned to the 26,000 pieces the training text uses (plus the marker tokens), then trained by distillation from a larger BERT-based punctuation model fine-tuned for the same task, together with the gold labels.

	## Training data

	Written Finnish with punctuation, and transcribed speech. For the speech, the input is the output of the Mateli recogniser, the punctuation and casing labels were transferred from human transcripts of the same audio by word alignment, and the pauses come from the recogniser's word times. Written text was lowercased and stripped of punctuation for the input, with simulated recognition noise. Windows were 16 to 160 words starting at random positions, so the model also handles text that does not begin at a sentence start.

	## Evaluation

	The test sets are held-out speech transcribed by the Mateli recogniser, with labels transferred from human transcripts of the same audio. The podcast set has shows not seen in training. The metrics follow syvai/danish-pnc. Each is an F1 score per word, for a sentence boundary (`.` `?` `!` interchangeably), comma, question mark and casing (capitalised or all caps vs lowercase).

	\| set \| input \| mode \| boundary F1 \| boundary P \| boundary R \| comma F1 \| question F1 \| casing F1 \|
	\|---\|---\|---\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| conversational podcasts \| words \| whole text \| 0.666 \| 0.638 \| 0.696 \| 0.700 \| 0.594 \| 0.702 \|
	\| conversational podcasts \| words + pauses \| whole text \| 0.680 \| 0.653 \| 0.710 \| 0.702 \| 0.614 \| 0.714 \|
	\| conversational podcasts \| words + pauses \| live \| 0.666 \| 0.644 \| 0.691 \| 0.699 \| 0.591 \| 0.705 \|
	\| TV programmes \| words \| whole text \| 0.709 \| 0.743 \| 0.678 \| 0.582 \| 0.552 \| 0.739 \|
	\| TV programmes \| words + pauses \| whole text \| 0.756 \| 0.780 \| 0.733 \| 0.598 \| 0.607 \| 0.772 \|
	\| TV programmes \| words + pauses \| live \| 0.753 \| 0.780 \| 0.727 \| 0.600 \| 0.602 \| 0.769 \|
	\| long conversations \| words \| whole text \| 0.683 \| 0.668 \| 0.698 \| 0.697 \| 0.669 \| 0.714 \|
	\| long conversations \| words \| live \| 0.657 \| 0.644 \| 0.672 \| 0.692 \| 0.659 \| 0.700 \|

	Live is the streaming policy of `Punctuator.stream()` (three words of lookahead, 64 words of context, a model call every five words). On the podcast set a word is committed on average 2.4 s after it is spoken (90th percentile 3.4 s).

	The scores are well below what the same kind of model reaches on written text, for two reasons. The input has recognition errors, and the reference labels come from human transcribers, who disagree among themselves on spontaneous speech, for example on whether ja, mut or nii starts a new sentence and where commas go in long spoken sentences.

	## Limitations

	- Trained for speech recognition output in Finnish. It also punctuates written text that has lost its punctuation, but that was not the target.
	- Sentence boundaries in disfluent conversation and question marks are the weakest points.
	- Casing is limited to a capital first letter or all capitals (no iPhone-style mixed case).
	- The pauses must come from the same kind of word timing as Mateli's (end of a word's last character to the start of the next word's first character, in 40 ms steps).

	## License

	The model weights and the code in this repository are licensed under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). The base model Finnish-NLP/electra-base-generator-finnish is Apache-2.0.