ekko-pnc / README.md

Add ONNX and Transformers Python quickstarts

d1372b2 verified 9 days ago

8.03 kB

---
language:
- da
license: cc-by-4.0
license_link: https://huggingface.co/RyeAI/ekko-pnc/blob/main/MODEL_LICENSE.md
thumbnail: https://huggingface.co/RyeAI/ekko-pnc/resolve/d3129544e4e0155093e192cac58a9e69a4dea1ff/cover.png
library_name: transformers
pipeline_tag: token-classification
base_model: jonfd/electra-small-nordic
base_model_relation: finetune
datasets:
- alexandrainst/nst-da
- alexandrainst/ftspeech
- CoRal-project/coral-v3
- wikimedia/wikipedia
tags:
- punctuation-restoration
- sentence-boundary-detection
- truecasing
- danish
- onnx
metrics:
- f1
---
![Ekko PnC v2 — Danish punctuation restoration](cover.png)

# Ekko PnC v2 — Danish punctuation restoration and truecasing

Ekko PnC v2 restores punctuation and capitalization in Danish speech transcripts.
It adds commas, full stops, question marks, and exclamation marks while
keeping the original words in the same order. This makes it suitable for
transcripts with word timestamps. The int8 ONNX model runs locally. Ekko v1 Tiny
is a separate speech recognizer; PnC v2 can process its output.

> **Input:** hej mit navn er emil hvordan går det i dag jeg har været på kontoret og bagefter skal jeg hjem
>
> **Output:** Hej, mit navn er Emil. Hvordan går det i dag? Jeg har været på kontoret, og bagefter skal jeg hjem.

## Quickstart

With [uv](https://docs.astral.sh/uv/) and Python 3.11+:

```bash
uvx --from 'ekko-stt[pnc]' ekko pnc "hej mit navn er emil hvordan går det i dag"
```

Expected output: `Hej, mit navn er Emil. Hvordan går det i dag?`

The command installs the [Ekko STT package](https://pypi.org/project/ekko-stt/).
The first PnC run downloads the pinned int8 model from Hugging Face; later runs use the cache.
For longer transcripts, the runtime uses 128-token windows with 12 words of overlap
and keeps every input word in order. Pass continuous text when possible so PnC can
use context across speech recognition chunks.

### Python with ONNX

Install `ekko-stt[pnc]` in your Python environment. This loads the int8 ONNX
export and applies the same windowing as the command above:

```python
from ekko import Punctuator
from ekko.artifacts import ensure_pnc_directory
pnc = Punctuator(ensure_pnc_directory())
print(pnc("hej mit navn er emil hvordan går det i dag"))
```

<details>
<summary>Python with Transformers (PyTorch)</summary>

Install `transformers` and `torch` to use the PyTorch weights directly. This
short-input example maps predictions from the first token of each word back to
the original words:

```python
import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer
repo = "RyeAI/ekko-pnc"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForTokenClassification.from_pretrained(repo).eval()
words = "hej mit navn er emil hvordan går det i dag".split()
inputs = tokenizer(words, is_split_into_words=True, return_tensors="pt")
with torch.inference_mode():
    predictions = model(**inputs).logits.argmax(dim=-1)[0].tolist()
output, seen, capitalize_next = [], set(), True
for word_id, prediction in zip(inputs.word_ids(), predictions):
    if word_id is None or word_id in seen:
        continue
    seen.add(word_id)
    label = model.config.id2label[prediction]
    mark = label.removesuffix("|U")
    word = words[word_id]
    if capitalize_next or label.endswith("|U"):
        word = word[:1].upper() + word[1:]
    output.append(word + ("" if mark == "O" else mark))
    capitalize_next = mark in {".", "?", "!"}
print(" ".join(output))
```

</details>

For longer transcripts, use the ONNX example above; it preserves all words
across overlapping windows.

Version 2 is tuned for **continuous speech** such as dictation and meetings.
The earlier version often missed sentence endings when given a long transcript;
it had mainly learned from short, isolated utterances. Version 2 was trained
on longer stretches of conversation and is substantially better at finding
sentence boundaries in that setting.

## Results

The main evaluation used 88 clips from 8 meetings (4,322 words), with
punctuation reviewed by a person. Both versions received the *same correct
words*, so these scores measure punctuation and casing rather than speech
recognition. The numbers below use the FP32 ONNX model.
F1 combines precision and recall; higher is better.

| Model | Sentence boundary F1 | Boundary recall | Boundary precision | Full stop F1 | Question F1 | Casing F1 |
|---|---:|---:|---:|---:|---:|---:|
| v1 | 0.402 | 0.264 | 0.849 | 0.350 | 0.333 | 0.523 |
| **v2** | **0.698** | **0.619** | 0.801 | **0.654** | **0.462** | **0.736** |

The v2 casing score includes a simple rule that capitalizes the word after a
predicted sentence ending. Without that rule, its casing F1 was 0.653.

On 55 held-out CoRal conversations, v2 improved sentence boundary F1 from
0.095 to 0.474. The punctuation reference for this set was machine-generated
and checked to preserve every word, so this result measures agreement with
that reference. This earlier comparison used the settings
`max_length=256, overlap_words=24`. In a later runtime comparison on the same
conversations, the same v2 int8 weights scored 0.471 with 256/24 and **0.599
with the recommended 128/12**.
The set has been used in earlier experiments, so this is guidance for runtime
configuration rather than an independent evaluation of new model weights.

Very short clips are a trade-off: on isolated 3–4 second utterances, boundary
F1 decreased from 0.865 (v1) to 0.818 (v2). Version 2 may add extra full stops
on such inputs.

## Speed

**1,000 Danish words in about 0.09 seconds** on an Apple M1 Pro CPU, after model
loading. The int8 ONNX benchmark includes tokenization, inference, and rendering;
it excludes download, speech recognition, and browser WASM. Speed varies by device
and input length.

## Files and use

| Format | Files | When to use |
|---|---|---|
| int8 ONNX | `pnc.int8.onnx`, `tokenizer.json`, `config.json` | Local and browser inference; about 22.4 MB for the model file |
| FP32 ONNX | `pnc.onnx`, `pnc.onnx.data`, `tokenizer.json`, `config.json` | Best measured quality when memory is less constrained |

On the meeting evaluation, int8 was behind FP32 by 0.006 boundary F1,
0.022 question F1, and 0.016 casing F1.

<details>
<summary>Exact revision and checksums</summary>

The int8 runtime pins revision
`d3129544e4e0155093e192cac58a9e69a4dea1ff`. SHA-256 checksums:

```text
pnc.int8.onnx  46bb979d409731872f10b6cb71298594595f16368d2aec2fcaf94d87b4266527
tokenizer.json a1e0d43ee307ad7a1ccc23e6e5398d877a68ca7f576067ac507625a0d50e9502
config.json    dc655c46f69eeb15dd43786ad58fb5862cc794b36cc9d2c426ab5bc4b01f23ca
```

The files in this repository also have checksums in `SHA256SUMS`. Version 1
remains available through the `v1` tag.

</details>

## Training and limitations

Ekko PnC builds on
[`jonfd/electra-small-nordic`](https://huggingface.co/jonfd/electra-small-nordic).
Its Danish training text included Wikipedia, NST, CoRal, and FT Speech.
Training combined human-punctuated source text with machine-generated
punctuation labels. The main meeting evaluation above uses human-reviewed
punctuation; the CoRal conversation evaluation uses machine-generated
punctuation references.

- Punctuation comes from text alone; the model cannot hear pauses or intonation.
  Question marks remain its weakest common punctuation class.
- It handles capitalization as an initial uppercase letter. It cannot create
  mixed-case words such as `iPhone` from lowercase input; existing capitals
  are preserved by the Ekko runtimes.
- The human-reviewed meeting set is small: 8 meetings and 324 reference full
  stops. Results on other speaking styles may differ.

## License and data terms

Ekko PnC v2 is released under CC BY 4.0. See
[`MODEL_LICENSE.md`](MODEL_LICENSE.md), [`NOTICE`](NOTICE), and
[`LICENSES/`](LICENSES/) for the terms and source notices. CoRal's use
restrictions also apply to this trained model. The FT Speech and NST notices
are included in the license bundle.

	---
	language:
	- da
	license: cc-by-4.0
	license_link: https://huggingface.co/RyeAI/ekko-pnc/blob/main/MODEL_LICENSE.md
	thumbnail: https://huggingface.co/RyeAI/ekko-pnc/resolve/d3129544e4e0155093e192cac58a9e69a4dea1ff/cover.png
	library_name: transformers
	pipeline_tag: token-classification
	base_model: jonfd/electra-small-nordic
	base_model_relation: finetune
	datasets:
	- alexandrainst/nst-da
	- alexandrainst/ftspeech
	- CoRal-project/coral-v3
	- wikimedia/wikipedia
	tags:
	- punctuation-restoration
	- sentence-boundary-detection
	- truecasing
	- danish
	- onnx
	metrics:
	- f1
	---

	![Ekko PnC v2 — Danish punctuation restoration](cover.png)

	# Ekko PnC v2 — Danish punctuation restoration and truecasing

	Ekko PnC v2 restores punctuation and capitalization in Danish speech transcripts.
	It adds commas, full stops, question marks, and exclamation marks while
	keeping the original words in the same order. This makes it suitable for
	transcripts with word timestamps. The int8 ONNX model runs locally. Ekko v1 Tiny
	is a separate speech recognizer; PnC v2 can process its output.

	> Input: hej mit navn er emil hvordan går det i dag jeg har været på kontoret og bagefter skal jeg hjem
	>
	> Output: Hej, mit navn er Emil. Hvordan går det i dag? Jeg har været på kontoret, og bagefter skal jeg hjem.

	## Quickstart

	With [uv](https://docs.astral.sh/uv/) and Python 3.11+:

	```bash
	uvx --from 'ekko-stt[pnc]' ekko pnc "hej mit navn er emil hvordan går det i dag"
	```

	Expected output: `Hej, mit navn er Emil. Hvordan går det i dag?`

	The command installs the [Ekko STT package](https://pypi.org/project/ekko-stt/).
	The first PnC run downloads the pinned int8 model from Hugging Face; later runs use the cache.
	For longer transcripts, the runtime uses 128-token windows with 12 words of overlap
	and keeps every input word in order. Pass continuous text when possible so PnC can
	use context across speech recognition chunks.

	### Python with ONNX

	Install `ekko-stt[pnc]` in your Python environment. This loads the int8 ONNX
	export and applies the same windowing as the command above:

	```python
	from ekko import Punctuator
	from ekko.artifacts import ensure_pnc_directory

	pnc = Punctuator(ensure_pnc_directory())
	print(pnc("hej mit navn er emil hvordan går det i dag"))
	```

	<details>
	<summary>Python with Transformers (PyTorch)</summary>

	Install `transformers` and `torch` to use the PyTorch weights directly. This
	short-input example maps predictions from the first token of each word back to
	the original words:

	```python
	import torch
	from transformers import AutoModelForTokenClassification, AutoTokenizer

	repo = "RyeAI/ekko-pnc"
	tokenizer = AutoTokenizer.from_pretrained(repo)
	model = AutoModelForTokenClassification.from_pretrained(repo).eval()
	words = "hej mit navn er emil hvordan går det i dag".split()
	inputs = tokenizer(words, is_split_into_words=True, return_tensors="pt")

	with torch.inference_mode():
	predictions = model(**inputs).logits.argmax(dim=-1)[0].tolist()

	output, seen, capitalize_next = [], set(), True
	for word_id, prediction in zip(inputs.word_ids(), predictions):
	if word_id is None or word_id in seen:
	continue
	seen.add(word_id)
	label = model.config.id2label[prediction]
	mark = label.removesuffix("\|U")
	word = words[word_id]
	if capitalize_next or label.endswith("\|U"):
	word = word[:1].upper() + word[1:]
	output.append(word + ("" if mark == "O" else mark))
	capitalize_next = mark in {".", "?", "!"}

	print(" ".join(output))
	```

	</details>

	For longer transcripts, use the ONNX example above; it preserves all words
	across overlapping windows.

	Version 2 is tuned for continuous speech such as dictation and meetings.
	The earlier version often missed sentence endings when given a long transcript;
	it had mainly learned from short, isolated utterances. Version 2 was trained
	on longer stretches of conversation and is substantially better at finding
	sentence boundaries in that setting.

	## Results

	The main evaluation used 88 clips from 8 meetings (4,322 words), with
	punctuation reviewed by a person. Both versions received the *same correct
	words*, so these scores measure punctuation and casing rather than speech
	recognition. The numbers below use the FP32 ONNX model.
	F1 combines precision and recall; higher is better.

	\| Model \| Sentence boundary F1 \| Boundary recall \| Boundary precision \| Full stop F1 \| Question F1 \| Casing F1 \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| v1 \| 0.402 \| 0.264 \| 0.849 \| 0.350 \| 0.333 \| 0.523 \|
	\| v2 \| 0.698 \| 0.619 \| 0.801 \| 0.654 \| 0.462 \| 0.736 \|

	The v2 casing score includes a simple rule that capitalizes the word after a
	predicted sentence ending. Without that rule, its casing F1 was 0.653.

	On 55 held-out CoRal conversations, v2 improved sentence boundary F1 from
	0.095 to 0.474. The punctuation reference for this set was machine-generated
	and checked to preserve every word, so this result measures agreement with
	that reference. This earlier comparison used the settings
	`max_length=256, overlap_words=24`. In a later runtime comparison on the same
	conversations, the same v2 int8 weights scored 0.471 with 256/24 and **0.599
	with the recommended 128/12**.
	The set has been used in earlier experiments, so this is guidance for runtime
	configuration rather than an independent evaluation of new model weights.

	Very short clips are a trade-off: on isolated 3–4 second utterances, boundary
	F1 decreased from 0.865 (v1) to 0.818 (v2). Version 2 may add extra full stops
	on such inputs.

	## Speed

	1,000 Danish words in about 0.09 seconds on an Apple M1 Pro CPU, after model
	loading. The int8 ONNX benchmark includes tokenization, inference, and rendering;
	it excludes download, speech recognition, and browser WASM. Speed varies by device
	and input length.

	## Files and use

	\| Format \| Files \| When to use \|
	\|---\|---\|---\|
	\| int8 ONNX \| `pnc.int8.onnx`, `tokenizer.json`, `config.json` \| Local and browser inference; about 22.4 MB for the model file \|
	\| FP32 ONNX \| `pnc.onnx`, `pnc.onnx.data`, `tokenizer.json`, `config.json` \| Best measured quality when memory is less constrained \|

	On the meeting evaluation, int8 was behind FP32 by 0.006 boundary F1,
	0.022 question F1, and 0.016 casing F1.

	<details>
	<summary>Exact revision and checksums</summary>

	The int8 runtime pins revision
	`d3129544e4e0155093e192cac58a9e69a4dea1ff`. SHA-256 checksums:

	```text
	pnc.int8.onnx 46bb979d409731872f10b6cb71298594595f16368d2aec2fcaf94d87b4266527
	tokenizer.json a1e0d43ee307ad7a1ccc23e6e5398d877a68ca7f576067ac507625a0d50e9502
	config.json dc655c46f69eeb15dd43786ad58fb5862cc794b36cc9d2c426ab5bc4b01f23ca
	```

	The files in this repository also have checksums in `SHA256SUMS`. Version 1
	remains available through the `v1` tag.

	</details>

	## Training and limitations

	Ekko PnC builds on
	[`jonfd/electra-small-nordic`](https://huggingface.co/jonfd/electra-small-nordic).
	Its Danish training text included Wikipedia, NST, CoRal, and FT Speech.
	Training combined human-punctuated source text with machine-generated
	punctuation labels. The main meeting evaluation above uses human-reviewed
	punctuation; the CoRal conversation evaluation uses machine-generated
	punctuation references.

	- Punctuation comes from text alone; the model cannot hear pauses or intonation.
	Question marks remain its weakest common punctuation class.
	- It handles capitalization as an initial uppercase letter. It cannot create
	mixed-case words such as `iPhone` from lowercase input; existing capitals
	are preserved by the Ekko runtimes.
	- The human-reviewed meeting set is small: 8 meetings and 324 reference full
	stops. Results on other speaking styles may differ.

	## License and data terms

	Ekko PnC v2 is released under CC BY 4.0. See
	[`MODEL_LICENSE.md`](MODEL_LICENSE.md), [`NOTICE`](NOTICE), and
	[`LICENSES/`](LICENSES/) for the terms and source notices. CoRal's use
	restrictions also apply to this trained model. The FT Speech and NST notices
	are included in the license bundle.