README.md

---
license: mit
language:
- ar
- az
- bg
- bn
- bs
- ca
- cs
- da
- de
- el
- en
- eo
- es
- et
- fa
- fi
- fr
- he
- hi
- hr
- hu
- hy
- id
- it
- ja
- ka
- kk
- ko
- ku
- lt
- lv
- mk
- mn
- mr
- ms
- my
- nb
- nl
- pl
- pt
- ro
- ru
- sk
- sl
- sq
- sr
- sv
- sw
- ta
- th
- tr
- uk
- ur
- vi
- zh
- af
- be
- cy
- eu
- ga
- gl
- is
- kn
- ml
- ne
- nn
- si
- te
- tl
- uz
- gu
- pa
- km
- la
- lb
- mt
- so
- am
- ky
pipeline_tag: token-classification
tags:
- punctuation
- punctuation-restoration
- truecasing
- capitalization
- asr
- speech-recognition-postprocessing
- multilingual
- ensemble
- modernbert
- xlm-roberta
base_model:
- jhu-clsp/mmBERT-base
- valkayuh/dewpoint
base_model_relation: quantized
datasets:
- wikimedia/wikipedia
- Helsinki-NLP/europarl
- HPLT/HPLT2.0_cleaned
- facebook/voxpopuli
- CohereLabs/aya_dataset
metrics:
- f1
model-index:
- name: Dewpoint
  results:
  - task:
      type: token-classification
      name: Punctuation restoration
    dataset:
      name: IWSLT 2017 TED, official tst2010-2015 (9 languages)
      type: IWSLT/iwslt2017
    metrics:
    - type: f1
      value: 0.7559
      name: Macro-F1 (COMMA, PERIOD, QUESTION), mean over languages
    - type: f1
      value: 0.6557
      name: Comma F1
    - type: f1
      value: 0.7923
      name: Period F1
    - type: f1
      value: 0.8197
      name: Question F1
  - task:
      type: token-classification
      name: Punctuation restoration
    dataset:
      name: WIT3 TED, same 90 talks (57 language tracks)
      type: wit3
    metrics:
    - type: f1
      value: 0.7303
      name: Macro-F1 (COMMA, PERIOD, QUESTION), mean over languages
    - type: f1
      value: 0.6482
      name: Comma F1
    - type: f1
      value: 0.7794
      name: Period F1
    - type: f1
      value: 0.7632
      name: Question F1
  - task:
      type: token-classification
      name: Punctuation restoration
    dataset:
      name: IWSLT TED through TTS and Whisper ASR (33 languages)
      type: iwslt-ted-asr
    metrics:
    - type: f1
      value: 0.6332
      name: Macro-F1 (COMMA, PERIOD, QUESTION), mean over languages
    - type: f1
      value: 0.5656
      name: Comma F1
    - type: f1
      value: 0.7102
      name: Period F1
    - type: f1
      value: 0.6238
      name: Question F1
---

# About

This model was quantized to BF16 from valkayuh/dewpoint. Refer to the [original model card](https://huggingface.co/valkayuh/dewpoint) for more details on the model.

# Original Model Card

![Dewpoint: punctuation and capitals for the spoken word](assets/banner.jpg)

# Dewpoint

**Punctuation and truecasing for speech-recognition output, in 79 languages.** Speech
recognisers hand back a stream of lowercase words with no punctuation, and Dewpoint
restores it: it adds the commas, full stops and question marks, and the capitals.
Open weights, MIT licence, free for everyone.

```text
so i said meet at three thirty tuesday what do you think the iphone was announced by steve jobs in san francisco
→ So I said meet at three thirty Tuesday. What do you think? The iPhone was announced by Steve Jobs in San Francisco.

ich weiß nicht ob wir das schaffen aber wir sollten es versuchen was meinst du
→ Ich weiß nicht, ob wir das schaffen, aber wir sollten es versuchen. Was meinst du?

πού είναι ο σταθμός ξέρεις να μου πεις
→ Πού είναι ο σταθμός; Ξέρεις να μου πεις;
```

These are unedited outputs of this checkpoint. Dewpoint is a **tagger, not a
generator**. For each word it decides which mark follows (none, `,` `.` `?` `!`) and how
the word is cased (lower, Capitalised, UPPER). It never adds, drops, reorders or rewrites
a word, so it cannot invent content.

## Results

On the IWSLT TED benchmark, Dewpoint scores higher than every other multilingual
punctuation model we tested, under all three test conditions. On the two text test
sets it is behind no other model in any language: **0 of 9** on the official
set and **0 of 57** on the extended one. Under ASR noise it trails in
**7 of 33**, 5 of them to raw Whisper.

![Macro-F1 on three benchmarks](assets/benchmarks.jpg)

| system | params | official, 9 lang | WIT3, 57 lang | ASR, 33 lang |
|---|---|---|---|---|
| **Dewpoint** | 869M | **0.7559** | **0.7303** | **0.6332** |
| Dewpoint, mmBERT-base half alone | 307M | 0.7392 | 0.7098 | 0.6197 |
| Dewpoint, XLM-R-large half alone | 561M | 0.7443 | 0.7241 | 0.6208 |
| [oliverguhr/fullstop-punctuation-multilang-large](https://huggingface.co/oliverguhr/fullstop-punctuation-multilang-large) | 559M | 0.6438 | 0.5984 | 0.5301 |
| [kredor/punctuate-all](https://huggingface.co/kredor/punctuate-all) | 278M | 0.6344 | 0.5950 | 0.5333 |
| [oliverguhr/fullstop-punctuation-multilingual-sonar-base](https://huggingface.co/oliverguhr/fullstop-punctuation-multilingual-sonar-base) | 278M | 0.6381 | 0.5653 | 0.5205 |
| [1-800-BAD-CODE/xlm-roberta_punctuation_fullstop_truecase](https://huggingface.co/1-800-BAD-CODE/xlm-roberta_punctuation_fullstop_truecase) | 278M | 0.6204 | 0.5710 | 0.5346 |
| raw [openai/whisper-large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo) punctuation | 809M | — | — | 0.4436 |

Macro-F1 over COMMA, PERIOD and QUESTION, averaged over languages. Every system runs
through the same harness and gets the same dev-set threshold calibration.

**Head to head on each model's own languages.** The averages above include languages the
other models were never trained on, which pulls their means down. Scored only on the
languages each model lists on its own model card, the margins are smaller:

| other model | languages it supports | official | WIT3 | ASR |
|---|---|---|---|---|
| [oliverguhr/fullstop-punctuation-multilang-large](https://huggingface.co/oliverguhr/fullstop-punctuation-multilang-large) | 4 | 0.755 vs **0.791** (4) | 0.758 vs **0.793** (5) | 0.741 vs **0.772** (4) |
| [oliverguhr/fullstop-punctuation-multilingual-sonar-base](https://huggingface.co/oliverguhr/fullstop-punctuation-multilingual-sonar-base) | 5 | 0.734 vs **0.786** (5) | 0.752 vs **0.797** (6) | 0.702 vs **0.753** (5) |
| [kredor/punctuate-all](https://huggingface.co/kredor/punctuate-all) | 12 | 0.726 vs **0.786** (5) | 0.733 vs **0.797** (14) | 0.695 vs **0.755** (9) |
| [1-800-BAD-CODE/xlm-roberta_punctuation_fullstop_truecase](https://huggingface.co/1-800-BAD-CODE/xlm-roberta_punctuation_fullstop_truecase) | 47 | 0.620 vs **0.756** (9) | 0.598 vs **0.744** (37) | 0.540 vs **0.636** (27) |

Dewpoint is ahead in every one of these languages except Bengali on Whisper ASR output against xlm-r truecase.

Each cell is that model's mean macro-F1 vs Dewpoint's on the same languages, with the
number of languages in brackets. This is the fairer comparison.

---

## Quick start

```bash
pip install torch transformers safetensors huggingface_hub numpy   # GPU or CPU
# or, with no torch at all:
pip install onnxruntime tokenizers huggingface_hub numpy
```

```python
import os, sys
from huggingface_hub import hf_hub_download
sys.path.insert(0, os.path.dirname(hf_hub_download("valkayuh/dewpoint", "dewpoint.py")))
from dewpoint import Punctuator

p = Punctuator.from_pretrained("valkayuh/dewpoint")   # torch if installed, else ONNX
p.restore("what time is it in tokyo right now", lang="en")
# 'What time is it in Tokyo right now?'
```

`from_pretrained` downloads only what the chosen backend needs. The torch backend
fetches the safetensors (3.5 GB, or 1.2 GB with `members=["mmbert-base"]`), the ONNX
backend fetches the ONNX graphs, and neither fetches the other.

**Always pass the language.** It selects the calibrated decision thresholds, turns
casing off for scripts that have none, and picks the right marks to write: `؟` for
Arabic, `;` for the Greek question mark, `。` and `，` for Chinese, `।` for Hindi.

**Single-model mode.** Pass `members=["mmbert-base"]` to download and run only the
307M-parameter mmBERT-base half. It uses its own calibration, runs in about half the
GPU time, and still beats every other model we tested on every benchmark:

```python
p = Punctuator.from_pretrained("valkayuh/dewpoint", members=["mmbert-base"])
```

**Streaming.** For live captions, each word is released only once it has right-hand
context and a stable label, so marks don't flicker:

```python
s = p.stream(lang="en", lag=6)
for chunk in asr_chunks:            # partial transcripts as they arrive
    print(s.push(chunk), end="", flush=True)
print(s.finish())
```

**Labels and probabilities.** `p.predict(words, lang)` returns per-word labels plus
the averaged posteriors (`punct_probs`, `case_probs`), in case you want confidence
scores or your own decision rule.

**Command line.**

```bash
echo "i called the office this morning but nobody answered did you get my message" | python dewpoint.py --lang en
# I called the office this morning, but nobody answered. Did you get my message?
```

## Formats and integration

| you want | use |
|---|---|
| Python with a GPU | `Punctuator(..., backend="torch")`: bf16 on CUDA, fp32 on CPU |
| Python without torch, e.g. a slim CPU server | `Punctuator(..., backend="onnx")`: needs only `onnxruntime`, `tokenizers`, `numpy` |
| a local HTTP API | `python serve.py` gives `POST /restore {"text", "lang"}` (standard library only) |
| a hosted API | deploy this repo as a [Hugging Face Inference Endpoint](https://huggingface.co/inference-endpoints); `handler.py` serves `{"inputs", "parameters": {"lang"}}` |
| the shell | `python dewpoint.py --lang de "…"` or pipe text on stdin |
| C#, Java, JavaScript, C++, mobile | the ONNX graphs in `onnx/`, with the pre- and post-processing below |

**ONNX graphs.** There is one per member, in `onnx/mmbert-base/` and
`onnx/xlm-roberta-large/` (the latter keeps its weights in `model.onnx.data` beside
the graph). Both are fp32, opset 17, with dynamic batch and sequence axes:

- inputs: `input_ids`, `attention_mask`, int64 `[batch, seq]`, from that member's
  `tokenizer.json` with the words passed pre-tokenised
- outputs: `punct_logits` `[batch, seq, 5]` and `case_logits` `[batch, seq, 3]`

Around the graph, `dewpoint.py` does four things, and a port needs the same four:
1. split the input into overlapping windows of at most 508 subwords, keeping each
   window's centre (`plan_windows`)
2. read each word's label from its first subword
3. average the two members' softmax posteriors
4. add the calibrated per-class bias from `ensemble_config.json` for the language, then
   take the argmax

**Verified.** Run over the whole official test set on CPU, the ONNX backend scores **0.7558** macro-F1 against the published 0.7559; the largest difference in any scored language is 0.0011, the gap between fp32 ONNX and the bf16 GPU run the published numbers come from. Its tokenisation matches transformers exactly on 323 documents in 57 languages.

### Speed

Measured on a 584-word TED passage, after warm-up, with the code in this repo:

| hardware | Dewpoint (869M) | mmBERT half (307M) |
|---|---|---|
| RTX 5080, torch bf16 | 45 ms (13k words/s) | 23 ms (25k words/s) |
| Ryzen 7 5800X3D, torch fp32 | 2.18 s | 632 ms (1k words/s) |
| Ryzen 7 5800X3D, ONNX Runtime fp32 | 2.13 s | 606 ms (1k words/s) |

Input of any length is handled with overlapping 512-token windows. Only the centre of
each window commits its predictions, because words near a window edge lack context and
score measurably worse.

---

## Model

Dewpoint is an **ensemble of two dual-head token classifiers**. Both members read the
same word list and produce one posterior per word, and those posteriors are averaged.

| member | encoder | architecture | parameters | of which embeddings |
|---|---|---|---|---|
| `mmbert-base/` | [jhu-clsp/mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) | ModernBERT | 307.5M | 196.6M |
| `xlm-roberta-large/` | [FacebookAI/xlm-roberta-large](https://huggingface.co/FacebookAI/xlm-roberta-large) | XLM-RoBERTa | 561.0M | 256.5M |

**Each member** is an encoder, then a residual shared trunk (Linear, GELU, LayerNorm),
then two linear heads:

- **punctuation**, 5 classes: `O` `COMMA` `PERIOD` `QUESTION` `EXCLAM`
- **case**, 3 classes: `LOWER` (*meeting*), `CAP` for a capitalised first letter
  (*Tuesday*, *Tokyo*, a sentence's first word), and `UPPER` for a word written
  entirely in capitals (*NASA*, *BBC*, *DNA*). Mixed-case words such as *iPhone* are
  handled by a word list, described below.

There is one decision per word, not per subword: the label sits on each word's first
subword. Punctuation and case are learned together because they are correlated: a
sentence end means a capital comes next.

**Combining the members.** Both members label the same words, so their outputs line up
without any tokenizer reconciliation. The average is taken in **probability space**, not
log space. With a geometric mean, one confident but wrong member could veto a class;
the arithmetic mean degrades gracefully instead.

**Calibration.** Class weighting stops the model predicting "no punctuation"
everywhere, but it buys recall at the cost of precision, so a plain argmax is not the
F1-optimal rule. Instead, a per-class additive bias on the log-posterior is fitted on
IWSLT dev2010, which is held out of every test table. Whether to add per-language
biases on top, and how far to trust them, is chosen by cross-validation *inside*
dev2010: its talks are split in half five times, and each strategy is fitted on one
half and scored on the other. An unshrunk bias per language, fitted on eight talks,
overfits: it was never the winning strategy, and for Dewpoint it scored worst. Dewpoint
uses per-language biases pulled three quarters of the way back to the pooled one. The ensemble is
calibrated as a system in its own right, and every other model in the tables went
through the identical procedure, each choosing its own strategy.

**Post-processing**, in `restore()` only (`predict()` returns the raw decision):
- a capital after every sentence-final mark
- a gazetteer of 4,896 mixed-case forms such as *iPhone*, *McDonald* and *GHz*
- a closing mark if the text would otherwise end on none

---

## Evaluation

- **Three test conditions:**
  - the IWSLT 2017 official tst2010–2015 sets, with 9 scorable languages
  - the WIT3 releases of the *same 90 talks* in 57 language tracks, so only the
    language changes
  - an ASR condition: TED text spoken by TTS, transcribed by Whisper
    large-v3-turbo, and aligned back to the reference, in 33 languages
- **Metric.** Macro-F1 over COMMA, PERIOD and QUESTION, averaged over languages.
  EXCLAM is folded into PERIOD, because no other model emits it. Accuracy is never
  reported: about 87% of words carry no mark, so a model that predicts nothing is
  "87% accurate".
- **Exclusions.** Japanese and Thai are excluded, and named here. Their TED references
  carry almost no punctuation (0.38% and 0.12% of tokens).
- **Reference audit.** Two errors in the reference's label maps were fixed for every
  system alike. Armenian writes its full stop as an ASCII colon; fixing that doubled
  *every* system's Armenian score. And the Burmese comma U+104A had been mapped to a
  full stop.
- **No TED data was used for training.** TED is the test set.

### Every language on the extended set

![Per-language macro-F1 on 57 WIT3 language tracks](assets/per_language_wit3.jpg)

### Under ASR noise

![Per-language gap to the best rival under ASR](assets/asr_condition_gap.jpg)

5 of the 7 are to raw Whisper (English, Spanish, French, Italian, Portuguese). Whisper punctuates from the audio itself, so it can use pauses and intonation that a text-only tagger never sees. The rest: Bengali (−0.044 to fullstop-sonar), Albanian (−0.025 to xlm-r truecase). Languages marked * in the chart have under 3,000 scored words, so their gaps are noisy.

### Per class, and truecasing

![Per-class F1](assets/per_class.jpg)

Mean per-class F1 against the strongest other model:

| test set | system | COMMA | PERIOD | QUESTION |
|---|---|---|---|---|
| IWSLT 2017 official (9 languages) | Dewpoint | **0.6557** | **0.7923** | **0.8197** |
| IWSLT 2017 official (9 languages) | fullstop-large | 0.5334 | 0.7206 | 0.6772 |
| WIT3, same 90 talks (57 language tracks) | Dewpoint | **0.6482** | **0.7794** | **0.7632** |
| WIT3, same 90 talks (57 language tracks) | fullstop-large | 0.5067 | 0.6988 | 0.5896 |
| Whisper ASR output (33 languages) | Dewpoint | **0.5656** | **0.7102** | **0.6238** |
| Whisper ASR output (33 languages) | fullstop-large | 0.4489 | 0.6556 | 0.4858 |

Truecasing F1 against the only other model that restores case, on cased scripts:

| test set | cased languages | Dewpoint CAP | UPPER | xlm-r truecase CAP | UPPER | languages behind |
|---|---|---|---|---|---|---|
| IWSLT 2017 official (9 languages) | 6 | **0.8772** | **0.8748** | 0.7708 | 0.7888 | 0 |
| WIT3, same 90 talks (57 language tracks) | 43 | **0.8533** | **0.8599** | 0.7173 | 0.8005 | 0 |
| Whisper ASR output (33 languages) | 25 | **0.8007** | **0.7584** | 0.7136 | 0.7083 | 0 |

**Reading the truecasing table.** `CAP` scores words whose first letter should be a
capital: names, places, and the first word of every sentence. `UPPER` scores words
written entirely in capitals, which in practice means acronyms and initialisms such as
*NASA*, *BBC* or *DNA*. These are much rarer, so they get their own score and are not
swamped by ordinary capitals. A single capital letter such as *I* counts as `CAP`.

The truecasing comparison covers cased scripts only. On caseless scripts (Arabic,
Chinese, Korean, Hindi and others) the case head is masked by design, so Latin loanwords
inside them are never capitalised. `xlm-roberta_punctuation_fullstop_truecase` does
capitalise them, so it wins on those words by construction. The native EXCLAM class
scores 0.11 F1 on the official set and 0.07 on WIT3. No other
model emits it at all.

### Size

![Parameters against macro-F1](assets/size_vs_score.jpg)

### Where the gain comes from

Every other model in these tables is an XLM-RoBERTa model, so a win could come from the
architecture or from the training recipe. To separate the two, we trained
**XLM-R-large, the encoder `fullstop-punctuation-multilang-large` uses, on this corpus
with this recipe**.

![Method against architecture](assets/method_vs_architecture.jpg)

The recipe is worth **+0.08 to +0.13** macro-F1. Swapping the encoder is worth −0.004
to +0.016. The corpus, the orthography fixes and the decision rule matter roughly an
order of magnitude more than the encoder. Neither encoder dominated the other, and
averaging exploits exactly that: members that do not make the same errors.

### Per-language tables

<details>
<summary>IWSLT 2017 official, 9 languages</summary>

| language | Dewpoint | mmBERT half alone | best other model | which | gap |
|---|---|---|---|---|---|
| Arabic (`ar`) | 0.6538 | 0.6229 | 0.5586 | fullstop-large | +0.095 |
| Chinese (`zh`) | 0.6551 | 0.6546 | 0.5502 | xlm-r truecase | +0.105 |
| Dutch (`nl`) | 0.7686 | 0.7587 | 0.7304 | fullstop-sonar | +0.038 |
| English (`en`) | 0.8033 | 0.7935 | 0.7430 | fullstop-large | +0.060 |
| French (`fr`) | 0.7956 | 0.7841 | 0.7581 | fullstop-large | +0.038 |
| German (`de`) | 0.8496 | 0.8402 | 0.8338 | fullstop-large | +0.016 |
| Italian (`it`) | 0.7145 | 0.6886 | 0.6838 | fullstop-large | +0.031 |
| Korean (`ko`) | 0.8010 | 0.7825 | 0.6553 | xlm-r truecase | +0.146 |
| Romanian (`ro`) | 0.7618 | 0.7274 | 0.6860 | fullstop-large | +0.076 |

</details>

<details>
<summary>WIT3 extended set, 57 language tracks</summary>

| language | Dewpoint | mmBERT half alone | best other model | which | gap |
|---|---|---|---|---|---|
| Albanian (`sq`) | 0.7152 | 0.6828 | 0.6036 | fullstop-large | +0.112 |
| Arabic (`ar`) | 0.6538 | 0.6225 | 0.5585 | fullstop-large | +0.095 |
| Armenian (`hy`) | 0.7902 | 0.7709 | 0.4924 | kredor | +0.298 |
| Azerbaijani (`az`) | 0.7286 | 0.7105 | 0.5832 | fullstop-sonar | +0.145 |
| Bengali (`bn`) | 0.6940 | 0.6497 | 0.5584 | fullstop-large | +0.136 |
| Bosnian (`bs`) | 0.7753 | 0.7540 | 0.6324 | fullstop-large | +0.143 |
| Bulgarian (`bg`) | 0.8089 | 0.7906 | 0.7491 | kredor | +0.060 |
| Burmese (`my`) | 0.4427 | 0.3569 | 0.3553 | xlm-r truecase | +0.087 |
| Catalan (`ca`) | 0.7189 | 0.6880 | 0.6339 | fullstop-large | +0.085 |
| Chinese (`zh`) | 0.6543 | 0.6349 | 0.4509 | xlm-r truecase | +0.203 |
| Chinese (Simplified) (`zh-cn`) | 0.6486 | 0.6483 | 0.5439 | xlm-r truecase | +0.105 |
| Chinese (Traditional) (`zh-tw`) | 0.5941 | 0.5960 | 0.5136 | xlm-r truecase | +0.081 |
| Croatian (`hr`) | 0.7712 | 0.7540 | 0.6942 | fullstop-large | +0.077 |
| Czech (`cs`) | 0.8120 | 0.7930 | 0.7461 | kredor | +0.066 |
| Danish (`da`) | 0.7661 | 0.7498 | 0.6975 | fullstop-large | +0.069 |
| Dutch (`nl`) | 0.8171 | 0.8060 | 0.7819 | fullstop-sonar | +0.035 |
| English (`en`) | 0.8036 | 0.7937 | 0.7431 | fullstop-large | +0.061 |
| Esperanto (`eo`) | 0.7663 | 0.7297 | 0.6385 | fullstop-sonar | +0.128 |
| Estonian (`et`) | 0.8146 | 0.7884 | 0.7178 | fullstop-large | +0.097 |
| Finnish (`fi`) | 0.8207 | 0.7996 | 0.7283 | kredor | +0.092 |
| French (`fr`) | 0.7961 | 0.7842 | 0.7588 | fullstop-large | +0.037 |
| French (Canada) (`fr-ca`) | 0.7778 | 0.7604 | 0.7445 | fullstop-sonar | +0.033 |
| Georgian (`ka`) | 0.7142 | 0.6583 | 0.6179 | fullstop-large | +0.096 |
| German (`de`) | 0.8472 | 0.8392 | 0.8302 | fullstop-large | +0.017 |
| Greek (`el`) | 0.7331 | 0.7071 | 0.6951 | fullstop-large | +0.038 |
| Hebrew (`he`) | 0.7398 | 0.7030 | 0.6506 | fullstop-large | +0.089 |
| Hindi (`hi`) | 0.5855 | 0.5600 | 0.4925 | fullstop-large | +0.093 |
| Hungarian (`hu`) | 0.7903 | 0.7637 | 0.6771 | fullstop-large | +0.113 |
| Indonesian (`id`) | 0.7350 | 0.7174 | 0.6401 | fullstop-large | +0.095 |
| Italian (`it`) | 0.7412 | 0.7211 | 0.7148 | fullstop-large | +0.026 |
| Kazakh (`kk`) | 0.7657 | 0.7556 | 0.6508 | xlm-r truecase | +0.115 |
| Korean (`ko`) | 0.8008 | 0.7825 | 0.6564 | xlm-r truecase | +0.144 |
| Kurdish (`ku`) | 0.1114 | 0.2843 | 0.0298 | xlm-r truecase | +0.082 |
| Latvian (`lv`) | 0.8174 | 0.7928 | 0.7117 | kredor | +0.106 |
| Lithuanian (`lt`) | 0.7935 | 0.7725 | 0.6945 | kredor | +0.099 |
| Macedonian (`mk`) | 0.7479 | 0.7190 | 0.6378 | fullstop-large | +0.110 |
| Malay (`ms`) | 0.7748 | 0.7426 | 0.6387 | fullstop-large | +0.136 |
| Marathi (`mr`) | 0.6624 | 0.6120 | 0.5224 | fullstop-large | +0.140 |
| Mongolian (`mn`) | 0.7962 | 0.7447 | 0.6349 | xlm-r truecase | +0.161 |
| Norwegian (`nb`) | 0.7704 | 0.7531 | 0.7399 | fullstop-large | +0.031 |
| Persian (`fa`) | 0.6895 | 0.6700 | 0.5903 | fullstop-large | +0.099 |
| Polish (`pl`) | 0.8181 | 0.7999 | 0.7696 | kredor | +0.049 |
| Portuguese (`pt`) | 0.7725 | 0.7555 | 0.7095 | kredor | +0.063 |
| Portuguese (Brazil) (`pt-br`) | 0.7550 | 0.7354 | 0.6701 | kredor | +0.085 |
| Romanian (`ro`) | 0.7623 | 0.7320 | 0.6975 | fullstop-large | +0.065 |
| Russian (`ru`) | 0.8354 | 0.8218 | 0.7312 | kredor | +0.104 |
| Serbian (`sr`) | 0.7783 | 0.7588 | 0.6984 | fullstop-large | +0.080 |
| Slovak (`sk`) | 0.8050 | 0.7787 | 0.7440 | kredor | +0.061 |
| Slovenian (`sl`) | 0.8386 | 0.8211 | 0.8100 | kredor | +0.029 |
| Spanish (`es`) | 0.7582 | 0.7463 | 0.7070 | fullstop-large | +0.051 |
| Swahili (`sw`) | 0.6403 | 0.5944 | 0.5180 | fullstop-sonar | +0.122 |
| Swedish (`sv`) | 0.7822 | 0.7611 | 0.7432 | fullstop-large | +0.039 |
| Tamil (`ta`) | 0.5091 | 0.4862 | 0.4771 | xlm-r truecase | +0.032 |
| Turkish (`tr`) | 0.7290 | 0.7082 | 0.5883 | xlm-r truecase | +0.141 |
| Ukrainian (`uk`) | 0.8149 | 0.7951 | 0.7241 | kredor | +0.091 |
| Urdu (`ur`) | 0.5309 | 0.5186 | 0.5042 | fullstop-large | +0.027 |
| Vietnamese (`vi`) | 0.7105 | 0.6830 | 0.5800 | fullstop-sonar | +0.131 |

</details>

<details>
<summary>ASR condition, 33 languages</summary>

| language | Dewpoint | mmBERT half alone | best other model | which | gap |
|---|---|---|---|---|---|
| Albanian (`sq`) | 0.3580 | 0.4005 | 0.3830 | xlm-r truecase | −0.025 |
| Arabic (`ar`) | 0.5309 | 0.5020 | 0.4840 | fullstop-large | +0.047 |
| Bengali (`bn`) | 0.0730 | 0.1404 | 0.1170 | fullstop-sonar | −0.044 |
| Bulgarian (`bg`) | 0.7247 | 0.6928 | 0.6776 | kredor | +0.047 |
| Catalan (`ca`) | 0.7112 | 0.6698 | 0.6025 | fullstop-sonar | +0.109 |
| Chinese (`zh`) | 0.7528 | 0.7620 | 0.7312 | xlm-r truecase | +0.022 |
| Dutch (`nl`) | 0.6765 | 0.6661 | 0.6115 | kredor | +0.065 |
| English (`en`) | 0.7837 | 0.7869 | 0.9382 | raw Whisper | −0.155 |
| Finnish (`fi`) | 0.7233 | 0.7053 | 0.6593 | kredor | +0.064 |
| French (`fr`) | 0.7702 | 0.7705 | 0.8240 | raw Whisper | −0.054 |
| German (`de`) | 0.8329 | 0.8204 | 0.8140 | fullstop-large | +0.019 |
| Greek (`el`) | 0.6437 | 0.5979 | 0.5943 | kredor | +0.049 |
| Hebrew (`he`) | 0.6332 | 0.5870 | 0.5265 | fullstop-sonar | +0.107 |
| Hindi (`hi`) | 0.6238 | 0.5805 | 0.5509 | fullstop-sonar | +0.073 |
| Hungarian (`hu`) | 0.7210 | 0.7041 | 0.6368 | fullstop-large | +0.084 |
| Indonesian (`id`) | 0.6723 | 0.6582 | 0.6249 | fullstop-large | +0.047 |
| Italian (`it`) | 0.7025 | 0.6750 | 0.7745 | raw Whisper | −0.072 |
| Kazakh (`kk`) | 0.4953 | 0.4933 | 0.3990 | raw Whisper | +0.096 |
| Latvian (`lv`) | 0.7218 | 0.7039 | 0.6108 | kredor | +0.111 |
| Malay (`ms`) | 0.6743 | 0.6516 | 0.5907 | fullstop-large | +0.084 |
| Marathi (`mr`) | 0.2584 | 0.2685 | 0.2000 | xlm-r truecase | +0.058 |
| Persian (`fa`) | 0.5209 | 0.4990 | 0.4512 | xlm-r truecase | +0.070 |
| Polish (`pl`) | 0.7608 | 0.7462 | 0.7409 | kredor | +0.020 |
| Portuguese (`pt`) | 0.7616 | 0.7535 | 0.8522 | raw Whisper | −0.091 |
| Romanian (`ro`) | 0.6462 | 0.6148 | 0.5553 | xlm-r truecase | +0.091 |
| Russian (`ru`) | 0.7627 | 0.7647 | 0.6927 | kredor | +0.070 |
| Spanish (`es`) | 0.7866 | 0.7844 | 0.8386 | raw Whisper | −0.052 |
| Swahili (`sw`) | 0.4165 | 0.3931 | 0.3596 | kredor | +0.057 |
| Swedish (`sv`) | 0.6646 | 0.6514 | 0.6435 | fullstop-large | +0.021 |
| Tamil (`ta`) | 0.4052 | 0.3825 | 0.2485 | fullstop-sonar | +0.157 |
| Turkish (`tr`) | 0.6670 | 0.6613 | 0.5448 | xlm-r truecase | +0.122 |
| Ukrainian (`uk`) | 0.7310 | 0.6955 | 0.6545 | xlm-r truecase | +0.076 |
| Vietnamese (`vi`) | 0.6895 | 0.6684 | 0.5873 | xlm-r truecase | +0.102 |

</details>

All per-language, per-class scores for every system are in
[`results/scores.json`](results/scores.json).

---

## Training

### Text: 79 languages, no TED

| source | licence | role |
|---|---|---|
| [Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) | CC-BY-SA-4.0 | volume, and most of the proper nouns truecasing needs |
| [Europarl](https://huggingface.co/datasets/Helsinki-NLP/europarl) | no known restrictions | spoken-style parliamentary prose |
| [Tatoeba](https://tatoeba.org/en/downloads) | CC-BY-2.0 FR | questions: 10–25% of its sentences are questions |
| [HPLT 2.0 cleaned](https://huggingface.co/datasets/HPLT/HPLT2.0_cleaned) | CC0-1.0 | fills six languages Wikipedia leaves thin |
| [VoxPopuli transcripts](https://github.com/facebookresearch/voxpopuli) | CC0-1.0 | verbatim spoken transcripts, 16 languages |
| [Aya dataset](https://huggingface.co/datasets/CohereLabs/aya_dataset) | Apache-2.0 | question-dense human prose for low-resource languages |

The corpus is temperature-sampled so that the largest languages don't swamp the rest.
Labels come free: strip the punctuation and case from clean prose, and what was
stripped is exactly what the model must predict. **Synthetic ASR noise** is applied to
about two thirds of training examples: filler words, dropped short words, character
substitutions, merged and split words, numbers written the way they're spoken, and
truncated utterances.

### Speech: round-trip ASR data

A slice of the training text was spoken aloud by TTS, transcribed by Whisper
large-v3-turbo, and aligned back to the original. That gives inputs with genuine
recognition errors but ground-truth labels. The corpus has 0.96M unique words across 23 languages, voiced by three permissively licensed engines. Oversampled to match the per-language dose of the original recipe, it makes up 6.8% of the final fine-tuning stage.

| engine | licence | languages (utterances kept) |
|---|---|---|
| [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) | Apache-2.0 | en (5,998), es (5,077), fr (3,996), hi (3,879), it (3,996), pt (3,996), zh (2,992) |
| [Piper voices](https://huggingface.co/rhasspy/piper-voices) | CC0 or CC-BY-4.0, per voice | bg (2,180), de (2,198), el (2,124), fa (1,057), fi (1,196), hu (1,184), kk (244), lv (1,067), nl (2,168), pl (2,189), ro (1,194), ru (1,177), sq (662), sv (2,183), uk (1,150) |
| [Chatterbox Multilingual](https://huggingface.co/ResembleAI/chatterbox) | MIT | ar (606) |

Every Piper voice's licence was read from its model card by code, and any voice whose
card said non-commercial, share-alike, "see URL" or nothing at all was refused.
The voice that produced a language's ASR *test* audio is never used for its training
audio. French is the exception, because Kokoro has only one French voice. No audio was
kept or is distributed.

### Procedure

Each member was trained in stages. Every stage starts from the weights the previous
stage ended with, and each later stage makes smaller, more targeted changes:

1. **Learn the task.** The encoder starts from its public pretrained weights, knowing
   language but nothing about punctuation, and learns to predict marks and capitals on
   a large clean-text corpus.
2. **Rebalance toward speech.** Training continues on a resample of the corpus that
   gives more weight to spoken-style sources (parliamentary transcripts, Tatoeba, Aya)
   and mixes short passages in with long ones, closer to what ASR produces.
3. **Harden against recognition errors.** A short final stage on a 55M-word mix that
   includes the round-trip ASR data above, at a low learning rate, so the model learns
   to punctuate text containing Whisper's mistakes without forgetting the rest.

| member | stage | starts from | data | steps | peak learning rate |
|---|---|---|---|---|---|
| mmBERT-base | learn the task | `jhu-clsp/mmBERT-base` | 1.02B-word corpus, long passages | 34,000 | 4e-5 |
| mmBERT-base | rebalance | previous stage | 735M-word resample | 13,000 | 1.8e-5 |
| mmBERT-base | rebalance, continued | previous stage | 735M-word resample | 3,500 | 8e-6 |
| mmBERT-base | harden | previous stage | 55M-word mix with round-trip data | 4,000 | 9e-6 |
| XLM-R-large | learn + rebalance | `FacebookAI/xlm-roberta-large` | 735M-word resample | 17,000 | 8e-6 |
| XLM-R-large | harden | previous stage | 55M-word mix with round-trip data | 4,000 | 3e-6 |

How to read the table:
- **Steps.** A step is one update of the weights, computed from a batch of 64 text
  windows of up to 512 subword tokens each, roughly 30,000 tokens. The first mmBERT
  stage, for example, saw about a billion tokens.
- **Peak learning rate.** This is the largest step size in that stage's schedule. It
  falls from stage to stage because later stages refine rather than relearn.
- **XLM-R-large has fewer stages** because it went straight to the rebalanced
  corpus. It was first trained as a controlled comparison against mmBERT-base, and
  later became the second member.

Settings shared by all stages:
- AdamW, with a learning rate that warms up and then decays along a cosine curve
- a higher learning rate for the two heads than for the encoder
- class-weighted cross-entropy on both heads, so the rare marks are not drowned out by
  the many words that take none
- label smoothing 0.02
- for XLM-R-large only, a frozen embedding matrix

Everything was trained on a single RTX 5080 with 16 GB.

### What did not work

| attempt | result |
|---|---|
| supervised contrastive loss (reported +1.9 to +3.2 F1 on this task) | +0.001 / +0.001 / −0.002; the rarest class got worse |
| joint punctuation–case decoding | cut head contradictions from 9.0% to 0.27% but lowered F1 at every setting |
| mDeBERTa-v3-base as a third encoder | lost on all three sets |
| distilling the ensemble into one mmBERT-base | in-domain validation +0.015, every held-out set flat |
| choosing case *given* the decided punctuation | lowered case F1 at every coupling strength |

The distillation result is why Dewpoint is an ensemble. Its robustness comes from the
two members disagreeing on each input. A single model trained to mimic their average
has no disagreement left to exploit. Two of these experiments were pre-registered:
their success criteria were written down before the run.

---

## Limitations

- **Two forward passes.** The ensemble has 2.8× the parameters of its mmBERT half, for +0.013 to +0.020 macro-F1. If latency matters more than that margin, use `members=["mmbert-base"]`.
- **Text only.** Raw Whisper still wins in English, Spanish, French, Italian, Portuguese under ASR conditions, most likely because it hears pauses and intonation. Adding a tagger after ASR only helps if your ASR doesn't already punctuate.
- **English is not state of the art against English-only systems.** On English tst2011, Dewpoint scores 0.800 macro-F1 (COMMA 0.715, PERIOD 0.823, QUESTION 0.860) without having seen any TED text. The strongest published English-only systems report about 0.85 overall F1 on IWSLT2011; they were trained in-domain on TED and scored on a differently preprocessed version of the set.
- **Weak languages.** Kurdish (0.11), Burmese (0.44), Tamil (0.51), Urdu (0.53) are still low in absolute terms, even where Dewpoint leads the other models. In Kurdish (0.11 against 0.28) the ensemble is below its own mmBERT half, so averaging in probability space does not help everywhere.
- **Japanese and Thai are unmeasured.** Their TED references are nearly unpunctuated, so no trustworthy number exists. The model is trained on both.
- **Very short inputs.** With only a few words there is little context, and the case head can guess a sentence boundary that the punctuation head didn't place (`hello how are you` → `Hello, How are You?`). On full passages it is rare: 0.4% of words on the official test set. Pass whole utterances, not fragments.
- **Caseless-script loanwords** are never capitalised (see truecasing above).
- **EXCLAM is weak**: 0.11 F1 on the official set and 0.07 on WIT3, and exclamation marks are rare there too.
- **Only five marks.** Colons, semicolons, dashes, quotes and parentheses are not restored, and French typographic spacing before `?` and `!` is not added.
- **Domain.** The benchmark is TED, which is prepared monologue. Conversational, overlapping, multi-speaker speech is harder, and isn't measured here.

---

## Licence and provenance

**Weights and code: MIT.** Both base encoders are MIT-licensed. Every TTS voice used for
training audio is MIT, Apache-2.0, CC0 or CC-BY-4.0.

A few things are worth knowing before commercial use:

- **Wikipedia is CC-BY-SA-4.0.** Whether share-alike terms reach model weights trained
  on such text is legally unsettled. Many encoders trained on Wikipedia, BERT among
  them, are released under permissive licences, and Dewpoint follows that practice.
- **Tatoeba is CC-BY-2.0 FR.** Sentences © Tatoeba contributors, https://tatoeba.org.
- **Some Piper voices are CC-BY-4.0**, which asks for attribution to the voice
  datasets. They are listed in [`results/rt2_voices.json`](results/rt2_voices.json).
- **Evaluation data.** IWSLT/WIT3 TED transcripts are CC-BY-NC-ND-4.0. They were used
  for measurement only and are not included here.

This is a description, not legal advice.

---

## Files

| path | contents |
|---|---|
| `dewpoint.py` | self-contained inference: `Punctuator` (torch or ONNX backend), streaming, command line |
| `onnx/` | both members as fp32 ONNX graphs |
| `serve.py` | a standard-library HTTP server |
| `handler.py` | the Hugging Face Inference Endpoints handler |
| `requirements.txt`, `requirements-onnx.txt` | dependencies for each backend |
| `ensemble_config.json` | member list, weights, the ensemble's calibrated biases |
| `mmbert-base/` | weights (safetensors), encoder config, tokenizer, its own calibration |
| `xlm-roberta-large/` | the same, for the XLM-R-large member |
| `gazetteer.json` | mixed-case surface forms, per language |
| `results/` | per-language scores for every system, and round-trip voice provenance |
| `assets/` | the images on this page |

The inference file reproduces the published official-set score exactly, per language,
against the evaluation harness it was measured with.

## Citation

```bibtex
@misc{dewpoint2026,
  title  = {Dewpoint: multilingual punctuation restoration and truecasing for speech},
  author = {valkayuh},
  year   = {2026},
  url    = {https://huggingface.co/valkayuh/dewpoint}
}
```

Please also cite the encoders Dewpoint builds on:
[mmBERT](https://huggingface.co/jhu-clsp/mmBERT-base) and
[XLM-RoBERTa](https://arxiv.org/abs/1911.02116).