Premiere publication

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,90 @@
+---
+language:
+- mg
+license: cc-by-sa-4.0
+library_name: transformers
+pipeline_tag: token-classification
+base_model: Davlan/xlm-roberta-base-finetuned-malagasy
+datasets:
+- badrex/malagasy-speech-full
+tags:
+- punctuation-restoration
+- truecasing
+- malagasy
+- asr-postprocessing
+- xlm-roberta
+---
+
+<p align="center"><a href="#results"><img src="https://img.shields.io/badge/formatted%20WER%20on%20ASR%20output-18.5%20%25%20vs%2021.7%20%25-2F7D55?style=flat-square&labelColor=1E2A5E" alt="formatted WER on ASR output: 18.5 % against 21.7 %"></a> <a href="#results"><img src="https://img.shields.io/badge/period%20F1-79-2F7D55?style=flat-square&labelColor=1E2A5E" alt="period F1: 79"></a> <a href="#what-it-does"><img src="https://img.shields.io/badge/words-never%20changed-E9C46A?style=flat-square&labelColor=1E2A5E" alt="words: never changed"></a> <a href="#license"><img src="https://img.shields.io/badge/license-CC--BY--SA--4.0-8A93A6?style=flat-square&labelColor=1E2A5E" alt="license: CC-BY-SA-4.0"></a> <a href="https://huggingface.co/spaces/Flo976/malagasy-asr-demo"><img src="https://img.shields.io/badge/demo-try%20it%20in%20the%20browser-E9C46A?style=flat-square&labelColor=1E2A5E" alt="demo: try it in the browser"></a></p>
+
+# Malagasy casing and punctuation restoration for speech recognition
+
+Speech recognition models such as [Flo976/w2v-bert-malagasy](https://huggingface.co/Flo976/w2v-bert-malagasy) output lowercase text without punctuation. This model puts back **capital letters, commas, periods, question and exclamation marks and colons, without ever changing a word**, and restores accented letters that ASR normalization strips (`tan na` → `tanàna`).
+
+```
+in:  vehivavy fohy volo iray ary miloko mainty manamboatra volo manoloana ny fitaratra
+out: Vehivavy fohy volo iray ary miloko mainty, manamboatra volo manoloana ny fitaratra.
+```
+
+**Résumé en français.** Remet les majuscules et la ponctuation dans la sortie d'un modèle de reconnaissance vocale malgache, sans jamais changer les mots. Sur la vraie sortie de l'ASR, le texte final mis en forme compte 18,5 % d'erreurs contre 21,7 % pour une mise en forme minimale (majuscule initiale et point final). Les points sont bien placés (F1 79), les virgules une fois sur deux environ (F1 52).
+
+## What it does
+
+For every word it predicts a case (lower, Capitalized, UPPER, mixed) and the mark that follows it: none, `,` `.` `?` `!` `:` or `;`. Since it classifies words instead of generating text, it cannot invent, drop or rewrite words: every call asserts that `normalize(output) == normalize(input)`, where `normalize` is the lowercase, punctuation-free form used to score the ASR. This assertion held on every test output and in a fuzz test.
+
+## Results
+
+Test split of [badrex/malagasy-speech-full](https://huggingface.co/datasets/badrex/malagasy-speech-full), 3,101 clips, speakers never seen in training. *Formatted WER* compares the final text to the original transcript **with** its casing and punctuation, without any normalization: it measures what a reader actually sees.
+
+| Input | System | Formatted WER | F1 macro `, . ? ! :` | F1 comma | F1 period | Case accuracy |
+| --- | --- | --- | --- | --- | --- | --- |
+| real ASR output (10.6 % WER) | capitalize first word + final period | 21.73 % | 13.1 | 0 | 65.6 | 96.0 % |
+| real ASR output (10.6 % WER) | **this model** | **18.47 %** | **45.2** | 51.9 | 79.0 | 97.1 % |
+| reference words | capitalize first word + final period | 13.09 % | 13.3 | 0 | 66.3 | 96.1 % |
+| reference words | **this model** | **9.18 %** | **51.2** | 52.7 | 79.8 | 97.3 % |
+
+On real ASR output the gain over the trivial baseline is **−3.26 points** of formatted WER (paired bootstrap, 95 % CI [−3.51, −3.02]). With perfect labels, the floor would be 2.4 %: quotes, brackets and some accents are out of scope.
+
+## Usage
+
+```python
+import os, sys
+from huggingface_hub import snapshot_download
+
+path = snapshot_download("Flo976/malagasy-punctuation")
+sys.path.insert(0, path)
+from punctuate import Restorer
+
+restorer = Restorer(path)  # CPU by default; Restorer(path, device="cuda") on GPU
+print(restorer.restore("tena mamy tokoa ity voankazo ity ary tsy mifidy"))
+```
+
+Requirements: `torch`, `transformers`, `sentencepiece`, `numpy`. Long texts are processed in sliding windows of 150 words. On two CPU threads, a 32-word utterance takes about 80 ms.
+
+The [demo](https://huggingface.co/spaces/Flo976/malagasy-asr-demo) also capitalizes the word after `.`, `?` and `!`, as 99.6 % of sentence starts are in the validation transcripts. That rule is not included in the measurements above.
+
+## Training
+
+| | |
+| --- | --- |
+| Base model | [Davlan/xlm-roberta-base-finetuned-malagasy](https://huggingface.co/Davlan/xlm-roberta-base-finetuned-malagasy), XLM-R base adapted to Malagasy |
+| Head | one token-classification head, 28 classes (4 cases × 7 marks), read on the first sub-token of each word |
+| Data | 0.93 M words of raw train-split transcripts of badrex/malagasy-speech-full + 1.5 M words of filtered Malagasy Wikipedia sentences |
+| Robustness | half of the training inputs receive synthetic CTC-style errors (substituted letters, merged or split words) |
+| Accent table | built from the training text, restores letters that normalization strips; same no-word-change guarantee |
+| Selection | on the validation split only; the test was used once |
+| Hardware | one RTX 5070 Ti, 13 minutes |
+
+## Limitations
+
+- **Commas remain hard** (F1 about 52): transcribers place them inconsistently.
+- **Question and exclamation marks are rare** in the test (20 and 35 cases): their scores are very uncertain. Semicolons are never predicted.
+- **Domain**: trained on read image descriptions and Wikipedia. Dialogue and questions, as in a voice assistant, are less well punctuated.
+- **Size**: 1.1 GB in fp32. Dynamic int8 quantization breaks it (comma F1 from 54 to 19): do not quantize.
+- Pauses in the audio are not used. Adding them helped a separate classifier, but they are not integrated in this model.
+
+## License
+
+**CC-BY-SA-4.0**, because the model is trained on Malagasy Wikipedia (CC BY-SA). The base model, Davlan/xlm-roberta-base-finetuned-malagasy, declares no license on its card; XLM-R itself is MIT. The speech dataset, badrex/malagasy-speech-full, declares no license either. Please credit Davlan (David Adelani) for the base model.
+
+Made for [Milo Voice](https://github.com/Flo976/milo) by [Flo976](https://huggingface.co/Flo976) — Florent Didelot / [Sooatek](https://sooatek.com).

common.py

@@ -0,0 +1,170 @@
+"""Briques communes du chantier ponctuation/majuscules.
+
+Schema d'etiquettes, par mot NORMALISE (normalize() d'evaluate_asr.py) :
+  - casse  : L (minuscule), C (Capitale initiale), U (TOUT EN MAJUSCULES, >= 2 lettres),
+             M (mixte, ex. "McDonald", "iPhone" : forme rendue via une table apprise)
+  - ponct. : signe qui suit le mot : O (aucun) , . ? ! : ;
+             ("..." est compte comme "." ; guillemets, parentheses, tirets
+             d'incise ne sont pas predits : voir README).
+
+Alignement texte brut -> mots normalises : normalize() agit caractere par
+caractere a l'interieur de chaque jeton separe par des blancs (minuscules,
+puis tout caractere hors [a-z0-9'-] devient un blanc, puis les jetons "-" et
+"'" isoles sont supprimes). On rejoue exactement cette operation en gardant la
+position de chaque caractere dans le texte brut : chaque mot normalise est
+alors une plage du texte brut, d'ou on lit la casse ; le "trou" entre deux mots
+(blancs, ponctuation, lettres accentuees supprimees) donne la ponctuation.
+Si la reconstruction ne redonne pas normalize(texte).split(), l'exemple est
+ignore et compte.
+"""
+import os
+import re
+import sys
+import unicodedata
+
+try:
+    from evaluate_asr import APOS, DASH, normalize  # noqa: E402
+except ImportError:  # hors du depot (Space, paquet du modele) : copie conforme
+    APOS = dict.fromkeys(map(ord, "’ʼʻ`´"), "'")
+    DASH = dict.fromkeys(map(ord, "‐‑‒–—―"), "-")
+    _KEEP_RE = re.compile(r"[^a-z0-9'\- ]+")
+
+    def normalize(text):
+        if not text:
+            return ""
+        text = unicodedata.normalize("NFC", text).translate(APOS).translate(DASH).lower()
+        text = re.sub(r"\s+", " ", _KEEP_RE.sub(" ", text)).strip()
+        return " ".join(w for w in text.split() if w not in {"-", "'"})
+
+CASES = ["L", "C", "U", "M"]
+PUNCTS = ["O", ",", ".", "?", "!", ":", ";"]
+CASE2ID = {c: i for i, c in enumerate(CASES)}
+P2ID = {p: i for i, p in enumerate(PUNCTS)}
+ALLOWED = set("abcdefghijklmnopqrstuvwxyz0123456789'- ")
+# Priorite quand un trou contient plusieurs signes (ex. '".' ou ').').
+PRIORITY = ["?", "!", ".", ":", ";", ","]
+
+
+def gap_punct(gap):
+    for p in PRIORITY:
+        if p in gap:
+            return p
+    return "O"
+
+
+def case_of(s):
+    letters = [c for c in s if c.isalpha()]
+    if not letters:
+        return "L"
+    if all(c.islower() for c in letters):
+        return "L"
+    if letters[0].isupper() and all(c.islower() for c in letters[1:]):
+        return "C"
+    if len(letters) >= 2 and all(c.isupper() for c in letters):
+        return "U"
+    if len(letters) == 1:
+        return "C"
+    return "M"
+
+
+def segments(raw):
+    """Rend [(debut, fin)] des mots normalises dans le texte brut (NFC), et le texte NFC.
+
+    Les positions renvoient au texte NFC + apostrophes/tirets unifies (meme
+    longueur que le NFC, la traduction etant caractere pour caractere)."""
+    t = unicodedata.normalize("NFC", raw).translate(APOS).translate(DASH)
+    segs, start = [], None
+    for i, ch in enumerate(t):
+        low = ch.lower()
+        keep = len(low) == 1 and low in ALLOWED and low != " "
+        if ch.isspace():
+            keep = False
+        if keep:
+            if start is None:
+                start = i
+        else:
+            if start is not None:
+                segs.append((start, i))
+                start = None
+    if start is not None:
+        segs.append((start, len(t)))
+    segs = [(a, b) for a, b in segs if t[a:b] not in ("-", "'")]
+    return segs, t
+
+
+def label_text(raw):
+    """Texte brut -> (mots normalises, casses, ponctuations, formes brutes) ou None si desaccord."""
+    segs, t = segments(raw)
+    words = [t[a:b].lower() for a, b in segs]
+    if words != normalize(raw).split():
+        return None
+    cases, puncts, forms = [], [], []
+    for k, (a, b) in enumerate(segs):
+        nxt = segs[k + 1][0] if k + 1 < len(segs) else len(t)
+        cases.append(case_of(t[a:b]))
+        puncts.append(gap_punct(t[b:nxt]))
+        forms.append(t[a:b])
+    return words, cases, puncts, forms
+
+
+def apply_case(word, c, mixed_table=None):
+    if c == "L":
+        return word
+    if c == "U":
+        return word.upper()
+    if c == "M" and mixed_table and word in mixed_table:
+        return mixed_table[word]
+    start = 0
+    if c == "M":
+        # M hors table : le cas typique est un nom propre apres elision ou
+        # trait d'union ("an'Antananarivo", "maha-Malagasy").
+        cut = max(word.rfind("'"), word.rfind("-"))
+        if 0 <= cut < len(word) - 1:
+            start = cut + 1
+    # C : premiere lettre en majuscule.
+    for i, ch in enumerate(word):
+        if i < start:
+            continue
+        if ch.isalpha():
+            return word[:i] + ch.upper() + word[i + 1 :]
+    return word
+
+
+def render(words, cases, puncts, mixed_table=None, lex_table=None):
+    """Assemble la sortie. lex_table (optionnelle) : n-gramme normalise -> forme brute
+    minuscule accentuee (ex. ("tan","na") -> "tanàna"), appliquee avant la casse.
+    La sortie verifie toujours normalize(sortie).split() == words."""
+    out, i, n = [], 0, len(words)
+    while i < n:
+        span = 1
+        form = None
+        if lex_table:
+            for k in (3, 2, 1):
+                if i + k <= n:
+                    key = " ".join(words[i : i + k])
+                    if key in lex_table:
+                        form, span = lex_table[key], k
+                        break
+        if form is None:
+            tok = apply_case(words[i], cases[i], mixed_table)
+        else:
+            tok = apply_case(form, cases[i], None)
+        p = puncts[i + span - 1]
+        out.append(tok + ("" if p == "O" else p))
+        i += span
+    return " ".join(out)
+
+
+def check_words(inp_words, text):
+    return normalize(text).split() == inp_words
+
+
+def trivial_baseline(norm_text):
+    """Baseline triviale : majuscule en debut + point final."""
+    w = norm_text.split()
+    if not w:
+        return ""
+    return render(w, ["C"] + ["L"] * (len(w) - 1), ["O"] * (len(w) - 1) + ["."])
+
+
+WORK = os.path.dirname(os.path.abspath(__file__))

config.json

@@ -0,0 +1,87 @@
+{
+  "architectures": [
+    "XLMRobertaForTokenClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "L|O",
+    "1": "L|,",
+    "2": "L|.",
+    "3": "L|?",
+    "4": "L|!",
+    "5": "L|:",
+    "6": "L|;",
+    "7": "C|O",
+    "8": "C|,",
+    "9": "C|.",
+    "10": "C|?",
+    "11": "C|!",
+    "12": "C|:",
+    "13": "C|;",
+    "14": "U|O",
+    "15": "U|,",
+    "16": "U|.",
+    "17": "U|?",
+    "18": "U|!",
+    "19": "U|:",
+    "20": "U|;",
+    "21": "M|O",
+    "22": "M|,",
+    "23": "M|.",
+    "24": "M|?",
+    "25": "M|!",
+    "26": "M|:",
+    "27": "M|;"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "C|!": 11,
+    "C|,": 8,
+    "C|.": 9,
+    "C|:": 12,
+    "C|;": 13,
+    "C|?": 10,
+    "C|O": 7,
+    "L|!": 4,
+    "L|,": 1,
+    "L|.": 2,
+    "L|:": 5,
+    "L|;": 6,
+    "L|?": 3,
+    "L|O": 0,
+    "M|!": 25,
+    "M|,": 22,
+    "M|.": 23,
+    "M|:": 26,
+    "M|;": 27,
+    "M|?": 24,
+    "M|O": 21,
+    "U|!": 18,
+    "U|,": 15,
+    "U|.": 16,
+    "U|:": 19,
+    "U|;": 20,
+    "U|?": 17,
+    "U|O": 14
+  },
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "xlm-roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "output_past": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.57.1",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 250002
+}

labels.json

@@ -0,0 +1,22 @@
+{
+ "schema": "etiquette combinee = indice_casse * 7 + indice_ponct, lue sur le premier sous-jeton de chaque mot normalise",
+ "cases": {
+  "L": "minuscule",
+  "C": "Capitale initiale",
+  "U": "TOUT EN MAJUSCULES",
+  "M": "mixte (forme lue dans tables.json['mixed'], sinon capitale apres la derniere apostrophe/trait d'union)"
+ },
+ "puncts": [
+  "O",
+  ",",
+  ".",
+  "?",
+  "!",
+  ":",
+  ";"
+ ],
+ "normalisation_entree": "evaluate_asr.normalize (minuscules, [a-z0-9'-], apostrophes/tirets unifies)",
+ "tables": "tables.json : 'mixed' = formes a casse mixte ; 'lex' = n-gramme normalise -> forme accentuee (ex. 'tan na' -> 'tanàna'), construite sur train + Wikipedia",
+ "base": "Davlan/xlm-roberta-base-finetuned-malagasy",
+ "entrainement": "work/punct/train.py --noise 0.5 --epochs 3 (checkpoint epoque 2.5)"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:13887a16c47f554d329ccfa6e88c081db0a1139bfa2b5c7096a79d57165eb415
+size 1109922400

punctuate.py

@@ -0,0 +1,197 @@
+#!/usr/bin/env python
+"""Restauration des majuscules et de la ponctuation pour la sortie ASR malgache.
+
+    from punctuate import restore
+    restore("lanonana ara-panambadiana iray ahitana karazan'olona mandihy")
+    -> "Lanonana ara-panambadiana iray ahitana karazan'olona mandihy."
+
+Garantie : normalize(sortie).split() == normalize(entree).split() (assert).
+Les mots ne sont jamais changes ; seules la casse et la ponctuation qui suit
+chaque mot sont predites (plus, en option, la remise des lettres accentuees
+qu'efface normalize(), ex. "tan na" -> "tanàna", qui respecte la meme garantie).
+
+Modele : XLM-R base (Davlan/xlm-roberta-base-finetuned-malagasy) + une tete de
+classification de jetons combinee (4 casses x 7 ponctuations), etiquette lue
+sur le premier sous-jeton de chaque mot. Textes longs : fenetres glissantes.
+
+CLI :
+  python punctuate.py "texte en minuscules"            # restaure
+  python punctuate.py --bench 100 --threads 2          # latence CPU
+"""
+import argparse
+import json
+import os
+import sys
+import time
+
+import numpy as np
+import torch
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, HERE)
+from common import CASES, PUNCTS, normalize, render  # noqa: E402
+
+DEFAULT_MODEL = os.environ.get("MG_PUNCT_DIR", HERE)
+NP = len(PUNCTS)
+
+
+def windows(n, size, step):
+    """Fenetres [a, b) couvrant n mots ; chaque mot sera lu dans la fenetre ou il est le plus central."""
+    if n <= size:
+        return [(0, n)]
+    out, a = [], 0
+    while True:
+        b = min(a + size, n)
+        out.append((a, b))
+        if b == n:
+            return out
+        a += step
+
+
+class Restorer:
+    def __init__(self, model_dir=DEFAULT_MODEL, device="cpu", model=None, tokenizer=None, use_lex=True, use_bias=True, quantize=False,
+                 win_words=150, win_step=100, max_tokens=512):
+        from transformers import AutoModelForTokenClassification, AutoTokenizer
+
+        self.device = device
+        self.tok = tokenizer or AutoTokenizer.from_pretrained(model_dir)
+        if model is None:
+            model = AutoModelForTokenClassification.from_pretrained(model_dir)
+            model.to(device).eval()
+            if quantize and device == "cpu":
+                # int8 dynamique sur les couches lineaires (option, CPU seulement).
+                model = torch.ao.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)
+        self.model = model
+        tables = {"mixed": {}, "lex": {}}
+        tpath = os.path.join(model_dir, "tables.json") if model_dir else ""
+        if tpath and os.path.exists(tpath):
+            tables = json.load(open(tpath))
+        self.mixed = tables.get("mixed", {})
+        self.lex = tables.get("lex", {}) if use_lex else {}
+        self.win_words, self.win_step, self.max_tokens = win_words, win_step, max_tokens
+        # Biais additif par signe sur log P(ponct.), regle sur la validation (tune_bias.py).
+        self.bias = np.zeros(NP, dtype=np.float32)
+        bpath = os.path.join(model_dir, "punct_bias.json") if model_dir else ""
+        if use_bias and bpath and os.path.exists(bpath):
+            b = json.load(open(bpath))
+            self.bias = np.array([b.get(p, 0.0) for p in PUNCTS], dtype=np.float32)
+
+    def predict_words(self, batch_words, batch_size=32):
+        """Liste de listes de mots -> liste de (casses, ponctuations). Argmax des marginales."""
+        out = []
+        for w, (pc, pp, seen) in zip(batch_words, self.predict_probs(batch_words, batch_size)):
+            cases = [CASES[i] for i in pc.argmax(1)] if len(w) else []
+            puncts = [PUNCTS[i] for i in (np.log(pp + 1e-9) + self.bias).argmax(1)] if len(w) else []
+            # Mot jamais vu (tronque) : pas de prediction -> minuscule, sans signe.
+            for g in range(len(w)):
+                if not seen[g]:
+                    cases[g], puncts[g] = "L", "O"
+            out.append((cases, puncts))
+        return out
+
+    @torch.no_grad()
+    def predict_probs(self, batch_words, batch_size=32):
+        """Liste de listes de mots -> [(P(casse) [n,4], P(ponct.) [n,7], vu [n])] par exemple."""
+        jobs = []  # (index exemple, a, b)
+        for k, w in enumerate(batch_words):
+            for a, b in windows(len(w), self.win_words, self.win_step):
+                jobs.append((k, a, b))
+        probs = [np.zeros((len(w), len(CASES) * NP), dtype=np.float32) for w in batch_words]
+        best = [np.full(len(w), -1.0) for w in batch_words]
+        jobs.sort(key=lambda j: j[2] - j[1])
+        for s in range(0, len(jobs), batch_size):
+            chunk = jobs[s : s + batch_size]
+            enc = self.tok([batch_words[k][a:b] for k, a, b in chunk], is_split_into_words=True,
+                           truncation=True, max_length=self.max_tokens, padding=True, return_tensors="pt")
+            logits = self.model(input_ids=enc["input_ids"].to(self.device),
+                                attention_mask=enc["attention_mask"].to(self.device)).logits.float()
+            p = torch.softmax(logits, -1).cpu().numpy()
+            for j, (k, a, b) in enumerate(chunk):
+                prev = None
+                for t, wid in enumerate(enc.word_ids(j)):
+                    if wid is None or wid == prev:
+                        prev = wid
+                        continue
+                    prev = wid
+                    g = a + wid
+                    # centralite : distance au bord de fenetre (les bords de texte ne comptent pas)
+                    left = wid if a > 0 else 10**6
+                    right = (b - a - 1 - wid) if b < len(batch_words[k]) else 10**6
+                    c = min(left, right)
+                    if c > best[k][g]:
+                        best[k][g] = c
+                        probs[k][g] = p[j, t]
+        out = []
+        for k, w in enumerate(batch_words):
+            pr = probs[k].reshape(len(w), len(CASES), NP)
+            out.append((pr.sum(2), pr.sum(1), best[k] >= 0))
+        return out
+
+    def restore_many(self, texts, batch_size=32):
+        words = [normalize(t).split() for t in texts]
+        preds = self.predict_words(words, batch_size=batch_size)
+        outs = []
+        for w, (c, p) in zip(words, preds):
+            o = render(w, c, p, self.mixed, self.lex) if w else ""
+            assert normalize(o).split() == w, (w, o)
+            outs.append(o)
+        return outs
+
+    def restore(self, text):
+        return self.restore_many([text])[0]
+
+
+_DEFAULT = None
+
+
+def restore(text: str) -> str:
+    """Restaure majuscules et ponctuation (CPU). Le modele est charge au premier appel."""
+    global _DEFAULT
+    if _DEFAULT is None:
+        _DEFAULT = Restorer()
+    return _DEFAULT.restore(text)
+
+
+def bench(n, threads, model_dir, quantize=False):
+    """Latence CPU, une phrase a la fois, sur n hypotheses ASR de la VALIDATION."""
+    torch.set_num_threads(threads)
+    r = Restorer(model_dir, quantize=quantize)
+    path = os.path.join(HERE, "asr_valid", "valid_lm_transcripts.jsonl")
+    if os.path.exists(path):
+        texts = [json.loads(l)["hyp"] for l in open(path)]
+        import random
+        texts = random.Random(0).sample(texts, min(n, len(texts)))  # le fichier est trie par duree
+    else:
+        texts = [" ".join(json.loads(l)["words"]) for l in open(os.path.join(HERE, "data", "valid.jsonl"))][:n]
+    for t in texts[:5]:
+        r.restore(t)
+    lat = []
+    for t in texts:
+        t0 = time.perf_counter()
+        r.restore(t)
+        lat.append(1000 * (time.perf_counter() - t0))
+    lat = np.array(lat)
+    words = np.mean([len(t.split()) for t in texts])
+    res = {"n": len(texts), "threads": threads, "int8": quantize, "mots_moyens": round(float(words), 1),
+           "median_ms": round(float(np.median(lat)), 1), "p90_ms": round(float(np.percentile(lat, 90)), 1),
+           "max_ms": round(float(lat.max()), 1), "moyenne_ms": round(float(lat.mean()), 1)}
+    print(json.dumps(res, ensure_ascii=False))
+    return res
+
+
+if __name__ == "__main__":
+    ap = argparse.ArgumentParser()
+    ap.add_argument("text", nargs="?")
+    ap.add_argument("--model", default=DEFAULT_MODEL)
+    ap.add_argument("--bench", type=int, default=0)
+    ap.add_argument("--threads", type=int, default=2)
+    ap.add_argument("--out", default=None)
+    ap.add_argument("--int8", action="store_true", help="quantification dynamique int8 (CPU)")
+    a = ap.parse_args()
+    if a.bench:
+        res = bench(a.bench, a.threads, a.model, a.int8)
+        if a.out:
+            json.dump(res, open(a.out, "w"), ensure_ascii=False, indent=2)
+    else:
+        torch.set_num_threads(a.threads)
+        print(Restorer(a.model, quantize=a.int8).restore(a.text or sys.stdin.read()))

sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfc8146abe2a0488e9e2a0c56de7952f7c11ab059eca145a0a727afce0db2865
+size 5069051

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:883b037111086fd4dfebbbc9b7cee11e1517b5e0c0514879478661440f137085
+size 17082987

tokenizer_config.json

@@ -0,0 +1,55 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "250001": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "XLMRobertaTokenizer",
+  "unk_token": "<unk>"
+}

valid_metrics.json

@@ -0,0 +1,50 @@
+{
+  "step": 8070,
+  "epoch": 2.5,
+  "min": 10.9,
+  "valid": {
+    "par_signe": {
+      ",": {
+        "P": 60.66,
+        "R": 49.17,
+        "F1": 54.31,
+        "n_ref": 4444
+      },
+      ".": {
+        "P": 75.29,
+        "R": 84.74,
+        "F1": 79.73,
+        "n_ref": 5885
+      },
+      "?": {
+        "P": 52.17,
+        "R": 42.86,
+        "F1": 47.06,
+        "n_ref": 28
+      },
+      "!": {
+        "P": 76.92,
+        "R": 47.62,
+        "F1": 58.82,
+        "n_ref": 42
+      },
+      ":": {
+        "P": 34.34,
+        "R": 24.27,
+        "F1": 28.44,
+        "n_ref": 375
+      },
+      ";": {
+        "P": 0.0,
+        "R": 0.0,
+        "F1": 0.0,
+        "n_ref": 581
+      }
+    },
+    "f1_macro_5": 53.67,
+    "f1_micro_tous_signes": 66.63,
+    "casse_exactitude": 97.24,
+    "majuscule_F1": 80.08
+  },
+  "score": 63.394
+}