| --- |
| license: mit |
| language: |
| - zh |
| library_name: onnxruntime |
| pipeline_tag: token-classification |
| tags: |
| - onnx |
| - punctuation-restoration |
| - input-method |
| - traditional-chinese |
| --- |
| |
| # 0c D512 Punctuation Restoration (Bidirectional Encoder) |
|
|
| **Project website:** [0c.biggo.com](https://0c.biggo.com) |
|
|
| Restores punctuation in unpunctuated Traditional Chinese text. The model inserts punctuation while preserving |
| all source characters. It does not append punctuation at the end of the text. |
| Labels: ， 。 ？ ！ 、 ；, plus a "no insertion" class. |
|
|
| - Architecture: a bidirectional Transformer encoder with 8 layers, hidden size 512, 8 attention heads, FFN size 768. A D512 backbone pretrained from random initialization was adapted to bidirectional attention and fine-tuned for punctuation restoration. |
| - Window: 128 characters; stride: 64 |
| - Format: ONNX (opset 18), CPU inference |
| - Vocabulary: 27,857 characters (`vocab.json`) |
|
|
| ## Files |
|
|
| | File | Description | |
| |---|---| |
| | `model.onnx` | Punctuation classification model; SHA-256 `4db3ed40df777347d9c455da01285b01de192d4c210c5f7cd9f0646c727880d8` | |
| | `vocab.json` | Special tokens and character vocabulary; SHA-256 `556dffaffbf4c05211a846dcee55ef735cea68c5754f9bffcc6f9588124172b3` | |
| | `manifest.json` | Labels, threshold, window settings, and file hashes | |
|
|
| ## Evaluation |
|
|
| Export validation used 1,171 development segments that had been seen during training, |
| rather than an unseen test set. At threshold 0.63, precision is 0.897, |
| recall is 0.755, and F1 is 0.820. ONNX and PyTorch produce identical punctuation insertions for every validation segment. |
|
|
| ## Installation |
|
|
| Requires Python 3.9 or later. The examples use CPU inference. |
|
|
| ```sh |
| pip install onnxruntime numpy huggingface_hub |
| ``` |
|
|
| The first run downloads the model from Hugging Face (approximately 130 MB). Subsequent runs use the local cache. |
|
|
| ## Usage |
|
|
| Save the complete example below as `punctuate.py` and run it. Threshold 0.63 is an operating point |
| with precision around 0.9 in the reported validation. Lower thresholds insert more punctuation but also increase incorrect insertions. |
|
|
| ```python |
| import json |
| import re |
| |
| import numpy as np |
| import onnxruntime as ort |
| from huggingface_hub import snapshot_download |
| |
| path = snapshot_download("Funmula/0c-punct-d512") |
| manifest = json.load(open(f"{path}/manifest.json", encoding="utf-8")) |
| vocab = json.load(open(f"{path}/vocab.json", encoding="utf-8")) |
| CHARS = {c: i + 8 for i, c in enumerate(vocab["characters"])} # IDs 0–7 are special tokens; 4 = UNK |
| LABELS = manifest["labels"] # Class 0 = no insertion |
| THRESHOLD, WINDOW, STRIDE = manifest["threshold"], manifest["window"], manifest["stride"] |
| session = ort.InferenceSession(f"{path}/model.onnx", providers=["CPUExecutionProvider"]) |
| |
| URL = re.compile(r"(?:https?://|www\.)[^\s，。！？、；「」『』（）【】]+", re.I) |
| ENUMERATORS = set("一二三四五六七八九十壹貳參肆伍陸柒捌玖拾甲乙丙丁戊己庚辛壬癸") |
| |
| |
| def is_han(c): |
| return "㐀" <= c <= "鿿" or "\U00020000" <= c <= "\U000323af" |
| |
| |
| def allowed(text): |
| # Insert after a Han character outside URLs, followed by Han text or whitespace; |
| # the next non-whitespace character must be Han or ASCII alphanumeric. |
| protected = {i for m in URL.finditer(text) for i in range(m.start(), m.end())} |
| out = [] |
| for i, c in enumerate(text): |
| nxt = text[i + 1:i + 2] |
| visible = text[i + 1:].lstrip()[:1] |
| out.append(is_han(c) and i not in protected |
| and (not nxt or is_han(nxt) or nxt.isspace()) |
| and (not visible or is_han(visible) or visible.isascii() and visible.isalnum())) |
| return out |
| |
| |
| def window_probs(text): |
| ids = np.array([[1, 7] + [CHARS.get(c, 4) for c in text]], dtype=np.int64) # [BOS][TASK] source text |
| return session.run(["probs"], {"ids": ids})[0][2:] # Row i: label probabilities after character i |
| |
| |
| def probabilities(text): |
| # For long text, use sliding windows and keep the most central prediction at each position. |
| n = len(text) |
| starts = [0] if n <= WINDOW else list(range(0, n - WINDOW, STRIDE)) + [n - WINDOW] |
| out, best = [None] * n, [-1] * n |
| for s in starts: |
| probs = window_probs(text[s:s + WINDOW]) |
| for j in range(len(probs)): |
| margin = min(j, len(probs) - 1 - j) if n > WINDOW else 0 |
| if margin > best[s + j]: |
| best[s + j], out[s + j] = margin, probs[j] |
| return np.stack(out) |
| |
| |
| def punctuate(text): |
| # Insert punctuation while preserving every source character; leave the final boundary unchanged. |
| if not text: |
| return text |
| ok, probs = allowed(text), probabilities(text) |
| out = [] |
| for i, c in enumerate(text): |
| out.append(c) |
| if i == len(text) - 1 or not ok[i]: |
| continue |
| k = int(probs[i, 1:].argmax()) + 1 |
| if i == 0 and LABELS[k] == "、" and c in ENUMERATORS: # Skip a list comma after a single initial enumerator |
| continue |
| if probs[i, k] >= THRESHOLD: |
| out.append(LABELS[k]) |
| return "".join(out) |
| |
| |
| print(punctuate("好的我知道了明天見")) # 好的我知道了，明天見 |
| ``` |
|
|
| The input-method integration processes at most 256 characters per request. The example itself |
| has no length limit and uses sliding windows for longer text. |
|
|
| ## License and Release Scope |
|
|
| Model weights and the accompanying inference examples are released under the MIT license (see `LICENSE`). |
| This release covers this D512 checkpoint only. Training code and training materials are not included. |
|
|
