README.md

---
license: unknown
language:
  - multilingual
tags:
  - tokenizer
  - sentencepiece
  - byte-fallback
  - multilingual
pipeline_tag: token-classification
---

# SWT — Unified SentencePiece BPE Tokenizer

> **Beta / early access.** This tokenizer is not in its final version. Request
> access via the repo's **Access** tab (approved by the Wonhui team); the
> vocabulary, normalizer and special-token layout may still change before the
> release.

Fast (`tokenizers`) port of the **SWT unified SentencePiece model** — a byte-fallback
BPE tokenizer trained over the Wonhui **SWT multilingual mixture** (464 languages,
37 scripts in the evaluation holdout).

This repository ships a **self-contained fast tokenizer** (`tokenizer.json`). It does **not** require the
`sentencepiece` library at inference time. The original `unified.model` is also
included for users who want raw SentencePiece parity.

## Usage

```python
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("wonhui-labs/swt-tokenizer", token=True)
ids = tok.encode("Morphologie et syntaxe")

```
**Note**: a user can signal the language by declaring a language ISO-693-3 tag

```python
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("wonhui-labs/swt-tokenizer", token=True)
tok.encode("<|lang:fra|> Morphologie et syntaxe")
```

## Run it on your own machine

### From the Hub (after access is approved)

```bash
pip install -r requirements.txt            # transformers, tokenizers
python - <<'PY'
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer
import sentencepiece as spm

# 1) Fast path (tokenizer.json only, no sentencepiece needed)
tok = AutoTokenizer.from_pretrained("wonhui-labs/swt-tokenizer", token=True)
ids = tok.encode("<|lang:fra|> Morphologie et syntaxe")

# 2) Raw SentencePiece path (exact spm parity)
path = hf_hub_download("wonhui-labs/swt-tokenizer", "unified.model", token=True)
sp = spm.SentencePieceProcessor(model_file=path)
assert tok.encode("Bonjour le monde") == sp.encode("Bonjour le monde")
PY
```

### Fully local (downloaded repo folder)

Download the repo (`git clone` or the web UI), then load straight from the folder —
only `tokenizer.json` and `tokenizer_config.json` are required:

```python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("./swt-tokenizer")
ids = tok.encode("Morphologie et syntaxe")
```