Buckets:
| # Normalizers | |
| **TODO -- coming soon.** Reference documentation on this page is not generated yet: the | |
| rc0 bindings do not expose all of these APIs. It comes back with them, along with the | |
| other bindings. See [`REQUIRED_FOR_V1.md`](https://github.com/huggingface/tokenizers/blob/main/REQUIRED_FOR_V1.md) for the full list. | |
| ## ByteLevel[[tokenizers.normalizers.ByteLevel]] | |
| tokenizers.normalizers.ByteLevel | |
| ## Lowercase[[tokenizers.normalizers.Lowercase]] | |
| tokenizers.normalizers.Lowercase | |
| ## NFC[[tokenizers.normalizers.NFC]] | |
| tokenizers.normalizers.NFC | |
| ## NFD[[tokenizers.normalizers.NFD]] | |
| tokenizers.normalizers.NFD | |
| ## NFKC[[tokenizers.normalizers.NFKC]] | |
| tokenizers.normalizers.NFKC | |
| ## NFKD[[tokenizers.normalizers.NFKD]] | |
| tokenizers.normalizers.NFKD | |
| ## Nmt[[tokenizers.normalizers.Nmt]] | |
| tokenizers.normalizers.Nmt | |
| ## Normalizer[[tokenizers.normalizers.Normalizer]] | |
| tokenizers.normalizers.Normalizer | |
| ## Precompiled[[tokenizers.normalizers.Precompiled]] | |
| tokenizers.normalizers.Precompiled | |
| ## Replace[[tokenizers.normalizers.Replace]] | |
| tokenizers.normalizers.Replace | |
| ## Sequence[[tokenizers.normalizers.Sequence]] | |
| tokenizers.normalizers.Sequence | |
| ## Strip[[tokenizers.normalizers.Strip]] | |
| tokenizers.normalizers.Strip | |
| ## StripAccents[[tokenizers.normalizers.StripAccents]] | |
| tokenizers.normalizers.StripAccents | |
| ## BertNormalizer[[tokenizers.normalizers.BertNormalizer]] | |
| tokenizers.normalizers.BertNormalizer | |
| The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website. | |
Xet Storage Details
- Size:
- 1.55 kB
- Xet hash:
- 2806cd66d4345e923b38a62fb77292edc26e0d93e8bf4f9e2c2aabbf471de3ef
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.