Buckets:
| # Pre-tokenizers | |
| **TODO -- coming soon.** Reference documentation on this page is not generated yet: the | |
| rc0 bindings do not expose all of these APIs. It comes back with them, along with the | |
| other bindings. See [`REQUIRED_FOR_V1.md`](https://github.com/huggingface/tokenizers/blob/main/REQUIRED_FOR_V1.md) for the full list. | |
| ## BertPreTokenizer[[tokenizers.pre_tokenizers.BertPreTokenizer]] | |
| tokenizers.pre_tokenizers.BertPreTokenizer | |
| ## ByteLevel[[tokenizers.pre_tokenizers.ByteLevel]] | |
| tokenizers.pre_tokenizers.ByteLevel | |
| ## CharDelimiterSplit[[tokenizers.pre_tokenizers.CharDelimiterSplit]] | |
| tokenizers.pre_tokenizers.CharDelimiterSplit | |
| ## Digits[[tokenizers.pre_tokenizers.Digits]] | |
| tokenizers.pre_tokenizers.Digits | |
| ## Metaspace[[tokenizers.pre_tokenizers.Metaspace]] | |
| tokenizers.pre_tokenizers.Metaspace | |
| ## PreTokenizer[[tokenizers.pre_tokenizers.PreTokenizer]] | |
| tokenizers.pre_tokenizers.PreTokenizer | |
| ## Punctuation[[tokenizers.pre_tokenizers.Punctuation]] | |
| tokenizers.pre_tokenizers.Punctuation | |
| ## Sequence[[tokenizers.pre_tokenizers.Sequence]] | |
| tokenizers.pre_tokenizers.Sequence | |
| ## Split[[tokenizers.pre_tokenizers.Split]] | |
| tokenizers.pre_tokenizers.Split | |
| ## UnicodeScripts[[tokenizers.pre_tokenizers.UnicodeScripts]] | |
| tokenizers.pre_tokenizers.UnicodeScripts | |
| ## Whitespace[[tokenizers.pre_tokenizers.Whitespace]] | |
| tokenizers.pre_tokenizers.Whitespace | |
| ## WhitespaceSplit[[tokenizers.pre_tokenizers.WhitespaceSplit]] | |
| tokenizers.pre_tokenizers.WhitespaceSplit | |
| The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website. | |
Xet Storage Details
- Size:
- 1.6 kB
- Xet hash:
- d76c20d2a9713f153f804b686a11160628b9e3af7e6f0b225c5d8f385f8aeba3
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.