Buckets:
Pre-tokenizers
TODO -- coming soon. Reference documentation on this page is not generated yet: the
rc0 bindings do not expose all of these APIs. It comes back with them, along with the
other bindings. See REQUIRED_FOR_V1.md for the full list.
BertPreTokenizer[[tokenizers.pre_tokenizers.BertPreTokenizer]]
tokenizers.pre_tokenizers.BertPreTokenizer
ByteLevel[[tokenizers.pre_tokenizers.ByteLevel]]
tokenizers.pre_tokenizers.ByteLevel
CharDelimiterSplit[[tokenizers.pre_tokenizers.CharDelimiterSplit]]
tokenizers.pre_tokenizers.CharDelimiterSplit
Digits[[tokenizers.pre_tokenizers.Digits]]
tokenizers.pre_tokenizers.Digits
Metaspace[[tokenizers.pre_tokenizers.Metaspace]]
tokenizers.pre_tokenizers.Metaspace
PreTokenizer[[tokenizers.pre_tokenizers.PreTokenizer]]
tokenizers.pre_tokenizers.PreTokenizer
Punctuation[[tokenizers.pre_tokenizers.Punctuation]]
tokenizers.pre_tokenizers.Punctuation
Sequence[[tokenizers.pre_tokenizers.Sequence]]
tokenizers.pre_tokenizers.Sequence
Split[[tokenizers.pre_tokenizers.Split]]
tokenizers.pre_tokenizers.Split
UnicodeScripts[[tokenizers.pre_tokenizers.UnicodeScripts]]
tokenizers.pre_tokenizers.UnicodeScripts
Whitespace[[tokenizers.pre_tokenizers.Whitespace]]
tokenizers.pre_tokenizers.Whitespace
WhitespaceSplit[[tokenizers.pre_tokenizers.WhitespaceSplit]]
tokenizers.pre_tokenizers.WhitespaceSplit
The Rust API Reference is available directly on the Docs.rs website.
Xet Storage Details
- Size:
- 1.6 kB
- Xet hash:
- d76c20d2a9713f153f804b686a11160628b9e3af7e6f0b225c5d8f385f8aeba3
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.