Buckets:

hf-doc-build/doc-dev / tokenizers /pr_2639 /en /api /pre-tokenizers.md
|
download
raw
1.6 kB
# Pre-tokenizers
**TODO -- coming soon.** Reference documentation on this page is not generated yet: the
rc0 bindings do not expose all of these APIs. It comes back with them, along with the
other bindings. See [`REQUIRED_FOR_V1.md`](https://github.com/huggingface/tokenizers/blob/main/REQUIRED_FOR_V1.md) for the full list.
## BertPreTokenizer[[tokenizers.pre_tokenizers.BertPreTokenizer]]
tokenizers.pre_tokenizers.BertPreTokenizer
## ByteLevel[[tokenizers.pre_tokenizers.ByteLevel]]
tokenizers.pre_tokenizers.ByteLevel
## CharDelimiterSplit[[tokenizers.pre_tokenizers.CharDelimiterSplit]]
tokenizers.pre_tokenizers.CharDelimiterSplit
## Digits[[tokenizers.pre_tokenizers.Digits]]
tokenizers.pre_tokenizers.Digits
## Metaspace[[tokenizers.pre_tokenizers.Metaspace]]
tokenizers.pre_tokenizers.Metaspace
## PreTokenizer[[tokenizers.pre_tokenizers.PreTokenizer]]
tokenizers.pre_tokenizers.PreTokenizer
## Punctuation[[tokenizers.pre_tokenizers.Punctuation]]
tokenizers.pre_tokenizers.Punctuation
## Sequence[[tokenizers.pre_tokenizers.Sequence]]
tokenizers.pre_tokenizers.Sequence
## Split[[tokenizers.pre_tokenizers.Split]]
tokenizers.pre_tokenizers.Split
## UnicodeScripts[[tokenizers.pre_tokenizers.UnicodeScripts]]
tokenizers.pre_tokenizers.UnicodeScripts
## Whitespace[[tokenizers.pre_tokenizers.Whitespace]]
tokenizers.pre_tokenizers.Whitespace
## WhitespaceSplit[[tokenizers.pre_tokenizers.WhitespaceSplit]]
tokenizers.pre_tokenizers.WhitespaceSplit
The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website.

Xet Storage Details

Size:
1.6 kB
·
Xet hash:
d76c20d2a9713f153f804b686a11160628b9e3af7e6f0b225c5d8f385f8aeba3

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.