Buckets:

hf-doc-build/doc-dev / tokenizers /pr_2649 /en /api /pre-tokenizers.md
|
download
raw
1.6 kB

Pre-tokenizers

TODO -- coming soon. Reference documentation on this page is not generated yet: the rc0 bindings do not expose all of these APIs. It comes back with them, along with the other bindings. See REQUIRED_FOR_V1.md for the full list.

BertPreTokenizer[[tokenizers.pre_tokenizers.BertPreTokenizer]]

tokenizers.pre_tokenizers.BertPreTokenizer

ByteLevel[[tokenizers.pre_tokenizers.ByteLevel]]

tokenizers.pre_tokenizers.ByteLevel

CharDelimiterSplit[[tokenizers.pre_tokenizers.CharDelimiterSplit]]

tokenizers.pre_tokenizers.CharDelimiterSplit

Digits[[tokenizers.pre_tokenizers.Digits]]

tokenizers.pre_tokenizers.Digits

Metaspace[[tokenizers.pre_tokenizers.Metaspace]]

tokenizers.pre_tokenizers.Metaspace

PreTokenizer[[tokenizers.pre_tokenizers.PreTokenizer]]

tokenizers.pre_tokenizers.PreTokenizer

Punctuation[[tokenizers.pre_tokenizers.Punctuation]]

tokenizers.pre_tokenizers.Punctuation

Sequence[[tokenizers.pre_tokenizers.Sequence]]

tokenizers.pre_tokenizers.Sequence

Split[[tokenizers.pre_tokenizers.Split]]

tokenizers.pre_tokenizers.Split

UnicodeScripts[[tokenizers.pre_tokenizers.UnicodeScripts]]

tokenizers.pre_tokenizers.UnicodeScripts

Whitespace[[tokenizers.pre_tokenizers.Whitespace]]

tokenizers.pre_tokenizers.Whitespace

WhitespaceSplit[[tokenizers.pre_tokenizers.WhitespaceSplit]]

tokenizers.pre_tokenizers.WhitespaceSplit

The Rust API Reference is available directly on the Docs.rs website.

Xet Storage Details

Size:
1.6 kB
·
Xet hash:
d76c20d2a9713f153f804b686a11160628b9e3af7e6f0b225c5d8f385f8aeba3

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.