tokenizers/pr_2449/en/api/normalizers.md

Normalizers

TODO -- coming soon. Reference documentation on this page is not generated yet: the rc0 bindings do not expose all of these APIs. It comes back with them, along with the other bindings. See REQUIRED_FOR_V1.md for the full list.

ByteLevel[[tokenizers.normalizers.ByteLevel]]

tokenizers.normalizers.ByteLevel

Lowercase[[tokenizers.normalizers.Lowercase]]

tokenizers.normalizers.Lowercase

NFC[[tokenizers.normalizers.NFC]]

tokenizers.normalizers.NFC

NFD[[tokenizers.normalizers.NFD]]

tokenizers.normalizers.NFD

NFKC[[tokenizers.normalizers.NFKC]]

tokenizers.normalizers.NFKC

NFKD[[tokenizers.normalizers.NFKD]]

tokenizers.normalizers.NFKD

Nmt[[tokenizers.normalizers.Nmt]]

tokenizers.normalizers.Nmt

Normalizer[[tokenizers.normalizers.Normalizer]]

tokenizers.normalizers.Normalizer

Precompiled[[tokenizers.normalizers.Precompiled]]

tokenizers.normalizers.Precompiled

Replace[[tokenizers.normalizers.Replace]]

tokenizers.normalizers.Replace

Sequence[[tokenizers.normalizers.Sequence]]

tokenizers.normalizers.Sequence

Strip[[tokenizers.normalizers.Strip]]

tokenizers.normalizers.Strip

StripAccents[[tokenizers.normalizers.StripAccents]]

tokenizers.normalizers.StripAccents

BertNormalizer[[tokenizers.normalizers.BertNormalizer]]

tokenizers.normalizers.BertNormalizer

The Rust API Reference is available directly on the Docs.rs website.