tokenizers/pr_2449/en/api/tokenizer.md

Tokenizer

TODO -- coming soon. Reference documentation on this page is not generated yet: the rc0 bindings do not expose all of these APIs. It comes back with them, along with the other bindings. See REQUIRED_FOR_V1.md for the full list.

Tokenizer[[tokenizers.Tokenizer]]

tokenizers.Tokenizer

Attributes: padding, truncation

Not yet available in the rc0 bindings: decoder, model, normalizer, post_processor, pre_tokenizer.

The Rust API Reference is available directly on the Docs.rs website.