tokenizers/pr_2449/en/api/tokenizer.md

# Tokenizer

**TODO -- coming soon.** Reference documentation on this page is not generated yet: the
rc0 bindings do not expose all of these APIs. It comes back with them, along with the
other bindings. See [`REQUIRED_FOR_V1.md`](https://github.com/huggingface/tokenizers/blob/main/REQUIRED_FOR_V1.md) for the full list.

## Tokenizer[[tokenizers.Tokenizer]]

tokenizers.Tokenizer

Attributes: `padding`, `truncation`

Not yet available in the rc0 bindings: `decoder`, `model`, `normalizer`, `post_processor`, `pre_tokenizer`.

The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website.