tokenizers/pr_2449/en/api/post-processors.md

Post-processors

TODO -- coming soon. Reference documentation on this page is not generated yet: the rc0 bindings do not expose all of these APIs. It comes back with them, along with the other bindings. See REQUIRED_FOR_V1.md for the full list.

Every post-processor is a template. BertProcessing, RobertaProcessing, ByteLevel and a Sequence of post-processors were spellings of one, so a tokenizer.json using any of them is rewritten into a TemplateProcessing when it is converted, and there is no separate class for them.

PostProcessor[[tokenizers.processors.PostProcessor]]

tokenizers.processors.PostProcessor

TemplateProcessing[[tokenizers.processors.TemplateProcessing]]

tokenizers.processors.TemplateProcessing

The Rust API Reference is available directly on the Docs.rs website.