tokenizers/pr_2449/en/api/models.md

Models

TODO -- coming soon. Reference documentation on this page is not generated yet: the rc0 bindings do not expose all of these APIs. It comes back with them, along with the other bindings. See REQUIRED_FOR_V1.md for the full list.

BPE[[tokenizers.models.BPE]]

tokenizers.models.BPE

Model[[tokenizers.models.Model]]

tokenizers.models.Model

Unigram[[tokenizers.models.Unigram]]

tokenizers.models.Unigram

WordLevel[[tokenizers.models.WordLevel]]

tokenizers.models.WordLevel

WordPiece[[tokenizers.models.WordPiece]]

tokenizers.models.WordPiece

The Rust API Reference is available directly on the Docs.rs website.