Buckets:

|
download
raw
911 Bytes

Trainers

TODO -- coming soon. Reference documentation on this page is not generated yet: the rc0 bindings do not expose all of these APIs. It comes back with them, along with the other bindings. See REQUIRED_FOR_V1.md for the full list.

BpeTrainer[[tokenizers.trainers.BpeTrainer]]

tokenizers.trainers.BpeTrainer

ParityBpeTrainer[[tokenizers.trainers.ParityBpeTrainer]]

tokenizers.trainers.ParityBpeTrainer

UnigramTrainer[[tokenizers.trainers.UnigramTrainer]]

tokenizers.trainers.UnigramTrainer

WordLevelTrainer[[tokenizers.trainers.WordLevelTrainer]]

tokenizers.trainers.WordLevelTrainer

WordPieceTrainer[[tokenizers.trainers.WordPieceTrainer]]

tokenizers.trainers.WordPieceTrainer

The Rust API Reference is available directly on the Docs.rs website.

Xet Storage Details

Size:
911 Bytes
·
Xet hash:
074c05f7634952d3a6eb5d003461592095a5c136c5a7aa17aced9324f6956ce9

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.