Buckets:
| # Encoding | |
| **TODO -- coming soon.** Reference documentation on this page is not generated yet: the | |
| rc0 bindings do not expose all of these APIs. It comes back with them, along with the | |
| other bindings. See [`REQUIRED_FOR_V1.md`](https://github.com/huggingface/tokenizers/blob/main/REQUIRED_FOR_V1.md) for the full list. | |
| ## Encoding[[tokenizers.Encoding]] | |
| tokenizers.Encoding | |
| Attributes: `attention_mask`, `ids`, `type_ids` | |
| Not yet available in the rc0 bindings: `n_sequences`, `offsets`, `overflowing`, `sequence_ids`, `special_tokens_mask`, `tokens`, `word_ids`, `words`. | |
| The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website. | |
Xet Storage Details
- Size:
- 697 Bytes
- Xet hash:
- 365ceda190c4d3db8db701b01358b7a5734f5c5939655035e7c90a803d92ccfe
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.