Buckets:

|
download
raw
1.26 kB

Decoders

TODO -- coming soon. Reference documentation on this page is not generated yet: the rc0 bindings do not expose all of these APIs. It comes back with them, along with the other bindings. See REQUIRED_FOR_V1.md for the full list.

DecodeStream[[tokenizers.decoders.DecodeStream]]

tokenizers.decoders.DecodeStream

BPEDecoder[[tokenizers.decoders.BPEDecoder]]

tokenizers.decoders.BPEDecoder

ByteFallback[[tokenizers.decoders.ByteFallback]]

tokenizers.decoders.ByteFallback

ByteLevel[[tokenizers.decoders.ByteLevel]]

tokenizers.decoders.ByteLevel

CTC[[tokenizers.decoders.CTC]]

tokenizers.decoders.CTC

Fuse[[tokenizers.decoders.Fuse]]

tokenizers.decoders.Fuse

Metaspace[[tokenizers.decoders.Metaspace]]

tokenizers.decoders.Metaspace

Replace[[tokenizers.decoders.Replace]]

tokenizers.decoders.Replace

Sequence[[tokenizers.decoders.Sequence]]

tokenizers.decoders.Sequence

Strip[[tokenizers.decoders.Strip]]

tokenizers.decoders.Strip

WordPiece[[tokenizers.decoders.WordPiece]]

tokenizers.decoders.WordPiece

The Rust API Reference is available directly on the Docs.rs website.

Xet Storage Details

Size:
1.26 kB
·
Xet hash:
3cf9ac7acc52b427c55ab090d4a5dce793923627ecdb712f21b958adee924aec

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.