tokenizer-128k / tokenizer.model

Upload 128k vocab SentencePiece BPE tokenizer (trained on 175GB)

43f50d5 verified 8 months ago

2.35 MB

This file is stored with Xet . It is too big to display, but you can still download it.

Xet efficiently stores Large Files inside Git, intelligently splitting files into unique chunks and accelerating uploads and downloads. More info.