LnL-AI
/

dbrx-base-tokenizer

Transformers.js

Model card Files Files and versions

dbrx-base-tokenizer / tokenizer.json

Qubitium's picture

add pad and extra_N special tokens

3a0d911 over 2 years ago

History Blame Contribute Delete

4.24 MB

This file is stored with Xet . It is too big to display, but you can still download it.

Xet Pointer Details

( Raw pointer file )

Xet hash:: 391b067f4ef562e02b040b31049639a45c9a8e4293c7dcb62ca414114ea9248b
Size of remote file:: 4.24 MB
SHA256:: a974ee932171faced419ceee9a4f8e7de782c4bbae3884a69bb5df59080e92e6

Xet efficiently stores Large Files inside Git, intelligently splitting files into unique chunks and accelerating uploads and downloads. More info.