RaymondLi
/

gpt-4-tokenizer

Transformers.js

Model card Files Files and versions

gpt-4-tokenizer

Commit History

Oct 11, 2023

15:58
set clean_up_tokenization_spaces to false by default

RaymondLi

Unverified signature
68ffc2f
15:25
adjust pretokenizer regex

RaymondLi

Unverified signature
a80ab01

Aug 4, 2023

10:13
Create README.md

Xenova HF Staff

c9238fc
09:43
Upload 3 files

Xenova HF Staff

9959d33
09:15
Upload 2 files

Xenova HF Staff

80bf8d0
09:14
initial commit

Xenova HF Staff

c6d34a3