Hugging Face's logo

procmarco
/
ita-en-code-bpe-48k

Italian
English
tiktoken
tokenizer
bpe
italian
english
code
fineweb-2
fineweb-edu
Model card Files
xet
 Community
ita-en-code-bpe-48k

Commit History

  1. add token_bytes.pt at root

    procmarco
    Verified signature
    1fbde5e

  2. add tokenizer.pkl at root

    procmarco
    Verified signature
    08e1e2d

  3. 48k IT/EN/code BPE (40/40/20), frontier-aligned special tokens

    procmarco
    Verified signature
    b4e272e

  4. initial commit

    procmarco
    Verified signature
    9929537