Tokenizer Eval

tokenizer-eval's activity

• All
• Models
• Datasets
• Spaces
• Buckets
• Papers
• Collections
• Community
• Posts
• Articles

christopher 

in tokenizer-eval/ud-treebank-tokens about 2 months ago

[bot] Conversion to Parquet

#1 opened about 2 months ago by

parquet-converter

christopher 

published a dataset about 2 months ago

tokenizer-eval/ud-treebank-tokens

Viewer • Updated Jul 4, 2023 • 1.81M • 56

christopher 

updated a dataset about 2 months ago

tokenizer-eval/tokenizer-census

Viewer • Updated Jun 2 • 448k • 19 • 1

christopher 

in tokenizer-eval/tokenizer-census 3 months ago

[bot] Conversion to Parquet

#1 opened 3 months ago by

parquet-converter

christopher 

published a dataset 3 months ago

tokenizer-eval/tokenizer-census

Viewer • Updated Jun 2 • 448k • 19 • 1

christopher 

in tokenizer-eval/universal-unigram 3 months ago

[bot] Conversion to Parquet

#1 opened 3 months ago by

parquet-converter

christopher 

updated a dataset 3 months ago

tokenizer-eval/universal-bpe

Viewer • Updated Apr 18 • 26M • 3

christopher 

published a dataset 3 months ago

tokenizer-eval/universal-bpe

Viewer • Updated Apr 18 • 26M • 3

christopher 

updated a dataset 3 months ago

tokenizer-eval/universal-unigram

Viewer • Updated Apr 18 • 17.9M • 10

christopher 

published a dataset 3 months ago

tokenizer-eval/universal-unigram

Viewer • Updated Apr 18 • 17.9M • 10

christopher 

updated a Space 3 months ago

Running

README

🚀

christopher 

published a Space 3 months ago

Running

README

🚀

christopher 

updated a dataset 3 months ago

tokenizer-eval/universal-wordpiece

Viewer • Updated Apr 18 • 11.3M • 3

christopher 

published a dataset 3 months ago

tokenizer-eval/universal-wordpiece

Viewer • Updated Apr 18 • 11.3M • 3

christopher 

updated a dataset 3 months ago

tokenizer-eval/universal-wordlevel

Viewer • Updated Apr 18 • 3.73M • 3

christopher 

published a dataset 3 months ago

tokenizer-eval/universal-wordlevel

Viewer • Updated Apr 18 • 3.73M • 3

christopher 

in tokenizer-eval/new-tokenizer-metadata 4 months ago

[bot] Conversion to Parquet

#1 opened 4 months ago by

parquet-converter

christopher 

updated 2 datasets 5 months ago

tokenizer-eval/tokenizer-data

Viewer • Updated Feb 26 • 24.8k • 4

tokenizer-eval/tokenizer-metadata

Viewer • Updated Feb 26 • 424k • 2

benjamin 

authored a paper 7 months ago

Bolmo: Byteifying the Next Generation of Language Models

Paper • 2512.15586 • Published Dec 17, 2025 • 19