toklens/language-compressibility-bpe128k
Updated
TokLens
community
AI & ML interests
None defined yet.
Recent Activity
View all activity
toklens/fineweb_edu_tokenizers
Updated
toklens/fineweb_data_optimal_143M_tok_T8
Updated • 34
toklens/fineweb_data_optimal_143M_tok_T6
Updated • 24
toklens/fineweb_data_optimal_143M_tok_T4
Updated • 3
toklens/fineweb_data_optimal_143M_tok_T2
Updated • 4
toklens/fineweb_data_optimal_143M_tok_T1
Updated • 24
toklens/fineweb_data_optimal_143M_tok_T18
Updated • 25
toklens/fineweb_data_optimal_143M_tok_T12
Updated • 3
toklens/fineweb_edu_12gb_tokenizer_no_pretok
Updated
toklens/fineweb_edu_12gb_tokenizer_sentencepiece
Updated
toklens/fineweb_edu_12gb_tokenizer_split_on_whitespace
Updated
toklens/data_optimal_143M_tok_T8_space
Updated
toklens/data_optimal_143M_tok_T6_space
Updated
toklens/data_optimal_143M_tok_T4_space
Updated
toklens/data_optimal_143M_tok_T2_space
Updated
toklens/data_optimal_143M_tok_T18_space
Updated
toklens/data_optimal_143M_tok_T12_space
Updated
toklens/maddition_eng_Latn_lambda_voc_1_lambda_tok_1_lambda_seg_1000_2000_minimal
Updated
toklens/maddition_ita_Latn_lambda_voc_1_lambda_tok_1_lambda_seg_1000_2000_minimal
Updated
toklens/maddition_ita_Latn_lambda_voc_1_lambda_tok_1_lambda_seg_5_2000_minimal
Updated
toklens/maddition_ita_Latn_lambda_voc_1_lambda_tok_1_lambda_seg_1_2000_minimal
Updated
toklens/maddition_ita_Latn_lambda_voc_1_lambda_tok_10_000_lambda_seg_1_2000_minimal
Updated
toklens/maddition_ita_Latn_lambda_voc_1_lambda_tok_100_000_lambda_seg_1_2000_minimal
Updated
toklens/maddition_ita_Latn_lambda_voc_1_lambda_tok_1_000_000_lambda_seg_1_2000_minimal
Updated
toklens/maddition_ita_Latn_lambda_voc_1_lambda_tok_100_lambda_seg_1_2000_minimal
Updated • 1
toklens/maddition_eng_Latn_lambda_voc_1_lambda_tok_1_lambda_seg_1_2000_minimal
Updated
toklens/maddition_tur_Latn_lambda_voc_1_lambda_tok_1_lambda_seg_1_2000_minimal
Updated • 1
toklens/maddition_eng_Latn_lambda_voc_5_lambda_tok_.5_lambda_seg_1_2000_minimal
Updated
toklens/maddition_tur_Latn_lambda_voc_5_lambda_tok_.5_lambda_seg_1_2000_minimal
Updated