MinGram, CapsCode

txt = """~350M model pretrained on ~12.9B tokens of DCLM-Edu, UltraData-Math, and other assorted datasets, then further trained on an extra ~4B tokens of a higher quality filtered mix of real and synthetic data designed to bring out better benchmarks while not entirely destroying general language modelling capabilities."""
a = tok.encode(txt)
print(a)
print(tok.decode(a))
print(f"matches: {txt == tok.decode(a)}")
print(tok.decode(a, direct=True))
print([tok.decode([b], direct=True) for b in a])
[126, 51, 53, 48, 634, 700, 7808, 114, 3670, 290, 5316, 49, 50, 46, 57, 489, 3480, 259, 892, 2997, 773, 45, 254, 5701, 256, 3585, 108, 116, 1903, 1038, 45, 1196, 256, 262, 386, 2700, 7560, 13609, 1413, 256, 398, 1725, 9207, 290, 310, 2327, 5316, 52, 489, 3480, 259, 263, 1657, 1862, 9241, 6204, 259, 966, 262, 6100, 3392, 345, 2900, 261, 2634, 371, 957, 9153, 10408, 599, 303, 5995, 13196, 2421, 1275, 1435, 2071, 3326, 10976, 46]
~350M model pretrained on ~12.9B tokens of DCLM-Edu, UltraData-Math, and other assorted datasets, then further trained on an extra ~4B tokens of a higher quality filtered mix of real and synthetic data designed to bring out better benchmarks while not entirely destroying general language modelling capabilities.
matches: True
~350↑¦m¦¦model¦¦pretrained¦¦on¦¦~12.9↑¦b¦¦tokens¦¦of¦⇑¦dclm¦-↑¦edu¦,¦¦UltraData¦-↑¦math¦,¦¦and¦¦other¦¦assorted¦¦datasets¦,¦¦then¦¦further¦¦trained¦¦on¦¦an¦¦extra¦¦~4↑¦b¦¦tokens¦¦of¦¦a¦¦higher¦¦quality¦¦filtered¦¦mix¦¦of¦¦real¦¦and¦¦synthetic¦¦data¦¦designed¦¦to¦¦bring¦¦out¦¦better¦¦benchmarks¦¦while¦¦not¦¦entirely¦¦destroying¦¦general¦¦language¦¦modelling¦¦capabilities¦.
['~', '3', '5', '0', '↑¦m¦', '¦model¦', '¦pret', 'r', 'ained¦', '¦on¦', '¦~', '1', '2', '.', '9', '↑¦b¦', '¦tokens¦', '¦of¦', '⇑¦d', 'cl', 'm¦', '-', '↑', '¦edu¦', ',¦', '¦U', 'l', 't', 'ra', 'Data¦', '-', '↑¦math¦', ',¦', '¦and¦', '¦other¦', '¦ass', 'orted¦', '¦datas', 'ets¦', ',¦', '¦then¦', '¦further¦', '¦trained¦', '¦on¦', '¦an¦', '¦extra¦', '¦~', '4', '↑¦b¦', '¦tokens¦', '¦of¦', '¦a¦', '¦higher¦', '¦quality¦', '¦filtered¦', '¦mix¦', '¦of¦', '¦real¦', '¦and¦', '¦synth', 'etic¦', '¦data¦', '¦designed¦', '¦to¦', '¦bring¦', '¦out¦', '¦better¦', '¦bench', 'marks¦', '¦while¦', '¦not¦', '¦entirely¦', '¦destro', 'ying¦', '¦general¦', '¦language¦', '¦mod', 'elling¦', '¦capabilities¦', '.']
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train glossia/Rambley-Tokenizer-v2next-16k