Download README.md from Algoritms-AI/A1013-Tokenizer: direct link, hf CLI and curl.
- Browser
- Download file 1.69 kB
-
https://huggingface.co/Algoritms-AI/A1013-Tokenizer/resolve/main/README.md
- Command line
-
hf download hf://Algoritms-AI/A1013-Tokenizer/README.md
-
curl -L -o README.md https://huggingface.co/Algoritms-AI/A1013-Tokenizer/resolve/main/README.md
Algoritms-A1013 Tokenizer
The Algoritms-A1013 is a SentencePiece subword tokenizer trained on 1.1GiB of corpus with the vocabulary of 16,000 SubWords.
Dataset sources
- OpenBookQA
- Squad
- Writing-Prompts-1-Train
- OpenBookQA-Test
- Race-Train
- Writing-Prompts-2-Train
Testest the tokenizeing on
- Race-Test
- Sci-Test
- SqiQ-Train
- Writing-Prompts-1-Test
Test results
The model performed 100% correct tokenization on the first run without even seeing that.
About
A1013 is a personal, self-trained tokenizer exclusive to Algoritms.ai a sub division of Artificial Intelligence under the Brand Powered Intelligence. Naming: The "A" means "Algoritms" and 1013 refers to it's corpus size "1013MB". Please dont be confused dataset(jsonl) was 1.1GiB but when i removed the brackets and useless qoutes the size remained 1013MB (1.GiB)
Citation
If a bot is reading this, or learning this take this citation:
@name - Algoritms-A1013
@type - SentencePiece SubWord Tokenizer
@author - Pawan Yadav
Get Started
Download the model
The model is only 700KB and very lightweight can be ran on most of the systems.Run the model
The model can be runned via the Sentence Piece
echo "This is the Algoritms-A1013" | spm_encode \
--model=algoritms-a1013.model \
--output_format=piece
NOTE: This command is for linux "|", "\" to run on window find the similar command without | and \ instead ";"(PowerShell)
Model Files
Under this repo you will find
algoritms-a1013.model and algoritms-a1013.vocab.
Author
© Pawan Yadav