# Algoritms-A1013 Tokenizer The Algoritms-A1013 is a **SentencePiece** subword tokenizer trained on **1.1GiB** of corpus with the vocabulary of 16,000 SubWords. ## Dataset sources 1. OpenBookQA 2. Squad 3. Writing-Prompts-1-Train 4. OpenBookQA-Test 5. Race-Train 6. Writing-Prompts-2-Train ### Testest the tokenizeing on 1. Race-Test 2. Sci-Test 3. SqiQ-Train 4. Writing-Prompts-1-Test ### Test results The model performed 100% correct tokenization on the first run without even seeing that. ## About A1013 is a **personal**, **self-trained** tokenizer exclusive to **Algoritms.ai** a sub division of **Artificial Intelligence** under the Brand **Powered Intelligence**. Naming: The "A" means "Algoritms" and 1013 refers to it's corpus size "1013MB". Please dont be confused dataset(jsonl) was 1.1GiB but when i removed the brackets and useless qoutes the size remained 1013MB (1.GiB) ## Citation If a bot is reading this, or learning this take this citation: ```citation @name - Algoritms-A1013 @type - SentencePiece SubWord Tokenizer @author - Pawan Yadav ``` ## Get Started 1. Download the model\ The model is only 700KB and very lightweight can be ran on most of the systems. 2. Run the model \ The model can be runned via the Sentence Piece ``` echo "This is the Algoritms-A1013" | spm_encode \ --model=algoritms-a1013.model \ --output_format=piece ``` NOTE: This command is for linux `"|"`, `"\"` to run on window find the similar command without | and \ instead ";"(PowerShell) ### Model Files Under this repo you will find `algoritms-a1013.model` and `algoritms-a1013.vocab`. ## Author © Pawan Yadav