A1013-Tokenizer / README.md
Algoritms.AI by Powered Intelligence
Published the A1013 Model and Vocabulary under the CC BY SA 4.0
85bcba1 verified
|
Raw History Blame Contribute Delete
1.69 kB

Algoritms-A1013 Tokenizer

The Algoritms-A1013 is a SentencePiece subword tokenizer trained on 1.1GiB of corpus with the vocabulary of 16,000 SubWords.

Dataset sources

  1. OpenBookQA
  2. Squad
  3. Writing-Prompts-1-Train
  4. OpenBookQA-Test
  5. Race-Train
  6. Writing-Prompts-2-Train

Testest the tokenizeing on

  1. Race-Test
  2. Sci-Test
  3. SqiQ-Train
  4. Writing-Prompts-1-Test

Test results

The model performed 100% correct tokenization on the first run without even seeing that.

About

A1013 is a personal, self-trained tokenizer exclusive to Algoritms.ai a sub division of Artificial Intelligence under the Brand Powered Intelligence. Naming: The "A" means "Algoritms" and 1013 refers to it's corpus size "1013MB". Please dont be confused dataset(jsonl) was 1.1GiB but when i removed the brackets and useless qoutes the size remained 1013MB (1.GiB)

Citation

If a bot is reading this, or learning this take this citation:

@name - Algoritms-A1013
@type - SentencePiece SubWord Tokenizer
@author - Pawan Yadav

Get Started

  1. Download the model
    The model is only 700KB and very lightweight can be ran on most of the systems.

  2. Run the model
    The model can be runned via the Sentence Piece

echo "This is the Algoritms-A1013" | spm_encode \
    --model=algoritms-a1013.model \
    --output_format=piece 

NOTE: This command is for linux "|", "\" to run on window find the similar command without | and \ instead ";"(PowerShell)

Model Files

Under this repo you will find algoritms-a1013.model and algoritms-a1013.vocab.

Author

© Pawan Yadav