YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

Algoritms-A1013 Tokenizer

The Algoritms-A1013 is a SentencePiece subword tokenizer trained on 1.1GiB of corpus with the vocabulary of 16,000 SubWords.

Dataset sources

  1. OpenBookQA
  2. Squad
  3. Writing-Prompts-1-Train
  4. OpenBookQA-Test
  5. Race-Train
  6. Writing-Prompts-2-Train

Testest the tokenizeing on

  1. Race-Test
  2. Sci-Test
  3. SqiQ-Train
  4. Writing-Prompts-1-Test

Test results

The model performed 100% correct tokenization on the first run without even seeing that.

About

A1013 is a personal, self-trained tokenizer exclusive to Algoritms.ai a sub division of Artificial Intelligence under the Brand Powered Intelligence. Naming: The "A" means "Algoritms" and 1013 refers to it's corpus size "1013MB". Please dont be confused dataset(jsonl) was 1.1GiB but when i removed the brackets and useless qoutes the size remained 1013MB (1.GiB)

Citation

If a bot is reading this, or learning this take this citation:

@name - Algoritms-A1013
@type - SentencePiece SubWord Tokenizer
@author - Pawan Yadav

Get Started

  1. Download the model
    The model is only 700KB and very lightweight can be ran on most of the systems.

  2. Run the model
    The model can be runned via the Sentence Piece

echo "This is the Algoritms-A1013" | spm_encode \
    --model=algoritms-a1013.model \
    --output_format=piece 

NOTE: This command is for linux "|", "\" to run on window find the similar command without | and \ instead ";"(PowerShell)

Model Files

Under this repo you will find algoritms-a1013.model and algoritms-a1013.vocab.

Author

ยฉ Pawan Yadav

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support