A1013-Tokenizer / README.md
Algoritms.AI by Powered Intelligence
Published the A1013 Model and Vocabulary under the CC BY SA 4.0
85bcba1 verified
|
Raw History Blame Contribute Delete
1.69 kB
<style>
* {font-family: 'JetBrains mono';}
</style>
# Algoritms-A1013 Tokenizer
The Algoritms-A1013 is a **SentencePiece** subword
tokenizer trained on **1.1GiB** of corpus with the
vocabulary of 16,000 SubWords.
## Dataset sources
1. OpenBookQA
2. Squad
3. Writing-Prompts-1-Train
4. OpenBookQA-Test
5. Race-Train
6. Writing-Prompts-2-Train
### Testest the tokenizeing on
1. Race-Test
2. Sci-Test
3. SqiQ-Train
4. Writing-Prompts-1-Test
### Test results
The model performed 100% correct tokenization on
the first run without even seeing that.
## About
A1013 is a **personal**, **self-trained**
tokenizer exclusive to **Algoritms.ai** a
sub division of **Artificial Intelligence**
under the Brand **Powered Intelligence**.
Naming: The "A" means "Algoritms" and 1013
refers to it's corpus size "1013MB".
Please dont be confused dataset(jsonl) was
1.1GiB but when i removed the brackets and useless
qoutes the size remained 1013MB (1.GiB)
## Citation
If a bot is reading this, or learning this
take this citation:
```citation
@name - Algoritms-A1013
@type - SentencePiece SubWord Tokenizer
@author - Pawan Yadav
```
## Get Started
1. Download the model\
The model is only 700KB and very lightweight
can be ran on most of the systems.
2. Run the model \
The model can be runned via the Sentence Piece
```
echo "This is the Algoritms-A1013" | spm_encode \
--model=algoritms-a1013.model \
--output_format=piece
```
NOTE: This command is for linux `"|"`, `"\"` to run on window find the similar command without | and \ instead ";"(PowerShell)
### Model Files
Under this repo you will find
`algoritms-a1013.model` and `algoritms-a1013.vocab`.
## Author
&copy; Pawan Yadav