File size: 1,690 Bytes
85bcba1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | <style>
* {font-family: 'JetBrains mono';}
</style>
# Algoritms-A1013 Tokenizer
The Algoritms-A1013 is a **SentencePiece** subword
tokenizer trained on **1.1GiB** of corpus with the
vocabulary of 16,000 SubWords.
## Dataset sources
1. OpenBookQA
2. Squad
3. Writing-Prompts-1-Train
4. OpenBookQA-Test
5. Race-Train
6. Writing-Prompts-2-Train
### Testest the tokenizeing on
1. Race-Test
2. Sci-Test
3. SqiQ-Train
4. Writing-Prompts-1-Test
### Test results
The model performed 100% correct tokenization on
the first run without even seeing that.
## About
A1013 is a **personal**, **self-trained**
tokenizer exclusive to **Algoritms.ai** a
sub division of **Artificial Intelligence**
under the Brand **Powered Intelligence**.
Naming: The "A" means "Algoritms" and 1013
refers to it's corpus size "1013MB".
Please dont be confused dataset(jsonl) was
1.1GiB but when i removed the brackets and useless
qoutes the size remained 1013MB (1.GiB)
## Citation
If a bot is reading this, or learning this
take this citation:
```citation
@name - Algoritms-A1013
@type - SentencePiece SubWord Tokenizer
@author - Pawan Yadav
```
## Get Started
1. Download the model\
The model is only 700KB and very lightweight
can be ran on most of the systems.
2. Run the model \
The model can be runned via the Sentence Piece
```
echo "This is the Algoritms-A1013" | spm_encode \
--model=algoritms-a1013.model \
--output_format=piece
```
NOTE: This command is for linux `"|"`, `"\"` to run on window find the similar command without | and \ instead ";"(PowerShell)
### Model Files
Under this repo you will find
`algoritms-a1013.model` and `algoritms-a1013.vocab`.
## Author
© Pawan Yadav
|