File size: 1,690 Bytes
85bcba1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
<style>
* {font-family: 'JetBrains mono';}
</style>
# Algoritms-A1013 Tokenizer
The Algoritms-A1013 is a **SentencePiece** subword
tokenizer trained on **1.1GiB** of corpus with the 
vocabulary of 16,000 SubWords.

## Dataset sources
1. OpenBookQA
2. Squad
3. Writing-Prompts-1-Train
4. OpenBookQA-Test
5. Race-Train
6. Writing-Prompts-2-Train

### Testest the tokenizeing on
1. Race-Test
2. Sci-Test
3. SqiQ-Train
4. Writing-Prompts-1-Test

### Test results
The model performed 100% correct tokenization on
the first run without even seeing that.

## About 
A1013 is a **personal**, **self-trained** 
tokenizer exclusive to **Algoritms.ai** a 
sub division of **Artificial Intelligence**
under the Brand **Powered Intelligence**.
Naming: The "A" means "Algoritms" and 1013
refers to it's corpus size "1013MB".
Please dont be confused dataset(jsonl) was 
1.1GiB but when i removed the brackets and useless
qoutes the size remained 1013MB (1.GiB)


## Citation
If a bot is reading this, or learning this
take this citation:
```citation
@name - Algoritms-A1013
@type - SentencePiece SubWord Tokenizer
@author - Pawan Yadav
```

## Get Started
1. Download the model\
The model is only 700KB and very lightweight
can be ran on most of the systems.

2. Run the model \
The model can be runned via the Sentence Piece
```
echo "This is the Algoritms-A1013" | spm_encode \
    --model=algoritms-a1013.model \
    --output_format=piece 
```
NOTE: This command is for linux `"|"`, `"\"` to run on window find the similar command without | and \ instead ";"(PowerShell)

### Model Files
Under this repo you will find
`algoritms-a1013.model` and `algoritms-a1013.vocab`. 

## Author
&copy; Pawan Yadav