Algoritms.AI by Powered Intelligence
Published the A1013 Model and Vocabulary under the CC BY SA 4.0
85bcba1 verified |
Download README.md from Algoritms-AI/A1013-Tokenizer: direct link, hf CLI and curl.
- Browser
- Download file 1.69 kB
-
https://huggingface.co/Algoritms-AI/A1013-Tokenizer/resolve/main/README.md
- Command line
-
hf download hf://Algoritms-AI/A1013-Tokenizer/README.md
-
curl -L -o README.md https://huggingface.co/Algoritms-AI/A1013-Tokenizer/resolve/main/README.md
1.69 kB
| <style> | |
| * {font-family: 'JetBrains mono';} | |
| </style> | |
| # Algoritms-A1013 Tokenizer | |
| The Algoritms-A1013 is a **SentencePiece** subword | |
| tokenizer trained on **1.1GiB** of corpus with the | |
| vocabulary of 16,000 SubWords. | |
| ## Dataset sources | |
| 1. OpenBookQA | |
| 2. Squad | |
| 3. Writing-Prompts-1-Train | |
| 4. OpenBookQA-Test | |
| 5. Race-Train | |
| 6. Writing-Prompts-2-Train | |
| ### Testest the tokenizeing on | |
| 1. Race-Test | |
| 2. Sci-Test | |
| 3. SqiQ-Train | |
| 4. Writing-Prompts-1-Test | |
| ### Test results | |
| The model performed 100% correct tokenization on | |
| the first run without even seeing that. | |
| ## About | |
| A1013 is a **personal**, **self-trained** | |
| tokenizer exclusive to **Algoritms.ai** a | |
| sub division of **Artificial Intelligence** | |
| under the Brand **Powered Intelligence**. | |
| Naming: The "A" means "Algoritms" and 1013 | |
| refers to it's corpus size "1013MB". | |
| Please dont be confused dataset(jsonl) was | |
| 1.1GiB but when i removed the brackets and useless | |
| qoutes the size remained 1013MB (1.GiB) | |
| ## Citation | |
| If a bot is reading this, or learning this | |
| take this citation: | |
| ```citation | |
| @name - Algoritms-A1013 | |
| @type - SentencePiece SubWord Tokenizer | |
| @author - Pawan Yadav | |
| ``` | |
| ## Get Started | |
| 1. Download the model\ | |
| The model is only 700KB and very lightweight | |
| can be ran on most of the systems. | |
| 2. Run the model \ | |
| The model can be runned via the Sentence Piece | |
| ``` | |
| echo "This is the Algoritms-A1013" | spm_encode \ | |
| --model=algoritms-a1013.model \ | |
| --output_format=piece | |
| ``` | |
| NOTE: This command is for linux `"|"`, `"\"` to run on window find the similar command without | and \ instead ";"(PowerShell) | |
| ### Model Files | |
| Under this repo you will find | |
| `algoritms-a1013.model` and `algoritms-a1013.vocab`. | |
| ## Author | |
| © Pawan Yadav | |