File size: 4,179 Bytes
0f3b9a0 350d202 0f3b9a0 34ddf30 350d202 35f95e4 350d202 35f95e4 350d202 35f95e4 350d202 35f95e4 350d202 35f95e4 350d202 0f3b9a0 350d202 35f95e4 350d202 35f95e4 350d202 35f95e4 350d202 35f95e4 350d202 35f95e4 350d202 35f95e4 350d202 35f95e4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 | ---
language:
- fa
license: cc-by-sa-4.0
library_name: spacy
pipeline_tag: feature-extraction
tags:
- spacy
- feature-extraction
- floret
- word-embeddings
- persian
- farsi
---
# fa_floret_400k
Persian floret static vector table: 50,000 rows x 300 dimensions, floret mode, minn=maxn=5, hash_count=2, trained with floret-torch on the first 400,000 articles of the Persian Wikipedia dump. Vectors only, no pipeline components. This is the table used by the fa_*_news_md tier.
## Install
```bash
pip install https://huggingface.co/Phazel/fa_floret_400k/resolve/main/fa_floret_400k-0.1.0-py3-none-any.whl
```
## Use
```python
import spacy
nlp = spacy.load("fa_floret_400k")
print(nlp.vocab.vectors.shape)
print(nlp("میرود")[0].has_vector) # floret hashes subwords: always True
```
To train your own pipeline against it, unpack the wheel and point spaCy at the
directory:
```bash
pip download --no-deps -d . https://huggingface.co/Phazel/fa_floret_400k/resolve/main/fa_floret_400k-0.1.0-py3-none-any.whl
python -m spacy train config.cfg --paths.vectors ./fa_floret_400k
```
## The table
| Property | Value |
| --- | --- |
| Rows | 50,000 |
| Dimensions | 300 |
| Mode | floret |
| `minn` / `maxn` | 5 / 5 |
| `hash_count` | 2 |
| Corpus | first 400,000 articles of the Persian Wikipedia (`fawiki`) dump, extracted with WikiExtractor `--no-templates`, sentence-tokenized with spaCy `blank("fa")` |
| Tokens | ~163M (project estimate for 400,000 articles; not measured for this run) |
| Epochs | 5 |
| Trained with | [floret-torch](https://github.com/Fazel94/floret-torch) (GPU port of [explosion/floret](https://github.com/explosion/floret)), cbow, on a Colab T4 |
Trained with:
```
python -m floret_torch.train --input fa.txt --output fa \
--model cbow --mode floret --dim 300 --minn 5 --maxn 5 \
--neg 10 --epoch 5 --hashCount 2 --bucket 50000 \
--minCount 20 --lr 0.05 --batch 8192 --dtype fp32 --device cuda --seed 0
```
## Other packages in this family
| Pipeline | Tier | LAS | ENTS_F | Wheel |
| --- | --- | ---: | ---: | ---: |
| [`fa_dep_news_sm`](https://huggingface.co/Phazel/fa_dep_news_sm) | `sm` | 85.15 | - | 7.9 MB |
| [`fa_core_news_sm`](https://huggingface.co/Phazel/fa_core_news_sm) | `sm` | 85.15 | 71.87 | 13.5 MB |
| [`fa_ent_news_sm`](https://huggingface.co/Phazel/fa_ent_news_sm) | `sm` | - | 71.87 | 5.9 MB |
| [`fa_dep_news_md`](https://huggingface.co/Phazel/fa_dep_news_md) | `md` | 86.34 | - | 62.6 MB |
| [`fa_core_news_md`](https://huggingface.co/Phazel/fa_core_news_md) | `md` | 86.34 | 74.71 | 68.5 MB |
| [`fa_ent_news_md`](https://huggingface.co/Phazel/fa_ent_news_md) | `md` | - | 74.71 | 60.6 MB |
| [`fa_dep_news_lg`](https://huggingface.co/Phazel/fa_dep_news_lg) | `lg` | 86.60 | - | 229.3 MB |
| [`fa_core_news_lg`](https://huggingface.co/Phazel/fa_core_news_lg) | `lg` | 86.60 | 75.94 | 235.2 MB |
| [`fa_ent_news_lg`](https://huggingface.co/Phazel/fa_ent_news_lg) | `lg` | - | 75.94 | 227.3 MB |
| [`fa_core_news_trf`](https://huggingface.co/Phazel/fa_core_news_trf) | `trf` | 90.79 | 82.89 | 608.2 MB |
Tiers: `sm` hash embeddings, no vectors, `md` 50k x 300d floret vectors, `lg` 200k x 300d floret vectors, `trf` fine-tuned transformer, GPU recommended.
Standalone vector tables, usable as `--paths.vectors` for your own training:
| Vectors | Rows | Used by | Wheel |
| --- | ---: | --- | ---: |
| `fa_floret_400k` (this one) | 50,000 | `md` tier | 54.5 MB |
| [`fa_floret_full_wiki`](https://huggingface.co/Phazel/fa_floret_full_wiki) | 50,000 | no shipped pipeline | 54.9 MB |
| [`fa_floret_wiki_200k`](https://huggingface.co/Phazel/fa-floret-wiki-vectors) | 200,000 | `lg` tier | 221.3 MB |
Training scripts, configs and evaluation: <https://github.com/Fazel94/spacy-persian>.
## Sources
| Source | Author | Licence |
| --- | --- | --- |
| [fa_floret static vectors (50k rows x 300d, floret mode, first 400,000 Persian Wikipedia articles, trained with floret-torch)](https://github.com/Fazel94/spacy-persian) | Kiyarash Fazeli | CC BY-SA 4.0 |
| [Persian Wikipedia dump (fawiki), the text the vector table is trained on](https://dumps.wikimedia.org/fawiki/) | Wikipedia contributors | CC BY-SA 4.0 |
|