| --- |
| language: |
| - fa |
| license: cc-by-sa-4.0 |
| library_name: spacy |
| pipeline_tag: feature-extraction |
| tags: |
| - spacy |
| - feature-extraction |
| - floret |
| - word-embeddings |
| - persian |
| - farsi |
| --- |
| |
| # fa_floret_400k |
|
|
| Persian floret static vector table: 50,000 rows x 300 dimensions, floret mode, minn=maxn=5, hash_count=2, trained with floret-torch on the first 400,000 articles of the Persian Wikipedia dump. Vectors only, no pipeline components. This is the table used by the fa_*_news_md tier. |
| |
| ## Install |
| |
| ```bash |
| pip install https://huggingface.co/Phazel/fa_floret_400k/resolve/main/fa_floret_400k-0.1.0-py3-none-any.whl |
| ``` |
| |
| ## Use |
| |
| ```python |
| import spacy |
| |
| nlp = spacy.load("fa_floret_400k") |
| print(nlp.vocab.vectors.shape) |
| print(nlp("میرود")[0].has_vector) # floret hashes subwords: always True |
| ``` |
| |
| To train your own pipeline against it, unpack the wheel and point spaCy at the |
| directory: |
| |
| ```bash |
| pip download --no-deps -d . https://huggingface.co/Phazel/fa_floret_400k/resolve/main/fa_floret_400k-0.1.0-py3-none-any.whl |
| python -m spacy train config.cfg --paths.vectors ./fa_floret_400k |
| ``` |
| |
| ## The table |
| |
| | Property | Value | |
| | --- | --- | |
| | Rows | 50,000 | |
| | Dimensions | 300 | |
| | Mode | floret | |
| | `minn` / `maxn` | 5 / 5 | |
| | `hash_count` | 2 | |
| | Corpus | first 400,000 articles of the Persian Wikipedia (`fawiki`) dump, extracted with WikiExtractor `--no-templates`, sentence-tokenized with spaCy `blank("fa")` | |
| | Tokens | ~163M (project estimate for 400,000 articles; not measured for this run) | |
| | Epochs | 5 | |
| | Trained with | [floret-torch](https://github.com/Fazel94/floret-torch) (GPU port of [explosion/floret](https://github.com/explosion/floret)), cbow, on a Colab T4 | |
| |
| Trained with: |
| |
| ``` |
| python -m floret_torch.train --input fa.txt --output fa \ |
| --model cbow --mode floret --dim 300 --minn 5 --maxn 5 \ |
| --neg 10 --epoch 5 --hashCount 2 --bucket 50000 \ |
| --minCount 20 --lr 0.05 --batch 8192 --dtype fp32 --device cuda --seed 0 |
| ``` |
| |
| ## Other packages in this family |
| |
| | Pipeline | Tier | LAS | ENTS_F | Wheel | |
| | --- | --- | ---: | ---: | ---: | |
| | [`fa_dep_news_sm`](https://huggingface.co/Phazel/fa_dep_news_sm) | `sm` | 85.15 | - | 7.9 MB | |
| | [`fa_core_news_sm`](https://huggingface.co/Phazel/fa_core_news_sm) | `sm` | 85.15 | 71.87 | 13.5 MB | |
| | [`fa_ent_news_sm`](https://huggingface.co/Phazel/fa_ent_news_sm) | `sm` | - | 71.87 | 5.9 MB | |
| | [`fa_dep_news_md`](https://huggingface.co/Phazel/fa_dep_news_md) | `md` | 86.34 | - | 62.6 MB | |
| | [`fa_core_news_md`](https://huggingface.co/Phazel/fa_core_news_md) | `md` | 86.34 | 74.71 | 68.5 MB | |
| | [`fa_ent_news_md`](https://huggingface.co/Phazel/fa_ent_news_md) | `md` | - | 74.71 | 60.6 MB | |
| | [`fa_dep_news_lg`](https://huggingface.co/Phazel/fa_dep_news_lg) | `lg` | 86.60 | - | 229.3 MB | |
| | [`fa_core_news_lg`](https://huggingface.co/Phazel/fa_core_news_lg) | `lg` | 86.60 | 75.94 | 235.2 MB | |
| | [`fa_ent_news_lg`](https://huggingface.co/Phazel/fa_ent_news_lg) | `lg` | - | 75.94 | 227.3 MB | |
| | [`fa_core_news_trf`](https://huggingface.co/Phazel/fa_core_news_trf) | `trf` | 90.79 | 82.89 | 608.2 MB | |
| |
| Tiers: `sm` hash embeddings, no vectors, `md` 50k x 300d floret vectors, `lg` 200k x 300d floret vectors, `trf` fine-tuned transformer, GPU recommended. |
| |
| Standalone vector tables, usable as `--paths.vectors` for your own training: |
| |
| | Vectors | Rows | Used by | Wheel | |
| | --- | ---: | --- | ---: | |
| | `fa_floret_400k` (this one) | 50,000 | `md` tier | 54.5 MB | |
| | [`fa_floret_full_wiki`](https://huggingface.co/Phazel/fa_floret_full_wiki) | 50,000 | no shipped pipeline | 54.9 MB | |
| | [`fa_floret_wiki_200k`](https://huggingface.co/Phazel/fa-floret-wiki-vectors) | 200,000 | `lg` tier | 221.3 MB | |
| |
| Training scripts, configs and evaluation: <https://github.com/Fazel94/spacy-persian>. |
| |
| ## Sources |
| |
| | Source | Author | Licence | |
| | --- | --- | --- | |
| | [fa_floret static vectors (50k rows x 300d, floret mode, first 400,000 Persian Wikipedia articles, trained with floret-torch)](https://github.com/Fazel94/spacy-persian) | Kiyarash Fazeli | CC BY-SA 4.0 | |
| | [Persian Wikipedia dump (fawiki), the text the vector table is trained on](https://dumps.wikimedia.org/fawiki/) | Wikipedia contributors | CC BY-SA 4.0 | |
| |
