fa_floret_full_wiki

Persian floret static vector table: 50,000 rows x 300 dimensions, floret mode, minn=maxn=5, hash_count=2, trained with floret-torch on the full Persian Wikipedia dump. Vectors only, no pipeline components. Same table size as fa_floret_400k over a larger corpus; the fa_*_news_lg tier uses the 200,000-row fa_floret_wiki_200k instead.

Install

pip install https://huggingface.co/Phazel/fa_floret_full_wiki/resolve/main/fa_floret_full_wiki-0.1.0-py3-none-any.whl

Use

import spacy

nlp = spacy.load("fa_floret_full_wiki")
print(nlp.vocab.vectors.shape)
print(nlp("می‌رود")[0].has_vector)   # floret hashes subwords: always True

To train your own pipeline against it, unpack the wheel and point spaCy at the directory:

pip download --no-deps -d . https://huggingface.co/Phazel/fa_floret_full_wiki/resolve/main/fa_floret_full_wiki-0.1.0-py3-none-any.whl
python -m spacy train config.cfg --paths.vectors ./fa_floret_full_wiki

The table

Property	Value
Rows	50,000
Dimensions	300
Mode	floret
minn / maxn	5 / 5
hash_count	2
Corpus	full Persian Wikipedia (fawiki) dump, extracted with WikiExtractor --no-templates, sentence-tokenized with spaCy blank("fa")
Trained with	floret-torch (GPU port of explosion/floret), cbow, on a Colab T4

Other packages in this family

Pipeline	Tier	LAS	ENTS_F	Wheel
fa_dep_news_sm	sm	85.15	-	7.9 MB
fa_core_news_sm	sm	85.15	71.87	13.5 MB
fa_ent_news_sm	sm	-	71.87	5.9 MB
fa_dep_news_md	md	86.34	-	62.6 MB
fa_core_news_md	md	86.34	74.71	68.5 MB
fa_ent_news_md	md	-	74.71	60.6 MB
fa_dep_news_lg	lg	86.60	-	229.3 MB
fa_core_news_lg	lg	86.60	75.94	235.2 MB
fa_ent_news_lg	lg	-	75.94	227.3 MB
fa_core_news_trf	trf	90.79	82.89	608.2 MB

Tiers: sm hash embeddings, no vectors, md 50k x 300d floret vectors, lg 200k x 300d floret vectors, trf fine-tuned transformer, GPU recommended.

Standalone vector tables, usable as --paths.vectors for your own training:

Vectors	Rows	Used by	Wheel
fa_floret_400k	50,000	md tier	54.5 MB
fa_floret_full_wiki (this one)	50,000	no shipped pipeline	54.9 MB
fa_floret_wiki_200k	200,000	lg tier	221.3 MB

Training scripts, configs and evaluation: https://github.com/Fazel94/spacy-persian.

Sources

Source	Author	Licence
fa_floret static vectors (50k rows x 300d, floret mode, full Persian Wikipedia dump, trained with floret-torch)	Kiyarash Fazeli	CC BY-SA 4.0
Persian Wikipedia dump (fawiki), the text the vector table is trained on	Wikipedia contributors	CC BY-SA 4.0