fa_core_news_trf

Persian pipeline built on a fine-tuned HooshvareLab/bert-base-parsbert-uncased transformer. Components: transformer, tagger, morphologizer, trainable_lemmatizer, parser, ner. Entity labels: PER, LOC, ORG, DAT, MON, TIM, PCT. GPU recommended.

Install

This pipeline runs on a transformer, so it also needs spacy-transformers and a GPU for usable throughput.

pip install spacy-transformers
pip install https://huggingface.co/Phazel/fa_core_news_trf/resolve/main/fa_core_news_trf-1.0.0-py3-none-any.whl

Use

import spacy

nlp = spacy.load("fa_core_news_trf")
doc = nlp("شرکت ایران خودرو اعلام کرد که تولید خود را افزایش می‌دهد.")

print([(t.text, t.pos_, t.tag_, t.lemma_, t.dep_) for t in doc])
print([(e.text, e.label_) for e in doc.ents])

Accuracy

Scored with spacy benchmark accuracy on the held-out PerDT test split.

Metric	Score
Tokenization accuracy	99.96
XPOS tag accuracy	97.62
UPOS tag accuracy	97.63
Morphological features	97.82
Lemma accuracy	97.31
Unlabelled attachment (UAS)	93.87
Labelled attachment (LAS)	90.79
Sentence segmentation F	97.35
NER precision	84.06
NER recall	81.76
NER F-score	82.89

Per entity label:

Label	P	R	F
DAT	79.73	85.51	82.52
LOC	88.76	86.81	87.78
MON	100.00	80.00	88.89
ORG	77.18	79.86	78.50
PCT	50.00	25.00	33.33
PER	84.84	79.12	81.88
TIM	57.14	44.44	50.00

Throughput

Median of repeated nlp.pipe passes over the 146-document PerDT test split (23,825 tokens), timing the pipe only. Warmup pass discarded.

Device	Batch	Words/s
cpu (Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz)	32	187
gpu:0 (NVIDIA GeForce 940MX, 2048 MiB)	32	1,106
cpu (Intel(R) Xeon(R) CPU @ 2.00GHz)	32	336
gpu:0 (Tesla T4, 15360 MiB)	32	8,320

Other packages in this family

Pipeline	Tier	LAS	ENTS_F	Wheel
fa_dep_news_sm	sm	85.15	-	7.9 MB
fa_core_news_sm	sm	85.15	71.87	13.5 MB
fa_ent_news_sm	sm	-	71.87	5.9 MB
fa_dep_news_md	md	86.34	-	62.6 MB
fa_core_news_md	md	86.34	74.71	68.5 MB
fa_ent_news_md	md	-	74.71	60.6 MB
fa_dep_news_lg	lg	86.60	-	229.3 MB
fa_core_news_lg	lg	86.60	75.94	235.2 MB
fa_ent_news_lg	lg	-	75.94	227.3 MB
fa_core_news_trf (this one)	trf	90.79	82.89	608.2 MB

Tiers: sm hash embeddings, no vectors, md 50k x 300d floret vectors, lg 200k x 300d floret vectors, trf fine-tuned transformer, GPU recommended.

Standalone vector tables, usable as --paths.vectors for your own training:

Vectors	Rows	Used by	Wheel
fa_floret_400k	50,000	md tier	54.5 MB
fa_floret_full_wiki	50,000	no shipped pipeline	54.9 MB
fa_floret_wiki_200k	200,000	lg tier	221.3 MB

Training scripts, configs and evaluation: https://github.com/Fazel94/spacy-persian.

Sources

Source	Author	Licence
UD_Persian-PerDT (PerUDT v1.0)	Mohammad Sadegh Rasooli, Pegah Safari, Amirsaeid Moloodi, Alireza Nourian	CC BY-SA 4.0
UD_Persian-PerDT NER layer (not-to-release/Dadegan with NER tag/)	PerDT authors, tagged with Beheshti-NER (Taher, Hoseini, Shamsfard 2020)	CC BY-SA 4.0
spaCy lang/fa language data (stop words originally from HAZM)	Explosion and spaCy contributors	MIT
HooshvareLab/bert-base-parsbert-uncased	Hooshvare Team	no licence stated on the model card

Notes

Trained on UD_Persian-PerDT, licensed CC BY-SA 4.0; this pipeline is therefore distributed under CC BY-SA 4.0 with attribution to the treebank authors. The ner component is trained on the NER layer shipped in UD_Persian-PerDT's not-to-release/ directory, so it shares the treebank's genre, tokenization and licence. Those labels are SILVER: the treebank README states they were produced by the BERT-based Beheshti-NER tagger (Taher et al., 2020) with manual corrections to extend recall. They were transferred onto this pipeline's tokenization by difflib alignment at a 99.86% transfer rate (scripts/transfer_perdt_ner.py); spans that could not be aligned exactly were dropped rather than guessed. Labels PER, LOC, ORG and DAT have 1,300 or more training examples each; MON (205), TIM (135) and PCT (121) are thin and their scores in performance.ents_per_type should be read before relying on them. Multiword tokens (pronominal clitics, enclitic copulas) were merged with spacy convert --merge-subtokens, so a small number of XPOS tags are composite (e.g. N_IANM_PR_JOPER) and ~1.5% of lemmas contain a space. doc.noun_chunks under-fires on this pipeline: spacy/lang/fa/syntax_iterators.py matches ClearNLP labels that do not exist in Universal Dependencies, see docs/upstream/fa-noun-chunks.md. This is the trf tier: no static vectors. Contextual embeddings come from a fine-tuned HooshvareLab/bert-base-parsbert-uncased (no licence stated on the model card) via spacy-transformers, shared by every component through a TransformerListener, so one encoder forward pass serves the tagger, morphologizer, lemmatizer, parser and ner. Unlike the sm/md/lg tiers the ner is trained jointly rather than sourced, because a shared encoder cannot be fine-tuned twice and then merged. GPU is strongly recommended for both training and inference. REDISTRIBUTION WARNING: HooshvareLab/bert-base-parsbert-uncased states no licence, so this wheel embeds weights whose terms are unknown and must not be republished. Retrain against HooshvareLab/roberta-fa-zwnj-base (Apache-2.0) for a publishable artifact.