--- language: - ne - en - hi license: cc-by-nc-4.0 library_name: onnx pipeline_tag: token-classification base_model: FacebookAI/xlm-roberta-base tags: - token-classification - named-entity-recognition - pii - pii-detection - privacy - redaction - anonymization - nepali - romanized-nepali - devanagari - english - onnx - rakshya - fiste-ai metrics: - f1 --- # Rakshya server (रक्षा) **The large model of [Rakshya](https://huggingface.co/fiste-ai/rakshya): personal data found in Nepali and English text, for servers. By [Fiste AI](https://fiste-ai.github.io/rakshya/).** It has the rules, the 21 labels and the code of the small model, with 278 million weights in place of 40 million. Use the small one in a web page or on a phone, and this one where 1.1 GB and a few tens of milliseconds a sentence are no trouble. ## Benchmark 882 real texts in seven test sets that no system had seen: news, web pages, government pages and comments. The number is the F1 of exact spans over every kind of personal data in a set. Higher is better. | System | Weights | News (ne) | Web (ne) | Government (ne) | Comments (ne, Latin) | News (en) | Web (en) | News (hi) | Mean | |---|---|---|---|---|---|---|---|---|---| | **[Rakshya 0.2](https://huggingface.co/fiste-ai/rakshya)** | 40M | **0.878** | 0.802 | 0.731 | **0.529** | 0.803 | 0.703 | **0.804** | 0.750 | | **[Rakshya 0.2 server](https://huggingface.co/fiste-ai/rakshya-server)** | 278M | 0.870 | **0.815** | **0.740** | 0.520 | **0.813** | **0.740** | 0.800 | **0.757** | | [Gopya 0.1](https://huggingface.co/fiste-ai/rakshya/tree/v0.1) | 34M | 0.855 | 0.683 | 0.636 | 0.233 | 0.729 | 0.563 | 0.725 | 0.632 | | [EU PII anonymisation](https://huggingface.co/bardsai/eu-pii-anonimization-multilang) (bards.ai) | 278M | 0.647 | 0.601 | 0.409 | 0.253 | 0.787 | 0.530 | 0.727 | 0.565 | | [XLM-R base NER HRL](https://huggingface.co/Davlan/xlm-roberta-base-ner-hrl) (Davlan) | 277M | 0.660 | 0.617 | 0.267 | 0.302 | 0.772 | 0.498 | 0.712 | 0.547 | | [XLM-R large, CoNLL-2003](https://huggingface.co/FacebookAI/xlm-roberta-large-finetuned-conll03-english) (FacebookAI) | 559M | 0.651 | 0.574 | 0.200 | 0.265 | 0.783 | 0.481 | 0.760 | 0.531 | | [EU PII anonymisation v3](https://huggingface.co/bardsai/eu-pii-anonimization-multilang-v3) (bards.ai) | 308M | 0.144 | 0.704 | 0.328 | 0.162 | 0.605 | 0.709 | 0.528 | 0.454 | | [Nym PII multilingual](https://huggingface.co/Wismut/nym-pii-multilingual) (Wismut) | 308M | 0.427 | 0.420 | 0.301 | 0.252 | 0.655 | 0.478 | 0.595 | 0.447 | None of the 25 published systems we ran scores higher than Rakshya 0.2 server on any of the seven sets. All 25 systems, the scores by kind, the false alarms and the method are in the [full benchmark](https://github.com/fiste-ai/rakshya/blob/main/BENCHMARK.md). ## Use it ```python # pip install git+https://github.com/fiste-ai/rakshya from rakshya import Rakshya pii = Rakshya(model="server") # fetches this model once, 1.1 GB pii.redact("मेरो नाम सीता थापा हो, फोन ९८४१२३४५६७।") ``` From a shell: `rakshya --server "some text"`. On a processor it reads a sentence of about 21 tokens in 40 ms on one thread and 15 ms on four (a laptop processor, Intel Core i5-13420H). With `onnxruntime-gpu` installed, `Rakshya(model="server", providers=["CUDAExecutionProvider"])` runs it on a graphics card. The repository also loads as a plain token classifier in `transformers`. Read that way the model gives its tags alone: the rules and the cut at the case marker are in the package. The scores above were measured with these weights on a graphics card. The ONNX file here holds the same weights and gives the same tag on 10,259 of 10,259 tokens of a check. ## Limits The limits of the small model hold here too: it is a helper and not a guarantee, Nepali in Latin letters is the hardest, Hindi is experimental, and identity numbers are tested mostly on made-up text. See the [card of the small model](https://huggingface.co/fiste-ai/rakshya#limits). Weights CC BY-NC 4.0, code MIT; see `LICENSE.md` and `NOTICE.md`. Part of the training data is for non-commercial use, so there is no commercial licence for this version. Questions: [contact us](https://fiste-ai.github.io/contact/).