NepLRL-models

Models and code for the paper Evaluating Cross-Lingual Transfer for Named Entity Recognition in Low-Resource Languages of Nepal.

The paper studies NER transfer from Nepali to Maithili and Newari (Nepal Bhasa). mBERT and XLM-R are adapted to Nepali, Maithili, and Newari with language-adaptive pretraining (LAPT), fine-tuned for NER on the Nepali EverestNER dataset, and evaluated on the NepLRL-NER benchmark in zero-shot and few-shot settings.

Models

Path in this repo	Model	Base model	Use
trained_models/mbert-lapt	mBERT after LAPT	google-bert/bert-base-multilingual-cased	encoder for fine-tuning on any task in Nepali, Maithili, or Newari
trained_models/xlmr-lapt	XLM-R after LAPT	FacebookAI/xlm-roberta-base	same
trained_models/ner/mbert-lapt-seed5	mBERT+LAPT fine-tuned on Nepali NER	trained_models/mbert-lapt	ready-to-use NER tagger
trained_models/ner/xlmr-lapt-seed3	XLM-R+LAPT fine-tuned on Nepali NER	trained_models/xlmr-lapt	ready-to-use NER tagger (best model in the paper)

The NER models predict five entity types in BIO format: PER, LOC, ORG, DAT (date), and EVT (event), following the EverestNER guidelines.

Entity-level F1 of the two NER models (Nepali: EverestNER test set; Maithili and Newari: NepLRL-NER test split, zero-shot). The paper reports the mean over five seeds; the published seed is the one closest to that mean.

Model	Nepali	Maithili	Newari
ner/mbert-lapt-seed5	88.9	59.2	63.5
ner/xlmr-lapt-seed3	90.5	72.5	66.0

Usage

All models live in subfolders, so pass subfolder= when loading.

Tagging entities with a NER model:

from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline

repo = "SughamKharel/NepLRL-models"
sub = "trained_models/ner/xlmr-lapt-seed3"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=sub)
model = AutoModelForTokenClassification.from_pretrained(repo, subfolder=sub)

ner = pipeline("token-classification", model=model, tokenizer=tokenizer, aggregation_strategy="simple")
print(ner("<a Nepali, Maithili, or Newari sentence>"))

Loading an adapted encoder for your own fine-tuning:

from transformers import AutoTokenizer, AutoModel

repo = "SughamKharel/NepLRL-models"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="trained_models/xlmr-lapt")
model = AutoModel.from_pretrained(repo, subfolder="trained_models/xlmr-lapt")

How the models were trained

LAPT. Continued masked language modeling on Nepali, Maithili, and Newari text: 100,000 documents from the Nepali-Text-Corpus, 20,000 Maithili documents from IndicCorp v2, the NepLRL corpus, and the indigenous-eng-translation dataset (33.5M words after cleaning and deduplication; 5% of the documents held out as a test split). Each mini-batch comes from a single language, sampled with probability proportional to its word count to the power 0.3. Settings: 512-token sequences, 15% masking, effective batch size 64, 10,000 steps, learning rate 5e-5. The full configuration is in lapt_config.json next to each encoder, and the loss history in trainer_state.json.

NER fine-tuning. On EverestNER (12,463 training sentences; a random 10% of the official training set held out as a development set). Learning rate 5e-5, batch size 32, 10 epochs, first-subword labeling; the epoch with the highest development F1 was kept. The paper trains each model with five seeds; ner_results.json in each NER model folder holds that run's scores.

Reproducing the paper

Run everything from the repository root, after installing the pinned dependencies:

pip install -r requirements.txt

1. LAPT. Builds the pretraining corpus in datasets/lapt/ on the first run, then trains. This step needs a large GPU and is only required to retrain the encoders; the published ones are already in trained_models/.

python train_lapt.py --model mbert
python train_lapt.py --model xlmr
python eval_lapt.py --model mbert
python eval_lapt.py --model xlmr

2. Nepali NER fine-tuning. Four models, five seeds each.

for m in mbert mbert-lapt xlmr xlmr-lapt; do
  for s in 1 2 3 4 5; do
    python train_ner.py --model $m --seed $s
  done
done

3. Zero-shot evaluation of the 20 Nepali NER models on the NepLRL-NER test splits.

python eval_zeroshot.py

4. Few-shot runs. Each call covers all values of k and both target languages for one setting (transfer: continue from the Nepali NER model; target: train on the target sentences only), one model, and one seed.

for st in transfer target; do
  for m in mbert mbert-lapt xlmr xlmr-lapt; do
    for s in 1 2 3 4 5; do
      python train_fewshot.py --setting $st --model $m --seed $s
    done
  done
done

Each script writes its results to outputs/, in the same layout as the result files included in this repository:

• outputs/lapt_eval/: perplexity and masked-token accuracy before and after LAPT (Table 3 in the paper)
• outputs/ner_eval/nepali/: EverestNER results per model and seed (Table 4, Nepali column)
• outputs/ner_eval/zeroshot/: zero-shot results per model and seed (Table 4)
• outputs/ner_eval/fewshot/: few-shot results per setting, language, model, k, and seed, with the sampled sentence ids (Figure 2 and the appendix table)

The datasets are loaded at pinned revisions (data/dataset.py), and EverestNER is downloaded from its authors' Kaggle release with checksums, so reruns use exactly the data of the paper. NepLRL-NER is split into a 250-sentence few-shot split and a test split with a fixed seed in data/ner_dataset.py.

Data

• NepLRL: unlabeled Maithili and Newari corpus (CC BY 4.0)
• NepLRL-NER: manually annotated NER benchmark for Maithili and Newari (CC BY 4.0)
• EverestNER (Niraula and Chapagain, 2022): Nepali NER dataset used for source-language fine-tuning; the scripts download it from the authors' Kaggle release (version 1)

License

This repository is released under the Apache License 2.0. See LICENSE.

Citation

The paper describing these models is under review. The citation will be added here once it is published.