LiLT-RE (Sinhala form understanding: relation extraction)

This model links each answer in a Sinhala form to its question. It was released with the SinFUND dataset (SriDoc/datasets).

Model

• LiLT with an XLM-R base text encoder, plus a biaffine relation-extraction head (LiLTRobertaLikeForRelationExtraction).
• This head comes from the LiLT reference implementation. It is not part of the transformers library, so loading the model requires that repository's code.
• Input: the words, word boxes and entities of a form (spans labelled QUESTION or ANSWER).
• The model scores every question-answer pair within each 512-token segment.

Training

Sequential cross-lingual fine-tuning in two stages:

1. On the Italian part of XFUND (kavg/LiLT-RE-IT).
2. On the SinFUND training split (80 forms) for 10,000 steps, with a batch size of 8, a learning rate of 1e-5 and 10% warm-up.

Results

Results on the SinFUND test split (396 question-answer links). The input was ground-truth words, boxes and entities. A predicted link counts as correct only if it connects exactly the annotated question and answer. Scores are micro-averaged over all links:

Checkpoint	Precision	Recall	F1
This model (checkpoint selected during training)	0.4898	0.6641	0.5638
Final training step (no selection)	0.4890	0.6742	0.5669
For comparison: fine-tuned on SinFUND only (final step)	0.2805	0.5051	0.3607

Important: SinFUND has no separate validation split. The "validation" scores in the training log below were measured on the SinFUND test split, and the released checkpoint was chosen by the test loss.

Limitations

• The scores assume perfect entities. With entities predicted by SriDoc/LiLT-SER, or with OCR output, accuracy will be lower.
• Precision is much lower than recall, so the model predicts many incorrect links.

Citation

@article{gunathilaka2025sinfund,
  title   = {SinFUND and SinOCR: Benchmarks for Sinhala Handwritten OCR and Template-Free Form Understanding},
  author  = {Gunathilaka, Kavishka and Hewagama, Danusha and Pushpakumara, Supul and Ambegoda, Thanuja D.},
  journal = {Research Square preprint},
  year    = {2025},
  doi     = {10.21203/rs.3.rs-6976719/v1}
}

Training log

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 8
• eval_batch_size: 2
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• training_steps: 10000

Training results

The "Validation" columns were measured on the SinFUND test split (see above). From step 8,500 onwards the training was resumed, and the numbers in each row are in the order Precision, Recall, F1, Validation Loss, not in the order of the column headings.

Training Loss	Epoch	Step	F1	Validation Loss	Precision	Recall
0.061	41.67	500	0.4764	0.2580	0.4142	0.5606
0.0332	83.33	1000	0.4906	0.3439	0.4181	0.5934
0.0264	125.0	1500	0.5194	0.3892	0.4436	0.6263
0.0104	166.67	2000	0.5250	0.4165	0.4468	0.6364
0.0064	208.33	2500	0.5245	0.4479	0.4460	0.6364
0.0013	250.0	3000	0.5204	0.4655	0.4532	0.6111
0.0019	291.67	3500	0.5342	0.4859	0.4630	0.6313
0.0009	333.33	4000	0.5420	0.5162	0.4640	0.6515
0.0006	375.0	4500	0.5515	0.5724	0.4795	0.6490
0.0039	416.67	5000	0.5470	0.5687	0.4662	0.6616
0.0012	458.33	5500	0.5595	0.5582	0.4860	0.6591
0.0001	500.0	6000	0.5730	0.5709	0.4981	0.6742
0.0022	541.67	6500	0.5578	0.5795	0.4877	0.6515
0.0012	583.33	7000	0.5674	0.5710	0.4953	0.6641
0.0009	625.0	7500	0.5607	0.5994	0.4879	0.6591
0.0002	666.67	8000	0.5616	0.5865	0.4879	0.6616
0.0016	708.33	8500	0.4972	0.6717	0.5714	0.5878
0.0	750.0	9000	0.4898	0.6641	0.5638	0.6049
0.0002	791.67	9500	0.4826	0.6641	0.5590	0.6223
0.0014	833.33	10000	0.4890	0.6742	0.5669	0.6318

Framework versions

• Transformers 4.38.2
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.15.1