Van Nyaya models
Collection
Tiny Aya Global fine-tuned (LoRA SFT + DPO) for grounded bilingual answers on Indian forest and environment law. Lexsi Hackathon Track 1. • 6 items • Updated
Van Nyaya: SFT LoRA adapter for Tiny Aya Global (headline model)
Van Nyaya (Hindi: "forest justice") is a grounded bilingual (English + Hindi) assistant for Indian forest and environment law. Given a statute section, it answers only from that section, cites it, and says plainly when the section does not answer the question.
Statutes covered (332 sections): Indian Forest Act 1927, Wildlife (Protection) Act 1972, Forest (Conservation) Act 1980 / Van (Sanrakshan Evam Samvardhan) Adhiniyam 2023, Scheduled Tribes and Other Traditional Forest Dwellers (Recognition of Forest Rights) Act 2006 and Rules, Environment (Protection) Act 1986.
Results (held-out sections never seen in training, n = 164, scripts/score.py)
| Base Tiny Aya Global | Van Nyaya SFT | Change | |
|---|---|---|---|
| Correct refusal on unanswerable questions (40) | 10.0% | 100.0% | +90 pts |
| English token F1, grounded (94) | 0.294 | 0.359 | +22% |
| Hindi token F1, grounded (30) | 0.265 | 0.306 | +15% |
| False refusals on answerable questions (124) | 0 | 3 (2.4%) | small over-refusal cost |
Ablation: DPO directly on the base model (200 grounded-vs-ungrounded pairs) lifted correct refusals only to 45% with no F1 gain, so supervised fine-tuning on curated grounded data is what moves the needle.
Pipeline
- CuratorKIT: section chunking, grounded QA generation, Hindi translation, unanswerable (mismatched-section) questions, DPO pairs. Split by section, so evaluation sections are never seen in training.
- AlignTune-style LoRA SFT on 511 curated examples (English, Hindi, refusal), 132 steps, CPU-only on free GitHub Actions runners (fp32, chained 2-hour jobs).
- AuditKIT-style before/after evaluation: same prompts, same decoding, GGUF via llama.cpp.
Use
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-global")
model = PeftModel.from_pretrained(base, "AkshayWagh/van-nyaya-tiny-aya-sft-lora")
tok = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-global")
System prompt: "You are Van Nyaya, an assistant on Indian forest and environment law. Answer only from the section given, cite it, and say so if the section does not answer the question." Put the section text and the question in the user turn.
Trained from the ungated mirror 1-800-LLMs/tiny-aya-global (same weights).
Links
- Data: https://huggingface.co/datasets/AkshayWagh/van-nyaya-grounded-qa
- Eval generations and scores: https://huggingface.co/datasets/AkshayWagh/van-nyaya-eval-outputs (folder sft/)
- Code: https://github.com/akshay-ira-2026/van-nyaya
- Write-up: https://huggingface.co/spaces/AkshayWagh/van-nyaya
Limitations
Not legal advice. Answers are only as good as the section supplied; state amendments and case law are not covered. Small eval set (164); token F1 underrates correct paraphrases. The adapter refuses 3 of 124 answerable questions (2.4%) where the base never refuses: the price of going from 10% to 100% correct refusals.
Evaluation interpretation and toolkit attribution
The refusal figure is a lexical refusal-marker proxy on 40 mismatched-context questions, not human-verified legal correctness or a semantic safety guarantee. Token F1 measures reference overlap, not legal validity. CuratorKIT was run for curation; custom LoRA scripts perform training (AlignTune itself was not executed). Actual AuditKIT F1Score and Abstention offline cross-checks are available in the evaluation dataset and the reproducible script. AuditKIT native normalization differs from the headline custom scorer. See the full write-up for limitations.
- Downloads last month
- -