BharatAddr / README.md
sakasurya's picture
Access note: verified organisation email
caf2a7c verified
|
Raw History Blame Contribute Delete
13.1 kB
---
license: cc-by-nc-sa-4.0
pretty_name: BharatAddr
language:
- en
- hi
- multilingual
library_name: transformers
pipeline_tag: feature-extraction
base_model:
- Qwen/Qwen3-Embedding-0.6B
datasets:
- judicialmind/BharatAddr-Gazetteer
tags:
- address-parsing
- geocoding
- entity-resolution
- india
- lgd
- pincode
- open-government-data
extra_gated_prompt: 'This model is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 licence (CC
BY-NC-SA 4.0): research, academic, personal and other non-commercial use is permitted with attribution; adaptations must
be shared under the same licence; commercial use is not permitted. The original public source data remains under the Government
Open Data License - India (GODL-India). Commercial use: contact surya@judicialmind.ai or hello@judicialmind.ai.'
extra_gated_fields:
Name: text
Email: text
Organization: text
I accept the CC BY-NC-SA licence terms (research and non-commercial use only): checkbox
extra_gated_button_content: Accept and request access
---
# BharatAddr
<!-- jm:top -->
Licence: CC BY-NC-SA 4.0 — research and non-commercial use only; derivatives must be shared under the same licence. Commercial use: contact surya@judicialmind.ai or hello@judicialmind.ai
Access is approved automatically when the email you enter matches your Hugging Face account email and belongs to the organisation you name; other requests are reviewed manually.
<!-- /jm:top -->
Messy Indian address text in, official place out: village or town, sub-district, district and state with their
Local Government Directory (LGD) codes, plus a PIN code. BharatAddr reads English, Hindi (Devanagari), other Indic
scripts, Roman transliterations, abbreviations (Vill., PO, Teh., Tq., Mandal, Dist./Distt., Zila, UP/M.P.), typos,
missing levels and wrong PINs. On 63 districts it never saw during training it picks the exact village **95.74%** of
the time, against 86.37% for a strong non-neural matcher, and gets the PIN right 98.08% of the time.
**Held-out districts (`test_district`, 63 districts never seen in training), exact match %, n = 10,000**
| System | Village / town | Sub-district | District | State | PIN |
|---|---|---|---|---|---|
| **BharatAddr** | **95.74** | **98.24** | **99.28** | **99.82** | **98.08** |
| Qwen3-Embedding-0.6B zero-shot + hierarchy reranking | 89.95 | 95.22 | 97.99 | 99.06 | 95.32 |
| Qwen3-Embedding-0.6B zero-shot, retrieval only | 67.47 | 78.11 | 85.36 | 90.97 | 76.53 |
| Non-neural baseline (normalise + fuzzy + hierarchy + PIN) | 86.37 | 92.56 | 96.06 | 97.83 | 92.73 |
Hindi-script addresses: 95.58% village accuracy (baseline 89.69%).
## Quickstart
The repository is the runtime: a small `bharataddr` package, the gazetteer, the fine-tuned encoder and a
pre-computed index of all 680,293 gazetteer records. Download it, install the requirements and run from the folder.
```bash
huggingface-cli download judicialmind/BharatAddr --local-dir BharatAddr
cd BharatAddr
pip install -r requirements.txt
```
Python:
```python
from bharataddr import BharatAddr
ba = BharatAddr.load(".") # neural resolver; uses a GPU when available, else CPU
for r in ba.resolve(["H.No 12, Vill Ram Nagar, Teh. Milak, Distt Rampur, UP 244701",
"ग्राम राम नगर, तहसील मिलक, जिला रामपुर"]):
print(r["name"], r["code"], r["subdistrict"], r["district"], r["state"], r["pin"])
fast = BharatAddr.load(".", neural=False) # baseline only: no model, CPU
print(fast.resolve(["Vill Ram Nagar, Teh. Milak, Distt Rampur, UP 244701"])[0])
```
Run it from inside the downloaded folder. Both addresses resolve to **Ram Nagar** (LGD village 117381), sub-district
Milak, district Rampur, Uttar Pradesh, PIN 244701, and the baseline returns the same village. Loading builds the
gazetteer index in about a minute on CPU; after that an address takes tens of milliseconds on CPU.
Command line (one JSON object per input line; also `--file addresses.txt` or stdin):
```bash
python -m bharataddr.resolve "H.No 12, Vill Ram Nagar, Teh. Milak, Distt Rampur, UP 244701"
python -m bharataddr.resolve --no-neural "ग्राम राम नगर, तहसील मिलक, जिला रामपुर"
```
Input: free-text address strings (any order of components, any of the scripts above). Output, per address:
| Field | Meaning |
|---|---|
| `input` | the address as given (CLI only) |
| `eid` | gazetteer id: `v<LGD village code>` or `u<LGD local-body code>` (towns) |
| `kind` | `village` or `town` |
| `code`, `name` | LGD code and official English name of the village / town |
| `sd_code`, `subdistrict` | LGD sub-district code and name (empty for towns without one) |
| `d_code`, `district` | LGD district code and name |
| `s_code`, `state` | LGD state / UT code and name |
| `pin` | PIN code: the PIN written in the address when it belongs to the chosen place, else the place's LGD PIN |
| `score` | encoder cosine similarity of the chosen record (neural) or hierarchy score (baseline) |
When the text has no usable village evidence the baseline returns `eid`/`code` as null and still fills the best
consistent district, state and sub-district.
## How it works
1. **Gazetteer** (`gazetteer/`, also published as
[BharatAddr-Gazetteer](https://huggingface.co/datasets/judicialmind/BharatAddr-Gazetteer)): 675,247 LGD villages
and 5,046 towns with hierarchy, PINs, local-script and Devanagari names, aliases and old names.
2. **Encoder** (`model/`): Qwen3-Embedding-0.6B fine-tuned to map a messy address close to its canonical record
"village (Devanagari), sub-district, district, state PIN". Queries are prefixed with the instruction stored in
`model/config_sentence_transformers.json`; embeddings use last-token pooling and L2 normalisation. The records are
pre-embedded in `index/doc_emb.f16.npy` (rows aligned with `index/docs.parquet`).
3. **Resolver** (`bharataddr/resolve.py`): the encoder's top-50 records plus the baseline's top-20 candidates are
ranked by encoder similarity, with the baseline's hierarchy score (village, sub-district, district, state and PIN
evidence found in the text) breaking ties.
4. **Baseline** (`bharataddr/baseline.py`, `--no-neural`): script-aware romanisation, phonetic and consonant-skeleton
keys, exact + fuzzy n-gram matching at every level, level-marker hints (Dist., Teh., Vill., PO), PIN lookup and a
hierarchy-consistent score per candidate. About 10 ms per address per CPU core.
## Files
| Path | What |
|---|---|
| `bharataddr/` | resolver package: `resolve.py` (API + CLI), `baseline.py`, `neural.py`, `text.py`, `evaluate.py` |
| `gazetteer/` | `entries.parquet`, `units.parquet`, `pins.parquet`, `stats.json` (same as BharatAddr-Gazetteer) |
| `model/` | fine-tuned encoder (safetensors, tokenizer, sentence-transformers config) |
| `index/` | `docs.parquet` (record id + canonical text) and `doc_emb.f16.npy` (680,293 × 1,024 fp16 embeddings) |
| `metrics.json` | all reported results, with breakdowns by script, PIN condition, place kind and district |
| `eval/` | BharatAddr predictions for `test_district` and `test_random` |
| `requirements.txt` | pinned runtime dependencies |
## Results
Benchmark: the synthetic messy-address splits of
[BharatAddr-Gazetteer](https://huggingface.co/datasets/judicialmind/BharatAddr-Gazetteer) (`synthetic/`). Exact match
of the LGD code at each level; PIN counts as correct when it is one of the PINs LGD lists for the gold village.
**Random held-out villages and towns (`test_random`, districts seen in training), exact match %, n = 10,000**
| System | Village / town | Sub-district | District | State | PIN |
|---|---|---|---|---|---|
| **BharatAddr** | **95.27** | **97.73** | **99.17** | **99.66** | **97.57** |
| Qwen3-Embedding-0.6B zero-shot + hierarchy reranking | 90.25 | 95.39 | 98.03 | 99.07 | 95.49 |
| Qwen3-Embedding-0.6B zero-shot, retrieval only | 65.65 | 75.97 | 83.35 | 89.30 | 74.68 |
| Non-neural baseline | 87.15 | 93.10 | 96.40 | 98.10 | 93.07 |
**Village accuracy on held-out districts by input script (`test_district`)**
| Script of the address | Addresses | BharatAddr | Baseline |
|---|---|---|---|
| English / Roman | 5,537 | 96.24 | 85.19 |
| Hindi (Devanagari) | 2,240 | 95.58 | 89.69 |
| Mixed Roman + Devanagari | 1,230 | 95.12 | 86.34 |
| Romanised from Devanagari / local script | 981 | 93.99 | 85.42 |
| Other Indic script | 12 | 100.00 | 91.67 |
**Village accuracy on held-out districts by PIN condition (`test_district`)**
| PIN in the address | Addresses | BharatAddr | Baseline |
|---|---|---|---|
| Correct PIN present | 5,497 | 97.18 | 92.36 |
| No PIN | 3,884 | 93.98 | 79.84 |
| Wrong PIN | 619 | 94.02 | 74.15 |
**Encoder retrieval recall on held-out districts (`test_district`), %**
| Encoder | R@1 | R@10 | R@50 |
|---|---|---|---|
| **BharatAddr** | **95.74** | **99.77** | **99.93** |
| Qwen3-Embedding-0.6B zero-shot | 67.47 | 87.85 | 93.46 |
Macro village accuracy over the 63 held-out districts: 96.36 (baseline 86.79).
Reproduce the tables from the shipped predictions and the benchmark files:
```bash
huggingface-cli download judicialmind/BharatAddr-Gazetteer --repo-type dataset --local-dir BharatAddr-Gazetteer
python -m bharataddr.evaluate --gold BharatAddr-Gazetteer/synthetic/test_district.parquet \
--pred eval/pred_bharataddr_test_district.parquet
python -m bharataddr.evaluate --gold BharatAddr-Gazetteer/synthetic/test_district.parquet --run --limit 500
```
The first command scores the shipped predictions (add `--json` for the breakdowns in `metrics.json`); the second
runs the resolver end to end on the first 500 addresses (add `--no-neural` for the baseline). To score your own
system, write a parquet with columns `qid`, `eid`, `sd_code`, `d_code`, `s_code`, `pin` and pass it as `--pred`.
## Coverage
All of India: 35 states and union territories, 793 districts, 7,295
sub-districts, 675,247 villages and 5,046 towns, 17,995 distinct PINs. Places are resolved to the village /
town level; street, house and landmark details are ignored. Benchmark addresses are synthetic and built from the same
official directories; no personal data is used anywhere.
## Source data
Contains data from the Local Government Directory (LGD), Ministry of Panchayati Raj, and the Locality based Pincode directory, Department of Posts, Government of India, published on data.gov.in under the Government Open Data License - India (GODL-India), https://data.gov.in/government-open-data-license-india .
| Source (data.gov.in resource id) | Publisher | Used for |
|---|---|---|
| LGD - Villages (`c967fe8f-69c4-42df-8afc-8a2c98057437`) | Ministry of Panchayati Raj | village / sub-district / district / state codes, English + local-script names |
| LGD - Villages with PIN Codes (`f17a1608-5f10-4610-bb50-a63c80d83974`) | Ministry of Panchayati Raj | village -> PIN |
| LGD - Local Bodies (`1a6c26ed-d67c-40ea-aa20-d38d35f341a5`) | Ministry of Panchayati Raj | gram-panchayat names, urban local bodies (towns) |
| Locality based Pincode, 15 Feb 2016 (`7eca2fa3-d6f5-444e-b3d6-faa441e35294`) | Department of Posts | post-office names per PIN, DoP village spellings (locality_detail columns not used) |
| District crosswalk (derived from the portals above) | - | observed district spellings |
Model weights derive from [Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B).
<!-- jm:source -->
Original data: published by Ministry of Panchayati Raj (Local Government Directory) and Department of Posts, Ministry of Communications (Locality based Pincode directory) on the Open Government Data Platform India ([data.gov.in](https://data.gov.in)) and, where marked, NITI Aayog's National Data and Analytics Platform ([NDAP](https://ndap.niti.gov.in)), under the [Government Open Data License - India](https://data.gov.in/government-open-data-license-india) (GODL-India). The original public data remains under GODL-India; the CC BY-NC-SA 4.0 licence of this repository covers JudicialMind's compilation, derived fields, annotations, benchmarks and models.
<!-- /jm:source -->
## Citation
```bibtex
@misc{judicialmind2026bharataddr,
title={BharatAddr: Resolving Messy Indian Addresses to LGD Villages, Districts and PIN Codes},
author={JudicialMind},
year={2026},
publisher={Hugging Face},
howpublished={\url{https://huggingface.co/judicialmind/BharatAddr}}
}
```
<!-- jm:license -->
## License
[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) (see `LICENSE`): research, academic, personal and other non-commercial use, with attribution to JudicialMind and to the original data publishers; adaptations must be shared under the same licence. Commercial use requires a separate licence.
Commercial use: contact surya@judicialmind.ai or hello@judicialmind.ai.
Base model(s): `Qwen/Qwen3-Embedding-0.6B` (apache-2.0). Their licence terms and notices continue to apply to the upstream components; see `THIRD_PARTY_NOTICES.md`.
<!-- /jm:license -->