Text Generation
Transformers
Safetensors
English
qwen3
table-cleaning
india
government-data
data-standardisation
conversational
text-generation-inference
Instructions to use judicialmind/BharatTableScribe-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use judicialmind/BharatTableScribe-4B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="judicialmind/BharatTableScribe-4B") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("judicialmind/BharatTableScribe-4B") model = AutoModelForCausalLM.from_pretrained("judicialmind/BharatTableScribe-4B", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use judicialmind/BharatTableScribe-4B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "judicialmind/BharatTableScribe-4B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "judicialmind/BharatTableScribe-4B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/judicialmind/BharatTableScribe-4B
- SGLang
How to use judicialmind/BharatTableScribe-4B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "judicialmind/BharatTableScribe-4B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "judicialmind/BharatTableScribe-4B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "judicialmind/BharatTableScribe-4B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "judicialmind/BharatTableScribe-4B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use judicialmind/BharatTableScribe-4B with Docker Model Runner:
docker model run hf.co/judicialmind/BharatTableScribe-4B
|
Download README.md from judicialmind/BharatTableScribe-4B: direct link, hf CLI and curl.
- Browser
- Download file 10.9 kB
-
https://huggingface.co/judicialmind/BharatTableScribe-4B/resolve/main/README.md
- Command line
-
hf download hf://judicialmind/BharatTableScribe-4B/README.md
-
curl -L -H "Authorization: Bearer $HF_TOKEN" -o README.md https://huggingface.co/judicialmind/BharatTableScribe-4B/resolve/main/README.md
10.9 kB
| license: cc-by-nc-sa-4.0 | |
| base_model: Qwen/Qwen3-4B-Instruct-2507 | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| tags: | |
| - table-cleaning | |
| - india | |
| - government-data | |
| - data-standardisation | |
| datasets: | |
| - judicialmind/BharatTableScribe-Data | |
| language: | |
| - en | |
| extra_gated_prompt: 'This model is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 licence (CC | |
| BY-NC-SA 4.0): research, academic, personal and other non-commercial use is permitted with attribution; adaptations must | |
| be shared under the same licence; commercial use is not permitted. The original public source data remains under the Government | |
| Open Data License - India (GODL-India). Commercial use: contact surya@judicialmind.ai or hello@judicialmind.ai.' | |
| extra_gated_fields: | |
| Name: text | |
| Email: text | |
| Organization: text | |
| I accept the CC BY-NC-SA licence terms (research and non-commercial use only): checkbox | |
| extra_gated_button_content: Accept and request access | |
| # BharatTableScribe-4B | |
| <!-- jm:top --> | |
| Licence: CC BY-NC-SA 4.0 — research and non-commercial use only; derivatives must be shared under the same licence. Commercial use: contact surya@judicialmind.ai or hello@judicialmind.ai | |
| Access is approved automatically when the email you enter matches your Hugging Face account email and belongs to the organisation you name; other requests are reviewed manually. | |
| <!-- /jm:top --> | |
| A full fine-tune of Qwen/Qwen3-4B-Instruct-2507 that turns a raw Indian government table into a clean long table | |
| by writing a short cleaning program. The model reads the table as published (title rows, flattened or merged | |
| headers, unit clauses such as '(Rs. in lakh)', Indian digit grouping, total rows, serial-number columns, transposed | |
| layouts) and replies with one JSON object `{"schema": {...}, "program": [...]}`. The program is executed on the | |
| source table by the bundled `tablescribe` package, so output values are the table's own cells, parsed and | |
| multiplied by the declared unit scale. | |
| **Headline result:** on held-out ministries BharatTableScribe-4B reaches cell F1 0.946 on synthetic tables and 0.974 on | |
| NDAP-relabelled tables, against 0.792 and 0.535 for the rule baseline, and its programs execute on 97.8% of test items. | |
| ## Quickstart | |
| ```bash | |
| pip install -r requirements.txt # torch, transformers, accelerate (vllm optional, for GPU serving) | |
| ``` | |
| ### transformers | |
| ```python | |
| from huggingface_hub import snapshot_download | |
| import sys | |
| MODEL = snapshot_download("judicialmind/BharatTableScribe-4B") | |
| sys.path.insert(0, MODEL) # infer.py + tablescribe/ ship with the weights | |
| from tablescribe import RawTable, render | |
| from infer import load, generate, run_program | |
| raw = RawTable( # toy table, illustrative values | |
| title="State-wise Production of Milk from 2020-21 to 2022-23 (in '000 tonnes)", | |
| ministry="Ministry of Fisheries, Animal Husbandry and Dairying", | |
| rows=[ | |
| ["Sl. No.", "State/UT", "2020-21", "2021-22", "2022-23"], | |
| ["1", "Bihar", "11,502", "12,320", "12,852"], | |
| ["2", "Gujarat", "15,862", "16,719", "17,282"], | |
| ["3", "Kerala", "2,544", "2,527", "2,518"], | |
| ["", "Total", "29,908", "31,566", "32,652"], | |
| ], | |
| ) | |
| print(render(raw)) # the user turn the model sees | |
| tok, model = load(MODEL) # GPU if available, else CPU | |
| reply = generate(tok, model, raw, max_new_tokens=512) | |
| print(reply) # {"schema": {...}, "program": [...]} | |
| schema, table = run_program(raw, reply) # program executed on the full raw table | |
| print(table.keys + ["value"]) | |
| for row in table.rows[:4]: | |
| print(row) | |
| ``` | |
| Output (CPU, greedy decoding; the long `drop_rows` pattern is the model's own total-row filter). The rendered | |
| user turn: | |
| ```text | |
| Title: State-wise Production of Milk from 2020-21 to 2022-23 (in '000 tonnes) | |
| Ministry: Ministry of Fisheries, Animal Husbandry and Dairying | |
| Grid: 5 rows x 5 columns | |
| r0 | Sl. No. | State/UT | 2020-21 | 2021-22 | 2022-23 | |
| r1 | 1 | Bihar | 11,502 | 12,320 | 12,852 | |
| r2 | 2 | Gujarat | 15,862 | 16,719 | 17,282 | |
| r3 | 3 | Kerala | 2,544 | 2,527 | 2,518 | |
| r4 | | Total | 29,908 | 31,566 | 32,652 | |
| ``` | |
| The model's reply: | |
| ```json | |
| {"schema":{"geo_level":"state","time":"financial_year","scales":{"*":1.0}},"program":[{"op":"header"},{"op":"drop_rows","empty":true},{"op":"drop_cols","cols":["Sl. No."]},{"op":"drop_rows","col":"State/UT","match":"^\\s*(?:grand\\s+|sub[\\s-]*)?total\\b|^\\s*all[\\s-]*india\\b|^\\s*india\\s*$"},{"op":"melt","id":["State/UT"],"into":["year"]},{"op":"number"},{"op":"geo","col":"State/UT","level":"state"},{"op":"time","col":"year","kind":"financial_year"}]} | |
| ``` | |
| **First four rows of the clean table ('000 tonnes)** | |
| | state name | year | value | | |
| |---|---|---:| | |
| | Bihar | 2020-21 | 11,502 | | |
| | Bihar | 2021-22 | 12,320 | | |
| | Bihar | 2022-23 | 12,852 | | |
| | Gujarat | 2020-21 | 15,862 | | |
| The total row is dropped, the serial-number column removed, the year columns melted into a canonical | |
| `financial_year` key and the place column marked as `state`. The declared scale is 1, so values stay in the | |
| table's own unit ('000 tonnes, from the title). | |
| CLI for a CSV saved exactly as published (all rows, including title and header rows): | |
| ```bash | |
| python infer.py --model judicialmind/BharatTableScribe-4B --csv table.csv --title "<table title>" --out clean.csv | |
| ``` | |
| ### vLLM | |
| ```python | |
| # MODEL and raw as in the transformers example | |
| from vllm import LLM, SamplingParams | |
| from tablescribe import RawTable, messages, parse_output, execute | |
| llm = LLM(model=MODEL, max_model_len=12288) | |
| outs = llm.chat([messages(raw)], SamplingParams(temperature=0.0, max_tokens=1536)) | |
| schema, program, err = parse_output(outs[0].outputs[0].text) | |
| table = execute(raw.rows, program) | |
| ``` | |
| or serve it (`vllm serve judicialmind/BharatTableScribe-4B --max-model-len 12288`) and send `messages(raw)` to the | |
| OpenAI-compatible `/v1/chat/completions` endpoint with `temperature=0`. | |
| ### Prompt format | |
| Two chat turns (Qwen3 chat template, no thinking block): | |
| - **system**: the fixed instruction `tablescribe.prompts.SYSTEM` (begins "You convert raw Indian government tables | |
| into a tidy long table. Write ONE JSON object: {"schema": {...}, "program": [...]} and nothing else." and lists | |
| the DSL steps). Use it verbatim. | |
| - **user**: `tablescribe.grid.render(raw)`: | |
| ``` | |
| Title: <table title> | |
| Ministry: <publisher> | |
| Grid: <n> rows x <m> columns | |
| r0 | <cell> | <cell> | ... | |
| r1 | <cell> | <cell> | ... | |
| ``` | |
| Every cell is text and header rows are ordinary rows (`r0`, `r1`, ...). Long tables show the first 24 and last 6 | |
| rows with `... k rows omitted ...` between them, at most 40 columns, cells clipped to 48 characters (`|` inside a | |
| cell becomes `/`); wide tables are shown with fewer rows until the text fits 9,000 characters. The program still | |
| runs on the full table. | |
| ### Output format | |
| One compact JSON object: | |
| ```json | |
| {"schema": {"geo_level": "state", "time": "financial_year", "scales": {"*": 1000.0}}, | |
| "program": [{"op": "header"}, {"op": "drop_cols", "cols": ["Sl. No."]}, ...]} | |
| ``` | |
| `schema.geo_level` is the finest place level (national, state, district, subdistrict, block, village, city or | |
| none), `schema.time` the time kind (year, financial_year, agri_year, year_month, month, date, quarter or none), | |
| `schema.scales` the unit multipliers per variable (`"*"` = all). `program` steps: `skip`, `header`, `drop_rows`, | |
| `drop_cols`, `fill_down`, `split_col`, `rename`, `map`, `transpose`, `melt`, `number`, `geo`, `time`, `const` | |
| (documented in `tablescribe/dsl.py`). `tablescribe.execute(raw.rows, program)` returns a `CleanTable`: key columns | |
| (`keys`), one numeric `value` column in base units (`rows` = keys + value), and column roles (`geo:<level>`, | |
| `time:<kind>`, `key`). | |
| ## Files | |
| | file | contents | | |
| |---|---| | |
| | `model.safetensors`, `config.json`, `generation_config.json`, tokenizer files, `chat_template.jinja` | weights (bf16) and tokenizer | | |
| | `infer.py` | `build_messages`, `load`, `generate`, `run_program`, `clean_table` + CLI | | |
| | `tablescribe/` | prompt rendering, output parsing, program executor (pure Python, no dependencies) | | |
| | `requirements.txt` | pinned runtime dependencies | | |
| ## Results | |
| Test split of [BharatTableScribe-Data](https://huggingface.co/datasets/judicialmind/BharatTableScribe-Data) (held-out | |
| ministries; greedy decoding): | |
| | model | synthetic cell F1 | synthetic schema acc. | NDAP-relabelled cell F1 | exec ok | | |
| |---|---|---|---|---| | |
| | Qwen3-30B-A3B, zero-shot | 0.009 | 0.644 | 0.000 | 0.118 | | |
| | Qwen3-4B, zero-shot | 0.001 | 0.487 | 0.000 | 0.034 | | |
| | rule baseline | 0.792 | 0.890 | 0.535 | 0.976 | | |
| | **BharatTableScribe-4B** | **0.946** | **0.945** | **0.974** | **0.978** | | |
| Cell F1 compares executed outputs with gold tables. The dataset ships every model's test replies and `evaluate.py`, | |
| which reproduces these numbers. | |
| ## Coverage | |
| Trained and evaluated on English-language tables from Indian central ministries, Parliament answers and State | |
| governments published on data.gov.in and NITI Aayog's NDAP (snapshot May 2026). | |
| ## Source data | |
| Training tables come from the Open Government Data Platform India (data.gov.in) and NITI Aayog NDAP under the | |
| Government Open Data License - India; per-table attribution is in `judicialmind/BharatTableScribe-Data` (GODL-India / NDAP | |
| attribution in its card). | |
| <!-- jm:source --> | |
| Original data: published by the ministries, departments, State governments and Parliament named per table in judicialmind/BharatTableScribe-Data (`attribution` column) on the Open Government Data Platform India ([data.gov.in](https://data.gov.in)) and, where marked, NITI Aayog's National Data and Analytics Platform ([NDAP](https://ndap.niti.gov.in)), under the [Government Open Data License - India](https://data.gov.in/government-open-data-license-india) (GODL-India). The original public data remains under GODL-India; the CC BY-NC-SA 4.0 licence of this repository covers JudicialMind's compilation, derived fields, annotations, benchmarks and models. | |
| <!-- /jm:source --> | |
| ## Citation | |
| ```bibtex | |
| @misc{judicialmind2026tablescribe, | |
| title={BharatTableScribe-4B: Cleaning Raw Indian Government Tables with Executable Programs}, | |
| author={JudicialMind}, | |
| year={2026}, | |
| publisher={Hugging Face}, | |
| howpublished={\url{https://huggingface.co/judicialmind/BharatTableScribe-4B}} | |
| } | |
| ``` | |
| <!-- jm:license --> | |
| ## License | |
| [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) (see `LICENSE`): research, academic, personal and other non-commercial use, with attribution to JudicialMind and to the original data publishers; adaptations must be shared under the same licence. Commercial use requires a separate licence. | |
| Commercial use: contact surya@judicialmind.ai or hello@judicialmind.ai. | |
| Base model(s): `Qwen/Qwen3-4B-Instruct-2507` (apache-2.0). Their licence terms and notices continue to apply to the upstream components; see `THIRD_PARTY_NOTICES.md`. | |
| <!-- /jm:license --> | |