ICMR + HITEK Full DB (Mixed) — Prebuilt Indexes + One-Click Setup

Prebuilt sorted indexes for the Kzr0xx/Icmr-and-hitek dataset (2.5B rows, 11 columns, ~104 GB raw parquet).

Building these indexes took ~17 hours of compute. This repo saves you that work: download + run = API live in ~1-2 hours (download speed dependent).

Contents

The indexes are stored as sorted parts (each < 50 GB, split at row-group boundaries, order preserved) because HuggingFace's classic HTTP upload path caps files at 50 GB. The API reads all parts of an index with one glob — zone-map pruning still makes lookups milliseconds-fast.

Raw data files (part1.parquet, part2a.parquet, part2b_new.parquet, 104 GB) live in the Icmr-and-hitek repo — the setup scripts download them from there automatically.

One-click restore

Pick your platform, run ONE file. It downloads data + indexes + code, creates a venv, installs deps, starts the API.

Windows: double-click setup.bat PowerShell: right-click → Run with PowerShell, or powershell -ExecutionPolicy Bypass -File setup.ps1 Linux/VPS:

chmod +x setup.sh
./setup.sh

Total download: ~305 GB (104 GB data + ~200 GB index parts). Scripts resume interrupted downloads (curl -C -), so a dropped connection is not a problem — just re-run.

API

Base: http://127.0.0.1:8001 (Linux: 0.0.0.0:8001)

All 11 columns searchable: name, fathersName, phoneNumber, aadharNumber, otherNumber, address, district, pincode, state, town, source. Duplicates capped at 2 per person. source = icmr | inddata (hitek data is labelled inddata).

Manual start (if scripts already ran)

# Windows
cd /d "C:\path\to\folder"
start "" .venv\Scripts\python.exe -m uvicorn main:app --host 127.0.0.1 --port 8001

# Linux
cd /path/to/folder
ICMR_DATA_DIR="$PWD/data" nohup .venv/bin/python -m uvicorn main:app --host 0.0.0.0 --port 8001 > api.log 2>&1 &

ICMR_DATA_DIR points at the data/ folder; if it is not set, data/ next to main.py is used. main.py auto-detects index parts (idx_phone.*.parquet) — no config needed.

Rebuilding indexes (rarely needed)

ICMR_DATA_DIR="$PWD/data" .venv/bin/python build_index.py

Each index takes 6-11 hours on a typical machine. After building, split_idx.py can split them into upload-sized parts again.