--- title: Digital KP 2030 Assistant emoji: ๐Ÿ“˜ colorFrom: blue colorTo: indigo sdk: streamlit sdk_version: 1.40.2 app_file: app.py pinned: false --- # Digital KP 2030 Assistant A retrieval-augmented (RAG) chatbot that answers questions about the **Khyber Pakhtunkhwa Digital Transformation Policy & Roadmap 2030**, citing the page number behind every claim and refusing to answer when the document does not cover the question. --- ## Why this is built the way it is The published Digital KP 2030 file is a **design-heavy, image-based document**. Its embedded text layer is nearly empty โ€” most of the 50 roadmap pages yield only 90โ€“120 characters (running headers), and several pages return mojibake from custom font encodings. Across all 93 pages there are roughly 18,800 extractable characters in total. A conventional `PyPDF โ†’ chunk โ†’ embed` pipeline therefore indexes almost nothing, and the resulting assistant answers confidently from thin air or fails outright. **OCR is not optional for this document.** Two design decisions follow: 1. **`ingest.py` extracts text first and falls back to OCR** for any page whose text layer is too thin or too corrupted. It also detects the case where the `.pdf` is really a ZIP archive of page images, which some distributions are. 2. **Indexing happens offline, not in the Space.** You run `ingest.py` once on your own machine and commit the resulting `index/` folder. The Space then starts in seconds, needs no OCR binaries at runtime, and behaves deterministically. --- ## Architecture ``` PDF (image-based) โ”‚ โ–ผ ingest.py โ”€โ”€ text extraction โ”€โ”€โ–บ thin or garbled? โ”€โ”€โ–บ OCR (tesseract) โ”‚ โ–ผ chunking 1,100 chars, 180-char overlap, page number kept per chunk โ”‚ โ–ผ embeddings sentence-transformers/all-MiniLM-L6-v2 (local, free) โ”‚ โ–ผ vector store FAISS IndexFlatIP over normalised vectors (= cosine) โ”‚ โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€ index/ committed to the repo โ”€โ”€โ”€โ”€โ”€ โ”‚ โ–ผ app.py / rag.py โ”‚ โ”œโ”€ retrieve top-6 chunks, drop anything below a relevance floor โ”œโ”€ build a prompt containing ONLY those page-tagged excerpts โ”œโ”€ LLM generates the answer under strict grounding rules โ””โ”€ return answer + expandable source passages with page numbers ``` Grounding is enforced in three places rather than one: a relevance floor that discards weak matches, a system prompt that forbids outside knowledge and mandates page citations, and a UI that always shows the retrieved passages so a user can check the answer themselves. --- ## Project structure ``` digital-kp-assistant/ โ”œโ”€โ”€ app.py Streamlit chat interface โ”œโ”€โ”€ rag.py retrieval, prompting, LLM call โ”œโ”€โ”€ ingest.py run once locally to build the index โ”œโ”€โ”€ requirements.txt โ”œโ”€โ”€ packages.txt apt packages (only needed if you ever OCR in the Space) โ”œโ”€โ”€ .gitignore โ”œโ”€โ”€ data/ โ”‚ โ””โ”€โ”€ digital_kp_2030.pdf โ””โ”€โ”€ index/ generated by ingest.py โ€” commit this โ”œโ”€โ”€ faiss.index โ”œโ”€โ”€ chunks.json โ””โ”€โ”€ meta.json ``` --- ## Setup ### 1. Add the PDF Download the document from the Khyber Pakhtunkhwa Digital Transformation Board site and save it as `data/digital_kp_2030.pdf`. The filename matters โ€” it is what `ingest.py` looks for. ### 2. Install and build the index (local machine) OCR needs two system tools in addition to the Python packages: ```bash # Ubuntu / Debian sudo apt install poppler-utils tesseract-ocr # macOS brew install poppler tesseract # Windows: install Tesseract and Poppler, then add both to PATH ``` ```bash pip install -r requirements.txt python ingest.py ``` Expect this to take several minutes โ€” OCR at 220 DPI across 93 pages is the slow part. You should end with roughly 400โ€“700 chunks. If you get fewer than 100, OCR is not running; check that `tesseract` is on your PATH. ### 3. Run locally ```bash export LLM_API_KEY="your-key" streamlit run app.py ``` ### 4. Deploy to Hugging Face Spaces 1. Create a new Space: **SDK = Streamlit**, hardware **CPU basic** is enough. 2. Push the repository, **including `index/`** (it is small โ€” a few MB). ```bash git remote add space https://huggingface.co/spaces//digital-kp-assistant git push space main ``` If `index/` is larger than 10 MB, track it with Git LFS first: `git lfs track "index/*.index"`. 3. Add the secrets below under **Settings โ†’ Variables and secrets**. 4. The Space rebuilds and starts automatically. --- ## Secrets and environment variables Set these as **Secrets** (not Variables) in the Space settings: | Name | Required | Default | Notes | |---|---|---|---| | `LLM_API_KEY` | yes | โ€” | Your provider key. Never commit it. | | `LLM_BASE_URL` | no | `https://api.groq.com/openai/v1` | Any OpenAI-compatible endpoint. | | `LLM_MODEL` | no | `llama-3.3-70b-versatile` | Must exist on your provider. | Because the client is OpenAI-compatible, you can switch providers by changing two values and nothing else: | Provider | `LLM_BASE_URL` | Example `LLM_MODEL` | |---|---|---| | Groq (free tier, fast) | `https://api.groq.com/openai/v1` | `llama-3.3-70b-versatile` | | OpenAI | `https://api.openai.com/v1` | `gpt-4o-mini` | | OpenRouter | `https://openrouter.ai/api/v1` | `anthropic/claude-3.5-sonnet` | | Together | `https://api.together.xyz/v1` | `meta-llama/Llama-3.3-70B-Instruct-Turbo` | Embeddings run locally via `sentence-transformers`, so they cost nothing and need no key. --- ## Test questions Use these to check the system before demonstrating it. The first group should produce confident, cited answers; the last group should be refused. **Should answer well** 1. What is the vision of Digital KP 2030? 2. Explain the DIGIT 2030 framework. 3. What are the nine strategic goals? 4. Who chairs the KP Digital Council, and how often does it meet? 5. How many interventions sit under each strategic pillar? 6. What are the 2030 KPIs for digital payments? 7. What is DASTAK and how does it relate to Khyber Pass? 8. What does the roadmap say about AI governance and ethics? 9. What is expected of a Departmental Implementation Forum? 10. What targets exist for upskilling government officials? **Should be refused โ€” these are not in the document** 11. What is the budget allocation for Digital KP 2030 in rupees? 12. How does this compare to the Punjab digital strategy? 13. Who won the contract to build DASTAK? **Should surface a contradiction rather than pick a side** 14. How many strategic pillars are there? *(p. 24 prints "05"; p. 25 states six.)* 15. How many interventions are there in total? *(p. 13 "more than 900"; p. 24 "815+"; p. 27 gives 924, which sums correctly from its own table.)* Questions 14 and 15 are the most instructive demonstration in the whole project โ€” they show why page-level citation matters, and they are a genuine property of the source document rather than a model failure. --- ## Improving it later **Retrieval quality** - Add hybrid search: combine the current dense vectors with BM25 keyword matching, which handles exact terms like "DASTAK 2.0" or "Q3 2026" far better than embeddings do. - Add a reranker (`cross-encoder/ms-marco-MiniLM-L-6-v2`) over the top 20 hits to lift precision before the LLM sees anything. - Store the pillar and thematic-domain name on each chunk during ingestion so users can filter by pillar, and so citations can read "Pillar 4 ยท Data Governance (p. 55)" rather than a bare page number. **Citations** - Render the cited page as an image next to the answer. Because the source is already page images, this is unusually easy and very convincing in a government demonstration. - Highlight the retrieved passage within the page image. **Conversation memory** - The app currently passes the last two exchanges. For longer sessions, add query rewriting: use the LLM to turn "what about the second one?" into a standalone question before retrieval, since follow-up pronouns retrieve badly. **Multilingual support (English / Urdu / Pashto)** - Swap the embedding model for a multilingual one such as `paraphrase-multilingual-MiniLM-L12-v2` or `BAAI/bge-m3`, so an Urdu question can retrieve English source text directly. This single change matters more than anything else on this list. - Add `+urd` to `OCR_LANGS` in `ingest.py` after installing `tesseract-ocr-urd`. Pashto OCR is weak in Tesseract; for Pashto, translating the query into English before retrieval works better than OCRing Pashto text. - Add a language selector so answers can be forced into Urdu or Pashto regardless of the input language โ€” useful for briefing mixed audiences. **Evaluation** - Build a small fixed set of question/expected-page pairs and measure retrieval recall@5 after any change. Without this you cannot tell whether a "better" model actually helped. --- ## Limitations, stated plainly - Answers are only as good as the OCR. Tables in the roadmap's implementation plans OCR imperfectly, so timelines and KPIs should be verified against the page before being quoted in any official document. - The assistant covers one document. It knows nothing about subsequent notifications, amendments or progress reports. - The document is described as a living strategy subject to review at least every three years, so rebuild the index when a revised version is published.