README.md

metadata

title: Digital KP 2030 Assistant
emoji: 📘
colorFrom: blue
colorTo: indigo
sdk: streamlit
sdk_version: 1.40.2
app_file: app.py
pinned: false

Digital KP 2030 Assistant

A retrieval-augmented (RAG) chatbot that answers questions about the Khyber Pakhtunkhwa Digital Transformation Policy & Roadmap 2030, citing the page number behind every claim and refusing to answer when the document does not cover the question.

---

Why this is built the way it is

The published Digital KP 2030 file is a design-heavy, image-based document. Its embedded text layer is nearly empty — most of the 50 roadmap pages yield only 90–120 characters (running headers), and several pages return mojibake from custom font encodings. Across all 93 pages there are roughly 18,800 extractable characters in total.

A conventional PyPDF → chunk → embed pipeline therefore indexes almost nothing, and the resulting assistant answers confidently from thin air or fails outright. OCR is not optional for this document.

Two design decisions follow:

1. ingest.py extracts text first and falls back to OCR for any page whose text layer is too thin or too corrupted. It also detects the case where the .pdf is really a ZIP archive of page images, which some distributions are.
2. Indexing happens offline, not in the Space. You run ingest.py once on your own machine and commit the resulting index/ folder. The Space then starts in seconds, needs no OCR binaries at runtime, and behaves deterministically.

---

Architecture

  PDF (image-based)
        │
        ▼
  ingest.py  ──  text extraction  ──► thin or garbled? ──► OCR (tesseract)
        │
        ▼
  chunking            1,100 chars, 180-char overlap, page number kept per chunk
        │
        ▼
  embeddings          sentence-transformers/all-MiniLM-L6-v2  (local, free)
        │
        ▼
  vector store        FAISS IndexFlatIP over normalised vectors (= cosine)
        │
   ─────┴─────  index/ committed to the repo  ─────
        │
        ▼
  app.py / rag.py
        │
        ├─ retrieve top-6 chunks, drop anything below a relevance floor
        ├─ build a prompt containing ONLY those page-tagged excerpts
        ├─ LLM generates the answer under strict grounding rules
        └─ return answer + expandable source passages with page numbers

Grounding is enforced in three places rather than one: a relevance floor that discards weak matches, a system prompt that forbids outside knowledge and mandates page citations, and a UI that always shows the retrieved passages so a user can check the answer themselves.

---

Project structure

digital-kp-assistant/
├── app.py              Streamlit chat interface
├── rag.py              retrieval, prompting, LLM call
├── ingest.py           run once locally to build the index
├── requirements.txt
├── packages.txt        apt packages (only needed if you ever OCR in the Space)
├── .gitignore
├── data/
│   └── digital_kp_2030.pdf
└── index/              generated by ingest.py — commit this
    ├── faiss.index
    ├── chunks.json
    └── meta.json

---

Setup

1. Add the PDF

Download the document from the Khyber Pakhtunkhwa Digital Transformation Board site and save it as data/digital_kp_2030.pdf. The filename matters — it is what ingest.py looks for.

2. Install and build the index (local machine)

OCR needs two system tools in addition to the Python packages:

# Ubuntu / Debian
sudo apt install poppler-utils tesseract-ocr
# macOS
brew install poppler tesseract
# Windows: install Tesseract and Poppler, then add both to PATH

pip install -r requirements.txt
python ingest.py

Expect this to take several minutes — OCR at 220 DPI across 93 pages is the slow part. You should end with roughly 400–700 chunks. If you get fewer than 100, OCR is not running; check that tesseract is on your PATH.

3. Run locally

export LLM_API_KEY="your-key"
streamlit run app.py

4. Deploy to Hugging Face Spaces

1. Create a new Space: SDK = Streamlit, hardware CPU basic is enough.
2. Push the repository, including index/ (it is small — a few MB).

   git remote add space https://huggingface.co/spaces/<username>/digital-kp-assistant
   git push space main
   

   If index/ is larger than 10 MB, track it with Git LFS first: git lfs track "index/*.index".
3. Add the secrets below under Settings → Variables and secrets.
4. The Space rebuilds and starts automatically.

---

Secrets and environment variables

Set these as Secrets (not Variables) in the Space settings:

Name	Required	Default	Notes
LLM_API_KEY	yes	—	Your provider key. Never commit it.
LLM_BASE_URL	no	https://api.groq.com/openai/v1	Any OpenAI-compatible endpoint.
LLM_MODEL	no	llama-3.3-70b-versatile	Must exist on your provider.

Because the client is OpenAI-compatible, you can switch providers by changing two values and nothing else:

Provider	LLM_BASE_URL	Example LLM_MODEL
Groq (free tier, fast)	https://api.groq.com/openai/v1	llama-3.3-70b-versatile
OpenAI	https://api.openai.com/v1	gpt-4o-mini
OpenRouter	https://openrouter.ai/api/v1	anthropic/claude-3.5-sonnet
Together	https://api.together.xyz/v1	meta-llama/Llama-3.3-70B-Instruct-Turbo

Embeddings run locally via sentence-transformers, so they cost nothing and need no key.

---

Test questions

Use these to check the system before demonstrating it. The first group should produce confident, cited answers; the last group should be refused.

Should answer well

1. What is the vision of Digital KP 2030?
2. Explain the DIGIT 2030 framework.
3. What are the nine strategic goals?
4. Who chairs the KP Digital Council, and how often does it meet?
5. How many interventions sit under each strategic pillar?
6. What are the 2030 KPIs for digital payments?
7. What is DASTAK and how does it relate to Khyber Pass?
8. What does the roadmap say about AI governance and ethics?
9. What is expected of a Departmental Implementation Forum?
10. What targets exist for upskilling government officials?

Should be refused — these are not in the document

11. What is the budget allocation for Digital KP 2030 in rupees?
12. How does this compare to the Punjab digital strategy?
13. Who won the contract to build DASTAK?

Should surface a contradiction rather than pick a side

14. How many strategic pillars are there? (p. 24 prints "05"; p. 25 states six.)
15. How many interventions are there in total? (p. 13 "more than 900"; p. 24 "815+"; p. 27 gives 924, which sums correctly from its own table.)

Questions 14 and 15 are the most instructive demonstration in the whole project — they show why page-level citation matters, and they are a genuine property of the source document rather than a model failure.

---

Improving it later

Retrieval quality

• Add hybrid search: combine the current dense vectors with BM25 keyword matching, which handles exact terms like "DASTAK 2.0" or "Q3 2026" far better than embeddings do.
• Add a reranker (cross-encoder/ms-marco-MiniLM-L-6-v2) over the top 20 hits to lift precision before the LLM sees anything.
• Store the pillar and thematic-domain name on each chunk during ingestion so users can filter by pillar, and so citations can read "Pillar 4 · Data Governance (p. 55)" rather than a bare page number.

Citations

• Render the cited page as an image next to the answer. Because the source is already page images, this is unusually easy and very convincing in a government demonstration.
• Highlight the retrieved passage within the page image.

Conversation memory

• The app currently passes the last two exchanges. For longer sessions, add query rewriting: use the LLM to turn "what about the second one?" into a standalone question before retrieval, since follow-up pronouns retrieve badly.

Multilingual support (English / Urdu / Pashto)

• Swap the embedding model for a multilingual one such as paraphrase-multilingual-MiniLM-L12-v2 or BAAI/bge-m3, so an Urdu question can retrieve English source text directly. This single change matters more than anything else on this list.
• Add +urd to OCR_LANGS in ingest.py after installing tesseract-ocr-urd. Pashto OCR is weak in Tesseract; for Pashto, translating the query into English before retrieval works better than OCRing Pashto text.
• Add a language selector so answers can be forced into Urdu or Pashto regardless of the input language — useful for briefing mixed audiences.

Evaluation

• Build a small fixed set of question/expected-page pairs and measure retrieval recall@5 after any change. Without this you cannot tell whether a "better" model actually helped.

---

Limitations, stated plainly

• Answers are only as good as the OCR. Tables in the roadmap's implementation plans OCR imperfectly, so timelines and KPIs should be verified against the page before being quoted in any official document.
• The assistant covers one document. It knows nothing about subsequent notifications, amendments or progress reports.
• The document is described as a living strategy subject to review at least every three years, so rebuild the index when a revised version is published.