DigitalKPAssistant / README.md
Sumayyea's picture
Upload 7 files
269c0ee verified
|
Raw History Blame Contribute Delete
9.62 kB
---
title: Digital KP 2030 Assistant
emoji: πŸ“˜
colorFrom: blue
colorTo: indigo
sdk: streamlit
sdk_version: 1.40.2
app_file: app.py
pinned: false
---
# Digital KP 2030 Assistant
A retrieval-augmented (RAG) chatbot that answers questions about the **Khyber
Pakhtunkhwa Digital Transformation Policy & Roadmap 2030**, citing the page
number behind every claim and refusing to answer when the document does not
cover the question.
---
## Why this is built the way it is
The published Digital KP 2030 file is a **design-heavy, image-based document**.
Its embedded text layer is nearly empty β€” most of the 50 roadmap pages yield
only 90–120 characters (running headers), and several pages return mojibake
from custom font encodings. Across all 93 pages there are roughly 18,800
extractable characters in total.
A conventional `PyPDF β†’ chunk β†’ embed` pipeline therefore indexes almost
nothing, and the resulting assistant answers confidently from thin air or
fails outright. **OCR is not optional for this document.**
Two design decisions follow:
1. **`ingest.py` extracts text first and falls back to OCR** for any page whose
text layer is too thin or too corrupted. It also detects the case where the
`.pdf` is really a ZIP archive of page images, which some distributions are.
2. **Indexing happens offline, not in the Space.** You run `ingest.py` once on
your own machine and commit the resulting `index/` folder. The Space then
starts in seconds, needs no OCR binaries at runtime, and behaves
deterministically.
---
## Architecture
```
PDF (image-based)
β”‚
β–Ό
ingest.py ── text extraction ──► thin or garbled? ──► OCR (tesseract)
β”‚
β–Ό
chunking 1,100 chars, 180-char overlap, page number kept per chunk
β”‚
β–Ό
embeddings sentence-transformers/all-MiniLM-L6-v2 (local, free)
β”‚
β–Ό
vector store FAISS IndexFlatIP over normalised vectors (= cosine)
β”‚
─────┴───── index/ committed to the repo ─────
β”‚
β–Ό
app.py / rag.py
β”‚
β”œβ”€ retrieve top-6 chunks, drop anything below a relevance floor
β”œβ”€ build a prompt containing ONLY those page-tagged excerpts
β”œβ”€ LLM generates the answer under strict grounding rules
└─ return answer + expandable source passages with page numbers
```
Grounding is enforced in three places rather than one: a relevance floor that
discards weak matches, a system prompt that forbids outside knowledge and
mandates page citations, and a UI that always shows the retrieved passages so a
user can check the answer themselves.
---
## Project structure
```
digital-kp-assistant/
β”œβ”€β”€ app.py Streamlit chat interface
β”œβ”€β”€ rag.py retrieval, prompting, LLM call
β”œβ”€β”€ ingest.py run once locally to build the index
β”œβ”€β”€ requirements.txt
β”œβ”€β”€ packages.txt apt packages (only needed if you ever OCR in the Space)
β”œβ”€β”€ .gitignore
β”œβ”€β”€ data/
β”‚ └── digital_kp_2030.pdf
└── index/ generated by ingest.py β€” commit this
β”œβ”€β”€ faiss.index
β”œβ”€β”€ chunks.json
└── meta.json
```
---
## Setup
### 1. Add the PDF
Download the document from the Khyber Pakhtunkhwa Digital Transformation Board
site and save it as `data/digital_kp_2030.pdf`. The filename matters β€” it is
what `ingest.py` looks for.
### 2. Install and build the index (local machine)
OCR needs two system tools in addition to the Python packages:
```bash
# Ubuntu / Debian
sudo apt install poppler-utils tesseract-ocr
# macOS
brew install poppler tesseract
# Windows: install Tesseract and Poppler, then add both to PATH
```
```bash
pip install -r requirements.txt
python ingest.py
```
Expect this to take several minutes β€” OCR at 220 DPI across 93 pages is the
slow part. You should end with roughly 400–700 chunks. If you get fewer than
100, OCR is not running; check that `tesseract` is on your PATH.
### 3. Run locally
```bash
export LLM_API_KEY="your-key"
streamlit run app.py
```
### 4. Deploy to Hugging Face Spaces
1. Create a new Space: **SDK = Streamlit**, hardware **CPU basic** is enough.
2. Push the repository, **including `index/`** (it is small β€” a few MB).
```bash
git remote add space https://huggingface.co/spaces/<username>/digital-kp-assistant
git push space main
```
If `index/` is larger than 10 MB, track it with Git LFS first:
`git lfs track "index/*.index"`.
3. Add the secrets below under **Settings β†’ Variables and secrets**.
4. The Space rebuilds and starts automatically.
---
## Secrets and environment variables
Set these as **Secrets** (not Variables) in the Space settings:
| Name | Required | Default | Notes |
|---|---|---|---|
| `LLM_API_KEY` | yes | β€” | Your provider key. Never commit it. |
| `LLM_BASE_URL` | no | `https://api.groq.com/openai/v1` | Any OpenAI-compatible endpoint. |
| `LLM_MODEL` | no | `llama-3.3-70b-versatile` | Must exist on your provider. |
Because the client is OpenAI-compatible, you can switch providers by changing
two values and nothing else:
| Provider | `LLM_BASE_URL` | Example `LLM_MODEL` |
|---|---|---|
| Groq (free tier, fast) | `https://api.groq.com/openai/v1` | `llama-3.3-70b-versatile` |
| OpenAI | `https://api.openai.com/v1` | `gpt-4o-mini` |
| OpenRouter | `https://openrouter.ai/api/v1` | `anthropic/claude-3.5-sonnet` |
| Together | `https://api.together.xyz/v1` | `meta-llama/Llama-3.3-70B-Instruct-Turbo` |
Embeddings run locally via `sentence-transformers`, so they cost nothing and
need no key.
---
## Test questions
Use these to check the system before demonstrating it. The first group should
produce confident, cited answers; the last group should be refused.
**Should answer well**
1. What is the vision of Digital KP 2030?
2. Explain the DIGIT 2030 framework.
3. What are the nine strategic goals?
4. Who chairs the KP Digital Council, and how often does it meet?
5. How many interventions sit under each strategic pillar?
6. What are the 2030 KPIs for digital payments?
7. What is DASTAK and how does it relate to Khyber Pass?
8. What does the roadmap say about AI governance and ethics?
9. What is expected of a Departmental Implementation Forum?
10. What targets exist for upskilling government officials?
**Should be refused β€” these are not in the document**
11. What is the budget allocation for Digital KP 2030 in rupees?
12. How does this compare to the Punjab digital strategy?
13. Who won the contract to build DASTAK?
**Should surface a contradiction rather than pick a side**
14. How many strategic pillars are there? *(p. 24 prints "05"; p. 25 states six.)*
15. How many interventions are there in total? *(p. 13 "more than 900"; p. 24
"815+"; p. 27 gives 924, which sums correctly from its own table.)*
Questions 14 and 15 are the most instructive demonstration in the whole
project β€” they show why page-level citation matters, and they are a genuine
property of the source document rather than a model failure.
---
## Improving it later
**Retrieval quality**
- Add hybrid search: combine the current dense vectors with BM25 keyword
matching, which handles exact terms like "DASTAK 2.0" or "Q3 2026" far better
than embeddings do.
- Add a reranker (`cross-encoder/ms-marco-MiniLM-L-6-v2`) over the top 20 hits
to lift precision before the LLM sees anything.
- Store the pillar and thematic-domain name on each chunk during ingestion so
users can filter by pillar, and so citations can read "Pillar 4 Β· Data
Governance (p. 55)" rather than a bare page number.
**Citations**
- Render the cited page as an image next to the answer. Because the source is
already page images, this is unusually easy and very convincing in a
government demonstration.
- Highlight the retrieved passage within the page image.
**Conversation memory**
- The app currently passes the last two exchanges. For longer sessions, add
query rewriting: use the LLM to turn "what about the second one?" into a
standalone question before retrieval, since follow-up pronouns retrieve badly.
**Multilingual support (English / Urdu / Pashto)**
- Swap the embedding model for a multilingual one such as
`paraphrase-multilingual-MiniLM-L12-v2` or `BAAI/bge-m3`, so an Urdu question
can retrieve English source text directly. This single change matters more
than anything else on this list.
- Add `+urd` to `OCR_LANGS` in `ingest.py` after installing
`tesseract-ocr-urd`. Pashto OCR is weak in Tesseract; for Pashto, translating
the query into English before retrieval works better than OCRing Pashto text.
- Add a language selector so answers can be forced into Urdu or Pashto
regardless of the input language β€” useful for briefing mixed audiences.
**Evaluation**
- Build a small fixed set of question/expected-page pairs and measure
retrieval recall@5 after any change. Without this you cannot tell whether a
"better" model actually helped.
---
## Limitations, stated plainly
- Answers are only as good as the OCR. Tables in the roadmap's implementation
plans OCR imperfectly, so timelines and KPIs should be verified against the
page before being quoted in any official document.
- The assistant covers one document. It knows nothing about subsequent
notifications, amendments or progress reports.
- The document is described as a living strategy subject to review at least
every three years, so rebuild the index when a revised version is published.