project_report.md

# NEET Biology Question & Answer Generation System — Project Report

## 1. Project Objective

Build an **end-to-end AI-powered Question Answering system** for NEET Biology that:

1. Ingests NCERT Class 11 & 12 Biology textbooks (PDFs)
2. Chunks, embeds, and stores them in a **vector database**
3. Uses a **Retrieval-Augmented Generation (RAG)** pipeline to answer student questions grounded in NCERT content
4. Fine-tunes a **BERT classifier** to predict which chapter a question belongs to — enabling targeted retrieval and routing
5. Exposes the pipeline via a **FastAPI backend** (planned)

The system is designed for NEET exam preparation, where every answer must be traceable back to specific NCERT textbook content.

---

## 2. High-Level Architecture

```
┌──────────────────────────────────────────────────────┐
│                    USER QUESTION                     │
└──────────────┬───────────────────────────────────────┘
               │
               ▼
┌──────────────────────────┐
│  BERT Chapter Classifier │  ← Classification/ (fine-tuned bert-base-uncased)
│  Predicts: chapter_id    │
└──────────┬───────────────┘
           │ chapter filter (planned integration)
           ▼
┌──────────────────────────┐     ┌─────────────────────────────┐
│   RAG Pipeline (rag/)    │────▶│  Supabase + pgvector        │
│  1. Embed question       │     │  Table: ncert_knowledge_    │
│  2. Retrieve top-k chunks│◀────│         chunks              │
│  3. Build prompt         │     │  RPC: match_ncert_chunks()  │
│  4. Generate answer (LLM)│     └─────────────────────────────┘
│  5. Justify + Verify     │
│  6. Confidence score     │
└──────────┬───────────────┘
           │
           ▼
┌──────────────────────────┐
│   FastAPI Backend (TBD)  │  ← /qa endpoint, /ingest, /health
└──────────────────────────┘
```

---

## 3. Complete Tech Stack

| Layer | Technology | Purpose |
|---|---|---|
| **Language** | Python 3.10+ | Core runtime |
| **PDF Extraction** | PyMuPDF 1.23.8 | Extract text from NCERT PDFs |
| **Chunking** | Custom ([clean_pdf_to_json.py](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/pdf_to_embedding/clean_pdf_to_json.py)) | Section-aware chunking with metadata |
| **Embedding Model** | `BAAI/bge-small-en-v1.5` (384-dim) | Local embedding generation via `sentence-transformers` |
| **Vector Database** | Supabase + pgvector | Cloud-hosted PostgreSQL with vector similarity search |
| **Vector Index** | HNSW (via pgvector) | Fast approximate nearest-neighbor search |
| **LLM Provider** | Groq API | Free, fast inference (llama-3.3-70b-versatile) |
| **Classification** | HuggingFace `bert-base-uncased` | Fine-tuned for chapter prediction |
| **Training Framework** | HuggingFace `Trainer` + PyTorch | BERT fine-tuning with early stopping |
| **Backend (planned)** | FastAPI + Uvicorn | REST API serving |
| **Data Validation** | Pydantic | Request/response schemas |
| **Environment** | python-dotenv | Secrets management via [.env](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/pdf_to_embedding/.env) |
| **Dependencies** | pip + [requirements.txt](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/requirements.txt) | All versions pinned for reproducibility |

---

## 4. Repository Structure

```
Question_AndAnswerGeneration/
├── .env                          # (gitignored) API keys
├── .gitignore                    # .venv, .env, __pycache__, *.pyc
├── requirements.txt              # All dependencies pinned
├── taxonomy_full.txt             # Complete NCERT syllabus taxonomy
│
├── pdf_to_embedding/             # 🟢 PHASE 1: Data Pipeline
│   ├── .env / .env.example       # Supabase + Groq credentials
│   ├── clean_pdf_to_json.py      # PDF → section-aware JSON chunks
│   ├── embed_and_upload.py       # JSON → embeddings → Supabase upsert
│   ├── supabase_schema.sql       # Table + RPC + HNSW index DDL
│   └── json_ouput/               # Intermediate JSON files
│       ├── 11th Ncert output/    #   19 files (Kebo101–Kebo119)
│       └── 12th Ncert Output/    #   13 files (Lebo101–Lebo113)
│
├── rag/                          # 🟢 PHASE 2: RAG Pipeline
│   ├── __init__.py
│   ├── retriever.py              # Embed question → Supabase RPC → top-k chunks
│   ├── prompt_builder.py         # Build answer/justification/verifier prompts
│   ├── generator.py              # Groq LLM calls (answer + justify + verify)
│   ├── confidence.py             # Confidence scorer (retrieval + verification)
│   └── pipeline.py               # Orchestrator with smoke test (--debug flag)
│
└── Classification/               # 🟡 PHASE 3: BERT Classifier (scaffold ready)
    ├── __init__.py
    ├── config.json               # All hyperparameters (model, lr, epochs, etc.)
    ├── dataset.py                # PyTorch Dataset + label_map builder
    ├── train.py                  # Full HuggingFace Trainer script
    ├── predict.py                # ClassifierPredictor inference wrapper
    └── data/
        ├── sample_questions.json # Placeholder labeled data (3 examples)
        └── label_map.json        # (auto-generated) chapter_id → chapter_name
```

---

## 5. What Is Completed ✅

### Phase 1 — Data Ingestion Pipeline ✅
| Component | File | Status |
|---|---|---|
| PDF text extraction | [clean_pdf_to_json.py](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/pdf_to_embedding/clean_pdf_to_json.py) | ✅ Complete |
| Section-aware chunking with metadata | [clean_pdf_to_json.py](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/pdf_to_embedding/clean_pdf_to_json.py) | ✅ Complete |
| Supabase schema + RPC + HNSW index | [supabase_schema.sql](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/pdf_to_embedding/supabase_schema.sql) | ✅ Deployed |
| Embedding generation (BGE-small) | [embed_and_upload.py](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/pdf_to_embedding/embed_and_upload.py) | ✅ Complete |
| Batch upsert to Supabase | [embed_and_upload.py](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/pdf_to_embedding/embed_and_upload.py) | ✅ Complete |
| All 32 NCERT chapters processed | `json_ouput/` | ✅ 19 (Class 11) + 13 (Class 12) |

**Data stats:** 32 JSON files → **~696 chunks** with metadata (chapter, section, section_title, chunk_text, embedding vector).

### Phase 2 — RAG Pipeline ✅
| Component | File | What It Does |
|---|---|---|
| Retriever | [retriever.py](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/rag/retriever.py) | Embeds query → calls `match_ncert_chunks` RPC → returns top-k relevant chunks |
| Prompt Builder | [prompt_builder.py](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/rag/prompt_builder.py) | Constructs 3 prompt types: answer, justification, verifier |
| Generator | [generator.py](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/rag/generator.py) | Calls Groq API for answer generation, step-by-step justification, and YES/NO faithfulness verification |
| Confidence Scorer | [confidence.py](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/rag/confidence.py) | Combines retrieval similarity + verifier output into a 0-1 confidence score |
| Pipeline Orchestrator | [pipeline.py](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/rag/pipeline.py) | End-to-end flow: question → retrieve → generate → justify → verify → confidence. Includes `--debug` smoke test |

**RAG design highlights:**
- **Multi-stage pipeline**: Retrieve → Answer → Justify → Verify → Confidence
- **Singleton pattern** for Supabase client, embedding model, and Groq client (no repeated initialization)
- **Similarity threshold** of 0.1 (tuned lower to avoid false negatives with limited data)
- **Groq model**: `llama-3.3-70b-versatile` — fast, free tier, high quality

### Phase 3 — BERT Classification Layer 🟡 (Scaffold Complete)
| Component | File | Status |
|---|---|---|
| Config | [config.json](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/Classification/config.json) | ✅ All hyperparameters defined |
| Dataset class | [dataset.py](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/Classification/dataset.py) | ✅ Tokenizer + label_map builder |
| Training script | [train.py](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/Classification/train.py) | ✅ Full Trainer with early stopping, F1 metric |
| Inference wrapper | [predict.py](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/Classification/predict.py) | ✅ [ClassifierPredictor](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/Classification/predict.py#26-109) with top-k + confidence |
| Labeled training data | `data/` | 🔴 Only 3 sample examples — needs real NEET questions |

**Classifier config:**
```json
{
  "model_name": "bert-base-uncased",
  "max_length": 128,
  "num_labels": 16,
  "batch_size": 16,
  "learning_rate": 2e-5,
  "num_epochs": 10,
  "warmup_ratio": 0.1,
  "eval_strategy": "epoch",
  "metric_for_best_model": "f1",
  "early_stopping_patience": 2
}
```

---

## 6. NCERT Dataset Summary

| Class | Chapters | Files | Total Chunks | Unique Sections |
|---|---|---|---|---|
| 11th | 19 (Ch 1–19) | Kebo101–Kebo119 | ~263 | ~200+ |
| 12th | 12 (Ch 2–13) + 1 front matter | Lebo101–Lebo113 | ~433 | ~150+ |
| **Total** | **32** | **32 JSON files** | **~696** | **~350+** |

> **Note:** Class 12 Chapter 1 (Reproduction in Organisms) is missing — [Lebo101.json](file:///d:/Users/Pranil/Github%20Repos/Question_AndAnswerGeneration/pdf_to_embedding/json_ouput/12th%20Ncert%20Output/Lebo101.json) contains only front matter. Needs re-processing.

---

## 7. Environment Variables

```env
SUPABASE_URL=https://xxxxx.supabase.co
SUPABASE_KEY=eyJhbG...                    # Service Role Key
GROQ_API_KEY=gsk_...                      # Free tier API key
GROQ_MODEL=llama-3.3-70b-versatile        # Default LLM
EMBEDDING_MODEL=BAAI/bge-small-en-v1.5    # 384-dim local embeddings
```

---

## 8. What Remains — Roadmap 🔜

### Immediate (Next Steps)

| # | Task | Priority | Effort |
|---|---|---|---|
| 1 | **Collect & label NEET questions** → map each to chapter_id | 🔴 Critical | Medium |
| 2 | **Run BERT training** (`python -m Classification.train`) | 🔴 Critical | Low (code ready) |
| 3 | **Evaluate classifier** — target >85% F1 on held-out set | 🔴 Critical | Low |
| 4 | **Fix Lebo101.json** — re-process Ch 1 Reproduction in Organisms PDF | 🟡 Medium | Low |

### Short-Term (Integration)

| # | Task | Priority |
|---|---|---|
| 5 | **Integrate classifier into RAG** — use predicted chapter to filter retrieval (pre-filter Supabase query by chapter metadata) | 🔴 Critical |
| 6 | **Build FastAPI backend** — `/qa` endpoint wrapping `pipeline.run()`, `/health`, `/ingest` | 🟡 Medium |
| 7 | **Add syllabus_taxonomy.json** — structured JSON of all chapters/sections for the classifier and frontend | 🟡 Medium |

### Medium-Term (Production Readiness)

| # | Task | Priority |
|---|---|---|
| 8 | Add unit + integration tests for RAG pipeline and classifier | 🟡 Medium |
| 9 | Build frontend UI (React / Streamlit) for question input + answer display | 🟢 Nice-to-have |
| 10 | Add question generation capability (generate practice MCQs from NCERT content) | 🟢 Future |
| 11 | Deploy to cloud (Render / Railway / Docker) | 🟢 Future |

---

## 9. Key Design Decisions

| Decision | Rationale |
|---|---|
| **BGE-small-en-v1.5** over larger models | 384-dim is fast and good enough for NCERT-level text; runs locally without GPU |
| **Supabase + pgvector** over Pinecone/Weaviate | Free tier, PostgreSQL ecosystem, full SQL alongside vectors |
| **Groq API** over OpenAI/Anthropic | Free tier with fast inference; `llama-3.3-70b` is competitive quality |
| **bert-base-uncased** for classification | Lightweight, well-studied for text classification; fine-tunes on a single GPU |
| **Section-aware chunking** | Each chunk carries chapter/section metadata → enables filtered retrieval |
| **Multi-stage RAG** (answer → justify → verify) | Reduces hallucination by adding a verification layer before presenting answers |
| **Singleton pattern** for clients | Avoids expensive re-initialization of embedding model, Supabase client, Groq client |

---

## 10. How to Run (Quick Reference)

```bash
# 1. Activate virtual environment
.\.venv\Scripts\activate

# 2. Process a new PDF
python pdf_to_embedding/clean_pdf_to_json.py --input "path/to/chapter.pdf"

# 3. Upload embeddings to Supabase
python pdf_to_embedding/embed_and_upload.py

# 4. Test RAG pipeline (smoke test)
python -m rag.pipeline --debug

# 5. Train BERT classifier (when data is ready)
python -m Classification.train

# 6. Test classifier predictions
python -m Classification.predict
```