Islam Mamedov commited on
Commit
b7c8476
·
1 Parent(s): e5e7d99

Rewrite README with complete project documentation

Browse files
Files changed (1) hide show
  1. README.md +673 -124
README.md CHANGED
@@ -8,146 +8,695 @@ app_file: app.py
8
  pinned: false
9
  ---
10
 
11
- # ⚡ FastAPI Codebase Q&A — an eval-driven RAG system
12
 
13
- Ask questions about the [FastAPI](https://github.com/fastapi/fastapi)
14
- codebase and get answers grounded in its actual source code, documentation,
15
- and GitHub issues — with citations linking to the exact lines, and honest
16
- refusals when the corpus doesn't contain the answer.
17
 
18
- **Live demo:** https://huggingface.co/spaces/islam-mamedov/fastapi-codebase-qa
19
 
20
- Every architectural decision in this project was made by measuring, not
21
- guessing: a hand-labeled 42-question eval set gates every change, and the
22
- ablation results below include the things that *didn't* work and why.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
23
 
24
  ## Architecture
25
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
  ```
27
- GitHub repo ──► Ingestion ──► Chunking ──► Indexing ──► Retrieval ──► Generation
28
- code clone AST-aware ChromaDB dense LLM with
29
- docs (en) API pull (tree-sitter) bge-small (winner of citations +
30
- issues resumable md-aware + BM25 6 ablations) refusal rules
31
- issue-aware
32
- │ │
33
- └────── Eval harness ◄─────┘
34
- recall@5 · MRR · faithfulness ·
35
- correctness · refusal accuracy
36
- ```
37
-
38
- - **Corpus:** 46 source files, 161 English docs, 175 closed issues →
39
- **1,352 chunks** (363 code, 814 doc, 175 issue)
40
- - **Chunking:** tree-sitter AST parsing — one chunk per function/class,
41
- classes >150 lines split per-method, every chunk carries a context header
42
- (`# File: fastapi/routing.py | Symbol: APIRouter`) and exact line numbers
43
- for citations
44
- - **Stack:** Python, ChromaDB, sentence-transformers (`bge-small-en-v1.5`),
45
- rank-bm25, Groq (LLM), Streamlit — total cost: $0
46
-
47
- ## Evaluation
48
-
49
- 42 hand-labeled questions across four categories: **api_usage** (16),
50
- **behavior** (11), **location** ("where is X defined", 8), and
51
- **unanswerable** (7 — questions FastAPI's corpus genuinely can't answer,
52
- to measure refusal). Gold labels mark which file/symbol contains the
53
- answer; retrieval is scored by recall@5 and MRR, generation by an
54
- LLM-as-judge (faithfulness, correctness) plus string-matched refusals.
55
-
56
- ### Retrieval ablations
57
-
58
- | Configuration | recall@5 | MRR | Verdict |
59
- |---|---|---|---|
60
- | **dense (bge-small)** | **0.91** | **0.71** | **winner — shipped** |
61
- | hybrid (dense + BM25, RRF) | 0.86 | 0.70 | BM25 noise displaces gold chunks |
62
- | hybrid, weighted RRF 2:1 | 0.86 | 0.71 | weighting doesn't recover recall |
63
- | dense + bge-reranker-base | 0.91 | 0.63 | reranker *hurts* ranking |
64
- | dense + bge-reranker-v2-m3 | 0.83 | 0.67 | newer reranker also hurts |
65
- | hybrid + bge-reranker-v2-m3 | 0.86 | 0.68 | — |
66
- | dense + LLM query rewriting | 0.91 | 0.69 | no gain; noisy expansions add drift |
67
-
68
- **Why hybrid lost:** diagnostic runs showed dense@20 = hybrid@20 = 0.97 —
69
- BM25 contributed zero unique gold candidates on this corpus, while its
70
- common-token matches (issue chunks full of "request", "body", "json")
71
- crowded correct chunks out of the top 5.
72
-
73
- **Why the rerankers lost:** both generations of cross-encoder demoted raw
74
- code chunks on "where is X defined" questions. A code chunk *is* the
75
- answer without *discussing* the answer — and rerankers are trained to
76
- score passages that discuss. dense@20 recall of 0.97 vs reranked@5 recall
77
- of 0.83–0.86 shows they were given the right candidates and ranked them
78
- worse.
79
-
80
- ### Generation (dense, llama-4-scout via Groq)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
81
 
82
  | Metric | Score |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
83
  |---|---|
84
- | Faithfulness (LLM judge) | 0.89 |
85
- | Correctness (LLM judge) | 0.91 |
86
- | Refusal accuracy (7 unanswerable questions) | 7/7 |
87
-
88
- Refusal was the most fragile metric: it swung 6/7 → 0/7 → 7/7 across three
89
- system-prompt variants. The fix was naming the failure mode in the prompt
90
- ("the context chunks are search results: they may be loosely RELATED to
91
- the question without actually answering it"). Without the eval harness
92
- this regression would have shipped silently.
93
-
94
- A score-threshold refusal gate was also tested and **rejected**: top-1
95
- similarity scores for answerable (0.74–0.89) and unanswerable (0.72–0.79)
96
- questions overlap heavily — semantic similarity can't distinguish "related
97
- to FastAPI" from "answerable from FastAPI's corpus".
98
-
99
- ## Failure analysis (what still breaks)
100
-
101
- 1. **Common-token queries.** "Where are the Query, Path, and Body parameter
102
- functions defined?" (`fastapi/param_functions.py`) misses even at k=20
103
- in every configuration — all its query tokens are ubiquitous. LLM query
104
- rewriting that injects likely identifiers is the targeted fix.
105
- 2. **Docs without code.** FastAPI's docs embed examples via
106
- `{* ../../docs_src/... *}` include-directives; the referenced snippets
107
- aren't inlined into chunks, so some answers describe code they can't
108
- show. Resolving the includes at chunking time is the top corpus
109
- improvement.
110
- 3. **Judge strictness.** Spot-checking answers flagged `[incorrect]` showed
111
- at least one (JWT auth) is actually correct — the LLM judge penalizes
112
- answers referencing code the truncated context implies but doesn't
113
- show. Judge scores are treated as directional, not absolute.
114
-
115
- ## Engineering notes
116
-
117
- - **Idempotent, resumable ingestion** — issues already fetched are skipped,
118
- so network failures mid-run just need a re-run
119
- - **Everything cached** — LLM answers, judgments, and query rewrites are
120
- cached to disk keyed by (model, prompt), making eval re-runs free on
121
- Groq's rate-limited free tier
122
- - **Rate-limit resilience** — retry with backoff; eval survived TPM/TPD
123
- limits and a mid-run model deprecation (Groq retired llama-3.3 mid-sprint)
124
- - **Self-building deploy** — the HF Space rebuilds the vector index from
125
- committed `chunks.jsonl` on first boot; no binary artifacts in git
126
- - **Unit-tested chunkers** and an eval suite that doubles as a regression
127
- gate
128
-
129
- ## Run it yourself
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
130
 
131
  ```bash
132
- git clone https://github.com/islam-mamedov/codebase-rag && cd codebase-rag
133
- python -m venv .venv && source .venv/bin/activate
134
- pip install -r requirements.txt PyGithub tree-sitter tree-sitter-python pytest
135
- echo "GROQ_API_KEY=gsk_..." > .env # free key: console.groq.com
136
 
137
- python src/ingest.py --repo fastapi/fastapi # needs GITHUB_TOKEN for issues
 
 
138
  python src/chunk.py
 
 
 
 
 
139
  python src/index.py
 
 
 
 
 
140
  python src/ask.py "How do I return a custom status code?"
141
- python src/eval.py --mode dense --answers # reproduce the numbers above
142
- streamlit run app.py
143
  ```
144
 
145
- ## What I'd build next
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
146
 
147
- - Resolve `docs_src` include-directives into doc chunks (highest-impact
148
- corpus fix)
149
- - Make ingestion repo-agnostic (`--repo any/repo`) — turn the project into
150
- a tool
151
- - Embedding-model ablation (bge-small vs bge-base vs code-specific models)
152
- - Ingest GitHub Discussions, where most FastAPI Q&A actually lives
153
- - CI job that fails the build if recall@5 drops below baseline
 
8
  pinned: false
9
  ---
10
 
11
+ <div align="center">
12
 
13
+ # ⚡ FastAPI Codebase Q&A
 
 
 
14
 
15
+ ### An evaluation-driven RAG system for exploring a real production codebase
16
 
17
+ Ask technical questions about FastAPI and receive answers grounded in its source code, documentation, and resolved GitHub issues.
18
+
19
+ [![Live Demo](https://img.shields.io/badge/Live_Demo-Hugging_Face-FFD21E?style=for-the-badge&logo=huggingface&logoColor=black)](https://huggingface.co/spaces/islam-mamedov/fastapi-codebase-qa)
20
+ [![GitHub](https://img.shields.io/badge/Source_Code-GitHub-181717?style=for-the-badge&logo=github)](https://github.com/islam-mamedov/codebase-rag)
21
+ [![Python](https://img.shields.io/badge/Python-3.10%2B-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/)
22
+ [![License](https://img.shields.io/badge/License-MIT-green?style=for-the-badge)](LICENSE)
23
+
24
+ </div>
25
+
26
+ ---
27
+
28
+ ## Overview
29
+
30
+ FastAPI Codebase Q&A is a Retrieval-Augmented Generation system designed to answer questions about the FastAPI repository.
31
+
32
+ Instead of relying only on an LLM's general knowledge, the system searches a custom knowledge base built from:
33
+
34
+ - FastAPI source code
35
+ - English documentation
36
+ - Closed GitHub issues
37
+
38
+ The retrieved evidence is passed to the language model, which generates an answer with references to the relevant files, symbols, and line numbers.
39
+
40
+ The project was built around one principle:
41
+
42
+ > Every important retrieval decision should be measured rather than guessed.
43
+
44
+ A hand-labelled evaluation set was used to compare dense retrieval, hybrid search, reranking, and query rewriting before selecting the final production configuration.
45
+
46
+ ## Live Demo
47
+
48
+ Try the deployed application:
49
+
50
+ **[Open FastAPI Codebase Q&A on Hugging Face Spaces](https://huggingface.co/spaces/islam-mamedov/fastapi-codebase-qa)**
51
+
52
+ Example questions:
53
+
54
+ ```text
55
+ How do I return a custom status code in FastAPI?
56
+ ```
57
+
58
+ ```text
59
+ Where is APIRouter defined?
60
+ ```
61
+
62
+ ```text
63
+ How does FastAPI validate request bodies?
64
+ ```
65
+
66
+ ```text
67
+ Can FastAPI automatically deploy my application to AWS?
68
+ ```
69
+
70
+ The final example tests the system's refusal behaviour. When the indexed corpus does not contain enough evidence, the application should say so instead of inventing an answer.
71
+
72
+ ---
73
+
74
+ ## What the System Does
75
+
76
+ The application follows a complete RAG workflow:
77
+
78
+ 1. Ingests content from the FastAPI repository.
79
+ 2. Splits code, documentation, and issues using content-aware chunking.
80
+ 3. Converts chunks into vector embeddings.
81
+ 4. Stores and retrieves relevant chunks using ChromaDB.
82
+ 5. Passes the retrieved evidence to an LLM.
83
+ 6. Generates a grounded answer with citations.
84
+ 7. Refuses questions that cannot be answered from the available evidence.
85
+ 8. Evaluates retrieval and answer quality using a labelled benchmark.
86
+
87
+ ---
88
 
89
  ## Architecture
90
 
91
+ ```text
92
+ ┌──────────────────────────────────────────────────────────────────────┐
93
+ │ FastAPI Repository │
94
+ │ │
95
+ │ Source Code · Documentation · GitHub Issues │
96
+ └───────────────────────────────┬──────────────────────────────────────┘
97
+ │
98
+ ▼
99
+ ┌──────────────────────────────────────────────────────────────────────┐
100
+ │ Ingestion │
101
+ │ │
102
+ │ Repository cloning · GitHub API · Resumable issue fetching │
103
+ └───────────────────────────────┬──────────────────────────────────────┘
104
+ │
105
+ ▼
106
+ ┌───────────────────────────────���──────────────────────────────────────┐
107
+ │ Chunking │
108
+ │ │
109
+ │ AST-aware code · Markdown-aware docs · Issue-aware chunks │
110
+ └───────────────────────────────┬──────────────────────────────────────┘
111
+ │
112
+ ▼
113
+ ┌──────────────────────────────────────────────────────────────────────┐
114
+ │ Indexing │
115
+ │ │
116
+ │ BGE embeddings · ChromaDB vector store · Chunk metadata │
117
+ └───────────────────────────────┬──────────────────────────────────────┘
118
+ │
119
+ ▼
120
+ ┌──────────────────────────────────────────────────────────────────────┐
121
+ │ Retrieval │
122
+ │ │
123
+ │ Dense semantic search selected through evaluation │
124
+ └───────────────────────────────┬──────────────────────────────────────┘
125
+ │
126
+ ▼
127
+ ┌──────────────────────────────────────────────────────────────────────┐
128
+ │ Generation │
129
+ │ │
130
+ │ Groq LLM · Grounded answers · Citations · Refusal rules │
131
+ └───────────────────────────────┬──────────────────────────────────────┘
132
+ │
133
+ ▼
134
+ ┌──────────────────────────────────────────────────────────────────────┐
135
+ │ Evaluation │
136
+ │ │
137
+ │ Recall@5 · MRR · Faithfulness · Correctness · Refusal accuracy │
138
+ └──────────────────────────────────────────────────────────────────────┘
139
  ```
140
+
141
+ ---
142
+
143
+ ## Indexed Corpus
144
+
145
+ The knowledge base contains:
146
+
147
+ | Source | Quantity |
148
+ |---|---:|
149
+ | FastAPI source files | 46 |
150
+ | English documentation files | 161 |
151
+ | Closed GitHub issues | 175 |
152
+ | Total chunks | 1,352 |
153
+
154
+ Chunk distribution:
155
+
156
+ | Chunk type | Quantity |
157
+ |---|---:|
158
+ | Code chunks | 363 |
159
+ | Documentation chunks | 814 |
160
+ | Issue chunks | 175 |
161
+
162
+ Every chunk includes metadata such as:
163
+
164
+ - File path
165
+ - Content type
166
+ - Symbol name
167
+ - Start and end lines
168
+ - Repository information
169
+ - Context header
170
+
171
+ Example context header:
172
+
173
+ ```text
174
+ # File: fastapi/routing.py | Symbol: APIRouter
175
+ ```
176
+
177
+ This metadata allows the application to produce answers that point back to the relevant part of the repository.
178
+
179
+ ---
180
+
181
+ ## Content-Aware Chunking
182
+
183
+ Different content types require different chunking strategies.
184
+
185
+ ### Source Code
186
+
187
+ Python files are parsed using Tree-sitter.
188
+
189
+ The chunker creates separate chunks for:
190
+
191
+ - Functions
192
+ - Classes
193
+ - Methods
194
+ - Module-level content
195
+
196
+ Large classes are split by method when necessary instead of being stored as one oversized block.
197
+
198
+ This preserves the structure of the code and improves retrieval for questions such as:
199
+
200
+ ```text
201
+ Where is the APIRouter class defined?
202
+ ```
203
+
204
+ ### Documentation
205
+
206
+ Markdown documents are divided using heading and section boundaries.
207
+
208
+ This avoids splitting related explanations in the middle of a section and preserves useful context.
209
+
210
+ ### GitHub Issues
211
+
212
+ Issues are stored with their:
213
+
214
+ - Title
215
+ - Body
216
+ - Resolution context
217
+ - Relevant metadata
218
+
219
+ Issue-aware chunking helps answer behavioural questions that may not be explained clearly in the source code or documentation.
220
+
221
+ ---
222
+
223
+ ## Evaluation Dataset
224
+
225
+ The system is evaluated using 42 manually labelled questions.
226
+
227
+ | Category | Questions |
228
+ |---|---:|
229
+ | API usage | 16 |
230
+ | Behaviour | 11 |
231
+ | Code location | 8 |
232
+ | Unanswerable | 7 |
233
+ | **Total** | **42** |
234
+
235
+ Each answerable question includes a gold reference identifying the file or symbol expected to contain the answer.
236
+
237
+ The evaluation measures two separate parts of the system.
238
+
239
+ ### Retrieval Metrics
240
+
241
+ - **Recall@5:** whether a correct chunk appears within the first five retrieved results
242
+ - **MRR:** how highly the first correct result is ranked
243
+
244
+ ### Generation Metrics
245
+
246
+ - **Faithfulness:** whether the answer is supported by the retrieved context
247
+ - **Correctness:** whether the response answers the question accurately
248
+ - **Refusal accuracy:** whether unsupported questions are rejected correctly
249
+
250
+ ---
251
+
252
+ ## Retrieval Experiments
253
+
254
+ Several retrieval configurations were tested before selecting the production approach.
255
+
256
+ | Configuration | Recall@5 | MRR | Result |
257
+ |---|---:|---:|---|
258
+ | **Dense retrieval — BGE Small** | **0.91** | **0.71** | **Selected for production** |
259
+ | Hybrid dense + BM25 using RRF | 0.86 | 0.70 | Lower recall |
260
+ | Hybrid weighted RRF at 2:1 | 0.86 | 0.71 | No recall improvement |
261
+ | Dense + BGE reranker base | 0.91 | 0.63 | Ranking became worse |
262
+ | Dense + BGE reranker v2-m3 | 0.83 | 0.67 | Lower recall and MRR |
263
+ | Hybrid + BGE reranker v2-m3 | 0.86 | 0.68 | No improvement |
264
+ | Dense + LLM query rewriting | 0.91 | 0.69 | No measurable gain |
265
+
266
+ ### Why Dense Retrieval Won
267
+
268
+ Dense retrieval produced the best overall combination of recall and ranking quality.
269
+
270
+ Diagnostic experiments showed:
271
+
272
+ ```text
273
+ Dense recall@20: 0.97
274
+ Hybrid recall@20: 0.97
275
+ ```
276
+
277
+ BM25 did not introduce additional correct candidates. Instead, common words such as `request`, `body`, and `json` caused unrelated GitHub issue chunks to move above more useful code and documentation chunks.
278
+
279
+ Because hybrid retrieval added noise without adding new correct results, the simpler dense configuration was selected.
280
+
281
+ ### Why Reranking Was Rejected
282
+
283
+ Both rerankers frequently pushed raw code chunks lower in the results.
284
+
285
+ This especially affected code-location questions.
286
+
287
+ A source-code definition may directly contain the answer without explaining it in natural language. Cross-encoders often prefer passages that discuss a topic rather than code that implements it.
288
+
289
+ The retrievers were already finding strong candidates at `k=20`, but the rerankers were reordering them incorrectly.
290
+
291
+ ### Why Query Rewriting Was Rejected
292
+
293
+ LLM query rewriting produced:
294
+
295
+ ```text
296
+ Recall@5: 0.91
297
+ MRR: 0.69
298
+ ```
299
+
300
+ It did not improve recall over the original dense query and slightly reduced ranking quality.
301
+
302
+ Some rewritten queries added useful-looking but incorrect identifiers, creating semantic drift. The feature was therefore not included in the deployed retrieval pipeline.
303
+
304
+ ---
305
+
306
+ ## Generation Results
307
+
308
+ The reported generation evaluation used dense retrieval with a Groq-hosted LLM.
309
 
310
  | Metric | Score |
311
+ |---|---:|
312
+ | Faithfulness | 0.89 |
313
+ | Correctness | 0.91 |
314
+ | Refusal accuracy | 7/7 |
315
+
316
+ ### Refusal Behaviour
317
+
318
+ Refusal accuracy was one of the most sensitive parts of the project.
319
+
320
+ Across three prompt versions, the result changed from:
321
+
322
+ ```text
323
+ 6/7 → 0/7 → 7/7
324
+ ```
325
+
326
+ The main problem was that retrieved chunks could be related to the question without actually containing enough information to answer it.
327
+
328
+ The final prompt explicitly explains this distinction to the model:
329
+
330
+ ```text
331
+ The context chunks are search results. They may be loosely related to the
332
+ question without actually containing the answer.
333
+ ```
334
+
335
+ This small prompt change prevented the model from treating all retrieved content as valid evidence.
336
+
337
+ ### Why a Similarity Threshold Was Not Used
338
+
339
+ A retrieval-score threshold was tested as an alternative refusal mechanism.
340
+
341
+ However, the score ranges overlapped:
342
+
343
+ ```text
344
+ Answerable questions: 0.74–0.89
345
+ Unanswerable questions: 0.72–0.79
346
+ ```
347
+
348
+ A high similarity score only shows that a chunk is related to the question. It does not prove that the chunk contains the answer.
349
+
350
+ For this reason, refusal is handled through evidence-aware prompting rather than a fixed similarity threshold.
351
+
352
+ ---
353
+
354
+ ## Engineering Features
355
+
356
+ ### AST-Aware Code Processing
357
+
358
+ Code is chunked using syntax structure instead of fixed character or token windows.
359
+
360
+ ### Grounded Citations
361
+
362
+ Answers include references to the source files and line ranges used to generate the response.
363
+
364
+ ### Honest Refusals
365
+
366
+ The system is instructed to refuse unsupported questions rather than produce confident but ungrounded answers.
367
+
368
+ ### Resumable Ingestion
369
+
370
+ Previously downloaded issues are skipped, allowing interrupted ingestion runs to continue without restarting from the beginning.
371
+
372
+ ### Evaluation Caching
373
+
374
+ LLM answers, judgments, and query rewrites are cached using the model and prompt as part of the cache key.
375
+
376
+ This reduces repeated API usage and makes evaluation reruns faster.
377
+
378
+ ### Rate-Limit Handling
379
+
380
+ API calls use retries and backoff to handle temporary rate limits and service interruptions.
381
+
382
+ ### Self-Building Deployment
383
+
384
+ The Hugging Face Space can rebuild its vector index from committed chunk data during startup.
385
+
386
+ Large generated database files do not need to be stored in Git.
387
+
388
+ ### Regression Testing
389
+
390
+ The project includes unit tests for the chunking pipeline and an evaluation set that can be used as a retrieval regression gate.
391
+
392
+ ---
393
+
394
+ ## Technology Stack
395
+
396
+ | Area | Technology |
397
  |---|---|
398
+ | Language | Python |
399
+ | User interface | Streamlit |
400
+ | Vector database | ChromaDB |
401
+ | Embeddings | `BAAI/bge-small-en-v1.5` |
402
+ | Sparse retrieval experiments | BM25 |
403
+ | Code parsing | Tree-sitter |
404
+ | LLM provider | Groq |
405
+ | GitHub ingestion | PyGithub |
406
+ | Testing | Pytest |
407
+ | Deployment | Hugging Face Spaces |
408
+
409
+ ---
410
+
411
+ ## Project Structure
412
+
413
+ ```text
414
+ codebase-rag/
415
+ ├── app.py
416
+ ├── README.md
417
+ ├── LICENSE
418
+ ├── requirements.txt
419
+ ├── .env
420
+ ├── .gitignore
421
+ │
422
+ ├── data/
423
+ │ ├── chunks.jsonl
424
+ │ ├── evaluation data
425
+ │ └── cached results
426
+ │
427
+ ├── src/
428
+ │ ├── ask.py
429
+ │ ├── chunk.py
430
+ │ ├── eval.py
431
+ │ ├── index.py
432
+ │ ├── ingest.py
433
+ │ └── retrieval.py
434
+ │
435
+ └── tests/
436
+ └── test_chunk.py
437
+ ```
438
+
439
+ ### Main Files
440
+
441
+ | File | Purpose |
442
+ |---|---|
443
+ | `app.py` | Streamlit chat interface |
444
+ | `src/ingest.py` | Downloads repository content and GitHub issues |
445
+ | `src/chunk.py` | Creates code, documentation, and issue chunks |
446
+ | `src/index.py` | Builds the vector index |
447
+ | `src/retrieval.py` | Runs retrieval strategies |
448
+ | `src/ask.py` | Generates grounded answers |
449
+ | `src/eval.py` | Runs retrieval and generation evaluation |
450
+ | `tests/test_chunk.py` | Tests chunking behaviour |
451
+
452
+ ---
453
+
454
+ ## Local Installation
455
+
456
+ ### 1. Clone the Repository
457
+
458
+ ```bash
459
+ git clone https://github.com/islam-mamedov/codebase-rag.git
460
+ cd codebase-rag
461
+ ```
462
+
463
+ ### 2. Create a Virtual Environment
464
+
465
+ macOS or Linux:
466
+
467
+ ```bash
468
+ python3 -m venv .venv
469
+ source .venv/bin/activate
470
+ ```
471
+
472
+ Windows:
473
+
474
+ ```bash
475
+ python -m venv .venv
476
+ .venv\Scripts\activate
477
+ ```
478
+
479
+ ### 3. Install Dependencies
480
+
481
+ ```bash
482
+ pip install --upgrade pip
483
+ pip install -r requirements.txt
484
+ ```
485
+
486
+ For ingestion, development, and testing:
487
+
488
+ ```bash
489
+ pip install PyGithub tree-sitter tree-sitter-python pytest
490
+ ```
491
+
492
+ ---
493
+
494
+ ## Environment Variables
495
+
496
+ Create a `.env` file in the project root:
497
+
498
+ ```env
499
+ GROQ_API_KEY=your_groq_api_key
500
+ GITHUB_TOKEN=your_github_token
501
+ ```
502
+
503
+ `GROQ_API_KEY` is required for answer generation.
504
+
505
+ `GITHUB_TOKEN` is used when downloading GitHub issues during ingestion.
506
+
507
+ Never commit `.env` or expose API keys in terminal screenshots, documentation, or chat messages.
508
+
509
+ You can confirm that `.env` is ignored with:
510
+
511
+ ```bash
512
+ git check-ignore .env
513
+ ```
514
+
515
+ ---
516
+
517
+ ## Running the Application
518
+
519
+ Start the Streamlit interface:
520
+
521
+ ```bash
522
+ streamlit run app.py
523
+ ```
524
+
525
+ Then open the local URL displayed in the terminal.
526
+
527
+ It is usually:
528
+
529
+ ```text
530
+ http://localhost:8501
531
+ ```
532
+
533
+ ---
534
+
535
+ ## Running the Pipeline
536
+
537
+ ### Ingest FastAPI Content
538
 
539
  ```bash
540
+ python src/ingest.py --repo fastapi/fastapi
541
+ ```
 
 
542
 
543
+ ### Create Chunks
544
+
545
+ ```bash
546
  python src/chunk.py
547
+ ```
548
+
549
+ ### Build the Vector Index
550
+
551
+ ```bash
552
  python src/index.py
553
+ ```
554
+
555
+ ### Ask a Question from the Terminal
556
+
557
+ ```bash
558
  python src/ask.py "How do I return a custom status code?"
 
 
559
  ```
560
 
561
+ ---
562
+
563
+ ## Running Tests
564
+
565
+ ```bash
566
+ pytest tests/ -v
567
+ ```
568
+
569
+ ---
570
+
571
+ ## Running Evaluation
572
+
573
+ Run dense retrieval evaluation:
574
+
575
+ ```bash
576
+ python src/eval.py --mode dense
577
+ ```
578
+
579
+ Run retrieval and answer-generation evaluation:
580
+
581
+ ```bash
582
+ python src/eval.py --mode dense --answers
583
+ ```
584
+
585
+ The evaluation results can be used to compare retrieval changes against the current baseline:
586
+
587
+ ```text
588
+ Recall@5: 0.91
589
+ MRR: 0.71
590
+ ```
591
+
592
+ A future CI pipeline could reject changes that reduce retrieval performance below these values.
593
+
594
+ ---
595
+
596
+ ## Current Limitations
597
+
598
+ ### Common-Token Queries
599
+
600
+ Some questions contain words that appear throughout the repository.
601
+
602
+ For example:
603
+
604
+ ```text
605
+ Where are the Query, Path, and Body parameter functions defined?
606
+ ```
607
+
608
+ The relevant file is:
609
+
610
+ ```text
611
+ fastapi/param_functions.py
612
+ ```
613
+
614
+ However, terms such as `query`, `path`, and `body` appear in many unrelated chunks.
615
+
616
+ LLM query rewriting was tested but did not improve aggregate retrieval performance. A stronger next step would be symbol-aware retrieval, exact identifier matching, or metadata filtering.
617
+
618
+ ### Documentation Include Directives
619
+
620
+ FastAPI documentation sometimes references external examples using include directives such as:
621
+
622
+ ```text
623
+ {* ../../docs_src/... *}
624
+ ```
625
+
626
+ The current chunker does not automatically insert the referenced source code into the documentation chunk.
627
+
628
+ As a result, some retrieved documentation explains an example without containing the complete implementation.
629
+
630
+ ### LLM-as-Judge Limitations
631
+
632
+ Generation metrics are useful for comparison, but they are not absolute.
633
+
634
+ During manual review, at least one response marked incorrect appeared to be valid. The judge penalised it because part of the supporting code was implied by truncated context rather than shown directly.
635
+
636
+ For this reason, judge scores are treated as directional measurements and are combined with manual inspection.
637
+
638
+ ### Repository-Specific Pipeline
639
+
640
+ The current system is designed around FastAPI.
641
+
642
+ Some ingestion and metadata assumptions would need to be generalised before the project could support arbitrary repositories.
643
+
644
+ ---
645
+
646
+ ## Future Improvements
647
+
648
+ - Resolve documentation include directives during ingestion
649
+ - Add symbol-aware and identifier-aware retrieval
650
+ - Make the ingestion pipeline repository-agnostic
651
+ - Compare BGE Small with larger and code-specific embedding models
652
+ - Add GitHub Discussions as another knowledge source
653
+ - Add conversation-aware follow-up questions
654
+ - Add automated retrieval regression tests in CI
655
+ - Track latency and token usage
656
+ - Add answer feedback and failure logging
657
+ - Evaluate retrieval separately for code, documentation, and issue questions
658
+
659
+ ---
660
+
661
+ ## Key Lessons
662
+
663
+ This project produced several practical findings:
664
+
665
+ 1. More retrieval components do not automatically produce better results.
666
+ 2. Hybrid search can reduce quality when sparse retrieval adds noisy candidates.
667
+ 3. Rerankers trained on natural-language passages may perform poorly on raw code.
668
+ 4. Similarity does not guarantee answerability.
669
+ 5. Refusal behaviour must be evaluated directly.
670
+ 6. Query rewriting can introduce semantic drift.
671
+ 7. A small labelled evaluation set can prevent weak architectural choices from reaching production.
672
+
673
+ ---
674
+
675
+ ## License
676
+
677
+ This project is released under the [MIT License](LICENSE).
678
+
679
+ ```text
680
+ Copyright (c) 2026 Islam Mamedov
681
+ ```
682
+
683
+ ---
684
+
685
+ ## Author
686
+
687
+ **Islam Mamedov**
688
+
689
+ AI Engineer focused on retrieval systems, agentic AI, computer vision, and production-oriented machine learning applications.
690
+
691
+ - GitHub: [islam-mamedov](https://github.com/islam-mamedov)
692
+ - Live project: [FastAPI Codebase Q&A](https://huggingface.co/spaces/islam-mamedov/fastapi-codebase-qa)
693
+
694
+ ---
695
+
696
+ <div align="center">
697
+
698
+ Built as an evaluation-driven AI engineering project.
699
+
700
+ **[Try the live demo](https://huggingface.co/spaces/islam-mamedov/fastapi-codebase-qa)**
701
 
702
+ </div>