Spaces:
Sleeping
A newer version of the Streamlit SDK is available: 1.61.1
title: FastAPI Codebase Q&A
emoji: ⚡
colorFrom: yellow
colorTo: gray
sdk: streamlit
app_file: app.py
pinned: false
⚡ FastAPI Codebase Q&A
An evaluation-driven RAG system for exploring a real production codebase
Ask technical questions about FastAPI and receive answers grounded in its source code, documentation, and resolved GitHub issues.
Question → Retrieval → Grounded answer → Source evidence
Overview
FastAPI Codebase Q&A is a Retrieval-Augmented Generation system designed to answer questions about the FastAPI repository.
Instead of relying only on an LLM's general knowledge, the system searches a custom knowledge base built from:
- FastAPI source code
- English documentation
- Closed GitHub issues
The retrieved evidence is passed to the language model, which generates an answer with references to the relevant files, symbols, and line numbers.
The project was built around one principle:
Every important retrieval decision should be measured rather than guessed.
A hand-labelled evaluation set was used to compare dense retrieval, hybrid search, reranking, and query rewriting before selecting the final production configuration.
Live Demo
Try the deployed application:
Open FastAPI Codebase Q&A on Hugging Face Spaces
Example questions:
How do I return a custom status code in FastAPI?
Where is APIRouter defined?
How does FastAPI validate request bodies?
Can FastAPI automatically deploy my application to AWS?
The final example tests the system's refusal behaviour. When the indexed corpus does not contain enough evidence, the application should say so instead of inventing an answer.
What the System Does
The application follows a complete RAG workflow:
- Ingests content from the FastAPI repository.
- Splits code, documentation, and issues using content-aware chunking.
- Converts chunks into vector embeddings.
- Stores and retrieves relevant chunks using ChromaDB.
- Passes the retrieved evidence to an LLM.
- Generates a grounded answer with citations.
- Refuses questions that cannot be answered from the available evidence.
- Evaluates retrieval and answer quality using a labelled benchmark.
Architecture
┌──────────────────────────────────────────────────────────────────────┐
│ FastAPI Repository │
│ │
│ Source Code · Documentation · GitHub Issues │
└───────────────────────────────┬──────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────┐
│ Ingestion │
│ │
│ Repository cloning · GitHub API · Resumable issue fetching │
└───────────────────────────────┬──────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────┐
│ Chunking │
│ │
│ AST-aware code · Markdown-aware docs · Issue-aware chunks │
└───────────────────────────────┬──────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────┐
│ Indexing │
│ │
│ BGE embeddings · ChromaDB vector store · Chunk metadata │
└───────────────────────────────┬──────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────┐
│ Retrieval │
│ │
│ Dense semantic search selected through evaluation │
└───────────────────────────────┬──────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────┐
│ Generation │
│ │
│ Groq LLM · Grounded answers · Citations · Refusal rules │
└───────────────────────────────┬──────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────┐
│ Evaluation │
│ │
│ Recall@5 · MRR · Faithfulness · Correctness · Refusal accuracy │
└──────────────────────────────────────────────────────────────────────┘
Indexed Corpus
The knowledge base contains:
| Source | Quantity |
|---|---|
| FastAPI source files | 46 |
| English documentation files | 161 |
| Closed GitHub issues | 175 |
| Total chunks | 1,352 |
Chunk distribution:
| Chunk type | Quantity |
|---|---|
| Code chunks | 363 |
| Documentation chunks | 814 |
| Issue chunks | 175 |
Every chunk includes metadata such as:
- File path
- Content type
- Symbol name
- Start and end lines
- Repository information
- Context header
Example context header:
# File: fastapi/routing.py | Symbol: APIRouter
This metadata allows the application to produce answers that point back to the relevant part of the repository.
Content-Aware Chunking
Different content types require different chunking strategies.
Source Code
Python files are parsed using Tree-sitter.
The chunker creates separate chunks for:
- Functions
- Classes
- Methods
- Module-level content
Large classes are split by method when necessary instead of being stored as one oversized block.
This preserves the structure of the code and improves retrieval for questions such as:
Where is the APIRouter class defined?
Documentation
Markdown documents are divided using heading and section boundaries.
This avoids splitting related explanations in the middle of a section and preserves useful context.
GitHub Issues
Issues are stored with their:
- Title
- Body
- Resolution context
- Relevant metadata
Issue-aware chunking helps answer behavioural questions that may not be explained clearly in the source code or documentation.
Evaluation Dataset
The system is evaluated using 42 manually labelled questions.
| Category | Questions |
|---|---|
| API usage | 16 |
| Behaviour | 11 |
| Code location | 8 |
| Unanswerable | 7 |
| Total | 42 |
Each answerable question includes a gold reference identifying the file or symbol expected to contain the answer.
The evaluation measures two separate parts of the system.
Retrieval Metrics
- Recall@5: whether a correct chunk appears within the first five retrieved results
- MRR: how highly the first correct result is ranked
Generation Metrics
- Faithfulness: whether the answer is supported by the retrieved context
- Correctness: whether the response answers the question accurately
- Refusal accuracy: whether unsupported questions are rejected correctly
Retrieval Experiments
Several retrieval configurations were tested before selecting the production approach.
| Configuration | Recall@5 | MRR | Result |
|---|---|---|---|
| Dense retrieval — BGE Small | 0.91 | 0.71 | Selected for production |
| Hybrid dense + BM25 using RRF | 0.86 | 0.70 | Lower recall |
| Hybrid weighted RRF at 2:1 | 0.86 | 0.71 | No recall improvement |
| Dense + BGE reranker base | 0.91 | 0.63 | Ranking became worse |
| Dense + BGE reranker v2-m3 | 0.83 | 0.67 | Lower recall and MRR |
| Hybrid + BGE reranker v2-m3 | 0.86 | 0.68 | No improvement |
| Dense + LLM query rewriting | 0.91 | 0.69 | No measurable gain |
Why Dense Retrieval Won
Dense retrieval produced the best overall combination of recall and ranking quality.
Diagnostic experiments showed:
Dense recall@20: 0.97
Hybrid recall@20: 0.97
BM25 did not introduce additional correct candidates. Instead, common words such as request, body, and json caused unrelated GitHub issue chunks to move above more useful code and documentation chunks.
Because hybrid retrieval added noise without adding new correct results, the simpler dense configuration was selected.
Why Reranking Was Rejected
Both rerankers frequently pushed raw code chunks lower in the results.
This especially affected code-location questions.
A source-code definition may directly contain the answer without explaining it in natural language. Cross-encoders often prefer passages that discuss a topic rather than code that implements it.
The retrievers were already finding strong candidates at k=20, but the rerankers were reordering them incorrectly.
Why Query Rewriting Was Rejected
LLM query rewriting produced:
Recall@5: 0.91
MRR: 0.69
It did not improve recall over the original dense query and slightly reduced ranking quality.
Some rewritten queries added useful-looking but incorrect identifiers, creating semantic drift. The feature was therefore not included in the deployed retrieval pipeline.
Generation Results
The reported generation evaluation used dense retrieval with a Groq-hosted LLM.
| Metric | Score |
|---|---|
| Faithfulness | 0.89 |
| Correctness | 0.91 |
| Refusal accuracy | 7/7 |
Refusal Behaviour
Refusal accuracy was one of the most sensitive parts of the project.
Across three prompt versions, the result changed from:
6/7 → 0/7 → 7/7
The main problem was that retrieved chunks could be related to the question without actually containing enough information to answer it.
The final prompt explicitly explains this distinction to the model:
The context chunks are search results. They may be loosely related to the
question without actually containing the answer.
This small prompt change prevented the model from treating all retrieved content as valid evidence.
Why a Similarity Threshold Was Not Used
A retrieval-score threshold was tested as an alternative refusal mechanism.
However, the score ranges overlapped:
Answerable questions: 0.74–0.89
Unanswerable questions: 0.72–0.79
A high similarity score only shows that a chunk is related to the question. It does not prove that the chunk contains the answer.
For this reason, refusal is handled through evidence-aware prompting rather than a fixed similarity threshold.
Engineering Features
AST-Aware Code Processing
Code is chunked using syntax structure instead of fixed character or token windows.
Grounded Citations
Answers include references to the source files and line ranges used to generate the response.
Honest Refusals
The system is instructed to refuse unsupported questions rather than produce confident but ungrounded answers.
Resumable Ingestion
Previously downloaded issues are skipped, allowing interrupted ingestion runs to continue without restarting from the beginning.
Evaluation Caching
LLM answers, judgments, and query rewrites are cached using the model and prompt as part of the cache key.
This reduces repeated API usage and makes evaluation reruns faster.
Rate-Limit Handling
API calls use retries and backoff to handle temporary rate limits and service interruptions.
Self-Building Deployment
The Hugging Face Space can rebuild its vector index from committed chunk data during startup.
Large generated database files do not need to be stored in Git.
Regression Testing
The project includes unit tests for the chunking pipeline and an evaluation set that can be used as a retrieval regression gate.
Technology Stack
| Area | Technology |
|---|---|
| Language | Python |
| User interface | Streamlit |
| Vector database | ChromaDB |
| Embeddings | BAAI/bge-small-en-v1.5 |
| Sparse retrieval experiments | BM25 |
| Code parsing | Tree-sitter |
| LLM provider | Groq |
| GitHub ingestion | PyGithub |
| Testing | Pytest |
| Deployment | Hugging Face Spaces |
Project Structure
codebase-rag/
├── app.py
├── README.md
├── LICENSE
├── requirements.txt
├── .env
├── .gitignore
│
├── data/
│ ├── chunks.jsonl
│ ├── evaluation data
│ └── cached results
│
├── src/
│ ├── ask.py
│ ├── chunk.py
│ ├── eval.py
│ ├── index.py
│ ├── ingest.py
│ └── retrieval.py
│
└── tests/
└── test_chunk.py
Main Files
| File | Purpose |
|---|---|
app.py |
Streamlit chat interface |
src/ingest.py |
Downloads repository content and GitHub issues |
src/chunk.py |
Creates code, documentation, and issue chunks |
src/index.py |
Builds the vector index |
src/retrieval.py |
Runs retrieval strategies |
src/ask.py |
Generates grounded answers |
src/eval.py |
Runs retrieval and generation evaluation |
tests/test_chunk.py |
Tests chunking behaviour |
Local Installation
1. Clone the Repository
git clone https://github.com/islam-mamedov/codebase-rag.git
cd codebase-rag
2. Create a Virtual Environment
macOS or Linux:
python3 -m venv .venv
source .venv/bin/activate
Windows:
python -m venv .venv
.venv\Scripts\activate
3. Install Dependencies
pip install --upgrade pip
pip install -r requirements.txt
For ingestion, development, and testing:
pip install PyGithub tree-sitter tree-sitter-python pytest
Environment Variables
Create a .env file in the project root:
GROQ_API_KEY=your_groq_api_key
GITHUB_TOKEN=your_github_token
GROQ_API_KEY is required for answer generation.
GITHUB_TOKEN is used when downloading GitHub issues during ingestion.
Never commit .env or expose API keys in terminal screenshots, documentation, or chat messages.
You can confirm that .env is ignored with:
git check-ignore .env
Running the Application
Start the Streamlit interface:
streamlit run app.py
Then open the local URL displayed in the terminal.
It is usually:
http://localhost:8501
Running the Pipeline
Ingest FastAPI Content
python src/ingest.py --repo fastapi/fastapi
Create Chunks
python src/chunk.py
Build the Vector Index
python src/index.py
Ask a Question from the Terminal
python src/ask.py "How do I return a custom status code?"
Running Tests
pytest tests/ -v
Running Evaluation
Run dense retrieval evaluation:
python src/eval.py --mode dense
Run retrieval and answer-generation evaluation:
python src/eval.py --mode dense --answers
The evaluation results can be used to compare retrieval changes against the current baseline:
Recall@5: 0.91
MRR: 0.71
A future CI pipeline could reject changes that reduce retrieval performance below these values.
Current Limitations
Common-Token Queries
Some questions contain words that appear throughout the repository.
For example:
Where are the Query, Path, and Body parameter functions defined?
The relevant file is:
fastapi/param_functions.py
However, terms such as query, path, and body appear in many unrelated chunks.
LLM query rewriting was tested but did not improve aggregate retrieval performance. A stronger next step would be symbol-aware retrieval, exact identifier matching, or metadata filtering.
Documentation Include Directives
FastAPI documentation sometimes references external examples using include directives such as:
{* ../../docs_src/... *}
The current chunker does not automatically insert the referenced source code into the documentation chunk.
As a result, some retrieved documentation explains an example without containing the complete implementation.
LLM-as-Judge Limitations
Generation metrics are useful for comparison, but they are not absolute.
During manual review, at least one response marked incorrect appeared to be valid. The judge penalised it because part of the supporting code was implied by truncated context rather than shown directly.
For this reason, judge scores are treated as directional measurements and are combined with manual inspection.
Repository-Specific Pipeline
The current system is designed around FastAPI.
Some ingestion and metadata assumptions would need to be generalised before the project could support arbitrary repositories.
Future Improvements
- Resolve documentation include directives during ingestion
- Add symbol-aware and identifier-aware retrieval
- Make the ingestion pipeline repository-agnostic
- Compare BGE Small with larger and code-specific embedding models
- Add GitHub Discussions as another knowledge source
- Add conversation-aware follow-up questions
- Add automated retrieval regression tests in CI
- Track latency and token usage
- Add answer feedback and failure logging
- Evaluate retrieval separately for code, documentation, and issue questions
Key Lessons
This project produced several practical findings:
- More retrieval components do not automatically produce better results.
- Hybrid search can reduce quality when sparse retrieval adds noisy candidates.
- Rerankers trained on natural-language passages may perform poorly on raw code.
- Similarity does not guarantee answerability.
- Refusal behaviour must be evaluated directly.
- Query rewriting can introduce semantic drift.
- A small labelled evaluation set can prevent weak architectural choices from reaching production.
License
This project is released under the MIT License.
Copyright (c) 2026 Islam Mamedov
Author
Islam Mamedov
AI Engineer focused on retrieval systems, agentic AI, computer vision, and production-oriented machine learning applications.
- GitHub: islam-mamedov
- Live project: FastAPI Codebase Q&A
Built as an evaluation-driven AI engineering project.