Merge branch 'master' of https://github.com/amt-danielboateng/ai-engineering-project

.gitignore

@@ -4,7 +4,8 @@ venv/
 .venv/
 
 # Project specifics
-project-rubrics
+project-rubric/
+project-rubrics/
 
 # Python
 __pycache__/

app.py

@@ -578,8 +578,8 @@ def get_general_response(user_query: str):
 
 @st.cache_resource(show_spinner="Loading policy index…")
 def init_rag_system():
-    from langchain_community.embeddings import HuggingFaceEmbeddings
-    from langchain_community.vectorstores import Chroma
+    from langchain_huggingface import HuggingFaceEmbeddings
+    from langchain_chroma import Chroma
     from langchain_openai import ChatOpenAI
 
     embeddings   = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
@@ -591,7 +591,7 @@ def init_rag_system():
         openai_api_key=OPENROUTER_API_KEY,
         model="openrouter/auto",
         temperature=0.0,
-        streaming=True,
+        max_tokens=512,
     )
     return retriever, llm
 

design-and-evaluation.md

@@ -7,15 +7,15 @@
 * **Chunking Strategy:** Documents were partitioned using LangChain's `RecursiveCharacterTextSplitter` set to a `chunk_size` of 500 characters and a `chunk_overlap` of 50 characters to guarantee complete coverage of semantic definitions near transition points.
 
 ## 📊 2. Evaluation Strategy and Results
-We evaluated our pipeline across a standardized benchmark test suite of 15 queries covering leave management, security protocols, and operational expenses.
+We evaluated our pipeline across a standardized benchmark test suite of 20 queries covering PTO, remote work, expenses, security, holidays, parental leave, sick leave, learning & development, performance review, travel, acceptable use, onboarding, health & safety, and code of conduct.
 
 ### Metric Results Table
 | Metric Category | Metric Evaluated | Performance Achieved |
 | :--- | :--- | :--- |
-| **Answer Quality** | Groundedness Rate | 100.0% |
-| **Answer Quality** | Citation Accuracy | 93.3% |
-| **System Performance** | $p50$ Latency | 1.14 seconds |
-| **System Performance** | $p95$ Latency | 2.45 seconds |
+| **Answer Quality** | Groundedness Rate | 95.0% |
+| **Answer Quality** | Citation Accuracy | 95.0% |
+| **System Performance** | $p50$ Latency | 2.69 seconds |
+| **System Performance** | $p95$ Latency | 7.91 seconds |
 
 ### Analysis
-Our 100% Groundedness rate indicates that the guardrails injected into our system prompt successfully prevented the model from hallucinating policies outside the given context chunks. The slight drop in Citation Accuracy occurred when a single policy was distributed across multiple consecutive chunks, and the model referenced only the first chronological block.
+Our 95% Groundedness and Citation Accuracy rates confirm that the guardrails in the system prompt effectively prevent hallucination. The single failing question (#18 — mandatory onboarding training) failed because the relevant content was split across two chunks from different policy documents (onboarding and code of conduct), causing the model to synthesize an answer that was not fully supported by any single retrieved chunk. The elevated p95 latency (7.91s) is explained by the cold-start on query #1, which triggered model weight loading; subsequent queries settled at a median of 2.69s.

evaluation.py

@@ -1,7 +1,10 @@
 # evaluation.py
 import os
+import random
 import time
 import numpy as np
+random.seed(42)
+np.random.seed(42)
 from app import rag_chain  # Import your configured chain from app.py
 from langchain_openai import ChatOpenAI
 from langchain_core.prompts import ChatPromptTemplate

ingestion.py

@@ -1,9 +1,13 @@
 # ingestion.py
 import os
+import random
+import numpy as np
+random.seed(42)
+np.random.seed(42)
 from langchain_community.document_loaders import DirectoryLoader, TextLoader
 from langchain_text_splitters import RecursiveCharacterTextSplitter
-from langchain_community.embeddings import HuggingFaceEmbeddings
-from langchain_community.vectorstores import Chroma
+from langchain_huggingface import HuggingFaceEmbeddings
+from langchain_chroma import Chroma
 
 def run_ingestion():
     # 1. Load documents from your local corpus

requirements.txt

@@ -3,6 +3,8 @@ langchain
 langchain-community
 langchain-openai
 langchain-text-splitters
+langchain-huggingface
+langchain-chroma
 chromadb
 sentence-transformers
 pydantic