Update README.md

README.md

@@ -12,107 +12,62 @@ pinned: false
 
 
 
-# Interview Q and A Dataset
-
-I built this dataset to solve a recurring problem I observed in data science and engineering education: the lack of realistic, challenging interview material for practice. Every interview is a performance, and to perform well, candidates need access to questions that reflect the actual hiring landscape across various industries and roles. 
-
-To create this material, I generated the entire dataset locally on my PC using an NVIDIA RTX 3080 GPU running Ollama. Running it locally gave me complete control over the generation pipeline without rate limits or data privacy concerns. I employed advanced prompt engineering techniques, including persona adoption, few-shot prompting, and strict JSON output formatting. I instructed the local language model to act as a senior talent acquisition specialist, dynamically injecting highly elaborated prompts with varying combinations of role, industry sector, interviewer persona, and difficulty tier. My goal was to guarantee that a Junior HR Specialist in Healthcare interviewed by a "Supportive" persona received a completely different and appropriate question compared to a Senior Data Scientist in Cybersecurity facing an "Aggressive" technical interviewer. The result is a structured, balanced question bank containing twenty thousand unique interview pairs.
-
-## Dataset Structure
-
-The dataset contains the following data structure:
-
-| Column | Type | Description |
-|---|---|---|
-| `question` | string | The interview question generated by the LLM |
-| `answer` | string | A model gold-standard answer |
-| `keywords` | string | Domain-specific searchable keywords |
-| `role` | string | Job title (e.g., Data Scientist, Software Engineer) |
-| `sector` | string | Industry sector (e.g., Finance, Healthcare) |
-| `interview_stage` | string | The type of interview round |
-| `interviewer` | string | The persona/tone of the interviewer |
-| `question_category` | string | Sub-category of the question (e.g., System Design) |
-| `question_level` | string | Difficulty / Seniority Level |
-
-After generating the data, I conducted a rigorous Exploratory Data Analysis to validate the quality and diversity of my synthetic dataset. 
-
-## Exploratory Data Analysis
-
-### Categorical Distributions
-
-![Categorical Distributions](eda_categorical_distributions.png)
-
-I began by checking the categorical variables to ensure balance across the dataset. First, I looked at the difficulty distribution across different seniority levels and learned that the model successfully balanced Easy, Medium, and Hard questions across all seniority tiers. Hard questions were slightly rarer, which perfectly mirrors real-world hiring pipelines where challenging questions are reserved for specific technical deep-dives. I also analyzed the distribution of question categories against interview types and discovered that the dataset did not over-represent any single skill. The questions were broadly distributed across twenty different categories, providing comprehensive coverage. Lastly, I verified the industry sector coverage and found a near-uniform distribution across all fourteen targeted sectors. This was a critical lesson: a dataset skewed entirely toward technology companies would be useless for a candidate applying to a hospital or a bank. My data remains truly industry-agnostic.
-
-### Text Length Analysis
-
-![Text Length Analysis](eda_text_length_analysis.png)
-
-I also needed to ensure the text itself felt natural, so I examined the text length distributions. I observed that the generated questions were concise and direct, while the answers provided enough depth to demonstrate expertise without turning into massive walls of text. 
-
-### Keyword Presence
-
-![Keyword Presence](eda_keyword_presence.png)
-
-To understand the technical vocabulary that dominated the dataset, I visualized the keyword presence. I observed that the language model successfully injected specific, relevant technical terms into the questions, avoiding overly generic fluff. This confirmed that the prompt engineering effectively anchored the questions in realistic technical scenarios.
-
-### Near Duplicate Detection
-
-![Near Duplicate Detection](eda_near_duplicates.png)
-
-A common pitfall with synthetic data is repetitive generation. To test for this, I implemented a near-duplicate detection script using TF-IDF cosine similarity. I was thrilled to see that almost all pairwise similarities were close to zero. There were absolutely no pairs crossing the high similarity thresholds, giving me mathematical proof that my dataset was clean and unique.
-
-### Role Keyword Alignment
+## 3. Embedding Model Selection
 
-![Role Keyword Alignment](eda_role_keyword_alignment.png)
+To power the retrieval engine, I evaluated three open-source, 384-dimensional sentence-embedding models from Hugging Face: `all-MiniLM-L6-v2`, `BAAI/bge-small-en-v1.5`, and `intfloat/e5-small-v2`. The evaluation ran on a random sample of 2,500 questions and measured three things:
 
-To take the analysis deeper, I designed a test to check the role and keyword alignment. I wanted to know if a Frontend Developer question actually contained frontend terminology. I initially found that some roles scored poorly because my keyword dictionary was too generic. Once I updated my script to search for actual technical domain words, the alignment rates soared. This taught me the importance of tailoring validation rules to the specific semantic domain being measured.
+1. **Encoding speed** — how long it takes to embed the sample.
+2. **Semantic separability** — Silhouette Score (cosine) of the embedding space against the 16 true question categories.
+3. **Retrieval quality** — Precision@1, Precision@5, and Mean Reciprocal Rank (MRR), where a retrieved question counts as relevant if it belongs to the same category as the query.
 
-### Sentiment Analysis
+One implementation detail worth noting: `e5-small-v2` is an asymmetric retrieval model and requires a `query: ` prefix on queries and a `passage: ` prefix on documents — without them its performance collapses. These prefixes were applied throughout the evaluation and the application.
 
-![Sentiment Analysis](eda_sentiment_analysis.png)
+![quality_vs_speed_evaluation](https://cdn-uploads.huggingface.co/production/uploads/69d7aa949910f34b03f98b9d/J6pWxJRDnjUYRQ0jyyF8s.png)
 
-Finally, I wanted to validate whether the model respected the "interviewer persona" I requested in my prompt. I used sentiment analysis to check the tone of the questions. I had to truncate the Y-axis labels because the prompt descriptions were too long and squashed the graph. Once fixed, the plot clearly showed that the "Aggressive" personas had distributions shifted toward negative polarity, while the "Friendly" personas were noticeably positive. This confirmed that my prompt engineering successfully altered the behavior and tone of the language model.
+![performance_compare](https://cdn-uploads.huggingface.co/production/uploads/69d7aa949910f34b03f98b9d/lOwBF7gaeFhsce5QH_tt2.png)
 
-With the data fully validated and explored, I am now moving forward to embed this dataset and build a recommendation engine.
+**Observations:**
 
+* **Speed:** `all-MiniLM-L6-v2` was by far the fastest (~28s for 2,500 questions), while `e5-small-v2` was the slowest (~84s, roughly 3× longer).
+* **Semantic separability:** all three silhouette scores were slightly negative — expected, since interview categories share a great deal of vocabulary and overlap heavily in meaning — but `e5-small-v2` scored best (−0.033), ahead of `bge-small-en-v1.5` (−0.040) and `all-MiniLM-L6-v2` (−0.046).
+* **Retrieval:** `e5-small-v2` won on all three metrics: Precision@1 = 0.18, Precision@5 = 0.16, MRR = 0.30. Interestingly, `all-MiniLM-L6-v2` came second (P@1 = 0.16, MRR = 0.28) and `bge-small-en-v1.5` last (P@1 = 0.15, MRR = 0.26), so raw model size was not the deciding factor. For context, these are strict proxy metrics — a match counts only if the retrieved question shares the exact category — and with 16 categories a random ranker would score P@1 ≈ 0.06, so the winner performs roughly 3× better than chance.
 
+**Conclusion:** `intfloat/e5-small-v2` won both quality evaluations (separability and retrieval), so despite being the slowest of the three, it was selected as the core embedding model. All 20,444 questions were then encoded with the `passage: ` prefix, L2-normalized, and saved as an `.npy` vector store that the application loads at startup.
 
-## 3. Embedding Model Selection
-To determine the most effective retrieval model for the application, three different open-source embedding models from Hugging Face were rigorously evaluated: `all-MiniLM-L6-v2`, `bge-small-en-v1.5`, and `e5-small-v2`.
+---
 
+## 4. Embeddings Analysis & Clustering
 
-![quality_vs_speed_evaluaation](https://cdn-uploads.huggingface.co/production/uploads/69d7aa949910f34b03f98b9d/J6pWxJRDnjUYRQ0jyyF8s.png)
+To validate that the chosen model organizes the interview domain in a meaningful way, the embedding space was analyzed with three complementary techniques: K-Means clustering with the Elbow Method, t-SNE projection, and hierarchical clustering.
 
+### K-Means & the Elbow Method
 
-![performance_compare](https://cdn-uploads.huggingface.co/production/uploads/69d7aa949910f34b03f98b9d/lOwBF7gaeFhsce5QH_tt2.png)
+![elbow_method](https://cdn-uploads.huggingface.co/production/uploads/69d7aa949910f34b03f98b9d/LQcobbqST6PAjn1-bVeMN.png)
 
+For each of the three candidate models, K-Means was fitted for K = 2 to 15 and the inertia (sum of squared distances) was plotted. The curves decline smoothly without a sharp elbow — another sign that interview questions form a continuum rather than isolated islands — but the bend suggested a different natural granularity per model: **K = 10** for MiniLM, **K = 8** for BGE, and **K = 6** for E5. Each model was then re-clustered at its own optimal K and projected to 2D (TruncatedSVD) for visual comparison:
 
-**Observation:** 
-The evaluation measured both execution speed and retrieval performance (Precision@1, Precision@5, and Mean Reciprocal Rank). 
-* `all-MiniLM-L6-v2` proved to be the fastest but lacked the semantic depth required for complex interview scenarios.
-* `e5-small-v2` consistently outperformed the others in semantic retrieval quality, achieving the highest Mean Reciprocal Rank (MRR: 0.30) and Precision@1 (0.18). 
 
-**Conclusion:** Despite a slightly longer execution time, `intfloat/e5-small-v2` was selected as the core embedding model due to its superior ability to capture the nuanced context of technical interview questions. All 18,144 questions were encoded and saved to an `.npy` vector database.
+![optimized_clusters](https://cdn-uploads.huggingface.co/production/uploads/69d7aa949910f34b03f98b9d/i7JxLx4iJJOgYMd2-8Bet.png)
 
----
+The E5 map shows the cleanest macro-structure: even at the smallest K, its clusters occupy distinct, coherent regions of the projection instead of interleaving.
 
-## 4. Embeddings Analysis & Clustering
-To validate that the chosen E5 model mathematically understands the interview domain, the 384-dimensional vectors were analyzed using dimensionality reduction (t-SNE/PCA) and clustering algorithms.
+### t-SNE Projection (True Categories)
 
 ![tSNE_projection](https://cdn-uploads.huggingface.co/production/uploads/69d7aa949910f34b03f98b9d/87YXdM2-F10u7gTdShgMz.png)
 
-**Dimensionality Reduction:** The t-SNE projections demonstrate how the models group questions by their true categories. The E5 model (right) shows distinct, organic clustering of topics like "Communication & Soft Skills" versus hard technical categories, validating its semantic mapping.
+To check how well each embedding space respects the *true* labels, the five most frequent question categories were projected with t-SNE (a non-linear method better suited to deep embeddings than PCA) and colored by their real category. As expected from the silhouette analysis, the categories are not linearly separable in any of the three spaces — interview questions genuinely blend topics (a "Coding & Problem Solving" question in FinTech shares vocabulary with "Tools & Frameworks"). However, clear **local neighborhood structure** is visible, and it is strongest in the E5 panel (right): "Prioritization Under Pressure" concentrates into a dense central region and "Tools & Frameworks" forms distinct pockets. This local coherence is exactly what a nearest-neighbor retrieval engine relies on, and it is consistent with E5 winning the Precision/MRR comparison.
+
+### Hierarchical Semantic Taxonomy
 
 ![SEMANTIC](https://cdn-uploads.huggingface.co/production/uploads/69d7aa949910f34b03f98b9d/rZ84igc2uMRVGqBcaE8g0.png)
 
-**Hierarchical Semantic Taxonomy:** 
-Using Ward's Linkage Variance, a hierarchical dendrogram was generated from the E5 embeddings. The algorithm automatically discovered the fundamental pillars of a tech interview without being provided any labels. It organically grouped the categories into distinct "branches":
-1. **Behavioral & HR:** Culture Fit, Soft Skills, and Salary naturally clustered together.
-2. **Core Engineering:** Coding, System Design, and Tools & Frameworks formed a distinct technical branch.
-3. **Operational Dynamics:** Incident Response, Conflict Resolution, and Cross-Team Work formed a behavioral-operational cluster.
+Finally, a mean (centroid) embedding was computed for each of the 16 question categories, L2-normalized, and clustered hierarchically using **Ward's linkage**. The category labels only define the 16 groups — the *relationships between them* are discovered purely from the geometry of the embedding space. The resulting dendrogram organizes the interview domain into three interpretable branches:
+
+1. **Behavioral & HR:** "Culture Fit & Values", "Salary & Expectations", "Communication & Soft Skills", and "Business Impact & Metrics" merge together at low distance.
+2. **Core Engineering:** "Coding & Problem Solving", "System Design & Architecture", "Scalability & Growth Planning", and "Innovation & Emerging Tech" form a distinct technical branch. Notably, "Motivation & Career Goals" attaches to this branch via "Tools & Frameworks" — the model apparently associates career-growth questions with technology-stack questions, a genuinely surprising discovery.
+3. **Operational & Situational:** "Incident Response & Ownership", "Prioritization Under Pressure", "Collaboration & Cross-Team Work", and "Conflict Resolution" cluster together with "Debugging & Troubleshooting" and "Technical Roadmap & Strategy" — the "how do you act under real conditions" side of interviewing.
 
-This proves the retrieval system searches within a highly structured, semantically logical vector space.
+The fact that this taxonomy matches how humans intuitively divide interviews — HR vs. hard skills vs. situational behavior — without ever being told those relationships confirms that the retrieval system operates in a semantically structured vector space.
 
 ---
 
@@ -125,5 +80,3 @@ The application utilizes a Vector-Based Retrieval strategy to recommend the most
 3. **Query Embedding:** The input query is passed through the selected `e5-small-v2` model, converting it into a 384-dimensional query vector.
 4. **Vector Similarity Search:** The system computes the **Cosine Similarity** between the user's query vector and the pre-computed question embeddings database.
 5. **Retrieval:** The system identifies the vector with the highest similarity score (argmax) and retrieves the exact matching interview scenario.
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference