AnishDev commited on
Commit
74d1c40
Β·
1 Parent(s): a2aef70

readme changes

Browse files
Files changed (1) hide show
  1. README.md +97 -164
README.md CHANGED
@@ -1,176 +1,56 @@
1
- # Coderound AI - Candidate Matching System
2
 
3
- This repository contains the backend for the Coderound AI Candidate Matching System. It's a sophisticated pipeline designed to intelligently match candidates to job descriptions (JDs) using a hybrid AI approach.
4
 
5
- ## Project Overview
6
 
7
- The system is built to address the challenge of efficiently and accurately screening a large volume of candidates for technical roles. It automates the parsing of unstructured data from JD PDFs and candidate resumes (in XLSX format), evaluates their suitability, and provides AI-generated explanations for its matching decisions.
8
 
9
- The core of the system is a multi-stage matching pipeline that combines traditional information retrieval techniques with modern NLP models:
10
-
11
- 1. **Data Parsing:** Ingests and structures data from various formats. JDs are parsed from PDFs to extract key requirements, while candidate data is parsed from XLSX files.
12
- 2. **BM25 Pre-filtering:** For large candidate pools (>200), the system first uses a BM25 relevance model to quickly identify a smaller, more relevant subset of candidates. This is a crucial first step for scalability.
13
- 3. **Semantic Matching:** The filtered candidates are then processed through a semantic search pipeline using sentence transformers (`all-MiniLM-L6-v2`). This model, run locally, generates embeddings for candidate experience and JD requirements to find semantic similarities.
14
- 4. **Proficiency-Aware Scoring:** The system doesn't just check for skill presence; it scores based on proficiency levels (e.g., "Expert" vs. "Familiar"), providing a more nuanced evaluation.
15
- 5. **AI-Powered Explanations:** For top matches, the system can generate a detailed explanation of why a candidate is a good fit, using a large language model hosted on Ollama Cloud.
16
-
17
- The entire process is exposed via a RESTful API built with FastAPI, featuring asynchronous processing for long-running matching tasks.
18
 
19
  ## Tech Stack
20
 
21
- | Category | Technology | Purpose |
22
- | --- | --- | --- |
23
- | **Framework** | FastAPI | High-performance Python web framework for building APIs. |
24
- | **Server** | Uvicorn | ASGI server for running the FastAPI application. |
25
- | **NLP & Matching** | Sentence-Transformers | For generating local sentence embeddings. |
26
- | | Rank-BM25 | Efficient keyword-based pre-filtering. |
27
- | | PyMuPDF | PDF parsing and text extraction. |
28
- | | Pandas, Openpyxl | Parsing and handling XLSX candidate files. |
29
- | **AI Explanations** | Ollama | For generating natural language explanations of matches. |
30
- | **Caching & State** | Redis | Used for caching results and managing task status. |
31
- | **Configuration** | Pydantic | For settings management and data validation. |
32
-
33
- ## Scaling to 100,000 Candidates
34
-
35
- The current architecture is designed with scalability in mind. Here’s how it would handle a load of 100,000 candidates and the changes that would be required to ensure performance.
36
-
37
- ### Current Scalability Features:
38
-
39
- * **Asynchronous Processing:** Matching jobs are run as background tasks, so the API remains responsive. The client can poll for results.
40
- * **BM25 Pre-filtering:** The pipeline intelligently activates a BM25 pre-filter when the candidate pool exceeds 200. This significantly reduces the number of candidates that need to go through the more computationally expensive embedding and scoring stages.
41
- * **Efficient Embeddings:** Using a lightweight local model like `all-MiniLM-L6-v2` avoids network latency and API costs associated with external embedding services.
42
-
43
- ### Architectural Changes for 100k+ Scale:
44
-
45
- 1. **Distributed Task Queue:**
46
- * **Current:** FastAPI's `BackgroundTasks` is suitable for a single-server setup.
47
- * **Proposed Change:** Replace `BackgroundTasks` with a dedicated distributed task queue like **Celery** with a **RabbitMQ** or **Redis** broker. This would allow us to distribute matching jobs across multiple worker nodes, scaling horizontally.
48
-
49
- 2. **Vector Database:**
50
- * **Current:** Embeddings are generated on-the-fly and stored in memory or a simple cache.
51
- * **Proposed Change:** Pre-compute and store candidate embeddings in a dedicated **vector database** like **Pinecone, Weaviate, or Milvus**. This would transform the matching process from a "search" to a "lookup," drastically reducing latency. The workflow would be:
52
- 1. When a candidate is uploaded, their profile is parsed, embedded, and stored in the vector DB.
53
- 2. When a JD is uploaded, its requirements are embedded.
54
- 3. The JD embedding is used to query the vector DB to find the top-K most similar candidates in milliseconds.
55
-
56
- 3. **Dedicated Microservices:**
57
- * **Current:** The application is a monolith.
58
- * **Proposed Change:** Break down the application into microservices:
59
- * `Parser Service`: Handles file parsing (PDF, XLSX).
60
- * `Matching Service`: Contains the core matching logic and workers.
61
- * `API Gateway`: The main entry point for all requests.
62
- * `Explanation Service`: Manages interaction with the Ollama LLM.
63
- This would allow independent scaling of each component. For example, we could scale up the `Matching Service` workers during peak hiring seasons without touching other services.
64
 
65
- 4. **Container Orchestration:**
66
- * **Current:** The app is containerized with Docker and can be deployed on a platform like Fly.io.
67
- * **Proposed Change:** Use **Kubernetes** to manage the containerized microservices. Kubernetes would handle auto-scaling, service discovery, and load balancing automatically.
68
 
69
- ### Diagram of Scaled Architecture
 
 
70
 
71
- ```mermaid
72
- graph TD
73
- subgraph "User"
74
- A[Recruiter]
75
- end
76
-
77
- subgraph "API Layer"
78
- B[API Gateway]
79
- end
80
-
81
- subgraph "Services"
82
- C[Parser Service]
83
- D[Matching Service]
84
- E[Explanation Service]
85
- end
86
 
87
- subgraph "Data & AI Layer"
88
- F[Vector Database e.g., Pinecone]
89
- G[Task Queue e.g., Celery + RabbitMQ]
90
- H[Redis Cache]
91
- I[Ollama LLM]
92
- end
93
 
94
- A --> B
95
- B --> C
96
- B --> D
97
- B --> E
98
 
99
- C --> G
100
- D -- Reads from --> F
101
- D -- Uses --> G
102
- D -- Caches in --> H
103
- E -- Calls --> I
104
- ```
 
 
 
 
 
105
 
106
- ## Sample Data for Testing
107
-
108
- ### Sample Job Description
109
-
110
- **Role:** Senior Backend Engineer (Python)
111
- **Experience:** 5+ years
112
- **Required Skills:**
113
- * Python (Expert)
114
- * Django or FastAPI (Expert)
115
- * PostgreSQL (Advanced)
116
- * Redis (Advanced)
117
- * Docker (Advanced)
118
- * Celery (Intermediate)
119
-
120
- **Preferred Skills:**
121
- * Kubernetes (Intermediate)
122
- * Terraform (Familiar)
123
- * AWS or GCP (Advanced)
124
-
125
- ### Sample Candidate Profiles
126
-
127
- **Candidate 1: Strong Match**
128
- * **Experience:** 7 years as a Backend Engineer.
129
- * **Skills:** Python (Expert), FastAPI (Expert), PostgreSQL (Advanced), Redis (Expert), Docker (Advanced), Kubernetes (Intermediate), AWS (Advanced).
130
- * **Summary:** Built and scaled a microservices-based system using FastAPI, deployed on AWS with Kubernetes.
131
-
132
- **Candidate 2: Medium Match**
133
- * **Experience:** 4 years as a Full-Stack Developer.
134
- * **Skills:** Python (Advanced), Django (Advanced), MySQL (Advanced), Docker (Intermediate), Heroku (Advanced).
135
- * **Summary:** Developed monolithic web applications with Django. Has some containerization experience but lacks cloud-native and distributed systems knowledge.
136
-
137
- **Candidate 3: Low Match**
138
- * **Experience:** 2 years as a Data Analyst.
139
- * **Skills:** Python (Intermediate), Pandas (Expert), SQL (Advanced), Tableau (Expert).
140
- * **Summary:** Focused on data analysis and visualization. Limited backend development experience.
141
-
142
- ## Setup and Deployment
143
-
144
- Follow the instructions below to set up the project locally or deploy it.
145
-
146
- ### Local Setup
147
-
148
- 1. **Prerequisites:** Python 3.11+
149
- 2. **Install dependencies:**
150
- ```bash
151
- pip install -r requirements.txt
152
- ```
153
- 3. **Configure Environment:**
154
- Create a `.env` file.
155
- 4. **Run the server:**
156
- ```bash
157
- uvicorn app.main:app --reload --port 8000
158
- ```
159
-
160
-
161
- | Variable | Required | Description | Example |
162
- |----------|----------|-------------|---------|
163
- | `OLLAMA_HOST` | No | Ollama server URL | `https://ollama.com` |
164
- | `OLLAMA_API_KEY` | No | Ollama cloud API key | `abc123...` |
165
- | `OLLAMA_MODEL` | No | Model name | `gpt-oss:120b-cloud` |
166
- | `REDIS_URL` | No | Redis connection string (use `rediss://` for TLS) | `rediss://default:pass@host:port` |
167
- | `CORS_ORIGINS` | Yes | Comma-separated allowed origins | `http://localhost:3000` |
168
- | `EMBEDDING_MODEL` | No | sentence-transformers model | `all-MiniLM-L6-v2` |
169
- | `BM25_PREFILTER_THRESHOLD` | No | Min candidates before BM25 activates | `200` |
170
- | `BM25_TOP_K` | No | Candidates kept after BM25 | `500` |
171
- | `LOG_LEVEL` | No | Logging level | `INFO` |
172
-
173
- > **Redis URL**: Upstash requires `rediss://` (double-s) for TLS. Using `redis://` will fail with SSL EOF errors.
174
 
175
  ## API Reference
176
 
@@ -207,7 +87,7 @@ GET /api/v1/match/{jd_id}/{candidate_id}?batch_id=&generate=true
207
  β†’ full score breakdown + AI explanation (generate=true triggers Ollama)
208
  ```
209
 
210
- ## Scoring Pipeline
211
 
212
  ```
213
  Hard filter (engineer type + years of experience)
@@ -235,7 +115,7 @@ Required columns:
235
  |--------|-------------|
236
  | `name` | Full name |
237
  | `current_position` | Job title / role |
238
- | `engineer_type` | "AI Engineer", "Backend Engineer", etc. |
239
  | `parsed_skills` | Skills text |
240
  | `parsed_summary` | Professional summary |
241
  | `parsed_work_experience` | Stringified Python list of `{company, role, duration, description}` dicts |
@@ -243,8 +123,61 @@ Required columns:
243
 
244
  Optional: `gen_ai_experience`, `current_title`, `is_actively_or_passively_looking`, `notice_period`, `parsed_metadata_most_recent_company`
245
 
246
- ## Storage Notes
 
 
247
 
248
- Candidate JSON is stored in **process memory**, not Redis. This avoids the Upstash free tier 10MB value limit. Process restart clears candidates β€” re-upload required. All match results and metadata are in Redis with 24-48h TTL.
249
 
250
- If Redis is unavailable, storage auto-falls back to `_MemStore` (in-process dict with TTL). System continues working without any external dependencies.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Backend β€” Candidate Matching System
2
 
3
+ FastAPI backend for the candidate matching task. Parses JD PDFs and candidate XLSX files, runs a hybrid BM25 + semantic matching pipeline, and returns ranked candidates with optional AI-generated explanations via Ollama.
4
 
5
+ ## How It Works
6
 
7
+ Matching runs as a background task through four stages:
8
 
9
+ 1. **Hard filter** β€” removes candidates that don't meet engineer type or minimum years of experience
10
+ 2. **BM25 pre-filter** β€” activated only when candidate pool exceeds 200; narrows down to top 500 before embedding
11
+ 3. **Semantic matching** β€” runs `all-MiniLM-L6-v2` locally to embed candidate work history and JD requirements
12
+ 4. **Scoring** β€” combines proficiency-aware skill matching, semantic similarity, title alignment, and domain score; applies penalties; stores ranked results in Redis
 
 
 
 
 
13
 
14
  ## Tech Stack
15
 
16
+ | Category | Technology |
17
+ | --- | --- |
18
+ | Framework | FastAPI + Uvicorn |
19
+ | NLP & Matching | Sentence-Transformers, Rank-BM25, PyMuPDF |
20
+ | Data Parsing | Pandas, Openpyxl |
21
+ | AI Explanations | Ollama |
22
+ | Caching & State | Redis |
23
+ | Config | Pydantic Settings |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
24
 
25
+ ## Setup
 
 
26
 
27
+ ### Prerequisites
28
+ - Python 3.11+
29
+ - Redis (or use Docker setup from root)
30
 
31
+ ### Run Locally
 
 
 
 
 
 
 
 
 
 
 
 
 
 
32
 
33
+ ```bash
34
+ pip install -r requirements.txt
35
+ cp .env.example .env # fill in values
36
+ uvicorn app.main:app --reload --port 8000
37
+ ```
 
38
 
39
+ ### Environment Variables
 
 
 
40
 
41
+ | Variable | Description | Example |
42
+ |----------|-------------|---------|
43
+ | `OLLAMA_HOST` | Ollama server URL | `https://ollama.com` |
44
+ | `OLLAMA_API_KEY` | Ollama cloud API key | `abc123...` |
45
+ | `OLLAMA_MODEL` | Model name | `gpt-oss:120b-cloud` |
46
+ | `REDIS_URL` | Redis connection string (use `rediss://` for TLS) | `rediss://default:pass@host:port` |
47
+ | `CORS_ORIGINS` | Comma-separated allowed origins | `http://localhost:3000` |
48
+ | `EMBEDDING_MODEL` | sentence-transformers model | `all-MiniLM-L6-v2` |
49
+ | `BM25_PREFILTER_THRESHOLD` | Min candidates before BM25 activates | `200` |
50
+ | `BM25_TOP_K` | Candidates kept after BM25 | `500` |
51
+ | `LOG_LEVEL` | Logging level | `INFO` |
52
 
53
+ > `REDIS_URL` with TLS (Upstash) must use `rediss://` (double-s). `redis://` will fail with SSL EOF errors.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
54
 
55
  ## API Reference
56
 
 
87
  β†’ full score breakdown + AI explanation (generate=true triggers Ollama)
88
  ```
89
 
90
+ ## Scoring Breakdown
91
 
92
  ```
93
  Hard filter (engineer type + years of experience)
 
115
  |--------|-------------|
116
  | `name` | Full name |
117
  | `current_position` | Job title / role |
118
+ | `engineer_type` | `"AI Engineer"`, `"Backend Engineer"`, etc. |
119
  | `parsed_skills` | Skills text |
120
  | `parsed_summary` | Professional summary |
121
  | `parsed_work_experience` | Stringified Python list of `{company, role, duration, description}` dicts |
 
123
 
124
  Optional: `gen_ai_experience`, `current_title`, `is_actively_or_passively_looking`, `notice_period`, `parsed_metadata_most_recent_company`
125
 
126
+ ## Storage
127
+
128
+ Candidate JSON lives in process memory, not Redis β€” avoids Upstash free tier 10MB value limit. Process restart clears candidates; re-upload required. Match results and metadata stored in Redis with 24–48h TTL.
129
 
130
+ If Redis is unavailable, falls back to `_MemStore` (in-process dict with TTL). System works without any external dependencies.
131
 
132
+ ## Scaling to 100,000 Candidates
133
+
134
+ ### What already helps at scale
135
+
136
+ - **Async background tasks** β€” API stays responsive; client polls for status
137
+ - **BM25 pre-filter** β€” skips expensive embedding for bulk of candidates when pool > 200
138
+ - **Local embeddings** β€” no network latency or API cost per request
139
+
140
+ ### What would need to change
141
+
142
+ **Distributed task queue** β€” replace `BackgroundTasks` with Celery + RabbitMQ/Redis broker to distribute matching jobs across worker nodes
143
+
144
+ **Vector database** β€” pre-compute and store candidate embeddings in Pinecone/Weaviate/Milvus instead of generating on-the-fly. Matching becomes a nearest-neighbour lookup instead of a full scan
145
+
146
+ **Microservices split** β€” break monolith into Parser, Matching, Explanation, and API Gateway services for independent scaling
147
+
148
+ **Kubernetes** β€” container orchestration for auto-scaling, service discovery, and load balancing across the above services
149
+
150
+ ```mermaid
151
+ graph TD
152
+ subgraph "User"
153
+ A[Recruiter]
154
+ end
155
+
156
+ subgraph "API Layer"
157
+ B[API Gateway]
158
+ end
159
+
160
+ subgraph "Services"
161
+ C[Parser Service]
162
+ D[Matching Service]
163
+ E[Explanation Service]
164
+ end
165
+
166
+ subgraph "Data & AI Layer"
167
+ F[Vector Database e.g., Pinecone]
168
+ G[Task Queue e.g., Celery + RabbitMQ]
169
+ H[Redis Cache]
170
+ I[Ollama LLM]
171
+ end
172
+
173
+ A --> B
174
+ B --> C
175
+ B --> D
176
+ B --> E
177
+
178
+ C --> G
179
+ D -- Reads from --> F
180
+ D -- Uses --> G
181
+ D -- Caches in --> H
182
+ E -- Calls --> I
183
+ ```