Spaces:
Sleeping
Sleeping
readme changes
Browse files
README.md
CHANGED
|
@@ -1,176 +1,56 @@
|
|
| 1 |
-
#
|
| 2 |
|
| 3 |
-
|
| 4 |
|
| 5 |
-
##
|
| 6 |
|
| 7 |
-
|
| 8 |
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
3. **Semantic Matching:** The filtered candidates are then processed through a semantic search pipeline using sentence transformers (`all-MiniLM-L6-v2`). This model, run locally, generates embeddings for candidate experience and JD requirements to find semantic similarities.
|
| 14 |
-
4. **Proficiency-Aware Scoring:** The system doesn't just check for skill presence; it scores based on proficiency levels (e.g., "Expert" vs. "Familiar"), providing a more nuanced evaluation.
|
| 15 |
-
5. **AI-Powered Explanations:** For top matches, the system can generate a detailed explanation of why a candidate is a good fit, using a large language model hosted on Ollama Cloud.
|
| 16 |
-
|
| 17 |
-
The entire process is exposed via a RESTful API built with FastAPI, featuring asynchronous processing for long-running matching tasks.
|
| 18 |
|
| 19 |
## Tech Stack
|
| 20 |
|
| 21 |
-
| Category | Technology |
|
| 22 |
-
| --- | --- |
|
| 23 |
-
|
|
| 24 |
-
|
|
| 25 |
-
|
|
| 26 |
-
|
|
| 27 |
-
|
|
| 28 |
-
|
|
| 29 |
-
| **AI Explanations** | Ollama | For generating natural language explanations of matches. |
|
| 30 |
-
| **Caching & State** | Redis | Used for caching results and managing task status. |
|
| 31 |
-
| **Configuration** | Pydantic | For settings management and data validation. |
|
| 32 |
-
|
| 33 |
-
## Scaling to 100,000 Candidates
|
| 34 |
-
|
| 35 |
-
The current architecture is designed with scalability in mind. Hereβs how it would handle a load of 100,000 candidates and the changes that would be required to ensure performance.
|
| 36 |
-
|
| 37 |
-
### Current Scalability Features:
|
| 38 |
-
|
| 39 |
-
* **Asynchronous Processing:** Matching jobs are run as background tasks, so the API remains responsive. The client can poll for results.
|
| 40 |
-
* **BM25 Pre-filtering:** The pipeline intelligently activates a BM25 pre-filter when the candidate pool exceeds 200. This significantly reduces the number of candidates that need to go through the more computationally expensive embedding and scoring stages.
|
| 41 |
-
* **Efficient Embeddings:** Using a lightweight local model like `all-MiniLM-L6-v2` avoids network latency and API costs associated with external embedding services.
|
| 42 |
-
|
| 43 |
-
### Architectural Changes for 100k+ Scale:
|
| 44 |
-
|
| 45 |
-
1. **Distributed Task Queue:**
|
| 46 |
-
* **Current:** FastAPI's `BackgroundTasks` is suitable for a single-server setup.
|
| 47 |
-
* **Proposed Change:** Replace `BackgroundTasks` with a dedicated distributed task queue like **Celery** with a **RabbitMQ** or **Redis** broker. This would allow us to distribute matching jobs across multiple worker nodes, scaling horizontally.
|
| 48 |
-
|
| 49 |
-
2. **Vector Database:**
|
| 50 |
-
* **Current:** Embeddings are generated on-the-fly and stored in memory or a simple cache.
|
| 51 |
-
* **Proposed Change:** Pre-compute and store candidate embeddings in a dedicated **vector database** like **Pinecone, Weaviate, or Milvus**. This would transform the matching process from a "search" to a "lookup," drastically reducing latency. The workflow would be:
|
| 52 |
-
1. When a candidate is uploaded, their profile is parsed, embedded, and stored in the vector DB.
|
| 53 |
-
2. When a JD is uploaded, its requirements are embedded.
|
| 54 |
-
3. The JD embedding is used to query the vector DB to find the top-K most similar candidates in milliseconds.
|
| 55 |
-
|
| 56 |
-
3. **Dedicated Microservices:**
|
| 57 |
-
* **Current:** The application is a monolith.
|
| 58 |
-
* **Proposed Change:** Break down the application into microservices:
|
| 59 |
-
* `Parser Service`: Handles file parsing (PDF, XLSX).
|
| 60 |
-
* `Matching Service`: Contains the core matching logic and workers.
|
| 61 |
-
* `API Gateway`: The main entry point for all requests.
|
| 62 |
-
* `Explanation Service`: Manages interaction with the Ollama LLM.
|
| 63 |
-
This would allow independent scaling of each component. For example, we could scale up the `Matching Service` workers during peak hiring seasons without touching other services.
|
| 64 |
|
| 65 |
-
|
| 66 |
-
* **Current:** The app is containerized with Docker and can be deployed on a platform like Fly.io.
|
| 67 |
-
* **Proposed Change:** Use **Kubernetes** to manage the containerized microservices. Kubernetes would handle auto-scaling, service discovery, and load balancing automatically.
|
| 68 |
|
| 69 |
-
###
|
|
|
|
|
|
|
| 70 |
|
| 71 |
-
|
| 72 |
-
graph TD
|
| 73 |
-
subgraph "User"
|
| 74 |
-
A[Recruiter]
|
| 75 |
-
end
|
| 76 |
-
|
| 77 |
-
subgraph "API Layer"
|
| 78 |
-
B[API Gateway]
|
| 79 |
-
end
|
| 80 |
-
|
| 81 |
-
subgraph "Services"
|
| 82 |
-
C[Parser Service]
|
| 83 |
-
D[Matching Service]
|
| 84 |
-
E[Explanation Service]
|
| 85 |
-
end
|
| 86 |
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
end
|
| 93 |
|
| 94 |
-
|
| 95 |
-
B --> C
|
| 96 |
-
B --> D
|
| 97 |
-
B --> E
|
| 98 |
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 105 |
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
### Sample Job Description
|
| 109 |
-
|
| 110 |
-
**Role:** Senior Backend Engineer (Python)
|
| 111 |
-
**Experience:** 5+ years
|
| 112 |
-
**Required Skills:**
|
| 113 |
-
* Python (Expert)
|
| 114 |
-
* Django or FastAPI (Expert)
|
| 115 |
-
* PostgreSQL (Advanced)
|
| 116 |
-
* Redis (Advanced)
|
| 117 |
-
* Docker (Advanced)
|
| 118 |
-
* Celery (Intermediate)
|
| 119 |
-
|
| 120 |
-
**Preferred Skills:**
|
| 121 |
-
* Kubernetes (Intermediate)
|
| 122 |
-
* Terraform (Familiar)
|
| 123 |
-
* AWS or GCP (Advanced)
|
| 124 |
-
|
| 125 |
-
### Sample Candidate Profiles
|
| 126 |
-
|
| 127 |
-
**Candidate 1: Strong Match**
|
| 128 |
-
* **Experience:** 7 years as a Backend Engineer.
|
| 129 |
-
* **Skills:** Python (Expert), FastAPI (Expert), PostgreSQL (Advanced), Redis (Expert), Docker (Advanced), Kubernetes (Intermediate), AWS (Advanced).
|
| 130 |
-
* **Summary:** Built and scaled a microservices-based system using FastAPI, deployed on AWS with Kubernetes.
|
| 131 |
-
|
| 132 |
-
**Candidate 2: Medium Match**
|
| 133 |
-
* **Experience:** 4 years as a Full-Stack Developer.
|
| 134 |
-
* **Skills:** Python (Advanced), Django (Advanced), MySQL (Advanced), Docker (Intermediate), Heroku (Advanced).
|
| 135 |
-
* **Summary:** Developed monolithic web applications with Django. Has some containerization experience but lacks cloud-native and distributed systems knowledge.
|
| 136 |
-
|
| 137 |
-
**Candidate 3: Low Match**
|
| 138 |
-
* **Experience:** 2 years as a Data Analyst.
|
| 139 |
-
* **Skills:** Python (Intermediate), Pandas (Expert), SQL (Advanced), Tableau (Expert).
|
| 140 |
-
* **Summary:** Focused on data analysis and visualization. Limited backend development experience.
|
| 141 |
-
|
| 142 |
-
## Setup and Deployment
|
| 143 |
-
|
| 144 |
-
Follow the instructions below to set up the project locally or deploy it.
|
| 145 |
-
|
| 146 |
-
### Local Setup
|
| 147 |
-
|
| 148 |
-
1. **Prerequisites:** Python 3.11+
|
| 149 |
-
2. **Install dependencies:**
|
| 150 |
-
```bash
|
| 151 |
-
pip install -r requirements.txt
|
| 152 |
-
```
|
| 153 |
-
3. **Configure Environment:**
|
| 154 |
-
Create a `.env` file.
|
| 155 |
-
4. **Run the server:**
|
| 156 |
-
```bash
|
| 157 |
-
uvicorn app.main:app --reload --port 8000
|
| 158 |
-
```
|
| 159 |
-
|
| 160 |
-
|
| 161 |
-
| Variable | Required | Description | Example |
|
| 162 |
-
|----------|----------|-------------|---------|
|
| 163 |
-
| `OLLAMA_HOST` | No | Ollama server URL | `https://ollama.com` |
|
| 164 |
-
| `OLLAMA_API_KEY` | No | Ollama cloud API key | `abc123...` |
|
| 165 |
-
| `OLLAMA_MODEL` | No | Model name | `gpt-oss:120b-cloud` |
|
| 166 |
-
| `REDIS_URL` | No | Redis connection string (use `rediss://` for TLS) | `rediss://default:pass@host:port` |
|
| 167 |
-
| `CORS_ORIGINS` | Yes | Comma-separated allowed origins | `http://localhost:3000` |
|
| 168 |
-
| `EMBEDDING_MODEL` | No | sentence-transformers model | `all-MiniLM-L6-v2` |
|
| 169 |
-
| `BM25_PREFILTER_THRESHOLD` | No | Min candidates before BM25 activates | `200` |
|
| 170 |
-
| `BM25_TOP_K` | No | Candidates kept after BM25 | `500` |
|
| 171 |
-
| `LOG_LEVEL` | No | Logging level | `INFO` |
|
| 172 |
-
|
| 173 |
-
> **Redis URL**: Upstash requires `rediss://` (double-s) for TLS. Using `redis://` will fail with SSL EOF errors.
|
| 174 |
|
| 175 |
## API Reference
|
| 176 |
|
|
@@ -207,7 +87,7 @@ GET /api/v1/match/{jd_id}/{candidate_id}?batch_id=&generate=true
|
|
| 207 |
β full score breakdown + AI explanation (generate=true triggers Ollama)
|
| 208 |
```
|
| 209 |
|
| 210 |
-
## Scoring
|
| 211 |
|
| 212 |
```
|
| 213 |
Hard filter (engineer type + years of experience)
|
|
@@ -235,7 +115,7 @@ Required columns:
|
|
| 235 |
|--------|-------------|
|
| 236 |
| `name` | Full name |
|
| 237 |
| `current_position` | Job title / role |
|
| 238 |
-
| `engineer_type` | "AI Engineer", "Backend Engineer", etc. |
|
| 239 |
| `parsed_skills` | Skills text |
|
| 240 |
| `parsed_summary` | Professional summary |
|
| 241 |
| `parsed_work_experience` | Stringified Python list of `{company, role, duration, description}` dicts |
|
|
@@ -243,8 +123,61 @@ Required columns:
|
|
| 243 |
|
| 244 |
Optional: `gen_ai_experience`, `current_title`, `is_actively_or_passively_looking`, `notice_period`, `parsed_metadata_most_recent_company`
|
| 245 |
|
| 246 |
-
## Storage
|
|
|
|
|
|
|
| 247 |
|
| 248 |
-
|
| 249 |
|
| 250 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Backend β Candidate Matching System
|
| 2 |
|
| 3 |
+
FastAPI backend for the candidate matching task. Parses JD PDFs and candidate XLSX files, runs a hybrid BM25 + semantic matching pipeline, and returns ranked candidates with optional AI-generated explanations via Ollama.
|
| 4 |
|
| 5 |
+
## How It Works
|
| 6 |
|
| 7 |
+
Matching runs as a background task through four stages:
|
| 8 |
|
| 9 |
+
1. **Hard filter** β removes candidates that don't meet engineer type or minimum years of experience
|
| 10 |
+
2. **BM25 pre-filter** β activated only when candidate pool exceeds 200; narrows down to top 500 before embedding
|
| 11 |
+
3. **Semantic matching** β runs `all-MiniLM-L6-v2` locally to embed candidate work history and JD requirements
|
| 12 |
+
4. **Scoring** β combines proficiency-aware skill matching, semantic similarity, title alignment, and domain score; applies penalties; stores ranked results in Redis
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
|
| 14 |
## Tech Stack
|
| 15 |
|
| 16 |
+
| Category | Technology |
|
| 17 |
+
| --- | --- |
|
| 18 |
+
| Framework | FastAPI + Uvicorn |
|
| 19 |
+
| NLP & Matching | Sentence-Transformers, Rank-BM25, PyMuPDF |
|
| 20 |
+
| Data Parsing | Pandas, Openpyxl |
|
| 21 |
+
| AI Explanations | Ollama |
|
| 22 |
+
| Caching & State | Redis |
|
| 23 |
+
| Config | Pydantic Settings |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
|
| 25 |
+
## Setup
|
|
|
|
|
|
|
| 26 |
|
| 27 |
+
### Prerequisites
|
| 28 |
+
- Python 3.11+
|
| 29 |
+
- Redis (or use Docker setup from root)
|
| 30 |
|
| 31 |
+
### Run Locally
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
|
| 33 |
+
```bash
|
| 34 |
+
pip install -r requirements.txt
|
| 35 |
+
cp .env.example .env # fill in values
|
| 36 |
+
uvicorn app.main:app --reload --port 8000
|
| 37 |
+
```
|
|
|
|
| 38 |
|
| 39 |
+
### Environment Variables
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
+
| Variable | Description | Example |
|
| 42 |
+
|----------|-------------|---------|
|
| 43 |
+
| `OLLAMA_HOST` | Ollama server URL | `https://ollama.com` |
|
| 44 |
+
| `OLLAMA_API_KEY` | Ollama cloud API key | `abc123...` |
|
| 45 |
+
| `OLLAMA_MODEL` | Model name | `gpt-oss:120b-cloud` |
|
| 46 |
+
| `REDIS_URL` | Redis connection string (use `rediss://` for TLS) | `rediss://default:pass@host:port` |
|
| 47 |
+
| `CORS_ORIGINS` | Comma-separated allowed origins | `http://localhost:3000` |
|
| 48 |
+
| `EMBEDDING_MODEL` | sentence-transformers model | `all-MiniLM-L6-v2` |
|
| 49 |
+
| `BM25_PREFILTER_THRESHOLD` | Min candidates before BM25 activates | `200` |
|
| 50 |
+
| `BM25_TOP_K` | Candidates kept after BM25 | `500` |
|
| 51 |
+
| `LOG_LEVEL` | Logging level | `INFO` |
|
| 52 |
|
| 53 |
+
> `REDIS_URL` with TLS (Upstash) must use `rediss://` (double-s). `redis://` will fail with SSL EOF errors.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
|
| 55 |
## API Reference
|
| 56 |
|
|
|
|
| 87 |
β full score breakdown + AI explanation (generate=true triggers Ollama)
|
| 88 |
```
|
| 89 |
|
| 90 |
+
## Scoring Breakdown
|
| 91 |
|
| 92 |
```
|
| 93 |
Hard filter (engineer type + years of experience)
|
|
|
|
| 115 |
|--------|-------------|
|
| 116 |
| `name` | Full name |
|
| 117 |
| `current_position` | Job title / role |
|
| 118 |
+
| `engineer_type` | `"AI Engineer"`, `"Backend Engineer"`, etc. |
|
| 119 |
| `parsed_skills` | Skills text |
|
| 120 |
| `parsed_summary` | Professional summary |
|
| 121 |
| `parsed_work_experience` | Stringified Python list of `{company, role, duration, description}` dicts |
|
|
|
|
| 123 |
|
| 124 |
Optional: `gen_ai_experience`, `current_title`, `is_actively_or_passively_looking`, `notice_period`, `parsed_metadata_most_recent_company`
|
| 125 |
|
| 126 |
+
## Storage
|
| 127 |
+
|
| 128 |
+
Candidate JSON lives in process memory, not Redis β avoids Upstash free tier 10MB value limit. Process restart clears candidates; re-upload required. Match results and metadata stored in Redis with 24β48h TTL.
|
| 129 |
|
| 130 |
+
If Redis is unavailable, falls back to `_MemStore` (in-process dict with TTL). System works without any external dependencies.
|
| 131 |
|
| 132 |
+
## Scaling to 100,000 Candidates
|
| 133 |
+
|
| 134 |
+
### What already helps at scale
|
| 135 |
+
|
| 136 |
+
- **Async background tasks** β API stays responsive; client polls for status
|
| 137 |
+
- **BM25 pre-filter** β skips expensive embedding for bulk of candidates when pool > 200
|
| 138 |
+
- **Local embeddings** β no network latency or API cost per request
|
| 139 |
+
|
| 140 |
+
### What would need to change
|
| 141 |
+
|
| 142 |
+
**Distributed task queue** β replace `BackgroundTasks` with Celery + RabbitMQ/Redis broker to distribute matching jobs across worker nodes
|
| 143 |
+
|
| 144 |
+
**Vector database** β pre-compute and store candidate embeddings in Pinecone/Weaviate/Milvus instead of generating on-the-fly. Matching becomes a nearest-neighbour lookup instead of a full scan
|
| 145 |
+
|
| 146 |
+
**Microservices split** β break monolith into Parser, Matching, Explanation, and API Gateway services for independent scaling
|
| 147 |
+
|
| 148 |
+
**Kubernetes** β container orchestration for auto-scaling, service discovery, and load balancing across the above services
|
| 149 |
+
|
| 150 |
+
```mermaid
|
| 151 |
+
graph TD
|
| 152 |
+
subgraph "User"
|
| 153 |
+
A[Recruiter]
|
| 154 |
+
end
|
| 155 |
+
|
| 156 |
+
subgraph "API Layer"
|
| 157 |
+
B[API Gateway]
|
| 158 |
+
end
|
| 159 |
+
|
| 160 |
+
subgraph "Services"
|
| 161 |
+
C[Parser Service]
|
| 162 |
+
D[Matching Service]
|
| 163 |
+
E[Explanation Service]
|
| 164 |
+
end
|
| 165 |
+
|
| 166 |
+
subgraph "Data & AI Layer"
|
| 167 |
+
F[Vector Database e.g., Pinecone]
|
| 168 |
+
G[Task Queue e.g., Celery + RabbitMQ]
|
| 169 |
+
H[Redis Cache]
|
| 170 |
+
I[Ollama LLM]
|
| 171 |
+
end
|
| 172 |
+
|
| 173 |
+
A --> B
|
| 174 |
+
B --> C
|
| 175 |
+
B --> D
|
| 176 |
+
B --> E
|
| 177 |
+
|
| 178 |
+
C --> G
|
| 179 |
+
D -- Reads from --> F
|
| 180 |
+
D -- Uses --> G
|
| 181 |
+
D -- Caches in --> H
|
| 182 |
+
E -- Calls --> I
|
| 183 |
+
```
|