Spaces:
Sleeping
Sleeping
File size: 6,433 Bytes
d31b9a2 74d1c40 ff7b177 74d1c40 ff7b177 74d1c40 ff7b177 74d1c40 ff7b177 74d1c40 ff7b177 74d1c40 ff7b177 74d1c40 ff7b177 74d1c40 ff7b177 74d1c40 ff7b177 74d1c40 ff7b177 74d1c40 ff7b177 74d1c40 ff7b177 74d1c40 ff7b177 74d1c40 ff7b177 74d1c40 ff7b177 74d1c40 ff7b177 74d1c40 ff7b177 74d1c40 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 | ---
title: Candidate Matching Backend
emoji: π€
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: false
---
# Backend β Candidate Matching System
FastAPI backend for the candidate matching task. Parses JD PDFs and candidate XLSX files, runs a hybrid BM25 + semantic matching pipeline, and returns ranked candidates with optional AI-generated explanations via Ollama.
## How It Works
Matching runs as a background task through four stages:
1. **Hard filter** β removes candidates that don't meet engineer type or minimum years of experience
2. **BM25 pre-filter** β activated only when candidate pool exceeds 200; narrows down to top 500 before embedding
3. **Semantic matching** β runs `all-MiniLM-L6-v2` locally to embed candidate work history and JD requirements
4. **Scoring** β combines proficiency-aware skill matching, semantic similarity, title alignment, and domain score; applies penalties; stores ranked results in Redis
## Tech Stack
| Category | Technology |
| --- | --- |
| Framework | FastAPI + Uvicorn |
| NLP & Matching | Sentence-Transformers, Rank-BM25, PyMuPDF |
| Data Parsing | Pandas, Openpyxl |
| AI Explanations | Ollama |
| Caching & State | Redis |
| Config | Pydantic Settings |
## Setup
### Prerequisites
- Python 3.11+
- Redis (or use Docker setup from root)
### Run Locally
```bash
pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000
```
### Environment Variables
| Variable | Description | Example |
|----------|-------------|---------|
| `OLLAMA_HOST` | Ollama server URL | `https://ollama.com` |
| `OLLAMA_API_KEY` | Ollama cloud API key | `abc123...` |
| `OLLAMA_MODEL` | Model name | `gpt-oss:120b-cloud` |
| `REDIS_URL` | Redis connection string (use `rediss://` for TLS) | `rediss://default:pass@host:port` |
| `CORS_ORIGINS` | Comma-separated allowed origins | `http://localhost:3000` |
| `EMBEDDING_MODEL` | sentence-transformers model | `all-MiniLM-L6-v2` |
| `BM25_PREFILTER_THRESHOLD` | Min candidates before BM25 activates | `200` |
| `BM25_TOP_K` | Candidates kept after BM25 | `500` |
| `LOG_LEVEL` | Logging level | `INFO` |
> `REDIS_URL` with TLS (Upstash) must use `rediss://` (double-s). `redis://` will fail with SSL EOF errors.
## API Reference
```
GET /health
β {status, redis, embedding_model_loaded}
POST /api/v1/jd/upload multipart: file=<PDF>
β {jd_id, title, engineer_type, min_years, required_skills_count}
GET /api/v1/jd/
β [{jd_id, title, engineer_type, uploaded_at}, ...]
GET /api/v1/jd/{jd_id}
β full JD with skills and responsibilities
POST /api/v1/candidates/upload multipart: file=<XLSX>
β {batch_id, total_count, engineer_type_distribution}
GET /api/v1/candidates/
β [{batch_id, total_count, uploaded_at}, ...]
POST /api/v1/match/{jd_id} body: {batch_id, top_n}
β {match_id, status: "pending"} 202 Accepted
GET /api/v1/status/{match_id}
β {status: "pending"|"running"|"completed"|"failed", progress, message}
GET /api/v1/match/{jd_id}?batch_id=&top_n=
β [{rank, candidate_id, name, final_score, tier, matched_skills,
missing_skills, below_level_skills, flags, ...}, ...]
GET /api/v1/match/{jd_id}/{candidate_id}?batch_id=&generate=true
β full score breakdown + AI explanation (generate=true triggers Ollama)
```
## Scoring Breakdown
```
Hard filter (engineer type + years of experience)
β
BM25 pre-filter (only if candidates > 200 β keeps top 500)
β
Local embeddings (all-MiniLM-L6-v2, candidate work history only)
β
Per-candidate score:
35% β proficiency-aware skill matching
40% β semantic similarity (JD vs work history)
10% β position title alignment
15% β domain score (GenAI depth for AI JDs, stack for Backend)
β
Penalties applied (type mismatch, years gap, missing fields)
β
Ranked β Redis ZSET β top-N retrieval O(log N)
```
## XLSX Format
Required columns:
| Column | Description |
|--------|-------------|
| `name` | Full name |
| `current_position` | Job title / role |
| `engineer_type` | `"AI Engineer"`, `"Backend Engineer"`, etc. |
| `parsed_skills` | Skills text |
| `parsed_summary` | Professional summary |
| `parsed_work_experience` | Stringified Python list of `{company, role, duration, description}` dicts |
| `parsed_metadata_calculated_years_experience` | Years of experience (float) |
Optional: `gen_ai_experience`, `current_title`, `is_actively_or_passively_looking`, `notice_period`, `parsed_metadata_most_recent_company`
## Storage
Candidate JSON lives in process memory, not Redis β avoids Upstash free tier 10MB value limit. Process restart clears candidates; re-upload required. Match results and metadata stored in Redis with 24β48h TTL.
If Redis is unavailable, falls back to `_MemStore` (in-process dict with TTL). System works without any external dependencies.
## Scaling to 100,000 Candidates
### What already helps at scale
- **Async background tasks** β API stays responsive; client polls for status
- **BM25 pre-filter** β skips expensive embedding for bulk of candidates when pool > 200
- **Local embeddings** β no network latency or API cost per request
### What would need to change
**Distributed task queue** β replace `BackgroundTasks` with Celery + RabbitMQ/Redis broker to distribute matching jobs across worker nodes
**Vector database** β pre-compute and store candidate embeddings in Pinecone/Weaviate/Milvus instead of generating on-the-fly. Matching becomes a nearest-neighbour lookup instead of a full scan
**Microservices split** β break monolith into Parser, Matching, Explanation, and API Gateway services for independent scaling
**Kubernetes** β container orchestration for auto-scaling, service discovery, and load balancing across the above services
```mermaid
graph TD
subgraph "User"
A[Recruiter]
end
subgraph "API Layer"
B[API Gateway]
end
subgraph "Services"
C[Parser Service]
D[Matching Service]
E[Explanation Service]
end
subgraph "Data & AI Layer"
F[Vector Database e.g., Pinecone]
G[Task Queue e.g., Celery + RabbitMQ]
H[Redis Cache]
I[Ollama LLM]
end
A --> B
B --> C
B --> D
B --> E
C --> G
D -- Reads from --> F
D -- Uses --> G
D -- Caches in --> H
E -- Calls --> I
```
|