File size: 6,433 Bytes
d31b9a2
 
 
 
 
 
 
 
 
74d1c40
ff7b177
74d1c40
ff7b177
74d1c40
ff7b177
74d1c40
ff7b177
74d1c40
 
 
 
ff7b177
 
 
74d1c40
 
 
 
 
 
 
 
ff7b177
74d1c40
ff7b177
74d1c40
 
 
ff7b177
74d1c40
ff7b177
74d1c40
 
 
 
ff7b177
74d1c40
ff7b177
74d1c40
 
 
 
 
 
 
 
 
 
 
ff7b177
74d1c40
ff7b177
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
74d1c40
ff7b177
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
74d1c40
ff7b177
 
 
 
 
 
 
74d1c40
 
 
ff7b177
74d1c40
ff7b177
74d1c40
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
---
title: Candidate Matching Backend
emoji: πŸ€–
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: false
---

# Backend β€” Candidate Matching System

FastAPI backend for the candidate matching task. Parses JD PDFs and candidate XLSX files, runs a hybrid BM25 + semantic matching pipeline, and returns ranked candidates with optional AI-generated explanations via Ollama.

## How It Works

Matching runs as a background task through four stages:

1. **Hard filter** β€” removes candidates that don't meet engineer type or minimum years of experience
2. **BM25 pre-filter** β€” activated only when candidate pool exceeds 200; narrows down to top 500 before embedding
3. **Semantic matching** β€” runs `all-MiniLM-L6-v2` locally to embed candidate work history and JD requirements
4. **Scoring** β€” combines proficiency-aware skill matching, semantic similarity, title alignment, and domain score; applies penalties; stores ranked results in Redis

## Tech Stack

| Category | Technology |
| --- | --- |
| Framework | FastAPI + Uvicorn |
| NLP & Matching | Sentence-Transformers, Rank-BM25, PyMuPDF |
| Data Parsing | Pandas, Openpyxl |
| AI Explanations | Ollama |
| Caching & State | Redis |
| Config | Pydantic Settings |

## Setup

### Prerequisites
- Python 3.11+
- Redis (or use Docker setup from root)

### Run Locally

```bash
pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000
```

### Environment Variables

| Variable | Description | Example |
|----------|-------------|---------|
| `OLLAMA_HOST` | Ollama server URL | `https://ollama.com` |
| `OLLAMA_API_KEY` | Ollama cloud API key | `abc123...` |
| `OLLAMA_MODEL` | Model name | `gpt-oss:120b-cloud` |
| `REDIS_URL` | Redis connection string (use `rediss://` for TLS) | `rediss://default:pass@host:port` |
| `CORS_ORIGINS` | Comma-separated allowed origins | `http://localhost:3000` |
| `EMBEDDING_MODEL` | sentence-transformers model | `all-MiniLM-L6-v2` |
| `BM25_PREFILTER_THRESHOLD` | Min candidates before BM25 activates | `200` |
| `BM25_TOP_K` | Candidates kept after BM25 | `500` |
| `LOG_LEVEL` | Logging level | `INFO` |

> `REDIS_URL` with TLS (Upstash) must use `rediss://` (double-s). `redis://` will fail with SSL EOF errors.

## API Reference

```
GET  /health
     β†’ {status, redis, embedding_model_loaded}

POST /api/v1/jd/upload                    multipart: file=<PDF>
     β†’ {jd_id, title, engineer_type, min_years, required_skills_count}

GET  /api/v1/jd/
     β†’ [{jd_id, title, engineer_type, uploaded_at}, ...]

GET  /api/v1/jd/{jd_id}
     β†’ full JD with skills and responsibilities

POST /api/v1/candidates/upload            multipart: file=<XLSX>
     β†’ {batch_id, total_count, engineer_type_distribution}

GET  /api/v1/candidates/
     β†’ [{batch_id, total_count, uploaded_at}, ...]

POST /api/v1/match/{jd_id}               body: {batch_id, top_n}
     β†’ {match_id, status: "pending"}     202 Accepted

GET  /api/v1/status/{match_id}
     β†’ {status: "pending"|"running"|"completed"|"failed", progress, message}

GET  /api/v1/match/{jd_id}?batch_id=&top_n=
     β†’ [{rank, candidate_id, name, final_score, tier, matched_skills,
         missing_skills, below_level_skills, flags, ...}, ...]

GET  /api/v1/match/{jd_id}/{candidate_id}?batch_id=&generate=true
     β†’ full score breakdown + AI explanation (generate=true triggers Ollama)
```

## Scoring Breakdown

```
Hard filter (engineer type + years of experience)
    ↓
BM25 pre-filter (only if candidates > 200 β†’ keeps top 500)
    ↓
Local embeddings (all-MiniLM-L6-v2, candidate work history only)
    ↓
Per-candidate score:
  35% β€” proficiency-aware skill matching
  40% β€” semantic similarity (JD vs work history)
  10% β€” position title alignment
  15% β€” domain score (GenAI depth for AI JDs, stack for Backend)
    ↓
Penalties applied (type mismatch, years gap, missing fields)
    ↓
Ranked β†’ Redis ZSET β†’ top-N retrieval O(log N)
```

## XLSX Format

Required columns:

| Column | Description |
|--------|-------------|
| `name` | Full name |
| `current_position` | Job title / role |
| `engineer_type` | `"AI Engineer"`, `"Backend Engineer"`, etc. |
| `parsed_skills` | Skills text |
| `parsed_summary` | Professional summary |
| `parsed_work_experience` | Stringified Python list of `{company, role, duration, description}` dicts |
| `parsed_metadata_calculated_years_experience` | Years of experience (float) |

Optional: `gen_ai_experience`, `current_title`, `is_actively_or_passively_looking`, `notice_period`, `parsed_metadata_most_recent_company`

## Storage

Candidate JSON lives in process memory, not Redis β€” avoids Upstash free tier 10MB value limit. Process restart clears candidates; re-upload required. Match results and metadata stored in Redis with 24–48h TTL.

If Redis is unavailable, falls back to `_MemStore` (in-process dict with TTL). System works without any external dependencies.

## Scaling to 100,000 Candidates

### What already helps at scale

- **Async background tasks** β€” API stays responsive; client polls for status
- **BM25 pre-filter** β€” skips expensive embedding for bulk of candidates when pool > 200
- **Local embeddings** β€” no network latency or API cost per request

### What would need to change

**Distributed task queue** β€” replace `BackgroundTasks` with Celery + RabbitMQ/Redis broker to distribute matching jobs across worker nodes

**Vector database** β€” pre-compute and store candidate embeddings in Pinecone/Weaviate/Milvus instead of generating on-the-fly. Matching becomes a nearest-neighbour lookup instead of a full scan

**Microservices split** β€” break monolith into Parser, Matching, Explanation, and API Gateway services for independent scaling

**Kubernetes** β€” container orchestration for auto-scaling, service discovery, and load balancing across the above services

```mermaid
graph TD
    subgraph "User"
        A[Recruiter]
    end

    subgraph "API Layer"
        B[API Gateway]
    end

    subgraph "Services"
        C[Parser Service]
        D[Matching Service]
        E[Explanation Service]
    end

    subgraph "Data & AI Layer"
        F[Vector Database e.g., Pinecone]
        G[Task Queue e.g., Celery + RabbitMQ]
        H[Redis Cache]
        I[Ollama LLM]
    end

    A --> B
    B --> C
    B --> D
    B --> E

    C --> G
    D -- Reads from --> F
    D -- Uses --> G
    D -- Caches in --> H
    E -- Calls --> I
```