AnishDev's picture
updated-readme for hf
d31b9a2
|
Raw History Blame Contribute Delete
6.43 kB
---
title: Candidate Matching Backend
emoji: πŸ€–
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: false
---
# Backend β€” Candidate Matching System
FastAPI backend for the candidate matching task. Parses JD PDFs and candidate XLSX files, runs a hybrid BM25 + semantic matching pipeline, and returns ranked candidates with optional AI-generated explanations via Ollama.
## How It Works
Matching runs as a background task through four stages:
1. **Hard filter** β€” removes candidates that don't meet engineer type or minimum years of experience
2. **BM25 pre-filter** β€” activated only when candidate pool exceeds 200; narrows down to top 500 before embedding
3. **Semantic matching** β€” runs `all-MiniLM-L6-v2` locally to embed candidate work history and JD requirements
4. **Scoring** β€” combines proficiency-aware skill matching, semantic similarity, title alignment, and domain score; applies penalties; stores ranked results in Redis
## Tech Stack
| Category | Technology |
| --- | --- |
| Framework | FastAPI + Uvicorn |
| NLP & Matching | Sentence-Transformers, Rank-BM25, PyMuPDF |
| Data Parsing | Pandas, Openpyxl |
| AI Explanations | Ollama |
| Caching & State | Redis |
| Config | Pydantic Settings |
## Setup
### Prerequisites
- Python 3.11+
- Redis (or use Docker setup from root)
### Run Locally
```bash
pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000
```
### Environment Variables
| Variable | Description | Example |
|----------|-------------|---------|
| `OLLAMA_HOST` | Ollama server URL | `https://ollama.com` |
| `OLLAMA_API_KEY` | Ollama cloud API key | `abc123...` |
| `OLLAMA_MODEL` | Model name | `gpt-oss:120b-cloud` |
| `REDIS_URL` | Redis connection string (use `rediss://` for TLS) | `rediss://default:pass@host:port` |
| `CORS_ORIGINS` | Comma-separated allowed origins | `http://localhost:3000` |
| `EMBEDDING_MODEL` | sentence-transformers model | `all-MiniLM-L6-v2` |
| `BM25_PREFILTER_THRESHOLD` | Min candidates before BM25 activates | `200` |
| `BM25_TOP_K` | Candidates kept after BM25 | `500` |
| `LOG_LEVEL` | Logging level | `INFO` |
> `REDIS_URL` with TLS (Upstash) must use `rediss://` (double-s). `redis://` will fail with SSL EOF errors.
## API Reference
```
GET /health
β†’ {status, redis, embedding_model_loaded}
POST /api/v1/jd/upload multipart: file=<PDF>
β†’ {jd_id, title, engineer_type, min_years, required_skills_count}
GET /api/v1/jd/
β†’ [{jd_id, title, engineer_type, uploaded_at}, ...]
GET /api/v1/jd/{jd_id}
β†’ full JD with skills and responsibilities
POST /api/v1/candidates/upload multipart: file=<XLSX>
β†’ {batch_id, total_count, engineer_type_distribution}
GET /api/v1/candidates/
β†’ [{batch_id, total_count, uploaded_at}, ...]
POST /api/v1/match/{jd_id} body: {batch_id, top_n}
β†’ {match_id, status: "pending"} 202 Accepted
GET /api/v1/status/{match_id}
β†’ {status: "pending"|"running"|"completed"|"failed", progress, message}
GET /api/v1/match/{jd_id}?batch_id=&top_n=
β†’ [{rank, candidate_id, name, final_score, tier, matched_skills,
missing_skills, below_level_skills, flags, ...}, ...]
GET /api/v1/match/{jd_id}/{candidate_id}?batch_id=&generate=true
β†’ full score breakdown + AI explanation (generate=true triggers Ollama)
```
## Scoring Breakdown
```
Hard filter (engineer type + years of experience)
↓
BM25 pre-filter (only if candidates > 200 β†’ keeps top 500)
↓
Local embeddings (all-MiniLM-L6-v2, candidate work history only)
↓
Per-candidate score:
35% β€” proficiency-aware skill matching
40% β€” semantic similarity (JD vs work history)
10% β€” position title alignment
15% β€” domain score (GenAI depth for AI JDs, stack for Backend)
↓
Penalties applied (type mismatch, years gap, missing fields)
↓
Ranked β†’ Redis ZSET β†’ top-N retrieval O(log N)
```
## XLSX Format
Required columns:
| Column | Description |
|--------|-------------|
| `name` | Full name |
| `current_position` | Job title / role |
| `engineer_type` | `"AI Engineer"`, `"Backend Engineer"`, etc. |
| `parsed_skills` | Skills text |
| `parsed_summary` | Professional summary |
| `parsed_work_experience` | Stringified Python list of `{company, role, duration, description}` dicts |
| `parsed_metadata_calculated_years_experience` | Years of experience (float) |
Optional: `gen_ai_experience`, `current_title`, `is_actively_or_passively_looking`, `notice_period`, `parsed_metadata_most_recent_company`
## Storage
Candidate JSON lives in process memory, not Redis β€” avoids Upstash free tier 10MB value limit. Process restart clears candidates; re-upload required. Match results and metadata stored in Redis with 24–48h TTL.
If Redis is unavailable, falls back to `_MemStore` (in-process dict with TTL). System works without any external dependencies.
## Scaling to 100,000 Candidates
### What already helps at scale
- **Async background tasks** β€” API stays responsive; client polls for status
- **BM25 pre-filter** β€” skips expensive embedding for bulk of candidates when pool > 200
- **Local embeddings** β€” no network latency or API cost per request
### What would need to change
**Distributed task queue** β€” replace `BackgroundTasks` with Celery + RabbitMQ/Redis broker to distribute matching jobs across worker nodes
**Vector database** β€” pre-compute and store candidate embeddings in Pinecone/Weaviate/Milvus instead of generating on-the-fly. Matching becomes a nearest-neighbour lookup instead of a full scan
**Microservices split** β€” break monolith into Parser, Matching, Explanation, and API Gateway services for independent scaling
**Kubernetes** β€” container orchestration for auto-scaling, service discovery, and load balancing across the above services
```mermaid
graph TD
subgraph "User"
A[Recruiter]
end
subgraph "API Layer"
B[API Gateway]
end
subgraph "Services"
C[Parser Service]
D[Matching Service]
E[Explanation Service]
end
subgraph "Data & AI Layer"
F[Vector Database e.g., Pinecone]
G[Task Queue e.g., Celery + RabbitMQ]
H[Redis Cache]
I[Ollama LLM]
end
A --> B
B --> C
B --> D
B --> E
C --> G
D -- Reads from --> F
D -- Uses --> G
D -- Caches in --> H
E -- Calls --> I
```