--- title: Candidate Matching Backend emoji: 🤖 colorFrom: blue colorTo: indigo sdk: docker pinned: false --- # Backend — Candidate Matching System FastAPI backend for the candidate matching task. Parses JD PDFs and candidate XLSX files, runs a hybrid BM25 + semantic matching pipeline, and returns ranked candidates with optional AI-generated explanations via Ollama. ## How It Works Matching runs as a background task through four stages: 1. **Hard filter** — removes candidates that don't meet engineer type or minimum years of experience 2. **BM25 pre-filter** — activated only when candidate pool exceeds 200; narrows down to top 500 before embedding 3. **Semantic matching** — runs `all-MiniLM-L6-v2` locally to embed candidate work history and JD requirements 4. **Scoring** — combines proficiency-aware skill matching, semantic similarity, title alignment, and domain score; applies penalties; stores ranked results in Redis ## Tech Stack | Category | Technology | | --- | --- | | Framework | FastAPI + Uvicorn | | NLP & Matching | Sentence-Transformers, Rank-BM25, PyMuPDF | | Data Parsing | Pandas, Openpyxl | | AI Explanations | Ollama | | Caching & State | Redis | | Config | Pydantic Settings | ## Setup ### Prerequisites - Python 3.11+ - Redis (or use Docker setup from root) ### Run Locally ```bash pip install -r requirements.txt uvicorn app.main:app --reload --port 8000 ``` ### Environment Variables | Variable | Description | Example | |----------|-------------|---------| | `OLLAMA_HOST` | Ollama server URL | `https://ollama.com` | | `OLLAMA_API_KEY` | Ollama cloud API key | `abc123...` | | `OLLAMA_MODEL` | Model name | `gpt-oss:120b-cloud` | | `REDIS_URL` | Redis connection string (use `rediss://` for TLS) | `rediss://default:pass@host:port` | | `CORS_ORIGINS` | Comma-separated allowed origins | `http://localhost:3000` | | `EMBEDDING_MODEL` | sentence-transformers model | `all-MiniLM-L6-v2` | | `BM25_PREFILTER_THRESHOLD` | Min candidates before BM25 activates | `200` | | `BM25_TOP_K` | Candidates kept after BM25 | `500` | | `LOG_LEVEL` | Logging level | `INFO` | > `REDIS_URL` with TLS (Upstash) must use `rediss://` (double-s). `redis://` will fail with SSL EOF errors. ## API Reference ``` GET /health → {status, redis, embedding_model_loaded} POST /api/v1/jd/upload multipart: file= → {jd_id, title, engineer_type, min_years, required_skills_count} GET /api/v1/jd/ → [{jd_id, title, engineer_type, uploaded_at}, ...] GET /api/v1/jd/{jd_id} → full JD with skills and responsibilities POST /api/v1/candidates/upload multipart: file= → {batch_id, total_count, engineer_type_distribution} GET /api/v1/candidates/ → [{batch_id, total_count, uploaded_at}, ...] POST /api/v1/match/{jd_id} body: {batch_id, top_n} → {match_id, status: "pending"} 202 Accepted GET /api/v1/status/{match_id} → {status: "pending"|"running"|"completed"|"failed", progress, message} GET /api/v1/match/{jd_id}?batch_id=&top_n= → [{rank, candidate_id, name, final_score, tier, matched_skills, missing_skills, below_level_skills, flags, ...}, ...] GET /api/v1/match/{jd_id}/{candidate_id}?batch_id=&generate=true → full score breakdown + AI explanation (generate=true triggers Ollama) ``` ## Scoring Breakdown ``` Hard filter (engineer type + years of experience) ↓ BM25 pre-filter (only if candidates > 200 → keeps top 500) ↓ Local embeddings (all-MiniLM-L6-v2, candidate work history only) ↓ Per-candidate score: 35% — proficiency-aware skill matching 40% — semantic similarity (JD vs work history) 10% — position title alignment 15% — domain score (GenAI depth for AI JDs, stack for Backend) ↓ Penalties applied (type mismatch, years gap, missing fields) ↓ Ranked → Redis ZSET → top-N retrieval O(log N) ``` ## XLSX Format Required columns: | Column | Description | |--------|-------------| | `name` | Full name | | `current_position` | Job title / role | | `engineer_type` | `"AI Engineer"`, `"Backend Engineer"`, etc. | | `parsed_skills` | Skills text | | `parsed_summary` | Professional summary | | `parsed_work_experience` | Stringified Python list of `{company, role, duration, description}` dicts | | `parsed_metadata_calculated_years_experience` | Years of experience (float) | Optional: `gen_ai_experience`, `current_title`, `is_actively_or_passively_looking`, `notice_period`, `parsed_metadata_most_recent_company` ## Storage Candidate JSON lives in process memory, not Redis — avoids Upstash free tier 10MB value limit. Process restart clears candidates; re-upload required. Match results and metadata stored in Redis with 24–48h TTL. If Redis is unavailable, falls back to `_MemStore` (in-process dict with TTL). System works without any external dependencies. ## Scaling to 100,000 Candidates ### What already helps at scale - **Async background tasks** — API stays responsive; client polls for status - **BM25 pre-filter** — skips expensive embedding for bulk of candidates when pool > 200 - **Local embeddings** — no network latency or API cost per request ### What would need to change **Distributed task queue** — replace `BackgroundTasks` with Celery + RabbitMQ/Redis broker to distribute matching jobs across worker nodes **Vector database** — pre-compute and store candidate embeddings in Pinecone/Weaviate/Milvus instead of generating on-the-fly. Matching becomes a nearest-neighbour lookup instead of a full scan **Microservices split** — break monolith into Parser, Matching, Explanation, and API Gateway services for independent scaling **Kubernetes** — container orchestration for auto-scaling, service discovery, and load balancing across the above services ```mermaid graph TD subgraph "User" A[Recruiter] end subgraph "API Layer" B[API Gateway] end subgraph "Services" C[Parser Service] D[Matching Service] E[Explanation Service] end subgraph "Data & AI Layer" F[Vector Database e.g., Pinecone] G[Task Queue e.g., Celery + RabbitMQ] H[Redis Cache] I[Ollama LLM] end A --> B B --> C B --> D B --> E C --> G D -- Reads from --> F D -- Uses --> G D -- Caches in --> H E -- Calls --> I ```