Spaces:
Sleeping
Sleeping
|
Download README.md from AnishDev/coderound-task-backend: direct link, hf CLI and curl.
- Browser
- Download file 6.43 kB
-
https://huggingface.co/spaces/AnishDev/coderound-task-backend/resolve/main/README.md
- Command line
-
hf download hf://spaces/AnishDev/coderound-task-backend/README.md
-
curl -L -o README.md https://huggingface.co/spaces/AnishDev/coderound-task-backend/resolve/main/README.md
6.43 kB
| title: Candidate Matching Backend | |
| emoji: π€ | |
| colorFrom: blue | |
| colorTo: indigo | |
| sdk: docker | |
| pinned: false | |
| # Backend β Candidate Matching System | |
| FastAPI backend for the candidate matching task. Parses JD PDFs and candidate XLSX files, runs a hybrid BM25 + semantic matching pipeline, and returns ranked candidates with optional AI-generated explanations via Ollama. | |
| ## How It Works | |
| Matching runs as a background task through four stages: | |
| 1. **Hard filter** β removes candidates that don't meet engineer type or minimum years of experience | |
| 2. **BM25 pre-filter** β activated only when candidate pool exceeds 200; narrows down to top 500 before embedding | |
| 3. **Semantic matching** β runs `all-MiniLM-L6-v2` locally to embed candidate work history and JD requirements | |
| 4. **Scoring** β combines proficiency-aware skill matching, semantic similarity, title alignment, and domain score; applies penalties; stores ranked results in Redis | |
| ## Tech Stack | |
| | Category | Technology | | |
| | --- | --- | | |
| | Framework | FastAPI + Uvicorn | | |
| | NLP & Matching | Sentence-Transformers, Rank-BM25, PyMuPDF | | |
| | Data Parsing | Pandas, Openpyxl | | |
| | AI Explanations | Ollama | | |
| | Caching & State | Redis | | |
| | Config | Pydantic Settings | | |
| ## Setup | |
| ### Prerequisites | |
| - Python 3.11+ | |
| - Redis (or use Docker setup from root) | |
| ### Run Locally | |
| ```bash | |
| pip install -r requirements.txt | |
| uvicorn app.main:app --reload --port 8000 | |
| ``` | |
| ### Environment Variables | |
| | Variable | Description | Example | | |
| |----------|-------------|---------| | |
| | `OLLAMA_HOST` | Ollama server URL | `https://ollama.com` | | |
| | `OLLAMA_API_KEY` | Ollama cloud API key | `abc123...` | | |
| | `OLLAMA_MODEL` | Model name | `gpt-oss:120b-cloud` | | |
| | `REDIS_URL` | Redis connection string (use `rediss://` for TLS) | `rediss://default:pass@host:port` | | |
| | `CORS_ORIGINS` | Comma-separated allowed origins | `http://localhost:3000` | | |
| | `EMBEDDING_MODEL` | sentence-transformers model | `all-MiniLM-L6-v2` | | |
| | `BM25_PREFILTER_THRESHOLD` | Min candidates before BM25 activates | `200` | | |
| | `BM25_TOP_K` | Candidates kept after BM25 | `500` | | |
| | `LOG_LEVEL` | Logging level | `INFO` | | |
| > `REDIS_URL` with TLS (Upstash) must use `rediss://` (double-s). `redis://` will fail with SSL EOF errors. | |
| ## API Reference | |
| ``` | |
| GET /health | |
| β {status, redis, embedding_model_loaded} | |
| POST /api/v1/jd/upload multipart: file=<PDF> | |
| β {jd_id, title, engineer_type, min_years, required_skills_count} | |
| GET /api/v1/jd/ | |
| β [{jd_id, title, engineer_type, uploaded_at}, ...] | |
| GET /api/v1/jd/{jd_id} | |
| β full JD with skills and responsibilities | |
| POST /api/v1/candidates/upload multipart: file=<XLSX> | |
| β {batch_id, total_count, engineer_type_distribution} | |
| GET /api/v1/candidates/ | |
| β [{batch_id, total_count, uploaded_at}, ...] | |
| POST /api/v1/match/{jd_id} body: {batch_id, top_n} | |
| β {match_id, status: "pending"} 202 Accepted | |
| GET /api/v1/status/{match_id} | |
| β {status: "pending"|"running"|"completed"|"failed", progress, message} | |
| GET /api/v1/match/{jd_id}?batch_id=&top_n= | |
| β [{rank, candidate_id, name, final_score, tier, matched_skills, | |
| missing_skills, below_level_skills, flags, ...}, ...] | |
| GET /api/v1/match/{jd_id}/{candidate_id}?batch_id=&generate=true | |
| β full score breakdown + AI explanation (generate=true triggers Ollama) | |
| ``` | |
| ## Scoring Breakdown | |
| ``` | |
| Hard filter (engineer type + years of experience) | |
| β | |
| BM25 pre-filter (only if candidates > 200 β keeps top 500) | |
| β | |
| Local embeddings (all-MiniLM-L6-v2, candidate work history only) | |
| β | |
| Per-candidate score: | |
| 35% β proficiency-aware skill matching | |
| 40% β semantic similarity (JD vs work history) | |
| 10% β position title alignment | |
| 15% β domain score (GenAI depth for AI JDs, stack for Backend) | |
| β | |
| Penalties applied (type mismatch, years gap, missing fields) | |
| β | |
| Ranked β Redis ZSET β top-N retrieval O(log N) | |
| ``` | |
| ## XLSX Format | |
| Required columns: | |
| | Column | Description | | |
| |--------|-------------| | |
| | `name` | Full name | | |
| | `current_position` | Job title / role | | |
| | `engineer_type` | `"AI Engineer"`, `"Backend Engineer"`, etc. | | |
| | `parsed_skills` | Skills text | | |
| | `parsed_summary` | Professional summary | | |
| | `parsed_work_experience` | Stringified Python list of `{company, role, duration, description}` dicts | | |
| | `parsed_metadata_calculated_years_experience` | Years of experience (float) | | |
| Optional: `gen_ai_experience`, `current_title`, `is_actively_or_passively_looking`, `notice_period`, `parsed_metadata_most_recent_company` | |
| ## Storage | |
| Candidate JSON lives in process memory, not Redis β avoids Upstash free tier 10MB value limit. Process restart clears candidates; re-upload required. Match results and metadata stored in Redis with 24β48h TTL. | |
| If Redis is unavailable, falls back to `_MemStore` (in-process dict with TTL). System works without any external dependencies. | |
| ## Scaling to 100,000 Candidates | |
| ### What already helps at scale | |
| - **Async background tasks** β API stays responsive; client polls for status | |
| - **BM25 pre-filter** β skips expensive embedding for bulk of candidates when pool > 200 | |
| - **Local embeddings** β no network latency or API cost per request | |
| ### What would need to change | |
| **Distributed task queue** β replace `BackgroundTasks` with Celery + RabbitMQ/Redis broker to distribute matching jobs across worker nodes | |
| **Vector database** β pre-compute and store candidate embeddings in Pinecone/Weaviate/Milvus instead of generating on-the-fly. Matching becomes a nearest-neighbour lookup instead of a full scan | |
| **Microservices split** β break monolith into Parser, Matching, Explanation, and API Gateway services for independent scaling | |
| **Kubernetes** β container orchestration for auto-scaling, service discovery, and load balancing across the above services | |
| ```mermaid | |
| graph TD | |
| subgraph "User" | |
| A[Recruiter] | |
| end | |
| subgraph "API Layer" | |
| B[API Gateway] | |
| end | |
| subgraph "Services" | |
| C[Parser Service] | |
| D[Matching Service] | |
| E[Explanation Service] | |
| end | |
| subgraph "Data & AI Layer" | |
| F[Vector Database e.g., Pinecone] | |
| G[Task Queue e.g., Celery + RabbitMQ] | |
| H[Redis Cache] | |
| I[Ollama LLM] | |
| end | |
| A --> B | |
| B --> C | |
| B --> D | |
| B --> E | |
| C --> G | |
| D -- Reads from --> F | |
| D -- Uses --> G | |
| D -- Caches in --> H | |
| E -- Calls --> I | |
| ``` | |