AnishDev's picture
updated-readme for hf
d31b9a2
|
Raw History Blame Contribute Delete
6.43 kB
metadata
title: Candidate Matching Backend
emoji: πŸ€–
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: false

Backend β€” Candidate Matching System

FastAPI backend for the candidate matching task. Parses JD PDFs and candidate XLSX files, runs a hybrid BM25 + semantic matching pipeline, and returns ranked candidates with optional AI-generated explanations via Ollama.

How It Works

Matching runs as a background task through four stages:

  1. Hard filter β€” removes candidates that don't meet engineer type or minimum years of experience
  2. BM25 pre-filter β€” activated only when candidate pool exceeds 200; narrows down to top 500 before embedding
  3. Semantic matching β€” runs all-MiniLM-L6-v2 locally to embed candidate work history and JD requirements
  4. Scoring β€” combines proficiency-aware skill matching, semantic similarity, title alignment, and domain score; applies penalties; stores ranked results in Redis

Tech Stack

Category Technology
Framework FastAPI + Uvicorn
NLP & Matching Sentence-Transformers, Rank-BM25, PyMuPDF
Data Parsing Pandas, Openpyxl
AI Explanations Ollama
Caching & State Redis
Config Pydantic Settings

Setup

Prerequisites

  • Python 3.11+
  • Redis (or use Docker setup from root)

Run Locally

pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000

Environment Variables

Variable Description Example
OLLAMA_HOST Ollama server URL https://ollama.com
OLLAMA_API_KEY Ollama cloud API key abc123...
OLLAMA_MODEL Model name gpt-oss:120b-cloud
REDIS_URL Redis connection string (use rediss:// for TLS) rediss://default:pass@host:port
CORS_ORIGINS Comma-separated allowed origins http://localhost:3000
EMBEDDING_MODEL sentence-transformers model all-MiniLM-L6-v2
BM25_PREFILTER_THRESHOLD Min candidates before BM25 activates 200
BM25_TOP_K Candidates kept after BM25 500
LOG_LEVEL Logging level INFO

REDIS_URL with TLS (Upstash) must use rediss:// (double-s). redis:// will fail with SSL EOF errors.

API Reference

GET  /health
     β†’ {status, redis, embedding_model_loaded}

POST /api/v1/jd/upload                    multipart: file=<PDF>
     β†’ {jd_id, title, engineer_type, min_years, required_skills_count}

GET  /api/v1/jd/
     β†’ [{jd_id, title, engineer_type, uploaded_at}, ...]

GET  /api/v1/jd/{jd_id}
     β†’ full JD with skills and responsibilities

POST /api/v1/candidates/upload            multipart: file=<XLSX>
     β†’ {batch_id, total_count, engineer_type_distribution}

GET  /api/v1/candidates/
     β†’ [{batch_id, total_count, uploaded_at}, ...]

POST /api/v1/match/{jd_id}               body: {batch_id, top_n}
     β†’ {match_id, status: "pending"}     202 Accepted

GET  /api/v1/status/{match_id}
     β†’ {status: "pending"|"running"|"completed"|"failed", progress, message}

GET  /api/v1/match/{jd_id}?batch_id=&top_n=
     β†’ [{rank, candidate_id, name, final_score, tier, matched_skills,
         missing_skills, below_level_skills, flags, ...}, ...]

GET  /api/v1/match/{jd_id}/{candidate_id}?batch_id=&generate=true
     β†’ full score breakdown + AI explanation (generate=true triggers Ollama)

Scoring Breakdown

Hard filter (engineer type + years of experience)
    ↓
BM25 pre-filter (only if candidates > 200 β†’ keeps top 500)
    ↓
Local embeddings (all-MiniLM-L6-v2, candidate work history only)
    ↓
Per-candidate score:
  35% β€” proficiency-aware skill matching
  40% β€” semantic similarity (JD vs work history)
  10% β€” position title alignment
  15% β€” domain score (GenAI depth for AI JDs, stack for Backend)
    ↓
Penalties applied (type mismatch, years gap, missing fields)
    ↓
Ranked β†’ Redis ZSET β†’ top-N retrieval O(log N)

XLSX Format

Required columns:

Column Description
name Full name
current_position Job title / role
engineer_type "AI Engineer", "Backend Engineer", etc.
parsed_skills Skills text
parsed_summary Professional summary
parsed_work_experience Stringified Python list of {company, role, duration, description} dicts
parsed_metadata_calculated_years_experience Years of experience (float)

Optional: gen_ai_experience, current_title, is_actively_or_passively_looking, notice_period, parsed_metadata_most_recent_company

Storage

Candidate JSON lives in process memory, not Redis β€” avoids Upstash free tier 10MB value limit. Process restart clears candidates; re-upload required. Match results and metadata stored in Redis with 24–48h TTL.

If Redis is unavailable, falls back to _MemStore (in-process dict with TTL). System works without any external dependencies.

Scaling to 100,000 Candidates

What already helps at scale

  • Async background tasks β€” API stays responsive; client polls for status
  • BM25 pre-filter β€” skips expensive embedding for bulk of candidates when pool > 200
  • Local embeddings β€” no network latency or API cost per request

What would need to change

Distributed task queue β€” replace BackgroundTasks with Celery + RabbitMQ/Redis broker to distribute matching jobs across worker nodes

Vector database β€” pre-compute and store candidate embeddings in Pinecone/Weaviate/Milvus instead of generating on-the-fly. Matching becomes a nearest-neighbour lookup instead of a full scan

Microservices split β€” break monolith into Parser, Matching, Explanation, and API Gateway services for independent scaling

Kubernetes β€” container orchestration for auto-scaling, service discovery, and load balancing across the above services

graph TD
    subgraph "User"
        A[Recruiter]
    end

    subgraph "API Layer"
        B[API Gateway]
    end

    subgraph "Services"
        C[Parser Service]
        D[Matching Service]
        E[Explanation Service]
    end

    subgraph "Data & AI Layer"
        F[Vector Database e.g., Pinecone]
        G[Task Queue e.g., Celery + RabbitMQ]
        H[Redis Cache]
        I[Ollama LLM]
    end

    A --> B
    B --> C
    B --> D
    B --> E

    C --> G
    D -- Reads from --> F
    D -- Uses --> G
    D -- Caches in --> H
    E -- Calls --> I