Two-Tower model:
- Switch from BCE to BPR loss with per-user negative pairing
- Remove weight decay from embedding parameters (sparse gradient fix)
- Collapse linear architecture (hidden_dims=[]) — MLP with ReLU collapsed
all inputs to same hypersphere direction on sparse book data
- Add BPRDataset with per-user negative sampling to dataset.py
- All 3 models now produce non-zero Recall@10 (was 0.000 for Two-Tower)
Evaluation results (5k sampled users, leave-last-out):
- Naive: Recall@10=0.0046, NDCG@10=0.0022, MRR=0.0020
- LightGBM: Recall@10=0.0026, NDCG@10=0.0012, MRR=0.0010
- Two-Tower: Recall@10=0.0024, NDCG@10=0.0014, MRR=0.0012
API / inference:
- Filter untitled items (42% of catalog) from all recommendation paths
- Use numpy fancy indexing for exclusion (was O(n) loop)
- Expand LightGBM candidate pool to 500 (was 100)
Frontend redesign:
- Single model UX — always use Two-Tower, remove ModelSelector/CompareView
- New MetricsTable component with real evaluation numbers
- page…
fix: resolve SIGSEGV and OpenMP deadlock for LightGBM + PyTorch coexistence
yash-b18
09fcfe1
- downloader.py: rewrite to use HfFileSystem streaming JSONL (datasets 4.x compat)
- user_features.py: fix .mode().iat[0] IndexError on empty Series
- deep.py: add weights_only=True to torch.load to prevent interactive hang in PyTorch 2.11
- serve_utils.py: lazy-import torch after LightGBM pickle load to prevent SIGSEGV;
call torch.set_num_threads(1) immediately after import to prevent OpenMP deadlock
when LightGBM and PyTorch share a process on Apple Silicon macOS
- app.py: fix personas endpoint to use rating column (not label); add detailed
model-loading logs to aid future debugging
- recommender.py: fix feat[item_idx] KeyError in classical recommendations;
fix nan title/category/brand display
feat: data pipeline — download, preprocess, split, and feature engineering
Sets up the complete project skeleton for the Explainable Multi-Stage
Recommendation System. Includes directory layout, central config module,
pipeline orchestrator, Makefile, report outline, and all package __init__
files. No model or data code yet — that comes in subsequent branches.
