samwjya commited on
Commit ·
e71840e
1
Parent(s): 4bbdc15
gitignore
Browse files
CLAUDE.md
DELETED
|
@@ -1,128 +0,0 @@
|
|
| 1 |
-
# PaperRank
|
| 2 |
-
|
| 3 |
-
An academic paper reranking system built for the Forward Data Lab research task at UIUC.
|
| 4 |
-
Adapts the Multi-Reranker approach from the ACM-ICAIF '24 paper
|
| 5 |
-
"Multi-Reranker: Maximizing performance of retrieval-augmented generation in the FinanceRAG challenge"
|
| 6 |
-
and applies it to academic literature search using the Semantic Scholar API.
|
| 7 |
-
|
| 8 |
-
## Project Goal
|
| 9 |
-
|
| 10 |
-
Compare BM25 baseline retrieval vs. multi-model cross-encoder reranking on academic papers.
|
| 11 |
-
The system fetches live papers from Semantic Scholar, ranks them with BM25, then reranks
|
| 12 |
-
using cross-encoder models inspired by the FinanceRAG paper. Evaluate using NDCG@10 and MRR.
|
| 13 |
-
|
| 14 |
-
## Stack
|
| 15 |
-
|
| 16 |
-
- **Language**: Python 3.10+
|
| 17 |
-
- **Data Source**: Semantic Scholar API (free, no key required)
|
| 18 |
-
- **Baseline Retrieval**: rank_bm25
|
| 19 |
-
- **Reranking**: sentence-transformers CrossEncoder (local, no API key)
|
| 20 |
-
- Primary model: BAAI/bge-reranker-v2-m3
|
| 21 |
-
- **Evaluation**: NDCG@10, MRR (manual relevance judgments)
|
| 22 |
-
- **CLI**: argparse
|
| 23 |
-
## Project Structure
|
| 24 |
-
|
| 25 |
-
```
|
| 26 |
-
paperrank/
|
| 27 |
-
├── CLAUDE.md # This file
|
| 28 |
-
├── main.py # CLI entry point
|
| 29 |
-
├── retriever.py # Semantic Scholar API fetching + BM25 ranking
|
| 30 |
-
├── reranker.py # CrossEncoder reranking logic (adapted from FinanceRAG)
|
| 31 |
-
├── evaluate.py # NDCG@10 and MRR metrics + side-by-side comparison
|
| 32 |
-
├── requirements.txt # Dependencies
|
| 33 |
-
└── README.md # Project documentation
|
| 34 |
-
```
|
| 35 |
-
|
| 36 |
-
## Key Files
|
| 37 |
-
|
| 38 |
-
- `retriever.py` — fetches top 50 candidate papers from Semantic Scholar API, ranks with BM25
|
| 39 |
-
- `reranker.py` — scores each (query, title + abstract) pair with CrossEncoder, returns reranked list
|
| 40 |
-
- `evaluate.py` — computes NDCG@10 and MRR, prints side-by-side BM25 vs reranked comparison table
|
| 41 |
-
- `main.py` — CLI entry: takes a query string, runs full pipeline, pretty prints results
|
| 42 |
-
## Architecture
|
| 43 |
-
|
| 44 |
-
```
|
| 45 |
-
User Query
|
| 46 |
-
│
|
| 47 |
-
▼
|
| 48 |
-
Semantic Scholar API (top 50 candidates)
|
| 49 |
-
│
|
| 50 |
-
▼
|
| 51 |
-
BM25 Baseline Ranking
|
| 52 |
-
│
|
| 53 |
-
▼
|
| 54 |
-
CrossEncoder Reranker (BAAI/bge-reranker-v2-m3, runs locally)
|
| 55 |
-
│
|
| 56 |
-
▼
|
| 57 |
-
Side-by-side Comparison (BM25 rank vs Reranked rank, with scores)
|
| 58 |
-
│
|
| 59 |
-
▼
|
| 60 |
-
NDCG@10 / MRR Evaluation
|
| 61 |
-
```
|
| 62 |
-
|
| 63 |
-
## Implementation Notes
|
| 64 |
-
|
| 65 |
-
- All models run **locally** — no OpenAI, Anthropic, or any paid API calls
|
| 66 |
-
- Reranker input: concatenate paper title + abstract as the document text
|
| 67 |
-
- BM25 tokenization: lowercase + whitespace split on title + abstract
|
| 68 |
-
- Fetch 50 candidates from Semantic Scholar, rerank to top 10 for evaluation
|
| 69 |
-
- Always show both BM25 and reranked results side-by-side for demo clarity
|
| 70 |
-
- Use `pandas` DataFrame for pretty printing comparison tables
|
| 71 |
-
## Reference Paper
|
| 72 |
-
|
| 73 |
-
- **Paper**: Multi-Reranker: Maximizing performance of RAG in the FinanceRAG challenge (ACM-ICAIF '24)
|
| 74 |
-
- **Authors**: Joohyun Lee, Minji Roh
|
| 75 |
-
- **ArXiv**: https://arxiv.org/abs/2411.16732
|
| 76 |
-
- **Original repo**: https://github.com/cv-lee/FinanceRAG
|
| 77 |
-
- **Key adaptation**: We replace their static finance corpus with live Semantic Scholar API ingestion,
|
| 78 |
-
and apply their multi-model reranking strategy to academic literature search instead of finance docs.
|
| 79 |
-
## Demo Queries (for testing + report)
|
| 80 |
-
|
| 81 |
-
1. "reducing cloud infrastructure costs using Kubernetes autoscaling"
|
| 82 |
-
2. "large language models for academic paper retrieval"
|
| 83 |
-
3. "graph neural networks for fraud detection"
|
| 84 |
-
4. "transformer models for information retrieval reranking"
|
| 85 |
-
## Claude Code Instructions
|
| 86 |
-
|
| 87 |
-
After writing each file, you MUST:
|
| 88 |
-
1. Print a plain English explanation of what the file does and why
|
| 89 |
-
2. Explain each function — what it takes as input, what it does, what it returns
|
| 90 |
-
3. Explain any non-obvious technical choices (e.g. why BM25, why concatenate title+abstract)
|
| 91 |
-
4. Show the exact command to run that file or test it
|
| 92 |
-
5. Show expected output so I know it's working correctly
|
| 93 |
-
Format each explanation like this:
|
| 94 |
-
|
| 95 |
-
---
|
| 96 |
-
### What this file does
|
| 97 |
-
...
|
| 98 |
-
|
| 99 |
-
### Function breakdown
|
| 100 |
-
- `function_name(args)` — what it does in plain English
|
| 101 |
-
### Why we made these choices
|
| 102 |
-
...
|
| 103 |
-
|
| 104 |
-
### How to run it
|
| 105 |
-
```bash
|
| 106 |
-
python filename.py --arg value
|
| 107 |
-
```
|
| 108 |
-
|
| 109 |
-
### Expected output
|
| 110 |
-
...
|
| 111 |
-
---
|
| 112 |
-
|
| 113 |
-
Do this for every file you create, no exceptions.
|
| 114 |
-
|
| 115 |
-
## Dependencies
|
| 116 |
-
|
| 117 |
-
```
|
| 118 |
-
requests
|
| 119 |
-
rank_bm25
|
| 120 |
-
sentence-transformers
|
| 121 |
-
pandas
|
| 122 |
-
torch
|
| 123 |
-
```
|
| 124 |
-
|
| 125 |
-
Install with:
|
| 126 |
-
```bash
|
| 127 |
-
pip install requests rank_bm25 sentence-transformers pandas torch
|
| 128 |
-
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|