Angshul commited on
Commit
4b7424b
·
verified ·
1 Parent(s): 6e3645d

Upload 7 files

Browse files
Files changed (7) hide show
  1. LICENSE +21 -0
  2. MODEL_CARD.md +56 -0
  3. PACKAGE_README.md +53 -0
  4. README.md +15 -6
  5. app.py +53 -0
  6. pyproject.toml +21 -0
  7. requirements.txt +4 -0
LICENSE ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Angshul Majumdar
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
MODEL_CARD.md ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ license: mit
5
+ tags:
6
+ - rag
7
+ - reranking
8
+ - evidence-subset-selection
9
+ - multi-hop-reasoning
10
+ pipeline_tag: text-ranking
11
+ ---
12
+
13
+ # Evidence-Graph Reranker: algorithm card
14
+
15
+ **Type:** training-free joint selection algorithm with external pretrained neural relevance scorers. **Not** a standalone trained neural checkpoint or language generator.
16
+
17
+ ## Formula
18
+
19
+ `score(candidate | already_selected, query) = 0.4 × minmax(relevance) + 0.5 × directed_title_link + 0.2 × query_title_match`
20
+
21
+ **One rule for all six datasets.** No dataset-specific presets or facet objectives. Scores are recomputed greedily after selection; selecting an informative first passage can make a complementary follow-up passage more attractive.
22
+
23
+ ## Neural models (third-party weights)
24
+
25
+ - `cross-encoder/ms-marco-MiniLM-L6-v2`: pretrained relevance logits for query/passage pairs, 256 WordPiece cap used in experiments.
26
+ - `sentence-transformers/all-MiniLM-L6-v2`: cosine similarity over pretrained normalized embeddings, 128 WordPiece cap used in experiments.
27
+
28
+ The code pulls these third-party weights separately from Hugging Face. The package's MIT license covers our selector code, not external pretrained weights.
29
+
30
+ ## Evaluation
31
+
32
+ Measured complete supporting evidence coverage at k=3 passages:
33
+
34
+ | Benchmark | Cross-encoder | Cross-encoder + fixed graph | Gain |
35
+ |---|---:|---:|---:|
36
+ | HotpotQA (600) | 60.83% | 80.50% | +19.67 pp |
37
+ | 2WikiMultiHopQA (600) | 44.50% | 70.17% | +25.67 pp |
38
+ | MuSiQue (300) | 19.33% | 27.00% | +7.67 pp |
39
+ | HoVer (300) | 57.00% | 72.67% | +15.67 pp |
40
+ | MultiHop-RAG (300) | 13.67% | 13.67% | +0.00 pp |
41
+ | QASC (300) | 13.00% | 13.00% | +0.00 pp |
42
+
43
+
44
+ Six datasets, four improvements and two ties at k=3. These are not the same candidate-pool protocol. HotpotQA and 2Wiki use previously computed neural model outputs and newly rerun unified graph selection, not newly recomputed forward passes. For k=2,5,8, exact timings, candidate availability, bootstrap intervals and limitations, see `results/methodology.md` and `results/six_dataset_results.csv`.
45
+
46
+ ## Known limitations
47
+
48
+ - Direct title/body string linking works best when meaningful titles and article cross-references exist. It cannot reliably model nonlexical semantic connections without additional features, which are *not part of this released single algorithm*.
49
+ - On QASC's untitled sentences, the graph features vanish and the method becomes ordinary neural ranking. MultiHop-RAG at k=3 is also tied with the cross-encoder.
50
+ - Gold-containing candidate pools on some benchmarks produce an easier, conditional reranking task. The method does not repair missing supporting evidence in the initial retrieval stage.
51
+ - **No LLM answer-quality or hallucination-reduction metric was computed.** Avoid equating recovered evidence with successful RAG generation.
52
+ - Graph's greedy heuristic does not carry a proven submodular approximation ratio.
53
+
54
+ ## Reuse
55
+
56
+ Install the package and run the Gradio demo. The graph-only API, `rank_with_scores`, does not download neural checkpoints. Cite the original pretrained models if using neural scoring. Verify the terms of all third-party model checkpoints and datasets.
PACKAGE_README.md ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Evidence-Graph Reranker
2
+
3
+ An installable **single-algorithm**, training-free evidence-subset reranker for RAG.
4
+
5
+ Unlike conventional independently scored query–document rerankers, it greedily selects a set of complementary passages using an explicitly directed title-reference graph. It can work with a pretrained MS MARCO MiniLM cross-encoder, a MiniLM dense encoder, or supplied relevance scores.
6
+
7
+ ## Install
8
+
9
+ ```bash
10
+ pip install -e '.[neural]'
11
+ ```
12
+
13
+ ## Quick start
14
+
15
+ ```python
16
+ from evidence_graph_reranker import EvidenceGraphReranker
17
+
18
+ query = 'Who founded the company that released the Atlas device?'
19
+ docs = [
20
+ {'title': 'Atlas device', 'text': 'The Atlas device was released by Novus Labs.'},
21
+ {'title': 'Novus Labs', 'text': 'Novus Labs was founded by Mira Chen.'},
22
+ {'title': 'Unrelated', 'text': 'A different device was invented elsewhere.'},
23
+ ]
24
+ reranker = EvidenceGraphReranker(backend='cross-encoder', device='cpu')
25
+ selected = reranker.rank(query, docs, k=2)
26
+ print([(x['index'], x['title']) for x in selected])
27
+ ```
28
+
29
+ **No model download:** if you have scores from an existing retriever, call `rank_with_scores(query, docs, scores, k)`. Neural scoring always happens before the unchanged fixed graph rule.
30
+
31
+ ## How selection works
32
+
33
+ At each greedy step, every remaining candidate receives:
34
+
35
+ `0.4 × per-query minmax neural relevance + 0.5 × directed link from an already selected passage + 0.2 × query/title match`
36
+
37
+ The function is deterministic, has no learned parameters of its own, and does not use gold facts or answers at inference. The title feature uses an exact normalized title string, and the link feature checks whether a candidate's title appears in the body of a previously selected passage. Documents without titles simply lack these two signals and are selected by the underlying neural scores.
38
+
39
+ Model checkpoints stay at their original locations:
40
+ - `cross-encoder/ms-marco-MiniLM-L6-v2` (query+passage WordPiece max length 256)
41
+ - `sentence-transformers/all-MiniLM-L6-v2` (dense WordPiece max length 128)
42
+
43
+ ## Public Hugging Face Space
44
+
45
+ The repository root has `app.py`, `README.md` with Space metadata and `requirements.txt`. Upload these files to a new **Gradio Space**. Model weights download on first use. No model weights or dataset copies are redistributed in this package.
46
+
47
+ ## Benchmark results
48
+
49
+ See `results/six_dataset_results.csv`, `MODEL_CARD.md`, and `results/methodology.md`. All six evaluations use the **identical feature extractor and greedy selector**. HotpotQA/2Wiki were reselected from cached pretrained neural logits and embeddings under the common algorithm, not rescored by rerunning the neural forward passes. This distinction is indicated in the timing report. The six benchmarks have different candidate pools and should not be pooled into a single headline accuracy number.
50
+
51
+ ## License
52
+
53
+ MIT for this source code; external model weights and benchmark data retain their original licenses and usage terms.
README.md CHANGED
@@ -1,14 +1,23 @@
1
  ---
2
  title: Evidence Graph Reranker
3
- emoji: 😻
4
- colorFrom: purple
5
- colorTo: yellow
6
  sdk: gradio
7
- sdk_version: 6.31.0
8
- python_version: '3.12'
9
  app_file: app.py
10
  pinned: false
11
  license: mit
 
 
 
 
 
12
  ---
13
 
14
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
1
  ---
2
  title: Evidence Graph Reranker
3
+ emoji: 🧩
4
+ colorFrom: blue
5
+ colorTo: indigo
6
  sdk: gradio
 
 
7
  app_file: app.py
8
  pinned: false
9
  license: mit
10
+ tags:
11
+ - rag
12
+ - reranking
13
+ - multi-hop-qa
14
+ - evidence-retrieval
15
  ---
16
 
17
+ # Evidence-Graph Reranker — Hugging Face Space
18
+
19
+ **A query-conditioned, training-free, joint evidence-subset reranker.** This Space scores query–passage pairs using the public MS MARCO MiniLM-L6 cross-encoder or all-MiniLM-L6 dense encoder, then selects passages with the **same fixed graph algorithm** evaluated on six evidence benchmarks.
20
+
21
+ For installation, API examples, full six-dataset results and caveats, see [PACKAGE_README.md](PACKAGE_README.md), [MODEL_CARD.md](MODEL_CARD.md), and [results/methodology.md](results/methodology.md).
22
+
23
+ **Not a new pretrained language model:** This repository provides an algorithm plus an optional Space. It downloads official neural scorer checkpoints on first use; no original model weights are redistributed.
app.py ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Hugging Face Space: live CPU demo for the exact fixed evidence-graph reranker."""
2
+ import json
3
+ import gradio as gr
4
+ from evidence_graph_reranker import EvidenceGraphReranker
5
+
6
+ _RERANKERS = {}
7
+ EXAMPLE = [
8
+ {"title": "Atlas device", "text": "The Atlas device was launched by Novus Labs in 2025."},
9
+ {"title": "Novus Labs", "text": "Novus Labs was founded by Mira Chen in 2018."},
10
+ {"title": "Orion spacecraft", "text": "The Orion spacecraft is designed for space missions."},
11
+ {"title": "Electronics", "text": "Electronics can be manufactured in many countries."},
12
+ ]
13
+
14
+
15
+ def demo(query, passages_json, budget, backend):
16
+ try:
17
+ docs = json.loads(passages_json)
18
+ if not isinstance(docs, list) or not 1 <= len(docs) <= 100:
19
+ raise ValueError("Enter a JSON list containing 1–100 passages.")
20
+ if not query.strip():
21
+ raise ValueError("Query cannot be empty.")
22
+ if backend not in _RERANKERS:
23
+ _RERANKERS[backend] = EvidenceGraphReranker(backend=backend, device="cpu", batch_size=16)
24
+ ranked = _RERANKERS[backend].rank(query, docs, k=int(budget))
25
+ table = [
26
+ [i + 1, r["index"], r["title"] or "(untitled)",
27
+ round(r["relevance_score"], 4), round(r["step_score"], 4),
28
+ round(r["directed_link_bonus"], 2)]
29
+ for i, r in enumerate(ranked)
30
+ ]
31
+ return table, json.dumps(ranked, indent=2, ensure_ascii=False)
32
+ except Exception as exc:
33
+ return [], json.dumps({"error": str(exc)})
34
+
35
+ with gr.Blocks(title="Evidence-Graph Reranker") as app:
36
+ gr.Markdown("# Evidence-Graph Reranker\nTraining-free, query-conditioned **joint evidence selection** after neural relevance scoring.\n\nA passage's usefulness can rise when it completes a link from a previously selected passage. This demo uses **one fixed graph rule** for every dataset. Model weights are fetched from their original Hugging Face repositories.")
37
+ query = gr.Textbox(label="Query", value="Who founded the company that released the Atlas device?")
38
+ passages = gr.Code(label="Candidate passages (JSON array; optional title)", language="json",
39
+ value=json.dumps(EXAMPLE, indent=2))
40
+ with gr.Row():
41
+ budget = gr.Slider(minimum=1, maximum=10, step=1, value=3, label="Passage budget (k)")
42
+ backend = gr.Dropdown(choices=["cross-encoder", "dense"], value="cross-encoder",
43
+ label="Pretrained neural relevance model")
44
+ go = gr.Button("Select complementary evidence", variant="primary")
45
+ table = gr.Dataframe(headers=["Rank", "Candidate index", "Title", "Neural score", "Selection score", "Link bonus"],
46
+ datatype=["number", "number", "str", "number", "number", "number"],
47
+ label="Selected passages", interactive=False)
48
+ output = gr.Code(label="Full selection JSON", language="json")
49
+ go.click(demo, inputs=[query, passages, budget, backend], outputs=[table, output])
50
+ gr.Markdown("**Interpretation:** `selection_score` is a step-dependent heuristic, not a probability. Gold labels are never used by the selector. Reported benchmarks measure complete evidence coverage, *not* LLM-generated answer accuracy. See `MODEL_CARD.md` for protocols and caveats.")
51
+
52
+ if __name__ == "__main__":
53
+ app.launch()
pyproject.toml ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [build-system]
2
+ requires = ["setuptools>=65", "wheel"]
3
+ build-backend = "setuptools.build_meta"
4
+
5
+ [project]
6
+ name = "evidence-graph-reranker"
7
+ version = "0.1.0"
8
+ description = "Training-free query-conditioned evidence subset selection using fixed directed article-title links"
9
+ readme = "PACKAGE_README.md"
10
+ requires-python = ">=3.10"
11
+ license = {text = "MIT"}
12
+ keywords = ["rag", "reranking", "multihop", "evidence selection", "huggingface"]
13
+ dependencies = ["numpy>=1.23"]
14
+
15
+ [project.optional-dependencies]
16
+ neural = ["sentence-transformers>=3.0", "torch>=2.1"]
17
+ demo = ["gradio>=4.0", "sentence-transformers>=3.0", "torch>=2.1"]
18
+ test = ["pytest>=7.0"]
19
+
20
+ [tool.setuptools.packages.find]
21
+ include = ["evidence_graph_reranker*"]
requirements.txt ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ numpy>=1.23
2
+ torch>=2.1
3
+ sentence-transformers>=3.0
4
+ gradio>=4.0