File size: 4,704 Bytes
c87d6ef ce3f6c1 c87d6ef ce3f6c1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 | ---
license: mit
tags:
- langchain
- rag
- bayesian
- probabilistic
- uncertainty
- retrieval
---
# Bayesian LangChain RAG
A corrected, Hugging Face-ready implementation of uncertainty-aware retrieval augmented generation built on **LangChain Core**.
This repository replaces the earlier heuristic "tool succeeded = high likelihood" and beam-style scoring with an explicit generalized-Bayesian retrieval posterior. For a query `q` and candidate document `d`,
`p(d | q) ∝ p(d) (similarity(q,d) + eps)^tau`
where the prior `p(d)` is built from a Beta source-reliability mean, source authority, and freshness metadata. The similarity term is deliberately described as a **generalized-Bayes/Gibbs likelihood**, not as a literal language-model likelihood.
## What was corrected
- Removed notebook/Colab shell commands and hard-coded paths from library code.
- Removed the non-Bayesian `p_langchain` beam scorer from the Bayesian runtime.
- Replaced deterministic identical-particle behavior with an importance-corrected SMC utility for workflows that need particles.
- Added proposal-density correction `log target - log q` to SMC weights.
- Fixed all-`-inf` log-weight normalization.
- Preserved final posterior weights by forbidding final-step resampling.
- Added a real `BaseRetriever` implementation and LangChain `Runnable` entry point.
- RAG now reasons over retrieved evidence quality rather than rewarding any successful tool call.
- Added explicit abstention for missing evidence and unresolved near-equal conflicts.
- Added citation-grounded answer synthesis and a reproducible end-to-end RAG benchmark.
## Install
```bash
pip install -e .
```
No GPU is required. The default evaluation generator is extractive and grounded. Any LangChain-compatible model can be supplied for generative answering.
## Quick start
```python
from bayesian_langchain import DocumentRecord, BayesianIndex, BayesianRAG
docs = [
DocumentRecord(
id="official",
title="Current portal policy",
text="The current HR portal is PeopleHub.",
source="official",
metadata={"reliability_alpha": 18, "reliability_beta": 2, "authority": 0.98, "freshness": 1.0},
),
DocumentRecord(
id="legacy",
title="HR portal guide",
text="The HR portal is WorkSphere.",
source="legacy-cache",
metadata={"reliability_alpha": 6, "reliability_beta": 6, "authority": 0.45, "freshness": 0.18},
),
]
rag = BayesianRAG(BayesianIndex(docs))
print(rag.invoke("What is the current HR portal?"))
```
`BayesianRAG.as_runnable()` returns a standard LangChain `RunnableLambda`.
## RAG validation
Run:
```bash
PYTHONPATH=src:. python evaluation/rag_benchmark.py
```
The committed result is in `artifacts/rag_benchmark_results.json`.
| Suite | System | Recall@1 | MRR | Answer accuracy | Abstention accuracy | End-to-end accuracy |
|---|---|---:|---:|---:|---:|---:|
| Clean RAG (8 QA) | Bayesian | 1.000 | 1.000 | 1.000 | n/a | 1.000 |
| Clean RAG (8 QA) | Plain TF-IDF | 1.000 | 1.000 | 1.000 | n/a | 1.000 |
| Conflict/missing RAG (18 QA) | Bayesian | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| Conflict/missing RAG (18 QA) | Plain TF-IDF | 0.250 | 0.611 | 0.250 | 0.333 | 0.278 |
The conflict suite deliberately contains stale high-overlap documents, current authoritative documents, questions with no supporting evidence, and equal-authority contradictions. The results are **controlled repository tests**, not a claim of universal RAG accuracy.
### Framework-level LangChain RAG test
The repository also contains `evaluation/framework_rag_benchmark.py`, which routes every case through the real LangChain path:
`BayesianRetriever(BaseRetriever) -> BayesianRAG.as_runnable() -> grounded generation/abstention`.
It was validated against **langchain-core 1.6.11**. Results: **8/8 clean cases** and **18/18 conflict/missing-evidence cases** passed end-to-end. The committed output is `artifacts/framework_rag_benchmark_results.json`.
Run it with:
```bash
PYTHONPATH=src:. python evaluation/framework_rag_benchmark.py
```
## Layout
- `src/bayesian_langchain/core.py` — posterior retrieval, abstention, grounded answerer
- `src/bayesian_langchain/retriever.py` — LangChain `BaseRetriever`
- `src/bayesian_langchain/rag.py` — end-to-end LangChain RAG and Runnable
- `src/bayesian_langchain/smc.py` — corrected importance-weighted SMC utility
- `evaluation/rag_benchmark.py` — controlled retrieval/grounding benchmark
- `evaluation/framework_rag_benchmark.py` — real LangChain `BaseRetriever`/Runnable RAG benchmark
- `tests/` — regression and integration tests
- `VALIDATION.md` — exact framework/version validation record
|