Shadowed in Vector Space: Zero-Label Detection and Resolution of Stale Version Collisions in Retrieval-Augmented Generation
Official repository for Semantic Collision Scorer ($\text{SCS}^$)*, a drop-in zero-latency pruner for stale document collisions in RAG pipelines.
β‘ The Problem: Stale Version Collisions & The MMR Diversity Paradox
In production enterprise RAG systems (finance, healthcare, cloud architecture, legal contracts), policies and numbers change constantly.
- The Pooling Dilution Bound ($\mathcal{O}(1/L^2)$): Dense embeddings pool tokens uniformly. When a policy changes by 1 token, the cosine shift is $\Delta \cos \le \mathcal{O}(1/L^2) < 0.0005$, making vector databases mathematically blind to the difference between current and obsolete documents.
- The MMR Diversity Paradox: Standard Maximal Marginal Relevance (MMR) actually worsens staleness by $+16.1%$ (from $31.8%$ to $47.9%$) because historical versions are treated as "valuable novel information".
- $\text{SCS}^$ Resolution*: A zero-label, sub-millisecond ($<0.8\text{ ms}$) geometric sieve running on CUDA tensor cores that drives stale hallucinations down to $0.00%$.
π Empirical Benchmarks Across Dual NVIDIA RTX 5080 GPUs
| Pipeline Strategy | Fresh Answer Acc (%) | Stale Hallucination Rate (%) | Pruning Latency |
|---|---|---|---|
| Vanilla Dense RAG | $50.00% - 52.08%$ | $25.00% - 31.77%$ | 110.0 ms |
| MMR Diversity RAG | $36.98% - 50.00%$ | $25.00% - 47.92%$ | 94.4 ms |
| Time-Aware (HoH) | $31.77%$ | $51.56%$ | 181.2 ms |
| PyTorch $\text{SCS}^$ (Ours)* | $75.00% - 86.46%$ | $0.00%$ | $< 0.8\text{ ms}$ |
π¦ Python Installation & Usage
pip install git+https://github.com/soham2710/scs-rag-collision-pruner.git#subdirectory=scs_pruner_pkg
Native Python
from scs_pruner import SCSPruner
pruner = SCSPruner(tau_collision=0.55)
retrieved_chunks = [
{"text": "2024 Travel Limit: $50 per day", "timestamp": 20240101},
{"text": "2025 Travel Limit: $100 per day", "timestamp": 20250101}
]
clean_chunks = pruner.prune(retrieved_chunks)
print(clean_chunks)
# [{'text': '2025 Travel Limit: $100 per day', 'timestamp': 20250101}]
LangChain Drop-In
from scs_pruner import LangChainSCSCompressor
from langchain.retrievers import ContextualCompressionRetriever
retriever = ContextualCompressionRetriever(
base_compressor=LangChainSCSCompressor(tau_collision=0.55),
base_retriever=vectorstore.as_retriever()
)
π Full Manuscripts & Figures
@article{sharma2026shadowed,
title={Shadowed in Vector Space: Zero-Label Detection and Resolution of Stale Version Collisions in Retrieval-Augmented Generation},
author={Sharma, Soham},
year={2026}
}