racore/mega-ram-100k

Mega-RAM — 85,654-parameter, training-free RAG model

Mega Retrieval Augmented Model. An 85,654-parameter controller with a 10M-token external memory. No weight in this repository was trained: every tensor is written from a declarative program (weights/program.json) by a deterministic compiler, and the Weights-Space-Programming loop edits the program — not gradients.

Parameters	85,654 (cap 100,000, enforced twice)
Trained parameters	0 — closed-form construction + least squares only
Encoder	256-d, 4 feature families, seeded orthogonal blocks
Context	10,000,000 tokens (external memory; not attention)
Weights size	130 KB compressed (weights/weights.npz), 343 KB as fp32
Program size	~2 KB (weights/program.json)
Latency (CPU)	3–20 ms/query, P50
Cost	~$0.0006 per 1,000 queries
Dependencies	numpy, scipy (no GPU, no transformers)

⚠️ This is not a language model

It does not generate free text. It retrieves, quotes and cites, and it can abstain. Every answer is a verbatim span from a document you index, with doc_id and character offsets. There is no world knowledge in these weights — that is the point of the design, and it is also its limitation.

Quick start

pip install numpy scipy huggingface_hub
hf download <your-username>/mega-ram-100k --local-dir ./mega-ram-100k

import sys, tempfile
sys.path.insert(0, "./mega-ram-100k")          # vendored, dependency-free loader

from modeling_megaram import load
rag = load("./mega-ram-100k")                  # or load("<username>/mega-ram-100k")

rag.index_documents([
    {"ext_id": "d1", "title": "BM25", "text": "BM25 ranks documents with term "
     "saturation k1 and length normalisation b. Typical values are k1=1.2, b=0.75."},
])

ans = rag.answer("What are the typical values of k1 and b in BM25?")
print(ans.text)             # 'k1=1.2, b=0.75'
print(ans.abstained)        # False
print(ans.citations[0].doc_id, ans.citations[0].quote)

If you have the full project installed (pip install -e . from the source repository), MegaRAM.load("<path>") gives the same object with the complete ingest toolkit (PDF, DOCX, images, video, code).

How the weights are written

block	shape	params	provenance (not training)
enc.W	256×256	65,536	seeded QR of an orthogonal block-diagonal matrix, one block per feature family
concept.W	256×72	18,432	encoder output of 72 hand-written glosses over 24 concepts
concept.b	72	72	zero
router.W	72×8	576	30 hand-written (concept, intent, weight) edges
router.b	8	8	intent priors
router.cue	24×8	192	24 regex cues → intent logits
span.W1	24×32	768	unit-norm random projections (reservoir)
span.b1	32	32	zero
span.W2	32	32	closed-form ridge distillation of the program's 24 span weights
span.b2	1	1	ridge intercept
fusion.w	5	5	program (bm25, char, vsa, proximity, concept)

weights/MANIFEST.json records the provenance, shape and parameter count of every block, and the compiler refuses to emit more than 100,000 parameters.

Retrieval fuses five signals with programmed weights: BM25, character 4-grams, a dense 256-d hashed-vector similarity, query-term proximity, and concept projection. Answering scores 24 surface features through the span scorer, then crops the best span (dates for temporal questions, currency for prices, subject-position entities for "who", quantities otherwise) and cites it.

Weights-Space Programming

Following Exploring Weight-Space Programming: Capabilities, Generalization, and Limitations, an editing loop proposes Δθ, evaluates it on a dev split, gates it on a second split, and keeps the original weights when improvement cannot be validated:

θ_{t+1} = θ_t + Δθ_t ,   Δθ_t = E_t(θ_t, H_t)

Two recorded runs:

run	evaluations	accepted	effect on held-out test
BEIR SciFact	41	1 (fusion[char] 0.30→0.10)	nDCG@10 0.6529 → 0.6447 (−0.0082)
synthetic fixture	79	0	unchanged (original retained)

The one accepted edit passed dev and gate and still failed to transfer — the same non-transfer the original paper reports. It is published here rather than hidden, because it is the honest result.

Benchmarks (measured, not claimed)

dataset	n	nDCG@10	R@5	MRR	EM	F1	Faithfulness	Citation acc.	P50
BEIR SciFact	300	0.6744	0.7536	0.6416	—	—	—	—	19 ms
BEIR NFCorpus	323	0.3275	0.1272	0.5380	—	—	—	—	9 ms
RAGBench hotpotqa	200	0.9192	1.0000	0.9329	—	—	1.0000	0.9286	7 ms
RAGBench msmarco	200	0.8696	0.7024	0.8452	—	—	1.0000	0.7010	9 ms
RGB	400	0.6651	0.3731	0.7928	0.1725	0.2503	1.0000	0.6963	6 ms
CRAG v4 (50 records)	50	—	—	—	0.0000	0.0318	1.0000	—	14 ms

Versus BM25 alone on the same index: SciFact nDCG@10 0.6744 vs 0.6644, NFCorpus 0.3275 vs 0.3134. So: slightly above BM25, far below large neural encoders. — means the benchmark cannot support that metric (no reference answers, no qrels), not zero.

CRAG and RAGBench results are also in the machine-readable model-index above. The BEIR numbers stay in this table because the Hub has no retrieval task id to attach them to, and inventing one would produce a broken link. RGB is not published as a Hub dataset, so that run reads github.com/chen700564/RGB directly.

Faithfulness is 1.0 by construction, because answers are quotes. Do not read that as "better than a generative RAG system": Exact Match on RGB is 0.17 and CRAG is 0.00 precisely because extraction cannot paraphrase, aggregate or compute.

Ingestion (full project)

txt, md, csv, json, any source code, PDF (text layer, OCR fallback for scans), DOCX, images (real OCR via rapidocr/tesseract), video (frame OCR with timestamps + optional ASR), audio (optional ASR). Every reader degrades explicitly — a file whose text cannot be extracted is marked, never silently invented.

Limitations

1. Cannot compose, calculate, or aggregate across documents (CRAG-style questions score 0).
2. Cannot paraphrase — Exact Match is structurally low.
3. Retrieval is BM25-class; no learned embeddings.
4. Video/images are handled through OCR text, not visual understanding.
5. This repository ships weights only — memory starts empty by design.

Reproduce

python scripts/run_bench.py --suite all --out reports/final
python scripts/run_wsp.py --dataset scifact --dev 60 --gate 40 --test 200
python scripts/scale_test.py --tokens 3000000

Full project, documentation and tests: docs/ARCHITECTURE.md, docs/WSP.md, docs/BENCHMARKS.md, 47 passing tests.

Citation

@software{megaram2026,
  title  = {Mega-RAM: an 85,654-parameter training-free retrieval-augmented model},
  year   = {2026},
  note   = {Weights written by Weights-Space Programming; no gradient training}
}

Related: Hu et al., Exploring Weight-Space Programming: Capabilities, Generalization, and Limitations, arXiv:2611.09382.

العربية

نموذج RAG صغير (85,654 بارامترًا) بذاكرة خارجية تصل إلى 10 ملايين توكن. لا يوجد أي وزن مُدرَّب: كل الأوزان مكتوبة برمجيًا من ملف program.json، والإجابات مقتبسة حرفيًا من مستنداتك مع استشهاد ورفض عند نقص الأدلة. لا يحسب ولا يعيد الصياغة — وهذا مذكور صراحة في قسم القيود، مع نتائج CRAG السيئة كما هي.