# QuantPPI: Energy-Aware Precision Allocation for Extreme Low-Bit Protein-Embedding Retrieval ## Project Overview QuantPPI investigates extreme compression of high-dimensional protein language model (PLM) embeddings (such as those from ESM-2) for approximate nearest neighbor retrieval. Storing and searching large collections of these embeddings creates substantial memory and bandwidth costs. This project examines the **Rotated Scaled Lloyd-Max (RSLM)** codec, a training-free vector quantization method. We identify a major failure mode in standard low-bit RSLM on ESM-2 embeddings: the native coordinate energy distribution is pathologically concentrated (a few coordinates contain most of the squared amplitude), leading to severe shared-scale quantization errors across the transformed vector block. To resolve this, we introduce and evaluate **Native Protection** (also referred to historically as Energy-Protected RSLM or RSLM-Outlier), a precision-allocation heuristic that isolates these high-energy coordinates in their native basis (storing them at FP16) before applying the RSLM transform to the remaining representation. The repository compares this strategy against Uniform RSLM, heterogeneous precision variants, and learned quantization baselines (Product Quantization (PQ) and Optimized Product Quantization (OPQ)) using candidate-pool recall and Inverted File Index (IVF) end-to-end evaluations. ## Repository Status - **Authoritative Papers:** The final results are documented in `QuantPPI_Paper_Trishal.pdf` (the primary 28-page research paper) and `QuantPPI_Mathematical_Methods_Report (1).pdf` (the external technical dossier). `paper/QuantPPI_Final_Report.pdf` contains the exact generated tables from our reproduction pipeline. - **Reproducibility Status:** **Partially Verified**. Static implementation and mathematical checks found no known contradictions in the audited paths. Full numerical reproducibility remains unverified because the expensive computations were not completed locally and their outputs were not exhaustively compared against the reported results to prevent heavy GPU/CPU workloads. - **Audit Documents:** Detailed methodological checks and limitations are documented in `results/final_reproducibility_audit.md` and `results/audit_checklist_resolved.md`. ## Directory Guide - [`docs/EXPERIMENTS_CHRONOLOGY_AND_RESULTS.md`](docs/EXPERIMENTS_CHRONOLOGY_AND_RESULTS.md) - **Master Chronology & Empirical Results**: Full historical progression of all experiments from inception to SCMF. - [`docs/SCMF_MATHEMATICAL_FOUNDATIONS.md`](docs/SCMF_MATHEMATICAL_FOUNDATIONS.md) - **SCMF Mathematical Foundations**: Exhaustive treatise covering prerequisites, formal proofs, basis architectures, and systems complexity. - [`docs/REPRODUCIBILITY.md`](docs/REPRODUCIBILITY.md) - Reproducibility runbook and documentation audit. - [`experiments/data_oblivious_quantization/`](experiments/data_oblivious_quantization/README.md) - **Data-Oblivious Transforms & Quantization**: Full investigation of Circulant, DCT, DST, Wavelet, Lattice ($E_8, D_4$), Block-scaled, and Dithered quantization for PLMs. - [`scripts/`](scripts/README.md) - Essential Python implementations, final experimental pipelines, and controls. - [`experiments/score_preserving_common_mode/`](experiments/score_preserving_common_mode/reports/README.md) - Complete SCMF experimental package, code, and LaTeX manuscript. - [`results/`](results/README.md) - JSON result artifacts, audit records, and reproducible benchmarks. - [`reports/`](reports/README.md) - Historical technical notes, validations, and provenances. - [`paper/`](paper/README.md) - Authoritative final PDFs and documentation. - [`data/`](data/README.md) - Evaluation datasets (SCOPe domains, Swiss-Prot human). - [`cache/`](cache/README.md) - HDF5 cached FP32 embeddings for ESM-2 variants. - [`explainer/`](explainer/README.md) - Node.js web explainer application. - [`pdf_converter/`](pdf_converter/README.md) - Node.js PDF conversion utility. ## Prerequisites and Setup ### Python Environment The project uses Python 3.14. It is recommended to use the existing virtual environment. ```bash # Activate the environment source ./venv/bin/activate ``` Dependencies include `torch`, `numpy`, `transformers`, `faiss-cpu`, and standard data science libraries (see `scripts/requirements.txt`). Do not install packages globally. ### Hardware Requirements - Core quantization, scoring, and evaluation scripts run entirely on CPU using `numpy` and `faiss-cpu`. - Generating new embedding caches (e.g., `prott5_geometry_gate.py`) requires downloading large neural network weights (~5GB for ProtT5) and ideally benefits from an MPS/CUDA GPU to execute the PyTorch transformer passes efficiently. ### Datasets and Caches The project relies on pre-computed `.h5` caches located in `cache/` (e.g., `esm2_650m_mean_l2norm_final_15166.h5`) and corresponding FASTA/TSV files in `data/` to avoid rerunning expensive PLM inferences repeatedly. ## Quick Start To safely verify your environment without triggering a massive computation, you can run a dry-run check of the scale diagnostic: ```bash source ./venv/bin/activate python scripts/test_energy_distortion_law.py --help ``` *Note: The actual execution of these scripts loads multi-gigabyte HDF5 files into memory. Ensure you have at least 16GB of RAM before running full benchmarks.* ## Reproducing the Main Paper A complete step-by-step reproduction sequence is available in the [Reproducibility Runbook](docs/REPRODUCIBILITY.md). ## Important Scientific Definitions - **Rotated Scaled Lloyd-Max (RSLM):** A training-free codec that transforms vectors via a blockwise Fast Walsh-Hadamard Transform (FWHT), determines a shared scale derived from transformed amplitudes, and quantizes the blocks using fixed Lloyd-Max tables (e.g., `C4D` mapping 4-dimensional blocks to 16 centroids for a 1 bit-per-dimension rate). - **Native Protection:** An algorithm that zeroes out a small, globally selected set of high-energy native coordinates *before* applying the FWHT. The selected coordinates are stored in FP16, and the remaining vector (the trunk) is quantized with standard RSLM. - **Recall@20@30:** Our candidate-pool evaluator isolates a pre-selected pool of the 500 nearest ground-truth neighbors. It ranks these 500 using compressed representations and reports the fraction of the true top-20 that successfully remain within the predicted top-30. - **Participation Ratio (PR) & Top-k Energy Fraction ($\gamma_k$):** PR is computed globally across the corpus variance vector. $\gamma_k$ is computed per-vector (squared sum of top $k$ divided by total squared sum) and then averaged across the corpus. - **Payload vs Amortized Storage:** Payload describes raw codec storage (e.g., 224 bytes/vector). Amortized storage divides model-level parameters (e.g., OPQ's 6.8MB rotation matrix) evenly across the $N$ dataset vectors to reflect true memory footprint. ## Reproducibility and Troubleshooting - **Configuration & Seeds:** Ensure you run the specific pipeline scripts listed in the Runbook, as they embed the exact parameters (e.g., $k=32$, $M=224$) used to generate the tables. - **Output Paths:** Result scripts output `.json` files exclusively to `results/`. Be cautious not to overwrite authoritative JSONs unless actively performing a reproduction audit. - **Memory Problems:** If a script crashes with an Out-of-Memory (OOM) error, verify that you are not loading multiple ESM-2 caches simultaneously. ## Current Limitations - **Numerical Verification:** Local full-scale generation of the exact JSON decimals has been deliberately deferred to avoid excessive compute overhead. - **Generalization:** Findings are strongly established for ESM-2 on SCOPe and Swiss-Prot structural retrieval. Cross-model audits indicate the pathology exists across some other PLMs, but universality is not guaranteed.