docs/strict_baseline_paper_notes.md

Strict Baseline Paper Notes

These notes summarize the detector designs used by the strict LLaMA baseline branch.

Sources

• EPR: "Learned Hallucination Detection in Black-Box LLMs using Token-level Entropy Production Rate" (arXiv:2509.04492), plus the Artefactual public implementation: https://github.com/artefactory/artefactual
• Lookback Lens: "Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps" (arXiv:2407.07071), plus the public implementation: https://github.com/voidism/Lookback-Lens
• Multi-View Attention: "Hallucinated Span Detection with Multi-View Attention Features" (arXiv:2504.04335, ACL Anthology 2025.starsem-1.31), plus the public implementation: https://github.com/Ogamon958/mva_hal_det

Strict Mapping

Method	Paper/reference approach	Our implementation
EPR	Computes top-k per-token entropy contribution -p log2(p) and sums rank contributions into a token score.	Teacher-forces the saved generated token IDs, extracts top-15 next-token log-probabilities, and computes the same -exp(logp) * log2(exp(logp)) contribution in float32.
Lookback Lens	Computes each layer/head's context_attention / (context_attention + generated_attention), then averages complete trailing windows before logistic regression.	Computes the same ratio from Hugging Face attention tensors, uses full trailing windows of size 8 for the strict run, max-pools hallucination-positive labels, and trains logistic regression with max_iter=1000.
Multi-View Attention	Uses three feature families per token: average incoming attention, incoming attention entropy, and outgoing attention entropy; feeds standardized features into a Transformer encoder with CRF decoding and selects hyperparameters by validation F1 search.	Extracts key_avg, key_entropy, and query_entropy by default; key_avg and key_entropy use the paper's adjusted incoming attention; standardizes the train split; runs a 200-trial Optuna search over the paper/public hyperparameter ranges; trains a Transformer-CRF with AdamW, ReduceLROnPlateau validation-loss scheduling, validation F1 early stopping, batch size 64, and up to 150 epochs.

Scope

The strict run evaluates these detector methods on our LLaMA 3.2 token-labelled dataset. It does not claim to reproduce the papers' benchmark numbers, because those depend on the original datasets, model sizes, prompt collections, and train/validation/test partitions.

The safe claim is: We implemented and evaluated strict detector-method paths for EPR, Lookback Lens, and Multi-View Attention on our LLaMA 3.2 token-labelled dataset.