| # Strict Baseline Paper Notes |
|
|
| These notes summarize the detector designs used by the strict LLaMA baseline |
| branch. |
|
|
| ## Sources |
|
|
| - EPR: "Learned Hallucination Detection in Black-Box LLMs using Token-level |
| Entropy Production Rate" (`arXiv:2509.04492`), plus the Artefactual public |
| implementation: https://github.com/artefactory/artefactual |
| - Lookback Lens: "Lookback Lens: Detecting and Mitigating Contextual |
| Hallucinations in Large Language Models Using Only Attention Maps" |
| (`arXiv:2407.07071`), plus the public implementation: |
| https://github.com/voidism/Lookback-Lens |
| - Multi-View Attention: "Hallucinated Span Detection with Multi-View Attention |
| Features" (`arXiv:2504.04335`, ACL Anthology `2025.starsem-1.31`), plus the |
| public implementation: https://github.com/Ogamon958/mva_hal_det |
|
|
| ## Strict Mapping |
|
|
| | Method | Paper/reference approach | Our implementation | |
| | --- | --- | --- | |
| | EPR | Computes top-k per-token entropy contribution `-p log2(p)` and sums rank contributions into a token score. | Teacher-forces the saved generated token IDs, extracts top-15 next-token log-probabilities, and computes the same `-exp(logp) * log2(exp(logp))` contribution in `float32`. | |
| | Lookback Lens | Computes each layer/head's `context_attention / (context_attention + generated_attention)`, then averages complete trailing windows before logistic regression. | Computes the same ratio from Hugging Face attention tensors, uses full trailing windows of size 8 for the strict run, max-pools hallucination-positive labels, and trains logistic regression with `max_iter=1000`. | |
| | Multi-View Attention | Uses three feature families per token: average incoming attention, incoming attention entropy, and outgoing attention entropy; feeds standardized features into a Transformer encoder with CRF decoding and selects hyperparameters by validation F1 search. | Extracts `key_avg`, `key_entropy`, and `query_entropy` by default; `key_avg` and `key_entropy` use the paper's adjusted incoming attention; standardizes the train split; runs a 200-trial Optuna search over the paper/public hyperparameter ranges; trains a Transformer-CRF with AdamW, ReduceLROnPlateau validation-loss scheduling, validation F1 early stopping, batch size 64, and up to 150 epochs. | |
|
|
| ## Scope |
|
|
| The strict run evaluates these detector methods on our LLaMA 3.2 |
| token-labelled dataset. It does not claim to reproduce the papers' benchmark |
| numbers, because those depend on the original datasets, model sizes, prompt |
| collections, and train/validation/test partitions. |
|
|
| The safe claim is: We implemented and evaluated strict detector-method paths |
| for EPR, Lookback Lens, and Multi-View Attention on our LLaMA 3.2 |
| token-labelled dataset. |
|
|
