# Fidelity Matrix This branch separates strict detector-method outputs from controls and local calibrations. A method is reported in the strict artifact only when its formula, feature shape, model/head, token alignment, prediction mode, and train/eval isolation match the selected paper or public reference target. ## Input Replay Gate - Faithful LLaMA runs must use `meta-llama/Llama-3.2-1B-Instruct`. - Faithful runs require saved `generated_token_ids` and teacher-force the exact generated answer tokens. - Reportable runs pin one immutable model/tokenizer commit, verify that both generation revisions match it, require the saved IDs to decode exactly to the answer, and recompute the generation-time effective prompt-token hash. - `--allow-token-id-model-mismatch` is only for diagnostic/non-faithful runs. - `--allow-unverified-replay` explicitly marks legacy provenance as diagnostic; it cannot appear in a strict result and is the only mode that permits missing answer IDs to be retokenized. - Prediction rows record whether saved answer-token IDs were required and used, whether either override was allowed, requested/resolved scoring revisions, and prompt replay verification status. - Every trained method consumes the same dataset-bound, duplicate-group-safe `--split-summary`; method-local row splits are diagnostic only. ## Method Status | Method | Current output method | Source target | Status | Required evidence | | --- | --- | --- | --- | --- | | EPR | response-level EPR | Paper response score | Pending method-faithful suite | Response mean token entropy and exact token replay | | Entropy-rank token probe | `entropy_rank_token_probe` | Local token control | Control, not EPR/WEPR | Token-label training must not be described as native WEPR | | Lookback Lens | `lookback_lens` | Lookback Lens repo | Versioned causal and method-faithful paths | Causal pre-token alignment for matched results; post-emission full windows only in the separately labeled method-faithful suite | | MVA logistic | `mva_feature_logistic_baseline` | MVA feature families only | Control baseline, not paper model | Output name and metadata must say logistic control | | MVA Transformer-CRF | `multi_view_attention_transformer_crf` | MVA paper method | Strict architecture/prediction path | Transformer encoder + CRF, paper feature families, CRF decode, train split only | | Hidden-state probe | `hidden_state_probe` | Control/comparator | Control baseline unless tied to a specific paper | Must not appear in the paper-faithful table | | Attention entropy | `attention_entropy` | Control heuristic | Control baseline | Must not appear in the paper-faithful table | ## Formula Gates ### EPR / WEPR - Native EPR is one response-level score: the mean token entropy. - Native WEPR is an unregularized response-label logistic model over the per-rank mean and max of top-15 entropic contributions (30 features). - The response-trained mean-rank weights may then localize token risk. - Use natural-log entropy for official-code parity; a base-2 sensitivity differs by a constant scale and must preserve ranking. ### Lookback Lens - For causal matched token `t`, read query row `prompt_len + t - 1`; token zero is prompt-only and the target token is never visible. - The separately labeled method-faithful suite may use each generated-token row: `context_attention / (context_attention + generated_attention)`. - With `window_size > 1` and `full_windows_only=True`, score only tokens with a complete trailing window. - The reference code min-pools factual labels where hallucination is `0`; this branch stores hallucination as `1`, so the equivalent window target is max-pooled hallucination labels. - Logistic classifier uses the train split only. The historical helper that silently used generated-token rows in the common comparison is retracted. Checkpoints now bind alignment, feature schema, and information horizon. ### Multi-View Attention - Feature families are `key_avg`, `key_entropy`, and `query_entropy`. - `key_avg` and `key_entropy` use the paper's adjusted incoming attention. - Paper feature columns are standardized from the train split. - The strict path uses the public repo Transformer block: GELU feed-forward, fixed 2048 feed-forward width, dropout before positional addition, `torchcrf` start/end-transition semantics, CRF transition initialization, and CRF Viterbi decode scores. - Training uses AdamW, ReduceLROnPlateau validation-loss scheduling, validation F1 early stopping with `delta=0.01`, batch size 64, up to 150 epochs, and 200 Optuna trials over `model_dim`, `num_heads`, `num_layers`, `dropout`, `lr`, and `weight_decay`. - The logistic feature detector is retained only as a control baseline. ## Reporting Gate The final report must split results into: - Strict reference-method outputs. - Local calibrations. - Controls/adaptations. Any method that fails a formula, feature, model/head, replay, or train/eval gate must be moved out of the paper-faithful section.