|
Download docs/fidelity_matrix.md from Anonymous1611/Lattice_Spike: direct link, hf CLI and curl.
- Browser
- Download file 5.05 kB
-
https://huggingface.co/Anonymous1611/Lattice_Spike/resolve/main/docs/fidelity_matrix.md
- Command line
-
hf download hf://Anonymous1611/Lattice_Spike/docs/fidelity_matrix.md
-
curl -L -o fidelity_matrix.md https://huggingface.co/Anonymous1611/Lattice_Spike/resolve/main/docs/fidelity_matrix.md
5.05 kB
| # Fidelity Matrix | |
| This branch separates strict detector-method outputs from controls and local | |
| calibrations. A method is reported in the strict artifact only when its formula, | |
| feature shape, model/head, token alignment, prediction mode, and train/eval | |
| isolation match the selected paper or public reference target. | |
| ## Input Replay Gate | |
| - Faithful LLaMA runs must use `meta-llama/Llama-3.2-1B-Instruct`. | |
| - Faithful runs require saved `generated_token_ids` and teacher-force the exact | |
| generated answer tokens. | |
| - Reportable runs pin one immutable model/tokenizer commit, verify that both | |
| generation revisions match it, require the saved IDs to decode exactly to the | |
| answer, and recompute the generation-time effective prompt-token hash. | |
| - `--allow-token-id-model-mismatch` is only for diagnostic/non-faithful runs. | |
| - `--allow-unverified-replay` explicitly marks legacy provenance as diagnostic; | |
| it cannot appear in a strict result and is the only mode that permits missing | |
| answer IDs to be retokenized. | |
| - Prediction rows record whether saved answer-token IDs were required and used, | |
| whether either override was allowed, requested/resolved scoring revisions, | |
| and prompt replay verification status. | |
| - Every trained method consumes the same dataset-bound, duplicate-group-safe | |
| `--split-summary`; method-local row splits are diagnostic only. | |
| ## Method Status | |
| | Method | Current output method | Source target | Status | Required evidence | | |
| | --- | --- | --- | --- | --- | | |
| | EPR | response-level EPR | Paper response score | Pending method-faithful suite | Response mean token entropy and exact token replay | | |
| | Entropy-rank token probe | `entropy_rank_token_probe` | Local token control | Control, not EPR/WEPR | Token-label training must not be described as native WEPR | | |
| | Lookback Lens | `lookback_lens` | Lookback Lens repo | Versioned causal and method-faithful paths | Causal pre-token alignment for matched results; post-emission full windows only in the separately labeled method-faithful suite | | |
| | MVA logistic | `mva_feature_logistic_baseline` | MVA feature families only | Control baseline, not paper model | Output name and metadata must say logistic control | | |
| | MVA Transformer-CRF | `multi_view_attention_transformer_crf` | MVA paper method | Strict architecture/prediction path | Transformer encoder + CRF, paper feature families, CRF decode, train split only | | |
| | Hidden-state probe | `hidden_state_probe` | Control/comparator | Control baseline unless tied to a specific paper | Must not appear in the paper-faithful table | | |
| | Attention entropy | `attention_entropy` | Control heuristic | Control baseline | Must not appear in the paper-faithful table | | |
| ## Formula Gates | |
| ### EPR / WEPR | |
| - Native EPR is one response-level score: the mean token entropy. | |
| - Native WEPR is an unregularized response-label logistic model over the | |
| per-rank mean and max of top-15 entropic contributions (30 features). | |
| - The response-trained mean-rank weights may then localize token risk. | |
| - Use natural-log entropy for official-code parity; a base-2 sensitivity differs | |
| by a constant scale and must preserve ranking. | |
| ### Lookback Lens | |
| - For causal matched token `t`, read query row `prompt_len + t - 1`; token zero | |
| is prompt-only and the target token is never visible. | |
| - The separately labeled method-faithful suite may use each generated-token row: | |
| `context_attention / (context_attention + generated_attention)`. | |
| - With `window_size > 1` and `full_windows_only=True`, score only tokens with a | |
| complete trailing window. | |
| - The reference code min-pools factual labels where hallucination is `0`; this | |
| branch stores hallucination as `1`, so the equivalent window target is | |
| max-pooled hallucination labels. | |
| - Logistic classifier uses the train split only. | |
| The historical helper that silently used generated-token rows in the common | |
| comparison is retracted. Checkpoints now bind alignment, feature schema, and | |
| information horizon. | |
| ### Multi-View Attention | |
| - Feature families are `key_avg`, `key_entropy`, and `query_entropy`. | |
| - `key_avg` and `key_entropy` use the paper's adjusted incoming attention. | |
| - Paper feature columns are standardized from the train split. | |
| - The strict path uses the public repo Transformer block: GELU feed-forward, | |
| fixed 2048 feed-forward width, dropout before positional addition, | |
| `torchcrf` start/end-transition semantics, CRF transition initialization, and | |
| CRF Viterbi decode scores. | |
| - Training uses AdamW, ReduceLROnPlateau validation-loss scheduling, validation | |
| F1 early stopping with `delta=0.01`, batch size 64, up to 150 epochs, and | |
| 200 Optuna trials over `model_dim`, `num_heads`, `num_layers`, `dropout`, | |
| `lr`, and `weight_decay`. | |
| - The logistic feature detector is retained only as a control baseline. | |
| ## Reporting Gate | |
| The final report must split results into: | |
| - Strict reference-method outputs. | |
| - Local calibrations. | |
| - Controls/adaptations. | |
| Any method that fails a formula, feature, model/head, replay, or train/eval gate | |
| must be moved out of the paper-faithful section. | |