|
Download docs/fidelity_matrix.md from Anonymous1611/Lattice_Spike: direct link, hf CLI and curl.
- Browser
- Download file 5.05 kB
-
https://huggingface.co/Anonymous1611/Lattice_Spike/resolve/main/docs/fidelity_matrix.md
- Command line
-
hf download hf://Anonymous1611/Lattice_Spike/docs/fidelity_matrix.md
-
curl -L -o fidelity_matrix.md https://huggingface.co/Anonymous1611/Lattice_Spike/resolve/main/docs/fidelity_matrix.md
5.05 kB
Fidelity Matrix
This branch separates strict detector-method outputs from controls and local calibrations. A method is reported in the strict artifact only when its formula, feature shape, model/head, token alignment, prediction mode, and train/eval isolation match the selected paper or public reference target.
Input Replay Gate
- Faithful LLaMA runs must use
meta-llama/Llama-3.2-1B-Instruct. - Faithful runs require saved
generated_token_idsand teacher-force the exact generated answer tokens. - Reportable runs pin one immutable model/tokenizer commit, verify that both generation revisions match it, require the saved IDs to decode exactly to the answer, and recompute the generation-time effective prompt-token hash.
--allow-token-id-model-mismatchis only for diagnostic/non-faithful runs.--allow-unverified-replayexplicitly marks legacy provenance as diagnostic; it cannot appear in a strict result and is the only mode that permits missing answer IDs to be retokenized.- Prediction rows record whether saved answer-token IDs were required and used, whether either override was allowed, requested/resolved scoring revisions, and prompt replay verification status.
- Every trained method consumes the same dataset-bound, duplicate-group-safe
--split-summary; method-local row splits are diagnostic only.
Method Status
| Method | Current output method | Source target | Status | Required evidence |
|---|---|---|---|---|
| EPR | response-level EPR | Paper response score | Pending method-faithful suite | Response mean token entropy and exact token replay |
| Entropy-rank token probe | entropy_rank_token_probe |
Local token control | Control, not EPR/WEPR | Token-label training must not be described as native WEPR |
| Lookback Lens | lookback_lens |
Lookback Lens repo | Versioned causal and method-faithful paths | Causal pre-token alignment for matched results; post-emission full windows only in the separately labeled method-faithful suite |
| MVA logistic | mva_feature_logistic_baseline |
MVA feature families only | Control baseline, not paper model | Output name and metadata must say logistic control |
| MVA Transformer-CRF | multi_view_attention_transformer_crf |
MVA paper method | Strict architecture/prediction path | Transformer encoder + CRF, paper feature families, CRF decode, train split only |
| Hidden-state probe | hidden_state_probe |
Control/comparator | Control baseline unless tied to a specific paper | Must not appear in the paper-faithful table |
| Attention entropy | attention_entropy |
Control heuristic | Control baseline | Must not appear in the paper-faithful table |
Formula Gates
EPR / WEPR
- Native EPR is one response-level score: the mean token entropy.
- Native WEPR is an unregularized response-label logistic model over the per-rank mean and max of top-15 entropic contributions (30 features).
- The response-trained mean-rank weights may then localize token risk.
- Use natural-log entropy for official-code parity; a base-2 sensitivity differs by a constant scale and must preserve ranking.
Lookback Lens
- For causal matched token
t, read query rowprompt_len + t - 1; token zero is prompt-only and the target token is never visible. - The separately labeled method-faithful suite may use each generated-token row:
context_attention / (context_attention + generated_attention). - With
window_size > 1andfull_windows_only=True, score only tokens with a complete trailing window. - The reference code min-pools factual labels where hallucination is
0; this branch stores hallucination as1, so the equivalent window target is max-pooled hallucination labels. - Logistic classifier uses the train split only.
The historical helper that silently used generated-token rows in the common comparison is retracted. Checkpoints now bind alignment, feature schema, and information horizon.
Multi-View Attention
- Feature families are
key_avg,key_entropy, andquery_entropy. key_avgandkey_entropyuse the paper's adjusted incoming attention.- Paper feature columns are standardized from the train split.
- The strict path uses the public repo Transformer block: GELU feed-forward,
fixed 2048 feed-forward width, dropout before positional addition,
torchcrfstart/end-transition semantics, CRF transition initialization, and CRF Viterbi decode scores. - Training uses AdamW, ReduceLROnPlateau validation-loss scheduling, validation
F1 early stopping with
delta=0.01, batch size 64, up to 150 epochs, and 200 Optuna trials overmodel_dim,num_heads,num_layers,dropout,lr, andweight_decay. - The logistic feature detector is retained only as a control baseline.
Reporting Gate
The final report must split results into:
- Strict reference-method outputs.
- Local calibrations.
- Controls/adaptations.
Any method that fails a formula, feature, model/head, replay, or train/eval gate must be moved out of the paper-faithful section.