Lattice_Spike / docs /fidelity_matrix.md
Md Saiful Islam Sajol
Upload Lattice_Spike project with full LFS weights
e2c9d42 verified
|
Raw History Blame Contribute Delete
5.05 kB

Fidelity Matrix

This branch separates strict detector-method outputs from controls and local calibrations. A method is reported in the strict artifact only when its formula, feature shape, model/head, token alignment, prediction mode, and train/eval isolation match the selected paper or public reference target.

Input Replay Gate

  • Faithful LLaMA runs must use meta-llama/Llama-3.2-1B-Instruct.
  • Faithful runs require saved generated_token_ids and teacher-force the exact generated answer tokens.
  • Reportable runs pin one immutable model/tokenizer commit, verify that both generation revisions match it, require the saved IDs to decode exactly to the answer, and recompute the generation-time effective prompt-token hash.
  • --allow-token-id-model-mismatch is only for diagnostic/non-faithful runs.
  • --allow-unverified-replay explicitly marks legacy provenance as diagnostic; it cannot appear in a strict result and is the only mode that permits missing answer IDs to be retokenized.
  • Prediction rows record whether saved answer-token IDs were required and used, whether either override was allowed, requested/resolved scoring revisions, and prompt replay verification status.
  • Every trained method consumes the same dataset-bound, duplicate-group-safe --split-summary; method-local row splits are diagnostic only.

Method Status

Method Current output method Source target Status Required evidence
EPR response-level EPR Paper response score Pending method-faithful suite Response mean token entropy and exact token replay
Entropy-rank token probe entropy_rank_token_probe Local token control Control, not EPR/WEPR Token-label training must not be described as native WEPR
Lookback Lens lookback_lens Lookback Lens repo Versioned causal and method-faithful paths Causal pre-token alignment for matched results; post-emission full windows only in the separately labeled method-faithful suite
MVA logistic mva_feature_logistic_baseline MVA feature families only Control baseline, not paper model Output name and metadata must say logistic control
MVA Transformer-CRF multi_view_attention_transformer_crf MVA paper method Strict architecture/prediction path Transformer encoder + CRF, paper feature families, CRF decode, train split only
Hidden-state probe hidden_state_probe Control/comparator Control baseline unless tied to a specific paper Must not appear in the paper-faithful table
Attention entropy attention_entropy Control heuristic Control baseline Must not appear in the paper-faithful table

Formula Gates

EPR / WEPR

  • Native EPR is one response-level score: the mean token entropy.
  • Native WEPR is an unregularized response-label logistic model over the per-rank mean and max of top-15 entropic contributions (30 features).
  • The response-trained mean-rank weights may then localize token risk.
  • Use natural-log entropy for official-code parity; a base-2 sensitivity differs by a constant scale and must preserve ranking.

Lookback Lens

  • For causal matched token t, read query row prompt_len + t - 1; token zero is prompt-only and the target token is never visible.
  • The separately labeled method-faithful suite may use each generated-token row: context_attention / (context_attention + generated_attention).
  • With window_size > 1 and full_windows_only=True, score only tokens with a complete trailing window.
  • The reference code min-pools factual labels where hallucination is 0; this branch stores hallucination as 1, so the equivalent window target is max-pooled hallucination labels.
  • Logistic classifier uses the train split only.

The historical helper that silently used generated-token rows in the common comparison is retracted. Checkpoints now bind alignment, feature schema, and information horizon.

Multi-View Attention

  • Feature families are key_avg, key_entropy, and query_entropy.
  • key_avg and key_entropy use the paper's adjusted incoming attention.
  • Paper feature columns are standardized from the train split.
  • The strict path uses the public repo Transformer block: GELU feed-forward, fixed 2048 feed-forward width, dropout before positional addition, torchcrf start/end-transition semantics, CRF transition initialization, and CRF Viterbi decode scores.
  • Training uses AdamW, ReduceLROnPlateau validation-loss scheduling, validation F1 early stopping with delta=0.01, batch size 64, up to 150 epochs, and 200 Optuna trials over model_dim, num_heads, num_layers, dropout, lr, and weight_decay.
  • The logistic feature detector is retained only as a control baseline.

Reporting Gate

The final report must split results into:

  • Strict reference-method outputs.
  • Local calibrations.
  • Controls/adaptations.

Any method that fails a formula, feature, model/head, replay, or train/eval gate must be moved out of the paper-faithful section.