Lattice_Spike / docs /fidelity_matrix.md
Md Saiful Islam Sajol
Upload Lattice_Spike project with full LFS weights
e2c9d42 verified
|
Raw History Blame Contribute Delete
5.05 kB
# Fidelity Matrix
This branch separates strict detector-method outputs from controls and local
calibrations. A method is reported in the strict artifact only when its formula,
feature shape, model/head, token alignment, prediction mode, and train/eval
isolation match the selected paper or public reference target.
## Input Replay Gate
- Faithful LLaMA runs must use `meta-llama/Llama-3.2-1B-Instruct`.
- Faithful runs require saved `generated_token_ids` and teacher-force the exact
generated answer tokens.
- Reportable runs pin one immutable model/tokenizer commit, verify that both
generation revisions match it, require the saved IDs to decode exactly to the
answer, and recompute the generation-time effective prompt-token hash.
- `--allow-token-id-model-mismatch` is only for diagnostic/non-faithful runs.
- `--allow-unverified-replay` explicitly marks legacy provenance as diagnostic;
it cannot appear in a strict result and is the only mode that permits missing
answer IDs to be retokenized.
- Prediction rows record whether saved answer-token IDs were required and used,
whether either override was allowed, requested/resolved scoring revisions,
and prompt replay verification status.
- Every trained method consumes the same dataset-bound, duplicate-group-safe
`--split-summary`; method-local row splits are diagnostic only.
## Method Status
| Method | Current output method | Source target | Status | Required evidence |
| --- | --- | --- | --- | --- |
| EPR | response-level EPR | Paper response score | Pending method-faithful suite | Response mean token entropy and exact token replay |
| Entropy-rank token probe | `entropy_rank_token_probe` | Local token control | Control, not EPR/WEPR | Token-label training must not be described as native WEPR |
| Lookback Lens | `lookback_lens` | Lookback Lens repo | Versioned causal and method-faithful paths | Causal pre-token alignment for matched results; post-emission full windows only in the separately labeled method-faithful suite |
| MVA logistic | `mva_feature_logistic_baseline` | MVA feature families only | Control baseline, not paper model | Output name and metadata must say logistic control |
| MVA Transformer-CRF | `multi_view_attention_transformer_crf` | MVA paper method | Strict architecture/prediction path | Transformer encoder + CRF, paper feature families, CRF decode, train split only |
| Hidden-state probe | `hidden_state_probe` | Control/comparator | Control baseline unless tied to a specific paper | Must not appear in the paper-faithful table |
| Attention entropy | `attention_entropy` | Control heuristic | Control baseline | Must not appear in the paper-faithful table |
## Formula Gates
### EPR / WEPR
- Native EPR is one response-level score: the mean token entropy.
- Native WEPR is an unregularized response-label logistic model over the
per-rank mean and max of top-15 entropic contributions (30 features).
- The response-trained mean-rank weights may then localize token risk.
- Use natural-log entropy for official-code parity; a base-2 sensitivity differs
by a constant scale and must preserve ranking.
### Lookback Lens
- For causal matched token `t`, read query row `prompt_len + t - 1`; token zero
is prompt-only and the target token is never visible.
- The separately labeled method-faithful suite may use each generated-token row:
`context_attention / (context_attention + generated_attention)`.
- With `window_size > 1` and `full_windows_only=True`, score only tokens with a
complete trailing window.
- The reference code min-pools factual labels where hallucination is `0`; this
branch stores hallucination as `1`, so the equivalent window target is
max-pooled hallucination labels.
- Logistic classifier uses the train split only.
The historical helper that silently used generated-token rows in the common
comparison is retracted. Checkpoints now bind alignment, feature schema, and
information horizon.
### Multi-View Attention
- Feature families are `key_avg`, `key_entropy`, and `query_entropy`.
- `key_avg` and `key_entropy` use the paper's adjusted incoming attention.
- Paper feature columns are standardized from the train split.
- The strict path uses the public repo Transformer block: GELU feed-forward,
fixed 2048 feed-forward width, dropout before positional addition,
`torchcrf` start/end-transition semantics, CRF transition initialization, and
CRF Viterbi decode scores.
- Training uses AdamW, ReduceLROnPlateau validation-loss scheduling, validation
F1 early stopping with `delta=0.01`, batch size 64, up to 150 epochs, and
200 Optuna trials over `model_dim`, `num_heads`, `num_layers`, `dropout`,
`lr`, and `weight_decay`.
- The logistic feature detector is retained only as a control baseline.
## Reporting Gate
The final report must split results into:
- Strict reference-method outputs.
- Local calibrations.
- Controls/adaptations.
Any method that fails a formula, feature, model/head, replay, or train/eval gate
must be moved out of the paper-faithful section.