Spaces:
Sleeping
Sleeping
Commit History
-
Add QK/OV equilibrium observables to correlations stageβ
Gated by correlation.equilibrium (off by default): compute per-head commutator, irreversibility, and symmetric/antisymmetric observables from the QK and OV operators. Extends extract_head_stores to materialize the OV operator when equilibrium is requested, and threads --equilibrium through the correlations rule and run_correlations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>1062742
-
Fix Pythia QKV extraction layout; add W_QK correctness testsβ
extract_pythia_qkv split the combined query_key_value.weight with qkv.chunk(3, dim=0), but HF GPT-NeoX stores QKV output features per-head-interleaved as [h0_Q, h0_K, h0_V, h1_Q, h1_K, h1_V, ...] (confirmed by reading GPTNeoXAttention.forward). The naive chunk scrambled Q, K, V across heads; downstream W_QK was the wrong bilinear form. Re-extract via per-head view + slice. Same fix applied to extract_pythia_biases. New tests under tests/test_wqk_* build tiny HF models with known weights, then verify (a) extracted W_Q/K/V match an architecture- correct reference, (b) x_i^T W_QK x_j equals the score HF's own projection computes on the same inputs, and (c) the factored d_head x d_head SVD path matches a full d_model x d_model SVD. Covers gpt2 and pythia-70m-deduped; same fixture pattern extends to Llama/Mistral/Gemma2 in a follow-up. tests/conftest.py prepends the worktree's src/ to sys.path so tests exercise the local source rather than the env-installed package (which resolves to the β¦5ecd8f7
-
Wire QK decomposition weight types into primary pipelineβ
W_Q_gram, W_K_gram, QK_alignment are already computed in head_analyzer.py. This commit exposes weight_types through config.yaml β run_primary.py β process_model() β primary.smk so they can be overridden at runtime. Tests updated to cover all 6 weight types; histogram test restricted to raw weight types since gram entries exceed standard bin range. compute_qk_decomposition.py removed (data now lives in primary dataset). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>a07e79a -
Rename src modules for head/pair symmetry; add pair_figures.smk; split eval_metricsβ
src/transformer_analysis/: weight_analysis β head_pipeline attn_head_analysis β head_analyzer metrics β head_metrics correlation_analysis β pair_pipeline head_correlations β pair_analyzer eval_metrics.py (new β implementation extracted from compute_perplexity.py) scripts/: plot_correlations.py β plot_pair_correlations.py plot_corr_figures.py β plot_pair_correlations_batch.py compute_perplexity.py β deleted (impl in eval_metrics; thin wrapper run_eval.py) run_eval.py (new thin wrapper) rules/: correlations.smk β switched to sentinel output (consistent with other rules) pair_figures.smk (new β regenerate pair figures from .npz without re-running correlations) eval.smk β updated to call run_eval.py Snakefile: added correlations_all, pair_figures_all convenience targets Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>a81fd74 -
Remove histogram_utils shim; update all imports to metrics/model_registryβ
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>e750974 -
feat: add Snakemake pipeline, MLflow tracking, metrics refactor, testsβ
Pipeline / workflow: - Snakefile + config.yaml orchestrate primary β transform β merge stages - rules/primary.smk, transform.smk, correlations.smk, merge.smk - Sentinel-file pattern (done/*.done) avoids directory-output conflicts - Incremental cross-model merge: appends new models without clobbering; --refresh flag for explicit replacement after re-transform - Per-model checkpoint merge rule for Pythia step-collapse CLI wrappers (scripts/): - run_primary.py β process_model() + MLflow logging - run_transform.py β reprocess_metrics() + MLflow logging (git sha tagged) - run_correlations.py β thin wrapper for run_multi_circuit_analysis() - run_merge.py β cross-model incremental merge + checkpoint merge Metrics refactor: - src/transformer_analysis/metrics.py: canonical home for all stat functions, bin configs, normality_metrics, singular_value_metrics, and adaptive binning helpers (make_weight_bins, make_sv_bins) - histogram_utils.py: backward-compat re-export shim; no logic cβ¦24e1932
-
feat: add eval metrics pipeline (perplexity, paper reference, dashboard overlay)β
- compute_perplexity.py: sliding-window eval on WikiText-103 or Pile subset, configurable via --corpus/--pile-tokens; uses eval_loop + NLLCollector pattern designed for future data-weighted statistics (dressed operators <W_QK>_{data}) - extract_paper_loss.py: fetch HF model-card benchmarks and compare vs reference - paper_reference_loss.csv: reference loss values from literature for validation - load_eval_metrics() in dashboard_utils.py: graceful parquet loader - step_evolution.py: optional secondary-axis overlay of eval metrics vs step Output schema: model, revision, step, metric, value, source, corpus 43 tests passing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>17def50
-
feat: add eigenvalue staircase sandbox dashboard pageβ
Standalone Streamlit page (dashboards/pages/sandbox.py) implementing RMT-style eigenvalue unfolding: staircase N(Ξ»), polynomial unfolded ΞΎα΅’, and spacing distribution P(s) with Wigner-Dyson and Poisson overlays. Runs via 'streamlit run dashboards/pages/sandbox.py' -- main app untouched. Pure numpy helpers (staircase, unfold, spacings) are module-level so tests can import them without Streamlit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>a5fcf5c -
feat: add KDE toggle for histogram plots in weights dashboardβ
Adds kde_from_histogram() to dashboard_utils β reconstructs a smooth KDE from stored histogram bin centers and counts using scipy gaussian_kde with Scott's bandwidth. Section 1 of the weights dashboard gets a 'Show as KDE' checkbox that replaces bar charts with filled line plots for both P(W) and P(Ξ») views. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>3a5adc4 -
feat: add configurable binning strategy hooks to histogram_utilsβ
Adds make_weight_bins() and make_sv_bins() with 'fixed' (default, preserves existing behavior), 'scott', and 'fd' strategies. The fixed path returns the same arrays as the existing defaults. Adaptive strategies require a data sample at call time. Wired through process_model() and the run_weight_analysis.py CLI as --binning-strategy. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>54eb2a2 -
feat: add W_Q_gram, W_K_gram, QK_alignment weight types to pipelineβ
Promotes the QK factorization decomposition from the side-table script into first-class weight types alongside W_Q/W_K/W_QK, as designed in docs/qk_decomposition_pipeline.md. Gram SVDs store sqrt(eigenvalues) so all existing SV metrics apply without modification. QK_alignment stores the d_head principal-angle cosines between the W_Q and W_K column spaces. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>2dc4a80