Spaces:
Sleeping
Sleeping
Commit History
-
Rename src modules for head/pair symmetry; add pair_figures.smk; split eval_metrics
src/transformer_analysis/: weight_analysis → head_pipeline attn_head_analysis → head_analyzer metrics → head_metrics correlation_analysis → pair_pipeline head_correlations → pair_analyzer eval_metrics.py (new — implementation extracted from compute_perplexity.py) scripts/: plot_correlations.py → plot_pair_correlations.py plot_corr_figures.py → plot_pair_correlations_batch.py compute_perplexity.py → deleted (impl in eval_metrics; thin wrapper run_eval.py) run_eval.py (new thin wrapper) rules/: correlations.smk — switched to sentinel output (consistent with other rules) pair_figures.smk (new — regenerate pair figures from .npz without re-running correlations) eval.smk — updated to call run_eval.py Snakefile: added correlations_all, pair_figures_all convenience targets Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>a81fd74
-
feat: add eval metrics pipeline (perplexity, paper reference, dashboard overlay)
- compute_perplexity.py: sliding-window eval on WikiText-103 or Pile subset, configurable via --corpus/--pile-tokens; uses eval_loop + NLLCollector pattern designed for future data-weighted statistics (dressed operators <W_QK>_{data}) - extract_paper_loss.py: fetch HF model-card benchmarks and compare vs reference - paper_reference_loss.csv: reference loss values from literature for validation - load_eval_metrics() in dashboard_utils.py: graceful parquet loader - step_evolution.py: optional secondary-axis overlay of eval metrics vs step Output schema: model, revision, step, metric, value, source, corpus 43 tests passing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>17def50