Spaces:
Sleeping
Sleeping
Commit History
-
Sanitize cache_dir: portable default + CACHE_DIR env override
Replace the hardcoded /Volumes/Flux cache path in config.yaml and the experiment_svd_truncation default with a portable "downloads" default, and add a CACHE_DIR env override in the Snakefile (mirroring OUTPUT_DIR). Point CACHE_DIR at a big external volume to reuse downloaded weights without committing a local path to the repo. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>15cbcc7
-
Add ultrametricity analysis stage for head-overlap matrices
Post-processing stage over the correlations Q_hh matrices. New module transformer_analysis.ultrametricity computes, per model and per metric (Frobenius chordal distance, Jensen-Shannon distance): - triple Rammal index u (3-point ultrametricity), per-head map, u-vs-layer-span, tight-pair co-occurrence, triangle-violation check - four-point additivity index u4 and Gromov delta-hyperbolicity - average-linkage dendrogram, cophenetic correlation, and the D=C+R residual decomposition (effective rank, eigenvalues/eigenvectors, sparsity verdict) - classical MDS / PCoA (eigenvalues + leading principal coordinates) - cophenetic correlation across linkage rules - an untrained null ensemble for the distance distribution run_ultrametricity.py reads the correlation .npz outputs (no model download), writes per-metric .npz + summary JSON, and logs to MLflow. Wired in as an optional snakemake stage (rules/ultrametricity.smk, ultrametricity_all target) with a config.yaml section. Co-Authored-By: …840479b
-
Add pipeline config for OLMo 2, Llama 3.2, Gemma 2, SmolLM2
Wire all 11 new models into config.yaml runs/targets and extend target_all in Snakefile to compose the four new family targets. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>3d53530 -
Namespace all outputs under experiment_dir, add MLflow run metadata
All pipeline outputs and sentinels now live under {output_dir}/{mlflow_experiment}/ instead of scattered across the project root and output_dir. Prevents clobbering across experiments and mirrors the MLflow hierarchy. Changes: - Snakefile: derive config["experiment_dir"], pre-create it, override eval.out dynamically; update all sentinel expand() paths - rules/*: sentinel touch paths and inter-rule deps use experiment_dir; correlations/figures dirs moved under experiment_dir - config.yaml: remove hardcoded eval.out (now derived in Snakefile) - scripts/run_*.py: add mlflow.note.content description, model tag, and DatasetInput (local source → output path) to every run Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>0bf8fb1
-
Add docstrings to all static rules for snakemake --list
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>fbffedd -
Add clean rules: clean, clean_transform, clean_mlflow, clean_run, clean_<family>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>92dcf5b -
Refactor eval to per-run parquets + merge_eval, remove -j1 constraint
Each eval job now writes outputs/eval/{run_key}.parquet independently. merge_eval concatenates them into the final artifact. eval_all depends on merge_eval output rather than individual sentinels. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>f49060a -
Make target_all compose from family rules instead of re-listing runs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>5717451 -
Add named family targets and pythia checkpoint sweep targets
config.yaml: - runs: expanded to full gpt2/pythia/llama+mistral model sets - targets: gpt2_family, pythia_family, llama_family, all - pythia_steps: disabled by default; enable + add runs to get target_pythia_{shortname}_steps covering all PYTHIA_REVISIONS Snakefile: - Loop over config["targets"] → target_{name} rules - Loop over config["pythia_steps"] → target_pythia_{shortname}_steps rules (reads PYTHIA_REVISIONS from model_registry) Usage: snakemake target_gpt2_family -j4 snakemake target_pythia_family -j4 snakemake target_llama_family -j4 snakemake target_all -j8 snakemake target_pythia_70m_steps -j4 # after enabling in pythia_steps Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>98c58da -
Merge worktree branch: module renames, eval/figures pipeline, histogram_utils removal
- src/: weight_analysis→head_pipeline, attn_head_analysis→head_analyzer, metrics→head_metrics, correlation_analysis→pair_pipeline, head_correlations→pair_analyzer, eval_metrics (new) - histogram_utils.py deleted; all imports updated to metrics/model_registry - rules/eval.smk, pair_figures.smk added; correlations.smk switched to sentinel - scripts/run_eval.py, plot_pair_correlations.py, plot_pair_correlations_batch.py - Snakefile: SQLite MLflow backend kept (resolves conflict with worktree file: URI) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>579f9be -
Pre-initialize MLflow DB in Snakefile to prevent parallel init race
With -j2, two primary jobs both tried to CREATE TABLE on the SQLite DB simultaneously. Initializing once in the Snakefile (single-threaded, before any jobs start) ensures the tables exist before workers connect. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>c9d164f -
Fix KeyError: derive mlflow_uri from output_dir in Snakefile
config.yaml no longer carries mlflow_uri; Snakefile derives it as file:{output_dir}/mlruns so runs and datasets travel together. Supports OUTPUT_DIR env var override at runtime. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>6e01ed5 -
Derive mlflow_uri from output_dir; support OUTPUT_DIR env var
mlruns/ now lives inside output_dir so datasets and their MLflow runs travel together. OUTPUT_DIR env var overrides output_dir at runtime: OUTPUT_DIR=/mnt/drive/ana-005 snakemake -j8 mlflow ui --backend-store-uri /mnt/drive/ana-005/mlruns mlflow_uri removed from config.yaml (derived in Snakefile). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>85dfcd2 -
Rename src modules for head/pair symmetry; add pair_figures.smk; split eval_metrics
src/transformer_analysis/: weight_analysis → head_pipeline attn_head_analysis → head_analyzer metrics → head_metrics correlation_analysis → pair_pipeline head_correlations → pair_analyzer eval_metrics.py (new — implementation extracted from compute_perplexity.py) scripts/: plot_correlations.py → plot_pair_correlations.py plot_corr_figures.py → plot_pair_correlations_batch.py compute_perplexity.py → deleted (impl in eval_metrics; thin wrapper run_eval.py) run_eval.py (new thin wrapper) rules/: correlations.smk — switched to sentinel output (consistent with other rules) pair_figures.smk (new — regenerate pair figures from .npz without re-running correlations) eval.smk — updated to call run_eval.py Snakefile: added correlations_all, pair_figures_all convenience targets Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>a81fd74 -
Add eval.smk, fix correlations input, remove superseded scripts
- rules/eval.smk: wraps compute_perplexity.py per run; sentinel-based, depends on transform; eval_all convenience target in Snakefile - rules/correlations.smk: fix input to use transform sentinel instead of bare dataset directory (which Snakemake couldn't track) - config.yaml: add eval block (corpus, pile_tokens, pile_cache, out) - Delete run_weight_analysis.py, run_all_correlations.py, merge_datasets.py (superseded by run_primary.py, run_correlations.py, run_merge.py) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>6ae3047 -
feat: add Snakemake pipeline, MLflow tracking, metrics refactor, tests
Pipeline / workflow: - Snakefile + config.yaml orchestrate primary → transform → merge stages - rules/primary.smk, transform.smk, correlations.smk, merge.smk - Sentinel-file pattern (done/*.done) avoids directory-output conflicts - Incremental cross-model merge: appends new models without clobbering; --refresh flag for explicit replacement after re-transform - Per-model checkpoint merge rule for Pythia step-collapse CLI wrappers (scripts/): - run_primary.py — process_model() + MLflow logging - run_transform.py — reprocess_metrics() + MLflow logging (git sha tagged) - run_correlations.py — thin wrapper for run_multi_circuit_analysis() - run_merge.py — cross-model incremental merge + checkpoint merge Metrics refactor: - src/transformer_analysis/metrics.py: canonical home for all stat functions, bin configs, normality_metrics, singular_value_metrics, and adaptive binning helpers (make_weight_bins, make_sv_bins) - histogram_utils.py: backward-compat re-export shim; no logic c…24e1932