Spaces:
Sleeping
Sleeping
Commit History
-
swap README for HF
github-actions[bot]e49f378 -
Merge claude/determined-kirch-620dbd into main
19c4fea -
README: badges, pipeline layout, commands, config reference
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>18d1a25 -
Add task manager app and model prefetch script
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>5e3dd78 -
Merge claude/friendly-lichterman-da3b5d into main
75ddf93
-
Sanitize cache_dir: portable default + CACHE_DIR env override
Replace the hardcoded /Volumes/Flux cache path in config.yaml and the experiment_svd_truncation default with a portable "downloads" default, and add a CACHE_DIR env override in the Snakefile (mirroring OUTPUT_DIR). Point CACHE_DIR at a big external volume to reuse downloaded weights without committing a local path to the repo. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>15cbcc7 -
Track previously-untracked utility scripts and example config
These were present locally but never added: the HF dataset builder, the MP statistics compute/inspect scripts, the viewer-index generator, and the example correlations config. Code/config only — no secrets. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>f51ee1f -
Ignore data, caches, worktrees, and private notes; untrack .snakemake
Public repo hygiene. Exclude large/local artifacts (outputs/, mlruns/, mlflow.db, snapshots/, mp_statistics/), generated campaign configs (config.*.yaml — config.yaml stays tracked), dev scratch/, the .claude/ worktree tree, and the .snakemake cache (also removed from tracking). Also exclude private research notes and personal prefs that should not land in this public repo's history: docs/, CLAUDE.md, tags.css, and the gram-matrices exploration notebook. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>698e9d9 -
Merge W_OV equilibrium observables into main
7939cd5 -
Add QK/OV equilibrium observables to correlations stage
Gated by correlation.equilibrium (off by default): compute per-head commutator, irreversibility, and symmetric/antisymmetric observables from the QK and OV operators. Extends extract_head_stores to materialize the OV operator when equilibrium is requested, and threads --equilibrium through the correlations rule and run_correlations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>1062742 -
Step Evolution: list only merged *_all_checkpoints datasets
The dataset selector was showing every per-step run; filter it to the merged checkpoint datasets and display them by model name. Falls back to the full list when no *_all_checkpoints datasets are present (HF-hosted Space). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>494c501 -
Speed up primary element stats and cut peak memory
Replace the per-head np.histogram + scipy stats with one fused pass: O(n) uniform-bin histogram, shared-central-moment mean/std/skew/kurtosis (bit-identical to scipy), and a histogram plug-in differential_entropy that drops scipy's O(n log n) Vasicek sort. ~18x faster per-head element stats. Build W_QK one head at a time instead of the full (n_heads, d_model, d_model) tensor, removing an ~838 MB transient and a pointless device round-trip (W_QK is only histogrammed on CPU). This is what relieves swap thrashing for the large models. Also add a gated svd_via_gram option (off by default) deriving W_Q/W_K U,S from the d_head Gram eigendecomposition instead of a wide MPS->CPU-fallback SVD. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>fa476b7
-
Add tight-pair bond ultrametricity scores
For each tight pair (a,b) — mutual nearest neighbors or the k smallest-D pairs — average the Rammal index u over triples where (a,b) is the shortest edge; the bond score 1-<u> separates the ultrametric backbone (high) from specific contacts (low). Compared against a null computed the same way on a distance matrix with shuffled off-diagonals; low-score pairs are expected to match the most-negative D-C residual pairs, and bimodality marks the backbone-vs-contact split. tight_pair_scores returns the pair indices, per-bond scores, and null so they can be cross-referenced against the R eigenvectors. Persisted to the .npz/summary and surfaced in the dashboard with live method (MNN / smallest-D) and k controls, a score-vs-null histogram, and a flagged low-score table tagged with residual overlap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>21da706 -
Fix corrupted mlflow_experiment name in config
mlflow_experiment held a fat-fingered value ('ana-006smakemake -open-production') which made experiment_dir resolve to a non-existent directory. Set it to the actual production experiment, ana-006-open-production. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>1b89601 -
Add ultrametricity page to the correlations app
Convert apps/correlations_app.py into a two-view app (figure gallery + ultrametricity) with a shared experiment selector; experiment detection now also picks up dirs with ultrametricity/ outputs. New self-contained apps/ultrametricity_view.py renders: - a d/C corner plot (log-default marginals, range-clipped log-color joint, untrained-null overlay on P(d)) plus d and C heatmaps - per-head u map, P(u) vs P(u4), u-vs-layer-span - model comparison (linkage rules), specific tree-residual pairs - residual structure: R heatmap, P(R), eigenvalue scree, top-N eigenvector maps - classical MDS: scree + leading principal-coordinate maps - tight-pair co-occurrence and per-head drill-down Heatmaps aggregate to layer-level above 256 heads; the per-head dendrogram is limited to small models (a per-head tree is the wrong view at large scale). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>6dd9345 -
Add ultrametricity analysis stage for head-overlap matrices
Post-processing stage over the correlations Q_hh matrices. New module transformer_analysis.ultrametricity computes, per model and per metric (Frobenius chordal distance, Jensen-Shannon distance): - triple Rammal index u (3-point ultrametricity), per-head map, u-vs-layer-span, tight-pair co-occurrence, triangle-violation check - four-point additivity index u4 and Gromov delta-hyperbolicity - average-linkage dendrogram, cophenetic correlation, and the D=C+R residual decomposition (effective rank, eigenvalues/eigenvectors, sparsity verdict) - classical MDS / PCoA (eigenvalues + leading principal coordinates) - cophenetic correlation across linkage rules - an untrained null ensemble for the distance distribution run_ultrametricity.py reads the correlation .npz outputs (no model download), writes per-metric .npz + summary JSON, and logs to MLflow. Wired in as an optional snakemake stage (rules/ultrametricity.smk, ultrametricity_all target) with a config.yaml section. Co-Authored-By: …840479b
-
Eval: optional layer offload via accelerate for >RAM models
Adds an opt-in --offload-layers path that loads the model with accelerate.init_empty_weights + load_checkpoint_and_dispatch, so layers stream from disk through a bounded memory budget instead of all sitting resident. Unlocks models that don't fit in unified memory in fp16 (e.g. pythia-2.8b on a 16 GB M4) at the cost of much slower per-window inference. Numerics are unchanged — perplexity matches a non-offload reference within fp16 noise (verified on pythia-70m, delta ppl < 0.1%). CLI: --offload-layers turn it on --offload-folder PATH spill directory; defaults to tempfile.mkdtemp() on the system disk (best per-window bandwidth, auto-cleaned). Override to put it on an external drive when the system disk is tight on space. --max-memory-gpu STR e.g. "4GiB"; budget for the accelerator --max-memory-cpu STR e.g. "2GiB"; budget for CPU before disk spill evaluate_model owns the t…66a0324 -
Eval: load in the model's native dtype, add --dtype CLI flag
The eval pipeline hardcoded torch_dtype=torch.float32, which made models >1-2B unrunnable on local Apple Silicon (3-7B in fp32 spills into swap and grinds to a halt). Switch to torch_dtype="auto" by default — Llama/Mistral/Gemma2/OLMo2/SmolLM2 load in bf16, Pythia in fp16, GPT-2 in fp32, matching each model's published dtype. NLL is still computed in fp32 inside HF's ForCausalLMLoss (it explicitly upcasts logits.float() before cross-entropy), so the perplexity numbers stay well-defined. low_cpu_mem_usage=True avoids the transient fp32 copy during from_pretrained, ~halving peak load memory. Add a --dtype CLI knob on run_eval.py (auto|bf16|fp16|fp32) threaded through to evaluate_model so a numerics reference run is one flag away. Also log dtype and device as MLflow params for reproducibility. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>937097a -
Pick best accelerator by default; route SVDs through MPS-safe path
get_device now uses CUDA -> MPS -> CPU priority so Apple Silicon picks MPS by default. The previous macOS->CPU branch made forward passes 10-50x slower than they need to be. CLI override (--device cuda|mps|cpu) still wins. Drop the stale "MPS has stability issues" comment; forward-only inference on MPS is solid in torch >= 2. torch.linalg.svdvals is NotImplemented on MPS in current torch, but torch.linalg.svd works. Add a _svdvals() helper in head_analyzer that delegates to the existing _svd() (and thus inherits its numpy fallback for ill-conditioned matrices). Swap the three svdvals call sites. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>2207aa9
-
Fix transform/primary path mismatch when revision is null
reprocess_metrics looked for {model} but primary writes {model}_main, so revision-less runs silently skipped (the Snakemake sentinel was still touched, hiding the breakage). Use the same "main" fallback string in both places. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>2dcdcdf -
Skip transform plotting cleanly when refined dataset is missing
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>e40b404 -
Log P(W) and std summary plots as MLflow artifacts in transform step
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>2999fae -
Fix small bugs in head_pipeline.py
- merge_datasets: isinstance(str, suffix) had args reversed; the branch was dead code. Now correctly checks isinstance(suffix, str). - process_model: revision.strip("step") strips a character set from both ends, not a prefix; silently truncates trailing s/t/e/p chars. Use removeprefix("step") so non-trivial revision names parse safely. - reprocess_metrics: drop the inner `import shutil as _shutil` that shadowed the file-level shutil import. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>cf0f677 -
Fix Pythia QKV extraction layout; add W_QK correctness tests
extract_pythia_qkv split the combined query_key_value.weight with qkv.chunk(3, dim=0), but HF GPT-NeoX stores QKV output features per-head-interleaved as [h0_Q, h0_K, h0_V, h1_Q, h1_K, h1_V, ...] (confirmed by reading GPTNeoXAttention.forward). The naive chunk scrambled Q, K, V across heads; downstream W_QK was the wrong bilinear form. Re-extract via per-head view + slice. Same fix applied to extract_pythia_biases. New tests under tests/test_wqk_* build tiny HF models with known weights, then verify (a) extracted W_Q/K/V match an architecture- correct reference, (b) x_i^T W_QK x_j equals the score HF's own projection computes on the same inputs, and (c) the factored d_head x d_head SVD path matches a full d_model x d_model SVD. Covers gpt2 and pythia-70m-deduped; same fixture pattern extends to Llama/Mistral/Gemma2 in a follow-up. tests/conftest.py prepends the worktree's src/ to sys.path so tests exercise the local source rather than the env-installed package (which resolves to the …5ecd8f7 -
Fix mlflow.log_input in all remaining pipeline scripts
Same fix as run_primary.py: mlflow.log_input expects mlflow.data.Dataset, not a raw DatasetInput entity. Use MlflowClient().log_inputs() throughout. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>88dbab4 -
Fix two OLMo 2 pipeline failures
mlflow.log_input expects an mlflow.data.Dataset, not a raw DatasetInput entity — use MlflowClient().log_inputs() which takes the lower-level type directly. torch.linalg.svd uses LAPACK gesdd which fails to converge on ill-conditioned weight matrices seen in OLMo 2. Add _svd() helper that catches LinAlgError and retries via numpy.linalg.svd (gesvd driver, always converges). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>7b59fb4
-
Add pipeline config for OLMo 2, Llama 3.2, Gemma 2, SmolLM2
Wire all 11 new models into config.yaml runs/targets and extend target_all in Snakefile to compose the four new family targets. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>3d53530 -
Fix cache-dir default to use existing downloads volume
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>43aa427 -
Make factored W_QK SVD the default; clean up experiment
Factor matrix SVD for W_QK is now unconditional — it replaces the full d_model×d_model SVD regardless of low_rank_svd_approximation. Experiment script no longer passes top_k_svd_d_head so the comparison is a pure A-B of the two SVD methods with no truncation confound. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>989c8e0 -
Match nominal revision when running truncated experiment
Read the revision from the nominal dataset's rows and pass it to process_model so both runs use identical model weights. Also update the truncated output path to reflect the actual revision label. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>abc312e -
Compute W_QK SVD via d_head×d_head factor matrices
When low_rank_svd_approximation is enabled, SVs of W_QK = W_Q^T W_K are computed from the thin SVDs of W_Q and W_K as: SVs(W_QK) = SVs(diag(S_Q) @ U_Q^T @ U_K @ diag(S_K)) a d_head×d_head matrix. This is exact (not approximate), eliminates the flat-spectrum failure mode from svd_lowrank, and shares the SVD computation already needed for alignment. The oversampling/niter path in fill_matrix is still used for W_Q and W_K individually. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>4bc6a22 -
Scale svd_lowrank oversampling with d_head and increase niter
Fixed oversampling of 10 fails for heads with flat singular value spectra (all d_head directions of similar magnitude). Use d_head//2 oversampling so the budget scales with the rank, and niter=4 for better convergence on near-degenerate spectra. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>90692ef -
Add oversampling to svd_lowrank to fix occasional accuracy failures
Use q = min(top_k_svd + 10, min(W.shape)) then truncate to top_k_svd, giving the randomized algorithm headroom to accurately resolve the top singular values at the rank boundary. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>e7412ba -
Update dashboards and correlations app for experiment-namespaced output layout
dashboard_utils: use DATA_PATH (was OUTPUT_DIR), add detect_experiments() and get_experiment_dir(); load_dataset_with_metadata and load_eval_metrics now resolve paths through the selected experiment directory. Cache key includes experiment_dir so switching experiments correctly invalidates cached data. streamlit_app: render experiment selectbox before page navigation; stores selection in session_state so utility functions pick it up without API changes in page files. correlations_app: fix sel_models → sel_runs reference in filter_key. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>3fe2586 -
Refactor experiment_svd_truncation to support multiple models
Accept --models and --nominal-dirs as parallel lists; run each model in sequence and write combined results.json and per_head_comparison.csv. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>2192ae0 -
Add d_head metadata, top-k-svd-d-head option, and SVD truncation experiment
- Log d_head explicitly in metadata.json (correct for GQA models where d_head != d_model/n_heads); strip redundant head_dim alias from serialized output - Add top_k_svd_d_head param to process_model: enables low-rank SVD with N=d_head without needing to know d_head at call time - Expose --low-rank-svd, --top-k-svd, --top-k-svd-d-head in run_primary.py; log all three to MLflow - _get_d_head in cross_model.py reads d_head directly from metadata when available - Add scripts/experiment_svd_truncation.py: runs d_head-truncated pipeline, compares compute time and per-head singular value accuracy against nominal results on disk Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>e9f64da -
Namespace all outputs under experiment_dir, add MLflow run metadata
All pipeline outputs and sentinels now live under {output_dir}/{mlflow_experiment}/ instead of scattered across the project root and output_dir. Prevents clobbering across experiments and mirrors the MLflow hierarchy. Changes: - Snakefile: derive config["experiment_dir"], pre-create it, override eval.out dynamically; update all sentinel expand() paths - rules/*: sentinel touch paths and inter-rule deps use experiment_dir; correlations/figures dirs moved under experiment_dir - config.yaml: remove hardcoded eval.out (now derived in Snakefile) - scripts/run_*.py: add mlflow.note.content description, model tag, and DatasetInput (local source → output path) to every run Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>0bf8fb1
-
Add OLMo 2, Llama 3.2, Gemma 2, SmolLM2 to model registry
Registers 11 new models across four families, all targeting the final checkpoint only (revisions=[]). OLMo 2 (4 sizes, allenai/, not gated): reuses extract_llama_qkv — architecture confirmed identical to LLaMA layout (separate q/k/v proj under model.layers.{i}.self_attn.*). Llama 3.2 1B and 3B (meta-llama/, gated): reuses extract_llama_qkv. Gemma 2 2B and 9B (google/, gated): adds extract_gemma2_qkv which reads head_dim explicitly from config.json (256 for both sizes) rather than deriving it from hidden_size // n_heads — these differ for Gemma 2. Reuses extract_llama_o and extract_llama_biases (same key prefix, bias-free). SmolLM2 135M/360M/1.7B (HuggingFaceTB/, not gated, instruct variants): reuses extract_llama_qkv — registered as LlamaForCausalLM in transformers. Also adds _read_head_dim() helper and fixes d_head computation in head_pipeline, pair_pipeline, and fuse_timesteps to use hf_config.head_dim when present (Gemma 2) rather than always computing d_model // n_heads. Adds…3c24019 -
Fix W_QK computation and dashboard data loading
head_analyzer.py: restore original W_Q^T @ W_K formula giving (d_model, d_model) per head — the true bilinear form in token space. Commit 8ff6453 had changed this to W_Q @ W_K^T → (head_dim, head_dim) for memory savings, but that changed the mathematical object entirely and corrupted statistics for all models. run_merge.py: fix incremental merge to read existing metadata before rmtree/move so Pythia entries aren't dropped when adding new models; also fix atomic save via temp-path to avoid HF datasets self-overwrite error. dashboards: remove campaign concept throughout; all pages now load from OUTPUT_DIR via load_dataset_with_metadata(ds_name=...) with no campaign param. get_available_campaigns() removed. cross_model, sandbox, animations, step_evolution, weights_dashboard, singular_values all updated. config.yaml: drop 'all' from targets (target_all already composes families). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>9c80428 -
Add smk_list.py: colorized snakemake --list wrapper
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>b9ecb4f -
Add docstrings to all static rules for snakemake --list
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>fbffedd -
Add MLflow logging to correlations and eval stages
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>dbd0f5c -
Add clean rules: clean, clean_transform, clean_mlflow, clean_run, clean_<family>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>92dcf5b -
Wire QK decomposition weight types into primary pipeline
W_Q_gram, W_K_gram, QK_alignment are already computed in head_analyzer.py. This commit exposes weight_types through config.yaml → run_primary.py → process_model() → primary.smk so they can be overridden at runtime. Tests updated to cover all 6 weight types; histogram test restricted to raw weight types since gram entries exceed standard bin range. compute_qk_decomposition.py removed (data now lives in primary dataset). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>a07e79a -
Remove append_to_parquet from eval_metrics
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>5fbc916 -
Refactor eval to per-run parquets + merge_eval, remove -j1 constraint
Each eval job now writes outputs/eval/{run_key}.parquet independently. merge_eval concatenates them into the final artifact. eval_all depends on merge_eval output rather than individual sentinels. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>f49060a -
Make target_all compose from family rules instead of re-listing runs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>5717451 -
Trim llama/mistral to llama-3.1-8b and mistral-7b-v0.3
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>38cd177 -
Add named family targets and pythia checkpoint sweep targets
config.yaml: - runs: expanded to full gpt2/pythia/llama+mistral model sets - targets: gpt2_family, pythia_family, llama_family, all - pythia_steps: disabled by default; enable + add runs to get target_pythia_{shortname}_steps covering all PYTHIA_REVISIONS Snakefile: - Loop over config["targets"] → target_{name} rules - Loop over config["pythia_steps"] → target_pythia_{shortname}_steps rules (reads PYTHIA_REVISIONS from model_registry) Usage: snakemake target_gpt2_family -j4 snakemake target_pythia_family -j4 snakemake target_llama_family -j4 snakemake target_all -j8 snakemake target_pythia_70m_steps -j4 # after enabling in pythia_steps Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>98c58da -
Add drop_columns option to transform stage for refined dataset reduction
- reprocess_metrics(): drop_columns param strips named columns before saving the refined dataset; primary is never touched so columns are always recoverable - run_transform.py: --drop-columns passthrough, logged to MLflow - transform.smk: reads transform.drop_columns from config, builds --drop-columns flag - config.yaml: transform.drop_columns: [] with note on common candidates (P_w, w_bins) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>2f68beb