Spaces:
Sleeping
Sleeping
Commit History
-
Eval: optional layer offload via accelerate for >RAM models
Adds an opt-in --offload-layers path that loads the model with accelerate.init_empty_weights + load_checkpoint_and_dispatch, so layers stream from disk through a bounded memory budget instead of all sitting resident. Unlocks models that don't fit in unified memory in fp16 (e.g. pythia-2.8b on a 16 GB M4) at the cost of much slower per-window inference. Numerics are unchanged — perplexity matches a non-offload reference within fp16 noise (verified on pythia-70m, delta ppl < 0.1%). CLI: --offload-layers turn it on --offload-folder PATH spill directory; defaults to tempfile.mkdtemp() on the system disk (best per-window bandwidth, auto-cleaned). Override to put it on an external drive when the system disk is tight on space. --max-memory-gpu STR e.g. "4GiB"; budget for the accelerator --max-memory-cpu STR e.g. "2GiB"; budget for CPU before disk spill evaluate_model owns the t…66a0324 -
Eval: load in the model's native dtype, add --dtype CLI flag
The eval pipeline hardcoded torch_dtype=torch.float32, which made models >1-2B unrunnable on local Apple Silicon (3-7B in fp32 spills into swap and grinds to a halt). Switch to torch_dtype="auto" by default — Llama/Mistral/Gemma2/OLMo2/SmolLM2 load in bf16, Pythia in fp16, GPT-2 in fp32, matching each model's published dtype. NLL is still computed in fp32 inside HF's ForCausalLMLoss (it explicitly upcasts logits.float() before cross-entropy), so the perplexity numbers stay well-defined. low_cpu_mem_usage=True avoids the transient fp32 copy during from_pretrained, ~halving peak load memory. Add a --dtype CLI knob on run_eval.py (auto|bf16|fp16|fp32) threaded through to evaluate_model so a numerics reference run is one flag away. Also log dtype and device as MLflow params for reproducibility. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>937097a
-
Fix mlflow.log_input in all remaining pipeline scripts
Same fix as run_primary.py: mlflow.log_input expects mlflow.data.Dataset, not a raw DatasetInput entity. Use MlflowClient().log_inputs() throughout. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>88dbab4
-
Namespace all outputs under experiment_dir, add MLflow run metadata
All pipeline outputs and sentinels now live under {output_dir}/{mlflow_experiment}/ instead of scattered across the project root and output_dir. Prevents clobbering across experiments and mirrors the MLflow hierarchy. Changes: - Snakefile: derive config["experiment_dir"], pre-create it, override eval.out dynamically; update all sentinel expand() paths - rules/*: sentinel touch paths and inter-rule deps use experiment_dir; correlations/figures dirs moved under experiment_dir - config.yaml: remove hardcoded eval.out (now derived in Snakefile) - scripts/run_*.py: add mlflow.note.content description, model tag, and DatasetInput (local source → output path) to every run Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>0bf8fb1
-
Add MLflow logging to correlations and eval stages
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>dbd0f5c -
Refactor eval to per-run parquets + merge_eval, remove -j1 constraint
Each eval job now writes outputs/eval/{run_key}.parquet independently. merge_eval concatenates them into the final artifact. eval_all depends on merge_eval output rather than individual sentinels. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>f49060a -
Rename src modules for head/pair symmetry; add pair_figures.smk; split eval_metrics
src/transformer_analysis/: weight_analysis → head_pipeline attn_head_analysis → head_analyzer metrics → head_metrics correlation_analysis → pair_pipeline head_correlations → pair_analyzer eval_metrics.py (new — implementation extracted from compute_perplexity.py) scripts/: plot_correlations.py → plot_pair_correlations.py plot_corr_figures.py → plot_pair_correlations_batch.py compute_perplexity.py → deleted (impl in eval_metrics; thin wrapper run_eval.py) run_eval.py (new thin wrapper) rules/: correlations.smk — switched to sentinel output (consistent with other rules) pair_figures.smk (new — regenerate pair figures from .npz without re-running correlations) eval.smk — updated to call run_eval.py Snakefile: added correlations_all, pair_figures_all convenience targets Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>a81fd74