traj-mc / code /docs /JOBS.md
ttishere's picture
Publish code/docs
49d95a5 verified
|
Raw History Blame Contribute Delete
4.58 kB

JOBS 台账

每提交一个 Slurm job 追加一行;job 完成后回填【状态】与【实际耗时】。

主实验 Slurm 依赖链(2026-07-23 提交;全部 sbatch,--dependency=afterok 硬门禁)。 ssh 直跑的前一轮 4 个压缩 job 被上一 session teardown 杀掉(nohup 未存活),改用 sbatch 重跑(Slurm job 存活于 teardown,且提供 job-id 供 afterok 链使用)。

Slurm id 依赖 脚本 作用 输入 输出 状态 耗时
152959 — compress.sbatch BASE all-linear r=0.7 (SVD-LLM t=0) base calib weights/base/all_r07 PENDING —
152960 — compress.sbatch OURS all-linear r=0.7 (Traj-MC) ours calib weights/ours/all_r07 PENDING —
152961 — compress.sbatch BASE attn r=0.8 — parked, no eval base calib weights/base/attn_r08 PENDING —
152962 — compress.sbatch OURS attn r=0.8 — parked, no eval ours calib weights/ours/attn_r08 PENDING —
152963 afterok:152959:152960 diff_gate.sbatch HARD GATE: BASE/OURS all_r07 summary diff; fail → cancels 152966-969 2×summary.json results/compress/ PENDING —
152964 — eval_lmeval.sbatch REF 5×ppl MCQ (arc_c/arc_e/piqa/wino/mmlu) dense results/eval/ref/ PENDING —
152965 — eval_lmeval.sbatch REF GSM8K (accelerate DP×4) dense results/eval/ref/ PENDING —
152966 afterok:152963 eval_lmeval.sbatch BASE 5×ppl MCQ weights/base/all_r07 results/eval/base/ PENDING —
152967 afterok:152963 eval_lmeval.sbatch BASE GSM8K (DP×4) weights/base/all_r07 results/eval/base/ PENDING —
152968 afterok:152963 eval_lmeval.sbatch OURS 5×ppl MCQ weights/ours/all_r07 results/eval/ours/ PENDING —
152969 afterok:152963 eval_lmeval.sbatch OURS GSM8K (DP×4) weights/ours/all_r07 results/eval/ours/ PENDING —

后续:全部 eval 落地 → analysis/mcnemar.py(主指标列) + analysis/holm.py → results/stats/。

GSM8K 并行说明:官方 harness 用 accelerate 数据并行(docs[rank::world] 分发 + 原子 gather)——即 strided 分片的 harness 原生形式,单 job/单 seed/单 git-hash,合并一致性由构造保证,无需 merge_shards 的跨分片一致性门禁。

前置 gate:REF harness 验证(piqa 74.33 / winogrande 74.33 / arc_c 43.0 @ n=300)复现官方 EVAL.md,已闭。


2026-07-24 现状(前一链 152959-969 因压缩 4h TIMEOUT 级联取消,已重来)

Slurm id 依赖 作用 输入 输出 状态
153565 — BASE all-linear r=0.7 压缩 (nsamples=256, 旧慢代码) base calib weights/base/all_r07 RUNNING(SVD) node3
153566 — OURS all-linear r=0.7 压缩 (nsamples=256) ours calib weights/ours/all_r07 RUNNING(SVD) node3
153709 — REF GSM8K (accelerate DP×4) dense results/eval/ref/lmeval/gsm8k PENDING
153710 — REF HellaSwag (accelerate DP×4) dense results/eval/ref/lmeval/hellaswag PENDING

REF 其余 5 项(arc_c/arc_e/piqa/wino/mmlu)已 COMPLETED(152964)并复现官方 EVAL.md(MMLU 65.99≈65.9)。 待 153565/566 完成 → diff_gate → BASE/OURS 全 7 任务评测。 压缩提速修复(GPU 常驻 XtX + 显存分箱)已进 compress.py,供后续/消融加速;153565/566 用的是旧代码但已近完成,不重跑。

主表 7-task 评测批(2026-07-24,压缩 gate PASS 后提交)

gate: summary_diff base vs ours all_r07 = PASS(ratio=0.7 n_compressed=224 kept=0.6998 逐字段一致)。

arm gsm8k(DP×4) hellaswag(DP×4) 5×MCQ(1gpu)
REF 153709 153710 ✅152964(done)
BASE 153738 153739 153740
OURS 153741 153742 153743

全部落地 → mcnemar(主指标, m=7) + holm → results/stats/。

多语种重做(2026-07-25):校准 en → multilingual(16-lang interleave),照 spec "不限语种"

REF 不用校准 → 结果复用,不重跑。C4-en base/ours 已归档 _c4en_archive/。 多语种 calib 已建(isolation OK),recompress base/ours all_r07 完成,diff-gate PASS(kept=0.6998)。

arm mcq5(1gpu) gsm8k(1gpu) hella(1gpu)
BASE 154648 154649 154650
OURS 154651 154652 154653
REF ✅done(152964) 154315(run) 154317(run)

single-process(避 DP 24GB prep OOM);broad partition;OOM 则改 pin genai。全落地 → McNemar+Holm(m=7)。