pjm-models / results /README.md
Lowca's picture
Add files using upload-large-folder tool
1ae786e verified
|
Raw History Blame Contribute Delete
3.12 kB

Test outputs and logs

Model outputs, references, metric logs and training logs behind the CrocoSign paper. No weights; those are in checkpoints/ and as ONNX in unisign/ and spamo/.

Uni-Sign (unisign/eval_<run>/)

test_tmp_pres.txt and test_tmp_refs.txt hold one line per test clip (sample: <key>, prediction: ... and sample: <key>, ground-truth: ...), log.txt the evaluation log with BLEU, ROUGE-L and chrF, bleurt_BLEURT-20.json the per-clip BLEURT-20 scores, train.log.gz the training log. Table 3 rows:

Row Directory
CSL-News MS / SI eval_pjm_phase3_full_ms, eval_pjm_phase3_full_si
How2Sign MS / SI eval_pjm_phase3_full_ms_how2sign, eval_pjm_phase3_full_si_how2sign
OpenASL MS / SI eval_pjm_phase3_full_ms_openasl, eval_pjm_phase3_full_si_openasl
scratch MS / SI eval_pjm_phase3_full_ms_scratch, eval_pjm_phase3_full_si_scratch

The other directories are the second seeds (_s43), the chained and distilled experiments, and the SI regularisation experiments (_aug, _drop03*). greedy_ens.log is the greedy ensemble selection.

SpaMo and Mod-SpaMo (spamo/<run>/)

test_outputs.txt holds reference and generated text per clip (lowercased, in test-set order without keys), plus the wandb summary, config and console log of the test run where they exist. Table 3 rows:

Row Directory
SpaMo PHOENIX-2014T MS 2026-05-05T23-00-55_test_upstream_pjm_ms_dual
SpaMo PHOENIX-2014T SI 2026-10-31T07-09-05_test_upstream_pjm_si_dual
SpaMo scratch MS / SI 2026-05-05T21-58-47_test_v30_dual_ms_scratch, 2026-05-06T15-27-35_test_v32_dual_si_scratch
Mod-SpaMo PHOENIX-2014T MS 2026-05-04T09-09-26_test_v27_quad_xattn
Mod-SpaMo PHOENIX-2014T SI 2026-10-31T15-20-57_test_v27_quad_si_filtered
Mod-SpaMo scratch MS / SI 2026-05-05T22-10-23_test_v31_quad_ms_scratch, 2026-05-07T04-18-55_test_v33_quad_si_scratch

run_phoenix_si.log.gz is the log of the SpaMo and Mod-SpaMo PHOENIX-2014T SI runs, finetune_pjm_ms.log.gz the training log of the SpaMo PHOENIX-2014T MS run. BLEURT for the older SpaMo rows was computed separately and is not stored here.

Polish targets (polish/)

pjm_pl_{ms,si}_how2sign: Uni-Sign How2Sign trained on Polish targets (pl_fixed of CrocoSign/pjm-data/texts/texts_fixed.csv), weights in checkpoints/unisign/pjm_pl_{ms,si}_how2sign/. en2pl_{ms,si}_how2sign: the English How2Sign outputs translated to Polish by a language model (Claude Sonnet). Both are scored against pl_fixed; see the "Polish targets" paragraph of the paper.

Training-size control (unisign/eval_pjm_ms_sub_openasl/)

Uni-Sign OpenASL trained on the MS training set cut to the SI size (10,062 pairs, every MS signer kept with about 42% of their clips, split_train_ms_sub.csv here) with the SI schedule, and tested on the full MS test set: 7.19 BLEU-4 and 39.75 BLEURT, against 8.53 and 42.43 with the full MS training set. Weights in checkpoints/unisign/pjm_ms_sub_openasl/; scripts make_ms_subset.py and queue_ms_subset.sh in Uni-Sign-PJM.