pjm-models / results /README.md
Lowca's picture
Add files using upload-large-folder tool
1ae786e verified
|
Raw History Blame Contribute Delete
3.12 kB
# Test outputs and logs
Model outputs, references, metric logs and training logs behind the CrocoSign paper. No weights; those are in
`checkpoints/` and as ONNX in `unisign/` and `spamo/`.
## Uni-Sign (`unisign/eval_<run>/`)
`test_tmp_pres.txt` and `test_tmp_refs.txt` hold one line per test clip (`sample: <key>, prediction: ...` and
`sample: <key>, ground-truth: ...`), `log.txt` the evaluation log with BLEU, ROUGE-L and chrF, `bleurt_BLEURT-20.json`
the per-clip BLEURT-20 scores, `train.log.gz` the training log. Table 3 rows:
| Row | Directory |
|---|---|
| CSL-News MS / SI | `eval_pjm_phase3_full_ms`, `eval_pjm_phase3_full_si` |
| How2Sign MS / SI | `eval_pjm_phase3_full_ms_how2sign`, `eval_pjm_phase3_full_si_how2sign` |
| OpenASL MS / SI | `eval_pjm_phase3_full_ms_openasl`, `eval_pjm_phase3_full_si_openasl` |
| scratch MS / SI | `eval_pjm_phase3_full_ms_scratch`, `eval_pjm_phase3_full_si_scratch` |
The other directories are the second seeds (`_s43`), the chained and distilled experiments, and the SI
regularisation experiments (`_aug`, `_drop03*`). `greedy_ens.log` is the greedy ensemble selection.
## SpaMo and Mod-SpaMo (`spamo/<run>/`)
`test_outputs.txt` holds reference and generated text per clip (lowercased, in test-set order without keys), plus
the wandb summary, config and console log of the test run where they exist. Table 3 rows:
| Row | Directory |
|---|---|
| SpaMo PHOENIX-2014T MS | `2026-05-05T23-00-55_test_upstream_pjm_ms_dual` |
| SpaMo PHOENIX-2014T SI | `2026-10-31T07-09-05_test_upstream_pjm_si_dual` |
| SpaMo scratch MS / SI | `2026-05-05T21-58-47_test_v30_dual_ms_scratch`, `2026-05-06T15-27-35_test_v32_dual_si_scratch` |
| Mod-SpaMo PHOENIX-2014T MS | `2026-05-04T09-09-26_test_v27_quad_xattn` |
| Mod-SpaMo PHOENIX-2014T SI | `2026-10-31T15-20-57_test_v27_quad_si_filtered` |
| Mod-SpaMo scratch MS / SI | `2026-05-05T22-10-23_test_v31_quad_ms_scratch`, `2026-05-07T04-18-55_test_v33_quad_si_scratch` |
`run_phoenix_si.log.gz` is the log of the SpaMo and Mod-SpaMo PHOENIX-2014T SI runs, `finetune_pjm_ms.log.gz`
the training log of the SpaMo PHOENIX-2014T MS run. BLEURT for the older SpaMo rows was computed separately and
is not stored here.
## Polish targets (`polish/`)
`pjm_pl_{ms,si}_how2sign`: Uni-Sign How2Sign trained on Polish targets (`pl_fixed` of
`CrocoSign/pjm-data/texts/texts_fixed.csv`), weights in `checkpoints/unisign/pjm_pl_{ms,si}_how2sign/`.
`en2pl_{ms,si}_how2sign`: the English How2Sign outputs translated to Polish by a language model (Claude Sonnet).
Both are scored against `pl_fixed`; see the "Polish targets" paragraph of the paper.
## Training-size control (`unisign/eval_pjm_ms_sub_openasl/`)
Uni-Sign OpenASL trained on the MS training set cut to the SI size (10,062 pairs, every MS signer kept with about
42% of their clips, `split_train_ms_sub.csv` here) with the SI schedule, and tested on the full MS test set: 7.19
BLEU-4 and 39.75 BLEURT, against 8.53 and 42.43 with the full MS training set. Weights in
`checkpoints/unisign/pjm_ms_sub_openasl/`; scripts `make_ms_subset.py` and `queue_ms_subset.sh` in Uni-Sign-PJM.