Added cipher encoders for 20 museum-only classes that previously had <500
training examples:
aeneas_tacticus, arnold_andre, babington, bacon_cipher, book_cipher,
commercial_code, culper_ring, homophonic, morse_code, navajo_code,
null_cipher, one_time_pad, pigpen, polybius, running_key, tap_code,
vernam, voynich_render, wallis_cipher, zimmermann
Key implementation details:
- record() now detects non-alphabetic ciphertext (alpha_ratio < 0.5) and
preserves raw format instead of calling with_spacing() which would strip
dots, numbers, and symbols — fixing babington ⟨wNN⟩, tap code dots, etc.
- _SYNTH_LABELS expanded from 20 → 40 labels; build_row() handles all 40.
- Generated 500 examples per new label + 300 extra atbash = 10 300 new rows.
- All 81 labels now have ≥ 513 examples (min: m94=513, max: monoalpha=3658)
- ATTACKS and EDU_NOTES registries updated for all new labels.
- tests/test_core.py ALLOWED_LABELS updated to include all 40 synthetic labels.
- 39 tests pass.
feat: expand to 81 ciphers — improved heuristic, 12 new cipher generators, focal-loss training
Paul Clark
76616a8
- core.py: multi-tier heuristic for all 81 ciphers; Tier 1 achieves 100% F1
on 12+ distinctive ciphers (tap_code, morse, ADFGVX, bacon, polybius, etc.);
plaintext detection now fires before affine brute-force to avoid misclassification
- scripts/generate_dataset.py: 12 new cipher implementations (beaufort, gronsfeld,
autokey, trithemius, porta, scytale, double_transposition, rot13, monoalphabetic,
stager_route); --append, --per-label, --labels CLI flags added
- data/cipher_examples.jsonl: 10 000 new synthetic examples appended (79 922 total)
- scripts/train_transformer.py: focal loss trainer, warmup_ratio, label_smoothing,
grad_accum, fp16, class-weighted loss on by default, 10 epoch default
- Makefile: train-best target with roberta-base + focal loss + optimal hyperparams
- tests/test_core.py: ALLOWED_LABELS expanded to 20 synthetic labels; caesar/caesar_rot
both accepted
chore: update cipher_examples dataset
Paul Clark
fe49700
Fix singleton class crash: normalise wallis_ciphers typo, guard train_test_split
Paul Clark
b108e4c
Replace synthetic 8-label dataset with cipher-museum 100k/82-label corpus
