Commit History
-
feat: ensemble + calibration + family/top-3 to lift classifier accuracy
Paul Clark +1: Claude Fable 5Five accuracy improvements: 1. Ensemble (combined_prediction): a verified-decode pass actually BREAKS the brute-forceable ciphers (Caesar/ROT13/Affine/Atbash) and auto-solves Vigenere, judging 'is it English?' by quadgram score (robust) rather than sparse word lists. A recovered plaintext is reported as certain. The heuristic wins on format-distinct ciphers (Morse/Polybius/tap code); else the more-confident calibrated predictor wins. Spot-check: 7/7 vs a model-only baseline that missed Caesar and Vigenere. 2. Family + top-3: the Detect report leads with the cipher family and lists the top-3 candidates with confidences (build_explanation). 3. Stronger-retrain recipe in DEPLOY.md (roberta-base --focal-loss --epochs 10). 4. Transformer confidence calibration (scripts/calibrate_transformer.py + transformer_calibration_map.json): isotonic map, applied in transformer_ predict. Val ECE 0.038 -> 0.014; measured 61.2% accuracy. 5. Character-native model recipe (googl…7e118c2
-
fix: bound char-level inference length to the model's token limit
Paul Clark +1: Claude Fable 5The transformers pipeline's truncation=True proved unreliable across versions, so cap raw characters before char-spacing (~1 token/char) to guarantee the sequence stays under 512 tokens. Prevents an embedding IndexError on long ciphertexts with a char-level model, in both the app and the evaluator. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>9b8dd68 -
feat: char-level inference parity + free-Colab training notebook
Paul Clark +1: Claude Fable 5Makes the --char-level model actually usable in production and gives a zero-cost path to train it. - char_tokenize_text() moved to core.py (pure string op, shared by trainer, app, and evaluator) - train_transformer.py records char_level in the model config so it serializes to config.json and travels to the Hub - app.py and evaluate_baseline.py detect a char-level model from its config and apply the SAME char spacing at inference. Without this a char-level model gets raw text in production and predicts garbage (train/serve skew). - notebooks/train_on_colab.ipynb: open in Colab, pick the free T4 GPU, paste a free HF write token, run — trains the char-level classifier and pushes it to the Hub with no paid GPU Space. DEPLOY.md documents this as the easy path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>dce1610
-
docs: add real app screenshots + Playwright capture script
Paul Clark +1: Claude Fable 5Captured from the running app with scripts/capture_screenshots.py (Playwright drives the live Gradio UI, fills a Caesar example in Detect, shoots each tab). The Detect hero shot shows the calibrated 76.7% confidence and full evidence report. README demo section now embeds the hero image with a regenerate recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>4ca187a -
Gold-standard hardening: calibration, new solvers, property tests, CI gates
Paul Clark +1: Claude Fable 5Confidence calibration (the honesty flagship): - The heuristic emitted hand-tuned confidences that were badly miscalibrated — a raw 26% meant ~4% real accuracy, a raw 86% meant ~97%. calibrate_confidence() applies an isotonic map (fit on validation by scripts/calibrate_confidence.py) inside heuristic_classify, cutting expected calibration error from 0.18 to 0.04 on the held-out test split. A reported N% now means right ~N% of the time. Two new auto-solvers + a dictionary pass: - beaufort_auto_solve (reciprocal Vigenere, same coordinate-descent refinement) - columnar_auto_solve (recovers column ORDER by hill-climbing the quadgram score; key letters are irrelevant) - Vigenere keyword-dictionary pass for short texts. Benchmarked with keys NOT in the dictionary so the reported statistical rates are honest. Property-based tests (tests/test_properties.py, hypothesis): - decode(encode(x))==x, reciprocity, and generator/core agreement across all reversible ciphers — the class o…77ec8ec -
Overhaul the auto-solvers: full n-gram model, working substitution solver
Paul Clark +1: Claude Fable 5The hill-climbing substitution solver never actually converged (0% success at any length). Root cause was the scoring model, not the search: the old hand-typed ~100-bigram/60-trigram tables were so sparse that their unigram backoff dominated and REWARDED frequency-correct letter salad, making the frequency-seeded start a local optimum of the solver's own metric. - ngram_tables.py (generated by scripts/build_ngram_tables.py): all 676 bigrams, top-3000 trigrams (99.9% coverage), top-5000 quadgrams (99.2%), derived from the corpus' 1.88M letters of English plaintext - english_quadgram_score() with trigram chain-rule backoff; trigram scorer now backs off via the bigram chain rule instead of unigram sums - hill_climb_substitution: Jakobsen-style greedy 325-swap sweeps from the frequency seed + random restarts (default 8), scoring 0.2*bigram + 0.3*trigram + 0.5*quadgram. Verified the true key is the global optimum of the new metric on test texts. - vigenere_auto_solve: coordinat…00a77fa -
fix: resolve all 10 ruff lint errors so CI passes again
Paul Clark +1: Claude Fable 5CI had been red since May on pre-existing lint errors (plus one new B905 in the Porta test). Fixes: ambiguous `l` variable names (E741), unsorted imports in training_space/app.py (I001), f-strings without placeholders (F541), zip() without strict= (B905), and a duplicate IT key in the bigram log-prob table (F601) that silently shadowed the -4.93 slot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>34cc0bd -
Improve accuracy and correctness: family layer, heuristic fixes, real bug fixes
Paul Clark +1: Claude Fable 5Bugs fixed: - columnar_transposition_decrypt read ciphertext segments in the wrong order and miscomputed irregular column lengths (Try Decode returned garbage) - Porta generator used a malformed non-permutation table; replaced with the authentic self-reciprocal formula (matches museum-corpus rows exactly) - Windows cp1252 console crash in three scripts printing non-ASCII output Heuristic (validated on fixed-seed A/B samples, full test split 27.1% -> 29.2%): - null_cipher requires spaces; run-together near-English is lorenz - removed kama_sutra rule (statistically identical to monoalphabetic; wrong 5x more often than right) and the stager_route IoC gate (right 21x, wrong ~600x) - exotic machine-class verdicts now require >= 60 letters - wheatstone requires run-together text New cipher-family layer: - CIPHER_FAMILIES / label_family() group all labels into 7 statistical families; evaluate_baseline.py reports family-level metrics as the honest headline (57.2% accuracy / 0.656 m…7733667
-
feat: train transformer on HF A10G + heuristic improvements
Paul ClarkTransformer training: - scripts/balance_dataset.py: cap majority classes at 1000/class, stratified train/val/test split (58k/6.5k/30k, 81 labels) - data/splits/: balanced train.jsonl + val.jsonl + test.jsonl - training_space/: Gradio Space that trains roberta-base on A10G GPU, pushes model to systemslibrarian/cipher-detective-classifier - Dataset uploaded to systemslibrarian/classical-cipher-corpus on HF Hub - Training Space live at: huggingface.co/spaces/systemslibrarian/cipher-detective-training Heuristic accuracy improvements (measured on 50 samples/class): - gronsfeld: 0% → 16% (moved check to correct IoC range 0.040–0.046) - columnar_transposition: 24% → 38% (word-score rail-fence disambiguation) - rail_fence: 16% → 26% (word-score disambiguator vs columnar) - double_transposition: 2% → 12% (length threshold 180→150) - beaufort: 2% → 6% (beaufort path when Kasiski support is 0) - too_short threshold lowered 20→12 letters (helps short polygraphic texts) - autokey: added runnin…4f5aa4e
-
fix: pin gradio>=4.44.1, modernize typing, clean ruff lint
Paul Clark +1: Claude Opus 4.7 (1M context)- Pin gradio>=4.44.1 in requirements.txt to fix HF Space startup (Gradio 4.44.0 + Starlette >=0.40 raises 'unhashable type: dict' in jinja2 template cache on every request). - Modernize typing across app.py, core.py, scripts/, tests/ to PEP 585 builtins (list/dict/tuple) — clears UP006/UP035. - Sort imports, drop unused word_score import, add zip(strict=) calls, rename ambiguous loop var l -> lbl/ln to clear E741. CI was failing on lint with 76 ruff errors; now clean (ruff check .) and 39/39 tests pass locally. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>2f76c8f -
fix(heuristic): range-based numeric cipher discrimination + Voynich detection
Paul ClarkTier 1b (numeric) improvements: - culper_ring: 3-digit tokens 70%+ in 800-999 range - vic: unspaced digits len>=35 -> vic; shorter -> straddling_checkerboard - nomenclator: bimodal distribution (50%+ in 1-26 PLUS 1+ in 50-99) - aeneas_tacticus: all tokens 1-26 only - wallis_cipher: 90 or 91 appears 2+ times as structural group markers - nihilist: all values >=22 (Polybius sums), strict below-22 exclusion - homophonic: fallback for broad 00-99 distribution Tier 1a addition: - voynich_render: all-lowercase alpha-space text with word_score < 2 Spot-check accuracy on 30 examples per label: culper_ring 28/30, aeneas_tacticus 30/30, nihilist 30/30, wallis_cipher 30/30, nomenclator 27/30, voynich_render 30/3012b96e8 -
feat(data): add 20 sparse-label synthetic generators; dataset now 90 222 rows
Paul ClarkAdded cipher encoders for 20 museum-only classes that previously had <500 training examples: aeneas_tacticus, arnold_andre, babington, bacon_cipher, book_cipher, commercial_code, culper_ring, homophonic, morse_code, navajo_code, null_cipher, one_time_pad, pigpen, polybius, running_key, tap_code, vernam, voynich_render, wallis_cipher, zimmermann Key implementation details: - record() now detects non-alphabetic ciphertext (alpha_ratio < 0.5) and preserves raw format instead of calling with_spacing() which would strip dots, numbers, and symbols — fixing babington ⟨wNN⟩, tap code dots, etc. - _SYNTH_LABELS expanded from 20 → 40 labels; build_row() handles all 40. - Generated 500 examples per new label + 300 extra atbash = 10 300 new rows. - All 81 labels now have ≥ 513 examples (min: m94=513, max: monoalpha=3658) - ATTACKS and EDU_NOTES registries updated for all new labels. - tests/test_core.py ALLOWED_LABELS updated to include all 40 synthetic labels. - 39 tests pass.78394ba -
feat(training): add early stopping, cosine LR schedule, and push-to-hub
Paul Clark- EarlyStoppingCallback with --early-stopping-patience (default 3 epochs) prevents overfitting on the tiny museum-only classes (book_cipher: 48, homophonic: 76, etc.) during the 10-epoch focal-loss run. - lr_scheduler_type=cosine: cosine decay outperforms linear for transformer fine-tuning with warmup, especially over 10 epochs. - --push-to-hub / --hub-model-id flags: after training, push the saved model directly to the HF Hub so the Space picks it up via CIPHER_MODEL_ID. - Makefile: added train-push target (train-best + hub push); updated PHONY and help text. Usage: make train-push HUB_MODEL_ID=yourname/cipher-model3ca03a7 -
fix: remove duplicate-tail dead code; harmonise app label vocabulary
Paul Clark +1: Claude Opus 4.7 (1M context)scripts/train_transformer.py had ~160 lines of botched-merge duplicate code after the new main(): a second def main() at line 296 shadowed the new one, and the second `if __name__ == "__main__": main()` invocation ran the OLD trainer right after the new one — silently re-training with different hyperparameters and overwriting the saved model. Removed the duplicate block; --focal-loss, --lr, --warmup-ratio, --label-smoothing, --grad-accum, fp16 now actually take effect. scripts/generate_dataset.py had a similar duplicate tail (~150 lines). No functional bug since the duplicates didn't redefine main(), but it caused 2× generation work and confused the file. Removed it. Also dropped the unused `string` import. app.py Challenge Mode used legacy labels (caesar_rot, columnar, substitution) that don't exist in the trained dataset, causing the generated answer label to disagree with what Detect Mode reported on the same ciphertext. Switched the dropdown and dispatch to the canonical names (c…c231f50 -
feat: expand to 81 ciphers — improved heuristic, 12 new cipher generators, focal-loss training
Paul Clark- core.py: multi-tier heuristic for all 81 ciphers; Tier 1 achieves 100% F1 on 12+ distinctive ciphers (tap_code, morse, ADFGVX, bacon, polybius, etc.); plaintext detection now fires before affine brute-force to avoid misclassification - scripts/generate_dataset.py: 12 new cipher implementations (beaufort, gronsfeld, autokey, trithemius, porta, scytale, double_transposition, rot13, monoalphabetic, stager_route); --append, --per-label, --labels CLI flags added - data/cipher_examples.jsonl: 10 000 new synthetic examples appended (79 922 total) - scripts/train_transformer.py: focal loss trainer, warmup_ratio, label_smoothing, grad_accum, fp16, class-weighted loss on by default, 10 epoch default - Makefile: train-best target with roberta-base + focal loss + optimal hyperparams - tests/test_core.py: ALLOWED_LABELS expanded to 20 synthetic labels; caesar/caesar_rot both accepted76616a8 -
Improve training: weighted loss, blind eval split, roberta-base defaults, 512 tokens
Paul Clark1a9a201 -
Fix singleton class crash: normalise wallis_ciphers typo, guard train_test_split
Paul Clarkb108e4c -
Replace synthetic 8-label dataset with cipher-museum 100k/82-label corpus
Paul Clark593878d -
Fix Trainer API: tokenizer -> processing_class (transformers 5.x)
Paul Clark39faac4
-
Make Space HF-launch-ready: fix boot crash, add solver, expand eval, polish UX
Paul Clark +1: Claude Opus 4.7 (1M context)The Space had a hard boot crash (unsupported aria_label= kwargs on Gradio 4.44 Textbox/Dropdown/Button) plus three pre-existing test failures and several open roadmap items. This bundles the launch-readiness work into one commit: - app.py: remove all 10 aria_label= kwargs (boot-blocker on HF Spaces); add a Solve Substitution tab driving a hill-climbing solver; add a Random Example button to Detect Mode; replace ad-hoc accessibility hints with Gradio info=. - core.py: hill_climb_substitution() with frequency-rank seed, English bigram log-prob scoring with unigram backoff, and random restarts. Tighten the heuristic so plaintext doesn't get misclassified as caesar_rot when the best Caesar shift is 0. - scripts/evaluate_baseline.py: per-difficulty and per-length-bucket accuracy + macro-F1 breakdowns alongside the overall metrics, for both heuristic and Transformer paths. - requirements.txt: pin torch>=2.2.0 so transformers.pipeline actually loads on a fresh Space. - tests/…fede36d -
ADA/mobile UI, pytest pythonpath, train_transformer PyArrow fix, enriched Explain Mode
Paul Clark5642612 -
10/10 polish: Compare Mode, Kasiski/Friedman, rich dataset schema, expanded tests/docs
Paul Clarkd634e85 -
Initial Cipher Detective AI public foundation
Paul Clarkf90a967