| ---
|
| tags:
|
| - reproducibility
|
| ---
|
|
|
| # eureka-digits-sol
|
|
|
| Experimental checkpoints. Input definitions, preprocessing and usage limitations are specified by the registered protocol and reproduction guide.
|
|
|
| | Experiment | Method | Seed | Split | Accuracy | Seconds | Device |
|
| |---|---|---:|---|---:|---:|---|
|
| | class_prior_s0 | class_prior | 0 | test | 0.102778 | 0.027 | cpu |
|
| | logistic_s0 | logistic | 0 | test | 0.955556 | 0.109 | cpu |
|
| | pilot_regularized_s0 | regularized | 0 | validation | 0.966574 | 6.083 | cuda |
|
| | regularized_s0 | regularized | 0 | test | 0.963889 | 0.373 | cuda |
|
| | regularized_s1 | regularized | 1 | test | 0.969444 | 5.740 | cuda |
|
| | regularized_s2 | regularized | 2 | test | 0.969444 | 2.151 | cuda |
|
| | sensitivity_s0 | sensitivity | 0 | test | 0.963889 | 2.098 | cuda |
|
| | sensitivity_s1 | sensitivity | 1 | test | 0.969444 | 2.105 | cuda |
|
| | sensitivity_s2 | sensitivity | 2 | test | 0.969444 | 2.121 | cuda |
|
| | shuffled_labels_s0 | shuffled_labels | 0 | test | 0.061111 | 2.190 | cuda |
|
| | shuffled_labels_s1 | shuffled_labels | 1 | test | 0.116667 | 2.194 | cuda |
|
| | shuffled_labels_s2 | shuffled_labels | 2 | test | 0.108333 | 2.103 | cuda |
|
| | unregularized_s0 | unregularized | 0 | test | 0.963889 | 2.032 | cuda |
|
| | unregularized_s1 | unregularized | 1 | test | 0.969444 | 2.142 | cuda |
|
| | unregularized_s2 | unregularized | 2 | test | 0.969444 | 2.203 | cuda |
|
|
|
| The mean paired accuracy difference is 0.000000, with conditional 95% bootstrap interval [0.000000, 0.000000]. The registered decision is **inconclusive**. The sample standard deviation across seed effects is 0.000000. The interval uses 10000 paired resamples of test examples and conditions on the trained models.
|
|
|
| - **analysis_kind**: "exploratory"
|
| - **prior_results_known**: true
|
| - **objective**: "Replicate a known fixed study to evaluate engineering completion; do not claim new scientific confirmation."
|
| - **question**: "Does AdamW weight decay improve held-out accuracy of a small MLP on the bundled digits data?"
|
| - **architecture**: [64, 128, 10]
|
| - **optimizer**: "AdamW"
|
| - **learning_rate**: 0.001
|
| - **batch_size**: 64
|
| - **epochs**: 50
|
| - **methods**: ["regularized", "unregularized", "sensitivity", "shuffled_labels", "logistic", "class_prior"]
|
| - **seeds**: [0, 1, 2]
|
| - **data_boundaries**: "Frozen stratified 60/20/20 train/validation/test partition. Split seed 42. Divide pixels by the documented constant 16. Pilot reports validation only. No learned preprocessing or selection uses test data. Generalization is limited to this random example split, not new writers or datasets. Prior results on this evaluation set are known. All replication observations and conclusions are exploratory; no untouched-data claim is made."
|
| - **analysis_plan**: "Use 10000 paired bootstrap resamples of test example indices with RNG seed 2026, preserving pairing across methods and all trained seeds. This interval conditions on the trained models and does not estimate training-population uncertainty. Separately show each seed effect and its sample standard deviation. Treat other comparisons as descriptive, with no post-hoc primary selection."
|
| - **hyperparameter_rationale**: "Small prespecified network and fixed training budget provide fast feedback on an established task. The three decay values are a descriptive local sensitivity check, not a test-selected sweep. Three seeds expose limited initialization sensitivity; they do not support broad population claims."
|
|
|
| - [src-matplotlib-plot: Matplotlib Axes.plot API](https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.plot.html)
|
| - [src-python-unittest: Python 3.14 unittest: discovery, assertions and isolated fixtures](https://docs.python.org/3.14/library/unittest.html)
|
| - [src-python-subprocess: Python 3.14 subprocess: argv sequences, pipes and process status](https://docs.python.org/3.14/library/subprocess.html)
|
| - [src-adamw-paper: Decoupled Weight Decay Regularization — Loshchilov and Hutter, ICLR 2019](https://arxiv.org/abs/1711.05101)
|
| - [src-adamw-code: AdamW-and-SGDW original implementation and license](https://github.com/loshchil/AdamW-and-SGDW)
|
| - [src-torch-adamw: PyTorch AdamW API and algorithm, version 2.14](https://docs.pytorch.org/docs/2.14/generated/torch.optim.AdamW.html)
|
| - [src-digits-loader: scikit-learn load_digits API](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html)
|
| - [src-uci-digits: Optical Recognition of Handwritten Digits — UCI, Alpaydin and Kaynak](https://archive.ics.uci.edu/dataset/80/optical+recognition+of+handwritten+digits)
|
| - [src-splits: scikit-learn train_test_split API](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)
|
| - [src-logistic: scikit-learn LogisticRegression API](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)
|
| - [src-bootstrap: SciPy bootstrap API: paired resampling and interval definitions](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html)
|
| - [src-reproducibility: PyTorch reproducibility notes, version 2.14](https://docs.pytorch.org/docs/2.14/notes/randomness.html)
|
| - [src-nanda-process: How I Think About My Research Process: Explore, Understand, Distill — Neel Nanda](https://www.alignmentforum.org/posts/hjMy4ZxS5ogA9cTYK/how-i-think-about-my-research-process-explore-understand)
|
| - [src-lones-pitfalls: How to avoid machine learning pitfalls: a guide for academic researchers — Michael A. Lones](https://arxiv.org/html/2108.02497v5)
|
| - [src-prior-haiku: eureka-digits-haiku prior supplied-fixture execution](https://github.com/BurnyCoder/eureka-digits-haiku)
|
| - [src-prior-luna: eureka-digits-luna prior supplied-fixture execution](https://github.com/BurnyCoder/eureka-digits-luna)
|
| - [src-haiku-disclosure: Haiku historical cutoff and subsequent publication disclosure](https://github.com/BurnyCoder/eureka-digits-haiku/blob/main/reports/post-run-publication.md)
|
| - [src-luna-disclosure: Luna post-publication provenance audit disclosure](https://github.com/BurnyCoder/eureka-digits-luna/blob/main/reports/post-run-publication.md)
|
| - [src-pandoc: Pandoc User Guide: PDF creation, Markdown extensions and resource paths](https://pandoc.org/MANUAL.html)
|
| - [src-node-vm: Node.js vm API](https://nodejs.org/api/vm.html) |
