| --- |
| license: apache-2.0 |
| tags: |
| - independent-component-analysis |
| - adversarial-learning |
| - representation-learning |
| - factorial-codes |
| - gradio |
| --- |
| |
| # Factorial Code Forge |
|
|
| Factorial Code Forge mixes two statistically independent non-Gaussian sources and |
| asks a tiny autoencoder to recover a useful two-dimensional code. In the experimental |
| variant, two adversarial predictors try to infer each latent coordinate from the |
| other while the encoder tries to make those predictions fail. Reconstruction and |
| variance constraints prevent a constant-code shortcut. |
|
|
| The historical anchor is Schmidhuber's 1992 |
| [Learning Factorial Codes by Predictability |
| Minimization](https://doi.org/10.1162/neco.1992.4.6.863). The modern experiment |
| compares predictability minimization with an architecture-matched reconstructive |
| encoder, PCA whitening, and FastICA. It is a new controlled benchmark, not a |
| reproduction of the original paper's implementation or datasets. |
|
|
| ## Verified results |
|
|
| Models trained on 20,000 mixed samples and were measured once on 5,000 new samples: |
|
|
| | Code | Absolute correlation | Symmetric MI estimate | |
| | --- | ---: | ---: | |
| | Autoencoder control | 0.02200 | 0.22345 | |
| | Online neural-adversarial ablation | 0.01886 | 0.22678 | |
| | Validation-selected predictability minimum | 0.01158 | 0.00280 | |
| | PCA whitening | 0.02039 | 0.22275 | |
| | FastICA | 0.00590 | 0.00326 | |
|
|
| The selected 12-parameter linear codec reconstructed the mixtures at |
| `1.08e-14` MSE. Its rotation was selected using polynomial cross-predictors on a |
| held-out portion of the training set, never the test set. The online neural |
| minimax game did not improve nonlinear dependence and is retained as a negative |
| stability result rather than omitted. |
|
|
| ## Reproduce |
|
|
| ```powershell |
| uv run python projects/factorial-code-forge/train.py |
| ``` |
|
|
