Claude Fable 6.9
A tiny 69-parameter neural network (fully connected, two tanh hidden layers) trained to approximate the function sin(x) + 0.5Β·cos(2x).
Architecture
A multilayer perceptron 1 β 6 β 7 β 1: one number in, one number out, two hidden layers with the tanh activation.
x βββΊ Linear(1β6) βββΊ tanh βββΊ Linear(6β7) βββΊ tanh βββΊ Linear(7β1) βββΊ Ε·
| Layer | Weights | Biases | Parameters |
|---|---|---|---|
| Linear(1 β 6) | 6 Γ 1 | 6 | 12 |
| Linear(6 β 7) | 7 Γ 6 | 7 | 49 |
| Linear(7 β 1) | 1 Γ 7 | 1 | 8 |
| Total | 69 |
| Property | Value |
|---|---|
| input / output size | 1 / 1 |
| hidden layers | 2 (sizes 6 and 7) |
| activation function | tanh on hidden layers, linear output |
| weight dtype | float32 |
| initialization | weights ~ N(0, 1/fan_in), biases = 0 |
Training
Data. A synthetic dataset:
xis uniformly distributed on [βΟ, Ο];y = sin(x) + 0.5Β·cos(2x) + Ξ΅, whereΞ΅is Gaussian noise with a standard deviation of 0.05;- 100 points for training and 50 points for validation, the random number generator is seeded (
numpy.random.default_rng(0)).
Loss function. Mean squared error (MSE): L = mean((Ε· β y)Β²).
Gradients are computed by hand with backpropagation (NumPy implementation, see train.py).
Optimization. Full-batch Adam (all 100 training points at every step).
| Hyperparameter | Value |
|---|---|
| optimizer | Adam (Ξ²β = 0.9, Ξ²β = 0.999, Ξ΅ = 1e-8) |
| learning rate | 0.02 (constant) |
| iterations | 5000 |
| batch size | 100 (full batch) |
Training result
| Iteration | train MSE | validation MSE |
|---|---|---|
| 1 | 1.39811 | 1.29456 |
| 500 | 0.00737 | 0.00772 |
| 1000 | 0.00714 | 0.00771 |
| 2000 | 0.00687 | 0.00729 |
| 3000 | 0.00674 | 0.00700 |
| 4000 | 0.00656 | 0.00688 |
| 5000 (final) | 0.00629 | 0.00682 |
The noise variance in the data (ΟΒ²) is 0.0025, so the final error is about 2.5 times higher. The model has not reached the noise level: the loss is still decreasing slowly at iteration 5000. With only 69 parameters the network most likely lacks the capacity to follow the function exactly, and it may also need more iterations; neither explanation was tested. Train and validation MSE are close, so there is no overfitting.