GPT-6.3 Astra
A 63,000-parameter neural network (fully connected, three tanh hidden layers) trained to approximate the function xΒ·sin(2x).
Architecture
A multilayer perceptron 1 β 125 β 249 β 125 β 1: one number in, one number out, three hidden layers with the tanh activation.
x βββΊ Linear(1β125) βββΊ tanh βββΊ Linear(125β249) βββΊ tanh βββΊ Linear(249β125) βββΊ tanh βββΊ Linear(125β1) βββΊ Ε·
| Layer | Weights | Biases | Parameters |
|---|---|---|---|
| Linear(1 β 125) | 125 Γ 1 | 125 | 250 |
| Linear(125 β 249) | 249 Γ 125 | 249 | 31,374 |
| Linear(249 β 125) | 125 Γ 249 | 125 | 31,250 |
| Linear(125 β 1) | 1 Γ 125 | 1 | 126 |
| Total | 63,000 |
| Property | Value |
|---|---|
| input / output size | 1 / 1 |
| hidden layers | 3 (sizes 125, 249, 125) |
| activation function | tanh on hidden layers, linear output |
| weight dtype | float32 |
| initialization | weights ~ N(0, 1/fan_in), biases = 0 |
The hidden layer sizes were chosen so that the total parameter count is exactly 63,000.
Training
Data. A synthetic dataset:
xis uniformly distributed on [β3, 3];y = xΒ·sin(2x) + Ξ΅, whereΞ΅is Gaussian noise with a standard deviation of 0.05;- 512 points for training and 256 points for validation, the random number generator is seeded (
numpy.random.default_rng(0)).
Loss function. Mean squared error (MSE): L = mean((Ε· β y)Β²).
Gradients are computed by hand with backpropagation (NumPy implementation, see train.py).
Optimization. Full-batch Adam (all 512 training points at every step).
| Hyperparameter | Value |
|---|---|
| optimizer | Adam (Ξ²β = 0.9, Ξ²β = 0.999, Ξ΅ = 1e-8) |
| learning rate | 0.002 at the start, cosine decay to 0 |
| iterations | 4000 |
| batch size | 512 (full batch) |
An earlier run with a constant learning rate of 0.002 was unstable: the loss spiked several times (for example, validation MSE 0.00300 at iteration 3200 and 0.00574 at iteration 4000), and the final model was saved right after a spike. The cosine decay was added to fix this and is what produced the final model.
Training result
| Iteration | train MSE | validation MSE |
|---|---|---|
| 1 | 5.36634 | 5.04776 |
| 400 | 0.00401 | 0.00531 |
| 800 | 0.00231 | 0.00326 |
| 1200 | 0.00212 | 0.00304 |
| 2000 | 0.00206 | 0.00297 |
| 4000 (final) | 0.00204 | 0.00293 |
The noise variance in the data (ΟΒ²) is 0.0025. The validation MSE is about 17% above it, and the train MSE is slightly below it (0.0020), which indicates mild fitting to the noise in the training points: the network has far more parameters (63,000) than training points (512). The gap between train and validation MSE is small, and both stay almost flat from iteration 2000 onwards.