GPT-6 Cyber
A 60,000-parameter neural network (fully connected, four tanh hidden layers) trained to approximate the function sin(3x)Β·cos(x) + 0.1Β·xΒ².
Architecture
A multilayer perceptron 1 β 144 β 128 β 127 β 192 β 1: one number in, one number out, four hidden layers with the tanh activation.
x βββΊ Linear(1β144) βββΊ tanh βββΊ Linear(144β128) βββΊ tanh βββΊ Linear(128β127) βββΊ tanh βββΊ Linear(127β192) βββΊ tanh βββΊ Linear(192β1) βββΊ Ε·
| Layer | Weights | Biases | Parameters |
|---|---|---|---|
| Linear(1 β 144) | 144 Γ 1 | 144 | 288 |
| Linear(144 β 128) | 128 Γ 144 | 128 | 18,560 |
| Linear(128 β 127) | 127 Γ 128 | 127 | 16,383 |
| Linear(127 β 192) | 192 Γ 127 | 192 | 24,576 |
| Linear(192 β 1) | 1 Γ 192 | 1 | 193 |
| Total | 60,000 |
| Property | Value |
|---|---|
| input / output size | 1 / 1 |
| hidden layers | 4 (sizes 144, 128, 127, 192) |
| activation function | tanh on hidden layers, linear output |
| weight dtype | float32 |
| initialization | weights ~ N(0, 1/fan_in), biases = 0 |
The hidden layer sizes were chosen so that the total parameter count is exactly 60,000.
Training
Data. A synthetic dataset:
xis uniformly distributed on [β3, 3];y = sin(3x)Β·cos(x) + 0.1Β·xΒ² + Ξ΅, whereΞ΅is Gaussian noise with a standard deviation of 0.05;- 512 points for training and 256 points for validation, the random number generator is seeded (
numpy.random.default_rng(0)).
Loss function. Mean squared error (MSE): L = mean((Ε· β y)Β²).
Gradients are computed by hand with backpropagation (NumPy implementation, see train.py).
Optimization. Full-batch Adam (all 512 training points at every step).
| Hyperparameter | Value |
|---|---|
| optimizer | Adam (Ξ²β = 0.9, Ξ²β = 0.999, Ξ΅ = 1e-8) |
| learning rate | 0.002 at the start, cosine decay to 0 |
| iterations | 4000 |
| batch size | 512 (full batch) |
Training result
| Iteration | train MSE | validation MSE |
|---|---|---|
| 1 | 3.17048 | 3.02635 |
| 400 | 0.00699 | 0.00620 |
| 800 | 0.00229 | 0.00300 |
| 1200 | 0.00208 | 0.00286 |
| 2000 | 0.00204 | 0.00286 |
| 4000 (final) | 0.00202 | 0.00286 |
The noise variance in the data (ΟΒ²) is 0.0025. The validation MSE is about 14% above it, and the train MSE is slightly below it (0.0020), which indicates mild fitting to the noise in the training points: the network has far more parameters (60,000) than training points (512). The gap between train and validation MSE is small, and both stay almost flat from iteration 1200 onwards.