GPT-6 Cyber

A 60,000-parameter neural network (fully connected, four tanh hidden layers) trained to approximate the function sin(3x)Β·cos(x) + 0.1Β·xΒ².

Architecture

A multilayer perceptron 1 β†’ 144 β†’ 128 β†’ 127 β†’ 192 β†’ 1: one number in, one number out, four hidden layers with the tanh activation.

x ──► Linear(1β†’144) ──► tanh ──► Linear(144β†’128) ──► tanh ──► Linear(128β†’127) ──► tanh ──► Linear(127β†’192) ──► tanh ──► Linear(192β†’1) ──► Ε·
Layer Weights Biases Parameters
Linear(1 β†’ 144) 144 Γ— 1 144 288
Linear(144 β†’ 128) 128 Γ— 144 128 18,560
Linear(128 β†’ 127) 127 Γ— 128 127 16,383
Linear(127 β†’ 192) 192 Γ— 127 192 24,576
Linear(192 β†’ 1) 1 Γ— 192 1 193
Total 60,000
Property Value
input / output size 1 / 1
hidden layers 4 (sizes 144, 128, 127, 192)
activation function tanh on hidden layers, linear output
weight dtype float32
initialization weights ~ N(0, 1/fan_in), biases = 0

The hidden layer sizes were chosen so that the total parameter count is exactly 60,000.

Training

Data. A synthetic dataset:

  • x is uniformly distributed on [βˆ’3, 3];
  • y = sin(3x)Β·cos(x) + 0.1Β·xΒ² + Ξ΅, where Ξ΅ is Gaussian noise with a standard deviation of 0.05;
  • 512 points for training and 256 points for validation, the random number generator is seeded (numpy.random.default_rng(0)).

Loss function. Mean squared error (MSE): L = mean((Ε· βˆ’ y)Β²). Gradients are computed by hand with backpropagation (NumPy implementation, see train.py).

Optimization. Full-batch Adam (all 512 training points at every step).

Hyperparameter Value
optimizer Adam (β₁ = 0.9, Ξ²β‚‚ = 0.999, Ξ΅ = 1e-8)
learning rate 0.002 at the start, cosine decay to 0
iterations 4000
batch size 512 (full batch)

Training result

Iteration train MSE validation MSE
1 3.17048 3.02635
400 0.00699 0.00620
800 0.00229 0.00300
1200 0.00208 0.00286
2000 0.00204 0.00286
4000 (final) 0.00202 0.00286

The noise variance in the data (σ²) is 0.0025. The validation MSE is about 14% above it, and the train MSE is slightly below it (0.0020), which indicates mild fitting to the noise in the training points: the network has far more parameters (60,000) than training points (512). The gap between train and validation MSE is small, and both stay almost flat from iteration 1200 onwards.

Downloads last month

-

Downloads are not tracked for this model. How to track
Safetensors
Model size
60k params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Collection including salgr/GPT-6-Cyber