GPT-6.3 Astra

A 63,000-parameter neural network (fully connected, three tanh hidden layers) trained to approximate the function xΒ·sin(2x).

Architecture

A multilayer perceptron 1 β†’ 125 β†’ 249 β†’ 125 β†’ 1: one number in, one number out, three hidden layers with the tanh activation.

x ──► Linear(1β†’125) ──► tanh ──► Linear(125β†’249) ──► tanh ──► Linear(249β†’125) ──► tanh ──► Linear(125β†’1) ──► Ε·
Layer Weights Biases Parameters
Linear(1 β†’ 125) 125 Γ— 1 125 250
Linear(125 β†’ 249) 249 Γ— 125 249 31,374
Linear(249 β†’ 125) 125 Γ— 249 125 31,250
Linear(125 β†’ 1) 1 Γ— 125 1 126
Total 63,000
Property Value
input / output size 1 / 1
hidden layers 3 (sizes 125, 249, 125)
activation function tanh on hidden layers, linear output
weight dtype float32
initialization weights ~ N(0, 1/fan_in), biases = 0

The hidden layer sizes were chosen so that the total parameter count is exactly 63,000.

Training

Data. A synthetic dataset:

  • x is uniformly distributed on [βˆ’3, 3];
  • y = xΒ·sin(2x) + Ξ΅, where Ξ΅ is Gaussian noise with a standard deviation of 0.05;
  • 512 points for training and 256 points for validation, the random number generator is seeded (numpy.random.default_rng(0)).

Loss function. Mean squared error (MSE): L = mean((Ε· βˆ’ y)Β²). Gradients are computed by hand with backpropagation (NumPy implementation, see train.py).

Optimization. Full-batch Adam (all 512 training points at every step).

Hyperparameter Value
optimizer Adam (β₁ = 0.9, Ξ²β‚‚ = 0.999, Ξ΅ = 1e-8)
learning rate 0.002 at the start, cosine decay to 0
iterations 4000
batch size 512 (full batch)

An earlier run with a constant learning rate of 0.002 was unstable: the loss spiked several times (for example, validation MSE 0.00300 at iteration 3200 and 0.00574 at iteration 4000), and the final model was saved right after a spike. The cosine decay was added to fix this and is what produced the final model.

Training result

Iteration train MSE validation MSE
1 5.36634 5.04776
400 0.00401 0.00531
800 0.00231 0.00326
1200 0.00212 0.00304
2000 0.00206 0.00297
4000 (final) 0.00204 0.00293

The noise variance in the data (σ²) is 0.0025. The validation MSE is about 17% above it, and the train MSE is slightly below it (0.0020), which indicates mild fitting to the noise in the training points: the network has far more parameters (63,000) than training points (512). The gap between train and validation MSE is small, and both stay almost flat from iteration 2000 onwards.

Downloads last month

-

Downloads are not tracked for this model. How to track
Safetensors
Model size
63k params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Collection including salgr/GPT-6.3-Astra