| --- |
| license: apache-2.0 |
| tags: |
| - tensorflow |
| - optimizer |
| - deep-learning |
| - machine-learning |
| - adaptive-learning-rate |
| library_name: tensorflow |
| --- |
| |
| # NEAT Optimizer |
|
|
| **NEAT (Noise-Enhanced Adaptive Training)** is a novel optimization algorithm for deep learning that combines adaptive learning rates with controlled noise injection to improve convergence and generalization. |
|
|
| ## Overview |
|
|
| The NEAT optimizer enhances traditional adaptive optimization methods by intelligently injecting noise into the gradient updates. This approach helps: |
| - Escape local minima more effectively |
| - Improve generalization performance |
| - Achieve faster and more stable convergence |
| - Reduce overfitting on training data |
|
|
| ## Installation |
|
|
| ### From PyPI (recommended) |
| ```bash |
| pip install neat-optimizer |
| ``` |
|
|
| ### From Source |
| ```bash |
| git clone https://github.com/yourusername/neat-optimizer.git |
| cd neat-optimizer |
| pip install -e . |
| ``` |
|
|
| ## Quick Start |
|
|
| ```python |
| import tensorflow as tf |
| from neat_optimizer import NEATOptimizer |
| |
| # Create your model |
| model = tf.keras.Sequential([ |
| tf.keras.layers.Dense(128, activation='relu'), |
| tf.keras.layers.Dense(10, activation='softmax') |
| ]) |
| |
| # Use NEAT optimizer |
| optimizer = NEATOptimizer( |
| learning_rate=0.001, |
| noise_scale=0.01, |
| beta_1=0.9, |
| beta_2=0.999 |
| ) |
| |
| # Compile and train |
| model.compile( |
| optimizer=optimizer, |
| loss='sparse_categorical_crossentropy', |
| metrics=['accuracy'] |
| ) |
| |
| model.fit(x_train, y_train, epochs=10, validation_data=(x_val, y_val)) |
| ``` |
|
|
| ## Key Features |
|
|
| - **Adaptive Learning Rates**: Automatically adjusts learning rates per parameter |
| - **Noise Injection**: Controlled stochastic perturbations for better exploration |
| - **TensorFlow Integration**: Drop-in replacement for standard TensorFlow optimizers |
| - **Hyperparameter Flexibility**: Customizable noise schedules and adaptation rates |
|
|
| ## Parameters |
|
|
| - `learning_rate` (float, default=0.001): Initial learning rate |
| - `noise_scale` (float, default=0.01): Scale of noise injection |
| - `beta_1` (float, default=0.9): Exponential decay rate for first moment estimates |
| - `beta_2` (float, default=0.999): Exponential decay rate for second moment estimates |
| - `epsilon` (float, default=1e-7): Small constant for numerical stability |
| - `noise_decay` (float, default=0.99): Decay rate for noise scale over time |
|
|
| ## Requirements |
|
|
| - Python >= 3.7 |
| - TensorFlow >= 2.4.0 |
| - NumPy >= 1.19.0 |
|
|
| ## Citation |
|
|
| If you use NEAT optimizer in your research, please cite: |
|
|
| ```bibtex |
| @software{neat_optimizer, |
| title={NEAT: Noise-Enhanced Adaptive Training Optimizer}, |
| author={Your Name}, |
| year={2025}, |
| url={https://github.com/yourusername/neat-optimizer} |
| } |
| ``` |
|
|
| ## References |
|
|
| - Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. |
| - Neelakantan, A., et al. (2015). Adding gradient noise improves learning for very deep networks. arXiv preprint arXiv:1511.06807. |
|
|
| ## License |
|
|
| This project is licensed under the Apache License 2.0 - see the LICENSE file for details. |
|
|
| ## Contributing |
|
|
| Contributions are welcome! Please feel free to submit a Pull Request. |
|
|
| ## Support |
|
|
| For issues, questions, or feature requests, please open an issue on GitHub. |
