psikosen/snow_spike

Snow Spike Project

Overview

Snow Spike is a Python project implementing entropy calculation, normalization, decay mechanisms, and living weights for a neural network system. The project focuses on implementing four key components:

1. True Discrete Entropy for Ternary Spikes - Calculate Shannon entropy for neurons with discrete states
2. Normalization of Update Terms - Ensure consistent scaling of different signals
3. Decay Discretization - Implement decay mechanisms for scores over time
4. Living Weights - Biologically-inspired adaptive weights that change based on activity patterns

MLX Integration

This project leverages the MLX framework for efficient computation. MLX is a machine learning framework designed for Apple Silicon, providing accelerated computation for array operations.

Architecture

Snow Spike is an experimental ML stack with a reusable signal-processing core and several training and application entry points:

• Signal primitives in src/entropy, src/normalization, src/decay, and src/weights implement ternary-spike entropy, normalized update signals, time-based decay, and activity-dependent weights.
• Text and data handling in src/nlp and src/data converts text into character/spike-oriented sequences and batches them for learning.
• Model composition in src/enhanced_networks and src/enhanced_architectures combines those primitives into shallow and deep adaptive networks, including MLX-backed variants and text generators.
• Training and evaluation are orchestrated by the root-level trainer scripts (such as train.py, enhanced_trainer.py, and enhanced_deep_trainer.py), with metrics in src/metrics and runnable checks in tests/ and examples/.
• Interactive use is exposed through app.py, gui/, and the model-loading helpers, which load saved artifacts for chat or inspection.

The typical data flow is:

text corpus -> character/spike encoding -> batched sequences
           -> adaptive MLX network -> entropy/decay/weight updates
           -> loss and metrics -> saved artifact -> generation or chat UI

The many top-level trainer, fix, and demo scripts are experiment entry points around this shared stack, rather than separate services.

Project Structure

snow_spike/
├── src/
│   ├── entropy/
│   │   ├── __init__.py
│   │   └── spike_entropy.py
│   ├── normalization/
│   │   ├── __init__.py
│   │   └── signal_normalization.py
│   ├── decay/
│   │   ├── __init__.py
│   │   └── decay_mechanisms.py
│   ├── weights/
│   │   ├── __init__.py
│   │   ├── living_weight.py
│   │   └── adaptive_network.py
│   └── utils/
│       ├── __init__.py
│       └── common.py
├── tests/
│   ├── entropy/
│   │   └── test_spike_entropy.py
│   ├── normalization/
│   │   └── test_signal_normalization.py
│   ├── decay/
│   │   └── test_decay_mechanisms.py
│   ├── weights/
│   │   └── test_living_weights.py
│   └── utils/
│       └── test_common.py
├── docs/
│   ├── entropy.md
│   ├── normalization.md
│   ├── decay.md
│   └── living_weights.md
├── examples/
│   ├── entropy_example.py
│   ├── normalize_example.py
│   ├── decay_example.py
│   ├── integrated_example.py
│   └── validate_living_weights.py
├── requirements.txt
├── setup.py
├── README.md
└── config.json

Installation

# Clone the repository
git clone https://github.com/username/snow_spike.git
cd snow_spike

# Create a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Install the package in development mode
pip install -e .

Usage

Entropy Calculation

from snow_spike.entropy import count_spike_occurrences, calculate_empirical_probabilities
from snow_spike.entropy import compute_neuron_entropy, aggregate_network_entropy
from snow_spike.utils.common import SpikeState

# Count spike occurrences
spike_data = [-1, 0, 1, 0, -1, 0, 0, 1, 0, 0]
counts = count_spike_occurrences(0, spike_data, len(spike_data))

# Calculate probabilities
probabilities = calculate_empirical_probabilities(counts, len(spike_data))

# Compute entropy
neuron_entropy = compute_neuron_entropy(probabilities)

Signal Normalization

from snow_spike.normalization import normalize_logic_check, normalize_cross_check
from snow_spike.normalization import normalize_corroboration_signal
from snow_spike.utils.common import CrossCheckType

# Normalize different signals
logic_norm = normalize_logic_check(1)
cross_norm = normalize_cross_check(0.75, CrossCheckType.COSINE_SIMILARITY)
corr_norm = normalize_corroboration_signal(5, kappa=0.5)

# Calculate trust delta
from snow_spike.normalization import calculate_trust_delta
delta_t = calculate_trust_delta(logic_norm, cross_norm, corr_norm, 0.3, 0.4, 0.3)

Decay Mechanisms

from snow_spike.decay import apply_continuous_decay, DecayManager
from snow_spike.utils.common import DecayStrategy
import time

# Apply continuous decay
initial_value = 1.0
initial_time = time.time() - 3600  # 1 hour ago
current_time = time.time()
decay_lambda = 0.01

decayed_value = apply_continuous_decay(initial_value, initial_time, current_time, decay_lambda)

# Use decay manager
manager = DecayManager(strategy=DecayStrategy.CONTINUOUS_ON_READ)
decayed_value = manager.decay_value(initial_value, initial_time)

Living Weights

from snow_spike.weights import AdaptiveNetwork, LivingWeight

# Create a network with living weights
network = AdaptiveNetwork(
    layer_sizes=[2, 4, 1],
    decay_lambda=0.001,
    growth_rate=0.05
)

# Train on XOR problem
inputs = [[1, 1], [1, -1], [-1, 1], [-1, -1]]
targets = [-1, 1, 1, -1]

# Train the network
history = network.train(inputs, targets, epochs=20)

# Make predictions
output = network.forward([1, -1])

# Calculate network entropy
entropy, layer_entropies = network.calculate_network_entropy()

Running Examples

You can run the provided examples to see each module in action:

# Run specific examples
./run.sh entropy     # Entropy calculation example
./run.sh normalize   # Normalization example 
./run.sh decay       # Decay mechanisms example

# Run the integrated example
python examples/integrated_example.py

# Validate living weights
python examples/validate_living_weights.py

Development

To set up the development environment and run tests:

# Build and test everything
./build_all.sh

# Run tests
pytest tests/

# Check code style
flake8 src tests

Documentation

Detailed documentation for each module is available in the docs/ directory:

• Entropy Calculation
• Normalization
• Decay Mechanisms
• Living Weights

References

• MLX Documentation
• MLX-LM GitHub Repository
• Absolute Zero Paper