docs/experiment_protocol_001.md

Experiment Protocol 001: MMEP Arithmetic Ablation

Project: BURT-IMMA
Contact: jessica@collectivekitty.com
License: BSL-1.1

---

1. Hypothesis

Primary hypothesis: MMEP (Matrix-Memory Equilibrium Propagation) with matrix memory outperforms standard backpropagation on arithmetic reasoning tasks.

Secondary hypotheses:

• H2: Matrix memory (C_global + C_expert) is necessary for strong arithmetic performance
• H3: Spectral and entropy constraints improve convergence stability without hurting accuracy
• H4: Sparse MoE routing enables specialization (e.g., one expert for addition, another for multiplication)
• H5: The equilibrium propagation training signal is at least as effective as backpropagation for this domain

---

2. Task Description

Arithmetic reasoning: Given an arithmetic expression as input, predict the correct result.

Expression types:

• Addition: a + b = ? (integers, 0-999)
• Subtraction: a - b = ? (integers, result >= 0)
• Multiplication: a * b = ? (integers, 0-99)
• Division: a / b = ? (exact division only, result is integer)
• Chained: a + b * c - d = ? (2-4 operations, PEMDAS)
• Nested: (a + b) * (c - d) = ? (parentheses)

Data generation: scripts/generate_arithmetic.py

# generate_arithmetic.py produces:
# - train.jsonl: 100,000 expressions
# - val.jsonl: 10,000 expressions
# - test.jsonl: 10,000 expressions
# Format: {"input": "23 + 45 = ?", "output": "68", "type": "addition", "difficulty": 1}

Difficulty levels:

1. Single operation, small numbers (0-99)
2. Single operation, larger numbers (0-999)
3. Two operations, small numbers
4. Two operations, larger numbers
5. Three+ operations with parentheses

---

3. Experiment Configurations

3.1 Configuration Table

#	Config Name	Description	Key Modification
1	full_mmep	Complete BURT-IMMA system	None (baseline)
2	no_memory	Remove all matrix memory	C_global = 0, C_expert = 0, no CIFG
3	no_constraint	Remove spectral + entropy constraints	lambda_max = inf, entropy_bound = inf
4	no_moe	Single expert, no routing	num_experts = 1, no router
5	standard_bp	Standard backpropagation training	Replace EP with Adam + backprop
6	high_T_free	Extended free phase	T_free = 100 (vs default 20)
7	low_beta	Very weak nudging	beta = 0.001 (vs default 0.1)
8	large_memory	Doubled memory capacity	d_model = 1024 (vs default 512)

3.2 Detailed Configuration Specifications

Config 1: full_mmep (Baseline)

model:
  d_model: 512
  num_layers: 4
  num_heads: 8
  num_experts: 4
  top_k: 2
  d_expert: 256

training:
  method: mmep
  T_free: 20
  T_nudge: 4
  beta: 0.1
  lr: 1e-4
  epochs: 50
  batch_size: 8

constraints:
  lambda_max: 0.95
  entropy_bound: 0.20
  spectral_projection: true
  entropy_projection: true

memory:
  C_global_dim: 512
  C_expert_dim: 256
  cifg_enabled: true
  forget_bias_init: 0.0

Config 2: no_memory

# Same as full_mmep except:
memory:
  C_global_dim: 0       # No global memory
  C_expert_dim: 0       # No expert memory
  cifg_enabled: false   # No CIFG gating
  # Forward pass skips memory retrieval step

Config 3: no_constraint

# Same as full_mmep except:
constraints:
  lambda_max: 1000.0       # Effectively no spectral bound
  entropy_bound: 100.0     # Effectively no entropy bound
  spectral_projection: false
  entropy_projection: false

Config 4: no_moe

# Same as full_mmep except:
model:
  num_experts: 1    # Single expert
  top_k: 1          # Must route to the only expert
  # No router, no load balancing, no sparsity penalty

Config 5: standard_bp

# Same model architecture as full_mmep, but:
training:
  method: backprop    # Standard backpropagation
  optimizer: adam
  lr: 3e-4
  weight_decay: 0.01
  epochs: 50
  batch_size: 32      # Larger batch (no EP memory overhead)
  # No T_free, T_nudge, or beta

Config 6: high_T_free

# Same as full_mmep except:
training:
  T_free: 100    # 5x longer free phase (default: 20)
  # Tests whether longer equilibration improves quality

Config 7: low_beta

# Same as full_mmep except:
training:
  beta: 0.001    # 100x weaker nudging (default: 0.1)
  # Tests biological plausibility limit
  # Lower beta = more accurate gradient but weaker signal

Config 8: large_memory

# Same as full_mmep except:
model:
  d_model: 1024       # 2x model dimension
  d_expert: 512       # 2x expert dimension
memory:
  C_global_dim: 1024  # 4x memory capacity (1024^2 vs 512^2)
  C_expert_dim: 512   # 4x expert memory

---

4. Metrics

4.1 Primary Metrics

Metric	Definition	Collection Frequency
Accuracy	Exact match of predicted vs. true answer	Every epoch
Convergence speed	Epoch at which accuracy first exceeds 80%	Once per run
Final accuracy	Test set accuracy at epoch 50	End of training

4.2 Secondary Metrics

Metric	Definition	Collection Frequency
Routing entropy	H(alpha) averaged over test set	Every epoch
Max spectral norm	max_W(sigma_max(W))	Every epoch
Memory utilization	effective_rank(C_global) / d_model	Every 5 epochs
Energy trajectory	E(t) over training (EP configs only)	Every step
Expert specialization	Accuracy per expert per operation type	End of training
Loss curve	Training loss per epoch	Every epoch

4.3 Per-Operation Breakdown

Report accuracy separately for each operation type:

• Addition accuracy
• Subtraction accuracy
• Multiplication accuracy
• Division accuracy
• Chained operations accuracy
• Nested operations accuracy

---

5. Falsification Criteria

The MMEP theory makes specific, testable claims. The following criteria define falsification:

5.1 Primary Falsification

If standard_bp (Config 5) achieves higher final accuracy than full_mmep (Config 1) by more than 2% (with p < 0.05), then:

The claim that "MMEP outperforms backpropagation on reasoning tasks" is falsified for arithmetic reasoning.

5.2 Secondary Falsification Criteria

Criterion	Condition	What it falsifies
F1	standard_bp > full_mmep + 2%	EP training advantage
F2	no_memory >= full_mmep	Matrix memory necessity
F3	no_constraint > full_mmep + 2%	Constraint benefit
F4	Energy non-monotonic for 10+ consecutive steps	Lyapunov guarantee
F5	Spectral norm > lambda_max after projection	Spectral projection correctness
F6	Entropy > 0.20 + 1e-3 after projection	Entropy projection correctness
F7	no_moe >= full_mmep	MoE specialization benefit

5.3 Non-Falsification Outcomes

If none of the falsification criteria are triggered:

• MMEP is validated for arithmetic reasoning (not universally)
• Results support the theoretical framework but do not prove it
• Generalization to other domains requires separate experiments

5.4 Interpretation Guide

Outcome	Interpretation	Action
All criteria pass	Theory supported	Proceed to harder tasks
F1 triggered	EP training not beneficial here	Investigate task properties
F2 triggered	Memory unnecessary for arithmetic	Reduce architecture
F3 triggered	Constraints too restrictive	Relax bounds
F4 triggered	Implementation bug or theory gap	Debug convergence
F5 or F6 triggered	Projection implementation error	Fix kernel bugs
F7 triggered	MoE overhead not justified	Simplify to single expert

---

6. Hardware Requirements

6.1 Minimum Hardware

• GPU: NVIDIA RTX 3080 (10 GB VRAM) or equivalent
• CPU: 8 cores (for data loading)
• RAM: 32 GB
• Storage: 20 GB free (checkpoints + logs)
• CUDA: 11.8 or later

6.2 VRAM Budget per Configuration

Config	Model Params	Memory Matrices	Activations	Total VRAM
full_mmep	200 MB	4 MB	500 MB	~2 GB
no_memory	195 MB	0 MB	450 MB	~1.5 GB
no_constraint	200 MB	4 MB	500 MB	~2 GB
no_moe	150 MB	2 MB	400 MB	~1.5 GB
standard_bp	200 MB	4 MB	300 MB	~1.5 GB
high_T_free	200 MB	4 MB	500 MB	~2 GB
low_beta	200 MB	4 MB	500 MB	~2 GB
large_memory	400 MB	16 MB	900 MB	~3.5 GB

All configurations fit within RTX 3080's 10 GB with ample headroom.

---

7. Expected Runtime

7.1 Per-Configuration Runtime

Config	Time per Epoch	Total (50 epochs)	Notes
full_mmep	3 min	2.5 hours	EP overhead: T_free + T_nudge passes
no_memory	2.5 min	2 hours	No memory ops
no_constraint	2.5 min	2 hours	No projection
no_moe	2 min	1.7 hours	No routing/dispatch
standard_bp	1 min	50 min	Standard training (no EP)
high_T_free	10 min	8.3 hours	5x free phase
low_beta	3 min	2.5 hours	Same as full_mmep
large_memory	6 min	5 hours	2x compute

7.2 Total Suite Runtime

Sequential (worst case): 2.5 + 2 + 2 + 1.7 + 0.8 + 8.3 + 2.5 + 5 = 24.8 hours
With 3 seeds each: 24.8 * 3 = 74.4 hours

Excluding high_T_free: (24.8 - 8.3) * 3 = 49.5 hours
Parallel (2 configs on RTX 3080): ~37 hours

Target: ~4 hours for single-seed pass (excluding high_T_free)
   = 2.5 + 2 + 2 + 1.7 + 0.8 + 2.5 + 5 = 16.5 hours single seed all configs
   
Achievable in ~4 hours: Run full_mmep + standard_bp + no_memory + no_moe (key comparisons)
   = 2.5 + 0.8 + 2 + 1.7 = 7 hours... 

Revised target for "core suite" (4 hours):
   - full_mmep: 2.5h
   - standard_bp: 0.8h  
   - no_memory: limited to 15 epochs = 0.75h
   Total core: ~4 hours

7.3 Recommended Execution Order

1. standard_bp (fastest, establishes backprop baseline)
2. full_mmep (primary comparison)
3. no_memory (test memory contribution)
4. no_moe (test routing contribution)
5. no_constraint (test constraint contribution)
6. low_beta (test nudging sensitivity)
7. large_memory (test scaling)
8. high_T_free (test convergence theory, run overnight)

---

8. Data

8.1 Data Generation

python scripts/generate_arithmetic.py \
    --train_size 100000 \
    --val_size 10000 \
    --test_size 10000 \
    --max_value 999 \
    --max_ops 4 \
    --seed 42 \
    --output_dir data/arithmetic/

8.2 Data Format

{"input": "23 + 45 = ?", "output": "68", "type": "addition", "difficulty": 1, "num_ops": 1}
{"input": "156 * 7 = ?", "output": "1092", "type": "multiplication", "difficulty": 2, "num_ops": 1}
{"input": "(12 + 8) * 3 = ?", "output": "60", "type": "nested", "difficulty": 3, "num_ops": 2}

8.3 Data Statistics

Split	Size	Difficulty Distribution	Operation Distribution
Train	100K	D1: 30%, D2: 25%, D3: 20%, D4: 15%, D5: 10%	+: 25%, -: 25%, *: 20%, /: 10%, chain: 12%, nest: 8%
Val	10K	Same as train	Same as train
Test	10K	Same as train	Same as train

8.4 Tokenization

• Character-level tokenization (digits, operators, parentheses, spaces)
• Vocabulary size: 20 tokens (0-9, +, -, *, /, (, ), =, ?, , <PAD>)
• Maximum sequence length: 32 (sufficient for all expressions)

---

9. Execution Script

#!/bin/bash
# run_experiment_001.sh

CONFIGS="full_mmep standard_bp no_memory no_constraint no_moe low_beta large_memory high_T_free"
SEEDS="42 137 256"
DATA_DIR="data/arithmetic"
OUTPUT_DIR="results/experiment_001"

# Generate data (if not exists)
if [ ! -f "$DATA_DIR/train.jsonl" ]; then
    python scripts/generate_arithmetic.py --output_dir $DATA_DIR --seed 42
fi

# Run experiments
for config in $CONFIGS; do
    for seed in $SEEDS; do
        echo "Running config=$config seed=$seed"
        python scripts/run_training.py \
            --config configs/experiment_001/${config}.yaml \
            --seed $seed \
            --data_dir $DATA_DIR \
            --output_dir $OUTPUT_DIR/${config}/seed_${seed} \
            --epochs 50 \
            --eval_every 1 \
            --checkpoint_every 10
    done
done

# Generate report
python scripts/analyze_results.py \
    --results_dir $OUTPUT_DIR \
    --output results/experiment_001_report.json

---

10. Analysis Plan

10.1 Statistical Tests

1. Paired t-test: Compare each config against full_mmep on final accuracy (3 seeds each)
2. Bonferroni correction: Adjust p-values for 7 comparisons (alpha = 0.05/7 = 0.0071)
3. Effect size: Report Cohen's d for each comparison
4. Confidence intervals: 95% CI on accuracy difference

10.2 Visualization

• Learning curves (accuracy vs. epoch) for all configs
• Energy trajectories (EP configs only)
• Expert specialization heatmap (operation type vs. expert)
• Spectral norm evolution over training
• Entropy evolution over training

10.3 Report Format

{
    "experiment": "001_mmep_arithmetic_ablation",
    "date": "YYYY-MM-DD",
    "hardware": "RTX 3080, 10GB",
    "results": {
        "full_mmep": {"accuracy": [s1, s2, s3], "mean": X, "std": Y},
        "standard_bp": {"accuracy": [s1, s2, s3], "mean": X, "std": Y},
        ...
    },
    "falsification": {
        "F1": {"triggered": false, "p_value": 0.XX, "effect_size": X.XX},
        ...
    },
    "conclusion": "Theory supported|falsified for criterion FN"
}

---

11. Reproducibility Checklist

• All random seeds documented and fixed
• Data generation script deterministic
• Model initialization deterministic
• CUDA deterministic mode enabled (torch.backends.cudnn.deterministic = True)
• All hyperparameters documented in config YAML
• PyTorch version recorded
• CUDA version recorded
• GPU model recorded
• Training logs saved (loss per step)
• Checkpoints saved at epoch 10, 20, 30, 40, 50
• Environment frozen (pip freeze > requirements_experiment_001.txt)