docs/experiment_protocol_001.md

# Experiment Protocol 001: MMEP Arithmetic Ablation

**Project:** BURT-IMMA  
**Contact:** jessica@collectivekitty.com  
**License:** BSL-1.1

---

## 1. Hypothesis

**Primary hypothesis:** MMEP (Matrix-Memory Equilibrium Propagation) with matrix memory outperforms standard backpropagation on arithmetic reasoning tasks.

**Secondary hypotheses:**
- H2: Matrix memory (C_global + C_expert) is necessary for strong arithmetic performance
- H3: Spectral and entropy constraints improve convergence stability without hurting accuracy
- H4: Sparse MoE routing enables specialization (e.g., one expert for addition, another for multiplication)
- H5: The equilibrium propagation training signal is at least as effective as backpropagation for this domain

---

## 2. Task Description

**Arithmetic reasoning:** Given an arithmetic expression as input, predict the correct result.

**Expression types:**
- Addition: `a + b = ?` (integers, 0-999)
- Subtraction: `a - b = ?` (integers, result >= 0)
- Multiplication: `a * b = ?` (integers, 0-99)
- Division: `a / b = ?` (exact division only, result is integer)
- Chained: `a + b * c - d = ?` (2-4 operations, PEMDAS)
- Nested: `(a + b) * (c - d) = ?` (parentheses)

**Data generation:** `scripts/generate_arithmetic.py`

```python

# generate_arithmetic.py produces:

# - train.jsonl: 100,000 expressions

# - val.jsonl: 10,000 expressions

# - test.jsonl: 10,000 expressions

# Format: {"input": "23 + 45 = ?", "output": "68", "type": "addition", "difficulty": 1}

```

**Difficulty levels:**
1. Single operation, small numbers (0-99)
2. Single operation, larger numbers (0-999)
3. Two operations, small numbers
4. Two operations, larger numbers
5. Three+ operations with parentheses

---

## 3. Experiment Configurations

### 3.1 Configuration Table

| # | Config Name | Description | Key Modification |
|---|-------------|-------------|------------------|
| 1 | `full_mmep` | Complete BURT-IMMA system | None (baseline) |
| 2 | `no_memory` | Remove all matrix memory | C_global = 0, C_expert = 0, no CIFG |
| 3 | `no_constraint` | Remove spectral + entropy constraints | lambda_max = inf, entropy_bound = inf |
| 4 | `no_moe` | Single expert, no routing | num_experts = 1, no router |

| 5 | `standard_bp` | Standard backpropagation training | Replace EP with Adam + backprop |
| 6 | `high_T_free` | Extended free phase | T_free = 100 (vs default 20) |

| 7 | `low_beta` | Very weak nudging | beta = 0.001 (vs default 0.1) |
| 8 | `large_memory` | Doubled memory capacity | d_model = 1024 (vs default 512) |



### 3.2 Detailed Configuration Specifications



#### Config 1: `full_mmep` (Baseline)

```yaml

model:

  d_model: 512

  num_layers: 4

  num_heads: 8

  num_experts: 4

  top_k: 2

  d_expert: 256



training:

  method: mmep

  T_free: 20

  T_nudge: 4

  beta: 0.1

  lr: 1e-4

  epochs: 50

  batch_size: 8



constraints:

  lambda_max: 0.95

  entropy_bound: 0.20

  spectral_projection: true

  entropy_projection: true



memory:

  C_global_dim: 512

  C_expert_dim: 256

  cifg_enabled: true

  forget_bias_init: 0.0

```

#### Config 2: `no_memory`



```yaml

# Same as full_mmep except:
memory:
  C_global_dim: 0       # No global memory
  C_expert_dim: 0       # No expert memory
  cifg_enabled: false   # No CIFG gating

  # Forward pass skips memory retrieval step

```



#### Config 3: `no_constraint`

```yaml

# Same as full_mmep except:

constraints:

  lambda_max: 1000.0       # Effectively no spectral bound

  entropy_bound: 100.0     # Effectively no entropy bound

  spectral_projection: false

  entropy_projection: false

```

#### Config 4: `no_moe`



```yaml

# Same as full_mmep except:
model:
  num_experts: 1    # Single expert

  top_k: 1          # Must route to the only expert
  # No router, no load balancing, no sparsity penalty
```



#### Config 5: `standard_bp`



```yaml

# Same model architecture as full_mmep, but:

training:

  method: backprop    # Standard backpropagation

  optimizer: adam

  lr: 3e-4

  weight_decay: 0.01

  epochs: 50

  batch_size: 32      # Larger batch (no EP memory overhead)

  # No T_free, T_nudge, or beta

```

#### Config 6: `high_T_free`

```yaml

# Same as full_mmep except:

training:

  T_free: 100    # 5x longer free phase (default: 20)

  # Tests whether longer equilibration improves quality

```

#### Config 7: `low_beta`



```yaml

# Same as full_mmep except:
training:
  beta: 0.001    # 100x weaker nudging (default: 0.1)
  # Tests biological plausibility limit
  # Lower beta = more accurate gradient but weaker signal
```



#### Config 8: `large_memory`



```yaml

# Same as full_mmep except:

model:

  d_model: 1024       # 2x model dimension

  d_expert: 512       # 2x expert dimension

memory:

  C_global_dim: 1024  # 4x memory capacity (1024^2 vs 512^2)

  C_expert_dim: 512   # 4x expert memory

```

---

## 4. Metrics

### 4.1 Primary Metrics

| Metric | Definition | Collection Frequency |
|--------|-----------|---------------------|
| **Accuracy** | Exact match of predicted vs. true answer | Every epoch |
| **Convergence speed** | Epoch at which accuracy first exceeds 80% | Once per run |
| **Final accuracy** | Test set accuracy at epoch 50 | End of training |

### 4.2 Secondary Metrics

| Metric | Definition | Collection Frequency |
|--------|-----------|---------------------|
| **Routing entropy** | H(alpha) averaged over test set | Every epoch |
| **Max spectral norm** | max_W(sigma_max(W)) | Every epoch |
| **Memory utilization** | effective_rank(C_global) / d_model | Every 5 epochs |

| **Energy trajectory** | E(t) over training (EP configs only) | Every step |

| **Expert specialization** | Accuracy per expert per operation type | End of training |

| **Loss curve** | Training loss per epoch | Every epoch |



### 4.3 Per-Operation Breakdown



Report accuracy separately for each operation type:

- Addition accuracy

- Subtraction accuracy

- Multiplication accuracy

- Division accuracy

- Chained operations accuracy

- Nested operations accuracy



---



## 5. Falsification Criteria



The MMEP theory makes specific, testable claims. The following criteria define falsification:



### 5.1 Primary Falsification



**If `standard_bp` (Config 5) achieves higher final accuracy than `full_mmep` (Config 1) by more than 2% (with p < 0.05), then:**



The claim that "MMEP outperforms backpropagation on reasoning tasks" is falsified for arithmetic reasoning.



### 5.2 Secondary Falsification Criteria



| Criterion | Condition | What it falsifies |

|-----------|-----------|-------------------|

| F1 | `standard_bp` > `full_mmep` + 2% | EP training advantage |

| F2 | `no_memory` >= `full_mmep` | Matrix memory necessity |

| F3 | `no_constraint` > `full_mmep` + 2% | Constraint benefit |

| F4 | Energy non-monotonic for 10+ consecutive steps | Lyapunov guarantee |

| F5 | Spectral norm > lambda_max after projection | Spectral projection correctness |
| F6 | Entropy > 0.20 + 1e-3 after projection | Entropy projection correctness |
| F7 | `no_moe` >= `full_mmep` | MoE specialization benefit |

### 5.3 Non-Falsification Outcomes

If none of the falsification criteria are triggered:
- MMEP is validated for arithmetic reasoning (not universally)
- Results support the theoretical framework but do not prove it
- Generalization to other domains requires separate experiments

### 5.4 Interpretation Guide

| Outcome | Interpretation | Action |
|---------|---------------|--------|
| All criteria pass | Theory supported | Proceed to harder tasks |
| F1 triggered | EP training not beneficial here | Investigate task properties |
| F2 triggered | Memory unnecessary for arithmetic | Reduce architecture |
| F3 triggered | Constraints too restrictive | Relax bounds |
| F4 triggered | Implementation bug or theory gap | Debug convergence |
| F5 or F6 triggered | Projection implementation error | Fix kernel bugs |
| F7 triggered | MoE overhead not justified | Simplify to single expert |

---

## 6. Hardware Requirements

### 6.1 Minimum Hardware

- **GPU:** NVIDIA RTX 3080 (10 GB VRAM) or equivalent
- **CPU:** 8 cores (for data loading)
- **RAM:** 32 GB
- **Storage:** 20 GB free (checkpoints + logs)
- **CUDA:** 11.8 or later

### 6.2 VRAM Budget per Configuration

| Config | Model Params | Memory Matrices | Activations | Total VRAM |
|--------|-------------|-----------------|-------------|-----------|
| `full_mmep` | 200 MB | 4 MB | 500 MB | ~2 GB |
| `no_memory` | 195 MB | 0 MB | 450 MB | ~1.5 GB |
| `no_constraint` | 200 MB | 4 MB | 500 MB | ~2 GB |
| `no_moe` | 150 MB | 2 MB | 400 MB | ~1.5 GB |
| `standard_bp` | 200 MB | 4 MB | 300 MB | ~1.5 GB |
| `high_T_free` | 200 MB | 4 MB | 500 MB | ~2 GB |
| `low_beta` | 200 MB | 4 MB | 500 MB | ~2 GB |
| `large_memory` | 400 MB | 16 MB | 900 MB | ~3.5 GB |

All configurations fit within RTX 3080's 10 GB with ample headroom.

---

## 7. Expected Runtime

### 7.1 Per-Configuration Runtime

| Config | Time per Epoch | Total (50 epochs) | Notes |
|--------|---------------|-------------------|-------|
| `full_mmep` | 3 min | 2.5 hours | EP overhead: T_free + T_nudge passes |
| `no_memory` | 2.5 min | 2 hours | No memory ops |
| `no_constraint` | 2.5 min | 2 hours | No projection |
| `no_moe` | 2 min | 1.7 hours | No routing/dispatch |
| `standard_bp` | 1 min | 50 min | Standard training (no EP) |
| `high_T_free` | 10 min | 8.3 hours | 5x free phase |
| `low_beta` | 3 min | 2.5 hours | Same as full_mmep |

| `large_memory` | 6 min | 5 hours | 2x compute |

### 7.2 Total Suite Runtime

```

Sequential (worst case): 2.5 + 2 + 2 + 1.7 + 0.8 + 8.3 + 2.5 + 5 = 24.8 hours

With 3 seeds each: 24.8 * 3 = 74.4 hours



Excluding high_T_free: (24.8 - 8.3) * 3 = 49.5 hours

Parallel (2 configs on RTX 3080): ~37 hours



Target: ~4 hours for single-seed pass (excluding high_T_free)

   = 2.5 + 2 + 2 + 1.7 + 0.8 + 2.5 + 5 = 16.5 hours single seed all configs

   

Achievable in ~4 hours: Run full_mmep + standard_bp + no_memory + no_moe (key comparisons)

   = 2.5 + 0.8 + 2 + 1.7 = 7 hours... 



Revised target for "core suite" (4 hours):

   - full_mmep: 2.5h

   - standard_bp: 0.8h  

   - no_memory: limited to 15 epochs = 0.75h

   Total core: ~4 hours

```

### 7.3 Recommended Execution Order

1. `standard_bp` (fastest, establishes backprop baseline)
2. `full_mmep` (primary comparison)
3. `no_memory` (test memory contribution)
4. `no_moe` (test routing contribution)
5. `no_constraint` (test constraint contribution)
6. `low_beta` (test nudging sensitivity)
7. `large_memory` (test scaling)
8. `high_T_free` (test convergence theory, run overnight)

---

## 8. Data

### 8.1 Data Generation

```bash

python scripts/generate_arithmetic.py \

    --train_size 100000 \

    --val_size 10000 \

    --test_size 10000 \

    --max_value 999 \

    --max_ops 4 \

    --seed 42 \

    --output_dir data/arithmetic/

```

### 8.2 Data Format

```json

{"input": "23 + 45 = ?", "output": "68", "type": "addition", "difficulty": 1, "num_ops": 1}

{"input": "156 * 7 = ?", "output": "1092", "type": "multiplication", "difficulty": 2, "num_ops": 1}

{"input": "(12 + 8) * 3 = ?", "output": "60", "type": "nested", "difficulty": 3, "num_ops": 2}

```

### 8.3 Data Statistics

| Split | Size | Difficulty Distribution | Operation Distribution |
|-------|------|------------------------|----------------------|
| Train | 100K | D1: 30%, D2: 25%, D3: 20%, D4: 15%, D5: 10% | +: 25%, -: 25%, *: 20%, /: 10%, chain: 12%, nest: 8% |

| Val | 10K | Same as train | Same as train |

| Test | 10K | Same as train | Same as train |



### 8.4 Tokenization



- Character-level tokenization (digits, operators, parentheses, spaces)

- Vocabulary size: 20 tokens (`0-9`, `+`, `-`, `*`, `/`, `(`, `)`, `=`, `?`, ` `, `<PAD>`)
- Maximum sequence length: 32 (sufficient for all expressions)

---

## 9. Execution Script

```bash

#!/bin/bash

# run_experiment_001.sh



CONFIGS="full_mmep standard_bp no_memory no_constraint no_moe low_beta large_memory high_T_free"

SEEDS="42 137 256"

DATA_DIR="data/arithmetic"

OUTPUT_DIR="results/experiment_001"



# Generate data (if not exists)

if [ ! -f "$DATA_DIR/train.jsonl" ]; then

    python scripts/generate_arithmetic.py --output_dir $DATA_DIR --seed 42

fi



# Run experiments

for config in $CONFIGS; do

    for seed in $SEEDS; do

        echo "Running config=$config seed=$seed"

        python scripts/run_training.py \

            --config configs/experiment_001/${config}.yaml \

            --seed $seed \

            --data_dir $DATA_DIR \

            --output_dir $OUTPUT_DIR/${config}/seed_${seed} \

            --epochs 50 \

            --eval_every 1 \

            --checkpoint_every 10

    done

done



# Generate report

python scripts/analyze_results.py \

    --results_dir $OUTPUT_DIR \

    --output results/experiment_001_report.json

```

---

## 10. Analysis Plan

### 10.1 Statistical Tests

1. **Paired t-test:** Compare each config against `full_mmep` on final accuracy (3 seeds each)
2. **Bonferroni correction:** Adjust p-values for 7 comparisons (alpha = 0.05/7 = 0.0071)
3. **Effect size:** Report Cohen's d for each comparison
4. **Confidence intervals:** 95% CI on accuracy difference

### 10.2 Visualization

- Learning curves (accuracy vs. epoch) for all configs
- Energy trajectories (EP configs only)
- Expert specialization heatmap (operation type vs. expert)
- Spectral norm evolution over training
- Entropy evolution over training

### 10.3 Report Format

```json

{

    "experiment": "001_mmep_arithmetic_ablation",

    "date": "YYYY-MM-DD",

    "hardware": "RTX 3080, 10GB",

    "results": {

        "full_mmep": {"accuracy": [s1, s2, s3], "mean": X, "std": Y},

        "standard_bp": {"accuracy": [s1, s2, s3], "mean": X, "std": Y},

        ...

    },

    "falsification": {

        "F1": {"triggered": false, "p_value": 0.XX, "effect_size": X.XX},

        ...

    },

    "conclusion": "Theory supported|falsified for criterion FN"

}

```

---

## 11. Reproducibility Checklist

- [ ] All random seeds documented and fixed
- [ ] Data generation script deterministic
- [ ] Model initialization deterministic
- [ ] CUDA deterministic mode enabled (`torch.backends.cudnn.deterministic = True`)
- [ ] All hyperparameters documented in config YAML
- [ ] PyTorch version recorded
- [ ] CUDA version recorded
- [ ] GPU model recorded
- [ ] Training logs saved (loss per step)
- [ ] Checkpoints saved at epoch 10, 20, 30, 40, 50
- [ ] Environment frozen (`pip freeze > requirements_experiment_001.txt`)