push from SNAPKITTYWEST/burt-imma

b88c26d verified about 2 months ago

14.4 kB

# BURT: Boolean Universal Reasoning Transformer

**Project:** BURT-IMMA  
**Contact:** jessica@collectivekitty.com  
**License:** BSL-1.1

---

## 1. Overview

BURT (Boolean Universal Reasoning Transformer) is a transformer-based architecture augmented with a Boolean algebra constraint layer and trained via Matrix-Memory Equilibrium Propagation (MMEP). It combines the representational power of attention mechanisms with the logical rigor of Boolean lattice operations, enforced through Huntington postulates at the architectural level.

BURT is designed for tasks requiring provably correct logical reasoning while maintaining the flexibility and scalability of modern transformer architectures.

---

## 2. Architecture Overview

```
Input -> Embedding -> [BURT Block x N] -> Output Head -> Output
                          |
                          v
                   +-------------+
                   | Transformer  |
                   | Backbone     |
                   +------+------+
                          |
                   +------v------+
                   | Boolean      |
                   | Constraint   |
                   | Layer        |
                   +------+------+
                          |
                   +------v------+
                   | MMEP         |
                   | Training     |
                   +-------------+
```

The architecture consists of three integrated systems:

1. **Transformer Backbone:** Multi-head attention with SmoothLeaky activations and GatesNormalization
2. **Boolean Algebra Constraint Layer:** Huntington postulate enforcement on intermediate representations
3. **MMEP Training:** Equilibrium propagation with matrix memory (see [MMEP_THEORY.md](./MMEP_THEORY.md))

---

## 3. Components

### 3.1 SmoothLeaky Activation

```python
SmoothLeaky(x) = x * sigmoid(k * x)    for x >= 0
               = alpha * x * sigmoid(k * x)    for x < 0
```

**Parameters:**
- `k`: smoothness parameter (default: 1.0, learnable)
- `alpha`: negative slope (default: 0.01, fixed)

**Properties:**
- Smooth everywhere (infinitely differentiable)
- Monotonically increasing
- Bounded gradient: `|dSmoothLeaky/dx| <= 1 + alpha`
- Converges to LeakyReLU as `k -> infinity`
- Required for MMEP convergence (energy function must be smooth)

### 3.2 GatesNormalization

```python
GatesNorm(x) = (x - mu) / sqrt(sigma^2 + eps) * gamma + beta
```

where `mu` and `sigma^2` are computed over the expert dimension, ensuring that the gating distribution is well-conditioned for sparse routing.

**Key difference from LayerNorm:** GatesNormalization operates on the gating logits before softmax, ensuring that:
1. No single expert dominates (prevents collapse)
2. The L1 norm of the resulting distribution is bounded
3. The entropy constraint (0.20) is satisfiable

**Parameters:**
- `gamma`: scale (learnable, per-expert)
- `beta`: bias (learnable, per-expert)
- `eps`: numerical stability (default: 1e-5)

### 3.3 CIFGMatrixMemory

The Coupled Input-Forget Gate Matrix Memory module:

```python
class CIFGMatrixMemory:
    C_global: Tensor[d_model, d_model]    # Shared memory
    C_expert: List[Tensor[d_expert, d_expert]]    # Per-expert memory
    W_f: Tensor[d_model, 2 * d_model]    # Forget gate weights
    b_f: Tensor[d_model]    # Forget gate bias

    def forward(self, h, x, expert_id=None):
        f = sigmoid(W_f @ concat(h, x) + b_f)
        candidate = outer(h, h)  # Rank-1 update
        C_global = f * C_global + (1 - f) * candidate
        if expert_id is not None:
            C_expert[expert_id] = f * C_expert[expert_id] + (1 - f) * candidate
        return C_global @ h  # Memory retrieval
```

**Properties:**
- Coupled gates: input gate = `1 - forget gate`
- Rank-1 updates preserve invertibility
- Forget bias schedule enables consolidation
- See [MMEP_THEORY.md](./MMEP_THEORY.md) Section 5 for full dynamics

### 3.4 GatesRouter

Sparse Mixture-of-Experts router with Gates normalization:

```python
class GatesRouter:
    W_gate: Tensor[num_experts, d_model]
    
    def forward(self, x):
        logits = W_gate @ x
        logits = gates_norm(logits)
        alpha = constrained_softmax(logits, entropy_bound=0.20)
        top_k_indices = topk(alpha, k=2)
        return top_k_indices, alpha[top_k_indices]
```

**Routing strategy:**
- Top-2 expert selection (default)
- Load balancing via auxiliary loss
- Entropy bounded at 0.20 nats
- Sparsity enforced via L1 penalty on `alpha`

### 3.5 SuperpositionedInductionHeads

Multi-head attention with superposition-aware induction:

```python
class SuperpositionedInductionHeads:
    W_Q, W_K, W_V: Tensor[num_heads, d_head, d_model]
    
    def forward(self, x, memory_context):
        Q = W_Q @ x
        K = W_K @ x
        V = W_V @ x
        
        # Standard attention
        attn = softmax(Q @ K^T / sqrt(d_head))
        
        # Induction: detect repeated patterns
        induction_score = detect_copy_pattern(Q, K, offset=1)
        
        # Superposition: combine with memory retrieval
        memory_V = memory_context @ V
        output = attn @ V + induction_score * memory_V
        
        return output
```

**Properties:**
- Detects and exploits repeated subsequences (induction)
- Superpositions memory-retrieved values with attention output
- Enables in-context learning through pattern completion
- Spectral norm constraint applied to all projection matrices

### 3.6 QuantumInterferenceResolver

Resolves conflicting expert outputs through interference-inspired combination:

```python
class QuantumInterferenceResolver:
    def forward(self, expert_outputs, routing_weights):
        # Represent each expert output as amplitude
        amplitudes = [sqrt(w) * exp(1j * phase(out)) for w, out in zip(routing_weights, expert_outputs)]
        
        # Interfere: sum amplitudes, then measure (take magnitude squared)
        superposition = sum(amplitudes)
        resolved = |superposition|^2 * sign(real(superposition))
        
        return resolved
```

**Purpose:** When multiple experts produce conflicting outputs, simple weighted averaging loses information. The interference resolver preserves phase relationships between expert outputs, allowing constructive and destructive interference to naturally select the most coherent answer.

---

## 4. Input/Output Specification

### 4.1 Input

| Field | Type | Shape | Description |
|-------|------|-------|-------------|
| `input_ids` | int64 | `[batch, seq_len]` | Tokenized input |
| `attention_mask` | float32 | `[batch, seq_len]` | Padding mask |
| `memory_state` | float32 | `[batch, d_model, d_model]` | Optional: pre-loaded memory |

### 4.2 Output

| Field | Type | Shape | Description |
|-------|------|-------|-------------|
| `logits` | float32 | `[batch, seq_len, vocab_size]` | Token predictions |
| `memory_state` | float32 | `[batch, d_model, d_model]` | Updated memory |
| `routing_decisions` | float32 | `[batch, seq_len, num_experts]` | Expert routing weights |
| `entropy` | float32 | `[batch]` | Routing entropy per sample |
| `spectral_norms` | float32 | `[num_layers]` | Max singular value per layer |

---

## 5. Layer-by-Layer Description

### Layer 0: Token Embedding + Positional Encoding

```
x_0 = Embed(input_ids) + PosEncode(positions)
```
- Embedding dimension: `d_model`
- Positional encoding: Rotary (RoPE)

### Layers 1 to N: BURT Blocks

Each BURT block consists of:

```
# Sub-layer 1: Attention with memory
h_attn = SuperpositionedInductionHeads(GatesNorm(x), memory_context=C_global @ x)
x = x + h_attn  # Residual

# Sub-layer 2: Boolean constraint
h_bool = BooleanConstraintLayer(GatesNorm(x))
x = x + h_bool  # Residual

# Sub-layer 3: MoE Feed-Forward
expert_ids, weights = GatesRouter(x)
h_ff = sum(weights[k] * Expert_k(x) for k in expert_ids)
h_ff = QuantumInterferenceResolver(expert_outputs, weights)
x = x + h_ff  # Residual

# Sub-layer 4: Memory update
C_global, C_expert = CIFGMatrixMemory.update(x, input)
```

### Layer N+1: Output Head

```
logits = Linear(GatesNorm(x_N))
```
- Entropy constraint enforced on softmax(logits) during training
- Spectral normalization on output projection

---

## 6. Inference Pipeline

```
1. Tokenize input
2. Load or initialize memory state (C_global, C_expert)
3. Forward pass through all BURT blocks
4. For each block:
   a. Compute attention with induction heads
   b. Apply Boolean constraints (Huntington verification)
   c. Route through sparse MoE (top-2)
   d. Resolve expert conflicts via interference
   e. Update memory via CIFG
5. Project to vocabulary via output head
6. Apply constrained decoding (entropy <= 0.20 on output distribution)
7. Return: tokens, updated memory, diagnostics
```

**Autoregressive generation:**
```
for each new token:
    logits, memory = BURT(context, memory)
    next_token = sample(constrained_softmax(logits, entropy_bound=0.20))
    context = append(context, next_token)
```

---

## 7. Training via MMEP

BURT is trained using Matrix-Memory Equilibrium Propagation as defined in [MMEP_THEORY.md](./MMEP_THEORY.md).

### 7.1 Free Phase

The network processes the input and relaxes to equilibrium:
- All layers compute forward pass
- Memory retrieval and routing decisions settle
- Hidden states converge: `||dh/dt|| < 1e-5`
- Duration: `T_free = 20` steps

### 7.2 Nudged Phase

The output is clamped toward the target:
- Nudging strength: `beta = 0.1` (annealed)
- Only output layer receives direct target signal
- Internal representations adjust via energy minimization
- Duration: `T_nudge = 4` steps

### 7.3 Parameter Update

```
dW = (1/beta) * (h_nudged @ h_nudged^T - h_free @ h_free^T)
W <- W - lr * dW
W <- project_spectral(W, lambda_max=0.95)
alpha <- project_entropy(alpha, bound=0.20)
```

---

## 8. Huntington Postulate Enforcement

The Boolean constraint layer enforces Huntington's postulates on intermediate representations:

### 8.1 Postulates

For any two elements `a`, `b` in the representation space:

1. **Commutativity:** `a + b = b + a` and `a * b = b * a`
2. **Distributivity:** `a * (b + c) = (a * b) + (a * c)` and `a + (b * c) = (a + b) * (a + c)`
3. **Identity:** There exist `0` and `1` such that `a + 0 = a` and `a * 1 = a`
4. **Complement:** For each `a`, there exists `a'` such that `a + a' = 1` and `a * a' = 0`

### 8.2 Enforcement Mechanism

```python
class BooleanConstraintLayer:
    def forward(self, x):
        # Project to Boolean lattice
        x_bool = hard_sigmoid(x)  # Approximate Boolean
        
        # Verify Huntington postulates
        complement = 1 - x_bool
        assert_approx(x_bool + complement, 1, tol=1e-3)
        assert_approx(x_bool * complement, 0, tol=1e-3)
        
        # Soft constraint loss (added to energy)
        huntington_loss = ||x_bool + complement - 1||^2 + ||x_bool * complement||^2
        
        # Return constrained representation
        return x + (x_bool - x).detach()  # Straight-through estimator
```

### 8.3 Constraint Satisfaction During Training

The Huntington constraint is enforced as a soft penalty during early training, gradually hardened:
```
lambda_huntington(epoch) = min(1.0, epoch / warmup_epochs)
```

After warmup, violations are treated as hard constraints and projected away.

---

## 9. Memory Management (CIFG Gating)

### 9.1 Memory Lifecycle

```
Phase 1 (Memorization): C_global absorbs corpus knowledge
    - Forget gate bias: low (high plasticity)
    - Update frequency: every batch

Phase 2 (Query Training): C_expert learns task-specific patterns
    - Forget gate bias: medium
    - Update frequency: when expert is selected

Phase 3 (Convergence): All memories consolidate
    - Forget gate bias: high (freeze)
    - Update frequency: decreasing schedule
```

### 9.2 Memory Capacity

- `C_global`: `d_model^2` parameters = effective capacity of `d_model` memories
- `C_expert_k`: `d_expert^2` parameters each, `K` experts total
- Total memory: `d_model^2 + K * d_expert^2`
- For `d_model=512`, `K=4`, `d_expert=256`: 262K + 262K = 524K memory parameters

### 9.3 Memory Overflow Handling

When memory approaches capacity (determined by condition number of C):
1. Compute SVD of memory matrix
2. Truncate smallest singular values (keep top `r` rank)
3. Re-normalize to maintain spectral bound
4. Log memory utilization metric

---

## 10. Hyperparameters

| Parameter | Default | Range | Description |
|-----------|---------|-------|-------------|
| `d_model` | 512 | [256, 2048] | Model dimension |
| `num_layers` | 6 | [4, 24] | Number of BURT blocks |
| `num_heads` | 8 | [4, 32] | Attention heads |
| `num_experts` | 4 | [2, 16] | MoE experts |
| `top_k` | 2 | [1, 4] | Experts per token |
| `d_expert` | 256 | [128, 1024] | Expert dimension |
| `lambda_max` | 0.95 | [0.9, 0.99] | Spectral bound |
| `entropy_bound` | 0.20 | [0.1, 0.5] | Routing entropy limit |
| `beta` | 0.1 | [0.01, 1.0] | Nudging strength |
| `T_free` | 20 | [10, 100] | Free phase steps |
| `T_nudge` | 4 | [2, 20] | Nudged phase steps |
| `smoothleaky_k` | 1.0 | [0.1, 10.0] | Activation smoothness |

---

## 11. Computational Complexity

| Operation | Complexity | Notes |
|-----------|-----------|-------|
| Attention | O(n^2 * d) | Standard quadratic |
| Boolean constraint | O(n * d) | Element-wise |
| MoE routing | O(n * K) | K = num_experts |
| Expert forward | O(n * d_expert^2 * top_k) | Only active experts |
| Memory update | O(d^2) | Per-step CIFG |
| Memory retrieval | O(d^2) | Matrix-vector product |
| EP free phase | O(T_free * forward_pass) | Iterative |
| EP nudged phase | O(T_nudge * forward_pass) | Iterative |

**Total training cost:** Approximately `(T_free + T_nudge)` times standard backprop cost per sample.

---

## 12. References

- [MMEP_THEORY.md](./MMEP_THEORY.md) - Mathematical foundation for training
- [IMMA_TRAINING.md](./IMMA_TRAINING.md) - Training protocol
- [SUM_INVERSION_THEORY.md](./SUM_INVERSION_THEORY.md) - Memory retrieval theory
- [kernel_api.md](./kernel_api.md) - CUDA kernel implementations

	# BURT: Boolean Universal Reasoning Transformer

	Project: BURT-IMMA
	Contact: jessica@collectivekitty.com
	License: BSL-1.1

	---

	## 1. Overview

	BURT (Boolean Universal Reasoning Transformer) is a transformer-based architecture augmented with a Boolean algebra constraint layer and trained via Matrix-Memory Equilibrium Propagation (MMEP). It combines the representational power of attention mechanisms with the logical rigor of Boolean lattice operations, enforced through Huntington postulates at the architectural level.

	BURT is designed for tasks requiring provably correct logical reasoning while maintaining the flexibility and scalability of modern transformer architectures.

	---

	## 2. Architecture Overview

	```
	Input -> Embedding -> [BURT Block x N] -> Output Head -> Output
	\|
	v
	+-------------+
	\| Transformer \|
	\| Backbone \|
	+------+------+
	\|
	+------v------+
	\| Boolean \|
	\| Constraint \|
	\| Layer \|
	+------+------+
	\|
	+------v------+
	\| MMEP \|
	\| Training \|
	+-------------+
	```

	The architecture consists of three integrated systems:

	1. Transformer Backbone: Multi-head attention with SmoothLeaky activations and GatesNormalization
	2. Boolean Algebra Constraint Layer: Huntington postulate enforcement on intermediate representations
	3. MMEP Training: Equilibrium propagation with matrix memory (see [MMEP_THEORY.md](./MMEP_THEORY.md))

	---

	## 3. Components

	### 3.1 SmoothLeaky Activation

	```python
	SmoothLeaky(x) = x * sigmoid(k * x) for x >= 0
	= alpha * x * sigmoid(k * x) for x < 0
	```

	Parameters:
	- `k`: smoothness parameter (default: 1.0, learnable)
	- `alpha`: negative slope (default: 0.01, fixed)

	Properties:
	- Smooth everywhere (infinitely differentiable)
	- Monotonically increasing
	- Bounded gradient: `\|dSmoothLeaky/dx\| <= 1 + alpha`
	- Converges to LeakyReLU as `k -> infinity`
	- Required for MMEP convergence (energy function must be smooth)

	### 3.2 GatesNormalization

	```python
	GatesNorm(x) = (x - mu) / sqrt(sigma^2 + eps) * gamma + beta
	```

	where `mu` and `sigma^2` are computed over the expert dimension, ensuring that the gating distribution is well-conditioned for sparse routing.

	Key difference from LayerNorm: GatesNormalization operates on the gating logits before softmax, ensuring that:
	1. No single expert dominates (prevents collapse)
	2. The L1 norm of the resulting distribution is bounded
	3. The entropy constraint (0.20) is satisfiable

	Parameters:
	- `gamma`: scale (learnable, per-expert)
	- `beta`: bias (learnable, per-expert)
	- `eps`: numerical stability (default: 1e-5)

	### 3.3 CIFGMatrixMemory

	The Coupled Input-Forget Gate Matrix Memory module:

	```python
	class CIFGMatrixMemory:
	C_global: Tensor[d_model, d_model] # Shared memory
	C_expert: List[Tensor[d_expert, d_expert]] # Per-expert memory
	W_f: Tensor[d_model, 2 * d_model] # Forget gate weights
	b_f: Tensor[d_model] # Forget gate bias

	def forward(self, h, x, expert_id=None):
	f = sigmoid(W_f @ concat(h, x) + b_f)
	candidate = outer(h, h) # Rank-1 update
	C_global = f * C_global + (1 - f) * candidate
	if expert_id is not None:
	C_expert[expert_id] = f * C_expert[expert_id] + (1 - f) * candidate
	return C_global @ h # Memory retrieval
	```

	Properties:
	- Coupled gates: input gate = `1 - forget gate`
	- Rank-1 updates preserve invertibility
	- Forget bias schedule enables consolidation
	- See [MMEP_THEORY.md](./MMEP_THEORY.md) Section 5 for full dynamics

	### 3.4 GatesRouter

	Sparse Mixture-of-Experts router with Gates normalization:

	```python
	class GatesRouter:
	W_gate: Tensor[num_experts, d_model]

	def forward(self, x):
	logits = W_gate @ x
	logits = gates_norm(logits)
	alpha = constrained_softmax(logits, entropy_bound=0.20)
	top_k_indices = topk(alpha, k=2)
	return top_k_indices, alpha[top_k_indices]
	```

	Routing strategy:
	- Top-2 expert selection (default)
	- Load balancing via auxiliary loss
	- Entropy bounded at 0.20 nats
	- Sparsity enforced via L1 penalty on `alpha`

	### 3.5 SuperpositionedInductionHeads

	Multi-head attention with superposition-aware induction:

	```python
	class SuperpositionedInductionHeads:
	W_Q, W_K, W_V: Tensor[num_heads, d_head, d_model]

	def forward(self, x, memory_context):
	Q = W_Q @ x
	K = W_K @ x
	V = W_V @ x

	# Standard attention
	attn = softmax(Q @ K^T / sqrt(d_head))

	# Induction: detect repeated patterns
	induction_score = detect_copy_pattern(Q, K, offset=1)

	# Superposition: combine with memory retrieval
	memory_V = memory_context @ V
	output = attn @ V + induction_score * memory_V

	return output
	```

	Properties:
	- Detects and exploits repeated subsequences (induction)
	- Superpositions memory-retrieved values with attention output
	- Enables in-context learning through pattern completion
	- Spectral norm constraint applied to all projection matrices

	### 3.6 QuantumInterferenceResolver

	Resolves conflicting expert outputs through interference-inspired combination:

	```python
	class QuantumInterferenceResolver:
	def forward(self, expert_outputs, routing_weights):
	# Represent each expert output as amplitude
	amplitudes = [sqrt(w) * exp(1j * phase(out)) for w, out in zip(routing_weights, expert_outputs)]

	# Interfere: sum amplitudes, then measure (take magnitude squared)
	superposition = sum(amplitudes)
	resolved = \|superposition\|^2 * sign(real(superposition))

	return resolved
	```

	Purpose: When multiple experts produce conflicting outputs, simple weighted averaging loses information. The interference resolver preserves phase relationships between expert outputs, allowing constructive and destructive interference to naturally select the most coherent answer.

	---

	## 4. Input/Output Specification

	### 4.1 Input

	\| Field \| Type \| Shape \| Description \|
	\|-------\|------\|-------\|-------------\|
	\| `input_ids` \| int64 \| `[batch, seq_len]` \| Tokenized input \|
	\| `attention_mask` \| float32 \| `[batch, seq_len]` \| Padding mask \|
	\| `memory_state` \| float32 \| `[batch, d_model, d_model]` \| Optional: pre-loaded memory \|

	### 4.2 Output

	\| Field \| Type \| Shape \| Description \|
	\|-------\|------\|-------\|-------------\|
	\| `logits` \| float32 \| `[batch, seq_len, vocab_size]` \| Token predictions \|
	\| `memory_state` \| float32 \| `[batch, d_model, d_model]` \| Updated memory \|
	\| `routing_decisions` \| float32 \| `[batch, seq_len, num_experts]` \| Expert routing weights \|
	\| `entropy` \| float32 \| `[batch]` \| Routing entropy per sample \|
	\| `spectral_norms` \| float32 \| `[num_layers]` \| Max singular value per layer \|

	---

	## 5. Layer-by-Layer Description

	### Layer 0: Token Embedding + Positional Encoding

	```
	x_0 = Embed(input_ids) + PosEncode(positions)
	```
	- Embedding dimension: `d_model`
	- Positional encoding: Rotary (RoPE)

	### Layers 1 to N: BURT Blocks

	Each BURT block consists of:

	```
	# Sub-layer 1: Attention with memory
	h_attn = SuperpositionedInductionHeads(GatesNorm(x), memory_context=C_global @ x)
	x = x + h_attn # Residual

	# Sub-layer 2: Boolean constraint
	h_bool = BooleanConstraintLayer(GatesNorm(x))
	x = x + h_bool # Residual

	# Sub-layer 3: MoE Feed-Forward
	expert_ids, weights = GatesRouter(x)
	h_ff = sum(weights[k] * Expert_k(x) for k in expert_ids)
	h_ff = QuantumInterferenceResolver(expert_outputs, weights)
	x = x + h_ff # Residual

	# Sub-layer 4: Memory update
	C_global, C_expert = CIFGMatrixMemory.update(x, input)
	```

	### Layer N+1: Output Head

	```
	logits = Linear(GatesNorm(x_N))
	```
	- Entropy constraint enforced on softmax(logits) during training
	- Spectral normalization on output projection

	---

	## 6. Inference Pipeline

	```
	1. Tokenize input
	2. Load or initialize memory state (C_global, C_expert)
	3. Forward pass through all BURT blocks
	4. For each block:
	a. Compute attention with induction heads
	b. Apply Boolean constraints (Huntington verification)
	c. Route through sparse MoE (top-2)
	d. Resolve expert conflicts via interference
	e. Update memory via CIFG
	5. Project to vocabulary via output head
	6. Apply constrained decoding (entropy <= 0.20 on output distribution)
	7. Return: tokens, updated memory, diagnostics
	```

	Autoregressive generation:
	```
	for each new token:
	logits, memory = BURT(context, memory)
	next_token = sample(constrained_softmax(logits, entropy_bound=0.20))
	context = append(context, next_token)
	```

	---

	## 7. Training via MMEP

	BURT is trained using Matrix-Memory Equilibrium Propagation as defined in [MMEP_THEORY.md](./MMEP_THEORY.md).

	### 7.1 Free Phase

	The network processes the input and relaxes to equilibrium:
	- All layers compute forward pass
	- Memory retrieval and routing decisions settle
	- Hidden states converge: `\|\|dh/dt\|\| < 1e-5`
	- Duration: `T_free = 20` steps

	### 7.2 Nudged Phase

	The output is clamped toward the target:
	- Nudging strength: `beta = 0.1` (annealed)
	- Only output layer receives direct target signal
	- Internal representations adjust via energy minimization
	- Duration: `T_nudge = 4` steps

	### 7.3 Parameter Update

	```
	dW = (1/beta) * (h_nudged @ h_nudged^T - h_free @ h_free^T)
	W <- W - lr * dW
	W <- project_spectral(W, lambda_max=0.95)
	alpha <- project_entropy(alpha, bound=0.20)
	```

	---

	## 8. Huntington Postulate Enforcement

	The Boolean constraint layer enforces Huntington's postulates on intermediate representations:

	### 8.1 Postulates

	For any two elements `a`, `b` in the representation space:

	1. Commutativity: `a + b = b + a` and `a * b = b * a`
	2. Distributivity: `a * (b + c) = (a * b) + (a * c)` and `a + (b * c) = (a + b) * (a + c)`
	3. Identity: There exist `0` and `1` such that `a + 0 = a` and `a * 1 = a`
	4. Complement: For each `a`, there exists `a'` such that `a + a' = 1` and `a * a' = 0`

	### 8.2 Enforcement Mechanism

	```python
	class BooleanConstraintLayer:
	def forward(self, x):
	# Project to Boolean lattice
	x_bool = hard_sigmoid(x) # Approximate Boolean

	# Verify Huntington postulates
	complement = 1 - x_bool
	assert_approx(x_bool + complement, 1, tol=1e-3)
	assert_approx(x_bool * complement, 0, tol=1e-3)

	# Soft constraint loss (added to energy)
	huntington_loss = \|\|x_bool + complement - 1\|\|^2 + \|\|x_bool * complement\|\|^2

	# Return constrained representation
	return x + (x_bool - x).detach() # Straight-through estimator
	```

	### 8.3 Constraint Satisfaction During Training

	The Huntington constraint is enforced as a soft penalty during early training, gradually hardened:
	```
	lambda_huntington(epoch) = min(1.0, epoch / warmup_epochs)
	```

	After warmup, violations are treated as hard constraints and projected away.

	---

	## 9. Memory Management (CIFG Gating)

	### 9.1 Memory Lifecycle

	```
	Phase 1 (Memorization): C_global absorbs corpus knowledge
	- Forget gate bias: low (high plasticity)
	- Update frequency: every batch

	Phase 2 (Query Training): C_expert learns task-specific patterns
	- Forget gate bias: medium
	- Update frequency: when expert is selected

	Phase 3 (Convergence): All memories consolidate
	- Forget gate bias: high (freeze)
	- Update frequency: decreasing schedule
	```

	### 9.2 Memory Capacity

	- `C_global`: `d_model^2` parameters = effective capacity of `d_model` memories
	- `C_expert_k`: `d_expert^2` parameters each, `K` experts total
	- Total memory: `d_model^2 + K * d_expert^2`
	- For `d_model=512`, `K=4`, `d_expert=256`: 262K + 262K = 524K memory parameters

	### 9.3 Memory Overflow Handling

	When memory approaches capacity (determined by condition number of C):
	1. Compute SVD of memory matrix
	2. Truncate smallest singular values (keep top `r` rank)
	3. Re-normalize to maintain spectral bound
	4. Log memory utilization metric

	---

	## 10. Hyperparameters

	\| Parameter \| Default \| Range \| Description \|
	\|-----------\|---------\|-------\|-------------\|
	\| `d_model` \| 512 \| [256, 2048] \| Model dimension \|
	\| `num_layers` \| 6 \| [4, 24] \| Number of BURT blocks \|
	\| `num_heads` \| 8 \| [4, 32] \| Attention heads \|
	\| `num_experts` \| 4 \| [2, 16] \| MoE experts \|
	\| `top_k` \| 2 \| [1, 4] \| Experts per token \|
	\| `d_expert` \| 256 \| [128, 1024] \| Expert dimension \|
	\| `lambda_max` \| 0.95 \| [0.9, 0.99] \| Spectral bound \|
	\| `entropy_bound` \| 0.20 \| [0.1, 0.5] \| Routing entropy limit \|
	\| `beta` \| 0.1 \| [0.01, 1.0] \| Nudging strength \|
	\| `T_free` \| 20 \| [10, 100] \| Free phase steps \|
	\| `T_nudge` \| 4 \| [2, 20] \| Nudged phase steps \|
	\| `smoothleaky_k` \| 1.0 \| [0.1, 10.0] \| Activation smoothness \|

	---

	## 11. Computational Complexity

	\| Operation \| Complexity \| Notes \|
	\|-----------\|-----------\|-------\|
	\| Attention \| O(n^2 * d) \| Standard quadratic \|
	\| Boolean constraint \| O(n * d) \| Element-wise \|
	\| MoE routing \| O(n * K) \| K = num_experts \|
	\| Expert forward \| O(n * d_expert^2 * top_k) \| Only active experts \|
	\| Memory update \| O(d^2) \| Per-step CIFG \|
	\| Memory retrieval \| O(d^2) \| Matrix-vector product \|
	\| EP free phase \| O(T_free * forward_pass) \| Iterative \|
	\| EP nudged phase \| O(T_nudge * forward_pass) \| Iterative \|

	Total training cost: Approximately `(T_free + T_nudge)` times standard backprop cost per sample.

	---

	## 12. References

	- [MMEP_THEORY.md](./MMEP_THEORY.md) - Mathematical foundation for training
	- [IMMA_TRAINING.md](./IMMA_TRAINING.md) - Training protocol
	- [SUM_INVERSION_THEORY.md](./SUM_INVERSION_THEORY.md) - Memory retrieval theory
	- [kernel_api.md](./kernel_api.md) - CUDA kernel implementations