docs/BURT_SPEC.md

BURT: Boolean Universal Reasoning Transformer

Project: BURT-IMMA
Contact: jessica@collectivekitty.com
License: BSL-1.1

---

1. Overview

BURT (Boolean Universal Reasoning Transformer) is a transformer-based architecture augmented with a Boolean algebra constraint layer and trained via Matrix-Memory Equilibrium Propagation (MMEP). It combines the representational power of attention mechanisms with the logical rigor of Boolean lattice operations, enforced through Huntington postulates at the architectural level.

BURT is designed for tasks requiring provably correct logical reasoning while maintaining the flexibility and scalability of modern transformer architectures.

---

2. Architecture Overview

Input -> Embedding -> [BURT Block x N] -> Output Head -> Output
                          |
                          v
                   +-------------+
                   | Transformer  |
                   | Backbone     |
                   +------+------+
                          |
                   +------v------+
                   | Boolean      |
                   | Constraint   |
                   | Layer        |
                   +------+------+
                          |
                   +------v------+
                   | MMEP         |
                   | Training     |
                   +-------------+

The architecture consists of three integrated systems:

1. Transformer Backbone: Multi-head attention with SmoothLeaky activations and GatesNormalization
2. Boolean Algebra Constraint Layer: Huntington postulate enforcement on intermediate representations
3. MMEP Training: Equilibrium propagation with matrix memory (see MMEP_THEORY.md)

---

3. Components

3.1 SmoothLeaky Activation

SmoothLeaky(x) = x * sigmoid(k * x)    for x >= 0
               = alpha * x * sigmoid(k * x)    for x < 0

Parameters:

• k: smoothness parameter (default: 1.0, learnable)
• alpha: negative slope (default: 0.01, fixed)

Properties:

• Smooth everywhere (infinitely differentiable)
• Monotonically increasing
• Bounded gradient: |dSmoothLeaky/dx| <= 1 + alpha
• Converges to LeakyReLU as k -> infinity
• Required for MMEP convergence (energy function must be smooth)

3.2 GatesNormalization

GatesNorm(x) = (x - mu) / sqrt(sigma^2 + eps) * gamma + beta

where mu and sigma^2 are computed over the expert dimension, ensuring that the gating distribution is well-conditioned for sparse routing.

Key difference from LayerNorm: GatesNormalization operates on the gating logits before softmax, ensuring that:

1. No single expert dominates (prevents collapse)
2. The L1 norm of the resulting distribution is bounded
3. The entropy constraint (0.20) is satisfiable

Parameters:

• gamma: scale (learnable, per-expert)
• beta: bias (learnable, per-expert)
• eps: numerical stability (default: 1e-5)

3.3 CIFGMatrixMemory

The Coupled Input-Forget Gate Matrix Memory module:

class CIFGMatrixMemory:
    C_global: Tensor[d_model, d_model]    # Shared memory
    C_expert: List[Tensor[d_expert, d_expert]]    # Per-expert memory
    W_f: Tensor[d_model, 2 * d_model]    # Forget gate weights
    b_f: Tensor[d_model]    # Forget gate bias

    def forward(self, h, x, expert_id=None):
        f = sigmoid(W_f @ concat(h, x) + b_f)
        candidate = outer(h, h)  # Rank-1 update
        C_global = f * C_global + (1 - f) * candidate
        if expert_id is not None:
            C_expert[expert_id] = f * C_expert[expert_id] + (1 - f) * candidate
        return C_global @ h  # Memory retrieval

Properties:

• Coupled gates: input gate = 1 - forget gate
• Rank-1 updates preserve invertibility
• Forget bias schedule enables consolidation
• See MMEP_THEORY.md Section 5 for full dynamics

3.4 GatesRouter

Sparse Mixture-of-Experts router with Gates normalization:

class GatesRouter:
    W_gate: Tensor[num_experts, d_model]
    
    def forward(self, x):
        logits = W_gate @ x
        logits = gates_norm(logits)
        alpha = constrained_softmax(logits, entropy_bound=0.20)
        top_k_indices = topk(alpha, k=2)
        return top_k_indices, alpha[top_k_indices]

Routing strategy:

• Top-2 expert selection (default)
• Load balancing via auxiliary loss
• Entropy bounded at 0.20 nats
• Sparsity enforced via L1 penalty on alpha

3.5 SuperpositionedInductionHeads

Multi-head attention with superposition-aware induction:

class SuperpositionedInductionHeads:
    W_Q, W_K, W_V: Tensor[num_heads, d_head, d_model]
    
    def forward(self, x, memory_context):
        Q = W_Q @ x
        K = W_K @ x
        V = W_V @ x
        
        # Standard attention
        attn = softmax(Q @ K^T / sqrt(d_head))
        
        # Induction: detect repeated patterns
        induction_score = detect_copy_pattern(Q, K, offset=1)
        
        # Superposition: combine with memory retrieval
        memory_V = memory_context @ V
        output = attn @ V + induction_score * memory_V
        
        return output

Properties:

• Detects and exploits repeated subsequences (induction)
• Superpositions memory-retrieved values with attention output
• Enables in-context learning through pattern completion
• Spectral norm constraint applied to all projection matrices

3.6 QuantumInterferenceResolver

Resolves conflicting expert outputs through interference-inspired combination:

class QuantumInterferenceResolver:
    def forward(self, expert_outputs, routing_weights):
        # Represent each expert output as amplitude
        amplitudes = [sqrt(w) * exp(1j * phase(out)) for w, out in zip(routing_weights, expert_outputs)]
        
        # Interfere: sum amplitudes, then measure (take magnitude squared)
        superposition = sum(amplitudes)
        resolved = |superposition|^2 * sign(real(superposition))
        
        return resolved

Purpose: When multiple experts produce conflicting outputs, simple weighted averaging loses information. The interference resolver preserves phase relationships between expert outputs, allowing constructive and destructive interference to naturally select the most coherent answer.

---

4. Input/Output Specification

4.1 Input

Field	Type	Shape	Description
input_ids	int64	[batch, seq_len]	Tokenized input
attention_mask	float32	[batch, seq_len]	Padding mask
memory_state	float32	[batch, d_model, d_model]	Optional: pre-loaded memory

4.2 Output

Field	Type	Shape	Description
logits	float32	[batch, seq_len, vocab_size]	Token predictions
memory_state	float32	[batch, d_model, d_model]	Updated memory
routing_decisions	float32	[batch, seq_len, num_experts]	Expert routing weights
entropy	float32	[batch]	Routing entropy per sample
spectral_norms	float32	[num_layers]	Max singular value per layer

---

5. Layer-by-Layer Description

Layer 0: Token Embedding + Positional Encoding

x_0 = Embed(input_ids) + PosEncode(positions)

• Embedding dimension: d_model
• Positional encoding: Rotary (RoPE)

Layers 1 to N: BURT Blocks

Each BURT block consists of:

# Sub-layer 1: Attention with memory
h_attn = SuperpositionedInductionHeads(GatesNorm(x), memory_context=C_global @ x)
x = x + h_attn  # Residual

# Sub-layer 2: Boolean constraint
h_bool = BooleanConstraintLayer(GatesNorm(x))
x = x + h_bool  # Residual

# Sub-layer 3: MoE Feed-Forward
expert_ids, weights = GatesRouter(x)
h_ff = sum(weights[k] * Expert_k(x) for k in expert_ids)
h_ff = QuantumInterferenceResolver(expert_outputs, weights)
x = x + h_ff  # Residual

# Sub-layer 4: Memory update
C_global, C_expert = CIFGMatrixMemory.update(x, input)

Layer N+1: Output Head

logits = Linear(GatesNorm(x_N))

• Entropy constraint enforced on softmax(logits) during training
• Spectral normalization on output projection

---

6. Inference Pipeline

1. Tokenize input
2. Load or initialize memory state (C_global, C_expert)
3. Forward pass through all BURT blocks
4. For each block:
   a. Compute attention with induction heads
   b. Apply Boolean constraints (Huntington verification)
   c. Route through sparse MoE (top-2)
   d. Resolve expert conflicts via interference
   e. Update memory via CIFG
5. Project to vocabulary via output head
6. Apply constrained decoding (entropy <= 0.20 on output distribution)
7. Return: tokens, updated memory, diagnostics

Autoregressive generation:

for each new token:
    logits, memory = BURT(context, memory)
    next_token = sample(constrained_softmax(logits, entropy_bound=0.20))
    context = append(context, next_token)

---

7. Training via MMEP

BURT is trained using Matrix-Memory Equilibrium Propagation as defined in MMEP_THEORY.md.

7.1 Free Phase

The network processes the input and relaxes to equilibrium:

• All layers compute forward pass
• Memory retrieval and routing decisions settle
• Hidden states converge: ||dh/dt|| < 1e-5
• Duration: T_free = 20 steps

7.2 Nudged Phase

The output is clamped toward the target:

• Nudging strength: beta = 0.1 (annealed)
• Only output layer receives direct target signal
• Internal representations adjust via energy minimization
• Duration: T_nudge = 4 steps

7.3 Parameter Update

dW = (1/beta) * (h_nudged @ h_nudged^T - h_free @ h_free^T)
W <- W - lr * dW
W <- project_spectral(W, lambda_max=0.95)
alpha <- project_entropy(alpha, bound=0.20)

---

8. Huntington Postulate Enforcement

The Boolean constraint layer enforces Huntington's postulates on intermediate representations:

8.1 Postulates

For any two elements a, b in the representation space:

1. Commutativity: a + b = b + a and a * b = b * a
2. Distributivity: a * (b + c) = (a * b) + (a * c) and a + (b * c) = (a + b) * (a + c)
3. Identity: There exist 0 and 1 such that a + 0 = a and a * 1 = a
4. Complement: For each a, there exists a' such that a + a' = 1 and a * a' = 0

8.2 Enforcement Mechanism

class BooleanConstraintLayer:
    def forward(self, x):
        # Project to Boolean lattice
        x_bool = hard_sigmoid(x)  # Approximate Boolean
        
        # Verify Huntington postulates
        complement = 1 - x_bool
        assert_approx(x_bool + complement, 1, tol=1e-3)
        assert_approx(x_bool * complement, 0, tol=1e-3)
        
        # Soft constraint loss (added to energy)
        huntington_loss = ||x_bool + complement - 1||^2 + ||x_bool * complement||^2
        
        # Return constrained representation
        return x + (x_bool - x).detach()  # Straight-through estimator

8.3 Constraint Satisfaction During Training

The Huntington constraint is enforced as a soft penalty during early training, gradually hardened:

lambda_huntington(epoch) = min(1.0, epoch / warmup_epochs)

After warmup, violations are treated as hard constraints and projected away.

---

9. Memory Management (CIFG Gating)

9.1 Memory Lifecycle

Phase 1 (Memorization): C_global absorbs corpus knowledge
    - Forget gate bias: low (high plasticity)
    - Update frequency: every batch

Phase 2 (Query Training): C_expert learns task-specific patterns
    - Forget gate bias: medium
    - Update frequency: when expert is selected

Phase 3 (Convergence): All memories consolidate
    - Forget gate bias: high (freeze)
    - Update frequency: decreasing schedule

9.2 Memory Capacity

• C_global: d_model^2 parameters = effective capacity of d_model memories
• C_expert_k: d_expert^2 parameters each, K experts total
• Total memory: d_model^2 + K * d_expert^2
• For d_model=512, K=4, d_expert=256: 262K + 262K = 524K memory parameters

9.3 Memory Overflow Handling

When memory approaches capacity (determined by condition number of C):

1. Compute SVD of memory matrix
2. Truncate smallest singular values (keep top r rank)
3. Re-normalize to maintain spectral bound
4. Log memory utilization metric

---

10. Hyperparameters

Parameter	Default	Range	Description
d_model	512	[256, 2048]	Model dimension
num_layers	6	[4, 24]	Number of BURT blocks
num_heads	8	[4, 32]	Attention heads
num_experts	4	[2, 16]	MoE experts
top_k	2	[1, 4]	Experts per token
d_expert	256	[128, 1024]	Expert dimension
lambda_max	0.95	[0.9, 0.99]	Spectral bound
entropy_bound	0.20	[0.1, 0.5]	Routing entropy limit
beta	0.1	[0.01, 1.0]	Nudging strength
T_free	20	[10, 100]	Free phase steps
T_nudge	4	[2, 20]	Nudged phase steps
smoothleaky_k	1.0	[0.1, 10.0]	Activation smoothness

---

11. Computational Complexity

Operation	Complexity	Notes
Attention	O(n^2 * d)	Standard quadratic
Boolean constraint	O(n * d)	Element-wise
MoE routing	O(n * K)	K = num_experts
Expert forward	O(n * d_expert^2 * top_k)	Only active experts
Memory update	O(d^2)	Per-step CIFG
Memory retrieval	O(d^2)	Matrix-vector product
EP free phase	O(T_free * forward_pass)	Iterative
EP nudged phase	O(T_nudge * forward_pass)	Iterative

Total training cost: Approximately (T_free + T_nudge) times standard backprop cost per sample.

---

12. References

• MMEP_THEORY.md - Mathematical foundation for training
• IMMA_TRAINING.md - Training protocol
• SUM_INVERSION_THEORY.md - Memory retrieval theory
• kernel_api.md - CUDA kernel implementations