Architecture & System Design

Q-TensorFormer decouples static parameter overprovisioning from dynamic inference execution by wrapping classical Transformer decoder blocks with closed-loop information tracking and nested tensor factorization.

1. The 8-Dimensional Token Information State (z_t)

In standard Transformers, token representations pass through each layer without internal monitoring. Q-TensorFormer deploys a normalized 8D feature probe at the entrance of each block (implemented in src/information_state.py):

S_t: Shannon Entropy

Measures the uncertainty of the token's predictive probability distribution over the vocabulary.

H_t: Hidden State L2 Norm

Quantifies representation magnitude, tracking whether activations are saturated or decaying.

U_t: Top-2 Prediction Margin

Difference between top-1 and top-2 token probabilities; small margins indicate epistemic ambiguity.

A_t: Normalized Position

Relative token position within the active context window (t / max_seq_len).

R_t: Residual Norm Delta

Rate of change between consecutive layer representations, detecting semantic convergence.

L_t: Latency Budget Drift

Tracks accumulated real-time inference latency relative to target SLO milestones.

M_t: Memory Footprint Stress

Monitors active KV-cache allocation relative to hardware DRAM/SRAM limits.

B_t: Dynamic Batch Pressure

Concurrency index to adjust shared hardware core contention on edge processors.

2. Nested Tensor-Train FFN (Zero-SVD Rank Slicing)

Standard linear feed-forward layers store an uncompressed 2D matrix W ∈ R^{D × 4D}. Q-TensorFormer factorizes W into a train of contracted 3-way cores G^{(1)}, G^{(2)}, ..., G^{(d)} using the Tensor-Train (TT) decomposition (src/tensor_layers.py):

W_{(i1..id), (o1..od)} = sum_{alpha_1..alpha_{d-1}} G^(1)_{1, o1, i1, alpha_1} * ... * G^(d)_{alpha_{d-1}, od, id, 1}

Zero-SVD Slicing: Because the TT cores are nested, reducing the rank from 16 to 8, 4, 2, or 1 simply requires slicing the tensor view core[..., :r] without executing costly Singular Value Decomposition (SVD) at runtime. This provides instantaneous capacity switching in zero memory allocation time.

3. Zero-Copy Grouped-Query Attention (GQA)

To prevent high KV-cache memory traffic, Q-TensorFormer pairs multi-head queries (n_heads) with grouped key-value heads (n_kv_heads).

Unlike naive implementations that allocate duplicated contiguous tensors via repeat_interleave, Q-TensorFormer uses 5D zero-copy broadcasting (src/attention.py):

# Zero-copy view broadcasting without expanding K/V in memory:
q_grouped = q.view(B, n_kv_heads, num_queries_per_kv, T, head_dim)
k_grouped = k.unsqueeze(2) # Shape: (B, n_kv_heads, 1, T_k, head_dim)
scores = torch.matmul(q_grouped, k_grouped.transpose(-2, -1)) * scale

This guarantees that K and V tensors share the exact same underlying storage pointer (last_kv_shared_storage = True), completely eliminating memory bloat.

4. IEEE-754 NaN-Safe Boolean Masking

Many open-source Transformer repositories contain a silent numerical bug where attention masks multiply padding zeros by negative infinity (mask * float("-inf")). In IEEE-754 arithmetic, 0.0 * -inf = NaN!

Q-TensorFormer replaces this pattern with boolean masked_fill and guards against completely padded query rows (e.g. left-padding in mixed batches):

row_all_masked = hide_mask.all(dim=-1, keepdim=True)
safe_hide = hide_mask & ~row_all_masked
masked_scores = scores.masked_fill(safe_hide, float("-inf"))
weights = F.softmax(masked_scores, dim=-1).masked_fill(row_all_masked, 0.0)