docs/ARCHITECTURE_AUDIT.md

# Comprehensive Architecture & Empirical Audit: Q-TensorFormer

**Repository**: `Premchan369/Q-TensorFormer`  
**Date of Audit**: September 2026  
**Auditor**: Antigravity (Google DeepMind Agentic Pair Programmer)  
**Objective**: Complete diagnostic of codebase components, mathematical formulations, implementation artifacts, empirical claims, and scientific rigor prior to architectural transformation.

---

## Executive Summary

Q-TensorFormer was conceived as a hybrid architecture combining Tensor-Train (TT) weight compression, quantum circuit embeddings (PennyLane), attention-entropy-driven rank scheduling, and hardware energy estimates.

While the conceptual vision is rich, the current repository exhibits severe scientific, architectural, and empirical discrepancies:
1. **Broken / Mocked Quantum Pathways**: The "Selective Quantum Router" in `src/router.py` does not execute quantum circuits; it routes tokens through a `GELU` + dynamically allocated linear projection (`# TODO: actual quantum circuit call goes here`). Similarly, `HybridQAttention` in `src/attention.py` uses standard classical dot-product (`# Quantum kernel attention (simplified: still dot-product with noise)`).
2. **Severe Perplexity Discrepancy**: While `README.md` reports WikiText-2 perplexity of ~68–72 vs dense baseline ~65, the empirical data recorded in `benchmark_results.json` records **430.96** for the baseline and **6,186.49** for Q-TensorFormer.
3. **Wall-Clock Latency Inversion**: Theoretical FLOP reductions are negated by runtime overhead: CPU latency is **+104.8% slower** (13.52 ms vs 6.60 ms in `benchmark_results.json`), primarily from tensor reshaping/einsum overhead and lack of optimized kernel paths.
4. **Purely Theoretical Energy Claims**: Energy numbers (e.g., "18–73% less energy") are calculated via an analytical linear multiplier ($2 \times \text{params} \times B \times T \times \varepsilon_{\text{HW}}$) rather than measured power or hardware counters.
5. **Absence of KV Cache & IO Awareness**: Modern LLM inference bottlenecks are memory-bandwidth and KV-cache bounded. The codebase has **zero** KV-cache compression, eviction, quantization, or memory-traffic profiling.
6. **No Hugging Face `transformers` Standard Integration**: The codebase lacks `PreTrainedModel`, `PretrainedConfig`, and `AutoModel` compatibility, relying instead on custom dict-based configs and a bespoke character tokenizer.

---

## 1. Inventory & Mapping of All Model Components

| Component | Source File | Mathematical Formulation | Actual Implementation Status |
| :--- | :--- | :--- | :--- |
| **TTLinear** | `src/tensor_layers.py`<br>`q_tensor_former_v2.py` | $W = G^{(1)} \cdots G^{(d)}$, $G^{(k)} \in \mathbb{R}^{r_{k-1} \times o_k \times i_k \times r_k}$ | Implemented via iterative `torch.bmm` / `torch.einsum`. Includes SVD rank truncation in `set_rank()`. |
| **TTFeedForward** | `src/tensor_layers.py` | $\text{Down}(\text{GELU}(\text{Up}(x)))$ with TT cores | Functional; up/down projections both decomposed into TT cores. |
| **Quantum Angle Embedding** | `src/quantum_layers.py` | $R_x(x) \to [R_y(\theta) \to \text{CNOT}] \to \langle Z \rangle$ | Functional PennyLane QNode using `default.qubit` and `TorchLayer`. Fails gracefully with classical fallback if PennyLane is absent. |
| **Entanglement Monitor** | `src/quantum_layers.py` | $S = -\text{Tr}(\rho \log \rho)$ proxied by $-\sum A \log A$ | Classical attention entropy across sequence dimension; averaged over heads. |
| **Quantum Router** | `src/router.py` | $h_t > \theta \implies \text{Quantum}$, else Classical | **Incomplete / Mock**: Gate is learned, but selected tokens pass through `F.gelu` and a lazy `nn.Linear`, not the quantum circuit. |
| **Hybrid Attention** | `src/attention.py` | $K(q,k) = \|\langle \phi(q)\|\phi(k)\rangle\|^2$ | **Mocked**: Computes standard scaled dot-product attention; fallback logic switches boolean flag. |
| **Rank Scheduler** | `src/scheduler.py` | $r_t = r_{\min} + \alpha \cdot \overline{S} \cdot (r_{\max} - r_{\min})$ | Implemented with EMA smoothing; in practice saturates at a single rank across sequence. |
| **DARUAN / QKAN** | `src/qkan.py` | $\sum W^{(r+1)} \sigma(w_r x + b_r)$ | Classical element-wise polynomial/activation emulator; functional. |
| **Energy Estimator V4** | `src/energy_v4.py` | $E = \text{FLOPs} \times \varepsilon_{\text{HW}} + M \times \mu_{\text{BW}}$ | Strictly analytical proxy based on parameter count. No hardware profiling hooks (RAPL / NVML / ROCm). |
| **Data & Tokenizer** | `src/data.py` | Character-level vocabulary dictionary | Toy character-level tokenizer; cannot tokenize subwords or evaluate standard benchmarks (MMLU, GSM8K). |
| **HF Integration** | N/A | HF `transformers` API | **Missing**: No `PretrainedConfig`, `PreTrainedModel`, or `AutoModel` registration. |

---

## 2. Detailed Mathematical & Algorithmic Audit

### 2.1 Tensor-Train Contraction & Rank Truncation
- **Contraction**:
  $$\mathbf{y} = \mathbf{x} \times_1 G^{(1)} \times_2 G^{(2)} \cdots \times_d G^{(d)}$$
  In `TTLinear.forward()`, the contraction is executed with sequential `torch.bmm` and `torch.einsum`. While mathematically correct, sequential contraction introduces Python loop overhead and non-contiguous memory allocations for intermediate tensors, contributing directly to the observed +104% latency penalty.
- **SVD Rank Truncation**:
  `TTLinear.set_rank()` contracts adjacent core pairs into $(r_k \cdot o_k \cdot i_k, r_{mid})$ and $(r_{mid}, o_{k+1} \cdot i_{k+1} \cdot r_{k+2})$, performs SVD via `torch.linalg.svd(..., full_matrices=False)`, and truncates to `new_rank`.
  *Critique*: SVD truncation at runtime is computationally expensive ($O(D^3)$) and blocks GPU execution streams. Dynamic rank switching must not perform online SVDs on the fly during decode; instead, nested core slices or pre-projected rank bases must be used.

### 2.2 Entropy Formulation & Rank Mapping
- **Entropy Formulation**:
  $$H_t = -\sum_{j=1}^T A_{i,j} \ln (A_{i,j} + \epsilon)$$
  The codebase claims this measures "Von Neumann Entanglement Entropy". In reality, this is classical Shannon entropy of the attention distribution. While attention dispersion correlates with syntactic branching or semantic ambiguity in some contexts, it does NOT represent quantum entanglement unless density matrices are explicitly reconstructed from quantum states.
- **Rank Policy Saturation**:
  $$r = \text{round}\left(\text{EMA}\left(r_{\min} + \alpha \cdot \frac{H}{\ln T} (r_{\max} - r_{\min})\right)\right)$$
  Because attention entropy is averaged globally across heads and sequence length, token-level specificity is discarded in `HybridBlock.forward()` (line 104-106):
  ```python
  mean_entropy = entropy.mean() if entropy.dim() > 0 else entropy
  new_rank = self.rank_scheduler(mean_entropy, seq_len=x.shape[1])
  self.tt_ffn.set_rank(new_rank)
  ```
  The entire batch and layer is forced to a single global rank. As confirmed in `benchmark_results.json`, 19 out of 20 tokens collapsed to rank 3.

### 2.3 Quantum Router & Circuit Simulation
- In `src/router.py`:
  ```python
  # Gate decision
  gate_logits = self.gate_proj(x).squeeze(-1)
  soft_mask = torch.sigmoid(gate_logits / self.temperature)
  hard_mask = (soft_mask > 0.5).float()
  mask = hard_mask.detach() + soft_mask - soft_mask.detach()
  q_input = self.q_proj(x)
  # TODO: actual quantum circuit call goes here
  quantum_out = F.gelu(q_input)
  if not hasattr(self, '_q_out_proj'):
      self._q_out_proj = nn.Linear(self.q_input_dim, d_model).to(x.device)
  quantum_out = self._q_out_proj(quantum_out)
  ```
  *Bugs & Weaknesses*:
  1. No quantum circuit is called despite claims in README.md.
  2. `self._q_out_proj` is instantiated dynamically in the forward pass rather than in `__init__`, breaking PyTorch module registration, parameter discovery, optimizer binding, and state dict serialization.

---

## 3. Measured vs. Estimated vs. Simulated vs. Projected Results

| Metric / Claim | Model Card / README Claim | Actual Code Implementation | Audit Classification |
| :--- | :--- | :--- | :--- |
| **WikiText-2 PPL** | ~68–72 (vs Dense 65) | `benchmark_results.json`: Baseline **430.96**, Q-TF **6,186.49** | **FABRICATED / DISCREPANT** |
| **Parameter Compression** | 2.0× – 8.1× | Baseline 1.55M vs Q-TF 0.79M (1.96× for d=128) | **MEASURED** (structural param count) |
| **CPU Latency** | +104% CPU overhead noted | `benchmark_results.json`: Baseline 6.60ms vs Q-TF 13.52ms (+104.8%) | **MEASURED** (validates CPU latency penalty) |
| **Energy / Query** | 60 μJ (CPU), 95 μJ (Mobile) | $2 \cdot N_{\text{params}} \cdot B \cdot T \cdot \varepsilon_{\text{HW}}$ analytical multiplication | **ESTIMATED** (Purely theoretical formula) |
| **Carbon / Query** | 7 ng (Global Avg) | $E_{\text{kWh}} \times 400\text{g CO}_2/\text{kWh}$ multiplication | **ESTIMATED** (Proxy conversion) |
| **Quantum Path Usage** | 20% quantum routing | `benchmark_results.json`: 4 / 20 tokens routed | **SIMULATED** (via classical straight-through gate) |
| **Scale-Up Projections** | 18.6× at 89.4M (XL) | Analytical table calculation | **PROJECTED** |
| **KV Cache Compression** | Implied in research overview | Not implemented in codebase | **ABSENT** |

---

## 4. Key Bottlenecks and Bugs Identified

1. **Lazy Layer Instantiation**:
   `src/router.py` creates `_q_out_proj` inside `forward()`. Optimizer and `model.to(device)` will miss this parameter until after the first forward pass.
2. **Import Failures in Standalone Scripts**:
   `benchmark_fast.py` imports `from qtensorformer import ...` and `from qtensorformer.core.quantum_layer import ...`, expecting a containerized root path `/app` that does not exist in standard repos.
3. **Hard Dependency on PennyLane without Guaranteed Classical Surrogacy**:
   `src/models.py` and `q_tensor_former.py` fail immediately if `pennylane` is not installed, even though real production environments require 100% reliable classical fallback.
4. **Attention Entropy Flattening**:
   Entropy is computed across sequence length and then averaged over all heads and tokens to produce a single scalar, wiping out token-level heterogeneity.
5. **No Memory Traffic (IO) Measurement**:
   Memory bandwidth pressure, DRAM transactions, and cache footprints are never measured.
6. **No Prefill vs. Decode Separation**:
   Inference is treated solely as batched sequence evaluation, neglecting single-token autoregressive decode dynamics where KV-cache bandwidth dominates.

---

## 5. Architectural Transformation Blueprint

To fulfill the research mandate, Q-TensorFormer must move from disjoint heuristics to a **closed-loop, hardware-aware Information-Value Resource Allocator**:

```
INPUT TOKENS
     │
     ▼
┌─────────────────────────────────────────────────────────────┐
│ 1. TOKEN INFORMATION STATE (z_t)                            │
│    - Predictive entropy (model confidence)                  │
│    - Entanglement / Attention dispersion (semantic context)  │
│    - Epistemic uncertainty                                  │
│    - Tensor reconstruction error (loss under low rank)      │
│    - Hardware resource pressure (latency, memory, bandwidth)│
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ 2. INFORMATION-VALUE RESOURCE ALLOCATOR                     │
│    Value(a | z_t, H_hw) = ΔQ(a|x) / (ΔC_lat + λ_m ΔC_mem +  │
│                                      λ_e ΔC_energy + ...)   │
│    Action a = (TT Rank, Attn Path, Depth, KV Policy)        │
│    + Hysteresis stabilization (anti-chattering)             │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ 3. HARDWARE-AWARE COST MODEL (Empirically Calibrated)        │
│    Predicts real wall-clock latency, peak DRAM bytes,       │
│    KV traffic, and Joules/token across CPU / GPU / Edge.    │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ 4. JOINT EXECUTION ENGINE                                   │
│    ├── Weight Path: Nested TT Cores (Rank 1 / 2 / 4 / 8)    │
│    ├── Compute Path: Classical Fast-Track vs QKSAM VQC      │
│    ├── Memory Path: Dynamic KV Cache (FP16/INT8/INT4/Evict) │
│    └── Quantum Backend: Hardware / Sim / Classical Surrogate│
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
                   HARDWARE-EFFICIENT OUTPUT
```

---

## 6. Audit Conclusion

The scientific moat of Q-TensorFormer lies not in isolated "quantum" or "tensor" keywords, but in **the closed-loop economic optimization of real hardware resources (compute, memory traffic, KV footprint, energy) driven by the marginal information value of each token**. The codebase will now be restructured to realize this vision with full mathematical integrity, reproducible empirical rigor, and seamless Hugging Face ecosystem compatibility.