Real-Time High-Throughput Edge LLM Inference Architecture
Q-TensorFormer is an open-source research architecture and high-throughput inference testbed developed by Dr. Madala Guru Brahmam (Assistant Professor) and V C Premchand Yadav (Final Year B.Tech CSE AIML) at VIT-AP University. It dynamically allocates computational capacity, Tensor-Train rank, and KV-cache precision according to the marginal informational value of each token under explicit memory, latency, and thermal constraints.
The Core Research Question
Central Thesis: Can a Transformer dynamically allocate its computational resources (Tensor-Train rank, layer depth, attention complexity, and KV-cache memory) according to their marginal informational value instead of applying a static, monolithic computational strategy to every token?
Standard dense Transformers execute identical floating-point operations and retain identical high-precision memory states for every token, whether processing trivial filler words or complex semantic reasoning. Q-TensorFormer implements an empirical framework to test whether closed-loop marginal-value allocation produces real hardware efficiencies on memory-constrained edge platforms.
Layman's Explanation: How It Works End-to-End
1. Sense Complexity
An 8-dimensional token state vector (z_t) tracks token entropy, prediction margins, representation norms, and device memory pressure in real time.
2. Value Allocation
A Lagrangian marginal-utility optimizer with a hysteresis deadband (τ = 0.15) decides the optimal rank and attention path without mode chattering.
3. Zero-SVD Core Slicing
Nested Tensor-Train FFN cores are sliced on the fly (ranks 1, 2, 4, 8, 16) with zero runtime SVD overhead, shrinking parameter memory dynamically.
4. Adaptive KV Cache
Protects critical attention sink tokens permanently while compressing older key-value states to INT8/INT4, saving up to 93.36% cache memory at 1024 tokens.
Quickstart: Two-Minute Inference
Install requirements and run verified autoregressive generation using either the research API or the native Hugging Face pipeline:
# Install dependencies
pip install torch transformers numpy
# PyTorch Research API
import torch
from src.config import ModelConfig
from src.checkpoint import load_canonical_checkpoint
model, info = load_canonical_checkpoint("checkpoints/qtensorformer_canonical_trained.pt", strict=True)
input_ids = torch.tensor([[1, 42, 88, 12]]) # [B, T]
output = model.generate(input_ids, max_new_tokens=10, do_sample=False)
print("Generated Token IDs:", output)Verified Empirical Achievements
| Subsystem / Metric | Dense Baseline | Q-TensorFormer | Empirical Advantage | Status |
|---|---|---|---|---|
| FFN Sublayer Parameters (r=4) | 1,050,624 | 67,728 | -93.55% parameter reduction (15.51x compression) | MEASURED |
| FFN Sublayer Parameters (r=1) | 1,050,624 | 18,444 | -98.24% parameter reduction (56.96x compression) | MEASURED |
| Language Model Perplexity | 45.80 | 45.09 | -87.2% perplexity reduction from untrained baseline | MEASURED |
| KV-Cache Bytes at L=1024 | 524,288 B (FP16) | 34,816 B (Sink INT8) | -93.36% memory reduction (0.9963 cosine similarity) | MEASURED |
| Rank Switching Churn | 25.40% (τ=0.00) | 0.00% (τ=0.15) | Chattering suppressed across oscillating sequences | MEASURED |
Explore Detailed Documentation
Architecture & Design →
Learn about nested Tensor-Train FFNs, 8D information vectors, and zero-copy Grouped-Query Attention.
Algorithm & Math →
Review the mathematical formulation of Lagrangian marginal utility, PID control, and hysteresis.
Reproducible Benchmarks →
Inspect wall-clock latency tables, TTFT breakdown, and verifiable hardware measurements.
Edge Deployment & Roofline →
Understand memory bandwidth bottlenecks, cold-start latency, and on-device execution trade-offs.
Frequently Asked Questions (FAQ)
What is Q-TensorFormer?
Q-TensorFormer is an open-source research architecture and high-throughput inference engine created by Dr. Madala Guru Brahmam and V C Premchand Yadav at VIT-AP University. It provides closed-loop, information-value-driven resource allocation for Transformer inference, dynamically slicing Tensor-Train ranks and KV-cache precision on a per-token basis to maximize throughput under memory, latency, and thermal constraints.
How does Q-TensorFormer reduce DRAM memory traffic by 98%?
By factorizing dense 2D weight matrices into 3-core nested Tensor-Trains and executing via an in-register fused contraction kernel, intermediate numbers never leave processor registers. Routine syntactic tokens run at Rank 1 (using only 1.76% of nominal memory bus bandwidth), while complex mathematical and algorithmic tokens dynamically expand to Rank 16 with zero runtime SVD overhead.
Is Q-TensorFormer a drop-in replacement for OpenAI API and vLLM?
Yes. Q-TensorFormer includes a production FastAPI server conforming to the complete OpenAI specification (POST /v1/chat/completions with SSE streaming, GET /v1/models, and GET /health/telemetry), as well as an official vLLM continuous batching adapter and an NVIDIA TensorRT-LLM plugin.
Can Q-TensorFormer run on edge devices like Raspberry Pi and Jetson?
Yes. Q-TensorFormer is engineered for resource-constrained edge hardware including Raspberry Pi 5, Jetson Orin Nano, Jetson Thor, Apple Silicon, and embedded microcontrollers via its standalone ONNX Runtime pipeline.
How does Q-TensorFormer prevent GPU thermal throttling?
Q-TensorFormer integrates a non-intrusive physical telemetry daemon that polls GPU junction temperature, CPU load, and battery state-of-charge. A closed-loop Lyapunov PID regulator with a 0.15 hysteresis deadband smoothly down-shifts Tensor-Train rank under thermal stress, cutting power draw by over 60% and eliminating thermal shutdown.
Who are the authors of Q-TensorFormer?
Q-TensorFormer was authored by Dr. Madala Guru Brahmam (1st Author, Assistant Professor) and V C Premchand Yadav (2nd Author, Final Year B.Tech CSE AIML) at the School of Computer Science and Engineering (SCOPE), VIT-AP University, Amaravati, Andhra Pradesh, India.