Edge Inference, Cold-Start & Roofline Model

Edge AI deployment is fundamentally constrained by memory capacity and memory bandwidth, not raw ALU computation. Q-TensorFormer investigates how Tensor-Train compression and KV-cache pruning impact cold-start latency and operational intensity.

1. The Cold-Start Loading Bottleneck

On resource-constrained platforms (such as point-of-care medical scanners, drones, and mobile devices), models cannot remain permanently resident in fast SRAM or VRAM. Instead, models must be loaded on-demand from slower flash storage or paged across PCIe buses.

Author Motivation: This research was directly inspired by the author's real-world developer experience deploying a 1.5-billion-parameter medical model (MedGemma 1.5B) on a memory-constrained machine, where cold-starting the model into memory required ~200 seconds before generating the first token. (Note: Cited strictly as motivating developer context, not as an in-repo benchmark.)

By factorizing dense FFN weights into Tensor-Train cores, checkpoint byte volume decreases by 30.6% to 49.3%, directly reducing the physical bytes that must be read during cold-start deserialization.

2. Hardware Roofline Model Analysis

The hardware roofline model characterizes whether an execution kernel is bounded by memory bandwidth or compute throughput based on its arithmetic intensity (FLOPs per byte transferred from DRAM):

Dense Linear (Compute-Bound)

Arithmetic Intensity: ~250.0 FLOP/byte

Standard dense GEMM matrices reuse weights extensively across batch tokens, achieving high ALU utilization on server GPUs.

Tensor-Train Linear (Memory-Bound)

Arithmetic Intensity: ~0.20 FLOP/byte

Contracting multi-core tensor views executes fewer total parameters but transfers intermediate tensors across memory caches.

Crucial Architectural Insight: Tensor-Train compression is not an automatic speedup on server GPUs with plentiful VRAM. TT compression is an edge deployment enabler for regimes where a dense model would otherwise fail to fit in memory or cause disk thrashing.

3. Honest Energy Measurement Disclosures

In accordance with rigorous scientific integrity standards, energy reporting is structured into four distinct levels (src/hardware_cost_model.py):

Level 1 (Analytical): Theoretical energy derived from FLOP count multiplied by nominal operational intensity (ESTIMATED).
Level 2 (Memory-Traffic): Energy modeled from simulated DRAM byte transfers multiplied by physical capacitor access cost (~15 pJ/bit, ESTIMATED).
Level 3 (Wall-Clock TDP): Measured real-time execution duration multiplied by processor nominal Thermal Design Power (ESTIMATED).
Level 4 (Physical Power Sensor): Direct physical readings from RAPL or NVML wattmeters. (Because the benchmark host is a CPU-only environment without attached inline hardware power meters, Level 4 is explicitly disclosed as UNVERIFIED / N/A).