Text Generation
Transformers
Safetensors
PyTorch
ONNX
English
qtensorformer
qtf
q-tensorformer
qtensor
high-throughput-inference
tensor-train
tensor-networks
tensorrt-llm
vllm
continuous-batching
paged-attention
speculative-decoding
zero-svd
adaptive-computation
dynamic-neural-networks
kv-cache
kv-cache-quantization
attention-sinks
lyapunov-stability
control-theory
contextual-bandits
edge-ai
edge-inference
on-device-ai
model-compression
hopper-h100
blackwell-b200
jetson-orin
jetson-thor
grouped-query-attention
rotary-position-embeddings
custom_code
academic-paper
ieee
neurips
vit-ap-university
deepseek
deepseek-r1
reasoning
chain-of-thought
gsm8k
openai-compatible
fastapi
rest-api
sse-streaming
onnx-runtime
embedded-ai
hardware-telemetry
power-efficiency
green-ai
raspberry-pi
smollm2
qwen2.5
llama-3.2
apple-silicon
Instructions to use Premchan369/Q-TensorFormer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Premchan369/Q-TensorFormer with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Premchan369/Q-TensorFormer", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("Premchan369/Q-TensorFormer", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Premchan369/Q-TensorFormer with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Premchan369/Q-TensorFormer" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Premchan369/Q-TensorFormer", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/Premchan369/Q-TensorFormer
- SGLang
How to use Premchan369/Q-TensorFormer with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Premchan369/Q-TensorFormer" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Premchan369/Q-TensorFormer", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Premchan369/Q-TensorFormer" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Premchan369/Q-TensorFormer", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use Premchan369/Q-TensorFormer with Docker Model Runner:
docker model run hf.co/Premchan369/Q-TensorFormer
Premchandyadav369
feat(seo): maximize discoverability, Google Scholar metadata, interactive HF widget, LLM crawler schemas, zero em dashes, and one-click quickstart downloads
5175adb Download site/edge-inference.html from Premchan369/Q-TensorFormer: direct link, hf CLI and curl.
- Browser
- Download file 6.79 kB
-
https://huggingface.co/Premchan369/Q-TensorFormer/resolve/main/site/edge-inference.html
- Command line
-
hf download hf://Premchan369/Q-TensorFormer/site/edge-inference.html
-
curl -L -o edge-inference.html https://huggingface.co/Premchan369/Q-TensorFormer/resolve/main/site/edge-inference.html
6.79 kB
| <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> | |
| <title>Edge Inference, Cold-Start & Roofline Model - Q-TensorFormer</title> | |
| <meta name="description" content="Technical analysis of cold-start model loading, edge memory bandwidth bottlenecks, and the hardware roofline model for Q-TensorFormer."> | |
| <link rel="canonical" href="https://premchan369.github.io/Q-TensorFormer/edge-inference.html"> | |
| <meta property="og:type" content="article"> | |
| <meta property="og:site_name" content="Q-TensorFormer Research Project"> | |
| <meta property="og:title" content="Edge Inference, Cold-Start & Roofline Model - Q-TensorFormer"> | |
| <meta property="og:description" content="Why model loading latency and memory bandwidth dominate edge AI deployment."> | |
| <meta property="og:url" content="https://premchan369.github.io/Q-TensorFormer/edge-inference.html"> | |
| <script type="application/ld+json"> | |
| { | |
| "@context": "https://schema.org", | |
| "@type": "TechArticle", | |
| "headline": "Q-TensorFormer Edge Inference, Cold-Start Dynamics, and Roofline Analysis", | |
| "description": "Analysis of Time-To-First-Token, DRAM memory bandwidth limits, and roofline regimes on embedded processors.", | |
| "url": "https://premchan369.github.io/Q-TensorFormer/edge-inference.html", | |
| "author": { "@type": "Person", "name": "Premchand Yadav" }, | |
| "datePublished": "2026-10-13" | |
| } | |
| </script> | |
| <link rel="stylesheet" href="assets/style.css"> | |
| </head> | |
| <body> | |
| <header> | |
| <div class="header-container"> | |
| <div class="logo"> | |
| <a href="index.html"><span>Q-TensorFormer</span> Research</a> | |
| </div> | |
| <nav> | |
| <a href="index.html">Overview</a> | |
| <a href="architecture.html">Architecture</a> | |
| <a href="methodology.html">Methodology</a> | |
| <a href="benchmarks.html">Benchmarks</a> | |
| <a href="edge-inference.html" class="active">Edge Inference</a> | |
| <a href="training.html">Training</a> | |
| <a href="documentation.html">Docs & API</a> | |
| <a href="research.html">Research</a> | |
| <a href="limitations.html">Limitations</a> | |
| <a href="contributing.html">Contributing</a> | |
| </nav> | |
| </div> | |
| </header> | |
| <main class="container"> | |
| <h1>Edge Inference, Cold-Start & Roofline Model</h1> | |
| <p class="lead"> | |
| Edge AI deployment is fundamentally constrained by memory capacity and memory bandwidth, not raw ALU computation. Q-TensorFormer investigates how Tensor-Train compression and KV-cache pruning impact cold-start latency and operational intensity. | |
| </p> | |
| <section> | |
| <h2>1. The Cold-Start Loading Bottleneck</h2> | |
| <p> | |
| On resource-constrained platforms (such as point-of-care medical scanners, drones, and mobile devices), models cannot remain permanently resident in fast SRAM or VRAM. Instead, models must be loaded on-demand from slower flash storage or paged across PCIe buses. | |
| </p> | |
| <div class="callout"> | |
| <p><strong>Author Motivation:</strong> This research was directly inspired by the author's real-world developer experience deploying a 1.5-billion-parameter medical model (<strong>MedGemma 1.5B</strong>) on a memory-constrained machine, where cold-starting the model into memory required <strong>~200 seconds</strong> before generating the first token. <em>(Note: Cited strictly as motivating developer context, not as an in-repo benchmark.)</em></p> | |
| </div> | |
| <p> | |
| By factorizing dense FFN weights into Tensor-Train cores, checkpoint byte volume decreases by <strong>30.6% to 49.3%</strong>, directly reducing the physical bytes that must be read during cold-start deserialization. | |
| </p> | |
| </section> | |
| <section> | |
| <h2>2. Hardware Roofline Model Analysis</h2> | |
| <p> | |
| The hardware roofline model characterizes whether an execution kernel is bounded by memory bandwidth or compute throughput based on its <strong>arithmetic intensity</strong> (FLOPs per byte transferred from DRAM): | |
| </p> | |
| <div class="grid"> | |
| <div class="card"> | |
| <h3>Dense Linear (Compute-Bound)</h3> | |
| <p><strong>Arithmetic Intensity:</strong> ~250.0 FLOP/byte</p> | |
| <p>Standard dense GEMM matrices reuse weights extensively across batch tokens, achieving high ALU utilization on server GPUs.</p> | |
| </div> | |
| <div class="card"> | |
| <h3>Tensor-Train Linear (Memory-Bound)</h3> | |
| <p><strong>Arithmetic Intensity:</strong> ~0.20 FLOP/byte</p> | |
| <p>Contracting multi-core tensor views executes fewer total parameters but transfers intermediate tensors across memory caches.</p> | |
| </div> | |
| </div> | |
| <div class="callout warning"> | |
| <p><strong>Crucial Architectural Insight:</strong> Tensor-Train compression is not an automatic speedup on server GPUs with plentiful VRAM. TT compression is an <em>edge deployment enabler</em> for regimes where a dense model would otherwise fail to fit in memory or cause disk thrashing.</p> | |
| </div> | |
| </section> | |
| <section> | |
| <h2>3. Honest Energy Measurement Disclosures</h2> | |
| <p> | |
| In accordance with rigorous scientific integrity standards, energy reporting is structured into four distinct levels (<code>src/hardware_cost_model.py</code>): | |
| </p> | |
| <ul> | |
| <li><strong>Level 1 (Analytical):</strong> Theoretical energy derived from FLOP count multiplied by nominal operational intensity (<code>ESTIMATED</code>).</li> | |
| <li><strong>Level 2 (Memory-Traffic):</strong> Energy modeled from simulated DRAM byte transfers multiplied by physical capacitor access cost (~15 pJ/bit, <code>ESTIMATED</code>).</li> | |
| <li><strong>Level 3 (Wall-Clock TDP):</strong> Measured real-time execution duration multiplied by processor nominal Thermal Design Power (<code>ESTIMATED</code>).</li> | |
| <li><strong>Level 4 (Physical Power Sensor):</strong> Direct physical readings from RAPL or NVML wattmeters. <em>(Because the benchmark host is a CPU-only environment without attached inline hardware power meters, Level 4 is explicitly disclosed as <code>UNVERIFIED / N/A</code>).</em></li> | |
| </ul> | |
| </section> | |
| </main> | |
| <footer> | |
| <div class="footer-links"> | |
| <a href="index.html">Home</a> | |
| <a href="architecture.html">Architecture</a> | |
| <a href="methodology.html">Methodology</a> | |
| <a href="benchmarks.html">Benchmarks</a> | |
| <a href="documentation.html">Docs</a> | |
| <a href="limitations.html">Limitations</a> | |
| <a href="sitemap.xml">Sitemap</a> | |
| <a href="llms.txt">llms.txt</a> | |
| <a href="https://huggingface.co/Premchan369/Q-TensorFormer">Hugging Face</a> | |
| </div> | |
| <p>© 2026 Premchand Yadav. Licensed under Apache-2.0.</p> | |
| </footer> | |
| </body> | |
| </html> | |