Text Generation
Transformers
Safetensors
PyTorch
ONNX
English
qtensorformer
q-tensorformer
high-throughput-inference
tensor-train
tensor-networks
tensorrt-llm
vllm
continuous-batching
paged-attention
speculative-decoding
zero-svd
adaptive-computation
dynamic-neural-networks
kv-cache
kv-cache-quantization
attention-sinks
lyapunov-stability
control-theory
contextual-bandits
edge-ai
edge-inference
on-device-ai
model-compression
hopper-h100
blackwell-b200
jetson-orin
jetson-thor
grouped-query-attention
rotary-position-embeddings
custom_code
academic-paper
ieee
neurips
vit-ap-university
deepseek
deepseek-r1
reasoning
chain-of-thought
gsm8k
openai-compatible
fastapi
rest-api
sse-streaming
onnx-runtime
embedded-ai
hardware-telemetry
power-efficiency
green-ai
raspberry-pi
smollm2
qwen2.5
llama-3.2
apple-silicon
Instructions to use Premchan369/Q-TensorFormer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Premchan369/Q-TensorFormer with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Premchan369/Q-TensorFormer", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("Premchan369/Q-TensorFormer", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Premchan369/Q-TensorFormer with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Premchan369/Q-TensorFormer" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Premchan369/Q-TensorFormer", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/Premchan369/Q-TensorFormer
- SGLang
How to use Premchan369/Q-TensorFormer with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Premchan369/Q-TensorFormer" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Premchan369/Q-TensorFormer", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Premchan369/Q-TensorFormer" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Premchan369/Q-TensorFormer", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use Premchan369/Q-TensorFormer with Docker Model Runner:
docker model run hf.co/Premchan369/Q-TensorFormer
Premchandyadav369
feat(systems): add NVIDIA TRT-LLM plugin, vLLM continuous batching adapter, multi-architecture roofline model, test suite, updated academic papers, and systems research dossier
383a1cd Download tests/test_systems_and_serving.py from Premchan369/Q-TensorFormer: direct link, hf CLI and curl.
- Browser
- Download file 7.97 kB
-
https://huggingface.co/Premchan369/Q-TensorFormer/resolve/main/tests/test_systems_and_serving.py
- Command line
-
hf download hf://Premchan369/Q-TensorFormer/tests/test_systems_and_serving.py
-
curl -L -o test_systems_and_serving.py https://huggingface.co/Premchan369/Q-TensorFormer/resolve/main/tests/test_systems_and_serving.py
7.97 kB
| """ | |
| tests/test_systems_and_serving.py | |
| Comprehensive test suite verifying Enterprise Systems & Serving Modules: | |
| 1. vLLM Continuous Batching and PagedAttention Adapter (src/vllm_adapter.py) | |
| 2. NVIDIA TensorRT-LLM Custom Plugin (src/trt_llm_plugin.py) | |
| 3. Multi-Architecture Hardware Roofline Analyzer (src/roofline_analyzer.py) | |
| """ | |
| import sys | |
| from pathlib import Path | |
| # Ensure repo root is on sys.path | |
| REPO_ROOT = Path(__file__).resolve().parent.parent | |
| if str(REPO_ROOT) not in sys.path: | |
| sys.path.insert(0, str(REPO_ROOT)) | |
| import pytest | |
| import torch | |
| import torch.nn as nn | |
| from src.config import ModelConfig | |
| from src.models import QTensorFormer | |
| from src.vllm_adapter import VLLMContinuousBatchingAdapter, QTensorFormerPagedAttentionMetadata | |
| from src.trt_llm_plugin import QTensorFormerTRTPlugin | |
| from src.roofline_analyzer import EnterpriseRooflineAnalyzer, DecodingRooflineProfile, HardwareRooflineSpec | |
| def tiny_qtf_model(): | |
| cfg = ModelConfig( | |
| d_model=64, | |
| n_heads=2, | |
| n_layers=2, | |
| ff_multiplier=2, | |
| max_seq_len=64, | |
| vocab_size=256, | |
| tt_rank=4, | |
| tt_min_rank=1, | |
| use_quantum=False, | |
| ) | |
| model = QTensorFormer(cfg) | |
| model.eval() | |
| return model | |
| # ========================================================================= | |
| # 1. vLLM Continuous Batching & PagedAttention Tests | |
| # ========================================================================= | |
| def test_vllm_paged_attention_metadata_creation(): | |
| """Validates creation and tensor properties of PagedAttention metadata.""" | |
| metadata = QTensorFormerPagedAttentionMetadata.create_dummy_metadata( | |
| batch_size=4, | |
| max_seq_len=32, | |
| block_size=16, | |
| device="cpu", | |
| ) | |
| assert metadata.block_tables.shape == (4, 2) | |
| assert metadata.slot_mapping.shape == (4 * 32,) | |
| assert metadata.context_lens.shape == (4,) | |
| assert metadata.active_ranks.shape == (4,) | |
| assert metadata.max_context_len == 32 | |
| def test_vllm_rank_scheduling_thermal_and_entropy(tiny_qtf_model): | |
| """Verifies dynamic rank scheduling across varied token entropies and thermal states.""" | |
| adapter = VLLMContinuousBatchingAdapter(tiny_qtf_model) | |
| # High, medium, low entropy tokens | |
| entropies = torch.tensor([3.2, 1.8, 0.4]) | |
| # Case 1: Normal thermal headroom (> 5.0 C margin) | |
| ranks_normal = adapter.schedule_batch_ranks( | |
| seq_entropies=entropies, | |
| hardware_thermal_margin_c=12.5, | |
| target_latency_budget_ms=25.0, | |
| ) | |
| assert ranks_normal == [16, 8, 4] | |
| # Case 2: Severe thermal throttling (< 5.0 C margin) | |
| ranks_throttled = adapter.schedule_batch_ranks( | |
| seq_entropies=entropies, | |
| hardware_thermal_margin_c=3.2, | |
| target_latency_budget_ms=25.0, | |
| ) | |
| assert ranks_throttled == [2, 2, 2] | |
| def test_vllm_paged_forward_execution(tiny_qtf_model): | |
| """Verifies that execute_paged_forward produces correct logits shapes without regression.""" | |
| adapter = VLLMContinuousBatchingAdapter(tiny_qtf_model) | |
| metadata = QTensorFormerPagedAttentionMetadata.create_dummy_metadata( | |
| batch_size=2, | |
| max_seq_len=16, | |
| block_size=8, | |
| ) | |
| input_ids = torch.randint(0, 256, (2, 16)) | |
| positions = torch.arange(16).unsqueeze(0).expand(2, -1) | |
| logits = adapter.execute_paged_forward( | |
| input_ids=input_ids, | |
| positions=positions, | |
| metadata=metadata, | |
| selected_ranks=[4, 8], | |
| ) | |
| assert logits.shape == (2, 16, 256) | |
| assert not torch.isnan(logits).any() | |
| # ========================================================================= | |
| # 2. NVIDIA TensorRT-LLM Custom Plugin Tests | |
| # ========================================================================= | |
| def test_trt_llm_plugin_initialization_and_metadata(): | |
| """Validates IPluginV2DynamicExt contracts and static characteristics.""" | |
| plugin = QTensorFormerTRTPlugin( | |
| in_features=64, | |
| out_features=64, | |
| max_tt_rank=8, | |
| in_modes=[4, 4, 4], | |
| out_modes=[4, 4, 4], | |
| dtype=torch.float32, | |
| ) | |
| assert plugin.get_plugin_type() == "QTensorFormerPlugin_TRT" | |
| assert plugin.get_plugin_version() == "2.1.0" | |
| # Zero-scratchpad in SMEM contract | |
| assert plugin.get_workspace_size(max_batch_size=8, max_seq_len=128) == 0 | |
| def test_trt_llm_plugin_smem_sizing(): | |
| """Verifies that SMEM allocation stays well below the 228 KB Hopper limit.""" | |
| plugin = QTensorFormerTRTPlugin( | |
| in_features=64, | |
| out_features=64, | |
| max_tt_rank=8, | |
| in_modes=[4, 4, 4], | |
| out_modes=[4, 4, 4], | |
| ) | |
| smem_r1 = plugin.compute_smem_requirement_bytes(active_rank=1) | |
| smem_r8 = plugin.compute_smem_requirement_bytes(active_rank=8) | |
| assert smem_r1 > 0 | |
| assert smem_r8 >= smem_r1 | |
| assert smem_r8 <= 65536 # Within hardware SM limits | |
| def test_trt_llm_plugin_dynamic_slicing_and_forward(): | |
| """Verifies that forward pass runs correctly across active rank configurations.""" | |
| plugin = QTensorFormerTRTPlugin( | |
| in_features=64, | |
| out_features=64, | |
| max_tt_rank=8, | |
| in_modes=[4, 4, 4], | |
| out_modes=[4, 4, 4], | |
| dtype=torch.float32, | |
| ) | |
| x = torch.randn(2, 10, 64) | |
| # Test full rank | |
| out_full = plugin(x, active_rank=8) | |
| assert out_full.shape == (2, 10, 64) | |
| assert not torch.isnan(out_full).any() | |
| # Test sliced sub-rank | |
| out_sliced = plugin(x, active_rank=2) | |
| assert out_sliced.shape == (2, 10, 64) | |
| assert not torch.isnan(out_sliced).any() | |
| # Core slicing checks | |
| c1, c2, c3 = plugin.slice_cores(active_rank=2) | |
| assert c1.shape == (1, 4, 4, 2) | |
| assert c2.shape == (2, 4, 4, 2) | |
| assert c3.shape == (2, 4, 4, 1) | |
| # ========================================================================= | |
| # 3. Enterprise Roofline Analyzer Tests | |
| # ========================================================================= | |
| def test_roofline_analyzer_initialization_all_targets(): | |
| """Verifies hardware specs for Hopper, Blackwell, Thor, Orin, and Raspberry Pi 5.""" | |
| targets = [ | |
| "blackwell_b200", | |
| "hopper_h100", | |
| "ampere_a100", | |
| "jetson_thor", | |
| "jetson_orin_nano", | |
| "raspberry_pi_5", | |
| ] | |
| for key in targets: | |
| analyzer = EnterpriseRooflineAnalyzer(key) | |
| assert analyzer.spec.peak_bandwidth_gb_s > 0 | |
| assert analyzer.spec.peak_tensor_tflops > 0 | |
| assert analyzer.spec.ridge_point > 0 | |
| def test_roofline_analyzer_invalid_key(): | |
| """Ensures unrecognized target raises KeyError with informative message.""" | |
| with pytest.raises(KeyError): | |
| EnterpriseRooflineAnalyzer("unknown_supercomputer") | |
| def test_roofline_token_decoding_simulation(): | |
| """Validates operational intensity and speedup calculations.""" | |
| analyzer = EnterpriseRooflineAnalyzer("hopper_h100") | |
| profile = analyzer.analyze_token_decoding( | |
| batch_size=1, | |
| seq_len=512, | |
| active_rank=4, | |
| d_model=512, | |
| n_layers=8, | |
| ) | |
| assert profile.hardware_name == "NVIDIA Hopper H100 (SXM5)" | |
| assert profile.speedup_ratio >= 1.0 | |
| assert profile.dram_traffic_reduction_pct > 80.0 | |
| assert profile.qtf_latency_ms_per_token > 0.0 | |
| assert profile.dense_latency_ms_per_token > profile.qtf_latency_ms_per_token | |
| assert "DRAM Bandwidth Limited" in profile.dense_regime | |
| def test_roofline_compare_all_and_markdown_table(): | |
| """Validates comparison across all targets and confirms zero em dashes in output.""" | |
| comparison = EnterpriseRooflineAnalyzer.compare_all_hardware(batch_size=1, seq_len=512, active_rank=4) | |
| assert len(comparison) == 6 | |
| for key, p in comparison.items(): | |
| assert p.speedup_ratio > 1.0 | |
| table_md = EnterpriseRooflineAnalyzer.generate_roofline_summary_table() | |
| assert "| NVIDIA Hopper H100" in table_md | |
| assert "| NVIDIA Blackwell B200" in table_md | |
| assert "—" not in table_md, "Found forbidden em dash in roofline summary table" | |