DenCT's picture
|
download
raw
70.2 kB

THE ULTIMATE GUIDE FOR INFERENCE OPTIMIZATION

MODULE 1: The Intuition of Silicon – GPU Execution Mechanics

Before optimizing serving engines or writing Triton/CUDA kernels, an AI engineer must possess a mechanical sympathy for the underlying silicon. Optimizing Small Language Models (SLMs) or scaling massive architectures requires understanding that the GPU is not a generic compute engine; it is a highly specialized throughput machine that aggressively punishes branch divergence and uncoalesced memory access.

1.1 The SIMT Paradigm: Warps, Threads, and the Illusion of Parallelism

The GPU operates on the Single Instruction, Multiple Threads (SIMT) paradigm. Unlike CPU threads, which possess independent instruction pointers and context (MIMD), GPU threads are physically bound together in groups of 32, known as a Warp.

Intuition: The Marching Band

Imagine a warp as a 32-person marching band. The conductor (the Warp Scheduler) holds a single sheet of music (the instruction). All 32 musicians must play the exact same note at the exact same time.

  • The Happy Path: If the instruction is a matrix multiplication step, all 32 threads compute their specific tile of the matrix simultaneously. Throughput is 100%.
  • Thread Divergence (The Penalty): If an if/else statement exists (e.g., if (token_id == pad_token)), and 16 threads evaluate to true while 16 evaluate to false, the marching band splits. The conductor must first have the true-evaluating threads play their part while the other 16 stand completely frozen. Then, the true threads freeze while the false threads play. Execution time doubles. Compute efficiency drops by 50%. When writing custom kernels for token generation or sequence packing, branch logic must be eradicated or designed to ensure all threads within a warp take the exact same branch.

1.2 Memory Hierarchy: The True Bottleneck

The execution of transformer models—particularly during the autoregressive decode phase—is almost entirely memory-bound. The ALU (Arithmetic Logic Unit) speed is irrelevant if it is starved for data.

The L1 / Shared Memory (The Workbench)

  • Size: Extremely small (e.g., 228 KB per Streaming Multiprocessor on H100).
  • Speed: Near-instantaneous (Registers) to extremely fast (~30 TB/s aggregate).
  • Intuition: This is the engineer's physical workbench. Data must be brought here to be operated upon by Tensor Cores. Shared Memory is software-managed. You must explicitly write code to load data from Global Memory into Shared Memory. FlashAttention derives its speed entirely from keeping the $Q$, $K$, and $V$ blocks on this workbench and computing the entire attention matrix before writing the final result back out, avoiding round-trips to the slower memory tiers.

The L2 Cache (The Local Warehouse)

  • Size: Medium (50 MB on H100).
  • Speed: Fast (~12 TB/s).
  • Intuition: Managed automatically by the hardware. When scaling models across Azure ND-series instances (e.g., NDm A100 v4 or H100 v5), understanding L2 cache behavior is critical. If your working set for a kernel exceeds L2 capacity, you suffer cache misses, dropping down to HBM speeds.

High Bandwidth Memory - HBM (The Global Shipping Hub)

  • Size: Large (80GB to 192GB).
  • Speed: "Slow" relative to compute (2.0 to 8.0 TB/s).
  • Intuition: This is where the model weights, KV cache, and activations reside. During text generation, for every single token produced, the entire model weight matrix must be transferred from HBM to the SMs. If you are serving an SLM like Qwen-7B at FP16, the model weights consume 14GB. To generate one token, 14GB of data must cross the HBM bus. On an A100 (2.0 TB/s bandwidth), the absolute theoretical maximum throughput for a batch size of 1 is $\frac{2000 \text{ GB/s}}{14 \text{ GB}} \approx 142 \text{ tokens/sec}$. No amount of kernel optimization can exceed this physics-bound limit. The only solutions are quantization (compressing the weights) or continuous batching (amortizing the memory read across multiple queries).

1.3 Tensor Core Architecture and Mathematical Intuition

Tensor Cores are highly specialized ALUs designed to do exactly one thing with terrifying efficiency: Fused Multiply-Add (FMA) operations on matrices. Specifically, they compute $D = A \times B + C$.

The Precision Hierarchy

The evolution of Tensor Cores has been a race to the bottom in bit precision, recognizing that deep neural networks are highly resilient to numerical noise.

Precision Bit Distribution (Sign, Exponent, Mantissa) Ideal Use Case Memory Footprint (vs FP32)
FP32 1, 8, 23 Gradient accumulation, Loss calculation 1x
BF16 1, 8, 7 Pre-training, Fine-tuning 0.5x
FP16 1, 5, 10 Inference (Legacy), Activations 0.5x
FP8 (E4M3) 1, 4, 3 High-throughput Inference (Weights & Activations) 0.25x
INT4 N/A (Linear quantization) Edge deployment, Weight-only inference 0.125x
Why Fine-Tuning Demands BF16:
When fine-tuning architectures like Nemotron or Qwen, gradients can explode or vanish. FP16 has a narrow dynamic range (max value ~65,504); overflowing it results in NaN losses, crashing the training run. BF16 (Brain Float) sacrifices precision (mantissa) to match the dynamic range (exponent) of FP32. You lose decimal accuracy, but you completely eliminate overflow risks.
Why Inference Prefers FP8/INT4:
Once weights are frozen, the dynamic range requirement drops. Utilizing FP8 (e.g., E4M3 for weights) on Hopper architectures engages 4th-generation Tensor Cores, immediately doubling the mathematical throughput compared to BF16, while simultaneously halving the HBM bandwidth requirement.

1.4 The Communications Topology: PCIe vs. NVLink vs. InfiniBand

When models exceed a single GPU, the bottleneck shifts from HBM bandwidth to interconnect bandwidth.

Intra-Node (Within the Server)

  • PCIe Gen5: ~64 GB/s bi-directional. Used strictly for Host-to-Device (CPU to GPU) transfers. Never shard a model across GPUs communicating solely via PCIe unless latency is completely irrelevant.
  • NVLink: 600 GB/s (A100) to 900 GB/s (H100) to 1.8 TB/s (Blackwell). This is a dedicated, ultra-high-speed mesh. When implementing Tensor Parallelism (TP), the TP degree must never exceed the NVLink domain (typically 4 or 8 GPUs). When an attention matrix is split across 4 GPUs, the resulting vectors must be synchronized via an AllReduce operation. NVLink makes this synchronization nearly invisible.

Inter-Node (Between Servers)

  • InfiniBand (RDMA/RoCE): ~400 Gbps (50 GB/s) to 800 Gbps. When scaling beyond a single node (e.g., multi-node cluster deployment), traffic hits the network switch. This is orders of magnitude slower than NVLink.
  • Rule of Scale: Pipeline Parallelism (PP) and Data Parallelism (DP) are mapped across InfiniBand links because they require infrequent, small communication payloads compared to the constant, dense matrix syncing of Tensor Parallelism.

MODULE 2: The Execution Matrix – CUDA Kernel Mastery & Memory Choreography

2.1 The Execution Hierarchy & Theoretical Occupancy

When launching a CUDA kernel, you define a Grid of Thread Blocks. Understanding how these blocks map to the Streaming Multiprocessors (SMs) dictates your absolute performance limit.

The Hierarchy Breakdown

  1. Thread: The fundamental unit. Processes one element (or a small tile).
  2. Warp: 32 threads executing in lockstep (SIMT).
  3. Thread Block: A group of warps (up to 1024 threads) that share an L1/Shared Memory space. Crucial: A thread block cannot be split across multiple SMs.
  4. Grid: The entire collection of Thread Blocks.
  5. Cluster (Hopper+): A group of Thread Blocks that can share data across the L1 caches of multiple SMs simultaneously.

The Occupancy Trap (Register Pressure)

Occupancy is the ratio of active warps on an SM to the maximum possible active warps. While 100% occupancy sounds ideal, it is a trap. An H100 SM has 65,536 registers. If a block has 1024 threads, and each thread requires 64 registers for a fused attention kernel, the block consumes $1024 \times 64 = 65,536$ registers.

  • Result: Only one block can fit on the SM.
  • If that block stalls waiting for HBM memory, the SM sits idle because there are no other resident blocks to switch context to.

    [!info] SENIOR ARCHITECT NOTES:

    For LLM inference kernels (like custom RMSNorm or RoPE), you often deliberately restrict block sizes to 128 or 256 threads. This keeps register usage low per block, allowing 4 to 8 blocks to reside on a single SM. If one block stalls on a KV-cache memory read, the warp scheduler instantly swaps to a warp in another block with zero overhead. Latency hiding > Raw Occupancy.

C++ cpp // PRO TIP: Always dynamically calculate optimal launch configurations int minGridSize; int blockSize; cudaOccupancyMaxPotentialBlockSize( &minGridSize, &blockSize, MyFusedAttentionKernel, dynamicSMemSize, 0 // max sequence length limit ); // Use 'blockSize' for your kernel launch ### 2.2 Memory Choreography: Resolving the Bottlenecks Data movement dictates inference speed. A kernel's efficiency is determined by how well it utilizes the memory tiers. #### Memory Tier Latency & Bandwidth Matrix (H100 Architecture) |Memory Tier|Access Latency|Bandwidth per SM|Best Used For| |---|---|---|---| |Registers|1-2 cycles|15 TB/s|Thread-local variables, Accumulators| |Shared (L1)|20-30 cycles|12 TB/s|Block-local broadcasting, Matrix tiles| |L2 Cache|200 cycles|5 TB/s|Cross-SM data reuse (e.g., model weights in batch > 1)| |Global (HBM)|500-800 cycles|3.3 TB/s|KV Cache, Full Model Weights, Activations| #### 1. Global Memory: The Coalescing Rule When threads in a warp access HBM, the hardware attempts to group (coalesce) the accesses into a single 32-byte, 64-byte, or 128-byte transaction. - Rule: Thread $i$ must access Memory Address $N + i$. - If threads access scattered memory (e.g., gathering specific non-contiguous tokens from a KV cache without PagedAttention), the transaction splits. A read that should take one 128-byte transaction might take 32 separate 32-byte transactions. Bandwidth drops by 90%. #### 2. Shared Memory: Bank Conflicts Shared memory is divided into 32 "banks". If multiple threads in a warp access different addresses within the same bank, a "Bank Conflict" occurs. The hardware serializes the requests. C++ cpp // BAD: Causes N-way bank conflicts during matrix transpose or QK^T __shared__ float tile[32][32]; float val = tile[threadIdx.y][threadIdx.x]; // Stride of 32 hits same bank // JUNIOR ENGINEER CHECKLIST: The Padding Trick // GOOD: Add a single padding column to stagger the bank mapping __shared__ float tile[32][33]; float val = tile[threadIdx.y][threadIdx.x]; // Bank conflict completely resolved! ### 2.3 Hopper Architecture Exclusives: Clusters and TMA If you are deploying LLMs on H100s, standard CUDA paradigms are insufficient. You must utilize Hopper's specific asynchronous features. #### Thread Block Clusters Prior to Hopper, if Block A needed data processed by Block B, Block B had to write to HBM, and Block A had to read from HBM. Hopper introduces Clusters. Blocks within a cluster can perform .st.async (asynchronous stores) directly into the Shared Memory of another SM within the cluster. #### Tensor Memory Accelerator (TMA) TMA is a dedicated hardware engine on the H100. Instead of threads calculating memory addresses and executing ld.global instructions in a loop, a single thread issues a TMA descriptor. The TMA hardware asynchronously fetches a massive multi-dimensional tile of data directly into Shared Memory while the SM's ALUs do other math. Code snippet mermaid sequenceDiagram participant SM_ALU as SM (Compute) participant TMA_Engine as TMA (Hardware) participant HBM as Global Memory participant SMem as Shared Memory SM_ALU->>TMA_Engine: "Fetch Q, K, V tiles [Issue Descriptor]" Note over SM_ALU: SM begins computing previous tile TMA_Engine->>HBM: Request Bulk DMA Transfer HBM-->>TMA_Engine: Stream Data TMA_Engine->>SMem: Write Directly to Shared Memory TMA_Engine->>SM_ALU: Async Arrive Barrier (Signal Completion) Note over SM_ALU: SM computes new tile ### 2.4 Asynchronous Execution: Streams & Overlapping An inference server (like vLLM) never processes one task at a time. To achieve maximum throughput, compute (kernel execution) must overlap with data transfer (PCIe/NVLink) and other compute. CUDA Streams are queues of GPU operations. Operations within a single stream execute sequentially. Operations in different streams can execute concurrently. C++ cpp // THE CONTINUOUS BATCHING OVERLAP PATTERN cudaStream_t computeStream, transferStream; cudaStreamCreate(&computeStream); cudaStreamCreate(&transferStream); for (int batch_idx = 0; batch_idx < num_batches; ++batch_idx) { // 1. Asynchronously copy next batch inputs to GPU cudaMemcpyAsync(d_input[batch_idx+1], h_input[batch_idx+1], size, cudaMemcpyHostToDevice, transferStream); // 2. Compute current batch simultaneously FusedTransformerLayer<<<grid, block, 0, computeStream>>>(d_input[batch_idx]); // 3. Sync points (Events) to ensure transfer completes before compute uses it cudaEventRecord(transferEvent, transferStream); cudaStreamWaitEvent(computeStream, transferEvent, 0); } ### 2.5 CUDA Graphs: The CPU Overhead Eradicator In Small Language Models (SLMs) or when generating tokens at high speeds, the GPU executes a kernel in 2 microseconds. However, the CPU takes 5 microseconds to launch the kernel. The GPU ends up starved, waiting for the CPU. CUDA Graphs allow you to "record" the entire sequence of hundreds of kernel launches (LayerNorm -> QKV Proj -> RoPE -> Attention -> MLP -> Logits) into a single execution graph. #### Advanced Graph Pattern: Node Updating You cannot use standard CUDA graphs if parameters change (like the sequence_length index incrementing during token generation). You must use Graph Node Updating. C++ cpp // 1. Initial capture (Done during server warmup) cudaStreamBeginCapture(stream, cudaStreamCaptureModeGlobal); RoPEKernel<<<grid, block, 0, stream>>>(q, k, current_seq_pos); // Capture with pos=0 cudaStreamEndCapture(stream, &graph); cudaGraphInstantiate(&instance, graph, NULL, NULL, 0); // 2. During Token Generation Loop (Iterative Decode) for (int seq_pos = 1; seq_pos < max_len; ++seq_pos) { // AVOID RECAPTURING: Update the kernel parameters directly in the instantiated graph cudaGraphExecKernelNodeSetParams(instance, ropeNode, &new_rope_params); // Launch the entire model step in ONE CPU call (~2µs overhead total) cudaGraphLaunch(instance, stream); cudaStreamSynchronize(stream); }

MODULE 3: Framework-to-Silicon Mapping & Architectural Deep Dives

To optimize at the highest level, you must understand how the structural idiosyncrasies of specific model families interact with the hardware constraints discussed in Modules 1 and 2.

3.1 SLM vs. LLM Deployment Physics

Small Language Models (SLMs) (e.g., in the 1B to 8B parameter range) exhibit fundamentally different performance bottlenecks compared to massive 70B+ LLMs. When deploying SLM-based architectures, your primary enemy shifts from Memory Capacity to Memory Bandwidth vs. Compute Bound thresholds.

The Arithmetic Intensity Threshold

Arithmetic Intensity is the ratio of mathematical operations (FLOPs) to memory bytes transferred.

  • Compute-Bound: The GPU spends more time doing math than waiting for data. (Ideal).
  • Memory-Bound: The GPU waits for data to arrive from HBM to the SMs. (Typical for inference). The SLM Math (e.g., Qwen-7B at FP16 on Azure NDm A100 v4):
  • Model Weights: ~14 GB.
  • A100 HBM Bandwidth: 2000 GB/s.
  • Max Theoretical Decode Speed (Batch = 1): $2000 / 14 = \sim 142$ tokens/second.
  • A100 FP16 Compute: 312 TFLOPS.
  • Actual Compute Used (Batch = 1): Roughly $2 \times 7 \text{ billion} = 14 \text{ GFLOPs}$ per token. At 142 tokens/sec, you are using $142 \times 14 \text{ GFLOPs} = 1.98 \text{ TFLOPS}$. You are utilizing less than 1% of the A100's compute capacity. The GPU is starving.

Continuous Batching: The SLM Savior

To shift an SLM from memory-bound to compute-bound, you must increase the batch size until the arithmetic intensity crosses the hardware's threshold. Continuous batching (iteration-level scheduling) ensures that as soon as one request finishes, another is injected into the very next forward pass. Code snippet

gantt
    title Static vs Continuous Batching Execution
    dateFormat  s
    axisFormat %S
    section Static Batching (Legacy)
    Req 1 (Short)    :active, a1, 0, 3s
    Req 2 (Medium)   :active, a2, 0, 5s
    Req 3 (Long)     :active, a3, 0, 8s
    GPU Idle Time    :crit, c1, 3, 8s
    section Continuous Batching (vLLM/TGI)
    Req 1 (Short)    :active, b1, 0, 3s
    Req 4 (New)      :done, b2, 3, 8s
    Req 2 (Medium)   :active, b3, 0, 5s
    Req 5 (New)      :done, b4, 5, 8s
    Req 3 (Long)     :active, b5, 0, 8s

3.2 Deep Dive: FlashAttention-2 vs. FlashAttention-3 Kernels

Standard attention is $O(N^2)$ in both time and memory. It requires materializing the entire $N \times N$ attention matrix in HBM, which is catastrophic for long-context windows.

FlashAttention-2 (Ampere/Hopper Base)

FlashAttention achieves $O(N)$ memory and significant speedups by computing the softmax incrementally and never materializing the $N \times N$ matrix in HBM. It uses Tiling.

  1. Load blocks of $Q$, $K$, $V$ from HBM to Shared Memory (SRAM).
  2. Compute $S = QK^T$ in SRAM.
  3. Update the online softmax running maximums and denominators.
  4. Compute $O = SV$ in SRAM.
  5. Write the block of $O$ back to HBM. The PTX Reality: FA2 relies heavily on the mma.sync (Matrix Multiply-Accumulate) PTX instructions. However, the SM's warp scheduler still has to issue these instructions and manage registers.

FlashAttention-3 (Hopper Exclusive - The WGMMA Leap)

If you are deploying on Azure H100 instances, FA3 is mandatory. It abuses the Hopper specific Warp Group Matrix Multiply Accumulate (WGMMA). Instead of a single warp (32 threads) doing a small matrix math operation, WGMMA allows a Warp Group (128 threads) to collaboratively issue a single, massive matrix multiplication instruction directly against data residing in Shared Memory, bypassing registers entirely. C++

// SENIOR ARCHITECT KERNEL CONCEPT: The WGMMA Pipeline in FA3
// 1. TMA initiates async fetch of Q, K, V tiles to Shared Memory
// 2. While fetching tile N+1, issue WGMMA instruction for tile N
// WGMMA PTX Assembly abstraction:
asm volatile(
    "wgmma.mma_async.sync.aligned.m64n64k16.f32.f16.f16 "
    "[%0], [%1], [%2], %3;\n"
    :
    : "r"(D_sram_ptr), "r"(A_sram_ptr), "r"(B_sram_ptr), "n"(scale)
);
// Notice how A and B are pointers to SRAM, not registers. 
// This frees up massive register space, allowing higher occupancy.

3.3 Advanced Quantization: The GPTQ vs. AWQ Kernel Architecture

When compressing models (like fine-tuned Nemotron or Qwen variants) down to 4-bit, the math remains in FP16/BF16. The "4-bit" refers only to the storage of the weights. The weights must be dequantized to FP16 in registers just before the FMA operation.

GPTQ (Post-Training Quantization via Hessian)

GPTQ quantizes weights row-by-row, adjusting the remaining weights in the row to compensate for the quantization error of the previous weights using inverse Hessian matrices.

  • Kernel Bottleneck: GPTQ kernels (like ExLlamaV2 or Marlin) have to do complex bit-shifting and masking to unpack the 4-bit integers back into FP16.
  • The Marlin Breakthrough: The Marlin kernel optimizes this by repacking the 4-bit weights offline into a specific memory layout that allows the GPU's ldmatrix instruction to load the 4-bit weights and instantly unpack them directly into the FP16 registers required by the Tensor Cores, without intermediate bit-manipulation steps.

AWQ (Activation-aware Weight Quantization)

AWQ realizes that a small fraction (e.g., 1%) of weights are highly salient based on activation magnitudes. It keeps these salient weights in FP16 (or scales them up before quantization).

  • Why AWQ is currently favored for deployment: AWQ's hardware implementation is simpler. It relies on standard GEMM (General Matrix Multiply) operations with a fused dequantization step. It maps perfectly to vLLM's custom CUDA kernels without needing the specialized layout repacking that highly optimized GPTQ (Marlin) requires.
Metric GPTQ (Standard) GPTQ (Marlin Kernel) AWQ (vLLM Native)
Quantization Time Slow (Hours) Slow (Hours) Fast (Minutes)
Memory Layout Standard Highly Custom (Repacked) Standard
Decode Speed (Batch=1) Fast Extremely Fast Fast
Prefill Speed Moderate Very Fast Fast
Ease of Multi-GPU Moderate Hard Very Easy (Native TP)

3.4 The W8A8 SmoothQuant Architecture: Solving Activation Outliers

SmoothQuant is the undisputed backbone of INT8/FP8 serving on Ampere and Hopper architectures. When scaling to 70B+ or multimodal models, Weight-Only Quantization (like AWQ/GPTQ) hits a ceiling. If you only quantize weights (W4A16), you compress the memory footprint, but the math is still executed in FP16. To double your compute throughput and utilize INT8/FP8 Tensor Cores, you must quantize both weights and activations (W8A8).

The Mathematical Roadblock: Activation Outliers

Quantizing weights to INT8 is easy; weights are statically distributed. Quantizing activations to INT8 historically destroyed model accuracy. Why? Large language models systematically develop massive "outlier channels" in their activations. For example, across 4096 hidden dimensions, channel #112 might consistently output values 100x larger than the other 4095 channels.

  • If you do per-tensor quantization (scaling the whole matrix by the largest value), the outlier forces the scaling factor so high that the other 4095 channels get crushed to zero.
  • If you do per-channel quantization on activations, the hardware cannot execute it efficiently because INT8 Tensor Cores require the inner dot-product dimensions to share the same scale.

The SmoothQuant Breakthrough: Migration of Difficulty

SmoothQuant solves this by realizing a mathematical equivalence: you can "smooth" the activations by dividing the outlier channels by a scaling factor $s$, as long as you multiply the corresponding input channels of the Weight matrix by that exact same $s$ offline. Given the standard linear layer $\mathbf{Y} = \mathbf{X} \mathbf{W}$: Y=(Xdiag(s)−1)⋅(diag(s)W)\mathbf{Y} = (\mathbf{X} \text{diag}(\mathbf{s})^{-1}) \cdot (\text{diag}(\mathbf{s}) \mathbf{W}) Let $\hat{\mathbf{X}} = \mathbf{X} \text{diag}(\mathbf{s})^{-1}$ and $\hat{\mathbf{W}} = \text{diag}(\mathbf{s}) \mathbf{W}$. We have successfully migrated the quantization difficulty from the dynamic activations to the static weights. $\hat{\mathbf{X}}$ is now smooth and can be trivially quantized to INT8 per-tensor, and $\hat{\mathbf{W}}$ can be quantized to INT8 per-channel.

CUDA Execution: The CUTLASS INT8 GEMM Pipeline

To extract actual speedups from W8A8, you cannot rely on PyTorch. You must drop down to CUTLASS (CUDA Templates for Linear Algebra Subroutines) to map the operation directly to INT8 mma.sync hardware instructions. C++

// SENIOR ARCHITECT: W8A8 SmoothQuant Fused GEMM + Dequantization Pipeline
// This executes the INT8 math and immediately converts back to FP16 in registers
// 1. The Core PTX Instruction for INT8 Tensor Cores (Ampere/Turing)
// mma.sync.aligned.m16n8k32.row.col.s32.s8.s8.s32
// Notice K=32: INT8 Tensor Cores consume 32 elements per instruction, 
// exactly double the 16 elements consumed by FP16 Tensor Cores!
template <typename ElementInputA, typename ElementInputB, typename ElementOutput>
__global__ void smoothquant_w8a8_gemm_kernel(
    const int8_t* __restrict__ A_quant, // Smoothed Activations
    const int8_t* __restrict__ B_quant, // Scaled Weights
    half* __restrict__ C_out,           // FP16 Output
    const float* __restrict__ alpha_col,// Weight scales
    const float alpha_act,              // Activation scale
    int M, int N, int K
) {
    // ... [CUTLASS Threadblock mapping boilerplate] ...
    // Step 1: Execute INT8 Tensor Core Math
    // Accumulator must be INT32 to prevent overflow during the dot product
    int32_t accumulator[TileSize] = {0}; 
    // -> Hardware executes: accumulator = A_quant * B_quant
    // Step 2: Fused Epilogue (Dequantization)
    // We do NOT write INT32 to global memory. We dequantize in registers.
    int col_idx = blockIdx.x * blockDim.x + threadIdx.x;
    float w_scale = alpha_col[col_idx];
    float combined_scale = w_scale * alpha_act;
    for(int i=0; i<TileSize; ++i) {
        // Dequantize: FP16_out = INT32_accum * (Weight_Scale * Act_Scale)
        float fp32_val = static_cast<float>(accumulator[i]) * combined_scale;
        // Write out FP16
        C_out[output_offset] = __float2half(fp32_val);
    }
}

The Migration to FP8 (Hopper Architecture)

If you are deploying on H100s, SmoothQuant INT8 is effectively being superseded by Native FP8 (E4M3 for weights/activations). FP8 maintains a non-linear exponent, giving it a dynamic range that inherently survives activation outliers without requiring the aggressive mathematical smoothing of SmoothQuant. However, for Ampere (A100) or Ada (L40S) deployments where FP8 Tensor Cores are either absent or less flexible, W8A8 SmoothQuant remains the absolute ceiling for throughput optimization.

Precision / Method Hardware Target Accumulator Activation Outlier Handling Compute Speedup Memory Speedup
FP16 Universal FP32 Natural (Large dynamic range) 1x 1x
INT8 (Naive) Ampere / Ada INT32 Fails (Values clipped to 127) 2x 2x
W8A8 SmoothQuant Ampere / Ada INT32 Mathematically Migrated to Weights 2x 2x
FP8 (E4M3) Hopper (H100) FP32 Natural (Exponent bits absorb outliers) 2x+ 2x

MODULE 4: Custom Kernel Fusion & Bare-Metal Distributed Execution

To achieve the "max in-depth" reality that bypasses high-level frameworks (like PyTorch or HuggingFace), we must bridge the gap between custom CUDA C++ kernel development and how those kernels physicalize across a high-speed NVLink network. This module exposes the exact mechanics used by engineers at OpenAI, Anthropic, and X.x to squeeze the final 20% of performance out of their clusters.

4.1 The Physics of Kernel Fusion: Why PyTorch is "Slow"

A standard transformer block contains operations like RMSNorm, SwiGLU (SiLU + Multiply), and RoPE (Rotary Positional Embeddings). If you write this in naive PyTorch:

x = rms_norm(x)
y = silu(y)
z = x * y

The Hardware Reality (The Memory Wall):

  1. PyTorch launches RMSNormKernel. It reads $X$ (14GB) from HBM to SRAM, computes, and writes $X'$ back to HBM.
  2. PyTorch launches SiLUKernel. It reads $Y$ (14GB) from HBM, computes, and writes $Y'$ back to HBM.
  3. PyTorch launches MultiplyKernel. It reads $X'$ and $Y'$ (28GB) from HBM, multiplies, and writes $Z$ back to HBM. Total HBM trip: 84 GB of memory movement for a few mathematical operations. This is devastating to throughput. Kernel Fusion writes a single custom CUDA kernel that reads $X$ and $Y$ into the SM's registers once, performs RMSNorm, SiLU, and the multiplication entirely inside the ultra-fast registers, and writes $Z$ out once. Total memory movement: 42 GB (A 2x speedup by simply deleting memory writes).

4.2 Bare-Metal CUDA: Fused RMSNorm + SwiGLU Implementation

Writing high-performance CUDA requires exploiting Warp-level primitives and Vectorized Memory Access.

1. Vectorized Memory Loads (float4)

If you load a single __half (16-bit float) per thread, you waste PCIe/HBM transaction bandwidth. You must cast your pointers to float4 (which holds 128 bits / eight 16-bit floats) so a single instruction fetches an entire 128-byte cache line per warp.

2. Warp-Level Reductions (__shfl_down_sync)

RMSNorm requires calculating the sum of squares across the entire embedding dimension (e.g., 4096). Instead of writing to Shared Memory and causing bank conflicts, threads pass data directly between each other's registers using the Warp Shuffle instruction.

// SENIOR ARCHITECT CODE: The Fused Kernel (Simplified for 1D Block)
#include <cuda_fp16.h>
// 1. The Warp Reduction Primitive (Math strictly in registers)
__inline__ __device__ float warpReduceSum(float val) {
    for (int offset = warpSize/2; offset > 0; offset /= 2) 
        val += __shfl_down_sync(0xffffffff, val, offset);
    return val;
}
// 2. The Fused Kernel
__global__ void FusedRMSNormSwiGLU_Kernel(
    const half4* __restrict__ X,  // Input 1 (Vectorized 64-bit load)
    const half4* __restrict__ Y,  // Input 2
    half4* __restrict__ Z,        // Output
    const float* __restrict__ gamma, // RMSNorm weights
    const float epsilon,
    const int hidden_dim_vec // hidden_dim / 8 (since half4 holds 8 x fp16)
) {
    int tid = threadIdx.x;
    int token_idx = blockIdx.x; // Each block handles one token
    // Calculate global memory offsets
    int offset = token_idx * hidden_dim_vec + tid;
    // Step 1: Vectorized Read (Bypasses L1 cache, goes straight to L2/Registers)
    half4 x_vec = X[offset];
    half4 y_vec = Y[offset];
    // Step 2: Unpack and calculate sum of squares for RMSNorm
    float sq_sum = 0.0f;
    // ... unpack half4 to floats ... 
    // sq_sum += x_float1 * x_float1 + ... + x_float8 * x_float8;
    // Step 3: Warp-level sum (No shared memory required!)
    sq_sum = warpReduceSum(sq_sum);
    // Cross-warp reduction (if block > 32 threads) requires 1 write to Shared Memory
    __shared__ float shared_sum[32];
    int lane = tid % warpSize;
    int warpId = tid / warpSize;
    if (lane == 0) shared_sum[warpId] = sq_sum;
    __syncthreads();
    // Calculate final variance denominator
    float variance = shared_sum[0] + ... ; // simplified
    float inv_rms = rsqrtf(variance / (hidden_dim_vec * 8) + epsilon);
    // Step 4: Compute RMSNorm, SiLU, and Multiply directly in registers
    // SwiGLU(y) = y * sigmoid(y)
    // Z = RMSNorm(x) * SwiGLU(y)
    // ... perform math ...
    // Step 5: Vectorized Write out
    Z[offset] = z_vec_final; 
}

[!info] PTX Introspection: If you compile this with nvcc -ptx, you will see ld.global.nc.v4.f16 (Load Global Non-Coherent Vector-4). The .nc flag tells the L1 cache not to bother caching the data, preventing cache pollution, because in LLM inference, activation data is read once and immediately discarded.

4.3 Network Topologies: Tensor Parallelism Across NVLink

When an LLM (e.g., Llama3-70B) does not fit on one GPU, or the math is too vast for one GPU's required latency SLA, we use Tensor Parallelism (TP). The TP Math: Instead of computing $Y = X \times W$ on one GPU, we split the weight matrix $W$ vertically into $W_1$ and $W_2$.

  • GPU 0 computes $Y_1 = X \times W_1$
  • GPU 1 computes $Y_2 = X \times W_2$ To get the final vector, the GPUs must communicate to combine the results. This requires an AllReduce operation.

The AllReduce Bottleneck and NVSwitch

In an 8xH100 HGX system, the GPUs are not connected in a ring. They are connected to 4 hardware NVSwitches.

graph TD
    subgraph "HGX Baseboard"
        G1[GPU 0] <==>|18 NVLinks| SW1[NVSwitch 1]
        G2[GPU 1] <==>|18 NVLinks| SW1
        G1 <==>|18 NVLinks| SW2[NVSwitch 2]
        G2[GPU 1] <==>|18 NVLinks| SW2
        SW1 <==> G8[GPU 7]
        SW2 <==> G8
        Note["⚠️ AllReduce happens IN THE SWITCH<br/>hardware (SHARP), not on the GPUs!"]
    end
    
    style Note fill:#ffffcc,stroke:#ffcc00,stroke-width:2px

The SHARP Advantage (Scalable Hierarchical Aggregation and Reduction Protocol): Historically, GPUs passed data in a ring to sum up the vectors. On modern HGX systems with NVLink-Network (NVL), GPU 0 and GPU 1 push their $Y_1$ and $Y_2$ vectors into the NVSwitch simultaneously. The NVSwitch hardware itself possesses ALUs that perform the addition operation in-flight, and it pushes the final $Y$ vector back down to all GPUs. This effectively halves the communication time for an AllReduce.

[!info] JUNIOR CHECKLIST - Verifying NVLink Health:

A single degraded NVLink degrades the entire AllReduce operation to the speed of the slowest link.

# Always run this before starting a serving engine on a new node
nvbandwidth -t device_to_device_bidirectional
# On 8xH100, EVERY intersection must read > 850 GB/s. 
# If you see a 400 GB/s, a cable is improperly seated or a lane is burnt.

4.4 Multi-Node Scaling: InfiniBand & Context Chunking

When scaling beyond 8 GPUs (e.g., serving a 405B model, or fine-tuning), you cross the node boundary into the InfiniBand network.

The Cross-Node Rule: Never Tensor Parallel

InfiniBand (even 400 Gbps NDR) operates at ~50 GB/s. NVLink operates at 900 GB/s.

  • If you map Tensor Parallelism across InfiniBand, the GPUs will spend 95% of their time waiting for the AllReduce over the network.
  • Solution: Map Pipeline Parallelism (PP) across InfiniBand. Node 1 holds Layers 1-40. Node 2 holds Layers 41-80. Node 1 passes its final activation vector over InfiniBand to Node 2. This is a point-to-point transfer, not an AllReduce, and happens only once per forward pass.

Chunked Prefill (The Ultimate Production Hack)

When serving LLMs, the "Prefill" phase (processing the user's prompt) takes massive compute and destroys the latency for any user currently in the "Decode" phase (generating tokens one by one) on the same server. The Problem: A user sends a 100K token document. The GPU locks up for 2 seconds to process it. All other 50 users currently generating tokens see their streams freeze for 2 seconds. The Solution (Chunked Prefill): We enforce a strict max_num_batched_tokens limit (e.g., 4096).

  1. The 100K document is split into 25 chunks of 4096 tokens.
  2. Step 1: Process Chunk 1 of the document + Generate 1 token for all 50 other users.
  3. Step 2: Process Chunk 2 of the document + Generate 1 token for all 50 other users.
  4. The long document's KV cache is incrementally built over 25 steps. The Result: The 50 users generating tokens experience zero stutter (latency remains flat at ~50ms/token), while the new user's Time-To-First-Token (TTFT) degrades gracefully, completely eradicating the "tail latency" spikes in production.

Infrastructure Mapping Matrix (Scale to Billion Parameters)

Model State Required VRAM Target Hardware Engine Strategy Parallelism Strategy
8B (FP16) ~16GB 1x L40S or 1x A100 vLLM (Eager mode) None (Batch size heavily)
70B (AWQ4) ~42GB 1x H100 (80GB) vLLM (CUDA Graphs) None (Max Context: 32K)
70B (FP16) ~140GB 2x H100 / 4x A100 TensorRT-LLM TP=2 or TP=4 (NVLink)
405B (FP8) ~405GB 8x H100 (One Node) vLLM / SGLang TP=8 (NVLink, SHARP)
1.5T (MoE) ~3.0TB 64x H100 (8 Nodes) Megatron-LM / TGI TP=8 (Intra) + PP=8 (Inter)

MODULE 5: Advanced Execution Architectures & Silicon Diagnostics

While Module 4 covered bare-metal CUDA C++ and multi-node execution, we must address the two final frontiers of inference optimization: Latency-Bound Decoding (breaking the batch=1 speed limit) and Micro-Architectural Profiling (proving your kernels actually utilize the silicon).

5.1 The Abstraction Layer: Bare-Metal CUDA vs. OpenAI Triton

Writing custom CUDA C++ (as shown in Module 4) requires manual management of Shared Memory allocations, thread synchronization (__syncthreads()), and bank conflict avoidance. OpenAI Triton is a domain-specific language that abstracts these concepts into "Block-level" programming. It compiles directly down to PTX (Parallel Thread Execution) assembly, often matching or beating hand-written CUDA C++ by heavily optimizing memory coalescing and TMA usage via MLIR (Multi-Level Intermediate Representation).

Triton vs. CUDA C++ for Custom Fused Kernels

Feature Pure CUDA C++ OpenAI Triton PyTorch torch.compile
Execution Paradigm Thread-level (SIMT) Block-level (Tile operations) Graph-level (AOTAutograd)
Memory Management Manual (__shared__, TMA) Compiler Automated Framework Automated
Hardware Control Absolute (WGMMA, PTX) High (Block sizes, stages) Low (Relies on Triton backend)
Dev Time for FA3 Weeks Days N/A (Calls FA backend)
Portability NVIDIA / AMD (HIP) Hardware agnostic (via backends) High
# SENIOR ARCHITECT PATTERN: Fused Softmax in Triton
# This perfectly maps to GPU L2/SRAM without the boilerplate of CUDA C++
import triton
import triton.language as tl
@triton.jit
def fused_softmax_kernel(
    output_ptr, input_ptr, input_row_stride, output_row_stride, 
    n_cols, BLOCK_SIZE: tl.constexpr
):
    # 1. Map thread block to a specific row
    row_idx = tl.program_id(0)
    row_start_ptr = input_ptr + row_idx * input_row_stride
    # 2. Load the row into SRAM (Triton handles the coalescing)
    col_offsets = tl.arange(0, BLOCK_SIZE)
    input_ptrs = row_start_ptr + col_offsets
    mask = col_offsets < n_cols
    row = tl.load(input_ptrs, mask=mask, other=-float('inf'))
    # 3. Compute Softmax entirely in Registers/SRAM
    row_minus_max = row - tl.max(row, axis=0)
    numerator = tl.exp(row_minus_max)
    denominator = tl.sum(numerator, axis=0)
    softmax_output = numerator / denominator
    # 4. Write back to HBM
    output_row_start_ptr = output_ptr + row_idx * output_row_stride
    output_ptrs = output_row_start_ptr + col_offsets
    tl.store(output_ptrs, softmax_output, mask=mask)

[!info] PRO TIP - PTX Inspection: Never blindly trust Triton. Always inspect the generated assembly by running print(fused_softmax_kernel.asm["ptx"]). Verify that memory loads utilize .v4 (128-bit vectorized loads) and that there are no register spills to local memory (.local).

5.2 Breaking the Memory Wall: Speculative Decoding Architectures

In Module 1, we established the absolute physics limit of LLM decoding: to generate 1 token, you must load the entire model weight matrix from HBM. If your batch size is 1 (e.g., a real-time voice assistant), you are crippled by memory bandwidth. Speculative Decoding solves this by exploiting the fact that modern GPUs have excess ALUs during the decode phase.

The Mechanics of Speculative Decoding

  1. Draft Phase: A tiny, fast SLM (e.g., a 68M parameter draft model) rapidly generates $K$ tokens (e.g., $K=4$).
  2. Verification Phase: The massive Target Model (e.g., 70B parameters) takes those $K$ tokens and processes them in a single forward pass as if it were a prefill operation.
  3. Acceptance: If the Target Model agrees with $N$ out of $K$ tokens, we successfully generated $N$ tokens for the time cost of loading the Target Model's weights once.
sequenceDiagram
    participant Draft Model (Small)
    participant Target Model (70B)
    participant HBM (Memory)
    Draft Model (Small)->>Draft Model (Small): Gen Token 1 (Fast)
    Draft Model (Small)->>Draft Model (Small): Gen Token 2 (Fast)
    Draft Model (Small)->>Draft Model (Small): Gen Token 3 (Fast)
    Draft Model (Small)-->>Target Model (70B): Proposes [T1, T2, T3]
    Target Model (70B)->>HBM: Load Weights ONCE (140GB)
    Target Model (70B)->>Target Model (70B): Parallel Verification of [T1, T2, T3]
    Target Model (70B)-->>Target Model (70B): Accepts [T1, T2], Rejects [T3]
    Note right of Target Model (70B): Generated 2 tokens in the time it usually takes to generate 1!

Medusa & Lookahead Decoding (No Draft Model Required)

Managing two separate models in production is a DevOps nightmare (requiring separate KV caches and complex memory scheduling).

  • Medusa Engine: Modifies the Target Model itself by adding extra "Heads" to the final transformer layer. While generating Token $T$, Head 1 guesses $T+1$, Head 2 guesses $T+2$.
  • The next forward pass verifies these guesses using a tree-based attention mask. This provides a 2x-3x speedup on batch=1 latency with zero draft-model overhead.

5.3 Prefix Caching & RadixAttention (SGLang Deep Dive)

When deploying multi-turn chat applications or agentic AI (where the system prompt or tool-call instructions are repeatedly sent), standard engines recompute the KV cache for the system prompt every single time. RadixAttention (Pioneered by SGLang, adopted by vLLM): Instead of assigning KV blocks linearly, the engine maintains an LRU (Least Recently Used) cache structured as a Radix Tree.

  1. Request 1: [System Prompt] + [User Query A]
  2. Request 2: [System Prompt] + [User Query B] When Request 2 arrives, the engine performs a tree search, hashes the tokens, and finds that [System Prompt] already exists in the GPU's physical memory.
  • It assigns the logical block table of Request 2 to simply point to the exact same physical HBM blocks as Request 1.
  • Result: Zero prefill compute for the system prompt. Time-To-First-Token drops from 500ms to 20ms.

5.4 Silicon Diagnostics: The Nsight Profiling Suite

"Optimization without profiling is just guessing." One requires mastering NVIDIA's profiling tools to prove your kernels are not bottlenecking.

Nsight Systems (NSYS) - Macro-Level Intersections

NSYS visualizes the interaction between CPU, PCIe, NVLink, and GPU. It is used to hunt down bubbles (idle GPU time).

# JUNIOR ENGINEER CHECKLIST - NSYS Capture
nsys profile \
  --trace=cuda,cudnn,cublas,nvtx,osrt \
  --output=llm_inference_profile \
  --force-overwrite=true \
  python serve_vllm.py

What to look for in the NSYS UI:

  • CUDA API Overheads: If the CPU row shows cudaLaunchKernel taking 15µs, but the GPU row shows the kernel executing in 2µs, your GPU is starving. Action: Implement CUDA Graphs (Module 2).
  • H2D / D2H Blocking: If you see a cudaMemcpy (Host to Device) executing, and all compute streams are empty, you are suffering synchronous blocking. Action: Use Pinned Memory and cudaMemcpyAsync (Module 2).

Nsight Compute (NCU) - Micro-Architectural Dissection

NCU is the scalpel. It attaches to a single kernel launch and reads the hardware counters directly off the SMs.

# SENIOR ARCHITECT - NCU Hardware Counter Deep Dive
ncu \
  --metrics smsp__inst_executed_pipe_tensor_op_hmma.avg.pct_of_peak_sustained_active,sm__throughput.avg.pct_of_peak_sustained_active,dram__bytes_read.sum \
  --target-processes all \
  -k "regex:.*fused_attention.*" \
  python benchmark_model.py

The Ultimate NCU Diagnostic Matrix

Metric Name in NCU Target Value If Low / Failing, It Means... Resolution Strategy
sm__throughput.avg.pct > 80% (Prefill)



< 10% (Decode)
Expected during decode. If low during prefill, kernel has massive divergence or stalls. Check occupancy, reduce register usage, optimize grid size.
dram__throughput.avg.pct > 90% (Decode) You are memory bound but failing to saturate the HBM bus. Fix memory coalescing. Ensure float4 vectorized loads are used.
l1tex__t_sectors_pipe_lsu_mem_global_op_ld.sum (Compare to theoretical) High uncoalesced memory reads (Bank conflicts or scattered access). Pad shared memory arrays. Align thread IDs to memory addresses.
smsp__inst_executed_pipe_tensor_op_hmma > 70% Tensor Cores are starving. Falling back to CUDA cores for FP16 math. Ensure matrix dimensions are multiples of 16 (or 64 for Hopper WGMMA). Pad sequence lengths.

[!info] THE ALIGNMENT RULE: Tensor Cores demand aligned memory. If your hidden dimension is 4095 instead of 4096, the hardware cannot execute optimal matrix multiplications and will fall back to slower execution paths or require massive padding overhead. Always design network dimensions as multiples of 8 (preferably 64 for Hopper).

MODULE 6: MULTIMODAL PHYSICS & AGENTIC STATE MACHINES (In-Depth Infrastructure)

The transition from text-only autoregressive generation to Multimodal Vision-Language Models (VLMs) and continuous, multi-step Agentic AI shatters the foundational assumptions of legacy inference engines. In text-only generation, sequence lengths grow predictably (by 1 token per step), and batching is uniform. VLMs introduce massive, immediate spatial token injection, while Agentic loops introduce branching, non-linear state trees and theoretically infinite context windows. This module maps the physical execution of these advanced paradigms down to the silicon level.

6.1 The VLM Spatial Injection Crisis: AnyRes and Heterogeneous Batching

Standard Vision Transformers (ViTs) expect a fixed-size square input (e.g., $336 \times 336$). To handle high-resolution, arbitrary-aspect-ratio images (e.g., $1920 \times 1080$), modern architectures (like LLaVA-1.5+ and Qwen-VL) use AnyRes (Any Resolution) dynamic patching. The image is sliced into a dynamic grid of $336 \times 336$ patches.

  • A $1920 \times 1080$ image becomes a $6 \times 3$ grid = 18 patches.
  • At 576 tokens per patch, this single image injects 10,368 tokens into the prefill phase instantly.

The Heterogeneous Sequence Rupture

In a batch of 4 requests:

  • Req 0: Text only (15 tokens)
  • Req 1: Text + 1 small image (600 tokens)
  • Req 2: Text + 1 AnyRes image (10,500 tokens)
  • Req 3: Text + 4 AnyRes images (41,000 tokens) The Naive PyTorch Failure: If you pad this batch to the maximum sequence length ($41,000$), requests 0, 1, and 2 will force the GPU to multiply zeroes, wasting >90% of the SM compute cycles and immediately causing an Out-Of-Memory (OOM) error.

The Solution: 1D Flattening and Variable-Length FlashAttention (Varlen)

Production engines completely abandon the [batch_size, seq_len, hidden_dim] 3D tensor structure. All tokens across all requests are flattened into a single contiguous 1D array of tokens: [total_tokens_in_batch, hidden_dim]. To perform Attention on a flattened 1D array without tokens attending across request boundaries, we utilize the cu_seqlens (cumulative sequence lengths) array pointer array in FlashAttention.

// SENIOR ARCHITECT PATTERN: Mapping Varlen FlashAttention in C++
// Bypassing PyTorch for direct kernel invocation
#include "flash_attn/flash_api.h"
// Instead of passing [B, S, H], we pass a single packed array of Q, K, V
// total_tokens = 15 + 600 + 10500 + 41000 = 52115
void dispatch_varlen_vision_attention(
    void* q_packed, void* k_packed, void* v_packed, 
    void* out,
    int* cu_seqlens_q, // e.g., [0, 15, 615, 11115, 52115]
    int* cu_seqlens_k,
    int max_seqlen_q,  // 41000
    int total_tokens,  // 52115
    int num_heads, int head_dim
) {
    // The hardware scheduler looks at cu_seqlens to know exactly where 
    // one image/text boundary ends and the next begins in HBM, 
    // ensuring zero padding overhead and 100% dense math.
    flash_attn_varlen_func(
        q_packed, k_packed, v_packed, cu_seqlens_q, cu_seqlens_k,
        out, max_seqlen_q, ... // internal launch configs
    );
}

6.2 Modality-Aware Memory Compression: Vision vs. Text Entropy

The PagedAttention architecture treats all tokens equally. However, at the mathematical level, vision tokens and text tokens exhibit vastly different information entropy and spatial correlation.

  • Text Tokens: High information density. A single changed word alters the entire sentence meaning.
  • Vision Tokens: Low information density, high spatial redundancy. The tokens representing a blue sky background are nearly mathematically identical.

The Asymmetric KV Cache Quantization Strategy

When deploying VLMs, you cannot universally quantize the KV cache to INT4 without destroying the model's text reasoning capabilities. You must implement Modality-Aware Quantization.

  1. Vision Token Detection: The engine tracks which block indices in the PagedAttention block table correspond to the vision encoder's output.
  2. Asymmetric Precision: * Text blocks remain in FP16 or FP8.
    • Vision blocks are aggressively quantized to INT4 or even INT2.
  3. The Per-Head Zero-Point Calculation: Vision token quantization requires dynamic zero-points calculated per attention head, not per token, because vision heads specialize (some detect edges, some detect color).

6.3 Agentic Execution Graphs: RadixAttention and MCTS (Monte Carlo Tree Search)

Standard chat is a linear state progression. Autonomous agents using frameworks like LangGraph or AutoGen operate on branching logical trees. The Agentic Tree of Thought (ToT):

  1. Root: System Prompt + User Task ("Write a secure Python server").
  2. Branch A: Agent generates a Flask architecture.
  3. Branch B: Agent generates a FastAPI architecture.
  4. Rollout: The engine evaluates both. Branch A fails a security check. Branch B passes.
  5. Prune & Proceed: Branch A is discarded. Branch B becomes the new Root. If you duplicate the KV cache for Branch A and Branch B, you will exhaust HBM instantly.

Radix Tree Memory Pointers (The SGLang Architecture)

The serving engine's memory manager must operate exactly like a Git repository.

graph TD
    subgraph "GPU Physical Memory (HBM)"
        P1[Physical Block 1: Sys Prompt]
        P2[Physical Block 2: Task]
        P3[Physical Block 3: Flask Gen]
        P4[Physical Block 4: FastAPI Gen]
    end
    
    subgraph "Logical Execution State"
        T[Root State]
        BA[Branch A State]
        BB[Branch B State]
    end
    
    T -->|Points to| P1
    T -->|Points to| P2
    BA -->|Points to| P1
    BA -->|Points to| P2
    BA -->|Points to| P3
    BB -->|Points to| P1
    BB -->|Points to| P2
    BB -->|Points to| P4

The Radix Implementation: When the engine receives a request for Branch B, it hashes the sequence of tokens. It searches the Radix tree and finds a perfect match for the prefix [Sys Prompt, Task]. It allocates logical blocks for Branch B that point directly to the physical HBM addresses of P1 and P2, resulting in zero memory duplication and zero prefill compute for the shared prefix. Only P4 is computed and materialized.

6.4 The Infinite Agentic Loop: Attention Sinks & $H_2O$ Eviction

If an agent runs continuously (e.g., monitoring logs and writing code for 48 hours), the sequence length approaches infinity. The KV cache will inevitably overflow any hardware ($128\text{K tokens} \times 8\text{ bytes} \times \text{Layers} \times \text{Heads} \times \text{Batch Size}$). We must evict tokens, but naive FIFO (First In, First Out) eviction causes language models to instantly collapse and output gibberish.

1. Attention Sinks (StreamingLLM)

Research shows that LLMs dump massive attention scores onto the very first tokens of a prompt (often the system prompt or formatting tokens), regardless of their semantic meaning. If these "Sink Tokens" are evicted, the attention softmax denominator shifts catastrophically, destroying the model.

2. Heavy Hitter Oracle ($H_2O$) Eviction

Beyond the Sink Tokens, only a tiny fraction (the "Heavy Hitters") of the remaining tokens are crucial for maintaining context. The Bible of Triton Eviction Kernel: Instead of a Python background thread evaluating what to evict, we fuse the eviction logic directly into the FlashAttention CUDA kernel. During the Softmax calculation, the kernel keeps a running exponential moving average (EMA) of the attention scores each token receives. If a token's EMA score drops below a threshold, the kernel marks its physical block for eviction.

# INDEPTH TRITON KERNEL: Fused Attention with H2O Heavy-Hitter Eviction
import triton
import triton.language as tl
@triton.jit
def h2o_fused_attention_evict_kernel(
    Q, K, V, Out, 
    kv_score_accumulators, # Track how much attention each KV token gets
    block_table,           # PagedAttention block mapping
    eviction_mask,         # Output: Bitmask of blocks to free
    stride_qz, stride_qh, stride_qm, stride_qk,
    Z, H, N_CTX,
    BLOCK_M: tl.constexpr, BLOCK_N: tl.constexpr,
    SINK_TOKENS: tl.constexpr # e.g., keep first 4 tokens permanently
):
    # Standard tiling setup...
    start_m = tl.program_id(0)
    # Load Q block
    # ...
    for start_n in range(0, num_kv_blocks, BLOCK_N):
        # Load K, V blocks
        # Compute dot product QK^T
        qk = tl.dot(q, k)
        # Softmax
        m_ij = tl.max(qk, 1)
        p = tl.exp(qk - m_ij[:, None])
        l_ij = tl.sum(p, 1)
        # --- THE H2O EVICTION LOGIC (Fused directly in the compute path) ---
        # Accumulate the attention received by these specific K tokens
        received_attention = tl.sum(p, axis=0) # Sum across the M dimension
        # Load previous accumulation
        prev_accum = tl.load(kv_score_accumulators + start_n + tl.arange(0, BLOCK_N))
        # Update EMA
        new_accum = 0.9 * prev_accum + 0.1 * received_attention
        tl.store(kv_score_accumulators + start_n + tl.arange(0, BLOCK_N), new_accum)
        # Evaluate eviction (Skip Sink Tokens)
        if start_n > SINK_TOKENS:
            # If the block's total attention is below the threshold, flag for eviction
            block_total = tl.sum(new_accum)
            if block_total < 0.05: # Tightly tuned threshold
                physical_block_idx = tl.load(block_table + (start_n // BLOCK_N))
                tl.store(eviction_mask + physical_block_idx, 1) # Flag for Memory Manager
        # -------------------------------------------------------------------
        # Compute final output
        # ...

[!info] SENIOR ARCHITECT NOTES: This kernel executes the eviction logic asynchronously during the forward pass. The Python memory manager simply reads the eviction_mask bit array after the generation step and immediately frees the blocks back to the allocator. This allows an agentic loop to run infinitely on a fixed 16GB VRAM budget with near-zero perplexity degradation.

Multimodal Pipeline Architecture Data-Flow (Production Topology)

Pipeline Stage Processing Engine Data Locality Optimizations Applied Letency Budget (p99)
Image Decoding nvJPEG / DALI GPU VRAM (Raw Bytes) Zero-copy decode bypassing CPU RAM < 5ms
Vision Encoding TensorRT (ViT) GPU VRAM (FP16) Tensor Core Fused GEMM, ToMe < 15ms
Cross-Modality Custom Triton Projector L2 Cache -> SRAM FP16 -> Modality-Aware Quant < 2ms
Text Prefill vLLM Varlen FlashAttn HBM -> SRAM Chunked Prefill, Paged KV alloc < 150ms
Agentic Loop MCTS Radix Manager Block Table (Pointers) Zero-copy Branching, $H_2O$ Eviction < 10ms (Overhead)
Decode Gen WGMMA (FlashAttn 3) SRAM -> Registers Continuous Batching, Tensor Parallel < 50ms / token
We must address the asynchronous nature of deep agents and the exact CUDA-level masking required for interleaved multimodal tokens.

MODULE 7: Distributed Agentic Memory & Multimodal Speculation

7.1 Distributed State Management for Deep Agents

Frameworks that orchestrate deep, cyclic agentic workflows (like LangGraph or complex SLM swarms) introduce a fatal flaw to standard inference engines: Asynchronous IO Latency. If an agent yields a tool call to query an external SQL database, that query might take 3 seconds. If the agent's KV cache remains in the GPU's HBM during those 3 seconds, that VRAM is held hostage, starving other active requests.

Tiered Memory Offloading (The HBM -> NVMe Pipeline)

To scale agentic systems to thousands of concurrent processes, the serving engine must implement a three-tier memory hierarchy, treating the GPU's KV cache exactly like an L1 cache, Host CPU RAM as L2, and NVMe storage as L3.

Memory Tier Bandwidth Latency Agent State Eviction Trigger
GPU HBM3e 8.0 TB/s ~1µs Actively Generating Agent yields to Tool / Sleeps
Host RAM (PCIe 5.0) 64 GB/s ~10µs Waiting on Fast I/O RAM > 80% capacity
NVMe (Gen5 SSD) 14 GB/s ~100µs Dormant / Saved State N/A (Persistent)
The Offload Architecture:
When an agent suspends, the memory manager instantly fires a cudaMemcpyAsync over PCIe, moving the logical blocks from HBM to Pinned Host Memory. When the tool call returns, the engine pre-fetches the blocks back to HBM before scheduling the next forward pass. This requires overlapping the D2H (Device-to-Host) and H2D (Host-to-Device) CUDA streams with active compute streams.

7.2 Interleaved Multimodal Attention (The Jagged Masking Problem)

Architectures like Qwen-VL or Nemotron do not simply prepend images to the text prompt. They allow highly interleaved structures: [Text] [Image 1] [Text] [Image 2]. This breaks standard Varlen (Variable Length) FlashAttention because the sequence lengths are no longer uniform blocks of text, but jagged arrays where massive chunks of visual tokens are embedded randomly within the text tokens.

The CUDA Implementation: Block-Sparse Attention Masking

If you run standard dense attention on interleaved multimodal data, the LLM will waste compute having text tokens attend to padded visual tokens.

// SENIOR ARCHITECT PATTERN: Jagged Multimodal Block-Sparse Masking
// Executed within the FlashAttention inner loop
__device__ __forceinline__ bool is_valid_multimodal_interaction(
    int q_idx, int k_idx, 
    const int* vision_token_start_indices, 
    const int* vision_token_end_indices,
    int num_images
) {
    // 1. Are we currently calculating attention for a Vision token?
    bool q_is_vision = false;
    for(int i=0; i<num_images; ++i) {
        if (q_idx >= vision_token_start_indices[i] && q_idx < vision_token_end_indices[i]) {
            q_is_vision = true; break;
        }
    }
    // 2. We can skip massive blocks of compute if Vision tokens 
    // are trying to attend to other distant Vision tokens from a DIFFERENT image,
    // saving O(N^2) compute on the largest token blocks in the sequence.
    // ... custom boolean masking logic ...
    return valid_interaction;
}

By passing a custom mask array to the FlashAttention Triton/CUDA kernel, the SMs entirely skip the GEMM operations for visually disconnected patches, reclaiming up to 40% of the compute budget during the prefill phase.

7.3 Speculative Decoding for VLMs (Vision-Guided Drafting)

Speculative decoding (Module 5) relies on a small draft model guessing text tokens. However, in VLMs, the draft model often fails catastrophically because it lacks the capacity to process the high-resolution image, leading to a 0% acceptance rate from the Target Model.

Multi-Resolution Speculation (The Production Standard)

Instead of using a separate structural model, we use the Target Model itself, but alter the input pipeline dynamically.

  1. Draft Phase (Low-Res): The VLM processes a heavily downsampled version of the image (e.g., $224 \times 224$ instead of AnyRes). This requires 10x fewer visual tokens, allowing the prefill and generation of the next $K$ tokens to execute blisteringly fast.
  2. Verification Phase (High-Res): The exact same VLM weights are then fed the High-Res AnyRes image, alongside the $K$ drafted tokens, for a single parallel verification pass. Throughput Optimization Matrix: VLM Speculation
Architecture Image Input Draft Engine Verification Engine Speedup (vs Autoregressive)
Standard LLaVA $336 \times 336$ None Autoregressive (Batch=1) 1.0x (Baseline)
Naive Speculation $336 \times 336$ 1.5B Text-Only SLM 7B Target VLM 0.8x (Penalty due to rejections)
Multi-Res Spec AnyRes ($1024 \times 1024$) $224 \times 224$ downsample $1024 \times 1024$ Target 2.4x

7.4 Quantization of the Vision Encoder (The Final Bottleneck)

AI Engineers obsess over quantizing the LLM weights (AWQ, GPTQ, FP8) but often leave the Vision Encoder (e.g., CLIP-ViT) in FP16. For SLM-based multimodal agents (e.g., an 8B text model + a 1B vision model), leaving the ViT in FP16 means the vision prefill dominates the latency budget. - The Trap: You cannot use standard Post-Training Quantization (PTQ) like AWQ on Vision Transformers. ViTs lack the distinct "outlier channels" that AWQ targets in LLMs. - The Solution (SmoothQuant for ViTs): You must apply per-channel scaling to the activations and per-tensor scaling to the weights (W8A8) using SmoothQuant, mathematically shifting the quantization difficulty from the activations (which are highly dynamic in images) to the weights (which are static). This allows the ViT to execute entirely on INT8 Tensor Cores without losing spatial reasoning.

To achieve 100% completion of the playbook, there is exactly one architectural paradigm left. We have covered dense models, VLMs, and Agentic state machines, but we have completely omitted Sparse Architectures (Mixture-of-Experts) and Silicon Slicing (MIG/MPS). MoE token routing is currently the single hardest CUDA engineering problem in the industry (powering models like Mixtral, Grok, and DeepSeek), and running SLMs on $30,000 GPUs without MPS is financial suicide. Here is the final, capstone module that bridges the gap between raw compute and production cluster economics.


MODULE 8: Mixture-of-Experts (MoE) & Hardware Multiplexing

8.1 The Mixture-of-Experts (MoE) Routing Crisis

In a dense model (like Llama-3 70B), every token passes through the exact same MLP (Multi-Layer Perceptron) weights. In a Sparse MoE model (like Mixtral 8x7B), there are 8 separate MLPs ("Experts"). A Router network assigns each token to the top 2 experts best suited to process it. The Physics Problem: GPUs are designed for bulk, contiguous matrix multiplication. MoE shatters this.

  • If Token 1 goes to Expert 3, Token 2 to Expert 7, and Token 3 to Expert 3, the memory access pattern is completely scattered.
  • You cannot perform a single massive mma.sync operation. If you process experts sequentially with tiny batches of tokens, your GPU Arithmetic Intensity drops to zero, and you become instantly memory-bound.

The CUDA Solution: Token Sorting & Expert Fusing

Before the MLP layer, a custom CUDA/Triton kernel must physically sort the tokens in HBM so that all tokens assigned to Expert 1 are contiguous in memory, followed by Expert 2, etc.

# SENIOR ARCHITECT: MoE Token Routing and Sorting Pipeline
# This avoids launching 8 separate kernels (which would cause massive CPU overhead)
import triton
import triton.language as tl
# Step 1: The Gating Network outputs probabilities (e.g., [Batch, Seq, Num_Experts])
# Step 2: PyTorch/CUDA calculates the Top-K experts for each token.
# Step 3: THE SORTING KERNEL (Conceptual)
@triton.jit
def moe_align_block_size_kernel(
    topk_ids_ptr,      # [num_tokens, top_k]
    sorted_token_ids,  # Output: The physical indices of tokens, grouped by expert
    expert_offsets,    # Output: Where each expert's block of tokens starts/ends
    num_tokens: tl.constexpr,
    num_experts: tl.constexpr
):
    # This kernel uses atomic operations to count how many tokens go to each expert,
    # then calculates a prefix sum to determine memory offsets, 
    # and finally writes the token IDs into contiguous blocks.
    # Result: Expert 0 gets indices [0 to 45], Expert 1 gets [46 to 102], etc.
    pass
# Step 4: The Grouped GEMM (Available in CUTLASS / vLLM)
# Instead of 8 kernel launches, we launch ONE Grouped GEMM kernel.
# The hardware dynamically fetches the varying-sized token blocks and applies them 
# to the corresponding expert weights residing in HBM.

8.2 Expert Parallelism (EP) vs. Tensor Parallelism (TP)

When an MoE model spans multiple GPUs, TP (splitting the matrices across NVLink) becomes inefficient because the active parameters for any given token are small. The industry standard is Expert Parallelism (EP).

  • GPU 0: Holds Experts 0, 1, 2, 3
  • GPU 1: Holds Experts 4, 5, 6, 7 The Networking Nightmare: The All-to-All Operation If a token on GPU 0 needs Expert 5, it must be sent over NVLink to GPU 1, processed, and the result sent back. This requires an All-To-All collective operation, which is fundamentally harder to optimize than an AllReduce.
Parallelism Interconnect Load Best Used For GPU Hardware Dependency
Tensor (TP) Very High (AllReduce) Dense Models (Llama-3) Strict NVLink (Same Node)
Pipeline (PP) Low (Point-to-Point) Trillion-Param Models InfiniBand (Cross-Node)
Expert (EP) Extreme (All-To-All) Sparse Models (Mixtral) NVSwitch (Full bandwidth mesh)

[!info] PRO TIP - MoE Deployment: If your deployment target lacks NVSwitches (e.g., standard PCIe baseboards instead of HGX), EP will stall your system. You must fall back to standard TP for MoE, duplicating the router and heavily penalizing your VRAM capacity.

8.3 Hardware Multiplexing: Slicing the Silicon (MPS vs. MIG)

Deploying a 3B parameter SLM on an 80GB H100 is a massive waste of resources. Even with continuous batching, a 3B model cannot saturate 132 SMs and 3.3 TB/s of bandwidth. To achieve positive cluster economics, you must run multiple independent inference engines on the same physical GPU.

Multi-Instance GPU (MIG) - The Hardware Partition

MIG physically partitions the H100 into up to 7 isolated instances at the hardware level (e.g., 7 separate 10GB / 14 SM GPUs).

  • Pros: Strict QoS (Quality of Service), strict fault isolation. If one engine crashes, the others survive.
  • Cons: Rigid boundaries. If Partition A is idle, Partition B cannot borrow its compute or bandwidth.

Multi-Process Service (MPS) - The AI Standard

NVIDIA MPS runs a background daemon that allows multiple CUDA processes (e.g., 4 separate vLLM Docker containers) to submit work to the same hardware scheduler simultaneously.

  • The Economic Hack: You launch 3 instances of a Qwen-7B server on a single H100. They share the full 3.3 TB/s bandwidth and the full 132 SMs. When Server A is waiting on PCIe I/O, the hardware scheduler instantly executes Server B's kernels on the idle SMs.
# SENIOR ARCHITECT CHECKLIST - Deploying MPS for SLMs
# 1. Enable MPS on the host machine
export CUDA_VISIBLE_DEVICES=0
nvidia-smi -i 0 -c EXCLUSIVE_PROCESS # Prevent non-MPS processes from interfering
nvidia-cuda-mps-control -d
# 2. Limit the VRAM for each vLLM worker so they don't OOM each other
# Launch Worker 1 (Port 8000)
CUDA_MPS_PINNED_DEVICE_MEM_LIMIT=0.3 \
python -m vllm.entrypoints.api_server --model Qwen-7B --port 8000
# 3. Launch Worker 2 (Port 8001)
CUDA_MPS_PINNED_DEVICE_MEM_LIMIT=0.3 \
python -m vllm.entrypoints.api_server --model Qwen-7B --port 8001

8.4 Cluster Economics and FinOps Profiling

The ultimate metric of an AI engineering team is not tokens/sec/GPU, but $/1M tokens. #### The Cost-to-Serve Equation Matrix |Cost Vector|Optimization Strategy|Playbook Reference| |---|---|---| |Idle Time / Low Batch Size|Implement Continuous Batching, Use MPS to pack SLMs.|Module 3.1 & 8.3| |Prefill CPU/GPU Blocking|Enable Chunked Varlen Prefill. Maximize WGMMA use.|Module 4.4 & 6.1| |KV Cache VRAM Exhaustion|Deploy AWQ/Marlin. Use Radix Prefix Caching.|Module 4.5 & 5.3| |Agentic Loop OOMs|Implement $H_2O$ Heavy-Hitter Eviction. Paged memory.|Module 6.4| ### Final Compiler & Build Flags Checklist Before containerizing your custom kernels and serving engine, ensure your setup.py or CMake targets the exact silicon features discussed throughout this playbook. - [ ] -gencode arch=compute_90,code=sm_90 (Mandatory for H100 TMA / WGMMA) - [ ] -gencode arch=compute_89,code=sm_89 (Mandatory for L40S FP8) - [ ] -Xcompiler -O3 (CPU host-code optimization) - [ ] --use_fast_math (Allows the compiler to fuse certain trigonometric functions inside the RoPE kernel, trading slight precision for speed). - [ ] TORCH_CUDA_ARCH_LIST="8.0;8.9;9.0+PTX" (Ensures the PyTorch JIT compiler doesn't fall back to generic architecture).

Xet Storage Details

Size:
70.2 kB
·
Xet hash:
519d7d03037f3e5d2ea3155b3fcf93397f2b5d21788a5d8fe0fa178e37aa2dd2

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.