cortexelus commited on
Commit
1804a18
·
verified ·
1 Parent(s): bb411c3

tensorRT: document the runtime LoRA / DoRA engine

Browse files
Files changed (1) hide show
  1. tensorRT/README.md +10 -0
tensorRT/README.md CHANGED
@@ -172,6 +172,16 @@ comparison — earlier builds of it degraded past about two minutes of audio bec
172
  RoPE angle was computed in BF16, where one ULP exceeds 2π at long sequence lengths; the
173
  published engine has the RoPE table baked in at FP32 precision.
174
 
 
 
 
 
 
 
 
 
 
 
175
  **`dit_fp8.trt` is sm_90 only** and must be built weakly-typed; strong typing breaks
176
  TensorRT's attention fusion. This is the *opposite* of the FP16-mixed rule, which requires
177
  `STRONGLY_TYPED` or the builder re-casts the FP32 islands.
 
172
  RoPE angle was computed in BF16, where one ULP exceeds 2π at long sequence lengths; the
173
  published engine has the RoPE table baked in at FP32 precision.
174
 
175
+ **`dit_fp16_lora.trt` serves LoRA / DoRA adapters at runtime.** Same DiT, plus a low-rank
176
+ branch on each of 229 linears whose operands are *network inputs* rather than weights — so an
177
+ adapter loads in milliseconds, several stack, and each keeps its own strength, with no rebuild.
178
+ It costs a flat ~4–6 ms per step (+13% at L=4096, +37% at 1292) and you pay that with an empty
179
+ stack too, so use `dit_fp16.trt` until an adapter is actually wanted. **sm_90 only** for now;
180
+ other architectures build it locally in ~8 min with
181
+ `scripts/lora/build_branch.py --model sa3-m --download`. The runtime, the adapter baking step
182
+ and the layer map all live in the stable-audio-3 repo under
183
+ `optimized/tensorRT/scripts/lora/` — this file is only the compiled plan.
184
+
185
  **`dit_fp8.trt` is sm_90 only** and must be built weakly-typed; strong typing breaks
186
  TensorRT's attention fusion. This is the *opposite* of the FP16-mixed rule, which requires
187
  `STRONGLY_TYPED` or the builder re-casts the FP32 islands.