Commit History
-
Add bf16 medium DiT TRT engine (sm_90) — FMHA-fused, medium default (#1)
- Add bf16 medium DiT TRT engine (sm_90) — FMHA-fused, medium default (c8cb8ba88ddea67804bcacbbb9265afcbff03b6f) Co-authored-by: CJ Carr <cortexelus@users.noreply.huggingface.co>9a38efb
-
Add FP32 SAME-L + SAME-S TRT decoders
Cortexelus +1: Claude Opus 4.7 (1M context)Pair the existing FP32 DiT engines (sa3-{m,sm-music,sm-sfx}/dit_fp32.trt) with FP32 decoders so the consumer-side `--precision fp32` flag has a full FP32-throughout pipeline available, bit-equivalent to PyTorch eager. - same-l/dec_dynamic_fp32.trt — built by upcasting dec_dynamic_triton_swa.onnx initializers + value_info + Constant nodes + Cast.to from FP16 to FP32 (STRONGLY_TYPED). The Triton SWA plugin already runs on the FP32 side of the attention island, so the plugin contract is unchanged. ~1.7 GB. - same-s/dec_dynamic_fp32.trt — analogous FP32 engine for SAME-S. ~210 MB. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>773ac48 -
Decoder ONNX: clip+scale in FP32 before int32 cast
Cortexelus +1: Claude Opus 4.7 (1M context)The PCM-baked tail (audio → clamp → ×32767 → int32) ran in the engine's FP16/BF16 trunk, where 32767 is not representable (rounds up to 32768), causing the int32 output to exceed the int16 range. The consumer code then wrapped 32768 → -32768 on int16 downcast, producing audible clicks on loud transients. Fix: cast audio to FP32 before the clip/scale chain in both build_same_l_decoder.py and build_same_s_decoder.py producers. 32767 is exact in FP32. A trailing clamp(-32767, 32767) is added as belt-and-suspenders against any residual numerical drift. Affected: dec_dynamic_triton_swa.{onnx,trt} (SAME-L), dec_dynamic_bf16.{onnx,trt} (SAME-S). Verified: 0 samples outside [-32767, +32767] on real-music decode (was ~800 with the old engines). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>a429ffa
-
DiT engines: replace BF16 with FP16-mixed (FP32 islands) + add FP32 variants
Cortexelus +1: Claude Opus 4.7 (1M context)Drop dit_bf16.trt for all three SA3 DiTs (sm-music, sm-sfx, sa3-m). BF16 quantization error compounds over 8 pingpong sampling steps, drifting cos-sim vs PT FP32 from 0.99 (single step) to 0.81 (final latent) and producing audibly degraded output. Replace with dit_fp16mixed.trt (the canonical from now on): FP16 trunk with FP32 islands around every RMSNorm chain (Pow+ReduceMean+Sqrt+Mul), every attention Softmax, and the RoPE region (anything reachable from a Cast(to=FP32) feeding Cos/Sin/Einsum). 140 RMSNorms + 40 Softmaxes per sm-music block, more for medium. Built with STRONGLY_TYPED so TRT honors the explicit dtypes (no auto-promotion). Matches MLX's "FP16 with implicit FP32 reductions via fused kernels" recipe. Per-step cos-sim vs PT FP32: 0.99997 single-step, 0.998 over 8 steps. RMS-curve correlation: 0.998. Audio basically indistinguishable from FP32 PyTorch reference at 1/15 the wall time (43 ms vs 630 ms). Also adds dit_fp32.trt (1.8 GB sm-*, 5.8 GB medium) for users who wan…97651ef
-
DiT TRT engines: rebuild with profile min=1 (sm_90)
Cortexelus +1: Claude Opus 4.7 (1M context)Lowered the dynamic-shape lower bound on `x` and `local_add_cond` from 256 → 1 for all three DiT engines (sa3-m, sa3-sm-music, sa3-sm-sfx). This unlocks short-form output below 24 s (down to ~93 ms at L=1) at zero measured perf cost — TRT picks identical tactics across [1, 4096] when opt stays at 1292. Output quality below the trained range (L<256, ~23.8 s) is undefined but the engine runs. Also refreshes tokenizer.json content for the T5Gemma engine. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>5f27021
-
T5Gemma: FP16-mixed (FP32 attention island) — fixes BF16 numerical bug
CortexelusThe previous BF16 build had a numerical bug: cos similarity vs PyTorch FP32 reference dropped to 0.17 at specific tokens (the ' beautiful' token id 4964 output activations of magnitude 26 instead of 52). Some tokens silently produced bad conditioning for downstream DiT. Switch to FP16 trunk + FP32 attention island (STRONGLY_TYPED network), the same strategy that fixed SAME-L decoder's accuracy. Cos vs FP32 PyTorch is now 0.999998; the worst-token cos across our test prompts is 0.9987. Engine filename: t5gemma_bf16.trt → t5gemma_fp16mixed.trt. Engine size: 564 MB → 623 MB (+10%). Latency: 0.78 ms → 0.91 ms (negligible vs DiT's 50+ ms). ONNX: re-exported with mixed dtypes; STRONGLY_TYPED build respects them. Consumer code (sa3_trt.py, install.sh, build.py, build_from_onnx.py) updated in the github repo to point at the new filename.23f5c64 -
Decoder TRT engines: PCM-baked-in (sm_90)
CortexelusCompanion to the previous commit's ONNX update — the published sm_90 .trt engines for SAME-S and SAME-L decoders now have the same clamp+scale+ cast(int32)+permute postprocess baked into the engine graph. Drop-in compatible: sa3_trt.py auto-detects engine flavor by output tensor name (pcm vs audio) and skips Stage 5 clip/cast/transpose when pcm-baked. Stage 5 wall time drops from ~110ms to ~21ms on sm-music + same-s, 30s audio.106e6fd
-
Drop L-range suffix from SAME-L decoder filename
CortexelusRename: tensorRT/sm_90/same-l/dec_dynamic_triton_swa_min32_opt1292_max4096.trt → tensorRT/sm_90/same-l/dec_dynamic_triton_swa.trt The (min,opt,max) L profile is baked into the engine itself; the verbose filename was a historical leftover from when only the decoder was dynamic. The encoder + other dynamic engines never had it. Now they all match.633bd26 -
Reorganize tensorRT/ into per-arch subdirs; rebuild DiT engines with conditioner baked in
Cortexelus- Move all engines under tensorRT/sm_90/ (room for sm_100, sm_120 later) - Co-locate T5Gemma tokenizer.json with the engine at tensorRT/sm_90/t5gemma/ - DiT engines now bundle per-DiT padding_embedding + seconds_total Linear as graph Constants, eliminating the runtime/padding_embeddings/ and runtime/num_conditioner.* sidecar files08c1abe -
Remove padding_embedding.pt — these ship with the inference code repo
CortexelusPer-DiT learned padding embeddings (4 KB each) are constants tied to the model architecture but tiny. Moving them out of this engines repo and into the inference code repo's runtime/padding_embeddings/ alongside the tokenizer + num_conditioner.e8c6e3e -
Remove pipeline_state from tensorRT/ — it ships with the inference code repo, not here
CortexelusThe TRT engines repo only holds engine binaries. The small config + tokenizer files (pipeline_state/cond_routing.json, dist_shift_config.json, model_info.json, num_conditioner.{pt,npz}, num_conditioner_config.json, tokenizer/*) live alongside sa3_trt.py in the inference code repo since they are required by the runtime code and tiny (~35 MB including the tokenizer).f6ada03 -
Add pipeline_state + per-model padding_embedding for SA3 TRT pipeline
CortexelusRequired runtime support for the TRT engines: - pipeline_state/tokenizer/ T5Gemma BPE tokenizer - pipeline_state/num_conditioner.* seconds_total conditioner (768D) - pipeline_state/dist_shift_config DistributionShift schedule warp (base_shift=0.5, max_shift=1.15) - pipeline_state/{cond_routing,model_info}.json pipeline metadata - sa3-{m,sm-music,sm-sfx}/padding_embedding.pt per-DiT learned padding embedding for T5 cross-attn267e477 -
Add SA3 TRT engines (LFS-tracked)
Cortexelus8 production TRT engines, ~7.5 GB total: - t5gemma_bf16.trt 539 MB text encoder - sa3-m/dit_bf16.trt 2.8 GB medium DiT (dynamic L=[256, 4096]) - sa3-sm-music/dit_bf16.trt 892 MB small music DiT - sa3-sm-sfx/dit_bf16.trt 892 MB small sfx DiT - same-l/enc_dynamic_triton_swa.trt 1.2 GB SAME-L encoder (FP16-mixed) - same-l/dec_dynamic_triton_swa_min32_opt1292_max4096.trt 1.2 GB SAME-L decoder - same-s/enc_dynamic_bf16.trt 107 MB SAME-S encoder (dynamic L=[32, 4096]) - same-s/dec_dynamic_bf16.trt 110 MB SAME-S decodere3a5ef0 -
Create tensorRT/README.md
Verified signature333a4ea