--- license: apache-2.0 language: - en library_name: pytorch pipeline_tag: image-to-image tags: - super-resolution - image-restoration - image-to-image - sisr - real-world-restoration - pytorch - onnx - ncnn - spandrel - chainner - edge - mobile datasets: - Phips/lucid-cc0-v2-hc-512 --- # NERVE **Norm-free Efficient Restoration for Various Edge devices** A lightweight super-resolution architecture that is simple to train and fast to run. Released models are **verified** with **traiNNer-redux** (training) and **ONNX Runtime** + **ncnn** (inference). The op set is standard, so other runtimes (**CoreML**, **TensorRT**, DirectML, ...) are expected to work too — but they are not tested here. Part of the **BODY** suite by **Philip Hofmann** — networks built to be real-world usable first, not benchmark-chasing. See [Relation to HEART](#relation-to-heart) for which one to pick. All NERVE models were trained on a single consumer GPU (**RTX 3060, 12 GB**) — this network trains and runs on ordinary consumer hardware, no server-class GPU required. **Links:** [NERVE](https://huggingface.co/Phips/NERVE) · [HEART](https://huggingface.co/Phips/HEART) · [NERVE demo](https://huggingface.co/spaces/Phips/nerve) · [HEART demo](https://huggingface.co/spaces/Phips/heart) · [convert a checkpoint](https://huggingface.co/spaces/Phips/nerve-convert) · [local app](https://github.com/Phhofm/nerve-app) · [train NERVE](https://github.com/Phhofm/nerve-train) · [train HEART](https://github.com/Phhofm/heart-train) > Short version: copy `nerve_arch.py` into traiNNer-redux, train with the sample > configs, then run the `.safetensors`, `.onnx`, or `.ncnn` models. One > architecture, one size, no fused/unfused pairs. Details below. ![NERVE 4x on anime — input | bicubic | release | OTF GAN](examples/ani_compare.png) *Input (as-is) | bicubic x4 | NERVE 4x release | NERVE 4x OTF GAN.* ![NERVE 4x on a noisy image](examples/noisy_compare.png) ![NERVE 4x on a Stable-Diffusion image](examples/stablediffusion_compare.png) --- ## Why NERVE? There are strong and established SR networks already. This is the honest case for when NERVE is worth your time — and when it is not. This project comes out of a trainer's frustration rather than a benchmark goal. After training hundreds of SISR models (see [Phhofm/models](https://github.com/Phhofm/models)), the recurring annoyance was networks that felt like *paper* networks: they chase leaderboard metrics with techniques that leave you juggling an unfused and a fused checkpoint, or whose dynamic-ONNX conversion is fragile. This network is the opposite of that — built so a trainer can pick it up, train it, and convert it to dynamic ONNX without headaches. **Strengths of NERVE** - **Real-world usability over metric chasing.** Trained on a CC0 dataset with the full Real-ESRGAN OTF degradation pipeline; the OTF GAN model restores degraded images, not just clean bicubic downsamples. - **One file, one variant.** A single readable architecture file and one model size. No S/M/L matrix, no fused-vs-unfused checkpoints, nothing to fuse before deploy — what you train is what you ship. - **Export just works.** Dynamic-shape ONNX (verified, ORT parity ~1e-5) and a clean ncnn conversion. Standard ops only (`Conv / Add / ReLU / DepthToSpace / Resize`). - **Boring, stable training.** Norm-free, bf16 AMP, no gradient clipping, a single Charbonnier loss for the base stage. - **Small and fast.** ~1.8M params — SPAN-class size but conv-only, suited to mobile, edge, web and game engines. - **Maintainable for years.** One op set, no exotic dependencies, ~85-line arch. **When to pick something else.** If the goal is maximum PSNR/LPIPS on academic benchmarks, or a large transformer, NERVE is not aiming there — use HEART, HAT, ATD, DAT2, or one of the community networks below. NERVE aims at simplicity, deployability and hassle-free training. **Credit where it is due.** None of this exists in a vacuum. SISR has many strong architectures and generous community members, for example HAT, ATD and DAT2, the community network work of [umzi](https://github.com/umzi2) and [Kim2091](https://github.com/Kim2091), fdat by [sharekhan](https://github.com/stinkybread), and [the-database](https://github.com/the-database) (traiNNer-redux, its benchmarks and its models). This list is far from complete — apologies to anyone not named; the SISR community has a great many people contributing a great deal. --- ## Quickstart ### As a user (just want to upscale / restore images) Pick a model and run it: | I want to... | Use | |---|---| | upscale a clean image 4x | `models/4x_NERVE_release.safetensors` | | upscale a clean image 2x | `models/2x_NERVE_release.safetensors` | | restore a blurry / noisy / JPEG-y image 4x | `models/4x_NERVE_OTF_gan.safetensors` | | run on-device / inside an engine | the `onnx/` or `ncnn/` files | - **ONNX Runtime** (verified): use `onnx/*.onnx` (dynamic H/W). TensorRT / DirectML should work too (same op set) but are untested here. - **ncnn** (verified, fp16): `ncnn/*.ncnn.param` + `*.ncnn.bin`. Feed images with `Mat.from_pixels`: ```python import ncnn net = ncnn.Net(); net.load_param("4x_NERVE_fp16.ncnn.param"); net.load_model("4x_NERVE_fp16.ncnn.bin") ex = net.create_extractor() ex.input("in0", ncnn.Mat.from_pixels(bgr_uint8, ncnn.Mat.PixelType.PIXEL_BGR2RGB, w, h)) out = ex.extract("out0")[1] # CHW float, 4x ``` (Use `from_pixels`; the ncnn Python binding's raw-array constructor misbehaves on large inputs. Outputs are fp16, so expect small deviations at hard edges.) - **chaiNNer**: load the `onnx/*.onnx` files directly. Loading the `.safetensors` via spandrel needs NERVE registered in your chaiNNer/spandrel build (not upstream yet). - Input: RGB, any size. Output: 2x or 4x. ### As a trainer (want to train your own) NERVE is a normal [traiNNer-redux](https://github.com/the-database/traiNNer-redux) architecture — no special steps. ```bash # 1. Add the architecture + its ICNR helper (auto-registers: traiNNer scans *_arch.py) cp nerve_arch.py /traiNNer/archs/ cp icnr.py /traiNNer/utils/ # ICNR init helper (not in upstream yet) # 2. Copy a training config and point it at your data cp configs/4x_NERVE_release.yml /options/ # -> edit dataroot_gt / dataroot_lq in that file # 3. Train cd python train.py -opt options/4x_NERVE_release.yml --auto_resume # 4. Export to optimized dynamic ONNX (official converter) cp configs/4x_NERVE_onnx.yml /options/ # -> set path.pretrain_network_g to your trained checkpoint python convert_to_onnx.py -opt options/4x_NERVE_onnx.yml ``` That is the whole workflow — 2 small files plus a config. The net is one file, the config is one file, the checkpoint is one file — **no fusion step, no paired fused/unfused weights.** (`icnr.py` is the only piece not in upstream traiNNer-redux yet; it is the ICNR initializer for the PixelShuffle head.) - `4x_NERVE_onnx.yml` exports dynamic-shape ONNX (opset 20) and can emit `fp32`, `fp16`, or `bf16` (`onnx.dtype`), with OnnxSlim optimization and a built-in PyTorch-vs-ONNX verify pass. - Prefer no extra dependencies? `scripts/export_dynamic.py` is a tiny standalone exporter (torch + onnxruntime only). **Start from a released pretrain** (skip the expensive early training) by setting `path.pretrain_network_g` in your training config to a file from `models/`: | training stage | warm-start from | |---|---| | `4x_NERVE_release.yml` | `models/4x_NERVE_release.safetensors` (or the 2x one with `strict_load_g: false`) | | `4x_NERVE_OTF_fidelity.yml` | `models/4x_NERVE_release.safetensors` | | `4x_NERVE_OTF_gan.yml` | your OTF fidelity checkpoint | **Training tips** (all NERVE models follow these): - bf16 AMP is fine; no gradient clipping needed (norm-free net). - `--auto_resume` is recommended for long runs. - For real-world input, follow the chain: clean release -> OTF fidelity -> OTF GAN. --- ## Models | File | Role | Notes | |---|---|---| | `models/2x_NERVE_release.safetensors` | **2x official pretrain** | clean bicubic, 31.84 dB / 0.9255 SSIM (Urban100) | | `models/4x_NERVE_release.safetensors` | **4x official pretrain** | clean bicubic, 25.09 dB / 0.7531 SSIM (Urban100) | | `models/4x_NERVE_OTF_fidelity.safetensors` | 4x OTF fidelity | restore degraded input; the pretrain for OTF GAN finetunes | | `models/4x_NERVE_OTF_gan.safetensors` | 4x OTF GAN | **sharpest real-world output** (topiq 0.430 / lpips 0.290, Urban100 OTF val) | | `models/2x_NERVE_OTF_gan.safetensors` | 2x OTF GAN | 2x restoration; **2x best** (PSNR 23.57 / SSIM 0.706 / TopIQ 0.576 / LPIPS 0.221, Urban100 OTF x2 val) | | `onnx/4x_NERVE_1x3xHxW_fp32_op20.onnx` | 4x release, dynamic ONNX | onnxslim-optimized, fp32 | | `onnx/2x_NERVE_1x3xHxW_fp32_op20.onnx` | 2x release, dynamic ONNX | onnxslim-optimized, fp32 | | `onnx/4x_NERVE_OTF_fidelity_1x3xHxW_fp32_op20.onnx` | 4x OTF fidelity, dynamic ONNX | onnxslim-optimized, fp32 | | `onnx/4x_NERVE_OTF_gan_1x3xHxW_fp32_op20.onnx` | 4x OTF GAN, dynamic ONNX | onnxslim-optimized, fp32 | | `onnx/2x_NERVE_OTF_gan_1x3xHxW_fp32_op20.onnx` | 2x OTF GAN, dynamic ONNX | onnxslim-optimized, fp32 | | `ncnn/4x_NERVE_fp16.ncnn.param` + `.bin` | 4x release, ncnn | fp16 weights | | `ncnn/2x_NERVE_fp16.ncnn.param` + `.bin` | 2x release, ncnn | fp16 weights | | `ncnn/4x_NERVE_OTF_fidelity_fp16.ncnn.param` + `.bin` | 4x OTF fidelity, ncnn | fp16 weights | | `ncnn/4x_NERVE_OTF_gan_fp16.ncnn.param` + `.bin` | 4x OTF GAN, ncnn | fp16 weights | | `ncnn/2x_NERVE_OTF_gan_fp16.ncnn.param` + `.bin` | 2x OTF GAN, ncnn | fp16 weights | `models/*.safetensors` are plain EMA state dicts (no wrapper) — load them with `nerve_arch.nerve(scale=N)`. All released models (2x/4x release, 4x OTF fidelity, 4x OTF GAN) are final. **Pretrain chain** (how the models relate): ``` 2x_NERVE_release ──→ 4x_NERVE_release ──→ OTF fidelity ──→ OTF GAN ``` The release models are the shared starting point for anyone training NERVE: warm-start from them (2x → 4x, or release → OTF fidelity → OTF GAN) to skip the expensive early training. --- ## Examples Real images are fed in **as-is** (not downscaled) and upscaled 4x. For each image there are two comparisons: - `examples/*_compare.png` — whole image at **full resolution**: **input** | **bicubic x4** | **NERVE 4x release** | **NERVE 4x OTF GAN**. - `examples/*_1to1.png` — a **1:1 native-pixel center crop**. The bicubic column is the baseline; the OTF GAN column is the released showcase model. The source inputs are in `examples/inputs/`. - `examples/realphoto_compare.png` (street scene) - `examples/ani_compare.png` (anime) - `examples/manga_compare.png` (manga) - `examples/text_compare.png` (text) - `examples/noisy_compare.png` (noisy input) - `examples/stablediffusion_compare.png` (AI-generated) ## Architecture ``` 5x5 conv stem → 24 × (Conv-ReLU-Conv + residual) → PixelShuffle → + bicubic(x) ``` - **~1.8M params**, pure convolution — **no attention, no normalization, no gating, no positional encoding, no reparameterization.** - A **bicubic input residual** means the net only learns what bicubic upscaling misses — that's why it stays small and trains fast. - **ICNR-initialized** PixelShuffle head (checkerboard-free by construction). - The op graph is trivial: `Conv, Add, ReLU, DepthToSpace, Resize`. That's the entire reason it exports to dynamic ONNX and converts to ncnn (verified) without custom ops or fused/unfused pairs; CoreML is expected but untested. ## What we actually tested Every design choice here came from a measured ablation, not a guess. The full development log — including dead ends and raw numbers — is in [`docs/ABLATIONS.md`](docs/ABLATIONS.md). Highlights: - **Normalization**: removing LayerNorm beat the normed baseline by **+1.15 dB** at ~2x the training speed -> norm-free. - **Depth/width sweep + capacity scaling**: quality improved with size; the sweep put the GAN quality knee at **1.80M** (0.69M -> 1.80M: lpips -0.015; 1.80M -> 2.39M: only -0.002 more). - **Dilation**: a dilated-conv A/B under the OTF recipe was **null** -> plain convs. - **Checkerboard**: diagnosed a PixelShuffle phase-lock, fixed with ICNR plus *keeping aliased (nearest-exact) LR*. An anti-aliased-LR + sub-pixel-jitter variant made thin lines *worse* and was reverted. - **OTF/GAN capability**: the small net trained productively under the full Real-ESRGAN OTF GAN recipe (no collapse). - **Runtime validation**: every released `.safetensors` loads (strict) and runs; every `.onnx` loads in ONNX Runtime; the ncnn models were run at multiple input sizes up to 720x720 (2880x2880 output). - **Deployability**: dynamic-ONNX export + ORT parity and ncnn conversion were verified at every stage (the numbers are in the log). - **Not tested here** (community welcome): TensorRT / DirectML *speed* and fixed-shape static exports, CoreML, and on-device benchmarks. Train more variants, benchmark, and share — the arch is standard ops only. ## Training data & license All released NERVE models were trained **only** on [`Phips/lucid-cc0-v2-hc-512`](https://huggingface.co/datasets/Phips/lucid-cc0-v2-hc-512), a **CC0** dataset — the released weights are clean to use commercially. We did **not** optimize for benchmark leaderboards. If you want higher PSNR/LPIPS numbers, training on larger academic datasets (DIV2K, Flickr2K, LSDIR, ...) is expected to score better; NERVE is a normal architecture, train it on whatever you like. ## Relation to HEART NERVE and HEART are the two members of the **BODY** suite (by Philip Hofmann): same philosophy (real-world usability > benchmark scores), different tiers. | | **HEART** | **NERVE** | |---|---|---| | size | ~16.7M params | ~1.8M params | | design | attention (FlashAttention-friendly) | pure convolution | | use when | you want the **best quality** and have the compute (desktop, server, high-end device) | you want **speed, small size, and dead-simple deployment** (mobile, edge, web, game engines) | | cost | heavier to run | a fraction of the cost | Rule of thumb: **HEART for maximum quality, NERVE for maximum practicality.** Both export to dynamic ONNX; NERVE additionally converts to ncnn cleanly because it has no attention ops. Side-by-side comparisons on the same inputs (`input | bicubic | both release models | both OTF GAN models`), each with a 1:1 crop: `examples/OST_009_nerve_vs_heart_*` and `examples/00003_nerve_vs_heart_*` (Real-ESRGAN test set), plus `examples/realphoto_nerve_vs_heart_*` and `examples/noisy_nerve_vs_heart_*`. --- ## Repository layout ``` nerve_arch.py the architecture (copy to traiNNer/archs/) icnr.py ICNR init helper (copy to traiNNer/utils/; not in upstream yet) configs/ example training configs (+ 4x_NERVE_onnx.yml for ONNX export) models/ pretrained checkpoints (see table above) onnx/ dynamic fp32 ONNX (opset 20, onnxslim-optimized) ncnn/ ncnn fp16 param/bin docs/ABLATIONS.md the full development/ablation log (what we tested) examples/ real-image comparisons (input vs bicubic vs release vs GAN) scripts/export_dynamic.py checkpoint → dynamic ONNX exporter (+ ORT verify) LICENSE Apache-2.0 ``` ## Help wanted (community welcome) The goal here is **real-world usability** — easy training, hassle-free dynamic ONNX, simple and robust to maintain — **not** leaderboard chasing. So NERVE's strength is *simplicity for trainers*; there's still plenty worth measuring. Pick anything below and open an issue/PR/discussion with your numbers. **Benchmarks** (deliberately not chased here) - Train on an academic dataset (DIV2K / Flickr2K / LSDIR) and compare against SPAN, SRVGGNetCompact, HAT, Real-ESRGAN (PSNR / SSIM / LPIPS). - Perceptual metrics (LPIPS / TopIQ / MUSIQ) on real-world degraded images. **Speed & deployment** (not benchmarked here) - TensorRT / DirectML latency + peak VRAM on the dynamic ONNX. - Static-shape ONNX (e.g. `1x3x256x256`) vs dynamic: speed + memory. - fp16 vs fp32 ONNX. - ncnn on-device (Android / iOS): latency + memory. - CoreML conversion + on-device test. - Mobile CPU (XNNPACK / ORT Mobile / ncnn). **Models & code** - More finetunes (illustration, manga, denoise, 1x restoration). - Architecture experiments — see [`docs/ABLATIONS.md`](docs/ABLATIONS.md) for what we already tested (and rejected). Small, measured contributions are welcome. ## License Apache-2.0 (this repository's code and pretrained weights). ## Update — 2026-09-22: speed benchmark (honest numbers) We ran a head-to-head **inference benchmark on an RTX 3060** using the traiNNer-redux benchmark protocol (input `1x3x480x640`, fp16, 50 runs), against the closest networks by parameter count: | model | params | PyTorch fp16 | TensorRT fp16 | ncnn (Vulkan GPU) | |---|---|---|---|---| | **NERVE 2x** | 1.78M | 90.0 ms | 35.3 ms | 117 ms | | SPAN_S 2x | 2.22M | 40.4 ms | **16.8 ms** | — | | PLKSR_tiny 2x | 2.35M | 110.1 ms | 380.6 ms | — | | compact 2x | 0.60M | 35.7 ms | 19.1 ms | — | **NERVE does not win on speed.** SPAN_S is ~2.1x faster at a similar parameter count, and compact is faster still. (PLKSR_tiny is slower in PyTorch and falls apart in ONNX/TensorRT because its large-kernel/CCM ops don't map, so NERVE does beat it there.) That is neither a surprise nor a problem for what NERVE is for. **NERVE is not a benchmark-chasing network.** It exists for ease of use and real-world usability: one variant, one file, no fused/unfused checkpoint pairs, and a checkpoint that converts to dynamic-shape ONNX, TensorRT and ncnn with no special handling. Speed is good enough, not the goal. Practical notes from the benchmark: - **TensorRT is the fastest runtime for NERVE** (~2.5x over PyTorch), and it wants a **static shape** — that is where static exports actually matter. - In ONNX Runtime, static vs dynamic made no measurable difference. - fp16 ≈ bf16. - **ncnn runs on the GPU via Vulkan** (117 ms vs 5.4 s on CPU), but that is the Python binding; native ncnn on a phone is the intended path, not a desktop race. Choose SPAN if you want maximum speed. Choose NERVE if you want the simplest thing to train, convert and ship. ## Update — 2026-09-25: 2x OTF GAN (and a patch/batch result) Added a **2x OTF GAN** restoration model: warm-started from the 2x OTF fidelity chain and finetuned with a **larger patch (`lq 128`) and batch (8)** for 150k iterations, then validated against the previous 2x GAN (lq96/batch4, 150k) on the same OTF-degraded Urban100 x2 set, full-frame: | model (2x, same val) | PSNR | SSIM | TopIQ | LPIPS | |---|---|---|---|---| | 2x release (clean) | 20.36 | 0.525 | 0.407 | 0.486 | | 2x OTF fidelity | **24.37** | **0.743** | 0.561 | 0.246 | | 2x OTF GAN (lq96/b4) | 23.46 | 0.700 | 0.571 | 0.224 | | **2x OTF GAN (lq128/b8)** | 23.57 | 0.706 | **0.576** | **0.221** | The larger patch + batch improved **every** metric, so this is the released 2x OTF GAN. (The tiled-vs-full-frame validation difference was measured and is negligible: ~0.007 dB PSNR.) ## Update — 2026-09-29: fairness A/B vs SPAN_S (training stability) We ran a controlled comparison against **SPAN_S**, a similar-sized network, to check NERVE's claim of being simple and stable to train. Two experiments, each with **identical configs for both networks** — only `network_g` differs, same data, same degradation, same losses, same schedule, both **from scratch**. **Read this fairly:** these are **short runs** (20k fidelity + 10k GAN on the clean test; 30k on the OTF test), so they measure *early convergence and stability*, not a fully-tuned best. SPAN_S was not tuned for itself, and NERVE has fewer parameters (1.80M vs 2.24M). The configs are public so you can reproduce or extend them. ### 1) Clean / bicubic — standard Urban100 x4 Training on standard bicubic HR/LR pairs; validation on the **standard `urban100/lr_x4`** set (no self-made degradation). Fidelity 20k → GAN 10k. | stage | NERVE (1.80M) | SPAN_S (2.24M) | difference | |---|---|---|---| | fidelity @20k (PSNR / SSIM) | **24.696 / 0.7368** | 23.991 / 0.7018 | **+0.70 dB / +0.035** | | GAN @10k (PSNR / SSIM) | **24.694 / 0.7369** (peak 24.79) | 23.898 / 0.6930 (peak 24.08) | **+0.80 dB / +0.044** | ### 2) OTF — real-world degraded Urban100 x4 Training and validation with the Real-ESRGAN-style degradation pipeline, 30k. | | NERVE (1.80M) | SPAN_S (2.24M) | difference | |---|---|---|---| | PSNR / SSIM @30k | **21.685 / 0.6266** | 21.413 / 0.6057 | **+0.27 dB / +0.021** | | colour bias across training | flat (~2.6 / 1.6 / 1.7) | early spike to **4.96**, drifts back up | steadier | ### Verdict * NERVE **converged faster and further** than SPAN_S in both settings, despite fewer parameters — and it was ahead at essentially every validation step. * **Training stability:** in the OTF run SPAN_S's per-channel colour bias swung noticeably (a spike to ~4.96 early, then a slow climb) while NERVE stayed flat. In the clean run SPAN_S's **GAN stage lost PSNR** (24.08 → 23.90) whereas NERVE's held (24.79 → 24.69). That matches our own experience that SPAN can be finicky with GAN finetuning. * **Caveat:** short runs and an untuned SPAN_S mean this is an indication, not a final verdict. It is, however, consistent across two independent setups. ### Evidence — clean / bicubic (standard Urban100 x4) Validation PSNR (left) and SSIM (right): ![clean validation](docs/stability/clean/validation.png) Training losses: ![clean losses](docs/stability/clean/losses.png) Per-channel colour stability — flat lines near 0 mean no colour cast develops: ![clean colour stability](docs/stability/clean/colour_stability.png) ### Evidence — OTF (real-world degraded) ![otf validation](docs/stability/otf/validation.png) ![otf losses](docs/stability/otf/losses.png) ![otf colour stability](docs/stability/otf/colour_stability.png) Raw numbers and the exact configs (reproduce it yourself): [`docs/stability/clean/configs`](docs/stability/clean/configs) · [`docs/stability/otf/configs`](docs/stability/otf/configs) · [`colour_bias.csv` (clean)](docs/stability/clean/colour_bias.csv) · [`colour_bias.csv` (otf)](docs/stability/otf/colour_bias.csv) ## Update — 2026-09-30: TensorRT builder options (how to run NERVE fastest) We swept TensorRT builder options for NERVE (via ONNX Runtime's TensorRT provider) and applied the **same options to the other networks** for a fair comparison. Input `1x3x480x640`, fp16, static shape, 50 runs, mean: | model | TRT default | TRT `opt5 + CUDA graph` | gain | |---|---|---|---| | NERVE 2x (1.78M) | 33.24 ms | **31.80 ms** (31 FPS) | +4.5% | | NERVE 4x (1.80M) | 38.43 ms | **33.82 ms** (30 FPS) | +13.6% | | SPAN_S 2x (2.24M) | 16.62 ms | 14.91 ms (67 FPS) | +11.5% | | compact 2x (0.60M) | 18.53 ms | 15.32 ms (65 FPS) | +20.9% | ![NERVE vs similar-sized networks — TensorRT fp16 speed](docs/nerve_speed.png) **Verdict (honest).** The fastest TensorRT configuration we found is **`trt_builder_optimization_level=5` plus CUDA graphs** — a free speedup for every network here. NERVE gains 4.5–13.6%; it does **not** gain more than the others, and it remains slower than SPAN_S and compact, as in the 2026-09-22 benchmark. TensorRT needs a **static shape**, so export one at the size you need. ### Exact command / config Export a static fp16 ONNX at your target size, then create the session like this: ```python import onnxruntime as ort sess = ort.InferenceSession( "4x_NERVE_OTF_gan_1x3x480x640_fp16.onnx", providers=[ ("TensorrtExecutionProvider", { "trt_fp16_enable": True, "trt_builder_optimization_level": 5, # cheapest single win "trt_cuda_graph_enable": True, # biggest win for small nets "trt_engine_cache_enable": True, # cache the built engine "trt_engine_cache_path": "trt_cache", "trt_timing_cache_enable": True, }), "CUDAExecutionProvider", ], ) ``` Notes from the sweep: * `trt_builder_optimization_level=5` alone was roughly neutral; the gain comes mostly from **CUDA graphs** (launch overhead matters for a net this small). * `trt_tactic_sources` made no difference here (and CUDNN/CUBLAS names need a `+`/`-` prefix). Left at default. * The first run builds the engine (can take ~1 min); the cache above makes subsequent starts instant. * TensorRT was driven through ONNX Runtime; the same option names apply to the `trtexec` equivalents (`--builderOptimizationLevel=5`, `--useCudaGraph`).