# LapSRN x4

Single-image super-resolution with a Laplacian Pyramid Super-Resolution Network. One forward
pass produces both a 2x and a 4x image, because the pyramid reconstructs each scale from the
previous one and both intermediate results are exposed as outputs.

The model was originally distributed as a frozen TensorFlow graph (`LapSRN_x4.pb`, the model
used by opencv_contrib's `dnn_superres` module) and converted to ONNX for use with OpenCV's
DNN module.

## Model Details
- **Architecture**: LapSRN — Laplacian pyramid, progressive upsampling with `DepthToSpace`
  and recursive residual blocks (24 `Conv`, 4 `DepthToSpace`)
- **Input**: `IteratorGetNext:0`, **NHWC** `[1, H, W, 1]` — a single **luma (Y)** channel,
  float32 normalized to `[0, 1]`. `H`/`W` are dynamic; the network is fully convolutional.
- **Outputs** (both produced in one pass, NCHW):
  - `NCHW_output_2x` — `[1, 1, H*2, W*2]`
  - `NCHW_output_4x` — `[1, 1, H*4, W*4]`
- **Framework**: ONNX (converted from the TensorFlow frozen graph via tf2onnx, opset 13)
- **Original weights**: https://github.com/fannymonori/TF-LapSRN/tree/master/export
- **Paper**: Lai, Huang, Ahuja, Yang, *Deep Laplacian Pyramid Networks for Fast and Accurate
  Super-Resolution*, CVPR 2017 — https://arxiv.org/abs/1704.03915

> **Note on layout.** The input is NHWC while the outputs are NCHW (hence their names). This
> is not a mistake in the conversion — it is what `dnn_superres` feeds and reads, so the
> conversion preserves it. Do not use `blobFromImage` here; it would produce NCHW input.

### Colour handling

The network only sees the Y channel. To get a colour image, the demos follow exactly what
`dnn_superres` does (`preprocess_YCrCb` / `reconstruct_YCrCb`):

1. Convert BGR to YCrCb, scale to float `[0, 1]`.
2. Feed **only Y**, reshaped to `[1, H, W, 1]`.
3. Bicubically upscale Cr and Cb by the same factor.
4. Merge `(Y_hr, Cr_hr, Cb_hr)`, scale back to 8-bit, convert YCrCb to BGR.

## Usage

### Python
```bash
python demo.py --model lapsrn_x4_2026sep.onnx --image example_outputs/input_image.png --output-dir example_outputs
```
Writes `output_image_2x.png` and `output_image_4x.png`. Pass `--scale 2` or `--scale 4` to
run just one head.

### C++
The C++ demo runs inference with OpenCV's DNN module (default engine — no ONNX Runtime
needed). Adjust the OpenCV paths to your setup:
```bash
OCV=/path/to/opencv                          # OpenCV source tree
OCVBUILD=/path/to/opencv/build               # OpenCV build directory (generated headers + libs)
g++ -std=c++17 demo.cpp -o demo \
  -I$OCV/include \
  -I$OCV/modules/core/include \
  -I$OCV/modules/dnn/include \
  -I$OCV/modules/imgproc/include \
  -I$OCV/modules/imgcodecs/include \
  -I$OCVBUILD \
  -L$OCVBUILD/lib -Wl,-rpath,$OCVBUILD/lib -lopencv_dnn -lopencv_imgcodecs -lopencv_imgproc -lopencv_core
./demo --model lapsrn_x4_2026sep.onnx --image example_outputs/input_image.png --output-dir example_outputs
```

The Python and C++ demos agree to within 1–2 intensity levels on under 1% of pixels — NumPy
and OpenCV break rounding ties differently, which the YCrCb to BGR conversion mildly amplifies.

### With the dnn_superres module

This is the same file the module's own test loads, so it also works through the module API:

```cpp
DnnSuperResImpl sr;
sr.readModel("lapsrn_x4_2026sep.onnx");
sr.setModel("lapsrn", 4);

std::vector<Mat> outputs;
sr.upsampleMultioutput(img, outputs, {2, 4}, {"NCHW_output_2x", "NCHW_output_4x"});
```

## Conversion

Exported from the frozen TensorFlow graph with tf2onnx (opset 13) via
[convert_to_onnx.py](./convert_to_onnx.py). Requires `tensorflow`, `tf2onnx` and `onnx`.

```bash
python convert_to_onnx.py --pb LapSRN_x4.pb
```

Three details are specific to this model and are handled by the script:

- **The weights are stored quantized and have to be folded back to float.** The frozen graph
  went through TensorFlow's `quantize_weights` transform, so each of the 20 weight tensors is
  a `quint8` const plus a min/max pair, reconstituted at runtime by a `Dequantize` node in
  `MIN_FIRST` mode. tf2onnx has no mapping for `Dequantize`, so the script evaluates those
  nodes in a TF session and rewrites each one as a plain float32 const. This is why the ONNX
  (2.7 MB) is about 4x the size of the frozen graph (703 KB).
- **The graph is pruned to the two heads that are used.** The frozen graph carries a third
  head, `NCHW_output`, that nothing reads. `extract_sub_graph` on `NCHW_output_2x` /
  `NCHW_output_4x` drops it — that head and its `perm` const are the only two nodes removed,
  so no compute changes (329 to 327 nodes).
- **The output names are stripped of their `:0` suffix.** tf2onnx names ONNX outputs after TF
  tensors, so the heads come out as `NCHW_output_2x:0`, and
  `net.forward("NCHW_output_2x")` cannot resolve that. The script renames them after
  conversion.

Note that `shape_override` uses `None`, not `-1`, for the dynamic H/W dims: the frozen graph's
placeholder has unknown rank, and tf2onnx's shape inference passes the override straight to
`set_shape`, which rejects `-1`.

Re-running the script on `LapSRN_x4.pb` reproduces the shipped model: same opset, node count
and initializers, and **bit-identical inference output**. The only differences are tf2onnx's
auto-generated constant names (`const_fold_opt__611` and friends), whose numbering depends on
graph traversal order.

The input is deliberately left NHWC — `--inputs-as-nchw` is *not* used here, unlike the other
`dnn_superres` conversions.

## Example

| | |
|---|---|
| `example_outputs/input_image.png` | 256x256 (`butterfly.png`, the `dnn_superres` test image) |
| `example_outputs/output_image_2x.png` | 512x512, from `NCHW_output_2x` |
| `example_outputs/output_image_4x.png` | 1024x1024, from `NCHW_output_4x` |

## License
See [LICENSE](./LICENSE) — the model is released by the TF-LapSRN author under the
Apache License 2.0.