+ CAIN and RealSR for ncnn (nihui's conversions, unchanged), fetched by MediaDroid on demand

59c0496 verified 11 days ago

10.6 kB

---
license: other
license_name: mit-apache-and-xvfi-research
license_link: https://huggingface.co/Hawkynt/mediadroid-models/blob/main/LICENSE-XVFI
tags:
  - tflite
  - onnx
  - video-frame-interpolation
  - rife
  - dain
  - ncnn
  - xvfi
  - flavr
  - cain
  - realsr
  - android
---
# MediaDroid Models

Pre-converted models for [MediaDroid](https://github.com/Hawkynt/MediaDroid) — an Android media player that plays
movies and shows from local storage and network shares.

MediaDroid offers them only where a self-test in its Settings finds the device fast enough for playback; elsewhere
the setting switches itself off again and says how long the device took. The ones too large to carry in the app
(`dain/`, `xvfi/`, `flavr/`, `cain/`, `realsr/`) are fetched from here when they are switched on, each file checked against its SHA-256 digest.

## Models

| File | Size | Purpose | Source | Input | Output |
|------|------|---------|--------|-------|--------|
| `rife-4.25-lite-576p.tflite` | ~29 MB | **Frame interpolation** — the picture halfway between two frames (RIFE 4.25 lite), TensorFlow Lite, float32 | [hzwer/Practical-RIFE](https://github.com/hzwer/Practical-RIFE) (MIT) | `img0`, `img1`: 1×640×768×3 float32 [0,1] (NHWC) | `middle`: 1×640×768×3 float32 [0,1] |
| `rife-4.25-lite-576p.onnx` | ~72 MB | The same network as ONNX (opset 17), before the TFLite conversion | Same as above | `img0`, `img1`: 1×3×640×768 float32 [0,1] (NCHW) | `middle`: 1×3×640×768 float32 |

768 × 640 is 576p (720 × 576) padded to a multiple of 128, as Practical-RIFE pads; other sizes can be resampled to it.
The in-between is at t = 0.5.

### How it was made, and checked

The 4.25.lite release's weights (`flownet.pkl`, under the same MIT licence as Practical-RIFE) were loaded as tensors
only into the network written out from its published architecture (`tools/rife_net.py`), exported to ONNX and
converted with onnx2tf. For the conversion — each checked to give the same result — the resampling uses sizes
instead of scale factors, the identity grid and the time plane are constants, and the warp's border padding is
clamped coordinates with zero padding (the same with `align_corners`).

Against PyTorch on a test pair, the TFLite in-between differs by 0.00008 on average; 0.08 % of values differ by
more than 1/255. A float16 conversion came out wrong and is not offered. Steps in `tools/CONVERSION.md`.

On a Galaxy S20 (Snapdragon 865) one in-between takes about 1.7 s on the CPU, its fastest unit for this network —
far from real time there; newer devices with stronger NPUs and GPUs are where it is meant to run.

## DAIN for ncnn (`dain/`)

**Frame interpolation weighing depth** — DAIN (Depth-Aware Video Frame Interpolation, CVPR 2019) as nihui converted
it for [ncnn](https://github.com/Tencent/ncnn) in [dain-ncnn-vulkan](https://github.com/nihui/dain-ncnn-vulkan)
(`models/best`), copied unchanged. Four networks, each as `.param` (structure) and `.bin` (weights), about 51 MB:

| Network | Input | Output |
|---------|-------|--------|
| `depthnet` | `input`: a frame, 3 planes BGR in 0..1 | `depth` |
| `flownet` | `input0`, `input1`: two frames | `flow` (from the first to the second) |
| `ctxnet` | `input`: a frame | `ctx` |
| `interpolation` | both frames, both depths, both flows, `flow0_w` = t and `flow1_w` = 1 − t, both contexts | `output_rectified`: the in-between, 3 planes BGR in 0..1 |

Any size, padded to a multiple of 32 with the edges repeated. The networks call four layers of DAIN's own —
`dain.Correlation`, `dain.OpticalFlowWarp`, `dain.DepthFlowProjection`, `dain.FilterInterpolation` — which
dain-ncnn-vulkan implements, and MediaDroid after it (CPU and Vulkan).

Licences: DAIN by Wenbo Bao (`LICENSE-DAIN`, MIT), dain-ncnn-vulkan by nihui (`LICENSE-dain-ncnn-vulkan`, MIT).

| File | Bytes | SHA-256 |
|------|-------|---------|
| `dain/depthnet.param` | 23625 | `e5b2824a8efd13fcc8875456df47f723682f3b38ebac897c7ad06c4450cd15c0` |
| `dain/depthnet.bin` | 10728564 | `4d063853f50ecb7835e9c28c04087f9256541a276478925d5b062e4ab8e07355` |
| `dain/flownet.param` | 17852 | `ce1e6c4e86e21de63e6e94ddf37d8bc1f01ae1a3d7a983b5513df338ccc54d91` |
| `dain/flownet.bin` | 22092552 | `f6d5f47e84d0b25d9fff06c5acb382bef8281097188fd3a240501e754e95e2cb` |
| `dain/ctxnet.param` | 1254 | `85fc9a1fbf2d398b02edceaae2081ff1716edb44afd980c4928dacb44856f929` |
| `dain/ctxnet.bin` | 313748 | `957faa2e2cd475648166f056f65963136d24e4ccfcc665e367d7868b2f56d3e4` |
| `dain/interpolation.param` | 9066 | `9475d3368d092e8074189c44baa11fd5e357bf930188797fb2b4b899333cfdc1` |
| `dain/interpolation.bin` | 18291116 | `485ca7ffb0eb86e9468e9841356c0af5c55070dde1901a3628ce98baf5d28b6d` |

## XVFI for ncnn (`xvfi/`) — research and education only

**Frame interpolation from flows found coarse to fine** — XVFI-Net (Sim, Oh, Kim: XVFI, ICCV 2021,
[JihyongOh/XVFI](https://github.com/JihyongOh/XVFI)), the Vimeo-trained weights (`XVFInet_Vimeo_exp1_latest.pt`,
module scale 2, one coarser level), its convolution blocks written out for ncnn by `tools/xvfi/export.py` from
`tools/xvfi/xvfi_net.py` — the network written out from its published architecture, the weights loaded as tensors only.
About 11 MB, weights as half floats:

| Block | Input | Output |
|-------|-------|--------|
| `extract` | a frame, 3 planes RGB in 0..1 | 64 features at half size |
| `context` | 64 features | the next coarser level's, at half that |
| `bottom` | both frames' coarsest features (128) | 6: the flows both ways, their confidences |
| `flow1` | features and the other frame's warped to them (128) | 64 |
| `flow2` | both `flow1` outputs and the coarser flows doubled (132) | 6 |
| `flow3` | features, warped features and the flows from t (260) | 4: the refined flows from t |
| `refine` | features shuffled to full size, the frames, the warped frames, the flows (80) | 4: occlusion, residual |

What joins them — the backward warp, the complementary flow reversal (a forward warp with Gaussian weights), the
blending — is not a network: `xvfi_net.py` does it in PyTorch, MediaDroid (`app/src/main/cpp/xvfi.cpp`) in C++. On the
self-test's frames (a non-repeating texture moved 10 x 6 pixels) the in-between is off by 1.27 per value from the
frame truly halfway — the same for the PyTorch network and the ncnn blocks — where a blend of the two is off by 29.68.

**Terms**: the XVFI repository's — "freely used for research and education only. Any commercial use should get
formal permission first." Published here with the authors' permission for MediaDroid, under those terms
(`LICENSE-XVFI`).

## FLAVR for ncnn (`flavr/`)

**Frame interpolation from four frames at once** — FLAVR (Kalluri, Pathak, Chandraker, Tran: WACV 2023,
[tarun005/FLAVR](https://github.com/tarun005/FLAVR), Apache-2.0), the 2x model (`FLAVR_2x.pth`), written out for ncnn
by `tools/flavr/export.py` from `tools/flavr/flavr_net.py` — the network written out from its published architecture,
the weights loaded as tensors only. About 84 MB, weights as half floats:

| Network | Input | Output |
|---------|-------|--------|
| `body` | 4D: 3 channels x 4 frames x H x W, in 0..1, each channel's mean over the frames taken away | 64 x 4 x H x W |
| `head` | 256 x H x W: body's four times folded into channels, time major | 3 x H x W, the mean still away |

H and W multiples of 8. The in-between is that of the middle two frames. The 3D layers are written as 2D ones over the
four frames — the frames either side of each stacked into its channels, the four frames one below the other with zero
rows between, one ordinary 2D layer, the weights stored once — so ncnn runs them on its fast 2D paths and on the GPU.
On the self-test's frames the in-between is off by 5.23 per value from the frame truly halfway (the PyTorch network
5.25, a blend 29.95); 1.5 s at 128 x 128 on a desktop CPU (46 s as ncnn's own 3D layers).

Changes (Apache-2.0, 4b): converted to ncnn as above — the 3D layers as 2D ones, their kernels' times as channels — weights as half floats. Licence: `LICENSE-FLAVR`.

## CAIN for ncnn (`cain/`)

**Frame interpolation by channel attention, without flow** — CAIN (Choi et al., AAAI 2020,
[myungsub/CAIN](https://github.com/myungsub/CAIN), MIT) as nihui converted it in
[cain-ncnn-vulkan](https://github.com/nihui/cain-ncnn-vulkan) (MIT, `models/cain`), copied unchanged; about 86 MB.
Inputs `x.1`, `x.3`: the two frames in RGB 0..1, each with its own per-channel mean taken away, the edges repeated to a
multiple of 32; output `4070`: the in-between halfway, the two means' average still to be put back. Licences:
`LICENSE-CAIN`, `LICENSE-cain-ncnn-vulkan`.

## RealSR for ncnn (`realsr/`)

**Upscaling four times, trained on real-world degradations** — RealSR (Ji et al., CVPR Workshops 2020,
[jixiaozhong/RealSR](https://github.com/jixiaozhong/RealSR), Apache-2.0), its DF2K_JPEG model as nihui converted it in
[realsr-ncnn-vulkan](https://github.com/nihui/realsr-ncnn-vulkan) (MIT, `models-DF2K_JPEG/x4`), copied unchanged; about
33 MB. Input `data`: RGB 0..1, the edges mirrored 10 pixels around; output `output`: four times the size, the picture
starting 40 pixels in. Licences: `LICENSE-RealSR`, `LICENSE-realsr-ncnn-vulkan`.

## Other models used by MediaDroid

| Model | Purpose | Source |
|-------|---------|--------|
| IFRNet-S (ncnn) | Frame interpolation, carried in the app | [ltkong218/IFRNet](https://github.com/ltkong218/IFRNet) (MIT) via [ifrnet-ncnn-vulkan](https://github.com/nihui/ifrnet-ncnn-vulkan) (MIT) |
| waifu2x CUnet, Real-CUGAN SE (ncnn) | Upscaling 2x, carried in the app | [waifu2x-ncnn-vulkan](https://github.com/nihui/waifu2x-ncnn-vulkan), [realcugan-ncnn-vulkan](https://github.com/nihui/realcugan-ncnn-vulkan) (MIT) |
| RIFE 4.25 lite (ncnn) | Frame interpolation at any size, carried in the app | [TNTwise/rife-ncnn-vulkan](https://github.com/TNTwise/rife-ncnn-vulkan) (MIT, after nihui's rife-ncnn-vulkan) |
| Real-ESRGAN General x4v3 (TFLite, 8-bit) | Neural upscaling, 4× | [qualcomm/Real-ESRGAN-General-x4v3](https://huggingface.co/qualcomm/Real-ESRGAN-General-x4v3) (BSD-3-Clause, from [xinntao/Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN)) |

## Licence

FLAVR and RealSR under Apache-2.0 (`LICENSE-FLAVR`, `LICENSE-RealSR`), CAIN under MIT (`LICENSE-CAIN`). RIFE and DAIN under MIT: `LICENSE-RIFE` (Copyright (c) 2021 hzwer), `LICENSE-DAIN` (Copyright (c) 2019 Wenbo
Bao), `LICENSE-dain-ncnn-vulkan` (Copyright (c) 2020 nihui). XVFI for research and education only: `LICENSE-XVFI`.

	---
	license: other
	license_name: mit-apache-and-xvfi-research
	license_link: https://huggingface.co/Hawkynt/mediadroid-models/blob/main/LICENSE-XVFI
	tags:
	- tflite
	- onnx
	- video-frame-interpolation
	- rife
	- dain
	- ncnn
	- xvfi
	- flavr
	- cain
	- realsr
	- android
	---

	# MediaDroid Models

	Pre-converted models for [MediaDroid](https://github.com/Hawkynt/MediaDroid) — an Android media player that plays
	movies and shows from local storage and network shares.

	MediaDroid offers them only where a self-test in its Settings finds the device fast enough for playback; elsewhere
	the setting switches itself off again and says how long the device took. The ones too large to carry in the app
	(`dain/`, `xvfi/`, `flavr/`, `cain/`, `realsr/`) are fetched from here when they are switched on, each file checked against its SHA-256 digest.

	## Models

	\| File \| Size \| Purpose \| Source \| Input \| Output \|
	\|------\|------\|---------\|--------\|-------\|--------\|
	\| `rife-4.25-lite-576p.tflite` \| ~29 MB \| Frame interpolation — the picture halfway between two frames (RIFE 4.25 lite), TensorFlow Lite, float32 \| [hzwer/Practical-RIFE](https://github.com/hzwer/Practical-RIFE) (MIT) \| `img0`, `img1`: 1×640×768×3 float32 [0,1] (NHWC) \| `middle`: 1×640×768×3 float32 [0,1] \|
	\| `rife-4.25-lite-576p.onnx` \| ~72 MB \| The same network as ONNX (opset 17), before the TFLite conversion \| Same as above \| `img0`, `img1`: 1×3×640×768 float32 [0,1] (NCHW) \| `middle`: 1×3×640×768 float32 \|

	768 × 640 is 576p (720 × 576) padded to a multiple of 128, as Practical-RIFE pads; other sizes can be resampled to it.
	The in-between is at t = 0.5.

	### How it was made, and checked

	The 4.25.lite release's weights (`flownet.pkl`, under the same MIT licence as Practical-RIFE) were loaded as tensors
	only into the network written out from its published architecture (`tools/rife_net.py`), exported to ONNX and
	converted with onnx2tf. For the conversion — each checked to give the same result — the resampling uses sizes
	instead of scale factors, the identity grid and the time plane are constants, and the warp's border padding is
	clamped coordinates with zero padding (the same with `align_corners`).

	Against PyTorch on a test pair, the TFLite in-between differs by 0.00008 on average; 0.08 % of values differ by
	more than 1/255. A float16 conversion came out wrong and is not offered. Steps in `tools/CONVERSION.md`.

	On a Galaxy S20 (Snapdragon 865) one in-between takes about 1.7 s on the CPU, its fastest unit for this network —
	far from real time there; newer devices with stronger NPUs and GPUs are where it is meant to run.

	## DAIN for ncnn (`dain/`)

	Frame interpolation weighing depth — DAIN (Depth-Aware Video Frame Interpolation, CVPR 2019) as nihui converted
	it for [ncnn](https://github.com/Tencent/ncnn) in [dain-ncnn-vulkan](https://github.com/nihui/dain-ncnn-vulkan)
	(`models/best`), copied unchanged. Four networks, each as `.param` (structure) and `.bin` (weights), about 51 MB:

	\| Network \| Input \| Output \|
	\|---------\|-------\|--------\|
	\| `depthnet` \| `input`: a frame, 3 planes BGR in 0..1 \| `depth` \|
	\| `flownet` \| `input0`, `input1`: two frames \| `flow` (from the first to the second) \|
	\| `ctxnet` \| `input`: a frame \| `ctx` \|
	\| `interpolation` \| both frames, both depths, both flows, `flow0_w` = t and `flow1_w` = 1 − t, both contexts \| `output_rectified`: the in-between, 3 planes BGR in 0..1 \|

	Any size, padded to a multiple of 32 with the edges repeated. The networks call four layers of DAIN's own —
	`dain.Correlation`, `dain.OpticalFlowWarp`, `dain.DepthFlowProjection`, `dain.FilterInterpolation` — which
	dain-ncnn-vulkan implements, and MediaDroid after it (CPU and Vulkan).

	Licences: DAIN by Wenbo Bao (`LICENSE-DAIN`, MIT), dain-ncnn-vulkan by nihui (`LICENSE-dain-ncnn-vulkan`, MIT).

	\| File \| Bytes \| SHA-256 \|
	\|------\|-------\|---------\|
	\| `dain/depthnet.param` \| 23625 \| `e5b2824a8efd13fcc8875456df47f723682f3b38ebac897c7ad06c4450cd15c0` \|
	\| `dain/depthnet.bin` \| 10728564 \| `4d063853f50ecb7835e9c28c04087f9256541a276478925d5b062e4ab8e07355` \|
	\| `dain/flownet.param` \| 17852 \| `ce1e6c4e86e21de63e6e94ddf37d8bc1f01ae1a3d7a983b5513df338ccc54d91` \|
	\| `dain/flownet.bin` \| 22092552 \| `f6d5f47e84d0b25d9fff06c5acb382bef8281097188fd3a240501e754e95e2cb` \|
	\| `dain/ctxnet.param` \| 1254 \| `85fc9a1fbf2d398b02edceaae2081ff1716edb44afd980c4928dacb44856f929` \|
	\| `dain/ctxnet.bin` \| 313748 \| `957faa2e2cd475648166f056f65963136d24e4ccfcc665e367d7868b2f56d3e4` \|
	\| `dain/interpolation.param` \| 9066 \| `9475d3368d092e8074189c44baa11fd5e357bf930188797fb2b4b899333cfdc1` \|
	\| `dain/interpolation.bin` \| 18291116 \| `485ca7ffb0eb86e9468e9841356c0af5c55070dde1901a3628ce98baf5d28b6d` \|

	## XVFI for ncnn (`xvfi/`) — research and education only

	Frame interpolation from flows found coarse to fine — XVFI-Net (Sim, Oh, Kim: XVFI, ICCV 2021,
	[JihyongOh/XVFI](https://github.com/JihyongOh/XVFI)), the Vimeo-trained weights (`XVFInet_Vimeo_exp1_latest.pt`,
	module scale 2, one coarser level), its convolution blocks written out for ncnn by `tools/xvfi/export.py` from
	`tools/xvfi/xvfi_net.py` — the network written out from its published architecture, the weights loaded as tensors only.
	About 11 MB, weights as half floats:

	\| Block \| Input \| Output \|
	\|-------\|-------\|--------\|
	\| `extract` \| a frame, 3 planes RGB in 0..1 \| 64 features at half size \|
	\| `context` \| 64 features \| the next coarser level's, at half that \|
	\| `bottom` \| both frames' coarsest features (128) \| 6: the flows both ways, their confidences \|
	\| `flow1` \| features and the other frame's warped to them (128) \| 64 \|
	\| `flow2` \| both `flow1` outputs and the coarser flows doubled (132) \| 6 \|
	\| `flow3` \| features, warped features and the flows from t (260) \| 4: the refined flows from t \|
	\| `refine` \| features shuffled to full size, the frames, the warped frames, the flows (80) \| 4: occlusion, residual \|

	What joins them — the backward warp, the complementary flow reversal (a forward warp with Gaussian weights), the
	blending — is not a network: `xvfi_net.py` does it in PyTorch, MediaDroid (`app/src/main/cpp/xvfi.cpp`) in C++. On the
	self-test's frames (a non-repeating texture moved 10 x 6 pixels) the in-between is off by 1.27 per value from the
	frame truly halfway — the same for the PyTorch network and the ncnn blocks — where a blend of the two is off by 29.68.

	Terms: the XVFI repository's — "freely used for research and education only. Any commercial use should get
	formal permission first." Published here with the authors' permission for MediaDroid, under those terms
	(`LICENSE-XVFI`).

	## FLAVR for ncnn (`flavr/`)

	Frame interpolation from four frames at once — FLAVR (Kalluri, Pathak, Chandraker, Tran: WACV 2023,
	[tarun005/FLAVR](https://github.com/tarun005/FLAVR), Apache-2.0), the 2x model (`FLAVR_2x.pth`), written out for ncnn
	by `tools/flavr/export.py` from `tools/flavr/flavr_net.py` — the network written out from its published architecture,
	the weights loaded as tensors only. About 84 MB, weights as half floats:

	\| Network \| Input \| Output \|
	\|---------\|-------\|--------\|
	\| `body` \| 4D: 3 channels x 4 frames x H x W, in 0..1, each channel's mean over the frames taken away \| 64 x 4 x H x W \|
	\| `head` \| 256 x H x W: body's four times folded into channels, time major \| 3 x H x W, the mean still away \|

	H and W multiples of 8. The in-between is that of the middle two frames. The 3D layers are written as 2D ones over the
	four frames — the frames either side of each stacked into its channels, the four frames one below the other with zero
	rows between, one ordinary 2D layer, the weights stored once — so ncnn runs them on its fast 2D paths and on the GPU.
	On the self-test's frames the in-between is off by 5.23 per value from the frame truly halfway (the PyTorch network
	5.25, a blend 29.95); 1.5 s at 128 x 128 on a desktop CPU (46 s as ncnn's own 3D layers).

	Changes (Apache-2.0, 4b): converted to ncnn as above — the 3D layers as 2D ones, their kernels' times as channels — weights as half floats. Licence: `LICENSE-FLAVR`.

	## CAIN for ncnn (`cain/`)

	Frame interpolation by channel attention, without flow — CAIN (Choi et al., AAAI 2020,
	[myungsub/CAIN](https://github.com/myungsub/CAIN), MIT) as nihui converted it in
	[cain-ncnn-vulkan](https://github.com/nihui/cain-ncnn-vulkan) (MIT, `models/cain`), copied unchanged; about 86 MB.
	Inputs `x.1`, `x.3`: the two frames in RGB 0..1, each with its own per-channel mean taken away, the edges repeated to a
	multiple of 32; output `4070`: the in-between halfway, the two means' average still to be put back. Licences:
	`LICENSE-CAIN`, `LICENSE-cain-ncnn-vulkan`.

	## RealSR for ncnn (`realsr/`)

	Upscaling four times, trained on real-world degradations — RealSR (Ji et al., CVPR Workshops 2020,
	[jixiaozhong/RealSR](https://github.com/jixiaozhong/RealSR), Apache-2.0), its DF2K_JPEG model as nihui converted it in
	[realsr-ncnn-vulkan](https://github.com/nihui/realsr-ncnn-vulkan) (MIT, `models-DF2K_JPEG/x4`), copied unchanged; about
	33 MB. Input `data`: RGB 0..1, the edges mirrored 10 pixels around; output `output`: four times the size, the picture
	starting 40 pixels in. Licences: `LICENSE-RealSR`, `LICENSE-realsr-ncnn-vulkan`.

	## Other models used by MediaDroid

	\| Model \| Purpose \| Source \|
	\|-------\|---------\|--------\|
	\| IFRNet-S (ncnn) \| Frame interpolation, carried in the app \| [ltkong218/IFRNet](https://github.com/ltkong218/IFRNet) (MIT) via [ifrnet-ncnn-vulkan](https://github.com/nihui/ifrnet-ncnn-vulkan) (MIT) \|
	\| waifu2x CUnet, Real-CUGAN SE (ncnn) \| Upscaling 2x, carried in the app \| [waifu2x-ncnn-vulkan](https://github.com/nihui/waifu2x-ncnn-vulkan), [realcugan-ncnn-vulkan](https://github.com/nihui/realcugan-ncnn-vulkan) (MIT) \|
	\| RIFE 4.25 lite (ncnn) \| Frame interpolation at any size, carried in the app \| [TNTwise/rife-ncnn-vulkan](https://github.com/TNTwise/rife-ncnn-vulkan) (MIT, after nihui's rife-ncnn-vulkan) \|
	\| Real-ESRGAN General x4v3 (TFLite, 8-bit) \| Neural upscaling, 4× \| [qualcomm/Real-ESRGAN-General-x4v3](https://huggingface.co/qualcomm/Real-ESRGAN-General-x4v3) (BSD-3-Clause, from [xinntao/Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN)) \|

	## Licence

	FLAVR and RealSR under Apache-2.0 (`LICENSE-FLAVR`, `LICENSE-RealSR`), CAIN under MIT (`LICENSE-CAIN`). RIFE and DAIN under MIT: `LICENSE-RIFE` (Copyright (c) 2021 hzwer), `LICENSE-DAIN` (Copyright (c) 2019 Wenbo
	Bao), `LICENSE-dain-ncnn-vulkan` (Copyright (c) 2020 nihui). XVFI for research and education only: `LICENSE-XVFI`.