README.md

metadata

license: other
license_name: mit-apache-and-xvfi-research
license_link: https://huggingface.co/Hawkynt/mediadroid-models/blob/main/LICENSE-XVFI
tags:
  - tflite
  - onnx
  - video-frame-interpolation
  - rife
  - dain
  - ncnn
  - xvfi
  - flavr
  - cain
  - realsr
  - android

MediaDroid Models

Pre-converted models for MediaDroid — an Android media player that plays movies and shows from local storage and network shares.

MediaDroid offers them only where a self-test in its Settings finds the device fast enough for playback; elsewhere the setting switches itself off again and says how long the device took. The ones too large to carry in the app (dain/, xvfi/, flavr/, cain/, realsr/) are fetched from here when they are switched on, each file checked against its SHA-256 digest.

Models

File	Size	Purpose	Source	Input	Output
rife-4.25-lite-576p.tflite	~29 MB	Frame interpolation — the picture halfway between two frames (RIFE 4.25 lite), TensorFlow Lite, float32	hzwer/Practical-RIFE (MIT)	img0, img1: 1×640×768×3 float32 [0,1] (NHWC)	middle: 1×640×768×3 float32 [0,1]
rife-4.25-lite-576p.onnx	~72 MB	The same network as ONNX (opset 17), before the TFLite conversion	Same as above	img0, img1: 1×3×640×768 float32 [0,1] (NCHW)	middle: 1×3×640×768 float32

768 × 640 is 576p (720 × 576) padded to a multiple of 128, as Practical-RIFE pads; other sizes can be resampled to it. The in-between is at t = 0.5.

How it was made, and checked

The 4.25.lite release's weights (flownet.pkl, under the same MIT licence as Practical-RIFE) were loaded as tensors only into the network written out from its published architecture (tools/rife_net.py), exported to ONNX and converted with onnx2tf. For the conversion — each checked to give the same result — the resampling uses sizes instead of scale factors, the identity grid and the time plane are constants, and the warp's border padding is clamped coordinates with zero padding (the same with align_corners).

Against PyTorch on a test pair, the TFLite in-between differs by 0.00008 on average; 0.08 % of values differ by more than 1/255. A float16 conversion came out wrong and is not offered. Steps in tools/CONVERSION.md.

On a Galaxy S20 (Snapdragon 865) one in-between takes about 1.7 s on the CPU, its fastest unit for this network — far from real time there; newer devices with stronger NPUs and GPUs are where it is meant to run.

DAIN for ncnn (dain/)

Frame interpolation weighing depth — DAIN (Depth-Aware Video Frame Interpolation, CVPR 2019) as nihui converted it for ncnn in dain-ncnn-vulkan (models/best), copied unchanged. Four networks, each as .param (structure) and .bin (weights), about 51 MB:

Network	Input	Output
depthnet	input: a frame, 3 planes BGR in 0..1	depth
flownet	input0, input1: two frames	flow (from the first to the second)
ctxnet	input: a frame	ctx
interpolation	both frames, both depths, both flows, flow0_w = t and flow1_w = 1 − t, both contexts	output_rectified: the in-between, 3 planes BGR in 0..1

Any size, padded to a multiple of 32 with the edges repeated. The networks call four layers of DAIN's own — dain.Correlation, dain.OpticalFlowWarp, dain.DepthFlowProjection, dain.FilterInterpolation — which dain-ncnn-vulkan implements, and MediaDroid after it (CPU and Vulkan).

Licences: DAIN by Wenbo Bao (LICENSE-DAIN, MIT), dain-ncnn-vulkan by nihui (LICENSE-dain-ncnn-vulkan, MIT).

File	Bytes	SHA-256
dain/depthnet.param	23625	e5b2824a8efd13fcc8875456df47f723682f3b38ebac897c7ad06c4450cd15c0
dain/depthnet.bin	10728564	4d063853f50ecb7835e9c28c04087f9256541a276478925d5b062e4ab8e07355
dain/flownet.param	17852	ce1e6c4e86e21de63e6e94ddf37d8bc1f01ae1a3d7a983b5513df338ccc54d91
dain/flownet.bin	22092552	f6d5f47e84d0b25d9fff06c5acb382bef8281097188fd3a240501e754e95e2cb
dain/ctxnet.param	1254	85fc9a1fbf2d398b02edceaae2081ff1716edb44afd980c4928dacb44856f929
dain/ctxnet.bin	313748	957faa2e2cd475648166f056f65963136d24e4ccfcc665e367d7868b2f56d3e4
dain/interpolation.param	9066	9475d3368d092e8074189c44baa11fd5e357bf930188797fb2b4b899333cfdc1
dain/interpolation.bin	18291116	485ca7ffb0eb86e9468e9841356c0af5c55070dde1901a3628ce98baf5d28b6d

XVFI for ncnn (xvfi/) — research and education only

Frame interpolation from flows found coarse to fine — XVFI-Net (Sim, Oh, Kim: XVFI, ICCV 2021, JihyongOh/XVFI), the Vimeo-trained weights (XVFInet_Vimeo_exp1_latest.pt, module scale 2, one coarser level), its convolution blocks written out for ncnn by tools/xvfi/export.py from tools/xvfi/xvfi_net.py — the network written out from its published architecture, the weights loaded as tensors only. About 11 MB, weights as half floats:

Block	Input	Output
extract	a frame, 3 planes RGB in 0..1	64 features at half size
context	64 features	the next coarser level's, at half that
bottom	both frames' coarsest features (128)	6: the flows both ways, their confidences
flow1	features and the other frame's warped to them (128)	64
flow2	both flow1 outputs and the coarser flows doubled (132)	6
flow3	features, warped features and the flows from t (260)	4: the refined flows from t
refine	features shuffled to full size, the frames, the warped frames, the flows (80)	4: occlusion, residual

What joins them — the backward warp, the complementary flow reversal (a forward warp with Gaussian weights), the blending — is not a network: xvfi_net.py does it in PyTorch, MediaDroid (app/src/main/cpp/xvfi.cpp) in C++. On the self-test's frames (a non-repeating texture moved 10 x 6 pixels) the in-between is off by 1.27 per value from the frame truly halfway — the same for the PyTorch network and the ncnn blocks — where a blend of the two is off by 29.68.

Terms: the XVFI repository's — "freely used for research and education only. Any commercial use should get formal permission first." Published here with the authors' permission for MediaDroid, under those terms (LICENSE-XVFI).

FLAVR for ncnn (flavr/)

Frame interpolation from four frames at once — FLAVR (Kalluri, Pathak, Chandraker, Tran: WACV 2023, tarun005/FLAVR, Apache-2.0), the 2x model (FLAVR_2x.pth), written out for ncnn by tools/flavr/export.py from tools/flavr/flavr_net.py — the network written out from its published architecture, the weights loaded as tensors only. About 84 MB, weights as half floats:

Network	Input	Output
body	4D: 3 channels x 4 frames x H x W, in 0..1, each channel's mean over the frames taken away	64 x 4 x H x W
head	256 x H x W: body's four times folded into channels, time major	3 x H x W, the mean still away

H and W multiples of 8. The in-between is that of the middle two frames. The 3D layers are written as 2D ones over the four frames — the frames either side of each stacked into its channels, the four frames one below the other with zero rows between, one ordinary 2D layer, the weights stored once — so ncnn runs them on its fast 2D paths and on the GPU. On the self-test's frames the in-between is off by 5.23 per value from the frame truly halfway (the PyTorch network 5.25, a blend 29.95); 1.5 s at 128 x 128 on a desktop CPU (46 s as ncnn's own 3D layers).

Changes (Apache-2.0, 4b): converted to ncnn as above — the 3D layers as 2D ones, their kernels' times as channels — weights as half floats. Licence: LICENSE-FLAVR.

CAIN for ncnn (cain/)

Frame interpolation by channel attention, without flow — CAIN (Choi et al., AAAI 2020, myungsub/CAIN, MIT) as nihui converted it in cain-ncnn-vulkan (MIT, models/cain), copied unchanged; about 86 MB. Inputs x.1, x.3: the two frames in RGB 0..1, each with its own per-channel mean taken away, the edges repeated to a multiple of 32; output 4070: the in-between halfway, the two means' average still to be put back. Licences: LICENSE-CAIN, LICENSE-cain-ncnn-vulkan.

RealSR for ncnn (realsr/)

Upscaling four times, trained on real-world degradations — RealSR (Ji et al., CVPR Workshops 2020, jixiaozhong/RealSR, Apache-2.0), its DF2K_JPEG model as nihui converted it in realsr-ncnn-vulkan (MIT, models-DF2K_JPEG/x4), copied unchanged; about 33 MB. Input data: RGB 0..1, the edges mirrored 10 pixels around; output output: four times the size, the picture starting 40 pixels in. Licences: LICENSE-RealSR, LICENSE-realsr-ncnn-vulkan.

Other models used by MediaDroid

Model	Purpose	Source
IFRNet-S (ncnn)	Frame interpolation, carried in the app	ltkong218/IFRNet (MIT) via ifrnet-ncnn-vulkan (MIT)
waifu2x CUnet, Real-CUGAN SE (ncnn)	Upscaling 2x, carried in the app	waifu2x-ncnn-vulkan, realcugan-ncnn-vulkan (MIT)
RIFE 4.25 lite (ncnn)	Frame interpolation at any size, carried in the app	TNTwise/rife-ncnn-vulkan (MIT, after nihui's rife-ncnn-vulkan)
Real-ESRGAN General x4v3 (TFLite, 8-bit)	Neural upscaling, 4×	qualcomm/Real-ESRGAN-General-x4v3 (BSD-3-Clause, from xinntao/Real-ESRGAN)

Licence

FLAVR and RealSR under Apache-2.0 (LICENSE-FLAVR, LICENSE-RealSR), CAIN under MIT (LICENSE-CAIN). RIFE and DAIN under MIT: LICENSE-RIFE (Copyright (c) 2021 hzwer), LICENSE-DAIN (Copyright (c) 2019 Wenbo Bao), LICENSE-dain-ncnn-vulkan (Copyright (c) 2020 nihui). XVFI for research and education only: LICENSE-XVFI.