| --- |
| license: bsd-3-clause-clear |
| language: |
| - en |
| --- |
| <!-- Copyright 2025 ByteDance Ltd. and/or its affiliates. |
| All rights reserved. |
| Licensed under the BSD 3-Clause Clear License (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| https://choosealicense.com/licenses/bsd-3-clause-clear/ |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
|
|
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted (subject to the limitations in the disclaimer |
| below) provided that the following conditions are met: |
|
|
| * Redistributions of source code must retain the above copyright notice, |
| this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright notice, |
| this list of conditions and the following disclaimer in the documentation |
| and/or other materials provided with the distribution. |
| * Neither the name of ByteDance Ltd. and/or its affiliates nor the names of its |
| contributors may be used to endorse or promote products derived from this |
| software without specific prior written permission. |
|
|
| NO EXPRESS OR IMPLIED LICENSES TO ANY PARTY'S PATENT RIGHTS ARE GRANTED BY |
| THIS LICENSE. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND |
| CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT |
| NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A |
| PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR |
| CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; |
| OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, |
| WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR |
| OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF |
| ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. --> |
|
|
| <div align="center"> |
|
|
| # **NEVC-1.0** <br>(EHVC: Efficient Hierarchical Reference and Quality Structure for Neural Video Coding) |
|
|
| <div align="center"> |
| <img src="./assets/performance.png" alt="Performance comparison" width="60%" style="max-width: 100%;" height="auto"> |
| </div> |
|
|
| </div> |
|
|
| <div align="left"> |
|
|
| ## 📝 Introduction |
| This repository provides the pretrained model weights for **NEVC-1.0**, which integrates contributions from **EHVC (Efficient Hierarchical Reference and Quality Structure for Neural Video Coding)** — one of the core components of the framework. |
| **EHVC** introduces a hierarchical reference and quality structure that significantly improves both compression efficiency and rate–distortion performance. |
| The corresponding code repository can be found here: [NEVC-1.0-EHVC](https://github.com/bytedance/NEVC). |
|
|
| Key designs of **EHVC** include: |
| - **Hierarchical multi-reference:** Resolves reference–quality mismatches using a hierarchical reference structure and a multi-reference scheme, optimized for low-delay configurations. |
| - **Lookahead mechanism:** Enhances encoder-side context by leveraging forward features, thereby improving prediction accuracy and compression. |
| - **Layer-wise quantization scale with random quality training:** Provides a flexible and efficient quality structure that adapts during training, resulting in improved encoding performance. |
|
|
| --- |
|
|
| ## 🔧 Models |
| EHVC uses two models: the intra model and the inter model. |
| - The **intra model** handles intra-frame coding. |
| - The **inter model** is responsible for inter-frame (predictive) coding. |
|
|
| ### Intra Model |
| The main contributions of NEVC-1.0 focus on inter coding. |
| For intra coding, we directly adopt the pretrained model **`cvpr2023_image_psnr.pth.tar`** from [DCVC-DC](https://github.com/microsoft/DCVC/blob/main/DCVC-family/DCVC-DC/checkpoints/download.py), without further training. |
|
|
| ### Inter Model |
| The inter model of NEVC-1.0 is provided at **`/models/nevc1.0_inter.pth.tar`**. |
| The architecture of the inter model is illustrated below: |
| |
| <div align="center"> |
| <img src="./assets/architecture.png" alt="Inter model architecture" width="50%" style="max-width: 100%;" height="auto"> |
| </div> |
| |
| --- |
| |
| ## 📊 Experimental Results |
| ### Objective Comparison |
| <div align="center"> |
| |
| **BD-Rate (%) comparison for PSNR** |
| Anchor: VTM-23.4 LDB. |
| All codecs tested with 96 frames and intra-period = 32. |
| |
| <img src="./assets/96F32G.png" alt="BD-Rate 96F32G" width="50%" style="max-width: 100%;" height="auto"> |
| |
| **Rate–Distortion curves** on HEVC B, HEVC C, UVG, and MCL-JCV datasets. |
| Tested with 96 frames and intra-period = 32. |
| |
| <img src="./assets/96F32G_curve.png" alt="RD curves 96F32G" width="80%" style="max-width: 100%;" height="auto"> |
| |
| **BD-Rate (%) comparison for PSNR** |
| Anchor: VTM-23.4 LDB. |
| All codecs tested with full sequences and intra-period = -1. |
| |
| <img src="./assets/allF-1G.png" alt="BD-Rate allF-1G" width="50%" style="max-width: 100%;" height="auto"> |
| |
| **Rate–Distortion curves** on HEVC B, HEVC C, UVG, and MCL-JCV datasets. |
| Tested with full sequences and intra-period = -1. |
| |
| <img src="./assets/allF-1G_curve.png" alt="RD curves allF-1G" width="80%" style="max-width: 100%;" height="auto"> |
| |
| </div> |
| |
| --- |
| |
| ## 📜 Citation |
| If you find **NEVC-1.0** useful in your research or projects, please cite the following paper: |
| |
| - **EHVC: Efficient Hierarchical Reference and Quality Structure for Neural Video Coding** |
| Junqi Liao, Yaojun Wu, Chaoyi Lin, Zhipin Deng, Li Li, Dong Liu, Xiaoyan Sun. |
| *Proceedings of the 33rd ACM International Conference on Multimedia (ACM MM 2025).* |
| |
| ```bibtex |
| @inproceedings{liao2025ehvc, |
| title={EHVC: Efficient Hierarchical Reference and Quality Structure for Neural Video Coding}, |
| author={Liao, Junqi and Wu, Yaojun and Lin, Chaoyi and Deng, Zhipin and Li, Li and Liu, Dong and Sun, Xiaoyan}, |
| booktitle={Proceedings of the 33rd ACM International Conference on Multimedia}, |
| year={2025} |
| } |
| ``` |
| |
| --- |
| |
| |
| ## 🙌 Acknowledgement |
| The intra model of this project is based on [DCVC-DC](https://github.com/microsoft/DCVC/blob/main/DCVC-family/DCVC-DC/checkpoints/download.py). |
