---
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3.5-4B-Base/blob/main/LICENSE
pipeline_tag: video-text-to-text
language:
- en
- zh
tags:
- video-understanding
- video-llm
- multimodal
- codec
- token-efficiency
- incremental-encoding
- vision-language
- video-question-answering
- pytorch
- qwen
extra_gated_prompt: >-
  The NativeCodec repositories use manual approval. Weights, configs, tokenizers and
  dataset archives are available after your access request is approved. Please describe
  your intended use; the model card and dataset card remain readable without approval.
extra_gated_fields:
  Full name: text
  Affiliation: text
  Intended use: text
---

# NativeCodec

**Encoding every video frame with variable-length causal visual tokens.**

NativeCodec is a causal visual encoder that feeds **every** video frame to a multimodal LLM at up to 16 fps without exceeding the visual-token budget. It does not sample frames or prune tokens after encoding. It learns **variable-length causal visual tokens**: full I-frame anchors plus compact P-frame suffixes whose length adapts to how much new information each frame carries relative to its history.

This repository holds the official code and the released codec checkpoints (`weights/`). The aligned 4B language models live in two companion model repositories, and the evaluation dataset lives in a companion dataset repository:

| Repository | Content |
|:---|:---|
| [`Tennineee/NativeCodec-4B-PredLen`](https://huggingface.co/Tennineee/NativeCodec-4B-PredLen) | 4B LLM aligned with the learned length predictor (dynamic suffix ratio) |
| [`Tennineee/NativeCodec-4B-FixedRatio`](https://huggingface.co/Tennineee/NativeCodec-4B-FixedRatio) | 4B LLM aligned for a constant P-frame suffix ratio (1/16 recommended) |
| [`Tennineee/NativeCodec-BallBench`](https://huggingface.co/datasets/Tennineee/NativeCodec-BallBench) | BallBench high-frame-rate identity-reasoning dataset |

The code is released under the MIT License. The released weights are distributed under Apache-2.0, inherited from the Qwen3.5-4B-Base base model. See [License](#license).

## News

| Date | Update |
|:---:|:---|
| 2026/09/23 | Initial release: two-stage codec training, downstream LLM alignment, five-benchmark evaluation pipeline, and **BallBench**. |

## Access request

All four repositories are public but **gated with manual approval**. Anonymous visitors can read this model card and the model cards of the companion repositories, and can see file names and sizes. Downloading any file other than the README requires access. Access is granted to individual users, not to organizations.

To request access, open the repository page on [huggingface.co](https://huggingface.co/Tennineee/NativeCodec), sign in, click **Agree and access repository** / **Request access**, and submit the form. The form collects your Hub username and primary email by default; the model card asks for your full name, affiliation and intended use. You also confirm that you will use the materials in accordance with the license terms.

Requests are reviewed manually by the maintainers. There is no automatic approval and no published turnaround guarantee. The Hub notifies you when a decision is made. A rejected request cannot be resubmitted from the same account on the Hub; if you believe the decision was a mistake, contact the maintainers through the repository page.

Once approved, install `huggingface_hub` and download with:

```bash
# Code and codec checkpoints (about 9.47 GiB)
hf download Tennineee/NativeCodec --local-dir NativeCodec

# Aligned LLM, predicted-length variant (about 8.7 GiB)
hf download Tennineee/NativeCodec-4B-PredLen --local-dir NativeCodec-4B-PredLen

# Aligned LLM, fixed-ratio variant (about 8.7 GiB, optional)
hf download Tennineee/NativeCodec-4B-FixedRatio --local-dir NativeCodec-4B-FixedRatio

# BallBench dataset (archive about 4.0 GB, optional)
hf download Tennineee/NativeCodec-BallBench --repo-type dataset --local-dir NativeCodec-BallBench
```

Every download above needs an authenticated session. Log in with a token that can see gated repositories:

```bash
hf auth login
# older huggingface_hub versions: huggingface-cli login
```

From mainland China, huggingface.co is often slow or unreachable. The commands also work through the community mirror, which proxies gated downloads after you log in:

```bash
HF_ENDPOINT=https://hf-mirror.com hf download Tennineee/NativeCodec --local-dir NativeCodec
```

The mirror does not host the login flow, so authenticate against huggingface.co first and only set `HF_ENDPOINT` for the download command. Use `--include` to fetch a subset, for example `--include "weights/*"`. If you use a fine-grained token, enable access to public gated repositories in the token settings, otherwise the Hub returns 403 for gated files even after approval.

## What is NativeCodec?

Video MLLMs face a dilemma: motion and identity evidence often lives in brief interactions that sparse frame sampling destroys. Raising the frame rate helps, yet encoding every frame independently floods the LLM with redundant tokens: the second frame of a static scene barely adds anything over the first, and still costs the same `H/32 x W/32` visual tokens.

The practical question is how much capacity each frame deserves given what came before. NativeCodec answers it with a predictive-coding-style interface:

- **Every admitted frame keeps its temporal position**, so transient events such as contacts, rebounds and identity swaps stay addressable.
- **Each P frame is encoded causally** against its history, and only a suffix of its representation is retained.
- **The suffix length is predicted per frame**, so static continuation costs a few tokens while scene changes cost many.
- **Anchor (I-frame) placement follows the stream itself**: once accumulated suffixes fill a frame's worth of budget, the next frame resets the causal context as a new anchor.

At inference the budget is enforced exactly: if a stream exceeds `max_video_tokens`, frames are uniformly subsampled across the clip and the stream is re-planned. A frame is never truncated mid-frame.

## Our solution

| Component | What it does |
|:---|:---|
| **Incremental Encoder** | A causal transformer over per-frame token grids (initialized from frozen LLM layers). Frame `t` attends to all history and itself, producing a full causal representation `H_t`. |
| **Truncatable Suffixes** | Train with random suffix ratios `{1, 1/2, ..., 1/64}`; pixel and feature reconstruction at every truncation length pushes each frame's residual information into the tail of its representation. |
| **Oracle-Guided Length Predictor** | An offline reconstruction teacher searches, per frame and in temporal order, the smallest ratio that stays within `delta` dB of full-ratio quality; a feed-forward head distills this search into one parallel pass. |
| **Adaptive GOP Stream** | Cumulative suffix budget determines anchor placement. Every retained token enters the LLM at its true `(t, y, x)` coordinates, with no pooling and no reordering. |
| **BallBench** | A new benchmark for high-frame-rate identity reasoning: 4 to 10 visually identical balls collide at 30 fps; answering requires tracking trajectories through collisions where appearance cues are useless. |

## Results

**Main comparison** (open-source video MLLMs, 64k visual-token budget; NativeCodec uses a constant suffix ratio of 1/16, **†** = predicted suffix lengths):

| Model | BallBench | TL-AN | TL-CH | TL-QV | TL-All | MS | VH | MME-v2 |
|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| Qwen3-VL-8B | 14.3 | 48.0 | 47.2 | 59.4 | 49.6 | 9.8 | 42.5 | **27.9** |
| Qwen3.5-9B | 14.7 | 49.9 | 45.6 | 64.0 | 50.7 | 12.8 | <u>46.3</u> | 26.0 |
| LLaVA-OV-2-8B | <u>17.1</u> | 53.8 | 53.5 | 66.4 | <u>55.8</u> | <u>19.6</u> | 32.5 | 19.9 |
| Qwen3.5-4B (base) | 12.2 | 49.4 | 46.0 | 64.6 | 50.7 | 18.5 | 39.5 | 24.1 |
| Qwen3.5-4B-SFT | 15.3 | 51.7 | 49.5 | 66.6 | 53.2 | 18.8 | 44.0 | 23.3 |
| **NativeCodec-4B** | <u>26.1</u> | **54.5** | **53.8** | **67.7** | **56.4** | **25.0** | 47.1 | 24.6 |
| **NativeCodec-4B†** | **28.2** | 52.7 | 51.4 | 66.2 | 54.4 | 24.7 | **48.7** | 23.7 |

Column keys: BallBench accuracy, TimeLens-Bench (TL) average/action (AN)/change (CH)/query (QV) mIoU, MomentSeeker (MS) mIoU, Video-Holmes (VH), Video-MME-v2 (MME-v2).

NativeCodec-4B is the best open-source model on TimeLens-Bench (56.4 mIoU) and every subset, best on MomentSeeker (25.0), best on Video-Holmes with predicted lengths (48.7), and +13.9 over its own Qwen3.5-4B base on BallBench. On BallBench every other open-source baseline sits within 4 points of the 13.9 random level.

**Tight budgets** show the largest margin over the native visual pathway of the same Qwen3.5-4B base (TimeLens-Bench mIoU):

| Visual-token budget | 4k | 8k | 16k | 32k | 64k |
|:---|:---:|:---:|:---:|:---:|:---:|
| Delta mIoU vs. native pathway | **+14.2** | +12.8 | +9.9 | +7.6 | +5.1 |

At 16k tokens NativeCodec reaches **54.4 mIoU**, above the **50.9** the native pathway reaches with four times the budget (64k).

**BallBench vs. frame rate** (accuracy, 490 questions; suffix ratio 1/16):

| Model | 16 fps | 8 fps | 4 fps | 2 fps |
|:---|:---:|:---:|:---:|:---:|
| Random | 13.9 | 13.9 | 13.9 | 13.9 |
| Qwen3.5-4B | 12.2 | 13.2 | 8.6 | 10.4 |
| LLaVA-OV-2-8B | 17.1 | 17.8 | 16.3 | 17.8 |
| **NativeCodec-4B** | **26.1** | **24.3** | **23.5** | **22.9** |

Every open-source baseline stays within four points of random at all frame rates. NativeCodec improves monotonically with temporal resolution: it admits high-frame-rate input and uses that temporal resolution.

## Repository structure

```
NativeCodec/
|-- README.md
|-- LICENSE                     # MIT, for the code
|-- LICENSE-APACHE              # Apache-2.0, for the released weights
|-- NOTICE.md                   # Qwen attribution and change statement
|-- requirements.txt
|-- weights/                    # released codec checkpoints (download from this Hub repository)
|   |-- inc_enc_stage1_step18750.pt
|   |-- pred_len_delta00_step9000.pt
|   `-- pred_len_delta08_step3150.pt
|-- stage1_recodec_pretrain/    # Stage 1: truncatable representations
|   |-- train_frame_token.py    #   training entry (pixel + feature reconstruction)
|   |-- config_real_phase1_4b.yaml  #   4B config (9 size x window curricula)
|   |-- run.sh
|   |-- recodec/                #   model / encoder / decoder / vision / rope / data
|   `-- analysis/               #   reconstruction sweeps and visualizations
|-- stage2_len_pred/            # Stage 2: oracle-guided length training
|   |-- train_len_pred.py       #   frozen-oracle sequential GT search + CE head
|   |-- config_*.yaml           #   delta = 0.0 / 0.8, multi-pooling and soft-RD variants
|   |-- PHASE2_TRAINING_PIPELINE_4B.md
|   |-- recodec/
|   `-- analysis/
|-- stage3_llm_sft/             # Downstream alignment (LLM trained, codec frozen)
|   |-- train.py                #   HF Trainer + DeepSpeed ZeRO-2 entry
|   |-- modeling_recodec_qwen35.py  # dynamic GOP + two-pass budget enforcement
|   |-- dataset.py              #   tensor-cache / decord loading, random fps
|   |-- recodec_budget.py       #   token-budget math (bisection frame planning)
|   |-- demo_recodec_inference.py   # minimal inference example
|   |-- configs/  deepspeed/  tools/
|   `-- start_predlen_tsfix150_training.sh
`-- stage4_benchmark_eval/      # Five-benchmark evaluation
    |-- evaluate_*.py           #   per-benchmark entry points
    |-- run_predlen_delta08_3150_5bench.sh   # serial 8-shard launcher (resumable)
    |-- watch_delta08_3150_5bench.py         # live leaderboard with subset breakdown
    |-- configs/                #   per-benchmark evaluation configs
    `-- EVAL_GUIDE.md
```

The code is identical to the source tree. `weights/` is added by the Hub upload; the Stage 1 and Stage 2 checks are released there so that the pipeline can be used without retraining.

## Released checkpoints

| File | Size (bytes) | Content |
|:---|:---:|:---|
| `weights/inc_enc_stage1_step18750.pt` | 7,907,749,717 | **Stage 1** incremental encoder, merge norm and reconstruction decoders (`ckpt["model"]`: `inc_enc`, `merge_norm`, `decoders`, `aux_predictors`) |
| `weights/pred_len_delta00_step9000.pt` | 1,132,687,741 | **Stage 2** length predictor trained with teacher tolerance delta = 0.0 (strict, more tokens) |
| `weights/pred_len_delta08_step3150.pt` | 1,132,687,677 | **Stage 2** length predictor trained with teacher tolerance delta = 0.8 (aggressive, about 45% fewer tokens) |

Both length predictors share the same architecture (7-class suffix-ratio head over the last spatial token of each P frame) and are interchangeable; delta only affects how aggressive the predicted compression is. The codec checkpoints are loaded by the codec front-end in `stage3_llm_sft/modeling_recodec_qwen35.py` and by the evaluation entry points in `stage4_benchmark_eval/`.

## Installation

```bash
# Create the environment (Python 3.11)
conda create -n nativacodec python=3.11 -y
conda activate nativacodec

# Install dependencies (PyTorch >= 2.4 with CUDA required)
pip install -r requirements.txt

# FlashAttention for the Qwen3.5 backbone
pip install flash-attn --no-build-isolation
```

## Data preparation

Training and evaluation rows are ShareGPT-style JSONL (`id`, `video_url`, `conversations`), one QA per line:

```json
{
  "id": "videoseek_268014_1",
  "video_url": "/path/to/videos/ytb_50l5HAWTjwE.mp4",
  "meta_info": {"source": "videoseek", "is_mcq": true},
  "conversations": [
    {"role": "user", "content": "<video>\nWhat is the final text shown in the video?\nOptions:\nA. ... D. ..."},
    {"role": "assistant", "content": "</think>\n\nTherefore, the final answer is: D"}
  ]
}
```

BallBench is available in the [`Tennineee/NativeCodec-BallBench`](https://huggingface.co/datasets/Tennineee/NativeCodec-BallBench) dataset repository; its card documents the `qa.jsonl` fields, the archive layout and the generation scripts.

Optionally pre-decode videos into a uint8 tensor cache (fps 16, long side <= 512, 32-aligned) for fast training I/O:

```bash
cd stage3_llm_sft
python tools/convert_video_tensor_cache.py --input /path/to/videos --output /path/to/video_tensor
```

## Quick start

Paths inside the configs (model directories, checkpoints, data roots) must be adapted to your machine first; every stage validates them at startup.

### 1. Use the released checkpoints

Download this repository and one of the companion LLM repositories (see [Access request](#access-request)). The codec front-end lives in `stage3_llm_sft/modeling_recodec_qwen35.py`; it takes the aligned LLM directory plus the two codec checkpoints:

```python
from modeling_recodec_qwen35 import ReCodecQwen35ForConditionalGeneration

model = ReCodecQwen35ForConditionalGeneration(
    # Released HF checkpoint: a complete Qwen3.5-4B model with the adapter merged
    model_name_or_path="NativeCodec-4B-PredLen",
    recodec_root="recodec",
    phase1_ckpt="NativeCodec/weights/inc_enc_stage1_step18750.pt",
    phase2_ckpt="NativeCodec/weights/pred_len_delta08_step3150.pt",
    video_pad_token_id=video_pad_id,
    max_gop_frames=64,
    max_frames=12288,
    max_video_tokens=64000,
    fixed_p_ratio=None,   # None = learned length predictor (PredLen variant)
    lora_cfg=None,        # released checkpoints have the adapter merged; do not wrap them again
)
```

The full inference path (frame sampling, codec encoding, prompt assembly, generation) is implemented in `stage3_llm_sft/demo_recodec_inference.py`. To run the demo on the released checkpoint, set `QWEN_DIR` to the downloaded `NativeCodec-4B-PredLen` directory, point `PHASE1_CKPT` and `PHASE2_CKPT` at the files under `NativeCodec/weights/`, pass `lora_cfg=None` in `build_model`, and skip the `load_safetensors` / `load_state_dict` block there (those lines load a Stage 3 trainer checkpoint, while the released files already contain the merged weights):

```bash
cd stage3_llm_sft
python demo_recodec_inference.py --video /path/to/video.mp4 --question "What happens in the video?"
```

For the fixed-ratio variant, load `Tennineee/NativeCodec-4B-FixedRatio` instead and pass `fixed_p_ratio=1/16`; the length predictor is then not used.

### 2. Training stages (optional)

The released checkpoints make inference possible without training. The original training pipeline is kept complete and runs on 8 GPUs.

**STAGE 1: truncatable representations** (needs the Qwen3.5-4B base):

```bash
cd stage1_recodec_pretrain
bash run.sh    # 8-GPU DDP; writes the Phase 1 checkpoint (inc_enc, decoders, aux)
```

**STAGE 2: oracle-guided length training** (needs the Phase 1 checkpoint):

```bash
cd stage2_len_pred
bash run.sh    # default: delta = 0.8, random fps {16,8,4,2}
```

**STAGE 3: downstream LLM alignment** (needs Phase 1 + Phase 2 checkpoints):

```bash
cd stage3_llm_sft
bash start_predlen_tsfix150_training.sh  # pred-len dynamic mode, 8-GPU DeepSpeed ZeRO-2

# Export an HF-format checkpoint for inference / evaluation
python tools/convert_full_ckpt_to_hf.py --ckpt <output>/checkpoint-150 --out <hf_dir>
```

### 3. Evaluation

```bash
cd stage4_benchmark_eval
bash run_predlen_delta08_3150_5bench.sh       # full chain: 01 to 05, 8 shards, 1 shard/GPU
bash run_predlen_delta08_3150_5bench.sh 02    # resume from stage 02

# Live progress and subset-level metrics while it runs
python watch_delta08_3150_5bench.py
```

The eval configs under `stage4_benchmark_eval/configs/` reference paths from the training cluster. Adapt `recodec.phase1_ckpt`, `recodec.phase2_ckpt`, `checkpoint_path`, `data_path` and `video_dir` to your environment. For BallBench, use the archive from `Tennineee/NativeCodec-BallBench`; the evaluation entry point is `evaluate_ballbench_predlen_hf.py`.

## Configuration

Key parameters (see each `config_*.yaml` for the full list):

| Parameter | Description | Default |
|:---|:---|:---:|
| `video.max_video_tokens` | Hard visual-token budget; oversize streams are re-planned with uniformly subsampled frames until they fit | `64000` |
| `recodec.phase2_ckpt` | Trained length predictor (Stage 2), defines the per-frame length policy | not set |
| `video.fixed_p_ratio` | Constant suffix ratio mode (`null` = use the learned length predictor) | `null` |
| `recodec.target_delta_db` | Stage-2 teacher quality tolerance in dB; larger values mean more aggressive compression | `0.8` |
| `video.video_fps` | Codec input frame rate | `16` |
| `video.max_gop_frames` | Max frames per GOP (anchor + suffixes) | `64` |
| `video.max_side` | Frame long side after resize (32-aligned) | `512` |
| `video.random_fps_choices` | Training-time fps augmentation pool | `[2,4,8,16]` |

## License

This repository contains two kinds of material with different licenses.

The **code** in `stage1_recodec_pretrain/`, `stage2_len_pred/`, `stage3_llm_sft/`, `stage4_benchmark_eval/` and the accompanying scripts is released under the MIT License; see [LICENSE](LICENSE).

The **released weights** in `weights/`, and the aligned models in the two companion repositories, are distributed under the Apache License, Version 2.0, following the license of [Qwen/Qwen3.5-4B-Base](https://huggingface.co/Qwen/Qwen3.5-4B-Base). The full text is in [LICENSE-APACHE](LICENSE-APACHE), and [NOTICE.md](NOTICE.md) records the upstream attribution for the Qwen3.5-4B-Base model and the changes made here (codec front-end added, language model fine-tuned). Redistribution must keep both license texts and the notice, and must state any further modifications.

Data materials (BallBench) carry their own terms; see the dataset card of [`Tennineee/NativeCodec-BallBench`](https://huggingface.co/datasets/Tennineee/NativeCodec-BallBench).

## Acknowledgements

- [Qwen](https://github.com/QwenLM) for the Qwen3.5 base multimodal LLM and its visual tower.
- TimeLens-Bench, MomentSeeker, Video-Holmes and Video-MME, the public video understanding benchmarks used in the evaluation.
- The open-source video-LLM community's HF-native inference tooling.

NativeCodec Team