| --- |
| license: apache-2.0 |
| license_link: https://huggingface.co/Qwen/Qwen3.5-4B-Base/blob/main/LICENSE |
| base_model: Qwen/Qwen3.5-4B-Base |
| base_model_relation: finetune |
| library_name: transformers |
| pipeline_tag: video-text-to-text |
| language: |
| - en |
| - zh |
| tags: |
| - video-understanding |
| - video-llm |
| - multimodal |
| - codec |
| - token-efficiency |
| - incremental-encoding |
| - vision-language |
| - video-question-answering |
| - pytorch |
| - qwen |
| extra_gated_prompt: >- |
| The NativeCodec repositories use manual approval. Weights, configs, tokenizers and |
| dataset archives are available after your access request is approved. Please describe |
| your intended use; the model card and dataset card remain readable without approval. |
| extra_gated_fields: |
| Full name: text |
| Affiliation: text |
| Intended use: text |
| --- |
| |
| # NativeCodec-4B-PredLen |
|
|
| **4B video MLLM aligned with NativeCodec's learned variable-length token predictor.** |
|
|
| This is the predicted-length variant of NativeCodec-4B: the language model is fine-tuned to consume visual tokens produced by the NativeCodec incremental encoder, and each P frame's suffix length is chosen at inference by the learned length predictor. Use it together with the codec checkpoints and code in [`Tennineee/NativeCodec`](https://huggingface.co/Tennineee/NativeCodec). |
|
|
| | Companion asset | Repository | |
| |:---|:---| |
| | Code, incremental encoder, prediction heads | [`Tennineee/NativeCodec`](https://huggingface.co/Tennineee/NativeCodec) | |
| | Fixed-ratio aligner for the same codec | [`Tennineee/NativeCodec-4B-FixedRatio`](https://huggingface.co/Tennineee/NativeCodec-4B-FixedRatio) | |
| | BallBench dataset | [`Tennineee/NativeCodec-BallBench`](https://huggingface.co/datasets/Tennineee/NativeCodec-BallBench) | |
|
|
| ## Access request |
|
|
| This repository is public but **gated with manual approval**. Anonymous visitors can read this model card and see the file list. Downloading files requires approval. |
|
|
| To request access, open the repository page, sign in, click **Agree and access repository** / **Request access**, and submit the form. Access is granted to individual users, not to organizations. Requests are reviewed manually by the maintainers without an automatic approval or a published turnaround guarantee; the Hub notifies you when a decision is made. A rejected request cannot be resubmitted from the same account on the Hub. |
|
|
| After approval, download the files with an authenticated session (the full model is about 9.34 GB / 8.70 GiB): |
|
|
| ```bash |
| hf auth login |
| hf download Tennineee/NativeCodec-4B-PredLen --local-dir NativeCodec-4B-PredLen |
| hf download Tennineee/NativeCodec --local-dir NativeCodec |
| ``` |
|
|
| The codec checkpoints required at inference (`weights/inc_enc_stage1_step18750.pt` and one `weights/pred_len_*.pt`) live in the code repository. The complete card of [`Tennineee/NativeCodec`](https://huggingface.co/Tennineee/NativeCodec) also covers access requests, mirror downloads and fine-grained token settings. |
|
|
| ## Files |
|
|
| Every file sits at the repository root, so the directory can be passed to `from_pretrained` directly. |
|
|
| | File | Size (bytes) | Content | |
| |:---|:---:|:---| |
| | `model.safetensors` | 9,319,821,272 | Qwen3.5-4B language model, aligned to NativeCodec tokens; the adapter is merged into the base weights (no separate LoRA checkpoint) | |
| | `config.json` | 3,161 | Qwen3.5-4B model config (architecture `Qwen3_5ForConditionalGeneration`) | |
| | `preprocessor_config.json` | 390 | image processor config, inherited from the base model | |
| | `video_preprocessor_config.json` | 385 | video processor config, inherited from the base model | |
| | `tokenizer.json` | 12,807,982 | tokenizer (vocabulary 248,320) | |
| | `tokenizer_config.json` | 16,710 | tokenizer settings, including the chat template reference | |
| | `vocab.json` | 6,722,759 | vocabulary | |
| | `merges.txt` | 3,353,259 | BPE merges | |
| | `chat_template.jinja` | 7,756 | chat template | |
| | `LICENSE-APACHE`, `NOTICE.md`, `README.md` | | license, attribution and this card | |
|
|
| The weights are a complete Qwen3.5-4B-sized model: about 4.66 B parameters in bfloat16 (the file size divided by two bytes), including the frozen visual tower inherited from the base model. NativeCodec does not route video through that visual tower; the codec front-end adds its own visual tokens to the language model input. |
|
|
| ## Loading |
|
|
| The checkpoint follows the Hugging Face Qwen3.5 layout, so the standard multimodal loader works once your environment provides Qwen3.5 support (the upstream config records `transformers_version: 4.57.0.dev0`): |
|
|
| ```python |
| from transformers import AutoModelForImageTextToText, AutoTokenizer |
| |
| tok = AutoTokenizer.from_pretrained("Tennineee/NativeCodec-4B-PredLen") |
| model = AutoModelForImageTextToText.from_pretrained( |
| "Tennineee/NativeCodec-4B-PredLen", |
| torch_dtype="bfloat16", |
| attn_implementation="flash_attention_2", |
| ) |
| ``` |
|
|
| Loading this directory with vanilla `transformers` gives the aligned multimodal language model itself. Video reasoning through NativeCodec additionally needs the incremental encoder, the length predictor and the codec front-end: |
|
|
| ```python |
| import sys |
| sys.path.insert(0, "NativeCodec/stage3_llm_sft") |
| from modeling_recodec_qwen35 import ReCodecQwen35ForConditionalGeneration |
| |
| model = ReCodecQwen35ForConditionalGeneration( |
| model_name_or_path="NativeCodec-4B-PredLen", |
| recodec_root="NativeCodec/stage3_llm_sft/recodec", |
| phase1_ckpt="NativeCodec/weights/inc_enc_stage1_step18750.pt", |
| phase2_ckpt="NativeCodec/weights/pred_len_delta08_step3150.pt", |
| video_pad_token_id=video_pad_id, # from the code repository's Stage 3 config |
| max_gop_frames=64, |
| max_frames=12288, |
| max_video_tokens=64000, |
| fixed_p_ratio=None, # learned length predictor |
| lora_cfg=None, # adapter already merged |
| ) |
| ``` |
|
|
| A complete runnable path, including frame sampling and prompt assembly, is provided by `stage3_llm_sft/demo_recodec_inference.py` in the code repository. |
|
|
| ## Two length predictors |
|
|
| The companion code repository ships two Stage 2 length predictors that share one architecture and are interchangeable: |
|
|
| | Checkpoint | Teacher tolerance | Behaviour | |
| |:---|:---:|:---| |
| | `weights/pred_len_delta00_step9000.pt` | delta = 0.0 dB | stricter reconstruction, more tokens per P frame | |
| | `weights/pred_len_delta08_step3150.pt` | delta = 0.8 dB | about 45% fewer tokens than delta = 0.0, recommended default | |
|
|
| The released checkpoints and results use the delta = 0.8 predictor unless stated otherwise. |
|
|
| ## Use, budget and results |
|
|
| On BallBench at 16 fps the aligned model reaches **28.2% accuracy** with predicted lengths, the highest of the open-source models compared. On TimeLens-Bench with a 64k visual-token budget the predicted-length variant reaches **54.4 mIoU** (TimeLens-Bench all) and the constant 1/16 suffix ratio reaches **56.4 mIoU**, against 50.7 for the Qwen3.5-4B base. The full comparison and the token-budget ablations are in the code repository's model card. |
|
|
| The visual-token budget is configured by the codec front-end, not by these weights; `max_video_tokens` in your config decides how many tokens the model sees. |
|
|
| ## License and attribution |
|
|
| The fine-tuned weights are distributed under the **Apache License, Version 2.0**, following the license of [Qwen/Qwen3.5-4B-Base](https://huggingface.co/Qwen/Qwen3.5-4B-Base). The full license text is in [LICENSE-APACHE](LICENSE-APACHE); [NOTICE.md](NOTICE.md) records the upstream attribution and the changes made here. The code in `Tennineee/NativeCodec` is released under the MIT License. |
|
|
| NativeCodec Team |
|
|
