thaddeusk/Cosmos3-Edge-NVFP4-Reasoner-FP8-Generator

Cosmos3-Edge: calibrated NVFP4 reasoner + FP8 generator

One joint original-style checkpoint, not two independent models. The root tensor index contains both towers; the same model directory serves understanding with standard vLLM and image/video generation with vLLM-Omni.

This community quantization is experimental. NVIDIA officially tests Edge in BF16 only. The original model card, OpenMDW-1.1 license and NVIDIA notices are retained. Modifications: calibrated quantization, layout-preserving indices, mixed-precision metadata and the isolated runtime compatibility launcher.

Precision and validation

• Source: nvidia/Cosmos3-Edge, revision 344d602b128d1bbdacb43b08d0a3626f46343e29.
• Quantization framework: 7cfaf7854b2114eda5deb36250a7e8f00c64d69e.
• Understanding: 168 NVFP4 projections, native W4A4.
• Generation: 168 FP8 projections, native W8A8, with A16 activations on the first/last three diffusion iterations.
• Vision, projector, embeddings, vocabulary head and sensitive modules remain BF16. 502 shared tensor payloads were byte-verified unchanged.
• 2,336 indexed tensors / 4,282,869,184 indexed tensor bytes. The package also contains the VAE and other original pipeline components.
• Final root: 85/115 final-answer evaluation cases, with all 112 fused native NVFP4 linears audited after real inference. Matched BF16: 82/115; original standalone NVFP4 run: 85/115.
• Live joint pipeline: 168 native NVFP4 + 168 native FP8 linears, no Marlin or BF16 weight fallback.
• Safety-enabled generation: decoded 256 x 256 image and 53-frame 256 x 256 video at 24 FPS, using 12 diffusion iterations.

See VALIDATION.json for scoped results. This small, regex-scored held-out benchmark does not establish general quality improvement. The single same-seed FP8 comparison is a pixel difference, not a perceptual quality metric. Larger resolutions, long videos and action outputs are untested. Edge has no audio generation. No speedup or production-readiness claim is made.

Reasoner calibration used 272 disjoint rows: 176 videos, 48 images and 48 self-authored reasoning examples, including 224 teacher-forced continuations and 25,178 tokens. Dataset attribution: NVIDIA Cosmos-Reason1 SFT/benchmark releases (CC-BY-4.0) and RoboVQA (Apache-2.0), pinned revisions 4691712532849e9d5c550618c2941856c0ef112e and dc73eeac52e2fba0ae237b82af5317911c5b0366. Generation calibration used four image and four 53-frame video prompts, 50 iterations each: 400 iterations and 800 finite-checked denoiser calls. Dataset media, raw evaluation traces and calibration pickles are not distributed.

Download once

The repository is private. Authenticate as an authorized account:

hf auth login
hf download thaddeusk/Cosmos3-Edge-NVFP4-Reasoner-FP8-Generator \
  --local-dir Cosmos3-Edge-NVFP4-FP8

Use Linux/WSL with a CUDA-capable NVIDIA GPU supporting native NVFP4/FP8. The verified hardware was an RTX 5090. Install the applicable pinned environment in a separate Python 3.12 virtual environment:

uv venv edge-reasoner-env --python 3.12 --managed-python
uv pip sync --python edge-reasoner-env/bin/python \
  Cosmos3-Edge-NVFP4-FP8/requirements-reasoner.lock
source edge-reasoner-env/bin/activate
export CUDA_HOME=/usr/local/cuda-13.0
export PATH="$CUDA_HOME/bin:$PATH"

Reasoner only: standard vLLM

vllm serve ./Cosmos3-Edge-NVFP4-FP8 \
  --quantization modelopt_fp4 --dtype bfloat16 \
  --max-model-len 8192 --max-num-seqs 1 \
  --gpu-memory-utilization 0.3 --kv-cache-memory-bytes 1073741824 \
  --enforce-eager \
  --limit-mm-per-prompt '{"image":1,"video":1}' \
  --mm-processor-kwargs '{"fps":4.0,"max_pixels":262144}' \
  --served-model-name cosmos3-edge-reasoner \
  --host 127.0.0.1 --port 8000

Alternatively supply the Hugging Face repository ID directly. Joint shards can download generation tensors, but the native text loader filters generation weights. Preserve the original chat template; outputs include reasoning followed by </think> and the final answer. The advertised maximum context is untested.

Generation: vLLM-Omni

Create another Python 3.12 environment using requirements-generation.lock. The pinned runtime is vLLM/vLLM-Omni 0.30.0. It supports Edge's native mixed linears, but its diffusion-step policy validator needs the included strict compatibility launcher:

python Cosmos3-Edge-NVFP4-FP8/serve_unified.py \
  --runtime-dir ./fresh-edge-omni-runtime \
  ./Cosmos3-Edge-NVFP4-FP8 --host 127.0.0.1 --port 8012 \
  --init-timeout 1800 --enable-layerwise-offload

Choose a fresh runtime directory each launch. The launcher copies the installed SDK; it does not modify installed packages, bypass weight validation or disable safety. The root NVFP4 contract is understanding-only; the transformer component declares the actual joint MIXED_PRECISION storage. The unsupported upstream Qwen-only PyTorch reasoner graph pickle is not included; vLLM consumes the serialized native weights/scales and metadata instead.

Safety remains enabled. Authorized access to nvidia/Cosmos-Guardrail1 is required. This release provides text blocklist/Qwen3Guard checks and RetinaFace face blur; the upstream package disables the separate video content-safety classifier. Enabled guardrails should not be interpreted as complete moderation. If the official Hugging Face Xet transport fails, use HF_HUB_DISABLE_XET=1 to select authenticated HTTP transport.

INT8-ConvRot is not used: its rotation/loading format is unsupported by this pinned runtime and has not been quality-tested on Cosmos3-Edge.