Alpamayo 2 Edge — FP8 weight-only (fits 8 GB consumer GPUs)

Code: FrankLiWright/alpamayo2-edge-fp8 · Upstream: nvidia/Alpamayo2-Edge

A weight-only FP8 (e4m3) quantization of NVIDIA Alpamayo 2 Edge, the 2 B-parameter in-vehicle reasoning vision-language-action model.

It exists for one reason: the BF16 release needs 5 GiB of VRAM for its weights alone, which is most of a consumer 8 GB card. This checkpoint holds 2.49 GiB on disk and **2.5 GiB** in VRAM, leaving room for the 22-camera input and the trajectory diffusion expert.

This is a community derivative, not an NVIDIA release. It is not the official FP8 recipe from NVlabs/alpamayo-recipes (TensorRT Edge-LLM, FP8 weights and activations and K/V cache, targeting DRIVE AGX Thor). See How this differs from the official FP8 path.

What was quantized

Scheme FP8 e4m3fn weights, one FP32 scale per output channel (per input row for embeddings)
Granularity 508 modules: 506 nn.Linear + 2 nn.Embedding
Stays in BF16 norms, biases, and the hand-written action_in_proj / action_space parameters
Compute unchanged — weights dequantize to the activation dtype at forward

Because compute is ordinary BF16 matmul after dequantizing, this checkpoint runs on any BF16-capable CUDA GPU. FP8 tensor cores are not required.

Measured results

RTX 5060 Laptop GPU (8 GB, sm_120), Windows, driver 621.24, CUDA 12.8, torch 2.8.0+cu128, transformers 5.17.0, Python 3.12.15. Single sample, warm timings (first call is slower because of kernel warm-up). Input: the public 22-frame Edge profile (3 261 prompt tokens).

BF16 release This checkpoint
Checkpoint on disk 5.30 GB 2.49 GiB
Weights in VRAM ~5.0 GiB ~2.5 GiB
1 trajectory — time 2.43 s 3.00 s
1 trajectory — peak VRAM 7.01 GiB 4.53 GiB
6 trajectories — time 2 842 s 6.10 s
6 trajectories — peak VRAM 8.55 GiB 6.07 GiB
Model load 5 s 26 s

Two things worth calling out:

  • 1 trajectory is ~20 % slower than BF16. The dequantize-on-forward step is not free, but on this class of GPU the difference is small.
  • 6 trajectories is ~465× faster than BF16 — but that is not a quantization speedup. BF16 peaks at 8.55 GiB on an 8 GiB card, so it spills into system memory through WDDM and crawls. FP8 simply fits. On a 24 GB card BF16 would be the faster of the two.

How this differs from the official FP8 path

This checkpoint recipes/alpamayo2edge_edgellm (official)
Weights FP8 e4m3, per-channel scales FP8
Activations / K/V cache BF16 FP8
Runtime PyTorch + transformers TensorRT Edge-LLM
Target hardware any BF16 CUDA GPU NVIDIA DRIVE AGX Thor
Calibration none (weight-only) per-clip calibration, host GPU ≥ 16 GB
Latency (upstream reference) 2.4–3.0 s on a laptop GPU ~200 ms per request on Thor

Use the official recipe if you are deploying on Thor and need real-time. Use this one if you want to run the model on a desktop/laptop GPU in PyTorch.

Requirements

  • Python 3.12
  • torch with CUDA (tested 2.8.0+cu128)
  • transformers==5.17.0 — the first release with the cosmos3_edge backbone
  • alpamayo2-edge (Apache-2.0) for the model class, tokenizer and input preparation
pip install torch --index-url https://download.pytorch.org/whl/cu128
pip install transformers==5.17.0 safetensors einops hydra-core
pip install "alpamayo2-edge @ git+https://github.com/NVlabs/alpamayo2-edge"

Usage

load_fp8.py builds the official Alpamayo2Edge skeleton, swaps nn.Linear / nn.Embedding for FP8 modules, and fills them from quant.safetensors. The upstream model code is not modified.

import sys, torch
sys.path.insert(0, "/path/to/Alpamayo2-Edge-FP8")   # this repo
from load_fp8 import load_fp8_model
from alpamayo2_edge.helper import prepare_model_inputs, to_device

model, info = load_fp8_model("/path/to/Alpamayo2-Edge-FP8", device="cuda")
print(info["swapped_modules"], "FP8 modules")   # 508

sample = {   # or load a real clip via alpamayo2_edge.load_physical_aiavdataset
    "image_frames": ..., "camera_indices": ..., "absolute_timestamps": ...,
    "ego_history_xyz": ..., "ego_history_rot": ..., "nav_text": ["Turn right in 30m"],
}
inputs = to_device(prepare_model_inputs(sample, model.config, model.tokenizer), device="cuda")

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    xyz, rotations, _, extra = model.sample_trajectories_from_data(
        inputs, num_traj_sets=1, num_traj_samples=6, max_generation_length=256,
        diffusion_kwargs={"inference_step": 10,
                          "generator": torch.Generator(device="cuda").manual_seed(43)},
        return_extra=True,
    )
print(xyz.shape)   # [1, 6, 1, 64, 3]  -> 64 poses at 0.1 s intervals

Or run the self-contained example (synthetic 22-frame capture):

python example.py --model . --traj-sets 6

Reproducing the quantization

Weights are produced from the public BF16 release — no calibration data:

hf download nvidia/Alpamayo2-Edge --local-dir ./Alpamayo2-Edge
python quantize_fp8.py --source ./Alpamayo2-Edge --target ./Alpamayo2-Edge-FP8

Limitations — please read before deploying

Not evaluated here: trajectory accuracy on real driving data. The upstream model reports minADE₆ at 5 s of 0.52 m on 1 392 PhysicalAI-AV samples; that number is for BF16 and has not been re-measured for this checkpoint, because the evaluation dataset requires accepting its own access terms. There is also no closed-loop (AlpaSim) evaluation. If you are using this for anything beyond research, run those evaluations yourself.

What was checked: the checkpoint loads, produces finite 64-point trajectories, and on one synthetic 22-frame input with a fixed seed, its output is close to BF16 — mean per-waypoint displacement of 0.32 m and 0.78 m in two runs, over a trajectory spanning ~35 m. Take this as a smoke test, not an accuracy benchmark: the model samples its chain-of-thought stochastically, so a small logit perturbation can change the sampled reasoning and with it the trajectory. Part of that displacement is sampling divergence, not quantization error.

Other caveats

  • Weight-only quantization; activations and K/V cache stay BF16. FP8 GEMM kernels are not used, so there is no FP8 speed benefit — only memory.
  • load_fp8.py suppresses the random weight init during skeleton construction (every parameter is overwritten by the checkpoint). Non-persistent derived buffers (rotary inv_freq, Fourier tables, action-space scalars) are computed by each module's __init__ and are checked to be finite after loading.
  • Pinned to transformers==5.17.0. Newer transformers may rename or restructure the cosmos3_edge backbone.

License and attribution

Model weights are derived from nvidia/Alpamayo2-Edge, developed by NVIDIA Corporation, and are distributed under OpenMDW-1.1 (see LICENSE).

New material — the quantization scheme, the quantized weights, and fp8_quant.py / load_fp8.py / quantize_fp8.py / example.py — is Apache License 2.0. See NOTICE for the full attribution.

This repository is not affiliated with, endorsed by, or supported by NVIDIA. For upstream issues and questions use the Alpamayo NV Developer Forum and NVlabs/alpamayo2-edge.

Downloads last month
25
Video Preview
loading

Model tree for FrankLiWright/Alpamayo2-Edge-FP8

Finetuned
(2)
this model

Dataset used to train FrankLiWright/Alpamayo2-Edge-FP8