Alpamayo 2 Edge — FP8 weight-only (fits 8 GB consumer GPUs)
Code: FrankLiWright/alpamayo2-edge-fp8 · Upstream: nvidia/Alpamayo2-Edge
A weight-only FP8 (e4m3) quantization of NVIDIA Alpamayo 2 Edge, the 2 B-parameter in-vehicle reasoning vision-language-action model.
It exists for one reason: the BF16 release needs 5 GiB of VRAM for its weights
alone, which is most of a consumer 8 GB card. This checkpoint holds 2.49 GiB
on disk and **2.5 GiB** in VRAM, leaving room for the 22-camera input and the
trajectory diffusion expert.
This is a community derivative, not an NVIDIA release. It is not the official FP8 recipe from
NVlabs/alpamayo-recipes(TensorRT Edge-LLM, FP8 weights and activations and K/V cache, targeting DRIVE AGX Thor). See How this differs from the official FP8 path.
What was quantized
| Scheme | FP8 e4m3fn weights, one FP32 scale per output channel (per input row for embeddings) |
| Granularity | 508 modules: 506 nn.Linear + 2 nn.Embedding |
| Stays in BF16 | norms, biases, and the hand-written action_in_proj / action_space parameters |
| Compute | unchanged — weights dequantize to the activation dtype at forward |
Because compute is ordinary BF16 matmul after dequantizing, this checkpoint runs on any BF16-capable CUDA GPU. FP8 tensor cores are not required.
Measured results
RTX 5060 Laptop GPU (8 GB, sm_120), Windows, driver 621.24, CUDA 12.8,
torch 2.8.0+cu128, transformers 5.17.0, Python 3.12.15.
Single sample, warm timings (first call is slower because of kernel warm-up).
Input: the public 22-frame Edge profile (3 261 prompt tokens).
| BF16 release | This checkpoint | |
|---|---|---|
| Checkpoint on disk | 5.30 GB | 2.49 GiB |
| Weights in VRAM | ~5.0 GiB | ~2.5 GiB |
| 1 trajectory — time | 2.43 s | 3.00 s |
| 1 trajectory — peak VRAM | 7.01 GiB | 4.53 GiB |
| 6 trajectories — time | 2 842 s | 6.10 s |
| 6 trajectories — peak VRAM | 8.55 GiB | 6.07 GiB |
| Model load | 5 s | 26 s |
Two things worth calling out:
- 1 trajectory is ~20 % slower than BF16. The dequantize-on-forward step is not free, but on this class of GPU the difference is small.
- 6 trajectories is ~465× faster than BF16 — but that is not a quantization speedup. BF16 peaks at 8.55 GiB on an 8 GiB card, so it spills into system memory through WDDM and crawls. FP8 simply fits. On a 24 GB card BF16 would be the faster of the two.
How this differs from the official FP8 path
| This checkpoint | recipes/alpamayo2edge_edgellm (official) |
|
|---|---|---|
| Weights | FP8 e4m3, per-channel scales | FP8 |
| Activations / K/V cache | BF16 | FP8 |
| Runtime | PyTorch + transformers |
TensorRT Edge-LLM |
| Target hardware | any BF16 CUDA GPU | NVIDIA DRIVE AGX Thor |
| Calibration | none (weight-only) | per-clip calibration, host GPU ≥ 16 GB |
| Latency (upstream reference) | 2.4–3.0 s on a laptop GPU | ~200 ms per request on Thor |
Use the official recipe if you are deploying on Thor and need real-time. Use this one if you want to run the model on a desktop/laptop GPU in PyTorch.
Requirements
- Python 3.12
torchwith CUDA (tested2.8.0+cu128)transformers==5.17.0— the first release with thecosmos3_edgebackbonealpamayo2-edge(Apache-2.0) for the model class, tokenizer and input preparation
pip install torch --index-url https://download.pytorch.org/whl/cu128
pip install transformers==5.17.0 safetensors einops hydra-core
pip install "alpamayo2-edge @ git+https://github.com/NVlabs/alpamayo2-edge"
Usage
load_fp8.py builds the official Alpamayo2Edge skeleton, swaps nn.Linear /
nn.Embedding for FP8 modules, and fills them from quant.safetensors. The
upstream model code is not modified.
import sys, torch
sys.path.insert(0, "/path/to/Alpamayo2-Edge-FP8") # this repo
from load_fp8 import load_fp8_model
from alpamayo2_edge.helper import prepare_model_inputs, to_device
model, info = load_fp8_model("/path/to/Alpamayo2-Edge-FP8", device="cuda")
print(info["swapped_modules"], "FP8 modules") # 508
sample = { # or load a real clip via alpamayo2_edge.load_physical_aiavdataset
"image_frames": ..., "camera_indices": ..., "absolute_timestamps": ...,
"ego_history_xyz": ..., "ego_history_rot": ..., "nav_text": ["Turn right in 30m"],
}
inputs = to_device(prepare_model_inputs(sample, model.config, model.tokenizer), device="cuda")
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
xyz, rotations, _, extra = model.sample_trajectories_from_data(
inputs, num_traj_sets=1, num_traj_samples=6, max_generation_length=256,
diffusion_kwargs={"inference_step": 10,
"generator": torch.Generator(device="cuda").manual_seed(43)},
return_extra=True,
)
print(xyz.shape) # [1, 6, 1, 64, 3] -> 64 poses at 0.1 s intervals
Or run the self-contained example (synthetic 22-frame capture):
python example.py --model . --traj-sets 6
Reproducing the quantization
Weights are produced from the public BF16 release — no calibration data:
hf download nvidia/Alpamayo2-Edge --local-dir ./Alpamayo2-Edge
python quantize_fp8.py --source ./Alpamayo2-Edge --target ./Alpamayo2-Edge-FP8
Limitations — please read before deploying
Not evaluated here: trajectory accuracy on real driving data. The upstream model reports minADE₆ at 5 s of 0.52 m on 1 392 PhysicalAI-AV samples; that number is for BF16 and has not been re-measured for this checkpoint, because the evaluation dataset requires accepting its own access terms. There is also no closed-loop (AlpaSim) evaluation. If you are using this for anything beyond research, run those evaluations yourself.
What was checked: the checkpoint loads, produces finite 64-point trajectories, and on one synthetic 22-frame input with a fixed seed, its output is close to BF16 — mean per-waypoint displacement of 0.32 m and 0.78 m in two runs, over a trajectory spanning ~35 m. Take this as a smoke test, not an accuracy benchmark: the model samples its chain-of-thought stochastically, so a small logit perturbation can change the sampled reasoning and with it the trajectory. Part of that displacement is sampling divergence, not quantization error.
Other caveats
- Weight-only quantization; activations and K/V cache stay BF16. FP8 GEMM kernels are not used, so there is no FP8 speed benefit — only memory.
load_fp8.pysuppresses the random weight init during skeleton construction (every parameter is overwritten by the checkpoint). Non-persistent derived buffers (rotaryinv_freq, Fourier tables, action-space scalars) are computed by each module's__init__and are checked to be finite after loading.- Pinned to
transformers==5.17.0. Newer transformers may rename or restructure thecosmos3_edgebackbone.
License and attribution
Model weights are derived from nvidia/Alpamayo2-Edge,
developed by NVIDIA Corporation, and are distributed under
OpenMDW-1.1 (see LICENSE).
New material — the quantization scheme, the quantized weights, and
fp8_quant.py / load_fp8.py / quantize_fp8.py / example.py — is
Apache License 2.0. See NOTICE for the full attribution.
This repository is not affiliated with, endorsed by, or supported by NVIDIA. For upstream issues and questions use the Alpamayo NV Developer Forum and NVlabs/alpamayo2-edge.
- Downloads last month
- 25