Ishinca-4B-DART75

Ishinca: Distilling NVIDIA Alpamayo 1.5 (10B) into a 4B Vision-Language-Action Model for Autonomous Driving

Author: Sunguk Kim

Ishinca-4B driving in AlpaSim closed loop: bird's-eye plan, front camera with the model's Chain-of-Causation, predicted trajectory. Three example rollouts picked for illustration (run on an earlier AlpaSim runtime with the released Ishinca-4B weights); aggregate numbers and failure cases are in the Evaluation section below.

Ishinca-4B retrained with DART vision-token pruning, keep ratio 75% (2,160 image tokens). Lower prefill / KV cost.

Teacher: nvidia/Alpamayo-1.5-10B. Backbone initialisation: nvidia/Cosmos-Reason2-2B (Qwen3-VL-2B architecture). All distillation targets come from the teacher; no ground-truth trajectories are mixed into the distillation targets. Code, training and evaluation scripts, and clip-ID manifests: https://github.com/kimsunguk0/Ishinca.

Built on NVIDIA Cosmos. This is a research model, not a production driving system.

Model family (HF Collection "Ishinca")

Repo Variant Image tokens
suukim/Ishinca-4B no pruning 2,880
suukim/Ishinca-4B-DART75 (this repo) DART keep 75% 2,160
suukim/Ishinca-4B-DART50 DART keep 50% 1,440

This repo: Ishinca-4B + DART keep 75%. DART vision-token pruning, keep ratio 75% (2,160 image tokens per sample instead of 2,880). The pruning is applied by the inference code (driver option dart_keep), not stored in the weights. Running a DART checkpoint without pruning, or Ishinca-4B with pruning, is not what was evaluated.

Architecture

  • Backbone: Qwen3-VL-2B-style, 28 layers, hidden 2048, vision tower 24 layers (patch 16, merge 2, DeepStack at layers 5/11/17), vocabulary 155,685 (Alpamayo trajectory tokens appended; the tokenizer differs from the release Alpamayo tokenizer, see backbone/).
  • Action expert (AE28): 28 layers, hidden 2048, intermediate 8256, 16 heads x 128, non-causal attention over the VLM prefix KV cache. Layers were initialised from teacher expert layers round(i*35/27) and trained with flow matching on teacher actions.
  • Total: 4.2 B parameters (4.23 B) in model.safetensors (VLM 2.45 B incl. vision tower 0.41 B, embeddings and lm_head; action expert + projections 1.77 B), all bfloat16.
  • Output: 64 waypoints at 10 Hz (6.4 s), unicycle acceleration/curvature action space (same as Alpamayo 1.5); optional Chain-of-Causation text.
  • Input: 4 cameras x 4 frames (front-wide, front-tele, cross-left, cross-right), 320x576 after processing, 16-step ego history, and an optional route instruction passed as nav_text / --nav-text, format <command> in <distance> meters with command one of go straight, turn left, turn right and an integer distance (e.g. "turn left in 20 meters"). About 21% of the training samples carried a route instruction (go straight 11.5%, turn right 4.9%, turn left 4.6%). These route instructions were created by us; they are not part of the PhysicalAI-AV dataset, not NVIDIA annotations, and not redistributed. All evaluations below were run without a route instruction (no CFG-nav).

Files

model.safetensors      # 947 tensors: vlm.* (626) + expert.* / action_in_proj.* / action_out_proj.* (321), bf16
config.json            # Alpamayo1_5 config (vlm_name_or_path: ./backbone, resolved relative to this folder by the loader)
PACKAGE.json           # tensor counts and load check (no missing / unexpected keys)
backbone/              # Qwen3-VL config, tokenizer, processor, chat template (no weights; they are inside model.safetensors)
LICENSE  NOTICE.md

How to run

Code: https://github.com/kimsunguk0/Ishinca (Apache-2.0). The loader ishinca.load_ishinca resolves ./backbone relative to the checkpoint folder and loads the student tokenizer; the backbone has no separate weights (model.safetensors holds the VLM and the action expert), so nothing else needs to be downloaded or edited.

pip install git+https://github.com/kimsunguk0/Ishinca      # also needs the upstream Alpamayo-1.5 inference code, see the repo README
hf download suukim/Ishinca-4B-DART75 --local-dir ./Ishinca-4B-DART75      # or pass the repo id to load_ishinca directly
python scripts/infer.py --checkpoint ./Ishinca-4B-DART75 --clip-id <uuid> --seed 42      # from a clone of the code repo
import torch
from ishinca import load_ishinca, build_model_inputs
from ishinca.data import load_clip
from ishinca.inputs import get_processor

model = load_ishinca("./Ishinca-4B-DART75")                  # local dir or HF repo id
processor = get_processor(model)
data = load_clip("<clip-id>", t0_us=5_100_000)  # image_frames [4 cams, 4, 3, H, W], ego_history_xyz / rot
inputs = build_model_inputs(model, processor, data["image_frames"], data["camera_indices"],
                            data["ego_history_xyz"], data["ego_history_rot"])
torch.manual_seed(42)
with torch.autocast("cuda", dtype=torch.bfloat16):
    xyz, rot, extra = model.sample_trajectories_from_data_with_vlm_rollout(
        data=inputs, top_p=0.98, temperature=0.6, num_traj_samples=1, max_generation_length=256, return_extra=True)
print(extra["cot"][0, 0, 0])   # Chain-of-Causation text
print(xyz[0, 0, 0, :, :2])     # 64 x 2 waypoints, ego frame at t0, metres

Verified on x86 (RTX 3090, torch 2.8 / CUDA 12.8, transformers 4.57.1, bf16, eager PyTorch): about 8.0 GiB peak VRAM, about 1.0-1.2 s per call (4 cams x 4 frames, full CoC + 10-step flow matching), load 4-16 s. Evaluation in the tables below used the AlpaSim driver (BF16 compiled runtime, 16-frame window, Euler 10 steps, one path, no nav).

DART. config.json of this repo carries "ishinca_dart_keep": 0.75; load_ishinca reads it and applies DART token pruning (per image ceil(180*keep) of 180 merged tokens, selected after the vision merger, original mRoPE positions kept). Override or disable with load_ishinca(..., dart_keep=...) or scripts/infer.py --dart-keep 0.75 / --no-dart. The pruning is not stored in the weights; running without it is not what was evaluated. The DART speedup only shows in compiled / TensorRT runtimes: the eager PyTorch rollout is not faster (on the RTX 3090 about 1.0 s vs 1.1-1.2 s). NVIDIA documents DART as unsupported by the VLA Action Runner; our results come from the PyTorch/AlpaSim path.

Training summary

  • Teacher: Alpamayo-1.5 10B. Targets: teacher greedy Chain-of-Causation text, teacher discrete trajectory tokens (logit KD), teacher action-expert trajectories (path 0). Where the student's CoC differs from the teacher's, the teacher re-labels the AE target on the student's CoC. No ground-truth trajectories are used as targets or rewards; GT is only a monitor in evaluation.
  • Backbone: full fine-tune, loss = 0.1 CE_cot + 1.0 CE_traj + 0.2 CE_format + 0.1 KL_cot + 1.0 KL_traj + image-token relation distillation (N x N token-relation matching of late vision features against the teacher), ViT frozen, 3 epochs.
  • AE28: flow matching to teacher actions on the student's own KV, 25,209 steps (3 epochs).
  • DART variants: Ishinca-4B weights as start, DART selection inside every training step, backbone 1 epoch then AE28 3 epochs on the pruned prefix.
  • Data: trained on clips of nvidia/PhysicalAI-Autonomous-Vehicles; see the code repository for details. No dataset content is redistributed here; clip-ID manifests are in the code repository (https://github.com/kimsunguk0/Ishinca).
  • No backbone + AE joint training (the AE loss is never back-propagated into the VLM).

Evaluation

Metric convention: single-path ADE and FDE, and minADE6 @ 6.4 s (best of 6 paths), in metres. "vs Teacher" uses the teacher action expert's path 0 on its own greedy CoC; same initial noise (x0) for all rows.

Open loop

Our validation set: 4,000 held-out PhysicalAI-AV samples we built (not used for training); same initial noise for every model, flow-matching expert with 10 Euler steps.

Model Image tokens vs GT: ADE vs GT: minADE6@6.4s vs Teacher: ADE vs Teacher: FDE vs Teacher: minADE6@6.4s
Teacher 10B 2,880 1.750 0.826 n/a n/a n/a
Ishinca-4B 2,880 1.957 0.986 1.041 3.172 0.670
Ishinca-4B-DART75 2,160 1.957 0.994 1.043 3.173 0.686
Ishinca-4B-DART50 1,440 1.972 1.003 1.068 3.250 0.698

gold644: the 644 public PhysicalAI-AV evaluation examples of the Alpamayo-R1 paper (arXiv:2511.00088), Table 10, Section 6.4 "Public Benchmark Evaluation". The clip list comes from NVIDIA's public evaluation recipe; we measured with the official protocol (t0 = 5.1 s, 6 samples, T = 0.6, top-p 0.98, no navigation). DART variants were not run on gold644.

Model ADE (1 sample) minADE6@6.4s
Alpamayo-R1-10B (reported in the paper, Table 10) n/a 0.849
Alpamayo-1.5-10B (Teacher, measured by us) 2.241 0.831
Ishinca-4B (measured by us) 2.381 1.045

Caveat: the same-x0 protocol matters. Evaluating with a mismatched noise stream inflates "vs Teacher" by 0.22-0.27 m.

Closed loop (AlpaSim)

Setup: 150 scenes x 2 rollouts, all models on the same runtime (BF16 compiled runtime, full 16-frame window, Euler 10 steps, one path, no nav). Score = 0 on an at-fault collision or off-road event, else min(progress/0.8, 1), averaged over all 150 scenes. Traffic is non-reactive (log replay).

Teacher 10B Ishinca-4B Ishinca-4B-DART75 Ishinca-4B-DART50
Score 0.728 0.751 0.731 0.776
Failed rollouts / 300 40 37 45 37
off-road 32 34 35 30
at-fault collision 8 3 10 7

The differences among the four models are within run-to-run noise (paired scene bootstrap, 2 rollouts per scene); the ranking should not be read as one model being better.

DART limitation: the DART variants show a non-significant trend towards more at-fault collisions (150-scene run: Teacher 8 / Ishinca-4B 3 / DART50 7 / DART75 10; a 9-scene rerun with 6 rollouts per model: 10 / 11 / 19 / 21, not significant). Treat DART as a cost/quality trade-off.

Limitations

  • Evaluated without navigation. The model accepts a route instruction, but every evaluation reported here was run without one. Without a route, at junctions and driveways it picks a plausible route, not necessarily the intended one; a navigation-conditioned evaluation is planned.
  • Closed-loop failure modes are scene-systematic and shared with the teacher (14 scenes fail for all four models): turns cutting corners, route ambiguity, lane-keeping drift, nudging around stopped vehicles. The plan, not the controller, is the cause (tracking error <= 0.01 m).
  • DART variants: non-significant trend toward more at-fault collisions (see above); not ruled out.
  • Non-reactive traffic in closed loop (log replay): collision numbers do not measure interaction; at-fault labels are imperfect when the ego is slower than the log.
  • Closed-loop sample is small (150 scenes x 2 rollouts); differences between models are within noise.
  • Trained and evaluated only with the 4-camera, 4-frame, 320x576 setting of PhysicalAI-AV; other camera rigs are untested, and camera mounting/calibration changes shift the predicted path.
  • Open-loop gap to the teacher vs GT remains (Ishinca-4B ADE vs GT 1.957 vs teacher 1.750).
  • Student CoC text is a by-product and was not evaluated for reasoning quality.
  • Not safety-validated.

Roadmap

  • Goal: 10 FPS on Jetson AGX Thor (100 ms per call). The current best full-window Thor latency is 383.3 ms (model compute only). Vision encoding, prefill and CoC decoding are still the main bottlenecks, so the next steps are compressing the vision side and the prefill, and speeding up decoding.
  • Reinforcement learning (not applied). We are still studying reinforcement learning for driving VLAs. GRPO on the VLM is the next step we plan to apply to Ishinca; early experiments are under way but are not part of this release. This checkpoint does not use VLM GRPO: it is trained by distillation from the teacher only.
  • Further training and experiments (GRPO, navigation-conditioned evaluation, larger closed-loop studies) will continue once GPU resources for training are available again.

Intended use

Research on distillation of VLA driving models, token-pruning studies, and evaluation in simulation (e.g., AlpaSim). Not validated or certified for deployment in vehicles; any real-world use is at the user's own risk and responsibility. Commercial use is permitted by the upstream license terms (below), without warranty.

License

Released under the NVIDIA Open Model License (license: other, license_name: nvidia-open-model-license); the full agreement is in LICENSE, together with the OpenMDW-1.1 text of the teacher Alpamayo-1.5-10B. Why: the backbone is initialised from Cosmos-Reason2-2B (NVIDIA Open Model License; commercial use allowed, derivative models may be created and distributed), and the action expert is initialised from and trained against Alpamayo-1.5-10B (OpenMDW-1.1, permissive). We follow the more restrictive of the two, the NVIDIA Open Model License. Use must be consistent with NVIDIA's Trustworthy AI terms; bypassing a model guardrail terminates the license. See NOTICE.md for attributions. Built on NVIDIA Cosmos.

Citation

@misc{kim2026ishinca,
  title        = {Ishinca: Distilling Alpamayo 1.5 (10B) into a 4B Vision-Language-Action Model for Autonomous Driving},
  author       = {Kim, Sunguk},
  year         = {2026},
  howpublished = {\url{https://github.com/kimsunguk0/Ishinca}}
}
Downloads last month
35
Safetensors
Model size
4B params
Tensor type
BF16
·
Video Preview
loading

Model tree for suukim/Ishinca-4B-DART75

Finetuned
(15)
this model

Dataset used to train suukim/Ishinca-4B-DART75

Collection including suukim/Ishinca-4B-DART75

Paper for suukim/Ishinca-4B-DART75