Instructions to use suukim/Ishinca-4B-DART50 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Cosmos
How to use suukim/Ishinca-4B-DART50 with Cosmos:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Ishinca-4B-DART50
Ishinca: Distilling NVIDIA Alpamayo 1.5 (10B) into a 4B Vision-Language-Action Model for Autonomous Driving
Author: Sunguk Kim
Ishinca-4B driving in AlpaSim closed loop: bird's-eye plan, front camera with the model's Chain-of-Causation, predicted trajectory. Three example rollouts picked for illustration (run on an earlier AlpaSim runtime with the released Ishinca-4B weights); aggregate numbers and failure cases are in the Evaluation section below.
Ishinca-4B retrained with DART vision-token pruning, keep ratio 50% (1,440 image tokens). Lowest prefill / KV cost; non-significant trend towards more at-fault collisions in our closed loop.
Teacher: nvidia/Alpamayo-1.5-10B. Backbone initialisation: nvidia/Cosmos-Reason2-2B (Qwen3-VL-2B architecture).
All distillation targets come from the teacher; no ground-truth trajectories are mixed into the distillation targets. Code, training and evaluation scripts, and clip-ID manifests: https://github.com/kimsunguk0/Ishinca.
Built on NVIDIA Cosmos. This is a research model, not a production driving system.
Model family (HF Collection "Ishinca")
| Repo | Variant | Image tokens |
|---|---|---|
suukim/Ishinca-4B |
no pruning | 2,880 |
suukim/Ishinca-4B-DART75 |
DART keep 75% | 2,160 |
suukim/Ishinca-4B-DART50 (this repo) |
DART keep 50% | 1,440 |
This repo: Ishinca-4B + DART keep 50%. DART vision-token pruning, keep ratio 50% (1,440 image tokens per sample instead of 2,880).
The pruning is applied by the inference code (driver option dart_keep), not stored in the weights. Running a DART checkpoint without pruning, or Ishinca-4B with pruning, is not what was evaluated.
Architecture
- Backbone: Qwen3-VL-2B-style, 28 layers, hidden 2048, vision tower 24 layers (patch 16, merge 2, DeepStack at layers 5/11/17), vocabulary 155,685 (Alpamayo trajectory tokens appended; the tokenizer differs from the release Alpamayo tokenizer, see
backbone/). - Action expert (AE28): 28 layers, hidden 2048, intermediate 8256, 16 heads x 128, non-causal attention over the VLM prefix KV cache. Layers were initialised from teacher expert layers
round(i*35/27)and trained with flow matching on teacher actions. - Total: 4.2 B parameters (4.23 B) in
model.safetensors(VLM 2.45 B incl. vision tower 0.41 B, embeddings and lm_head; action expert + projections 1.77 B), all bfloat16. - Output: 64 waypoints at 10 Hz (6.4 s), unicycle acceleration/curvature action space (same as Alpamayo 1.5); optional Chain-of-Causation text.
- Input: 4 cameras x 4 frames (front-wide, front-tele, cross-left, cross-right), 320x576 after processing, 16-step ego history, and an optional route instruction passed as
nav_text/--nav-text, format<command> in <distance> meterswith command one ofgo straight,turn left,turn rightand an integer distance (e.g."turn left in 20 meters"). About 21% of the training samples carried a route instruction (go straight 11.5%, turn right 4.9%, turn left 4.6%). These route instructions were created by us; they are not part of the PhysicalAI-AV dataset, not NVIDIA annotations, and not redistributed. All evaluations below were run without a route instruction (no CFG-nav).
Files
model.safetensors # 947 tensors: vlm.* (626) + expert.* / action_in_proj.* / action_out_proj.* (321), bf16
config.json # Alpamayo1_5 config (vlm_name_or_path: ./backbone, resolved relative to this folder by the loader)
PACKAGE.json # tensor counts and load check (no missing / unexpected keys)
backbone/ # Qwen3-VL config, tokenizer, processor, chat template (no weights; they are inside model.safetensors)
LICENSE NOTICE.md
How to run
Code: https://github.com/kimsunguk0/Ishinca (Apache-2.0). The loader ishinca.load_ishinca resolves ./backbone relative to the checkpoint folder and loads the student tokenizer; the backbone has no separate weights (model.safetensors holds the VLM and the action expert), so nothing else needs to be downloaded or edited.
pip install git+https://github.com/kimsunguk0/Ishinca # also needs the upstream Alpamayo-1.5 inference code, see the repo README
hf download suukim/Ishinca-4B-DART50 --local-dir ./Ishinca-4B-DART50 # or pass the repo id to load_ishinca directly
python scripts/infer.py --checkpoint ./Ishinca-4B-DART50 --clip-id <uuid> --seed 42 # from a clone of the code repo
import torch
from ishinca import load_ishinca, build_model_inputs
from ishinca.data import load_clip
from ishinca.inputs import get_processor
model = load_ishinca("./Ishinca-4B-DART50") # local dir or HF repo id
processor = get_processor(model)
data = load_clip("<clip-id>", t0_us=5_100_000) # image_frames [4 cams, 4, 3, H, W], ego_history_xyz / rot
inputs = build_model_inputs(model, processor, data["image_frames"], data["camera_indices"],
data["ego_history_xyz"], data["ego_history_rot"])
torch.manual_seed(42)
with torch.autocast("cuda", dtype=torch.bfloat16):
xyz, rot, extra = model.sample_trajectories_from_data_with_vlm_rollout(
data=inputs, top_p=0.98, temperature=0.6, num_traj_samples=1, max_generation_length=256, return_extra=True)
print(extra["cot"][0, 0, 0]) # Chain-of-Causation text
print(xyz[0, 0, 0, :, :2]) # 64 x 2 waypoints, ego frame at t0, metres
Verified on x86 (RTX 3090, torch 2.8 / CUDA 12.8, transformers 4.57.1, bf16, eager PyTorch): about 8.0 GiB peak VRAM, about 1.0-1.2 s per call (4 cams x 4 frames, full CoC + 10-step flow matching), load 4-16 s. Evaluation in the tables below used the AlpaSim driver (BF16 compiled runtime, 16-frame window, Euler 10 steps, one path, no nav).
DART. config.json of this repo carries "ishinca_dart_keep": 0.5; load_ishinca reads it and applies DART token pruning (per image ceil(180*keep) of 180 merged tokens, selected after the vision merger, original mRoPE positions kept). Override or disable with load_ishinca(..., dart_keep=...) or scripts/infer.py --dart-keep 0.5 / --no-dart. The pruning is not stored in the weights; running without it is not what was evaluated. The DART speedup only shows in compiled / TensorRT runtimes: the eager PyTorch rollout is not faster (on the RTX 3090 about 1.0 s vs 1.1-1.2 s). NVIDIA documents DART as unsupported by the VLA Action Runner; our results come from the PyTorch/AlpaSim path.
Training summary
- Teacher: Alpamayo-1.5 10B. Targets: teacher greedy Chain-of-Causation text, teacher discrete trajectory tokens (logit KD), teacher action-expert trajectories (path 0). Where the student's CoC differs from the teacher's, the teacher re-labels the AE target on the student's CoC. No ground-truth trajectories are used as targets or rewards; GT is only a monitor in evaluation.
- Backbone: full fine-tune, loss = 0.1 CE_cot + 1.0 CE_traj + 0.2 CE_format + 0.1 KL_cot + 1.0 KL_traj + image-token relation distillation (N x N token-relation matching of late vision features against the teacher), ViT frozen, 3 epochs.
- AE28: flow matching to teacher actions on the student's own KV, 25,209 steps (3 epochs).
- DART variants: Ishinca-4B weights as start, DART selection inside every training step, backbone 1 epoch then AE28 3 epochs on the pruned prefix.
- Data: trained on clips of
nvidia/PhysicalAI-Autonomous-Vehicles; see the code repository for details. No dataset content is redistributed here; clip-ID manifests are in the code repository (https://github.com/kimsunguk0/Ishinca). - No backbone + AE joint training (the AE loss is never back-propagated into the VLM).
Evaluation
Metric convention: single-path ADE and FDE, and minADE6 @ 6.4 s (best of 6 paths), in metres. "vs Teacher" uses the teacher action expert's path 0 on its own greedy CoC; same initial noise (x0) for all rows.
Open loop
Our validation set: 4,000 held-out PhysicalAI-AV samples we built (not used for training); same initial noise for every model, flow-matching expert with 10 Euler steps.
| Model | Image tokens | vs GT: ADE | vs GT: minADE6@6.4s | vs Teacher: ADE | vs Teacher: FDE | vs Teacher: minADE6@6.4s |
|---|---|---|---|---|---|---|
| Teacher 10B | 2,880 | 1.750 | 0.826 | n/a | n/a | n/a |
| Ishinca-4B | 2,880 | 1.957 | 0.986 | 1.041 | 3.172 | 0.670 |
| Ishinca-4B-DART75 | 2,160 | 1.957 | 0.994 | 1.043 | 3.173 | 0.686 |
| Ishinca-4B-DART50 | 1,440 | 1.972 | 1.003 | 1.068 | 3.250 | 0.698 |
gold644: the 644 public PhysicalAI-AV evaluation examples of the Alpamayo-R1 paper (arXiv:2511.00088), Table 10, Section 6.4 "Public Benchmark Evaluation". The clip list comes from NVIDIA's public evaluation recipe; we measured with the official protocol (t0 = 5.1 s, 6 samples, T = 0.6, top-p 0.98, no navigation). DART variants were not run on gold644.
| Model | ADE (1 sample) | minADE6@6.4s |
|---|---|---|
| Alpamayo-R1-10B (reported in the paper, Table 10) | n/a | 0.849 |
| Alpamayo-1.5-10B (Teacher, measured by us) | 2.241 | 0.831 |
| Ishinca-4B (measured by us) | 2.381 | 1.045 |
Caveat: the same-x0 protocol matters. Evaluating with a mismatched noise stream inflates "vs Teacher" by 0.22-0.27 m.
Closed loop (AlpaSim)
Setup: 150 scenes x 2 rollouts, all models on the same runtime (BF16 compiled runtime, full 16-frame window, Euler 10 steps, one path, no nav). Score = 0 on an at-fault collision or off-road event, else min(progress/0.8, 1), averaged over all 150 scenes. Traffic is non-reactive (log replay).
| Teacher 10B | Ishinca-4B | Ishinca-4B-DART75 | Ishinca-4B-DART50 | |
|---|---|---|---|---|
| Score | 0.728 | 0.751 | 0.731 | 0.776 |
| Failed rollouts / 300 | 40 | 37 | 45 | 37 |
| off-road | 32 | 34 | 35 | 30 |
| at-fault collision | 8 | 3 | 10 | 7 |
The differences among the four models are within run-to-run noise (paired scene bootstrap, 2 rollouts per scene); the ranking should not be read as one model being better.
DART limitation: the DART variants show a non-significant trend towards more at-fault collisions (150-scene run: Teacher 8 / Ishinca-4B 3 / DART50 7 / DART75 10; a 9-scene rerun with 6 rollouts per model: 10 / 11 / 19 / 21, not significant). Treat DART as a cost/quality trade-off.
Limitations
- Evaluated without navigation. The model accepts a route instruction, but every evaluation reported here was run without one. Without a route, at junctions and driveways it picks a plausible route, not necessarily the intended one; a navigation-conditioned evaluation is planned.
- Closed-loop failure modes are scene-systematic and shared with the teacher (14 scenes fail for all four models): turns cutting corners, route ambiguity, lane-keeping drift, nudging around stopped vehicles. The plan, not the controller, is the cause (tracking error <= 0.01 m).
- DART variants: non-significant trend toward more at-fault collisions (see above); not ruled out.
- Non-reactive traffic in closed loop (log replay): collision numbers do not measure interaction; at-fault labels are imperfect when the ego is slower than the log.
- Closed-loop sample is small (150 scenes x 2 rollouts); differences between models are within noise.
- Trained and evaluated only with the 4-camera, 4-frame, 320x576 setting of PhysicalAI-AV; other camera rigs are untested, and camera mounting/calibration changes shift the predicted path.
- Open-loop gap to the teacher vs GT remains (Ishinca-4B ADE vs GT 1.957 vs teacher 1.750).
- Student CoC text is a by-product and was not evaluated for reasoning quality.
- Not safety-validated.
Roadmap
- Goal: 10 FPS on Jetson AGX Thor (100 ms per call). The current best full-window Thor latency is 383.3 ms (model compute only). Vision encoding, prefill and CoC decoding are still the main bottlenecks, so the next steps are compressing the vision side and the prefill, and speeding up decoding.
- Reinforcement learning (not applied). We are still studying reinforcement learning for driving VLAs. GRPO on the VLM is the next step we plan to apply to Ishinca; early experiments are under way but are not part of this release. This checkpoint does not use VLM GRPO: it is trained by distillation from the teacher only.
- Further training and experiments (GRPO, navigation-conditioned evaluation, larger closed-loop studies) will continue once GPU resources for training are available again.
Intended use
Research on distillation of VLA driving models, token-pruning studies, and evaluation in simulation (e.g., AlpaSim). Not validated or certified for deployment in vehicles; any real-world use is at the user's own risk and responsibility. Commercial use is permitted by the upstream license terms (below), without warranty.
License
Released under the NVIDIA Open Model License (license: other, license_name: nvidia-open-model-license); the full agreement is in LICENSE, together with the OpenMDW-1.1 text of the teacher Alpamayo-1.5-10B. Why: the backbone is initialised from Cosmos-Reason2-2B (NVIDIA Open Model License; commercial use allowed, derivative models may be created and distributed), and the action expert is initialised from and trained against Alpamayo-1.5-10B (OpenMDW-1.1, permissive). We follow the more restrictive of the two, the NVIDIA Open Model License. Use must be consistent with NVIDIA's Trustworthy AI terms; bypassing a model guardrail terminates the license. See NOTICE.md for attributions. Built on NVIDIA Cosmos.
Citation
@misc{kim2026ishinca,
title = {Ishinca: Distilling Alpamayo 1.5 (10B) into a 4B Vision-Language-Action Model for Autonomous Driving},
author = {Kim, Sunguk},
year = {2026},
howpublished = {\url{https://github.com/kimsunguk0/Ishinca}}
}
- Downloads last month
- 42