Bonsai Vineyard Sample LoRA (Cosmos3-Nano), iteration 150

7a5c912 verified 18 days ago

3.99 kB

---
license: cc-by-nc-4.0
base_model: nvidia/Cosmos3-Nano
base_model_relation: finetune
datasets:
  - BonsaiRobotics/bonsai-vineyard-sample
pipeline_tag: image-to-video
tags:
  - cosmos
  - lora
  - agriculture
  - vineyard
  - robotics
  - video-generation
---
# Bonsai Vineyard Sample LoRA (Cosmos3-Nano)

A rank-32 LoRA on [`nvidia/Cosmos3-Nano`](https://huggingface.co/nvidia/Cosmos3-Nano) that turns a seed image, a
segmentation video, a depth video, or segmentation + depth together into a 720p, 15 fps video of a farm robot driving
through vineyard rows. It is the checkpoint produced by the Cosmos3-Nano multi-control transfer LoRA recipe in
[NVIDIA/cosmos-framework](https://github.com/NVIDIA/cosmos-framework) (`docs/transfer_multicontrol_lora_nano.md`),
trained for 150 iterations on the 50 training clips of
[`BonsaiRobotics/bonsai-vineyard-sample`](https://huggingface.co/datasets/BonsaiRobotics/bonsai-vineyard-sample).

This repository holds the exported checkpoint (base weights plus the LoRA tensors), in the format
`cosmos_framework.scripts.inference --checkpoint-path` loads. One checkpoint serves all four modes.

## Use

From a cosmos-framework checkout (see the recipe for the pinned commit and setup):

```bash
uvx hf@latest download BonsaiRobotics/bonsai-vineyard-sample-lora --local-dir examples/checkpoints/bonsai-vineyard-sample-lora
python -m cosmos_framework.scripts.inference --parallelism-preset=latency --seed 2026 \
    -i <spec>.json -o outputs/bonsai-vineyard-sample-lora \
    --checkpoint-path examples/checkpoints/bonsai-vineyard-sample-lora
```

The recipe's `examples/transfer_multicontrol_lora_nano/make_specs.py` writes the specs used for its results (seed
image -> video: 149 frames, guidance 6; control modes: 61 frames, guidance 3, control guidance 0.9, 35 steps,
shift 10). The Wan2.2 VAE is fetched automatically on first use.

## Results

Held-out day of the dataset, 6 clips, one seed, against the base checkpoint with identical inputs (mean over clips):

| Mode | PSNR (dB), base -> LoRA | LPIPS, base -> LoRA |
|---|---|---|
| Seg + depth -> video | 11.82 -> **15.65** | 0.463 -> **0.323** |
| Seg -> video | 12.13 -> **15.30** | 0.466 -> **0.332** |
| Depth -> video | 12.31 -> **15.32** | 0.476 -> **0.334** |
| Seed image -> video | 19.27 -> 19.39 | 0.333 -> 0.318 |

The gain is mostly appearance: the LoRA renders the site's bright, dry daylight where the base model renders a
generic low-sun vineyard. Full tables, the before/after comparison and a check on orchard and strawberry clips are in
the recipe.

## Limitations

- Trained on a small sample dataset: 50 ten-second clips from one camera rig in one vineyard, daylight only. The
  results show what a small dataset can do, not the best this recipe can reach.
- Objects the segmentation marks but the training clips rarely show (buildings at the row end, people) are not drawn.
- Where the segmentation leaves part of the frame unassigned, such as a distant horizon, the LoRA fills it with the
  training site's background (dry hills, a fence).
- In the control modes, leaves near the camera change more from frame to frame than in real video.

## Licence

The fine-tuned checkpoint is released under the
[Creative Commons Attribution-NonCommercial 4.0 International licence (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/),
the same terms as the dataset it was trained on: non-commercial use, with credit to Bonsai Robotics.

It is derived from `nvidia/Cosmos3-Nano`, which NVIDIA releases under
[OpenMDW-1.1](https://openmdw.ai/license/1-1/); the base model's terms and notices continue to apply to the parts of
the weights that come from it.

## Citation

```bibtex
@misc{bonsai_vineyard_sample_lora_2026,
  title        = {Bonsai Vineyard Sample LoRA (Cosmos3-Nano)},
  author       = {Zachariah, Sandeep Sam and {Bonsai Robotics}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/BonsaiRobotics/bonsai-vineyard-sample-lora}}
}
```

	---
	license: cc-by-nc-4.0
	base_model: nvidia/Cosmos3-Nano
	base_model_relation: finetune
	datasets:
	- BonsaiRobotics/bonsai-vineyard-sample
	pipeline_tag: image-to-video
	tags:
	- cosmos
	- lora
	- agriculture
	- vineyard
	- robotics
	- video-generation
	---

	# Bonsai Vineyard Sample LoRA (Cosmos3-Nano)

	A rank-32 LoRA on [`nvidia/Cosmos3-Nano`](https://huggingface.co/nvidia/Cosmos3-Nano) that turns a seed image, a
	segmentation video, a depth video, or segmentation + depth together into a 720p, 15 fps video of a farm robot driving
	through vineyard rows. It is the checkpoint produced by the Cosmos3-Nano multi-control transfer LoRA recipe in
	[NVIDIA/cosmos-framework](https://github.com/NVIDIA/cosmos-framework) (`docs/transfer_multicontrol_lora_nano.md`),
	trained for 150 iterations on the 50 training clips of
	[`BonsaiRobotics/bonsai-vineyard-sample`](https://huggingface.co/datasets/BonsaiRobotics/bonsai-vineyard-sample).

	This repository holds the exported checkpoint (base weights plus the LoRA tensors), in the format
	`cosmos_framework.scripts.inference --checkpoint-path` loads. One checkpoint serves all four modes.

	## Use

	From a cosmos-framework checkout (see the recipe for the pinned commit and setup):

	```bash
	uvx hf@latest download BonsaiRobotics/bonsai-vineyard-sample-lora --local-dir examples/checkpoints/bonsai-vineyard-sample-lora
	python -m cosmos_framework.scripts.inference --parallelism-preset=latency --seed 2026 \
	-i <spec>.json -o outputs/bonsai-vineyard-sample-lora \
	--checkpoint-path examples/checkpoints/bonsai-vineyard-sample-lora
	```

	The recipe's `examples/transfer_multicontrol_lora_nano/make_specs.py` writes the specs used for its results (seed
	image -> video: 149 frames, guidance 6; control modes: 61 frames, guidance 3, control guidance 0.9, 35 steps,
	shift 10). The Wan2.2 VAE is fetched automatically on first use.

	## Results

	Held-out day of the dataset, 6 clips, one seed, against the base checkpoint with identical inputs (mean over clips):

	\| Mode \| PSNR (dB), base -> LoRA \| LPIPS, base -> LoRA \|
	\|---\|---\|---\|
	\| Seg + depth -> video \| 11.82 -> 15.65 \| 0.463 -> 0.323 \|
	\| Seg -> video \| 12.13 -> 15.30 \| 0.466 -> 0.332 \|
	\| Depth -> video \| 12.31 -> 15.32 \| 0.476 -> 0.334 \|
	\| Seed image -> video \| 19.27 -> 19.39 \| 0.333 -> 0.318 \|

	The gain is mostly appearance: the LoRA renders the site's bright, dry daylight where the base model renders a
	generic low-sun vineyard. Full tables, the before/after comparison and a check on orchard and strawberry clips are in
	the recipe.

	## Limitations

	- Trained on a small sample dataset: 50 ten-second clips from one camera rig in one vineyard, daylight only. The
	results show what a small dataset can do, not the best this recipe can reach.
	- Objects the segmentation marks but the training clips rarely show (buildings at the row end, people) are not drawn.
	- Where the segmentation leaves part of the frame unassigned, such as a distant horizon, the LoRA fills it with the
	training site's background (dry hills, a fence).
	- In the control modes, leaves near the camera change more from frame to frame than in real video.

	## Licence

	The fine-tuned checkpoint is released under the
	[Creative Commons Attribution-NonCommercial 4.0 International licence (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/),
	the same terms as the dataset it was trained on: non-commercial use, with credit to Bonsai Robotics.

	It is derived from `nvidia/Cosmos3-Nano`, which NVIDIA releases under
	[OpenMDW-1.1](https://openmdw.ai/license/1-1/); the base model's terms and notices continue to apply to the parts of
	the weights that come from it.

	## Citation

	```bibtex
	@misc{bonsai_vineyard_sample_lora_2026,
	title = {Bonsai Vineyard Sample LoRA (Cosmos3-Nano)},
	author = {Zachariah, Sandeep Sam and {Bonsai Robotics}},
	year = {2026},
	howpublished = {\url{https://huggingface.co/BonsaiRobotics/bonsai-vineyard-sample-lora}}
	}
	```