One model for camera traps and Galápagos animals (LoRA), LLaVA-OneVision 0.5B

One model for two jobs: camera traps, and the Galápagos animals Google's SpeciesNet has no label for. It scores 93.33% on mockingbird versus brown noddy, 91.69% on marine iguana versus green iguana and 87.58% on land iguana versus green iguana, and names 88.20% of invasive rats as rats, while holding camera traps at 79.22% (64.09% unedited). Frontier AI started from the camera-trap edit (an earlier direct edit of the weights for camera-trap answers), built this model and a direct-edit rival from the same 2,902 labelled practice photos, and measured both. This one is a standard LoRA fine-tune (a small trained add-on to the weights, merged back in), trained in about three hours on one desktop computer.

Explainer · Paper · Code, data and results

Results

Model Mockingbird vs brown noddy Land iguana vs green iguana Marine iguana vs green iguana Invasive rats found Camera traps
Island edit 98.33% 81.22% 80.37% 81.23% 72.14%
One model for both jobs (LoRA) 93.33% 87.58% 91.69% 88.20% 79.22%
One model for both jobs (direct edit) 90.83% 55.49% 55.49% 17.70% 80.70%
Camera-trap edit 91.67% 51.83% 51.83% 13.75% 81.76%
Unedited LLaVA-OneVision 0.5B 70.83% 50.00% 50.00% 2.96% 64.09%
SpeciesNet (standard output) 3.33% 12.44% 12.44% 8.11% —

Island scores come from 697 test crops from places not used to build any model. Each native-versus-look-alike score averages two shares, natives named as natives and look-alikes named as look-alikes, with each photo location counting equally; rat recall also weights locations equally. The camera-trap column is balanced accuracy on 1,611 crops from 45 cameras, each with its own list of names, with every camera and every animal within it counted equally. SpeciesNet has no label for the natives, and its standard output rarely names the look-alikes, so it scores below 50%, the score of guessing. SpeciesNet was scored on the island test only.

How it was made

A LoRA fine-tune of the camera-trap edit, merged into full weights. Rank-8 updates (alpha 16) were trained on the attention query and value matrices of all 24 language blocks: 540,672 parameters. One seeded pass covered 2,902 labelled practice crops, 2,095 from camera traps and 807 island photos, with each domain carrying half the loss. There was one fixed recipe and no search over settings. Training took about three hours on one Mac Studio. The Codex research assistant (OpenAI GPT) built and evaluated it. A direct edit for both jobs, built in about an hour from the same crops, scored 80.70% on camera traps, 1.48 points above this model, within the test's noise (95% interval −2.54 to +5.80), but lost most of the island gains.

How to use it

import torch
from PIL import Image
from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration

repo = "andrewxhill/camera-island-lora"
processor = AutoProcessor.from_pretrained(repo)
model = LlavaOnevisionForConditionalGeneration.from_pretrained(repo, dtype=torch.float32).eval()

names = ["Galapagos land iguana", "Santa Fe land iguana", "Galapagos pink land iguana",
         "marine iguana", "common green iguana", "black rat", "house mouse",
         "domestic cat", "domestic goat", "pig", "European rabbit"]
question = ("Which animal is shown in this crop?\n"
            + "\n".join(f"{chr(65 + i)}. {n}" for i, n in enumerate(names))
            + "\nAnswer with the animal's name from the choices, and no other text.")
messages = [{"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": [{"type": "image", "image": Image.open("crop.jpg").convert("RGB")},
                                         {"type": "text", "text": question}]}]
inputs = processor.apply_chat_template(messages, tokenize=True, add_generation_prompt=True,
                                       return_dict=True, return_tensors="pt")
out = model.generate(**inputs, do_sample=False, max_new_tokens=1024, repetition_penalty=1.1)
print(processor.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Give it a crop around one animal. The reply may be a name, a letter or both; the study's parser, code/eval/parser.py, accepts either. The study used three fixed 11-name lists, for iguanas, birds and rodents; all three are in code/island/menus.json. The bird list is: Galapagos mockingbird, San Cristobal mockingbird, Floreana mockingbird, Espanola mockingbird, Galapagos dove, yellow warbler, blue-footed booby, brown noddy, Galapagos petrel, domestic cat, black rat. The rodent list is: large Fernandina rice rat, Santiago Galapagos mouse, small Fernandina rice rat, Galapagos rice rat, black rat, brown rat, Polynesian rat, house mouse, domestic cat, South American coati, European rabbit. The scores above used float32, greedy decoding, 1,024 maximum new tokens and repetition penalty 1.1.

Intended use

Naming animals in cropped camera-trap frames from a per-camera list, and Galápagos iguanas, mockingbirds and invasive rats from the 11-name island lists, for research on place-specific wildlife models.

Data terms. Each iNaturalist photo used to fit this model keeps its own licence (CC0, CC-BY or CC-BY-SA); the data manifests list them with credits.

Rebuild from the public base

The delta/ folder rebuilds these weights exactly from lmms-lab/llava-onevision-qwen2-0.5b-ov at revision 381d9947148efb1e58a577f451c05705ceec666e. The delta holds the camera-trap edit's changes plus the 48 merged query and value matrices (91 MB). The full weights are stored in four shards; the rebuild writes two, with identical tensors.

git clone https://github.com/andrewxhill/frontier-model-weight-edits && cd frontier-model-weight-edits/code
pip install -r requirements.txt
python apply_edit.py --base lmms-lab/llava-onevision-qwen2-0.5b-ov \
  --revision 381d9947148efb1e58a577f451c05705ceec666e \
  --delta path/to/delta/delta.safetensors --manifest path/to/delta/delta_manifest.json --output rebuilt

The script checks the base and every replaced tensor against SHA256 hashes and confirms the finished state.

Licence and citation

CC BY-NC 4.0; see LICENSE. The weights were fitted on iNaturalist photos, and iNaturalist's terms forbid using its data to train AI for commercial purposes. The base model and its Qwen2 and SigLIP components are Apache-2.0; their licence and the changes made are in upstream-licenses/ and NOTICE.md. Please cite:

Andrew Hill (2026). Can weight-space programming surface the domain expertise hidden in a small vision-language model? Evidence from camera traps and Galápagos species https://andrewxhill.com/frontier-model-weight-edits/paper.pdf

Downloads last month
-
Safetensors
Model size
0.9B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for andrewxhill/camera-island-lora

Adapter
(8)
this model