Instructions to use andrewxhill/camera-island-lora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use andrewxhill/camera-island-lora with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="andrewxhill/camera-island-lora") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("andrewxhill/camera-island-lora") model = AutoModelForMultimodalLM.from_pretrained("andrewxhill/camera-island-lora", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use andrewxhill/camera-island-lora with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "andrewxhill/camera-island-lora" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "andrewxhill/camera-island-lora", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/andrewxhill/camera-island-lora
- SGLang
How to use andrewxhill/camera-island-lora with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "andrewxhill/camera-island-lora" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "andrewxhill/camera-island-lora", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "andrewxhill/camera-island-lora" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "andrewxhill/camera-island-lora", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use andrewxhill/camera-island-lora with Docker Model Runner:
docker model run hf.co/andrewxhill/camera-island-lora
One model for camera traps and Galápagos animals (LoRA), LLaVA-OneVision 0.5B
One model for two jobs: camera traps, and the Galápagos animals Google's SpeciesNet has no label for. It scores 93.33% on mockingbird versus brown noddy, 91.69% on marine iguana versus green iguana and 87.58% on land iguana versus green iguana, and names 88.20% of invasive rats as rats, while holding camera traps at 79.22% (64.09% unedited). Frontier AI started from the camera-trap edit (an earlier direct edit of the weights for camera-trap answers), built this model and a direct-edit rival from the same 2,902 labelled practice photos, and measured both. This one is a standard LoRA fine-tune (a small trained add-on to the weights, merged back in), trained in about three hours on one desktop computer.
Explainer · Paper · Code, data and results
Results
| Model | Mockingbird vs brown noddy | Land iguana vs green iguana | Marine iguana vs green iguana | Invasive rats found | Camera traps |
|---|---|---|---|---|---|
| Island edit | 98.33% | 81.22% | 80.37% | 81.23% | 72.14% |
| One model for both jobs (LoRA) | 93.33% | 87.58% | 91.69% | 88.20% | 79.22% |
| One model for both jobs (direct edit) | 90.83% | 55.49% | 55.49% | 17.70% | 80.70% |
| Camera-trap edit | 91.67% | 51.83% | 51.83% | 13.75% | 81.76% |
| Unedited LLaVA-OneVision 0.5B | 70.83% | 50.00% | 50.00% | 2.96% | 64.09% |
| SpeciesNet (standard output) | 3.33% | 12.44% | 12.44% | 8.11% | — |
Island scores come from 697 test crops from places not used to build any model. Each native-versus-look-alike score averages two shares, natives named as natives and look-alikes named as look-alikes, with each photo location counting equally; rat recall also weights locations equally. The camera-trap column is balanced accuracy on 1,611 crops from 45 cameras, each with its own list of names, with every camera and every animal within it counted equally. SpeciesNet has no label for the natives, and its standard output rarely names the look-alikes, so it scores below 50%, the score of guessing. SpeciesNet was scored on the island test only.
How it was made
A LoRA fine-tune of the camera-trap edit, merged into full weights. Rank-8 updates (alpha 16) were trained on the attention query and value matrices of all 24 language blocks: 540,672 parameters. One seeded pass covered 2,902 labelled practice crops, 2,095 from camera traps and 807 island photos, with each domain carrying half the loss. There was one fixed recipe and no search over settings. Training took about three hours on one Mac Studio. The Codex research assistant (OpenAI GPT) built and evaluated it. A direct edit for both jobs, built in about an hour from the same crops, scored 80.70% on camera traps, 1.48 points above this model, within the test's noise (95% interval −2.54 to +5.80), but lost most of the island gains.
How to use it
import torch
from PIL import Image
from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration
repo = "andrewxhill/camera-island-lora"
processor = AutoProcessor.from_pretrained(repo)
model = LlavaOnevisionForConditionalGeneration.from_pretrained(repo, dtype=torch.float32).eval()
names = ["Galapagos land iguana", "Santa Fe land iguana", "Galapagos pink land iguana",
"marine iguana", "common green iguana", "black rat", "house mouse",
"domestic cat", "domestic goat", "pig", "European rabbit"]
question = ("Which animal is shown in this crop?\n"
+ "\n".join(f"{chr(65 + i)}. {n}" for i, n in enumerate(names))
+ "\nAnswer with the animal's name from the choices, and no other text.")
messages = [{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": [{"type": "image", "image": Image.open("crop.jpg").convert("RGB")},
{"type": "text", "text": question}]}]
inputs = processor.apply_chat_template(messages, tokenize=True, add_generation_prompt=True,
return_dict=True, return_tensors="pt")
out = model.generate(**inputs, do_sample=False, max_new_tokens=1024, repetition_penalty=1.1)
print(processor.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
Give it a crop around one animal. The reply may be a name, a letter or both; the study's parser, code/eval/parser.py, accepts either. The study used three fixed 11-name lists, for iguanas, birds and rodents; all three are in code/island/menus.json. The bird list is: Galapagos mockingbird, San Cristobal mockingbird, Floreana mockingbird, Espanola mockingbird, Galapagos dove, yellow warbler, blue-footed booby, brown noddy, Galapagos petrel, domestic cat, black rat. The rodent list is: large Fernandina rice rat, Santiago Galapagos mouse, small Fernandina rice rat, Galapagos rice rat, black rat, brown rat, Polynesian rat, house mouse, domestic cat, South American coati, European rabbit. The scores above used float32, greedy decoding, 1,024 maximum new tokens and repetition penalty 1.1.
Intended use
Naming animals in cropped camera-trap frames from a per-camera list, and Galápagos iguanas, mockingbirds and invasive rats from the 11-name island lists, for research on place-specific wildlife models.
Data terms. Each iNaturalist photo used to fit this model keeps its own licence (CC0, CC-BY or CC-BY-SA); the data manifests list them with credits.
Rebuild from the public base
The delta/ folder rebuilds these weights exactly from lmms-lab/llava-onevision-qwen2-0.5b-ov at revision 381d9947148efb1e58a577f451c05705ceec666e. The delta holds the camera-trap edit's changes plus the 48 merged query and value matrices (91 MB). The full weights are stored in four shards; the rebuild writes two, with identical tensors.
git clone https://github.com/andrewxhill/frontier-model-weight-edits && cd frontier-model-weight-edits/code
pip install -r requirements.txt
python apply_edit.py --base lmms-lab/llava-onevision-qwen2-0.5b-ov \
--revision 381d9947148efb1e58a577f451c05705ceec666e \
--delta path/to/delta/delta.safetensors --manifest path/to/delta/delta_manifest.json --output rebuilt
The script checks the base and every replaced tensor against SHA256 hashes and confirms the finished state.
Licence and citation
CC BY-NC 4.0; see LICENSE. The weights were fitted on iNaturalist photos, and iNaturalist's terms forbid using its data to train AI for commercial purposes. The base model and its Qwen2 and SigLIP components are Apache-2.0; their licence and the changes made are in upstream-licenses/ and NOTICE.md. Please cite:
Andrew Hill (2026). Can weight-space programming surface the domain expertise hidden in a small vision-language model? Evidence from camera traps and Galápagos species https://andrewxhill.com/frontier-model-weight-edits/paper.pdf
- Downloads last month
- -
Model tree for andrewxhill/camera-island-lora
Base model
lmms-lab/llava-onevision-qwen2-0.5b-ov