Instructions to use andrewxhill/camera-island-direct-edit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use andrewxhill/camera-island-direct-edit with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="andrewxhill/camera-island-direct-edit") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("andrewxhill/camera-island-direct-edit") model = AutoModelForMultimodalLM.from_pretrained("andrewxhill/camera-island-direct-edit", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use andrewxhill/camera-island-direct-edit with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "andrewxhill/camera-island-direct-edit" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "andrewxhill/camera-island-direct-edit", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/andrewxhill/camera-island-direct-edit
- SGLang
How to use andrewxhill/camera-island-direct-edit with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "andrewxhill/camera-island-direct-edit" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "andrewxhill/camera-island-direct-edit", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "andrewxhill/camera-island-direct-edit" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "andrewxhill/camera-island-direct-edit", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use andrewxhill/camera-island-direct-edit with Docker Model Runner:
docker model run hf.co/andrewxhill/camera-island-direct-edit
Direct combined edit for camera traps and Galápagos animals, LLaVA-OneVision 0.5B
The direct-edit route to one model for camera traps and Galápagos animals. Frontier AI started from the camera-trap edit (an earlier direct edit for camera-trap answers) and solved one joint ridge regression, a least-squares fit with a penalty that keeps the numbers small. It wrote island-name hints into 54 projector units that the earlier edit had left unused. The projector is the small network between the image encoder and the language model. There was no training run, and the build took about an hour on one desktop computer. It holds camera traps at 80.70% and raises mockingbird versus brown noddy to 90.83%, but reaches only 55.49% on the iguana contrasts and names 17.70% of invasive rats as rats. The LoRA model built from the same photos scores 87.58% to 93.33% on the island contrasts; use it for island work.
Explainer · Paper · Code, data and results
Results
| Model | Mockingbird vs brown noddy | Land iguana vs green iguana | Marine iguana vs green iguana | Invasive rats found | Camera traps |
|---|---|---|---|---|---|
| Island edit | 98.33% | 81.22% | 80.37% | 81.23% | 72.14% |
| One model for both jobs (LoRA) | 93.33% | 87.58% | 91.69% | 88.20% | 79.22% |
| One model for both jobs (direct edit) | 90.83% | 55.49% | 55.49% | 17.70% | 80.70% |
| Camera-trap edit | 91.67% | 51.83% | 51.83% | 13.75% | 81.76% |
| Unedited LLaVA-OneVision 0.5B | 70.83% | 50.00% | 50.00% | 2.96% | 64.09% |
| SpeciesNet (standard output) | 3.33% | 12.44% | 12.44% | 8.11% | — |
Island scores come from 697 test crops from places not used to build any model. Each native-versus-look-alike score averages two shares, natives named as natives and look-alikes named as look-alikes, with each photo location counting equally; rat recall also weights locations equally. The camera-trap column is balanced accuracy on 1,611 crops from 45 cameras, each with its own list of names, with every camera and every animal within it counted equally. SpeciesNet has no label for the natives, and its standard output rarely names the look-alikes, so it scores below 50%, the score of guessing. SpeciesNet was scored on the island test only.
How it was made
Starting from the camera-trap edit, one joint ridge fit added island-name hints to projector units the earlier edit had left unused, while asking for no change on 2,095 camera practice crops. It used 807 island practice photos. Eight candidates crossed four camera-to-island weight ratios with two hint magnitudes. Selection on 194 camera and 426 island crops chose ratio 0.1 and magnitude 0.6. The fit took under a minute and the whole build about an hour on one Mac Studio. The Codex research assistant (OpenAI GPT) built and evaluated it.
How to use it
import torch
from PIL import Image
from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration
repo = "andrewxhill/camera-island-direct-edit"
processor = AutoProcessor.from_pretrained(repo)
model = LlavaOnevisionForConditionalGeneration.from_pretrained(repo, dtype=torch.float32).eval()
names = ["Galapagos land iguana", "Santa Fe land iguana", "Galapagos pink land iguana",
"marine iguana", "common green iguana", "black rat", "house mouse",
"domestic cat", "domestic goat", "pig", "European rabbit"]
question = ("Which animal is shown in this crop?\n"
+ "\n".join(f"{chr(65 + i)}. {n}" for i, n in enumerate(names))
+ "\nAnswer with the animal's name from the choices, and no other text.")
messages = [{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": [{"type": "image", "image": Image.open("crop.jpg").convert("RGB")},
{"type": "text", "text": question}]}]
inputs = processor.apply_chat_template(messages, tokenize=True, add_generation_prompt=True,
return_dict=True, return_tensors="pt")
out = model.generate(**inputs, do_sample=False, max_new_tokens=1024, repetition_penalty=1.1)
print(processor.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
Give it a crop around one animal. The reply may be a name, a letter or both; the study's parser, code/eval/parser.py, accepts either. The study used three fixed 11-name lists, for iguanas, birds and rodents; all three are in code/island/menus.json. The bird list is: Galapagos mockingbird, San Cristobal mockingbird, Floreana mockingbird, Espanola mockingbird, Galapagos dove, yellow warbler, blue-footed booby, brown noddy, Galapagos petrel, domestic cat, black rat. The rodent list is: large Fernandina rice rat, Santiago Galapagos mouse, small Fernandina rice rat, Galapagos rice rat, black rat, brown rat, Polynesian rat, house mouse, domestic cat, South American coati, European rabbit. The scores above used float32, greedy decoding, 1,024 maximum new tokens and repetition penalty 1.1.
Intended use
Comparing a direct edit with the LoRA model. For Galápagos animals, use the island edit or the LoRA model.
Data terms. Each iNaturalist photo used to fit this model keeps its own licence (CC0, CC-BY or CC-BY-SA); the data manifests list them with credits.
Rebuild from the public base
The delta/ folder rebuilds these weights exactly from lmms-lab/llava-onevision-qwen2-0.5b-ov at revision 381d9947148efb1e58a577f451c05705ceec666e. The delta holds the camera-trap edit's changes with the new projector tensors (7.4 MB).
git clone https://github.com/andrewxhill/frontier-model-weight-edits && cd frontier-model-weight-edits/code
pip install -r requirements.txt
python apply_edit.py --base lmms-lab/llava-onevision-qwen2-0.5b-ov \
--revision 381d9947148efb1e58a577f451c05705ceec666e \
--delta path/to/delta/delta.safetensors --manifest path/to/delta/delta_manifest.json --output rebuilt
The script checks the base and every replaced tensor against SHA256 hashes and confirms the finished state.
Licence and citation
CC BY-NC 4.0; see LICENSE. The weights were fitted on iNaturalist photos, and iNaturalist's terms forbid using its data to train AI for commercial purposes. The base model and its Qwen2 and SigLIP components are Apache-2.0; their licence and the changes made are in upstream-licenses/ and NOTICE.md. Please cite:
Andrew Hill (2026). Can weight-space programming surface the domain expertise hidden in a small vision-language model? Evidence from camera traps and Galápagos species https://andrewxhill.com/frontier-model-weight-edits/paper.pdf
- Downloads last month
- -
Model tree for andrewxhill/camera-island-direct-edit
Base model
lmms-lab/llava-onevision-qwen2-0.5b-ov