Image-Text-to-Text
Transformers
Safetensors
English
qwen3_5
optics
photonics
physics
scientific-computing
code
reinforcement-learning
conversational
Instructions to use HololabTHU/HoloModel-Research with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use HololabTHU/HoloModel-Research with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="HololabTHU/HoloModel-Research") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("HololabTHU/HoloModel-Research") model = AutoModelForMultimodalLM.from_pretrained("HololabTHU/HoloModel-Research", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use HololabTHU/HoloModel-Research with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "HololabTHU/HoloModel-Research" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "HololabTHU/HoloModel-Research", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/HololabTHU/HoloModel-Research
- SGLang
How to use HololabTHU/HoloModel-Research with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "HololabTHU/HoloModel-Research" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "HololabTHU/HoloModel-Research", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "HololabTHU/HoloModel-Research" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "HololabTHU/HoloModel-Research", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use HololabTHU/HoloModel-Research with Docker Model Runner:
docker model run hf.co/HololabTHU/HoloModel-Research
|
Download README.md from HololabTHU/HoloModel-Research: direct link, hf CLI and curl.
- Browser
- Download file 7.41 kB
-
https://huggingface.co/HololabTHU/HoloModel-Research/resolve/main/README.md
- Command line
-
hf download hf://HololabTHU/HoloModel-Research/README.md
-
curl -L -o README.md https://huggingface.co/HololabTHU/HoloModel-Research/resolve/main/README.md
7.41 kB
| license: cc-by-nc-4.0 | |
| base_model: Qwen/Qwen3.8-27B | |
| library_name: transformers | |
| pipeline_tag: image-text-to-text | |
| language: | |
| - en | |
| tags: | |
| - optics | |
| - photonics | |
| - physics | |
| - scientific-computing | |
| - code | |
| - reinforcement-learning | |
| <div align="center"> | |
| <img src="assets/banner.webp" width="85%" alt="HoloModel"/> | |
| </div> | |
| <div align="center" style="line-height: 1;"> | |
| <a href="https://huggingface.co/HololabTHU/HoloModel-Research"><img alt="Model" src="https://img.shields.io/badge/%F0%9F%A4%97%20Model-HoloModel--Research-5799D1"/></a> | |
| <a href="https://huggingface.co/datasets/HololabTHU/HoloModel-Bench"><img alt="Benchmark" src="https://img.shields.io/badge/%F0%9F%93%8A%20Benchmark-HoloModel--Bench-5799D1"/></a> | |
| <a href="https://huggingface.co/HololabTHU"><img alt="Org" src="https://img.shields.io/badge/Hugging%20Face-HololabTHU-FFD21E"/></a> | |
| <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/License-CC%20BY--NC%204.0-F6DC8A"/></a> | |
| </div> | |
| ## 1. Model Introduction | |
| **HoloModel-Research** is an open-weight language model for autonomous research in optical physics. It writes correct numerical code for optics and physics problems, and it is the reinforcement-learning branch of **HoloModel**, the model family that drives the *Optics Scientists* agent team. The family has two models: **HoloModel-Knowledge**, enhanced with optics knowledge, and **HoloModel-Research**. The team consists of agents that can be deployed locally and carries open optics questions from the literature to verified discoveries in holography, metasurfaces, spectral sensing and diffractive optics. | |
| ### Background | |
| Frontier AI scientists depend on proprietary models that are costly, closed and hard to reproduce. HoloModel takes the opposite route: HoloModel-Research is a 27-billion-parameter open model (Qwen3.8-27B) post-trained with reinforcement learning, and it runs on a single GPU inside the laboratory. | |
| ### Key Features | |
| - 🔬 **Built for optical physics.** On **Optics-Phy-Bench**, 217 multi-step computations reproduced from 217 published papers, HoloModel-Research solves **69.1 %** of problems against **46.5 %** for its base model: **+22.6 points, a 49 % relative gain** (95 % CI +15.7 to +29.5). | |
| - 💻 **Stronger coder, not only in optics.** VCP code dev **+8.8** (P = 0.0015) and optics coding proxy **+5.4** over the base model. | |
| - 🥊 **Punches above its size.** With thinking off, HoloModel-Research is ahead of **DeepSeek V4.1 Flash** (552B total parameters) on Optics-Phy-Bench (**69.1 %** against 62.7 % of problems; +2.9 points on steps, P = 0.025) and by **11.3 points** on VCP code dev; it leads **GLM-5.3** and **Kimi-K3.1-Pro** on all five optics and coding sets. | |
| - 🔒 **Private and reproducible.** Fixed weights, temperature-0 evaluation and fully local deployment: data and unpublished designs never leave the lab. | |
| ## 2. Model Summary | |
| | | HoloModel-Research | | |
| |---|---| | |
| | **Developers** | Jinwen Wei, Liangcai Cao (Department of Precision Instrument, Tsinghua University) | | |
| | **Architecture** | Causal language model with vision encoder; 64 layers, hybrid linear and full attention | | |
| | **Parameters** | 27B (bfloat16, 18 safetensors shards, 52 GB) | | |
| | **Context length** | 262,144 tokens | | |
| | **Base model** | Qwen3.8-27B | | |
| | **Post-training** | Reinforcement learning | | |
| | **License** | CC BY-NC 4.0 (base model: Apache-2.0) | | |
| ## 3. Evaluation Results | |
| <div align="center"> | |
| <img src="assets/bench_bars.png" width="95%" alt="Optics and coding benchmarks"/> | |
| </div> | |
| ### Optics, physics and coding | |
| <div align="center"> | |
| | Benchmark | Items | Qwen3.8-27B | Kimi-K3.1-Pro | GLM-5.3 | DeepSeek V4.1 Flash | **HoloModel-Research** | | |
| |:---|---:|---:|---:|---:|---:|---:| | |
| | Optics-Phy-Bench, problems | 217 | 46.5 | 41.9 | 39.2 | 62.7 | **69.1** | | |
| | Optics-Phy-Bench, steps | 833 | 82.0 | 79.5 | 77.8 | 88.1 | **91.0** | | |
| | VCP code dev | 194 | 77.8 | 42.3 | 72.2 | 75.3 | **86.6** | | |
| | Optics coding proxy | 300 | 81.3 | 71.0 | 69.7 | 83.7 | **86.7** | | |
| | Physics dev | 198 | 85.4 | 72.7 | 65.7 | 84.3 | **87.9** | | |
| </div> | |
| <details> | |
| <summary><b>Evaluation protocol</b></summary> | |
| - All models: thinking disabled (GLM-5.3 at its lowest reasoning setting), temperature 0 (Kimi-K3.1-Pro: 0.6, the lowest its API accepts with thinking off), 14,000-token limit, one run per benchmark. | |
| - HoloModel-Research and the base model are compared item by item; confidence intervals are paired bootstrap 95 % intervals (10,000 resamples) and P values exact McNemar tests. | |
| - Optics-Phy-Bench: test inputs are passed to each step function as numpy arrays; a problem counts as solved only when every step function reproduces the reference outputs within tolerance. | |
| - **Decontamination:** no training prompt shares a 13-word sequence with Optics-Phy-Bench, physics dev or the coding proxy. VCP code dev is a split of the public `open-r1/verifiable-coding-problems-python` dataset; six items with more than half of their text inside one training prompt were removed, leaving 194. | |
| - The checkpoint (iteration 30) was chosen by a rule fixed before training, not by any benchmark. | |
| - Frontier models with reasoning enabled still score higher on Optics-Phy-Bench. | |
| </details> | |
| ## 4. Quickstart | |
| ### Transformers | |
| ```python | |
| from transformers import AutoProcessor, AutoModelForImageTextToText | |
| model_id = "HololabTHU/HoloModel-Research" | |
| processor = AutoProcessor.from_pretrained(model_id) | |
| model = AutoModelForImageTextToText.from_pretrained(model_id, torch_dtype="auto", device_map="auto") | |
| messages = [{"role": "user", "content": [{"type": "text", "text": | |
| "Write a Python function that propagates a 2D complex field by distance z with the angular spectrum method."}]}] | |
| inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, | |
| return_dict=True, return_tensors="pt", | |
| enable_thinking=False).to(model.device) | |
| out = model.generate(**inputs, max_new_tokens=4096, do_sample=False) | |
| print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) | |
| ``` | |
| ### vLLM | |
| ```bash | |
| vllm serve HololabTHU/HoloModel-Research --max-model-len 32768 | |
| ``` | |
| Send requests with `"chat_template_kwargs": {"enable_thinking": false}` and `"temperature": 0` to match the evaluation setting. One 80–96 GB GPU holds the bfloat16 weights. | |
| ## 5. Intended Use | |
| - Writing and checking numerical code for optics and physics: diffraction and propagation, holography and phase retrieval, metasurfaces and gratings, spectroscopy, thin films, resonators and coupled-mode models. | |
| - The language-model core of local research agents. | |
| - Generated code should be run and checked against physics (solvers, conservation laws, known limits) before its results are trusted. | |
| ## 6. License | |
| The HoloModel-Research weights and this card are released under **CC BY-NC 4.0**: free to use, share and adapt for **non-commercial** purposes, with attribution. The base model Qwen3.8-27B is licensed by the Qwen team under Apache-2.0; its license is included as `LICENSE-Qwen` and its original card as `README-Qwen3.8-27B.md`. | |
| ## 7. Citation | |
| ```bibtex | |
| @misc{holomodel2026, | |
| title = {HoloModel: language models for optics and physics research and knowledge}, | |
| author = {Wei, Jinwen and Cao, Liangcai}, | |
| year = {2026}, | |
| howpublished = {\url{https://huggingface.co/HololabTHU/HoloModel-Research}} | |
| } | |
| ``` | |