cmriat/lion-ER-4B

Lion-ER: LionRock Embodied Reasoner

Lion-ER-4B (LionRock Embodied Reasoner) is a multimodal reasoning model from Lionrock Artificial Intelligence Laboratory for understanding scenes, spatial relationships, object interactions, and action-oriented visual questions. Built on Qwen3.5-4B, it uses a dense architecture.

Training mixture

The supervised training mixture spans complementary visual and embodied reasoning capabilities. The inner ring summarizes broad capability areas; the middle ring shows finer skills, and the outer ring groups existing task labels within each skill. Descendants share their parent area's color.

Training mixture by capability

Shares are calculated over 312,380 supervised training examples, not unique questions, tokens, or measured ability. If a question has multiple retained reasoning trajectories, each resulting training example is counted separately. Each example is assigned one primary capability using its existing task annotation; broad or insufficiently specific annotations remain in Mixed / unspecified. This grouping is descriptive, not an independent semantic annotation of every example.

Values are rounded to two decimal places, so the displayed shares sum to 100.02%; the unrounded total is 100%. Small sectors are retained even when their labels do not fit.

Benchmark results

The main benchmark and ERQA figures show Lion-ER-35B-A3B. Results for Lion-ER-4B and Lion-ER-9B are reported in the “Lion-ER across model sizes” table.

The main comparison uses avg@4: the mean score over four sampled responses, not majority voting, pass@4, or best-of-four selection.

All benchmark figures and the score table below use the same ten benchmarks. RealWorldQA contributes once to the equal-weight Average. Lion-ER-35B-A3B's Average is 65.29.

Per-benchmark comparison

Benchmark scorecard

Cross-benchmark profile

The radar uses the same ten benchmark values on a shared 0–100 scale. Average is not an extra axis. Line styles and markers distinguish models; radar area is not a separate ranking metric.

RoboBench aggregates the same five rule-scored tasks. BLINK preserves its native subtask aggregation. Where2Place reports placement score on a 0–100 scale; not every column represents multiple-choice accuracy.

RealWorldQA combines rule-scored multiple-choice questions with short answers evaluated by Qwen3.8-27B using a strict three-judgment agreement criterion, weighted by question count. Scoring and result-selection details are documented in the evaluation notes.

Small score differences should not be interpreted as statistically significant.

Lion-ER across model sizes

Lion-ER also adapts the Qwen3.5-4B and Qwen3.5-9B backbones with the same training recipe. Each trained model is shown alongside its corresponding official base model. Δ reports the absolute score difference in percentage points.

ERQA capability breakdown

The eight axes preserve the original ERQA categories under the aligned evaluation protocol. All curves use avg@4; Overall is reported separately and is not an unweighted average of the eight categories.

ERQA capability profile

ERQA capability scorecard

The scorecard reports the same eight categories plus the original Overall score. Bold, underlined entries indicate the highest displayed score in each category; the blue column identifies Lion-ER-35B-A3B.

Quick start

Evaluation system prompt and ERQA example

The examples below use the system prompt from our Lion-ER and Qwen comparison runs. Benchmark-specific answer-format instructions are added to the user question. This does not imply that all historical reference models used the same system prompt.

You are answering questions about scenes shown in one or more images. Reason carefully about what the images actually show before committing to an answer. When the question is multiple-choice (the options are labeled A, B, C, ...), follow the answer-format instruction given with the question to present your final choice; if the question does not specify a format, end your response with the chosen letter wrapped in \boxed{}, for example \boxed{X}. For non-multiple-choice questions, answer directly and concisely.

We use ERQA_155, a single-image trajectory-reasoning question from ERQA, provided by Google DeepMind under CC BY 4.0 and accessed through the FlagEval dataset conversion. The reference answer is A — the orange trajectory; it is not supplied to the model. The question and choices are preserved, with the answer-format instruction used in our ERQA evaluation. The image is fetched from the public dataset at runtime and is not bundled with this model.

Transformers

Install the dependencies in a dedicated environment. Use matching PyTorch and torchvision builds appropriate for your hardware:

pip install torch torchvision transformers accelerate pillow

Load the model and answer the ERQA example:

import json
from io import BytesIO
from urllib.request import urlopen

import torch
from PIL import Image
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration

system_prompt = (
    'You are answering questions about scenes shown in one or more images. Reason'
    ' carefully about what the images actually show before committing to an answer. When'
    ' the question is multiple-choice (the options are labeled A, B, C, ...), follow the'
    ' answer-format instruction given with the question to present your final choice; if'
    ' the question does not specify a format, end your response with the chosen letter'
    ' wrapped in \\boxed{}, for example \\boxed{X}. For non-multiple-choice questions, answer'
    ' directly and concisely.'
)

# Fetch a fresh image URL; dataset-viewer image links expire.
example_url = (
    "https://datasets-server.huggingface.co/rows?"
    "dataset=FlagEval%2FERQA&config=default&split=test&offset=154&length=1"
)
with urlopen(example_url, timeout=60) as response:
    example = json.load(response)["rows"][0]["row"]
assert example["question_id"] == "ERQA_155" and len(example["images"]) == 1
with urlopen(example["images"][0]["src"], timeout=60) as response:
    image_bytes = response.read()

# Preserve the question and choices; use the ERQA evaluation answer format.
question = example["question"].replace(
    " Please answer directly with only the letter of the correct option and nothing else.",
    "",
)
question += "\n" + 'End your response with the chosen option letter wrapped in \\boxed{}, for example \\boxed{A}.'
image = Image.open(BytesIO(image_bytes)).convert("RGB")

model_id = "cmriat/lion-ER-4B"
model = Qwen3_5ForConditionalGeneration.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
).eval()
processor = AutoProcessor.from_pretrained(model_id)

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": question},
    ]},
]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        do_sample=True,
        temperature=1.0,
        top_p=0.95,
        top_k=20,
        max_new_tokens=65536,
    )

generated = output[:, inputs["input_ids"].shape[-1]:]
print(processor.batch_decode(generated, skip_special_tokens=True)[0])

Serving with vLLM

Install vLLM in a separate environment:

pip install vllm openai

Start an OpenAI-compatible API server:

vllm serve cmriat/lion-ER-4B \
  --host 127.0.0.1 \
  --port 8000 \
  --dtype bfloat16 \
  --max-model-len 262144 \
  --reasoning-parser qwen3

Adjust --tensor-parallel-size and --max-model-len to suit your available GPU memory. Keep the service local or behind an authenticated gateway.

Image input via the API

With the server running, send the same ERQA image and question:

import base64
import json
from urllib.request import urlopen
from openai import OpenAI

system_prompt = (
    'You are answering questions about scenes shown in one or more images. Reason'
    ' carefully about what the images actually show before committing to an answer. When'
    ' the question is multiple-choice (the options are labeled A, B, C, ...), follow the'
    ' answer-format instruction given with the question to present your final choice; if'
    ' the question does not specify a format, end your response with the chosen letter'
    ' wrapped in \\boxed{}, for example \\boxed{X}. For non-multiple-choice questions, answer'
    ' directly and concisely.'
)

# Fetch a fresh image URL; dataset-viewer image links expire.
example_url = (
    "https://datasets-server.huggingface.co/rows?"
    "dataset=FlagEval%2FERQA&config=default&split=test&offset=154&length=1"
)
with urlopen(example_url, timeout=60) as response:
    example = json.load(response)["rows"][0]["row"]
assert example["question_id"] == "ERQA_155" and len(example["images"]) == 1
with urlopen(example["images"][0]["src"], timeout=60) as response:
    image_bytes = response.read()

# Preserve the question and choices; use the ERQA evaluation answer format.
question = example["question"].replace(
    " Please answer directly with only the letter of the correct option and nothing else.",
    "",
)
question += "\n" + 'End your response with the chosen option letter wrapped in \\boxed{}, for example \\boxed{A}.'
image_b64 = base64.b64encode(image_bytes).decode("ascii")
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY", timeout=1800)

response = client.chat.completions.create(
    model="cmriat/lion-ER-4B",
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": [
            {"type": "image_url", "image_url": {
                "url": f"data:image/jpeg;base64,{image_b64}"
            }},
            {"type": "text", "text": question},
        ]},
    ],
    temperature=1.0,
    top_p=0.95,
    max_tokens=65536,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": True},
    },
)

message = response.choices[0].message
reasoning = getattr(message, "reasoning", None) or getattr(message, "reasoning_content", None)
if reasoning:
    print("Thinking:", reasoning)
print("Answer:", message.content or "")

These examples generate one response per request. The reported avg@4 scores average four sampled responses per question using benchmark-specific prompts and scoring. Model weights are provided in BF16 Safetensors format.

Intended use and limitations

Intended for research and development in visual and embodied reasoning. Predictions and explanations can be incorrect, and spatial outputs should not be treated as verified geometry. The model has not been validated for safety-critical decisions or autonomous robot control. Benchmark performance does not guarantee transfer to new environments, and comprehensive train–test contamination freedom has not been independently established.

Attribution and terms

Lion-ER-4B is released under the Apache License 2.0.

Based on Qwen3.5-4B, developed by the Qwen team. This is an independently fine-tuned derivative with modified model weights, not an official Qwen release, and implies no endorsement. The upstream model is also provided under Apache 2.0; its license is preserved in UPSTREAM_LICENSE. See ATTRIBUTION.md for attribution and modification details. This release does not grant rights to source datasets or images, which are not included.