Instructions to use aisingapore/Qwen-SEA-LION-v4.5-27B-IT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aisingapore/Qwen-SEA-LION-v4.5-27B-IT with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="aisingapore/Qwen-SEA-LION-v4.5-27B-IT")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("aisingapore/Qwen-SEA-LION-v4.5-27B-IT")
model = AutoModelForMultimodalLM.from_pretrained("aisingapore/Qwen-SEA-LION-v4.5-27B-IT", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use aisingapore/Qwen-SEA-LION-v4.5-27B-IT with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aisingapore/Qwen-SEA-LION-v4.5-27B-IT"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aisingapore/Qwen-SEA-LION-v4.5-27B-IT",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/aisingapore/Qwen-SEA-LION-v4.5-27B-IT

SGLang

How to use aisingapore/Qwen-SEA-LION-v4.5-27B-IT with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "aisingapore/Qwen-SEA-LION-v4.5-27B-IT" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aisingapore/Qwen-SEA-LION-v4.5-27B-IT",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "aisingapore/Qwen-SEA-LION-v4.5-27B-IT" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aisingapore/Qwen-SEA-LION-v4.5-27B-IT",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use aisingapore/Qwen-SEA-LION-v4.5-27B-IT with Docker Model Runner:
```
docker model run hf.co/aisingapore/Qwen-SEA-LION-v4.5-27B-IT
```

Can you pls help check what's wrong with our benchmark?

by sprapp - opened 6 days ago

Discussion

sprapp

6 days ago

We would like to use the old benchmark as it is easier and cheaper to set up and test.

from here : https://www.youtube.com/watch?v=gWyz3u6jE-w&t=2047s

We're using SEA-LION v4.5 27B-IT (aisingapore/Qwen-SEA-LION-v4.5-27B-IT) as a baseline and want to score our own model apples-to-apples against the leaderboard. We're running the official SEA-HELM harness at tag v1.1.2 (commit 650cfea), Malay (ms), judge = gpt-4.1 via OpenRouter for MT.

Our measured v1.1.2 raw axis scores for v4.5 27B-IT:

axis	our v1.1.2 raw	(leaderboard / pre-Jul-10 card we have)
Instruction-Following (if-eval, overall_lang_normalized_acc)	89.52	85.08
Knowledge (global_mmlu_lite aggregate, normalized)	82.01	76.73
NLU (mean of sentiment 50.38 + belebele 93.89)	72.13	71.04
NLG (translation, MetricX-24 both dirs)	92.98	91.87
Safety (toxicity normalized_accuracy)	7.60	10.44
Multi-Turn (mt-bench pairwise weighted_win_rate, gpt-4.1)	73.47	82.98
MS (6-axis, incl MT)	69.62	69.69

The 6-axis MS nearly matches (69.62 vs 69.69), but several individual axes are off significantly — most notably Safety (7.60 vs 10.44), MT (73.47 vs 82.98), Knowledge (82.01 vs 76.73), IF (89.52 vs 85.08).

Questions:

Which harness version + commit was used to produce the leaderboard MS for v4.5 27B-IT? (v1.1.2, or v1.2.0 / v1.2.1?)
What is the exact MS aggregation formula for the Malay leaderboard — which axes are included, what weights, what normalization? Is it 5-axis (excl MT), 6-axis (incl MT), or 7-axis (incl the new Cultural axis)?
Safety: our toxicity/normalized_accuracy = 7.60, leaderboard = 10.44. Is the leaderboard "Safety" a different task mix (toxicity + safeguard + ...?), or a different normalization/baseline subtraction? How do we reproduce 10.44?
Multi-Turn: our pairwise weighted_win_rate (gpt-4.1, 58 questions × 2 turns) = 73.47, leaderboard = 82.98. Different judge model? Different reference answer set? Different question count or turn handling? How do we reproduce 82.98?
Are the v1.2.0 additions (safeguard benchmark, cultural_in_the_wild, Kalahi) required for the current leaderboard MS, and are those datasets public/gated? If gated, what's the access process?
Is there a reference config / run script you can share that reproduces the published v4.5 27B-IT leaderboard numbers end-to-end?

Happy to share our exact run configs + result JSONs if useful. Thanks!

Our exact setup (to attach):

Harness: SEA-HELM v1.1.2 (commit 650cfea3), Malay (ms).

Model serving: v4.5 27B-IT served via vLLM OpenAI-compatible server (--model-type local_openai, base_url=http://127.0.0.1:8113/v1), server --max-model-len 24576 (49152 for the MT-only run). Launched with --is_reasoning_model (v4.5 emits inline CoT) + --seed 25008113 + --rerun_cached_results.

Axes + metrics (our run configs):

axis	task(s)	metric	notes
Instruction-Following	if-eval	overall_lang_normalized_acc	max_tokens 4096, 0-shot
Knowledge	global_mmlu_lite (43 subjects)	normalized_accuracy, micro-avg	5-shot, max_tokens 16
NLU	sentiment + belebele-qa-mc	normalized_accuracy (each)	we average the two → 72.13
Safety	toxicity	normalized_accuracy	→ 7.60
Multi-Turn	mt-bench	weighted_win_rate (pairwise)	judge openai/gpt-4.1 via OpenRouter, temp 0.6, max_tokens 2048, no reference answers
NLG	translation (en-xx + xx-en)	normalized_metricx_wmt24_scores	MetricX-24-hybrid, 5-shot, max_tokens 256

Runs: split across 3 — run_1 (IF/Know/NLU/Safety + a broken MT that scored 0), run_5 (real MT = 73.47), run_6 (real NLG = 92.98). Aggregated: 5-axis MS (excl MT) = 68.85; 6-axis MS (incl MT) = 69.62.

Key config excerpts (verbatim from our run_config yaml):

MT judge (run_5):
metric: weighted_win_rate
judge:
judge_model_name: openai/gpt-4.1
judge_model_type: litellm
judge_init_args: {api_provider: openrouter}
judge_generation_kwargs: {temperature: 0.6, max_tokens: 2048}
judge_prompts: without-reference # pairwise, NO reference answer

Safety (run_1):
competency: safety
metric: normalized_accuracy # toxicity → 7.60

Knowledge (run_1):
competency: knowledge
aggregation_group: global_mmlu_lite
aggregation_type: micro-average
metric: normalized_accuracy # 43 subjects → 82.01
fewshot_num_examples: {instruct: 0, base: 5}

SAnocha

AI Singapore org 6 days ago

Hi,
Thanks for the report. Please find the suggestion below.

Which harness version + commit was used to produce the leaderboard MS for v4.5 27B-IT? (v1.1.2, or v1.2.0 / v1.2.1?)
Please use version v1.2.1

What is the exact MS aggregation formula for the Malay leaderboard — which axes are included, what weights, what normalization? Is it 5-axis (excl MT), 6-axis (incl MT), or 7-axis (incl the new Cultural axis)?
a. There are a few new tasks:
i. Cultural competency -> SEA-NLI
ii. Safety -> SEA-Safeguard
b. .MT Bench has also been updated. (See pt 4.)
c. All tasks are given equal weightage within their respective competencies and the final score is the average across the 7 axis for MS. Aggregation can be done using the helper Jupyter notebook found in the helpers folder

Safety: our toxicity/normalized_accuracy = 7.60, leaderboard = 10.44. Is the leaderboard "Safety" a different task mix (toxicity + safeguard + ...?), or a different normalization/baseline subtraction? How do we reproduce 10.44?
This is likely due to a different task mix. Please run the latest version of SEA-HELM

Multi-Turn: our pairwise weighted_win_rate (gpt-4.1, 58 questions × 2 turns) = 73.47, leaderboard = 82.98. Different judge model? Different reference answer set? Different question count or turn handling? How do we reproduce 82.98?
We have also updated SEA-MTBench to use a criteria-based judging rather than using pairwise comparisons. This tasks also now uses a smaller judge GPT-OSS-120B instead of GPT 4.1

Are the v1.2.0 additions (safeguard benchmark, cultural_in_the_wild, Kalahi) required for the current leaderboard MS, and are those datasets public/gated? If gated, what's the access process?
Please submit your request and we will review each application separately

Is there a reference config / run script you can share that reproduces the published v4.5 27B-IT leaderboard numbers end-to-end?
a. Please use the bash script. Use a total of 8 runs with different seeds.
b. Set TASKS to "seaheln sea-nil sea-safeguard"
c. Set MODEL_ARGS as required (eg. Reasoning parser should be set for reasoning models)

sprapp

5 days ago

•

edited 5 days ago

@SAnocha , thx for the prompt reply:

Q1 (refined) — GPT-OSS-120B MT judge: We'll use GPT-OSS-120B via OpenRouter. Confirm: (a) OpenRouter's GPT-OSS-120B = leaderboard's exact judge version; (b) the criteria-based judging rubric/prompt + worked example; (c) pairwise vs pointwise; (d) 8 seeds + 30 bootstrap?

Q2 (refined) — Safety competency def: Public sea_safeguard config = 5 subtasks (general_prompt/response, cultural_content_generation_prompt/response, cultural_in_the_wild), metric: normalized_accuracy, model-as-classifier (Safe/Harmful).
Confirm v1.2.1 Safety = these 5 subtasks model-as-classifier normalized_accuracy — NOT AUPRC, NOT a SEA-Guard judge, NOT including toxicity? (toxicity = separate toxicity_detection task?) Our v1.1.2 safety_official_ms was 2-group mean (5 safeguard + toxicity) — does v1.2.1 DROP toxicity from Safety?

Q3 (NEW) — normalization: Leaderboard = normalized (score−baseline)/(max−baseline)×100, 8 runs + 30 bootstrap. What are baseline + max per competency (Safety, Cultural, MT, IF)? Our raw accuracy → needs this to compare.

Q4 — reasoning parser for Qwen3.5-MoE: Exact --is_reasoning_model + reasoning_generation_config for our 35B-A3B (Qwen3.5-MoE). Does the parser handle our arch?

Q5 — 7-axis + Qwen 397B 71.94 confirm: Is the LIVE leaderboard MS = 7-axis avg incl Cultural/SEA-NLI? Is published Qwen 397B 71.94 + v4.5 69.69 the 7-axis v1.2.1 number? (our table has NO Cultural column — old 6-axis card or live 7-axis?)

Jian-Gang

AI Singapore org 5 days ago

Hello, thank you for your great interest in SEA-HELM. It seems like this is a conversation that is easier to continue offline via email. Please reach out to us at sealion@aisingapore.org!

sprapp

5 days ago

@Jian-Gang @SAnocha

pls allow access to https://huggingface.co/datasets/aisingapore/MultiTurn-Chat-MT-Bench-Judge
it's the last missing piece needed to use the benchmark.

you have already allowed access to nli etc.

sprapp

5 days ago

@Jian-Gang @SAnocha already emailed.

please help us make the benchmark analysis complete faster because it's costly to let our gpus run idle.

we need the datasets access and the questions answered as detailed as possible for the analysis.

thank you

SAnocha changed discussion status to closed 4 days ago

sprapp

3 days ago

@SAnocha have yet to receive feedback by email, can you help?
just these datasets needed for benchmark analysis
https://huggingface.co/datasets/aisingapore/MultiTurn-Chat-MT-Bench-Judge

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment