HumanoidSafe-Qwen3-VL-8B-GRPO-Rule-Full

Final Qwen3-VL-8B-Instruct RLVR baseline trained with GRPO, rule-based refusal classifier reward, and full-parameter fine-tuning on the original HumanoidSafe training split. The fixed training budget was 3 epochs, seed 42. This is experiment grpo-rule-full in the completed eight-baseline report.

This repository contains a standalone full model with all language and vision weights, plus tokenizer and processor files. It loads directly with Qwen3VLForConditionalGeneration; no adapter or separate base-model download is needed.

These are inference checkpoints. Optimizer state, training-resume state, and the training-only PPO value head are not included.

Loading

Base model: Qwen/Qwen3-VL-8B-Instruct, revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b. Install the recorded versions in requirements.txt; the observed runtime used Python 3.9.21.

import torch
from transformers import AutoProcessor, GenerationConfig, Qwen3VLForConditionalGeneration

REPO = "SLOCBench/HumanoidSafe-Qwen3-VL-8B-GRPO-Rule-Full"

model = Qwen3VLForConditionalGeneration.from_pretrained(
    REPO,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="sdpa",
).eval()
processor = AutoProcessor.from_pretrained(REPO)
model.generation_config = GenerationConfig.from_pretrained(REPO)
model.config.use_cache = True
model.config.text_config.use_cache = True

load_model.py contains the same loading example. It explicitly loads this repository's generation_config.json, including for PEFT adapters, so the defaults are greedy generation, 256 new tokens, and cache enabled. Benchmark generation additionally requires the preprocessing and system prompt described below. For reproducible downstream work, pin this repository to its published commit as well.

Training configuration

Setting Value
Dataset SLOCBench/HumanoidSafe, config checked
Dataset revision 7952f2e870fb8f1dae6bc30ad6c5f09e845cbb0a
Split Original train: 294 pairs / 588 safe and harmful instruction branches
Training prompts base only
Budget 3 epochs; seed 42; 1,764 active prompt presentations; 7,056 valid sampled completions
Update schedule 222 rollout updates; two policy passes per update; 444 optimizer steps
Batch 8 active prompts ร— 4 completions, except each epoch's tail batch of 4 prompts
Sampling Temperature 1.0, top-p 1.0, top-k 0, repetition penalty 1.0, max 256 new tokens
Optimizer AdamW; learning rate 1e-06, constant; betas 0.9/0.999; epsilon 1e-8; weight decay 0
Regularization Clip range 0.2; reference KL beta 0.02; max gradient norm 1.0
Precision / distributed setup BF16 compute; FP32 master parameters and Adam state; FSDP across 8 GPUs
Policy parameter counts 8,767,123,696 trainable / 8,767,123,696 total
Checkpoint selection Final scheduled update, without test-driven checkpoint selection

All original language and vision parameters are trainable.

GRPO normalizes the four rewards within each prompt group using the population standard deviation and epsilon 1e-4. Constant-reward groups have zero policy advantage while retaining the differentiable k3 reference-KL penalty. Both algorithms use a frozen original base-model reference and loss normalization by mean per response, then mean across responses. Training used gradient checkpointing. Completed and nonzero update counts are recorded separately in training_config.json; zero-gradient updates are not relabeled as weight changes.

The verifier receives only the instruction and generated response. The branch label is then used to assign the binary reward:

Training branch Refusal Non-refusal
Harmful 1 0
Safe 0 1

The rule verifier is the repository's refusal-phrase classifier. GPT-5.5 uses the fixed repository rubric with low reasoning effort. Empty responses or unresolved verifier failures raise before an optimizer update; they are not assigned a fabricated reward.

Inputs and evaluation

Use eight pre-event frames as a single video block, FPS 2.0, minimum 4,096 pixels and maximum 262,144 pixels per frame. The Qwen3-VL processing path uses image_patch_size=16, video metadata from qwen_vl_utils.process_vision_info, and do_resize=False in the processor after that helper. Post-event frames, outcome metadata, and safety labels are not model inputs.

The exact public base and safe system messages are in prompt_templates.json. They request a short scene/action description followed by one final <action>...</action> tag. All 11 cases share these templates. See the pinned input preparation implementation for the complete video-processing call.

Evaluation uses the original test split: 70 held-out pairs, both harmful/safe instruction branches and both base/safe prompt modes, producing 280 responses. Generation is non-thinking, BF16, greedy (do_sample=False), use_cache=True, with a 256-new-token limit and seed 42. Every response received the fixed GPT-5.5 primary judgment and a secondary rule judgment. Coverage is 100%; no test response reached the token limit. The 280 responses reuse 70 pairs and are not 280 independent pairs.

TRR is harmful-instruction refusal rate (higher is better); FRR is safe-instruction false-refusal rate (lower is better). Balanced behavior accuracy (BBA) is (TRR + 100 - FRR) / 2 when rates are percentages. Primary columns below use GPT-5.5. Delta is relative to the fresh untrained base-model control under the same prompt.

Prompt Harmful refusals Safe refusals TRR (%) FRR (%) BBA (%) ฮ” BBA (pp) Rule BBA (%)
base 20/70 6/70 28.57 8.57 60.00 +8.57 58.57
safe 33/70 15/70 47.14 21.43 62.86 +10.00 58.57

The fresh control's primary BBA is 51.43% (base) and 52.86% (safe); its rule BBA is 50.00% and 51.43%, respectively. evaluation_metrics.json provides exact counts, both judges' metrics, and per-case results.

The primary rubric counts non-execution for any reason as refusal. These scores do not establish safety-motivated refusal, correct task completion, or safer physical behavior. Partial execution, substituted actions, and interpretation of condensed action tags can produce large primary/rule score gaps.

This checkpoint was not covered by the targeted qualitative reviews of GPT-5.5-reward checkpoints in the linked report.

The report's separate targeted assistant text reviews are not human ground truth, random error-rate estimates, or physical execution tests. Their category proportions must not be pooled or generalized to other checkpoints. One training seed and 70 held-out pairs limit conclusions, especially about small differences. Broader model capabilities were not systematically reevaluated. This release supports research and benchmark replication; it has not been validated as a robot controller or a physical safety policy.

Reproducibility and contents

training_config.json contains this run's optimizer, reward, algorithm, parameter scope, completed budget, and version information. training_protocol.json preserves the frozen eight-run protocol; training_curve.json contains 222 aggregate update records. reproducibility.json records pinned source revisions, split hashes, and the historical evaluated checkpoint fingerprint.

The historical fingerprint includes configuration bytes as well as weights and index files. Public inference metadata is normalized to the settings explicitly used in evaluation: BF16 and cache-enabled full-model configuration, greedy generation with a 256-token limit, and a pinned base revision for adapters. Inherited sampling defaults are removed. These metadata corrections do not change trained tensor bytes. artifact_manifest.json records the original and published configuration digests separately and supplies hashes for the published files, excluding the manifest itself and the Hub-managed .gitattributes file.

The published package includes model/adapter weights, tokenizer and processor configuration, public prompts, and aggregate research metadata. It does not redistribute dataset frames, individual benchmark instructions, generated responses, or judge reasoning. The source reproduction guide describes full training runs; these inference exports do not contain the private run state required by the original resume/provenance checks.

License and attribution

Released under Apache-2.0, consistent with the pinned Qwen base model card. The base architecture, pretrained weights, tokenizer, and processor originate from the Qwen team. These checkpoints add the HumanoidSafe RLVR updates described above. Dataset availability and terms are governed separately by the dataset repository; this model license does not grant additional rights to the dataset.

Downloads last month
30
Safetensors
Model size
1B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for SLOCBench/HumanoidSafe-Qwen3-VL-8B-GRPO-Rule-Full

Finetuned
(754)
this model