Image-Text-to-Text
Transformers
Safetensors
English
gamescaling
game-agent
embodied-agent
keyboard-mouse
continued-pretraining
qwen3_5
Instructions to use THUSI-Lab/GameScaling with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use THUSI-Lab/GameScaling with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="THUSI-Lab/GameScaling")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("THUSI-Lab/GameScaling", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use THUSI-Lab/GameScaling with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "THUSI-Lab/GameScaling" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "THUSI-Lab/GameScaling", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/THUSI-Lab/GameScaling
- SGLang
How to use THUSI-Lab/GameScaling with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "THUSI-Lab/GameScaling" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "THUSI-Lab/GameScaling", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "THUSI-Lab/GameScaling" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "THUSI-Lab/GameScaling", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use THUSI-Lab/GameScaling with Docker Model Runner:
docker model run hf.co/THUSI-Lab/GameScaling
Add a Doom test frame with the model's reply and an end-to-end example script
Browse files- .gitattributes +1 -0
- README.md +36 -1
- examples/doom_battle1.jpg +3 -0
- examples/doom_battle1_action.txt +5 -0
- examples/run_example.py +74 -0
- tests/test_core.py +9 -0
.gitattributes
CHANGED
|
@@ -43,3 +43,4 @@ models/9b_s1_1600h_gs6736/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
|
| 43 |
models/9b_s3_1600h_gs6760/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 44 |
assets/teaser.jpg filter=lfs diff=lfs merge=lfs -text
|
| 45 |
assets/logos/affiliations.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 43 |
models/9b_s3_1600h_gs6760/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 44 |
assets/teaser.jpg filter=lfs diff=lfs merge=lfs -text
|
| 45 |
assets/logos/affiliations.png filter=lfs diff=lfs merge=lfs -text
|
| 46 |
+
examples/doom_battle1.jpg filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -106,6 +106,10 @@ THUSI-Lab/GameScaling
|
|
| 106 |
├── prompts/
|
| 107 |
│ ├── system_prompt_template.txt standard prompt, 3 slots (§2)
|
| 108 |
│ └── example_doom_battle1.txt complete Doom prompt (§2)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 109 |
├── gamescaling/ inference client
|
| 110 |
│ ├── prompt.py system prompt builders
|
| 111 |
│ ├── policy.py client and request body
|
|
@@ -126,7 +130,7 @@ Download only the checkpoint you need:
|
|
| 126 |
pip install -U huggingface_hub
|
| 127 |
huggingface-cli download THUSI-Lab/GameScaling \
|
| 128 |
--include "models/9b_s1_1600h_gs6736/*" "gamescaling/*" "prompts/*" "scripts/*" \
|
| 129 |
-
"tests/*" "requirements.txt" \
|
| 130 |
--local-dir ./GameScaling
|
| 131 |
cd GameScaling
|
| 132 |
```
|
|
@@ -294,6 +298,37 @@ is `X/1000 * screen width` and `Y/1000 * screen height`. You can spread it over
|
|
| 294 |
send it at once. The frame is sent at its native size; set `PolicyConfig(image_size=(w, h))` to
|
| 295 |
resize it first.
|
| 296 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 297 |
## 4. Reasoning modes and message structure
|
| 298 |
|
| 299 |
The assistant turn is always prefilled, and the request continues it
|
|
|
|
| 106 |
├── prompts/
|
| 107 |
│ ├── system_prompt_template.txt standard prompt, 3 slots (§2)
|
| 108 |
│ └── example_doom_battle1.txt complete Doom prompt (§2)
|
| 109 |
+
├── examples/
|
| 110 |
+
│ ├── doom_battle1.jpg test frame (1280x720)
|
| 111 |
+
│ ├── doom_battle1_action.txt the model's reply on that frame
|
| 112 |
+
│ └── run_example.py frame in, action out (§3)
|
| 113 |
├── gamescaling/ inference client
|
| 114 |
│ ├── prompt.py system prompt builders
|
| 115 |
│ ├── policy.py client and request body
|
|
|
|
| 130 |
pip install -U huggingface_hub
|
| 131 |
huggingface-cli download THUSI-Lab/GameScaling \
|
| 132 |
--include "models/9b_s1_1600h_gs6736/*" "gamescaling/*" "prompts/*" "scripts/*" \
|
| 133 |
+
"examples/*" "tests/*" "requirements.txt" \
|
| 134 |
--local-dir ./GameScaling
|
| 135 |
cd GameScaling
|
| 136 |
```
|
|
|
|
| 298 |
send it at once. The frame is sent at its native size; set `PolicyConfig(image_size=(w, h))` to
|
| 299 |
resize it first.
|
| 300 |
|
| 301 |
+
### Test frame
|
| 302 |
+
|
| 303 |
+
`examples/doom_battle1.jpg` is a frame from a Doom Battle-1 episode played by GameScaling-9B-S3.
|
| 304 |
+
A monster stands just left of the crosshair.
|
| 305 |
+
|
| 306 |
+
<p align="center">
|
| 307 |
+
<img src="https://huggingface.co/THUSI-Lab/GameScaling/resolve/main/examples/doom_battle1.jpg" alt="Doom Battle-1 test frame" width="100%">
|
| 308 |
+
</p>
|
| 309 |
+
|
| 310 |
+
The model's reply on this frame (`examples/doom_battle1_action.txt`), with the system prompt from
|
| 311 |
+
§2:
|
| 312 |
+
|
| 313 |
+
```text
|
| 314 |
+
<think>\n\n</think>\n\n<|action_start|>-37 -3 0 ; LB S ; LB ; LB ; LB ; LB ; LB<|action_end|>
|
| 315 |
+
```
|
| 316 |
+
|
| 317 |
+
It acts without thinking, turns left by 37/1000 of the screen width, and holds fire for the
|
| 318 |
+
whole 200 ms (with a short step back in the first 33 ms). The shot killed the monster.
|
| 319 |
+
|
| 320 |
+
Run the same frame against your server:
|
| 321 |
+
|
| 322 |
+
```bash
|
| 323 |
+
python examples/run_example.py --offline # parse the reference reply, no server
|
| 324 |
+
python examples/run_example.py # default: prefill <think>
|
| 325 |
+
python examples/run_example.py --no-reasoning # prefill <think>\n\n</think>
|
| 326 |
+
```
|
| 327 |
+
|
| 328 |
+
Each reply should parse into a valid action chunk (`parsed=True`); the script exits non-zero
|
| 329 |
+
otherwise. Sampling is at temperature 1.0 and the reference reply had five earlier frames in
|
| 330 |
+
context, so live replies differ in detail.
|
| 331 |
+
|
| 332 |
## 4. Reasoning modes and message structure
|
| 333 |
|
| 334 |
The assistant turn is always prefilled, and the request continues it
|
examples/doom_battle1.jpg
ADDED
|
Git LFS Details
|
examples/doom_battle1_action.txt
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<think>
|
| 2 |
+
|
| 3 |
+
</think>
|
| 4 |
+
|
| 5 |
+
<|action_start|>-37 -3 0 ; LB S ; LB ; LB ; LB ; LB ; LB<|action_end|>
|
examples/run_example.py
ADDED
|
@@ -0,0 +1,74 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""One Doom frame in, one action out.
|
| 2 |
+
|
| 3 |
+
`examples/doom_battle1.jpg` is a frame from a Doom Battle-1 episode played by
|
| 4 |
+
GameScaling-9B-S3, and `examples/doom_battle1_action.txt` is the reply the model
|
| 5 |
+
gave on that frame (a monster just left of the crosshair: turn left and fire).
|
| 6 |
+
|
| 7 |
+
python examples/run_example.py --offline # parse the reference reply only
|
| 8 |
+
bash scripts/serve_vllm.sh models/9b_s3_1600h_gs6760 8000
|
| 9 |
+
python examples/run_example.py # prefill <think>
|
| 10 |
+
python examples/run_example.py --no-reasoning # prefill <think>\\n\\n</think>
|
| 11 |
+
|
| 12 |
+
Sampling is at temperature 1.0 and the reference came with five earlier frames
|
| 13 |
+
in context, so a live reply will differ in detail. The check is that every
|
| 14 |
+
reply parses into a valid action chunk; the script also reports whether it
|
| 15 |
+
fires (LB), as the reference does.
|
| 16 |
+
"""
|
| 17 |
+
|
| 18 |
+
import argparse
|
| 19 |
+
import os
|
| 20 |
+
import sys
|
| 21 |
+
|
| 22 |
+
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
| 23 |
+
sys.path.insert(0, ROOT)
|
| 24 |
+
|
| 25 |
+
from PIL import Image # noqa: E402
|
| 26 |
+
|
| 27 |
+
from gamescaling import GameScalingPolicy, PolicyConfig, parse_action # noqa: E402
|
| 28 |
+
|
| 29 |
+
EXAMPLES = os.path.join(ROOT, "examples")
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
def describe(action) -> str:
|
| 33 |
+
fires = any("LB" in g for g in action.steps)
|
| 34 |
+
return f"parsed={action.parsed} mouse={action.mouse} steps={action.steps} fires={fires}"
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def main() -> int:
|
| 38 |
+
ap = argparse.ArgumentParser()
|
| 39 |
+
ap.add_argument("--base-url", default="http://127.0.0.1:8000/v1")
|
| 40 |
+
ap.add_argument("--image", default=os.path.join(EXAMPLES, "doom_battle1.jpg"))
|
| 41 |
+
ap.add_argument("--prompt", default=os.path.join(ROOT, "prompts", "example_doom_battle1.txt"))
|
| 42 |
+
ap.add_argument("--no-reasoning", action="store_true")
|
| 43 |
+
ap.add_argument("--trials", type=int, default=3)
|
| 44 |
+
ap.add_argument("--offline", action="store_true", help="only parse the reference reply")
|
| 45 |
+
args = ap.parse_args()
|
| 46 |
+
|
| 47 |
+
reference = open(os.path.join(EXAMPLES, "doom_battle1_action.txt")).read()
|
| 48 |
+
print("reference reply:", repr(reference))
|
| 49 |
+
print("reference action:", describe(parse_action(reference)))
|
| 50 |
+
if args.offline:
|
| 51 |
+
return 0
|
| 52 |
+
|
| 53 |
+
system = open(args.prompt).read().rstrip("\n")
|
| 54 |
+
policy = GameScalingPolicy(system, PolicyConfig(base_url=args.base_url,
|
| 55 |
+
reasoning=not args.no_reasoning))
|
| 56 |
+
if not policy.health_check():
|
| 57 |
+
print(f"server not reachable at {args.base_url}")
|
| 58 |
+
return 2
|
| 59 |
+
|
| 60 |
+
frame = Image.open(args.image)
|
| 61 |
+
ok = 0
|
| 62 |
+
for i in range(args.trials):
|
| 63 |
+
policy.reset()
|
| 64 |
+
d = policy.act(frame)
|
| 65 |
+
print(f"--- trial {i + 1} ({d.latency_ms:.0f} ms, {d.completion_tokens} tokens)")
|
| 66 |
+
print(repr(d.text))
|
| 67 |
+
print(describe(d.action))
|
| 68 |
+
ok += d.action.parsed and not d.action.truncated
|
| 69 |
+
print(f"{ok}/{args.trials} replies parsed")
|
| 70 |
+
return 0 if ok == args.trials else 1
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
if __name__ == "__main__":
|
| 74 |
+
sys.exit(main())
|
tests/test_core.py
CHANGED
|
@@ -115,6 +115,15 @@ def test_doom_example_matches_builder():
|
|
| 115 |
assert example == build_system_prompt("Doom Battle-1", rules=rules, key_bindings=bindings)
|
| 116 |
|
| 117 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 118 |
def test_translate_controls():
|
| 119 |
text = "- ArrowLeft: move left\n- a/d: strafe\n- 1-9: pick item\nPress Escape to pause."
|
| 120 |
out = translate_controls(text)
|
|
|
|
| 115 |
assert example == build_system_prompt("Doom Battle-1", rules=rules, key_bindings=bindings)
|
| 116 |
|
| 117 |
|
| 118 |
+
def test_example_pair():
|
| 119 |
+
root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
| 120 |
+
assert Image.open(os.path.join(root, "examples", "doom_battle1.jpg")).size == (1280, 720)
|
| 121 |
+
with open(os.path.join(root, "examples", "doom_battle1_action.txt")) as f:
|
| 122 |
+
a = parse_action(f.read())
|
| 123 |
+
assert a.parsed and a.mouse == (-37.0, -3.0, 0.0)
|
| 124 |
+
assert all("LB" in g for g in a.steps)
|
| 125 |
+
|
| 126 |
+
|
| 127 |
def test_translate_controls():
|
| 128 |
text = "- ArrowLeft: move left\n- a/d: strafe\n- 1-9: pick item\nPress Escape to pause."
|
| 129 |
out = translate_controls(text)
|