kobkrit commited on
Commit
8367d31
·
verified ·
1 Parent(s): 8937819

OpenThai-SystemOne-Vision v1.0

Browse files
.gitattributes CHANGED
@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ assets/fb_launch_v3_1x1.png filter=lfs diff=lfs merge=lfs -text
37
+ assets/openthai-systemone-launch.jpg filter=lfs diff=lfs merge=lfs -text
38
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,102 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language: [th, en]
4
+ pipeline_tag: image-text-to-text
5
+ tags: [system-one, decision-model, gui-grounding, computer-use, thai, qwen3.5, vision]
6
+ base_model: Qwen/Qwen3.5-0.8B-Base
7
+ ---
8
+
9
+ # OpenThai-SystemOne-Vision (v1.0)
10
+
11
+ The vision variant of [OpenThai-SystemOne](https://huggingface.co/iapp/OpenThai-SystemOne): the same open Thai + English
12
+ System One decision model, now with **images in the state** and a fourth question type, **`point`**, that returns where
13
+ on a screenshot something is. Screenshot in, click out, in one forward pass, no text generation.
14
+
15
+ | | OpenThai-SystemOne (text) | OpenThai-SystemOne-Vision |
16
+ |---|---|---|
17
+ | parameters | 0.75B | 0.86B (ViT 100M + point head 9M) |
18
+ | question types | choice / score / noul | choice / score / noul / **point** |
19
+ | state | text or JSON | text or JSON + up to 4 images |
20
+ | typical use | routing, moderation, RAG relevance, tool choice, agents with element tables | GUI agents from pixels, Thai screen reading, game frames, documents |
21
+ | text-only behaviour | v0.3 | identical to v0.3 (same tower and head; the vision path is skipped when no image is sent) |
22
+
23
+ ## API (superset of the TypeSafe `/v1/systemone` contract)
24
+
25
+ ```json
26
+ {
27
+ "state": "เป้าหมาย: ค้นหาเที่ยวบิน BKK→CNX",
28
+ "images": [{"id": "screen", "data": "data:image/png;base64,...", "role": "screenshot"}],
29
+ "questions": {
30
+ "where": {"type": "point", "instructions": "ปุ่มค้นหาเที่ยวบิน", "image": "screen"},
31
+ "op": {"type": "choice", "instructions": "next operation", "criteria": {"CLICK": null, "TYPE_TEXT": null, "SCROLL_DOWN": null, "DONE": null}},
32
+ "loaded": {"type": "noul", "instructions": "is the page fully loaded?"}
33
+ }
34
+ }
35
+ ```
36
+
37
+ `point` answers: `{"x": 0.63, "y": 0.41, "confidence": 0.86, "abstain": 0.02, "candidates": [{"x", "y", "p", "bbox"}, ...]}`
38
+ (normalised image coordinates; `abstain` = probability that the target is not on the image; candidates = the
39
+ above-threshold token clusters of the attention heat-map, GUI-Actor style).
40
+
41
+ Limits: ≤ 4 images, ≤ 8k visual tokens per request (a 1280×800 screenshot ≈ 1,000 tokens at 32 px/token; a 320×240 game
42
+ frame ≈ 80); `"detail": "high"` allows up to 1920×1200.
43
+
44
+ ## Usage
45
+
46
+ ```python
47
+ from openthai_systemone import SystemOneClient
48
+ c = SystemOneClient("iapp/OpenThai-SystemOne-Vision")
49
+ r = c.system_one("goal: open the login form", {"where": {"type": "point", "instructions": "ปุ่มเข้าสู่ระบบ", "image": "s"}},
50
+ images=[{"id": "s", "path": "screen.png", "role": "screenshot"}])
51
+ print(r.answers["where"].x, r.answers["where"].y, r.answers["where"].confidence)
52
+ ```
53
+
54
+ ## Evaluation
55
+
56
+ Numbers are for the released checkpoint (calibrated round 3); single option order (no permutation averaging).
57
+
58
+ ### GUI grounding (`point`: predicted click inside the target box)
59
+ | benchmark | acc | notes |
60
+ |---|---|---|
61
+ | ScreenSpot-v2 (1,272) | **82.7** | desktop 90.2 text / 71.4 icon · mobile 95.9 / 71.1 · web 87.6 / 70.9 · GUI-Actor-2B 88.6 |
62
+ | Thai GUI held-out domains, `point` (1,611) | **80.1** | teacher-written Thai/English instructions on unseen Thai sites |
63
+ | Thai GUI set-of-marks `choice` (476) | 92.9 | pick the element index from the element table |
64
+ | Thai GUI `operation` choice (476) | 99.8 | CLICK / TYPE_TEXT / SELECT |
65
+ | Thai screen QA `noul` / `choice` (623 / 330) | 94.9 / 85.0 | is a dialog open, which page type, is there a form… |
66
+ | "not on screen" abstention (179) | 45.8 | known weak spot: check `abstain` before clicking |
67
+
68
+ ### Doom from pixels (`demo/doom_demo.py --mode pixels`, defend_the_center, skill 3, 20 episodes)
69
+ | input | kills / episode | ms / decision (H100, idle) |
70
+ |---|---|---|
71
+ | two 320×240 frames only | 5.5 | 57 |
72
+ | frames + symbolic state | 6.0 | 51 |
73
+ | symbolic state only (text model behaviour) | 7.3 | 41 |
74
+ | the privileged scripted expert it imitates | 6.8 | – |
75
+
76
+ ### Text-only tasks (must not regress vs the text model v0.3)
77
+ | suite | v0.3 | this model |
78
+ |---|---|---|
79
+ | public 13-subset bench, 10 accuracy subsets, macro | 82.7 | 82.8 |
80
+ | Thai held-out sets (8), macro | 77.5 | 79.1 |
81
+
82
+ ### Latency (batch 1, H100, end-to-end `SystemOneClient`)
83
+ | request | tokens | ms |
84
+ |---|---|---|
85
+ | two 320×240 game frames + short state, 2 questions | 298 (160 visual) | 52 |
86
+ | 1280×800 screenshot + goal, `point` + `choice` | 1,062 (1,000 visual) | 55 |
87
+ | text only (3-question Thai ticket) | 166 | ~40 |
88
+
89
+ ## How it was built
90
+
91
+ 1. Text tower + slot head from OpenThai-SystemOne v0.3 (Qwen3.5-0.8B, Thai CPT 4.5B tokens, decision SFT + calibration).
92
+ 2. Vision tower + merger from Qwen3.5-0.8B-Base, re-aligned with an LM loss on 200k+ image→text pairs (Thai rendered
93
+ OCR pages in 55 fonts, Thai web screenshots → element tables, Doom frames → object tables, public screen/document QA).
94
+ 3. Decision SFT on the v0.3 text mix plus Thai GUI grounding (Playwright crawl of 300+ Thai sites, teacher-written and
95
+ blind-verified Thai/English instructions), public English grounding (Wave-UI, GroundUI), Doom rollouts (privileged
96
+ expert, 1.5M frames) with `move` / `attack` / `in_danger` / `nearest_enemy` questions, and screen QA.
97
+ 4. Calibration of the four per-type temperatures.
98
+
99
+ ## License & credits
100
+
101
+ Apache-2.0. Built by iApp Technology / OpenThaiGPT on Qwen3.5-0.8B-Base. GUI grounding recipe after GUI-Actor; System One
102
+ contract after TypeSafe AI's Jev.
assets/fb_launch_v3_1x1.png ADDED

Git LFS Details

  • SHA256: 5f25d5d8fc0f44b9ded71dc58dc8ab94d46eae3bfae1c9e254c547519127df15
  • Pointer size: 132 Bytes
  • Size of remote file: 4.48 MB
assets/openthai-systemone-launch.jpg ADDED

Git LFS Details

  • SHA256: b00d5f9e7e4f3b7588abfe03a684fc1ae3b529f29905d60c4bbb20f36c844d3f
  • Pointer size: 131 Bytes
  • Size of remote file: 141 kB
assets/siamai-logo.png ADDED
chat_template.jinja ADDED
@@ -0,0 +1,154 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- set image_count = namespace(value=0) %}
2
+ {%- set video_count = namespace(value=0) %}
3
+ {%- macro render_content(content, do_vision_count, is_system_content=false) %}
4
+ {%- if content is string %}
5
+ {{- content }}
6
+ {%- elif content is iterable and content is not mapping %}
7
+ {%- for item in content %}
8
+ {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
9
+ {%- if is_system_content %}
10
+ {{- raise_exception('System message cannot contain images.') }}
11
+ {%- endif %}
12
+ {%- if do_vision_count %}
13
+ {%- set image_count.value = image_count.value + 1 %}
14
+ {%- endif %}
15
+ {%- if add_vision_id %}
16
+ {{- 'Picture ' ~ image_count.value ~ ': ' }}
17
+ {%- endif %}
18
+ {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
19
+ {%- elif 'video' in item or item.type == 'video' %}
20
+ {%- if is_system_content %}
21
+ {{- raise_exception('System message cannot contain videos.') }}
22
+ {%- endif %}
23
+ {%- if do_vision_count %}
24
+ {%- set video_count.value = video_count.value + 1 %}
25
+ {%- endif %}
26
+ {%- if add_vision_id %}
27
+ {{- 'Video ' ~ video_count.value ~ ': ' }}
28
+ {%- endif %}
29
+ {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
30
+ {%- elif 'text' in item %}
31
+ {{- item.text }}
32
+ {%- else %}
33
+ {{- raise_exception('Unexpected item type in content.') }}
34
+ {%- endif %}
35
+ {%- endfor %}
36
+ {%- elif content is none or content is undefined %}
37
+ {{- '' }}
38
+ {%- else %}
39
+ {{- raise_exception('Unexpected content type.') }}
40
+ {%- endif %}
41
+ {%- endmacro %}
42
+ {%- if not messages %}
43
+ {{- raise_exception('No messages provided.') }}
44
+ {%- endif %}
45
+ {%- if tools and tools is iterable and tools is not mapping %}
46
+ {{- '<|im_start|>system\n' }}
47
+ {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
48
+ {%- for tool in tools %}
49
+ {{- "\n" }}
50
+ {{- tool | tojson }}
51
+ {%- endfor %}
52
+ {{- "\n</tools>" }}
53
+ {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
54
+ {%- if messages[0].role == 'system' %}
55
+ {%- set content = render_content(messages[0].content, false, true)|trim %}
56
+ {%- if content %}
57
+ {{- '\n\n' + content }}
58
+ {%- endif %}
59
+ {%- endif %}
60
+ {{- '<|im_end|>\n' }}
61
+ {%- else %}
62
+ {%- if messages[0].role == 'system' %}
63
+ {%- set content = render_content(messages[0].content, false, true)|trim %}
64
+ {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
65
+ {%- endif %}
66
+ {%- endif %}
67
+ {%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
68
+ {%- for message in messages[::-1] %}
69
+ {%- set index = (messages|length - 1) - loop.index0 %}
70
+ {%- if ns.multi_step_tool and message.role == "user" %}
71
+ {%- set content = render_content(message.content, false)|trim %}
72
+ {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
73
+ {%- set ns.multi_step_tool = false %}
74
+ {%- set ns.last_query_index = index %}
75
+ {%- endif %}
76
+ {%- endif %}
77
+ {%- endfor %}
78
+ {%- if ns.multi_step_tool %}
79
+ {{- raise_exception('No user query found in messages.') }}
80
+ {%- endif %}
81
+ {%- for message in messages %}
82
+ {%- set content = render_content(message.content, true)|trim %}
83
+ {%- if message.role == "system" %}
84
+ {%- if not loop.first %}
85
+ {{- raise_exception('System message must be at the beginning.') }}
86
+ {%- endif %}
87
+ {%- elif message.role == "user" %}
88
+ {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
89
+ {%- elif message.role == "assistant" %}
90
+ {%- set reasoning_content = '' %}
91
+ {%- if message.reasoning_content is string %}
92
+ {%- set reasoning_content = message.reasoning_content %}
93
+ {%- else %}
94
+ {%- if '</think>' in content %}
95
+ {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
96
+ {%- set content = content.split('</think>')[-1].lstrip('\n') %}
97
+ {%- endif %}
98
+ {%- endif %}
99
+ {%- set reasoning_content = reasoning_content|trim %}
100
+ {%- if loop.index0 > ns.last_query_index %}
101
+ {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
102
+ {%- else %}
103
+ {{- '<|im_start|>' + message.role + '\n' + content }}
104
+ {%- endif %}
105
+ {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
106
+ {%- for tool_call in message.tool_calls %}
107
+ {%- if tool_call.function is defined %}
108
+ {%- set tool_call = tool_call.function %}
109
+ {%- endif %}
110
+ {%- if loop.first %}
111
+ {%- if content|trim %}
112
+ {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
113
+ {%- else %}
114
+ {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
115
+ {%- endif %}
116
+ {%- else %}
117
+ {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
118
+ {%- endif %}
119
+ {%- if tool_call.arguments is defined %}
120
+ {%- for args_name, args_value in tool_call.arguments|items %}
121
+ {{- '<parameter=' + args_name + '>\n' }}
122
+ {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
123
+ {{- args_value }}
124
+ {{- '\n</parameter>\n' }}
125
+ {%- endfor %}
126
+ {%- endif %}
127
+ {{- '</function>\n</tool_call>' }}
128
+ {%- endfor %}
129
+ {%- endif %}
130
+ {{- '<|im_end|>\n' }}
131
+ {%- elif message.role == "tool" %}
132
+ {%- if loop.previtem and loop.previtem.role != "tool" %}
133
+ {{- '<|im_start|>user' }}
134
+ {%- endif %}
135
+ {{- '\n<tool_response>\n' }}
136
+ {{- content }}
137
+ {{- '\n</tool_response>' }}
138
+ {%- if not loop.last and loop.nextitem.role != "tool" %}
139
+ {{- '<|im_end|>\n' }}
140
+ {%- elif loop.last %}
141
+ {{- '<|im_end|>\n' }}
142
+ {%- endif %}
143
+ {%- else %}
144
+ {{- raise_exception('Unexpected message role.') }}
145
+ {%- endif %}
146
+ {%- endfor %}
147
+ {%- if add_generation_prompt %}
148
+ {{- '<|im_start|>assistant\n' }}
149
+ {%- if enable_thinking is defined and enable_thinking is true %}
150
+ {{- '<think>\n' }}
151
+ {%- else %}
152
+ {{- '<think>\n\n</think>\n\n' }}
153
+ {%- endif %}
154
+ {%- endif %}
config.json ADDED
@@ -0,0 +1,153 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "abstain_slot": 255,
3
+ "answer_token_id": 248082,
4
+ "architectures": [
5
+ "OpenThaiSystemOneForDecision"
6
+ ],
7
+ "dtype": "bfloat16",
8
+ "head_bias": true,
9
+ "image_token_id": 248056,
10
+ "model_type": "openthai_systemone",
11
+ "n_slots": 256,
12
+ "n_temperatures": 4,
13
+ "pad_token_id": 248044,
14
+ "point_head": {
15
+ "dim": 256,
16
+ "n_heads": 8,
17
+ "n_sa_layers": 1
18
+ },
19
+ "point_token_id": 248339,
20
+ "text_config": {
21
+ "_name_or_path": "runs/cpt/latest",
22
+ "architectures": [
23
+ "Qwen3_5ForCausalLM"
24
+ ],
25
+ "attention_bias": false,
26
+ "attention_dropout": 0.0,
27
+ "attn_output_gate": true,
28
+ "bos_token_id": null,
29
+ "chunk_size_feed_forward": 0,
30
+ "dtype": "bfloat16",
31
+ "eos_token_id": 248044,
32
+ "full_attention_interval": 4,
33
+ "head_dim": 256,
34
+ "hidden_act": "silu",
35
+ "hidden_size": 1024,
36
+ "id2label": {
37
+ "0": "LABEL_0",
38
+ "1": "LABEL_1"
39
+ },
40
+ "initializer_range": 0.02,
41
+ "intermediate_size": 3584,
42
+ "is_encoder_decoder": false,
43
+ "label2id": {
44
+ "LABEL_0": 0,
45
+ "LABEL_1": 1
46
+ },
47
+ "layer_types": [
48
+ "linear_attention",
49
+ "linear_attention",
50
+ "linear_attention",
51
+ "full_attention",
52
+ "linear_attention",
53
+ "linear_attention",
54
+ "linear_attention",
55
+ "full_attention",
56
+ "linear_attention",
57
+ "linear_attention",
58
+ "linear_attention",
59
+ "full_attention",
60
+ "linear_attention",
61
+ "linear_attention",
62
+ "linear_attention",
63
+ "full_attention",
64
+ "linear_attention",
65
+ "linear_attention",
66
+ "linear_attention",
67
+ "full_attention",
68
+ "linear_attention",
69
+ "linear_attention",
70
+ "linear_attention",
71
+ "full_attention"
72
+ ],
73
+ "linear_conv_kernel_dim": 4,
74
+ "linear_key_head_dim": 128,
75
+ "linear_num_key_heads": 16,
76
+ "linear_num_value_heads": 16,
77
+ "linear_value_head_dim": 128,
78
+ "mamba_ssm_dtype": "float32",
79
+ "max_position_embeddings": 262144,
80
+ "mlp_only_layers": [],
81
+ "model_type": "qwen3_5_text",
82
+ "mtp_num_hidden_layers": 1,
83
+ "mtp_use_dedicated_embeddings": false,
84
+ "num_attention_heads": 8,
85
+ "num_hidden_layers": 24,
86
+ "num_key_value_heads": 2,
87
+ "output_attentions": false,
88
+ "output_hidden_states": false,
89
+ "pad_token_id": null,
90
+ "partial_rotary_factor": 0.25,
91
+ "problem_type": null,
92
+ "return_dict": true,
93
+ "rms_norm_eps": 1e-06,
94
+ "rope_parameters": {
95
+ "mrope_interleaved": true,
96
+ "mrope_section": [
97
+ 11,
98
+ 11,
99
+ 10
100
+ ],
101
+ "partial_rotary_factor": 0.25,
102
+ "rope_theta": 10000000,
103
+ "rope_type": "default"
104
+ },
105
+ "tie_word_embeddings": false,
106
+ "use_cache": true,
107
+ "vocab_size": 248341
108
+ },
109
+ "transformers_version": "5.17.0",
110
+ "vision_config": {
111
+ "_name_or_path": "",
112
+ "architectures": null,
113
+ "chunk_size_feed_forward": 0,
114
+ "deepstack_visual_indexes": [],
115
+ "depth": 12,
116
+ "dtype": "bfloat16",
117
+ "hidden_act": "gelu_pytorch_tanh",
118
+ "hidden_size": 768,
119
+ "id2label": {
120
+ "0": "LABEL_0",
121
+ "1": "LABEL_1"
122
+ },
123
+ "in_channels": 3,
124
+ "initializer_range": 0.02,
125
+ "intermediate_size": 3072,
126
+ "is_encoder_decoder": false,
127
+ "label2id": {
128
+ "LABEL_0": 0,
129
+ "LABEL_1": 1
130
+ },
131
+ "model_type": "qwen3_5_vision",
132
+ "num_heads": 12,
133
+ "num_position_embeddings": 2304,
134
+ "out_hidden_size": 1024,
135
+ "output_attentions": false,
136
+ "output_hidden_states": false,
137
+ "patch_size": 16,
138
+ "problem_type": null,
139
+ "return_dict": true,
140
+ "rope_parameters": {
141
+ "rope_theta": 10000.0,
142
+ "rope_type": "axial"
143
+ },
144
+ "spatial_merge_size": 2,
145
+ "temporal_patch_size": 2
146
+ },
147
+ "vision_end_token_id": 248054,
148
+ "vision_start_token_id": 248053,
149
+ "auto_map": {
150
+ "AutoConfig": "configuration.OpenThaiSystemOneConfig",
151
+ "AutoModel": "modeling.OpenThaiSystemOneForDecision"
152
+ }
153
+ }
configuration.py ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ from typing import Any, Dict, Optional
4
+
5
+ from transformers import AutoConfig, PretrainedConfig
6
+
7
+
8
+ class OpenThaiSystemOneConfig(PretrainedConfig):
9
+ """Config = a text-tower config (Qwen3.5 text) + slot-head settings [+ vision tower + point head for the -Vision variant].
10
+
11
+ vision_config is None for the text-only line (v0.x): the model then wraps `Qwen3_5TextModel` and its state-dict
12
+ layout is unchanged. With a vision_config the model wraps the multimodal `Qwen3_5Model` (visual + language_model)
13
+ and gains a PointHead for `point` questions.
14
+ """
15
+
16
+ model_type = "openthai_systemone"
17
+ sub_configs = {"text_config": AutoConfig, "vision_config": AutoConfig}
18
+
19
+ def __init__(
20
+ self,
21
+ text_config: Optional[Dict[str, Any] | PretrainedConfig] = None,
22
+ n_slots: int = 256,
23
+ abstain_slot: int = 255,
24
+ answer_token_id: Optional[int] = None,
25
+ head_bias: bool = True,
26
+ n_temperatures: int = 3, # per question type: choice / score / noul (/ point)
27
+ vision_config: Optional[Dict[str, Any] | PretrainedConfig] = None,
28
+ image_token_id: Optional[int] = None,
29
+ vision_start_token_id: Optional[int] = None,
30
+ vision_end_token_id: Optional[int] = None,
31
+ point_token_id: Optional[int] = None,
32
+ point_head: Optional[Dict[str, Any]] = None, # {"dim": 256, "n_sa_layers": 1, "n_heads": 8}
33
+ **kwargs,
34
+ ):
35
+ if isinstance(text_config, dict):
36
+ text_config = AutoConfig.for_model(**text_config) if "model_type" in text_config else AutoConfig.for_model("qwen3_5_text", **text_config)
37
+ if isinstance(vision_config, dict):
38
+ vc = dict(vision_config)
39
+ vision_config = AutoConfig.for_model(**vc) if "model_type" in vc else AutoConfig.for_model("qwen3_5_vision", **vc)
40
+ self.text_config = text_config
41
+ self.vision_config = vision_config
42
+ self.n_slots = n_slots
43
+ self.abstain_slot = abstain_slot
44
+ self.answer_token_id = answer_token_id
45
+ self.head_bias = head_bias
46
+ self.n_temperatures = n_temperatures
47
+ self.image_token_id = image_token_id
48
+ self.vision_start_token_id = vision_start_token_id
49
+ self.vision_end_token_id = vision_end_token_id
50
+ self.point_token_id = point_token_id
51
+ self.point_head = point_head
52
+ super().__init__(**kwargs)
53
+
54
+ @property
55
+ def is_vision(self) -> bool:
56
+ return self.vision_config is not None
57
+
58
+ @property
59
+ def hidden_size(self) -> int:
60
+ return self.text_config.hidden_size
61
+
62
+ @property
63
+ def vocab_size(self) -> int:
64
+ return self.text_config.vocab_size
formatting.py ADDED
@@ -0,0 +1,543 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Turn (state, questions) into token ids + slot bookkeeping.
2
+
3
+ Layout (one sequence, causal):
4
+
5
+ <|ts_state|> {state text}
6
+ <|ts_q|><|ts_choice|> {instructions}
7
+ <|ts_opt_0|> {option name}: {description}
8
+ <|ts_opt_1|> {option name}
9
+ ...
10
+ <|ts_answer|> <- hidden state here -> SlotHead (256 logits)
11
+ <|ts_q|><|ts_noul|> {instructions}
12
+ <|ts_opt_0|> no
13
+ <|ts_opt_1|> yes
14
+ <|ts_answer|>
15
+ ...
16
+
17
+ Slot i (0..254) means "the option introduced by <|ts_opt_i|>"; slot 255 = abstain.
18
+ All answers for all questions are read out from one forward pass.
19
+
20
+ Vision variant (OpenThai-SystemOne-Vision) adds, before the state (or inline where the state text says <image:id>):
21
+
22
+ <|ts_img|> [image:screen 1280x800 screenshot]
23
+ <|vision_start|><|image_pad|> x N <|vision_end|>
24
+
25
+ and a fourth question type whose readout is the PointHead over that image's N visual tokens:
26
+
27
+ <|ts_q|><|ts_point|> [image:screen] {instructions}
28
+ <|ts_answer|>
29
+ """
30
+ from __future__ import annotations
31
+
32
+ import json
33
+ import random
34
+ from dataclasses import dataclass, field
35
+ from typing import Any, Dict, List, Optional, Sequence, Tuple, Union
36
+
37
+ from .types import Choice, Noul, Point, Score, Question, MAX_OPTIONS
38
+
39
+ N_SLOTS = 256
40
+ ABSTAIN_SLOT = 255
41
+
42
+ TOK_STATE = "<|ts_state|>"
43
+ TOK_Q = "<|ts_q|>"
44
+ TOK_CHOICE = "<|ts_choice|>"
45
+ TOK_SCORE = "<|ts_score|>"
46
+ TOK_NOUL = "<|ts_noul|>"
47
+ TOK_ANSWER = "<|ts_answer|>"
48
+ TOK_OPT = [f"<|ts_opt_{i}|>" for i in range(N_SLOTS)]
49
+ TOK_POINT = "<|ts_point|>" # vision variant only
50
+ TOK_IMG = "<|ts_img|>" # vision variant only
51
+
52
+ SPECIAL_TOKENS: List[str] = [TOK_STATE, TOK_Q, TOK_CHOICE, TOK_SCORE, TOK_NOUL, TOK_ANSWER] + TOK_OPT
53
+ VISION_TOKENS: List[str] = [TOK_POINT, TOK_IMG] # added on top of SPECIAL_TOKENS for the -Vision variant
54
+ # Qwen's own multimodal tokens (already in the base tokenizer)
55
+ QWEN_VISION_START, QWEN_VISION_END, QWEN_IMAGE_PAD = "<|vision_start|>", "<|vision_end|>", "<|image_pad|>"
56
+ QTYPES = ("choice", "score", "noul", "point")
57
+
58
+ NOUL_OPTIONS = ("no", "yes") # slot 0 = no, slot 1 = yes -> noul = p(slot 1)
59
+
60
+ DEFAULT_MAX_TOTAL_TOKENS = 65536
61
+ DEFAULT_MAX_STATE_TOKENS = 32768
62
+
63
+
64
+ def add_special_tokens(tokenizer, *, vision: bool = False) -> int:
65
+ """Register the control tokens (plus the vision ones when vision=True). Returns number of tokens added."""
66
+ existing = set(tokenizer.get_vocab())
67
+ wanted = SPECIAL_TOKENS + (VISION_TOKENS if vision else [])
68
+ new = [t for t in wanted if t not in existing]
69
+ if not new:
70
+ return 0
71
+ return tokenizer.add_tokens(new, special_tokens=True)
72
+
73
+
74
+ def sanitize(text: str) -> str:
75
+ """Stop user content from smuggling control tokens into the sequence."""
76
+ return text.replace("<|ts_", "<​|ts_") if "<|ts_" in text else text
77
+
78
+
79
+ def state_to_text(state: Union[str, Dict[str, Any], List[Any]], *, indent: Optional[int] = None) -> str:
80
+ if isinstance(state, str):
81
+ return state
82
+ return json.dumps(state, ensure_ascii=False, indent=indent)
83
+
84
+
85
+ @dataclass
86
+ class QuestionSpec:
87
+ """A question flattened to option strings + slot bookkeeping."""
88
+
89
+ qid: str
90
+ qtype: str # choice | score | noul
91
+ instructions: str
92
+ option_names: List[str] # in slot order (after any permutation)
93
+ option_descs: List[Optional[str]]
94
+ perm: List[int] # perm[slot] = original index of the option at that slot
95
+ label_slot: Optional[int] = None # training only
96
+ image: Optional[str] = None # point questions: id of the referenced image (None = first image)
97
+ point_bbox: Optional[List[float]] = None # training only: normalised [x0,y0,x1,y1]; None with point_label_given -> null
98
+ point_label_given: bool = False
99
+
100
+
101
+ def question_to_spec(
102
+ qid: str,
103
+ q: Question,
104
+ *,
105
+ label: Optional[Union[str, int, bool]] = None,
106
+ shuffle: bool = False,
107
+ rng: Optional[random.Random] = None,
108
+ drop_label: bool = False,
109
+ perm: Optional[Sequence[int]] = None,
110
+ ) -> QuestionSpec:
111
+ """Flatten a typed question.
112
+
113
+ label: for training. Choice -> option name; Score -> level index (int); Noul -> bool.
114
+ shuffle: permute option order (Choice only; Score/Noul order is semantic).
115
+ drop_label: remove the correct option from a Choice so the target becomes ABSTAIN_SLOT.
116
+ perm: explicit option order for a Choice (list of original indices), e.g. a cyclic shift for order-invariant inference.
117
+ """
118
+ if isinstance(q, Choice):
119
+ names = list(q.criteria.keys())
120
+ descs = [q.criteria[n] for n in names]
121
+ idx = list(range(len(names)))
122
+ label_idx = None
123
+ if label is not None:
124
+ if label not in q.criteria:
125
+ raise ValueError(f"label {label!r} is not one of the options")
126
+ label_idx = names.index(str(label))
127
+ if drop_label and label_idx is not None:
128
+ if len(idx) < 2:
129
+ raise ValueError("cannot drop the only option")
130
+ idx.remove(label_idx)
131
+ label_idx = None
132
+ if perm is not None:
133
+ idx = [i for i in perm if i in idx]
134
+ elif shuffle:
135
+ (rng or random).shuffle(idx)
136
+ names_p = [names[i] for i in idx]
137
+ descs_p = [descs[i] for i in idx]
138
+ if label is None:
139
+ slot = None
140
+ elif label_idx is None:
141
+ slot = ABSTAIN_SLOT
142
+ else:
143
+ slot = idx.index(label_idx)
144
+ return QuestionSpec(qid, "choice", q.instructions, names_p, descs_p, idx, slot)
145
+
146
+ if isinstance(q, Score):
147
+ names = [str(i) for i in range(len(q.criteria))]
148
+ descs = list(q.criteria)
149
+ slot = int(label) if label is not None else None
150
+ if slot is not None and not (0 <= slot < len(descs)):
151
+ raise ValueError("score label out of range")
152
+ return QuestionSpec(qid, "score", q.instructions, names, descs, list(range(len(names))), slot)
153
+
154
+ if isinstance(q, Noul):
155
+ c = q.criteria or {}
156
+ descs = [c.get("false"), c.get("true")]
157
+ slot = None if label is None else int(bool(label))
158
+ return QuestionSpec(qid, "noul", q.instructions, list(NOUL_OPTIONS), descs, [0, 1], slot)
159
+
160
+ if isinstance(q, Point):
161
+ # label: {"bbox": [x0,y0,x1,y1]} | [x0,y0,x1,y1] | {"bbox": None} (= not on screen) | None (inference)
162
+ bbox, given = None, False
163
+ if label is not None:
164
+ given = True
165
+ bbox = label.get("bbox") if isinstance(label, dict) else list(label)
166
+ if bbox is not None:
167
+ bbox = [float(v) for v in bbox]
168
+ if len(bbox) != 4:
169
+ raise ValueError("point label bbox needs 4 numbers")
170
+ return QuestionSpec(qid, "point", q.instructions, [], [], [], None, image=q.image, point_bbox=bbox, point_label_given=given)
171
+
172
+ raise TypeError(type(q))
173
+
174
+
175
+ def spec_to_text(spec: QuestionSpec, *, image_id: Optional[str] = None) -> str:
176
+ if spec.qtype == "point":
177
+ tag = f"[image:{sanitize(str(image_id if image_id is not None else spec.image or ''))}] "
178
+ return f"{TOK_Q}{TOK_POINT} {tag}{sanitize(spec.instructions).strip()}\n{TOK_ANSWER}\n"
179
+ head = {"choice": TOK_CHOICE, "score": TOK_SCORE, "noul": TOK_NOUL}[spec.qtype]
180
+ lines = [f"{TOK_Q}{head} {sanitize(spec.instructions).strip()}"]
181
+ for i, (name, desc) in enumerate(zip(spec.option_names, spec.option_descs)):
182
+ name = sanitize(str(name)).strip()
183
+ if desc:
184
+ lines.append(f"{TOK_OPT[i]} {name}: {sanitize(str(desc)).strip()}")
185
+ else:
186
+ lines.append(f"{TOK_OPT[i]} {name}")
187
+ lines.append(TOK_ANSWER)
188
+ return "\n".join(lines) + "\n"
189
+
190
+
191
+ @dataclass
192
+ class Encoded:
193
+ input_ids: List[int]
194
+ answer_positions: List[int] # index of each <|ts_answer|> token, question order
195
+ option_counts: List[int] # k per question (valid slots 0..k-1)
196
+ specs: List[QuestionSpec]
197
+ labels: List[int] = field(default_factory=list) # -100 if unknown
198
+ truncated_state: bool = False
199
+ # vision variant
200
+ image_ids: List[str] = field(default_factory=list)
201
+ image_grid_thw: List[List[int]] = field(default_factory=list) # per image (t, h, w) in patches
202
+ image_spans: List[Tuple[int, int]] = field(default_factory=list) # [start, end) positions of each image's tokens
203
+ image_sizes: List[Tuple[int, int]] = field(default_factory=list) # original (W, H)
204
+ pixel_values: Any = None # torch.Tensor (sum patches, C*T*P*P) or None
205
+ merge_size: int = 2
206
+ point_image_index: List[int] = field(default_factory=list) # per question: image index or -1
207
+ point_targets: List[Any] = field(default_factory=list) # per question: tensor (n_tokens+1) or None
208
+
209
+ @property
210
+ def n_tokens(self) -> int:
211
+ return len(self.input_ids)
212
+
213
+ @property
214
+ def n_visual_tokens(self) -> int:
215
+ return sum(e - s for s, e in self.image_spans)
216
+
217
+ def merged_grid(self, i: int) -> Tuple[int, int]:
218
+ t, h, w = self.image_grid_thw[i]
219
+ return h // self.merge_size, w // self.merge_size
220
+
221
+ def n_tokens_of_image(self, i: int) -> int:
222
+ s, e = self.image_spans[i]
223
+ return e - s
224
+
225
+
226
+ class Formatter:
227
+ """Tokenizer-aware encoder shared by training and inference."""
228
+
229
+ def __init__(
230
+ self,
231
+ tokenizer,
232
+ *,
233
+ max_total_tokens: int = DEFAULT_MAX_TOTAL_TOKENS,
234
+ max_state_tokens: int = DEFAULT_MAX_STATE_TOKENS,
235
+ image_processor=None,
236
+ max_pixels: Optional[int] = None,
237
+ ):
238
+ self.tok = tokenizer
239
+ self.image_processor = image_processor
240
+ self.vision = image_processor is not None
241
+ add_special_tokens(self.tok, vision=self.vision)
242
+ self.max_total_tokens = max_total_tokens
243
+ self.max_state_tokens = max_state_tokens
244
+ self.max_pixels = max_pixels
245
+ self.image_device = None # set by the client: patchify maths runs on the model device
246
+ self.answer_id = self.tok.convert_tokens_to_ids(TOK_ANSWER)
247
+ self.state_id = self.tok.convert_tokens_to_ids(TOK_STATE)
248
+ self.opt_ids = self.tok.convert_tokens_to_ids(TOK_OPT)
249
+ assert self.answer_id is not None and self.answer_id != self.tok.unk_token_id
250
+ if self.vision:
251
+ self.image_pad_id = self.tok.convert_tokens_to_ids(QWEN_IMAGE_PAD)
252
+ self.vision_start_id = self.tok.convert_tokens_to_ids(QWEN_VISION_START)
253
+ self.vision_end_id = self.tok.convert_tokens_to_ids(QWEN_VISION_END)
254
+ self.point_id = self.tok.convert_tokens_to_ids(TOK_POINT)
255
+ for v in (self.image_pad_id, self.vision_start_id, self.vision_end_id, self.point_id):
256
+ assert v is not None and v != self.tok.unk_token_id, "tokenizer lacks the Qwen vision tokens"
257
+
258
+ def _image_block(self, ref, n_tokens: int, size: Tuple[int, int]) -> List[int]:
259
+ """<|ts_img|> [image:id WxH role]\n<|vision_start|> pad*n <|vision_end|>\n -> ids; the pad run is contiguous."""
260
+ role = f" {sanitize(str(ref.role))}" if getattr(ref, "role", None) else ""
261
+ head = self._ids(f"{TOK_IMG} [image:{sanitize(ref.id)} {size[0]}x{size[1]}{role}]\n")
262
+ return head + [self.vision_start_id] + [self.image_pad_id] * n_tokens + [self.vision_end_id] + self._ids("\n")
263
+
264
+ def _ids(self, text: str) -> List[int]:
265
+ return self.tok(text, add_special_tokens=False)["input_ids"]
266
+
267
+ def encode(
268
+ self,
269
+ state: Union[str, Dict[str, Any], List[Any]],
270
+ questions: Dict[str, Question],
271
+ *,
272
+ labels: Optional[Dict[str, Union[str, int, bool]]] = None,
273
+ shuffle_options: bool = False,
274
+ shuffle_questions: bool = False,
275
+ drop_label_for: Optional[Sequence[str]] = None,
276
+ rng: Optional[random.Random] = None,
277
+ state_indent: Optional[int] = None,
278
+ option_orders: Optional[Dict[str, Sequence[int]]] = None,
279
+ images: Optional[Sequence[Any]] = None,
280
+ processed_images=None,
281
+ ) -> Encoded:
282
+ """images: list of ImageRef / dicts (vision variant). processed_images: a ProcessedImages to reuse (e.g. across
283
+ the option-order permutations of one request) instead of running the image processor again."""
284
+ rng = rng or random.Random()
285
+ labels = labels or {}
286
+ drop = set(drop_label_for or [])
287
+ option_orders = option_orders or {}
288
+ qids = list(questions.keys())
289
+ if shuffle_questions:
290
+ rng.shuffle(qids)
291
+
292
+ # ---- images (vision variant)
293
+ proc = processed_images
294
+ refs: List[Any] = []
295
+ if images:
296
+ if not self.vision:
297
+ raise ValueError("this model is text-only; images and point questions need OpenThai-SystemOne-Vision")
298
+ from .images import ImageRef, process_images
299
+
300
+ refs = [ImageRef.parse(im) for im in images]
301
+ if proc is None:
302
+ kw = {"max_pixels": self.max_pixels} if self.max_pixels else {}
303
+ proc = process_images(self.image_processor, refs, device=self.image_device, **kw)
304
+ has_point = any(isinstance(questions[q], Point) for q in qids)
305
+ if has_point and not refs:
306
+ raise ValueError("point questions need at least one image")
307
+ image_index = {r.id: i for i, r in enumerate(refs)}
308
+
309
+ specs = [
310
+ question_to_spec(
311
+ qid,
312
+ questions[qid],
313
+ label=labels.get(qid),
314
+ shuffle=shuffle_options,
315
+ rng=rng,
316
+ drop_label=qid in drop,
317
+ perm=option_orders.get(qid),
318
+ )
319
+ for qid in qids
320
+ ]
321
+ point_image_index: List[int] = []
322
+ for sp in specs:
323
+ if sp.qtype != "point":
324
+ point_image_index.append(-1)
325
+ continue
326
+ if sp.image is None:
327
+ point_image_index.append(0)
328
+ elif sp.image in image_index:
329
+ point_image_index.append(image_index[sp.image])
330
+ else:
331
+ raise ValueError(f"point question {sp.qid!r} references unknown image {sp.image!r}")
332
+ q_texts = [spec_to_text(s, image_id=(refs[point_image_index[i]].id if s.qtype == "point" else None)) for i, s in enumerate(specs)]
333
+ q_ids = [self._ids(t) for t in q_texts]
334
+ q_total = sum(len(x) for x in q_ids)
335
+
336
+ # image blocks: inline where the state text says <image:id>, otherwise all before the state
337
+ blocks: Dict[str, List[int]] = {}
338
+ if refs:
339
+ for i, r in enumerate(refs):
340
+ blocks[r.id] = self._image_block(r, proc.n_tokens[i], proc.sizes[i])
341
+ state_text = sanitize(state_to_text(state, indent=state_indent)).strip()
342
+ inline = [r.id for r in refs if f"<image:{r.id}>" in state_text]
343
+ prefix_ids: List[int] = []
344
+ for r in refs:
345
+ if r.id not in inline:
346
+ prefix_ids += blocks[r.id]
347
+ body_parts: List[List[int]] = []
348
+ if inline:
349
+ import re
350
+
351
+ pieces = re.split("(" + "|".join(re.escape(f"<image:{i}>") for i in inline) + ")", state_text)
352
+ first = True
353
+ for piece in pieces:
354
+ if piece.startswith("<image:") and piece[7:-1] in blocks:
355
+ body_parts.append(self._ids("\n") + blocks[piece[7:-1]])
356
+ elif piece:
357
+ body_parts.append(self._ids((TOK_STATE + " " if first else "") + piece))
358
+ first = False
359
+ if first:
360
+ body_parts.insert(0, self._ids(TOK_STATE + " "))
361
+ state_ids = [t for part in body_parts for t in part] + self._ids("\n")
362
+ else:
363
+ state_ids = self._ids(TOK_STATE + " " + state_text + "\n")
364
+ budget = min(self.max_state_tokens, self.max_total_tokens - q_total - len(prefix_ids))
365
+ truncated = False
366
+ if len(state_ids) > budget and not inline:
367
+ # keep the head (state token) and the tail of the state; the end is usually the most recent info
368
+ keep_tail = max(budget - 1, 0)
369
+ state_ids = state_ids[:1] + state_ids[len(state_ids) - keep_tail :]
370
+ truncated = True
371
+
372
+ ids: List[int] = prefix_ids + list(state_ids)
373
+ # locate each image's contiguous pad run (in `refs` order = the order the blocks were emitted)
374
+ image_spans: List[Tuple[int, int]] = []
375
+ if refs:
376
+ runs = []
377
+ i = 0
378
+ while i < len(ids):
379
+ if ids[i] == self.image_pad_id:
380
+ j = i
381
+ while j < len(ids) and ids[j] == self.image_pad_id:
382
+ j += 1
383
+ runs.append((i, j))
384
+ i = j
385
+ else:
386
+ i += 1
387
+ # runs appear in emission order: prefix blocks first (refs order minus inline), then inline ones in text order
388
+ order = [r.id for r in refs if r.id not in inline] + [m for m in re.findall(r"<image:([^>]+)>", state_text) if m in inline] if inline else [r.id for r in refs]
389
+ by_id = dict(zip(order, runs))
390
+ image_spans = [by_id[r.id] for r in refs]
391
+ assert all(e - s == proc.n_tokens[i] for i, (s, e) in enumerate(image_spans))
392
+ answer_positions: List[int] = []
393
+ for qi in q_ids:
394
+ ids.extend(qi)
395
+ # the answer token is the last non-newline token of each question block
396
+ pos = len(ids) - 1
397
+ while ids[pos] != self.answer_id:
398
+ pos -= 1
399
+ answer_positions.append(pos)
400
+
401
+ point_targets: List[Any] = []
402
+ if refs:
403
+ from .images import bbox_to_token_target
404
+
405
+ for sp, ii in zip(specs, point_image_index):
406
+ if sp.qtype == "point" and sp.point_label_given:
407
+ t, h, w = proc.grid_thw[ii]
408
+ point_targets.append(bbox_to_token_target(sp.point_bbox, h // proc.merge, w // proc.merge))
409
+ else:
410
+ point_targets.append(None)
411
+ else:
412
+ point_targets = [None] * len(specs)
413
+
414
+ return Encoded(
415
+ input_ids=ids,
416
+ answer_positions=answer_positions,
417
+ option_counts=[len(s.option_names) for s in specs],
418
+ specs=specs,
419
+ labels=[(-100 if s.label_slot is None else s.label_slot) for s in specs],
420
+ truncated_state=truncated,
421
+ image_ids=[r.id for r in refs],
422
+ image_grid_thw=list(proc.grid_thw) if refs else [],
423
+ image_spans=image_spans,
424
+ image_sizes=list(proc.sizes) if refs else [],
425
+ pixel_values=proc.pixel_values if refs else None,
426
+ merge_size=proc.merge if refs else 2,
427
+ point_image_index=point_image_index,
428
+ point_targets=point_targets,
429
+ )
430
+
431
+
432
+ def slot_mask(option_counts: Sequence[int], *, include_abstain: bool = True, n_slots: int = N_SLOTS):
433
+ """Boolean mask (Q, n_slots): True where a slot is valid for that question."""
434
+ import torch
435
+
436
+ k = torch.as_tensor(list(option_counts), dtype=torch.long)
437
+ ar = torch.arange(n_slots)
438
+ mask = ar[None, :] < k[:, None]
439
+ if include_abstain:
440
+ mask[:, ABSTAIN_SLOT] = True
441
+ return mask
442
+
443
+
444
+ def mrope_position_ids(encoded: Sequence[Encoded], T: int):
445
+ """Qwen3.5 3-D (t, h, w) position ids for a right-padded batch, computed on CPU without per-token python loops.
446
+
447
+ Text tokens: t = h = w = running position. An image with merged grid (t, hm, wm) placed at running position p gets
448
+ t = p + frame index, h = p + row, w = p + col, and advances the running position by max(hm, wm). Equals
449
+ `Qwen3_5Model.get_rope_index` (tests/test_vision.py checks it) but costs ~0.1 ms instead of tens of ms.
450
+ """
451
+ import torch
452
+
453
+ B = len(encoded)
454
+ pos = torch.zeros((3, B, T), dtype=torch.long)
455
+ for b, e in enumerate(encoded):
456
+ n = e.n_tokens
457
+ cur = 0 # running position
458
+ idx = 0 # token index
459
+ spans = sorted(zip(e.image_spans, range(len(e.image_spans))))
460
+ for (s0, e0), i in spans:
461
+ if s0 > idx: # text before the image
462
+ ar = torch.arange(s0 - idx) + cur
463
+ pos[:, b, idx:s0] = ar
464
+ cur += s0 - idx
465
+ t, h, w = e.image_grid_thw[i]
466
+ hm, wm = h // e.merge_size, w // e.merge_size
467
+ tt = torch.arange(t).view(t, 1, 1).expand(t, hm, wm)
468
+ hh = torch.arange(hm).view(1, hm, 1).expand(t, hm, wm)
469
+ ww = torch.arange(wm).view(1, 1, wm).expand(t, hm, wm)
470
+ pos[0, b, s0:e0] = tt.reshape(-1) + cur
471
+ pos[1, b, s0:e0] = hh.reshape(-1) + cur
472
+ pos[2, b, s0:e0] = ww.reshape(-1) + cur
473
+ cur += max(hm, wm)
474
+ idx = e0
475
+ if n > idx:
476
+ pos[:, b, idx:n] = torch.arange(n - idx) + cur
477
+ return pos
478
+
479
+
480
+ def collate(encoded: Sequence[Encoded], pad_id: int, *, max_questions: Optional[int] = None):
481
+ """Right-pad a batch. Returns dict of tensors for OpenThaiSystemOneForDecision.forward."""
482
+ import torch
483
+
484
+ B = len(encoded)
485
+ T = max(e.n_tokens for e in encoded)
486
+ Q = max_questions or max(len(e.answer_positions) for e in encoded)
487
+ input_ids = torch.full((B, T), pad_id, dtype=torch.long)
488
+ attention_mask = torch.zeros((B, T), dtype=torch.long)
489
+ answer_positions = torch.zeros((B, Q), dtype=torch.long)
490
+ option_counts = torch.zeros((B, Q), dtype=torch.long)
491
+ labels = torch.full((B, Q), -100, dtype=torch.long)
492
+ qtypes = torch.full((B, Q), -1, dtype=torch.long)
493
+ for b, e in enumerate(encoded):
494
+ n = e.n_tokens
495
+ input_ids[b, :n] = torch.tensor(e.input_ids)
496
+ attention_mask[b, :n] = 1
497
+ q = len(e.answer_positions)
498
+ answer_positions[b, :q] = torch.tensor(e.answer_positions)
499
+ option_counts[b, :q] = torch.tensor(e.option_counts)
500
+ qtypes[b, :q] = torch.tensor([QTYPES.index(s.qtype) for s in e.specs])
501
+ if e.labels:
502
+ labels[b, :q] = torch.tensor(e.labels)
503
+ out = {
504
+ "input_ids": input_ids,
505
+ "attention_mask": attention_mask,
506
+ "answer_positions": answer_positions,
507
+ "option_counts": option_counts,
508
+ "labels": labels,
509
+ "qtypes": qtypes,
510
+ }
511
+ if any(e.pixel_values is not None for e in encoded):
512
+ I = max(len(e.image_spans) for e in encoded)
513
+ pv = [e.pixel_values for e in encoded if e.pixel_values is not None]
514
+ grid = [torch.tensor(e.image_grid_thw, dtype=torch.long) for e in encoded if e.image_grid_thw]
515
+ image_spans = torch.full((B, I, 2), -1, dtype=torch.long)
516
+ mm_token_type_ids = torch.zeros((B, T), dtype=torch.long)
517
+ point_image_index = torch.full((B, Q), -1, dtype=torch.long)
518
+ L = max([e.n_tokens_of_image(i) for e in encoded for i in range(len(e.image_spans))] + [1])
519
+ point_targets = torch.zeros((B, Q, L + 1))
520
+ point_has_target = torch.zeros((B, Q), dtype=torch.bool)
521
+ for b, e in enumerate(encoded):
522
+ for i, (s0, e0) in enumerate(e.image_spans):
523
+ image_spans[b, i] = torch.tensor([s0, e0])
524
+ mm_token_type_ids[b, s0:e0] = 1
525
+ for qi, ii in enumerate(e.point_image_index):
526
+ point_image_index[b, qi] = ii
527
+ tgt = e.point_targets[qi] if qi < len(e.point_targets) else None
528
+ if tgt is not None:
529
+ n = tgt.shape[0] - 1
530
+ point_targets[b, qi, :n] = tgt[:n]
531
+ point_targets[b, qi, L] = tgt[n]
532
+ point_has_target[b, qi] = True
533
+ out.update({
534
+ "pixel_values": torch.cat(pv, 0),
535
+ "image_grid_thw": torch.cat(grid, 0),
536
+ "mm_token_type_ids": mm_token_type_ids,
537
+ "position_ids": mrope_position_ids(encoded, T),
538
+ "image_spans": image_spans,
539
+ "point_image_index": point_image_index,
540
+ "point_targets": point_targets,
541
+ "point_has_target": point_has_target,
542
+ })
543
+ return out
images.py ADDED
@@ -0,0 +1,234 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Image inputs for the vision variant: loading, resizing to the ViT token budget, and the patch-grid geometry
2
+ that the `point` head works in.
3
+
4
+ Geometry: the vision tower emits one token per (patch_size * spatial_merge_size)^2 pixel block (32x32 px for
5
+ Qwen3.5), row-major over the merged grid (hm, wm) = (h / merge, w / merge) where (t, h, w) = image_grid_thw.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ import base64
10
+ import io
11
+ from dataclasses import dataclass, field
12
+ from typing import Any, Dict, List, Optional, Sequence, Tuple, Union
13
+
14
+ DEFAULT_MAX_PIXELS = 1280 * 800 # ~1000 visual tokens at 32 px/token
15
+ HIGH_DETAIL_MAX_PIXELS = 1920 * 1200 # ~2280 tokens
16
+ DEFAULT_MIN_PIXELS = 64 * 64
17
+ MAX_IMAGES_PER_REQUEST = 4
18
+ MAX_VISUAL_TOKENS = 8192
19
+
20
+
21
+ def load_image(src: Any):
22
+ """PIL image from: PIL.Image, bytes, data-URI / base64 string, http(s) URL, or a filesystem path."""
23
+ from PIL import Image
24
+
25
+ if isinstance(src, Image.Image):
26
+ return src.convert("RGB")
27
+ if isinstance(src, (bytes, bytearray)):
28
+ return Image.open(io.BytesIO(src)).convert("RGB")
29
+ if hasattr(src, "shape"): # numpy array (H, W, 3)
30
+ return Image.fromarray(src).convert("RGB")
31
+ if not isinstance(src, str):
32
+ raise TypeError(f"unsupported image source {type(src)}")
33
+ if src.startswith("data:"):
34
+ _, _, payload = src.partition(",")
35
+ return Image.open(io.BytesIO(base64.b64decode(payload))).convert("RGB")
36
+ if src.startswith(("http://", "https://")):
37
+ import urllib.request
38
+
39
+ req = urllib.request.Request(src, headers={"User-Agent": "openthai-systemone/1.0"})
40
+ with urllib.request.urlopen(req, timeout=20) as r:
41
+ return Image.open(io.BytesIO(r.read())).convert("RGB")
42
+ if len(src) > 512 and "/" not in src[:16] and not src.startswith("."):
43
+ try: # bare base64
44
+ return Image.open(io.BytesIO(base64.b64decode(src, validate=True))).convert("RGB")
45
+ except Exception:
46
+ pass
47
+ return Image.open(src).convert("RGB")
48
+
49
+
50
+ @dataclass
51
+ class ImageRef:
52
+ """One image attached to a request."""
53
+
54
+ id: str
55
+ image: Any # anything load_image accepts
56
+ detail: str = "auto" # auto | high
57
+ role: Optional[str] = None # free text tag, shown in the sequence (screenshot, frame, photo, document)
58
+
59
+ @classmethod
60
+ def parse(cls, obj: Union["ImageRef", Dict[str, Any]]) -> "ImageRef":
61
+ if isinstance(obj, ImageRef):
62
+ return obj
63
+ d = dict(obj)
64
+ src = d.get("image", d.get("data", d.get("url", d.get("path"))))
65
+ if src is None:
66
+ raise ValueError("image needs one of: image / data / url / path")
67
+ return cls(id=str(d.get("id", "img")), image=src, detail=str(d.get("detail", "auto")), role=d.get("role"))
68
+
69
+
70
+ @dataclass
71
+ class ProcessedImages:
72
+ ids: List[str]
73
+ pixel_values: Any # torch.Tensor (sum_patches, C*T*P*P)
74
+ grid_thw: List[List[int]] # per image (t, h, w) in patches
75
+ sizes: List[Tuple[int, int]] # original (W, H)
76
+ n_tokens: List[int] = field(default_factory=list)
77
+ merge: int = 2
78
+
79
+ def tokens_for(self, i: int) -> int:
80
+ t, h, w = self.grid_thw[i]
81
+ return t * h * w // (self.merge * self.merge)
82
+
83
+ def merged_grid(self, i: int) -> Tuple[int, int]:
84
+ t, h, w = self.grid_thw[i]
85
+ return h // self.merge, w // self.merge
86
+
87
+
88
+ def fast_patchify(im, image_processor, *, min_pixels: int, max_pixels: int, device=None):
89
+ """Qwen2-VL preprocessing without the HF processor's overhead (~3 ms vs ~40 ms per frame on a busy CPU).
90
+
91
+ Same maths: smart_resize to multiples of patch*merge within [min_pixels, max_pixels], bicubic resize, rescale,
92
+ normalise with the processor's mean/std, duplicate the frame temporal_patch_size times, and flatten patches in the
93
+ processor's (t, h/merge, w/merge, merge, merge, C, T, p, p) order. tests/test_vision.py checks equality with the HF
94
+ processor output.
95
+ """
96
+ import numpy as np
97
+ import torch
98
+ from PIL import Image
99
+ from transformers.models.qwen2_vl.image_processing_qwen2_vl import smart_resize
100
+
101
+ patch = int(getattr(image_processor, "patch_size", 16))
102
+ merge = int(getattr(image_processor, "merge_size", 2))
103
+ tp = int(getattr(image_processor, "temporal_patch_size", 2))
104
+ mean = torch.tensor(image_processor.image_mean, dtype=torch.float32).view(3, 1, 1)
105
+ std = torch.tensor(image_processor.image_std, dtype=torch.float32).view(3, 1, 1)
106
+ w, h = im.size
107
+ rh, rw = smart_resize(h, w, factor=patch * merge, min_pixels=min_pixels, max_pixels=max_pixels)
108
+ if (rw, rh) != (w, h):
109
+ im = im.resize((rw, rh), Image.BICUBIC)
110
+ x = torch.from_numpy(np.asarray(im, dtype=np.uint8))
111
+ if device is not None: # normalise + patchify on the GPU: the CPU only resizes and uploads 3 bytes/pixel
112
+ x = x.to(device, non_blocking=True)
113
+ mean, std = mean.to(device), std.to(device)
114
+ x = x.permute(2, 0, 1).float() / 255.0 # (C, H, W)
115
+ x = (x - mean) / std
116
+ gh, gw = rh // patch, rw // patch
117
+ x = x.unsqueeze(0).expand(tp, -1, -1, -1) # (T, C, H, W)
118
+ x = x.reshape(tp, 3, gh // merge, merge, patch, gw // merge, merge, patch)
119
+ # HF order: (grid_t, temporal, C, gh//m, m, p, gw//m, m, p) -> transpose(0, 3, 6, 4, 7, 2, 1, 5, 8)
120
+ x = x.unsqueeze(0) # (1, T, C, gh/m, m, p, gw/m, m, p)
121
+ x = x.permute(0, 3, 6, 4, 7, 2, 1, 5, 8).reshape(gh * gw, 3 * tp * patch * patch)
122
+ return x, [1, gh, gw]
123
+
124
+
125
+ def process_images(image_processor, refs: Sequence[ImageRef], *, max_pixels: int = DEFAULT_MAX_PIXELS,
126
+ min_pixels: int = DEFAULT_MIN_PIXELS, fast: bool = True, device=None) -> ProcessedImages:
127
+ """Preprocess each image (own detail cap each) and concatenate. fast=True uses fast_patchify (identical output)."""
128
+ import torch
129
+
130
+ if len(refs) > MAX_IMAGES_PER_REQUEST:
131
+ raise ValueError(f"at most {MAX_IMAGES_PER_REQUEST} images per request")
132
+ merge = int(getattr(image_processor, "merge_size", 2))
133
+ pv, grids, sizes, ids, ntoks = [], [], [], [], []
134
+ for r in refs:
135
+ im = load_image(r.image)
136
+ sizes.append(im.size)
137
+ cap = HIGH_DETAIL_MAX_PIXELS if r.detail == "high" else max_pixels
138
+ if fast:
139
+ x, g = fast_patchify(im, image_processor, min_pixels=min_pixels, max_pixels=cap, device=device)
140
+ pv.append(x)
141
+ else:
142
+ out = image_processor(images=[im], size={"shortest_edge": min_pixels, "longest_edge": cap}, return_tensors="pt")
143
+ pv.append(out["pixel_values"])
144
+ g = [int(v) for v in out["image_grid_thw"][0].tolist()]
145
+ grids.append(g)
146
+ ids.append(r.id)
147
+ ntoks.append(g[0] * g[1] * g[2] // (merge * merge))
148
+ if sum(ntoks) > MAX_VISUAL_TOKENS:
149
+ raise ValueError(f"images need {sum(ntoks)} visual tokens > {MAX_VISUAL_TOKENS}; lower detail or send fewer images")
150
+ return ProcessedImages(ids=ids, pixel_values=torch.cat(pv, 0), grid_thw=grids, sizes=sizes, n_tokens=ntoks, merge=merge)
151
+
152
+
153
+ def bbox_to_token_target(bbox: Optional[Sequence[float]], hm: int, wm: int) -> "torch.Tensor":
154
+ """Soft target over the hm*wm tokens (+1 null slot at the end) for a normalised bbox [x0,y0,x1,y1] in 0..1.
155
+
156
+ Cells whose area overlaps the box get weight = overlap fraction of the cell (GUI-Actor style multi-patch
157
+ supervision); normalised to sum 1. bbox None -> all mass on the null slot ("not on screen").
158
+ """
159
+ import torch
160
+
161
+ tgt = torch.zeros(hm * wm + 1)
162
+ if bbox is None:
163
+ tgt[-1] = 1.0
164
+ return tgt
165
+ x0, y0, x1, y1 = [float(v) for v in bbox]
166
+ x0, x1 = sorted((max(0.0, min(1.0, x0)), max(0.0, min(1.0, x1))))
167
+ y0, y1 = sorted((max(0.0, min(1.0, y0)), max(0.0, min(1.0, y1))))
168
+ # a degenerate (point) box marks the cell that contains it
169
+ if x1 - x0 < 1e-6:
170
+ x0, x1 = max(0.0, x0 - 0.5 / wm), min(1.0, x0 + 0.5 / wm)
171
+ if y1 - y0 < 1e-6:
172
+ y0, y1 = max(0.0, y0 - 0.5 / hm), min(1.0, y0 + 0.5 / hm)
173
+ for r in range(hm):
174
+ cy0, cy1 = r / hm, (r + 1) / hm
175
+ oy = max(0.0, min(cy1, y1) - max(cy0, y0))
176
+ if oy <= 0:
177
+ continue
178
+ for c in range(wm):
179
+ cx0, cx1 = c / wm, (c + 1) / wm
180
+ ox = max(0.0, min(cx1, x1) - max(cx0, x0))
181
+ if ox > 0:
182
+ tgt[r * wm + c] = ox * oy * hm * wm # fraction of the cell covered
183
+ if tgt[:-1].sum() <= 0:
184
+ tgt[-1] = 1.0
185
+ else:
186
+ tgt = tgt / tgt.sum()
187
+ return tgt
188
+
189
+
190
+ def token_xy(index: int, hm: int, wm: int) -> Tuple[float, float]:
191
+ """Centre of merged token `index` in normalised image coordinates."""
192
+ r, c = divmod(index, wm)
193
+ return (c + 0.5) / wm, (r + 0.5) / hm
194
+
195
+
196
+ def heatmap_to_candidates(probs: Sequence[float], hm: int, wm: int, *, rel_threshold: float = 0.3, max_candidates: int = 5):
197
+ """GUI-Actor decoding: threshold at rel_threshold * max, 8-connected components, mass-weighted centroids.
198
+
199
+ probs: hm*wm token probabilities (null slot already removed). Returns list of {x, y, p, bbox} sorted by p.
200
+ """
201
+ n = hm * wm
202
+ p = [float(v) for v in probs[:n]]
203
+ if n == 0 or max(p) <= 0:
204
+ return []
205
+ thr = rel_threshold * max(p)
206
+ keep = {i for i, v in enumerate(p) if v >= thr}
207
+ seen, comps = set(), []
208
+ for start in sorted(keep, key=lambda i: -p[i]):
209
+ if start in seen:
210
+ continue
211
+ comp, stack = [], [start]
212
+ seen.add(start)
213
+ while stack:
214
+ i = stack.pop()
215
+ comp.append(i)
216
+ r, c = divmod(i, wm)
217
+ for dr in (-1, 0, 1):
218
+ for dc in (-1, 0, 1):
219
+ rr, cc = r + dr, c + dc
220
+ if 0 <= rr < hm and 0 <= cc < wm:
221
+ j = rr * wm + cc
222
+ if j in keep and j not in seen:
223
+ seen.add(j)
224
+ stack.append(j)
225
+ mass = sum(p[i] for i in comp)
226
+ xs = [token_xy(i, hm, wm) for i in comp]
227
+ x = sum(px * p[i] for (px, _), i in zip(xs, comp)) / mass
228
+ y = sum(py * p[i] for (_, py), i in zip(xs, comp)) / mass
229
+ rows = [i // wm for i in comp]
230
+ cols = [i % wm for i in comp]
231
+ bbox = [min(cols) / wm, min(rows) / hm, (max(cols) + 1) / wm, (max(rows) + 1) / hm]
232
+ comps.append({"x": round(x, 4), "y": round(y, 4), "p": round(mass, 4), "bbox": [round(v, 4) for v in bbox]})
233
+ comps.sort(key=lambda d: -d["p"])
234
+ return comps[:max_candidates]
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9a69dba19b55dfe7ed3c80c24e16e708ce2571961ae844e4de5dd0da71fa7bbf
3
+ size 1724713898
modeling.py ADDED
@@ -0,0 +1,461 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """OpenThai-SystemOne decision model.
2
+
3
+ text tower (Qwen3.5-0.8B, LM head removed) -> hidden state at every <|ts_answer|> -> SlotHead (256 logits)
4
+ mask slots >= k -> softmax -> probabilities over the k options
5
+
6
+ -Vision variant: the tower is the multimodal Qwen3.5 model (ViT + merger + the same text tower). Images enter as
7
+ runs of <|image_pad|> tokens; `point` questions are read out by the PointHead, which attends from the <|ts_answer|>
8
+ hidden state over the LLM's final hidden states of that image's visual tokens (GUI-Actor recipe) plus a learned
9
+ "null" key for "not on the image". With no pixel_values the forward path is identical to the text-only model.
10
+ """
11
+ from __future__ import annotations
12
+
13
+ import math
14
+ from dataclasses import dataclass
15
+ from typing import Optional
16
+
17
+ import torch
18
+ import torch.nn as nn
19
+ import torch.nn.functional as F
20
+ from transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer, PreTrainedModel
21
+ from transformers.utils import ModelOutput
22
+
23
+ from .configuration import OpenThaiSystemOneConfig
24
+ from .formatting import SPECIAL_TOKENS, TOK_ANSWER, TOK_POINT, QWEN_IMAGE_PAD, QWEN_VISION_END, QWEN_VISION_START, add_special_tokens
25
+
26
+ QTYPE_INDEX = {"choice": 0, "score": 1, "noul": 2, "point": 3}
27
+
28
+
29
+ @dataclass
30
+ class DecisionOutput(ModelOutput):
31
+ loss: Optional[torch.Tensor] = None
32
+ logits: Optional[torch.Tensor] = None # (B, Q, n_slots), masked with -inf
33
+ probs: Optional[torch.Tensor] = None # (B, Q, n_slots)
34
+ hidden_states: Optional[torch.Tensor] = None # (B, Q, H) at answer positions
35
+ point_logits: Optional[torch.Tensor] = None # (B, Q, L+1): per visual token of the referenced image + null; -inf elsewhere
36
+ point_probs: Optional[torch.Tensor] = None
37
+ slot_loss: Optional[torch.Tensor] = None
38
+ point_loss: Optional[torch.Tensor] = None
39
+
40
+
41
+ class PointHead(nn.Module):
42
+ """Attention readout over one image's visual tokens (final-layer LLM states).
43
+
44
+ q = MLP_T(h_answer), k_i = MLP_V(SA(v_i)), logits_i = q.k_i / sqrt(d) / T ; extra null key = "not on the image".
45
+ """
46
+
47
+ def __init__(self, hidden: int, dim: int = 256, n_sa_layers: int = 1, n_heads: int = 8):
48
+ super().__init__()
49
+ self.sa = nn.ModuleList(
50
+ [nn.TransformerEncoderLayer(hidden, n_heads, dim_feedforward=2 * hidden, dropout=0.0, batch_first=True, norm_first=True) for _ in range(n_sa_layers)]
51
+ )
52
+ self.q_proj = nn.Sequential(nn.Linear(hidden, dim), nn.GELU(), nn.Linear(dim, dim))
53
+ self.k_proj = nn.Sequential(nn.Linear(hidden, dim), nn.GELU(), nn.Linear(dim, dim))
54
+ self.null_key = nn.Parameter(torch.zeros(dim))
55
+ self.log_temperature = nn.Parameter(torch.zeros(()))
56
+ self.dim = dim
57
+
58
+ def forward(self, h_answer: torch.Tensor, visual: torch.Tensor, visual_mask: torch.Tensor) -> torch.Tensor:
59
+ """h_answer (N, H); visual (N, L, H) right-padded; visual_mask (N, L) True = real token. Returns (N, L+1) logits."""
60
+ x = visual
61
+ for layer in self.sa:
62
+ x = layer(x, src_key_padding_mask=~visual_mask)
63
+ k = self.k_proj(x) # (N, L, d)
64
+ q = self.q_proj(h_answer) # (N, d)
65
+ logits = torch.einsum("nd,nld->nl", q, k) / math.sqrt(self.dim)
66
+ null = (q @ self.null_key)[:, None] / math.sqrt(self.dim)
67
+ logits = torch.cat([logits, null], dim=1) / self.log_temperature.exp()
68
+ pad = torch.cat([~visual_mask, torch.zeros_like(visual_mask[:, :1])], dim=1)
69
+ return logits.masked_fill(pad, float("-inf"))
70
+
71
+
72
+ class OpenThaiSystemOneForDecision(PreTrainedModel):
73
+ config_class = OpenThaiSystemOneConfig
74
+ base_model_prefix = "model"
75
+ supports_gradient_checkpointing = True
76
+ _supports_flash_attn = True
77
+ _supports_sdpa = True
78
+
79
+ def __init__(self, config: OpenThaiSystemOneConfig):
80
+ super().__init__(config)
81
+ if config.is_vision:
82
+ from transformers.models.qwen3_5.configuration_qwen3_5 import Qwen3_5Config
83
+ from transformers.models.qwen3_5.modeling_qwen3_5 import Qwen3_5Model
84
+
85
+ mm = Qwen3_5Config(
86
+ text_config=config.text_config,
87
+ vision_config=config.vision_config,
88
+ image_token_id=config.image_token_id,
89
+ vision_start_token_id=config.vision_start_token_id,
90
+ vision_end_token_id=config.vision_end_token_id,
91
+ tie_word_embeddings=False,
92
+ )
93
+ self.model = Qwen3_5Model(mm)
94
+ enable_vision_helper_caches()
95
+ ph = dict(config.point_head or {})
96
+ self.point_head = PointHead(config.hidden_size, ph.get("dim", 256), ph.get("n_sa_layers", 1), ph.get("n_heads", 8))
97
+ else:
98
+ self.model = AutoModel.from_config(config.text_config)
99
+ self.point_head = None
100
+ self.slot_head = nn.Linear(config.hidden_size, config.n_slots, bias=config.head_bias)
101
+ # log-temperatures per question type (choice/score/noul[/point]); learned in the calibration stage
102
+ self.log_temperature = nn.Parameter(torch.zeros(config.n_temperatures))
103
+ self.post_init()
104
+
105
+ @property
106
+ def language_model(self):
107
+ return self.model.language_model if self.config.is_vision else self.model
108
+
109
+ def get_input_embeddings(self):
110
+ return self.language_model.get_input_embeddings()
111
+
112
+ def set_input_embeddings(self, value):
113
+ self.language_model.set_input_embeddings(value)
114
+
115
+ # ------------------------------------------------------------------ construction
116
+ @classmethod
117
+ def from_causal_lm(
118
+ cls,
119
+ path: str,
120
+ *,
121
+ tokenizer=None,
122
+ n_slots: int = 256,
123
+ torch_dtype=torch.bfloat16,
124
+ **kwargs,
125
+ ):
126
+ """Build a text-only decision model from a (text-only) causal-LM checkpoint: drop lm_head, add tokens + head."""
127
+ tok = tokenizer or AutoTokenizer.from_pretrained(path)
128
+ added = add_special_tokens(tok)
129
+ lm = AutoModelForCausalLM.from_pretrained(path, dtype=torch_dtype, **kwargs)
130
+ base = lm.model if hasattr(lm, "model") else lm.base_model
131
+ text_cfg = base.config
132
+ if added:
133
+ lm.resize_token_embeddings(len(tok), mean_resizing=False)
134
+ text_cfg.vocab_size = lm.get_input_embeddings().weight.shape[0]
135
+ _init_new_token_embeddings(lm.get_input_embeddings().weight, tok, added)
136
+ cfg = OpenThaiSystemOneConfig(
137
+ text_config=text_cfg,
138
+ n_slots=n_slots,
139
+ answer_token_id=tok.convert_tokens_to_ids(TOK_ANSWER),
140
+ pad_token_id=tok.pad_token_id,
141
+ )
142
+ cfg.text_config.tie_word_embeddings = False # there is no LM head any more
143
+ model = cls(cfg).to(torch_dtype)
144
+ missing, unexpected = model.model.load_state_dict(base.state_dict(), strict=False)
145
+ assert not unexpected, unexpected
146
+ _init_slot_head(model.slot_head)
147
+ model.model.config = cfg.text_config
148
+ return model, tok
149
+
150
+ @classmethod
151
+ def from_text_decision_model(
152
+ cls,
153
+ text_path: str,
154
+ vision_source: str,
155
+ *,
156
+ tokenizer=None,
157
+ point_head: Optional[dict] = None,
158
+ torch_dtype=torch.bfloat16,
159
+ ):
160
+ """Build the -Vision model: text tower + slot head from a text-only decision checkpoint (e.g. v0.3), the ViT +
161
+ merger from a Qwen3.5 multimodal checkpoint (base/Qwen3.5-0.8B-Base), a fresh PointHead, and the two extra
162
+ control tokens. Text-only forward of the result equals the source model."""
163
+ import glob
164
+ import json
165
+ import os
166
+
167
+ from safetensors import safe_open
168
+ from transformers import AutoConfig
169
+
170
+ text_model = cls.from_pretrained(text_path, dtype=torch_dtype)
171
+ assert not text_model.config.is_vision, "source must be the text-only model"
172
+ tok = tokenizer or AutoTokenizer.from_pretrained(text_path)
173
+ added = add_special_tokens(tok, vision=True)
174
+ src_cfg = AutoConfig.from_pretrained(vision_source)
175
+ vcfg = src_cfg.vision_config
176
+ vcfg.out_hidden_size = text_model.config.hidden_size
177
+ cfg = OpenThaiSystemOneConfig(
178
+ text_config=text_model.config.text_config,
179
+ n_slots=text_model.config.n_slots,
180
+ abstain_slot=text_model.config.abstain_slot,
181
+ answer_token_id=text_model.config.answer_token_id,
182
+ head_bias=text_model.config.head_bias,
183
+ n_temperatures=4,
184
+ vision_config=vcfg,
185
+ image_token_id=tok.convert_tokens_to_ids(QWEN_IMAGE_PAD),
186
+ vision_start_token_id=tok.convert_tokens_to_ids(QWEN_VISION_START),
187
+ vision_end_token_id=tok.convert_tokens_to_ids(QWEN_VISION_END),
188
+ point_token_id=tok.convert_tokens_to_ids(TOK_POINT),
189
+ point_head=point_head or {"dim": 256, "n_sa_layers": 1, "n_heads": 8},
190
+ pad_token_id=tok.pad_token_id,
191
+ )
192
+ cfg.text_config.vocab_size = len(tok)
193
+ model = cls(cfg).to(torch_dtype)
194
+ # text tower + heads
195
+ old_vocab = text_model.get_input_embeddings().weight.shape[0]
196
+ if len(tok) > old_vocab:
197
+ text_model.model.resize_token_embeddings(len(tok), mean_resizing=False)
198
+ _init_new_token_embeddings(text_model.get_input_embeddings().weight, tok, len(tok) - old_vocab)
199
+ missing, unexpected = model.model.language_model.load_state_dict(text_model.model.state_dict(), strict=False)
200
+ assert not unexpected and not missing, (missing, unexpected)
201
+ model.slot_head.load_state_dict(text_model.slot_head.state_dict())
202
+ with torch.no_grad():
203
+ n = min(text_model.log_temperature.numel(), model.log_temperature.numel())
204
+ model.log_temperature[:n] = text_model.log_temperature[:n]
205
+ # vision tower from the multimodal checkpoint
206
+ files = sorted(glob.glob(os.path.join(vision_source, "*.safetensors")))
207
+ sd = {}
208
+ for f in files:
209
+ with safe_open(f, "pt") as fh:
210
+ for k in fh.keys():
211
+ for prefix in ("model.visual.", "visual."):
212
+ if k.startswith(prefix):
213
+ sd[k[len(prefix):]] = fh.get_tensor(k)
214
+ break
215
+ missing, unexpected = model.model.visual.load_state_dict({k: v.to(torch_dtype) for k, v in sd.items()}, strict=False)
216
+ assert not unexpected and not missing, (missing, unexpected)
217
+ model.model.config.text_config = cfg.text_config
218
+ return model, tok
219
+
220
+ # ------------------------------------------------------------------ forward
221
+ def gather_answer_states(self, hidden: torch.Tensor, answer_positions: torch.Tensor) -> torch.Tensor:
222
+ idx = answer_positions.clamp(min=0).unsqueeze(-1).expand(-1, -1, hidden.shape[-1])
223
+ return torch.gather(hidden, 1, idx) # (B, Q, H)
224
+
225
+ def slot_logits(
226
+ self,
227
+ answer_hidden: torch.Tensor,
228
+ option_counts: torch.Tensor,
229
+ *,
230
+ include_abstain: bool = True,
231
+ qtypes: Optional[torch.Tensor] = None,
232
+ apply_temperature: bool = True,
233
+ ) -> torch.Tensor:
234
+ logits = self.slot_head(answer_hidden.to(self.slot_head.weight.dtype)).float()
235
+ if apply_temperature:
236
+ if qtypes is None:
237
+ t = self.log_temperature[0].exp()
238
+ else:
239
+ t = self.log_temperature.exp()[qtypes.clamp(min=0, max=self.log_temperature.numel() - 1)] # (B, Q)
240
+ t = t.unsqueeze(-1)
241
+ logits = logits / t
242
+ ar = torch.arange(logits.shape[-1], device=logits.device)
243
+ valid = ar[None, None, :] < option_counts.unsqueeze(-1)
244
+ if include_abstain:
245
+ valid = valid.clone()
246
+ valid[..., self.config.abstain_slot] = True
247
+ # questions that are padding or point questions (option_counts == 0) keep slot 0 valid to avoid NaNs
248
+ valid[..., 0] |= option_counts.unsqueeze(-1).squeeze(-1) == 0
249
+ return logits.masked_fill(~valid, float("-inf"))
250
+
251
+ def point_logits(
252
+ self,
253
+ hidden: torch.Tensor,
254
+ answer_hidden: torch.Tensor,
255
+ image_spans: torch.Tensor,
256
+ point_image_index: torch.Tensor,
257
+ *,
258
+ max_tokens: Optional[int] = None,
259
+ ) -> Optional[torch.Tensor]:
260
+ """(B, Q, L+1) logits for point questions (-inf rows elsewhere)."""
261
+ B, Q = point_image_index.shape
262
+ sel = (point_image_index >= 0).nonzero(as_tuple=False)
263
+ L = max_tokens or int((image_spans[..., 1] - image_spans[..., 0]).clamp(min=0).max().item())
264
+ out = torch.full((B, Q, L + 1), float("-inf"), device=hidden.device)
265
+ if sel.numel() == 0:
266
+ return out
267
+ n = sel.shape[0]
268
+ vis = hidden.new_zeros((n, L, hidden.shape[-1]))
269
+ mask = torch.zeros((n, L), dtype=torch.bool, device=hidden.device)
270
+ for r, (b, q) in enumerate(sel.tolist()):
271
+ s0, e0 = image_spans[b, point_image_index[b, q]].tolist()
272
+ vis[r, : e0 - s0] = hidden[b, s0:e0]
273
+ mask[r, : e0 - s0] = True
274
+ h = answer_hidden[sel[:, 0], sel[:, 1]]
275
+ pl = self.point_head(h.to(vis.dtype), vis, mask).float()
276
+ out[sel[:, 0], sel[:, 1]] = pl
277
+ return out
278
+
279
+ def forward(
280
+ self,
281
+ input_ids: torch.Tensor,
282
+ attention_mask: Optional[torch.Tensor] = None,
283
+ answer_positions: Optional[torch.Tensor] = None,
284
+ option_counts: Optional[torch.Tensor] = None,
285
+ labels: Optional[torch.Tensor] = None,
286
+ soft_labels: Optional[torch.Tensor] = None,
287
+ qtypes: Optional[torch.Tensor] = None,
288
+ include_abstain: bool = True,
289
+ label_smoothing: float = 0.0,
290
+ brier_weight: float = 0.0,
291
+ apply_temperature: bool = True,
292
+ pixel_values: Optional[torch.Tensor] = None,
293
+ image_grid_thw: Optional[torch.Tensor] = None,
294
+ mm_token_type_ids: Optional[torch.Tensor] = None,
295
+ image_spans: Optional[torch.Tensor] = None,
296
+ point_image_index: Optional[torch.Tensor] = None,
297
+ point_targets: Optional[torch.Tensor] = None,
298
+ point_has_target: Optional[torch.Tensor] = None,
299
+ point_loss_weight: float = 1.0,
300
+ position_ids: Optional[torch.Tensor] = None,
301
+ **kwargs,
302
+ ) -> DecisionOutput:
303
+ if pixel_values is not None:
304
+ if not self.config.is_vision:
305
+ raise ValueError("this checkpoint is text-only; images need OpenThai-SystemOne-Vision")
306
+ # position_ids (3, B, T) from formatting.mrope_position_ids skips the slow per-token get_rope_index;
307
+ # image_grid_thw may stay on the CPU (no GPU syncs in the python bookkeeping) — helper outputs go to the pixel device
308
+ _VISION_TARGET_DEVICE[0] = pixel_values.device
309
+ out = self.model(
310
+ input_ids=input_ids, attention_mask=attention_mask, pixel_values=pixel_values,
311
+ image_grid_thw=image_grid_thw, mm_token_type_ids=mm_token_type_ids, position_ids=position_ids, **kwargs,
312
+ )
313
+ else:
314
+ out = self.model(input_ids=input_ids, attention_mask=attention_mask, **kwargs)
315
+ hidden = out.last_hidden_state
316
+ if answer_positions is None:
317
+ answer_positions = (input_ids == self.config.answer_token_id).nonzero()[:, 1].unsqueeze(0)
318
+ if option_counts is None:
319
+ raise ValueError("option_counts required")
320
+ h = self.gather_answer_states(hidden, answer_positions)
321
+ logits = self.slot_logits(h, option_counts, include_abstain=include_abstain, qtypes=qtypes, apply_temperature=apply_temperature)
322
+ probs = logits.softmax(-1)
323
+
324
+ p_logits = p_probs = None
325
+ if self.point_head is not None and point_image_index is not None and (point_image_index >= 0).any():
326
+ p_logits = self.point_logits(hidden, h, image_spans, point_image_index,
327
+ max_tokens=(point_targets.shape[-1] - 1) if point_targets is not None else None)
328
+ p_probs = p_logits.softmax(-1)
329
+ p_probs = p_probs.masked_fill(torch.isinf(p_logits).all(-1, keepdim=True), 0.0)
330
+
331
+ loss = slot_loss = point_loss = None
332
+ if labels is not None or soft_labels is not None:
333
+ logp = logits.log_softmax(-1)
334
+ if soft_labels is not None:
335
+ valid = (option_counts > 0)
336
+ tgt = soft_labels.float()
337
+ nll = -(tgt * logp.masked_fill(torch.isinf(logp), 0.0)).sum(-1)
338
+ slot_loss = (nll * valid).sum() / valid.sum().clamp(min=1)
339
+ else:
340
+ flat_logp = logp.reshape(-1, logp.shape[-1])
341
+ flat_lab = labels.reshape(-1)
342
+ keep = flat_lab != -100
343
+ if keep.any():
344
+ lp = flat_logp[keep]
345
+ lb = flat_lab[keep]
346
+ nll = -lp.gather(1, lb[:, None]).squeeze(1)
347
+ if label_smoothing > 0:
348
+ n_valid = torch.isfinite(lp).sum(-1).clamp(min=1).float()
349
+ smooth = -(lp.masked_fill(torch.isinf(lp), 0.0)).sum(-1) / n_valid
350
+ nll = (1 - label_smoothing) * nll + label_smoothing * smooth
351
+ slot_loss = nll.mean()
352
+ if brier_weight > 0:
353
+ p = lp.exp()
354
+ onehot = F.one_hot(lb, p.shape[-1]).float()
355
+ slot_loss = slot_loss + brier_weight * ((p - onehot) ** 2).sum(-1).mean()
356
+ else:
357
+ slot_loss = logits.sum() * 0.0
358
+ loss = slot_loss
359
+ if p_logits is not None and point_targets is not None and point_has_target is not None and point_has_target.any():
360
+ lp = p_logits.log_softmax(-1).masked_fill(torch.isinf(p_logits), 0.0)
361
+ tgt = point_targets.float()
362
+ nll = -(tgt * lp).sum(-1) # cross-entropy against the soft mask (= KL up to the target entropy)
363
+ point_loss = (nll * point_has_target).sum() / point_has_target.sum().clamp(min=1)
364
+ loss = point_loss * point_loss_weight if loss is None else loss + point_loss_weight * point_loss
365
+ return DecisionOutput(loss=loss, logits=logits, probs=probs, hidden_states=h, point_logits=p_logits, point_probs=p_probs,
366
+ slot_loss=slot_loss, point_loss=point_loss)
367
+
368
+
369
+ _VISION_CACHE_ON = False
370
+ _VISION_TARGET_DEVICE = [None] # set by the forward: cached ViT helper outputs are moved here (lets image_grid_thw stay on the CPU)
371
+
372
+
373
+ def enable_vision_helper_caches(maxsize: int = 512):
374
+ """Memoise the Qwen3.5 ViT's per-call helpers by image grid.
375
+
376
+ `get_vision_interpolation_indices_and_weights`, `get_vision_position_ids` and `get_vision_attention_seqlens` rebuild
377
+ the same index tensors on every forward (tens of ms of small ops for a 320x240 frame). Their outputs depend only on
378
+ the grid (t, h, w) and the device, so they are cached; a Doom loop or a fixed-size screenshot stream hits the cache
379
+ every step. Idempotent; called automatically when a vision model is built.
380
+ """
381
+ global _VISION_CACHE_ON
382
+ if _VISION_CACHE_ON:
383
+ return
384
+ try:
385
+ from transformers.models.qwen3_5 import modeling_qwen3_5 as mq
386
+ except Exception: # pragma: no cover
387
+ return
388
+
389
+ def cached(fn, key_extra=lambda *a, **k: ()):
390
+ cache = {}
391
+
392
+ def wrapper(grid_thw, *args, **kwargs):
393
+ kw = {k: v for k, v in kwargs.items() if k != "kwargs"}
394
+ target = _VISION_TARGET_DEVICE[0] or grid_thw.device
395
+ key = (tuple(map(tuple, grid_thw.tolist())), str(target), tuple(sorted((k, str(v)) for k, v in kw.items())), tuple(str(a) for a in args))
396
+ hit = cache.get(key)
397
+ if hit is None:
398
+ hit = fn(grid_thw, *args, **kwargs)
399
+
400
+ def mv(x):
401
+ return x.to(target) if torch.is_tensor(x) else x
402
+
403
+ hit = tuple(mv(x) for x in hit) if isinstance(hit, tuple) else mv(hit)
404
+ if len(cache) >= maxsize:
405
+ cache.clear()
406
+ cache[key] = hit
407
+ return hit
408
+
409
+ wrapper.__wrapped__ = fn
410
+ return wrapper
411
+
412
+ for name in ("get_vision_interpolation_indices_and_weights", "get_vision_position_ids", "get_vision_attention_seqlens"):
413
+ fn = getattr(mq, name, None)
414
+ if fn is not None and not hasattr(fn, "__wrapped__"):
415
+ setattr(mq, name, cached(fn))
416
+ _VISION_CACHE_ON = True
417
+
418
+
419
+ def use_reference_kernels():
420
+ """Force the pure-PyTorch Gated-DeltaNet / causal-conv paths.
421
+
422
+ transformers routes `chunk_gated_delta_rule` & co. to the Triton kernels (flash-linear-attention, causal-conv1d)
423
+ whenever those packages are importable, without checking the tensor device, which crashes on CPU/MPS.
424
+ Call this before running on a non-CUDA device.
425
+ """
426
+ try:
427
+ from transformers.models.qwen3_5 import modeling_qwen3_5 as m
428
+ except Exception: # pragma: no cover
429
+ return
430
+ for name in ("torch_chunk_gated_delta_rule", "torch_recurrent_gated_delta_rule", "chunk_gated_delta_rule",
431
+ "fused_recurrent_gated_delta_rule", "causal_conv1d_fn", "causal_conv1d_update"):
432
+ fn = getattr(m, name, None)
433
+ if fn is not None and hasattr(fn, "__wrapped__"):
434
+ setattr(m, name, fn.__wrapped__)
435
+
436
+
437
+ def _init_slot_head(head: nn.Linear):
438
+ nn.init.normal_(head.weight, std=0.02)
439
+ if head.bias is not None:
440
+ nn.init.zeros_(head.bias)
441
+
442
+
443
+ @torch.no_grad()
444
+ def _init_new_token_embeddings(weight: torch.Tensor, tok, n_added: int):
445
+ """New control tokens start near the mean of digit-token embeddings + small noise."""
446
+ digit_ids = [tok.convert_tokens_to_ids(d) for d in "0123456789"]
447
+ digit_ids = [i for i in digit_ids if i is not None and i != tok.unk_token_id]
448
+ mean = weight[digit_ids].float().mean(0) if digit_ids else weight[: weight.shape[0] - n_added].float().mean(0)
449
+ std = weight[: weight.shape[0] - n_added].float().std()
450
+ new = mean[None, :] + torch.randn(n_added, weight.shape[1]) * std * 0.1
451
+ weight[-n_added:] = new.to(weight.dtype)
452
+
453
+
454
+ def confidence_from_probs(p: torch.Tensor, k: int) -> float:
455
+ """1 - normalised entropy over the k valid options."""
456
+ if k <= 1:
457
+ return 1.0
458
+ p = p[:k].clamp(min=1e-12)
459
+ p = p / p.sum()
460
+ h = -(p * p.log()).sum().item()
461
+ return float(max(0.0, min(1.0, 1.0 - h / math.log(k))))
preprocessor_config.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "do_convert_rgb": true,
3
+ "do_normalize": true,
4
+ "do_rescale": true,
5
+ "do_resize": true,
6
+ "image_mean": [
7
+ 0.5,
8
+ 0.5,
9
+ 0.5
10
+ ],
11
+ "image_processor_type": "Qwen2VLImageProcessor",
12
+ "image_std": [
13
+ 0.5,
14
+ 0.5,
15
+ 0.5
16
+ ],
17
+ "merge_size": 2,
18
+ "patch_size": 16,
19
+ "resample": 3,
20
+ "rescale_factor": 0.00392156862745098,
21
+ "size": {
22
+ "longest_edge": 16777216,
23
+ "shortest_edge": 65536
24
+ },
25
+ "temporal_patch_size": 2
26
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:348e2df3e5578d617b0556a58289dfa78d0f3c50b604f1a0f6c0b0f2d670671a
3
+ size 20039618
tokenizer_config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "audio_bos_token": "<|audio_start|>",
4
+ "audio_eos_token": "<|audio_end|>",
5
+ "audio_token": "<|audio_pad|>",
6
+ "backend": "tokenizers",
7
+ "bos_token": null,
8
+ "clean_up_tokenization_spaces": false,
9
+ "eos_token": "<|endoftext|>",
10
+ "errors": "replace",
11
+ "image_token": "<|image_pad|>",
12
+ "is_local": true,
13
+ "local_files_only": false,
14
+ "model_max_length": 262144,
15
+ "model_specific_special_tokens": {
16
+ "audio_bos_token": "<|audio_start|>",
17
+ "audio_eos_token": "<|audio_end|>",
18
+ "audio_token": "<|audio_pad|>",
19
+ "image_token": "<|image_pad|>",
20
+ "video_token": "<|video_pad|>",
21
+ "vision_bos_token": "<|vision_start|>",
22
+ "vision_eos_token": "<|vision_end|>"
23
+ },
24
+ "pad_token": "<|endoftext|>",
25
+ "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
26
+ "split_special_tokens": false,
27
+ "tokenizer_class": "Qwen2Tokenizer",
28
+ "unk_token": null,
29
+ "video_token": "<|video_pad|>",
30
+ "vision_bos_token": "<|vision_start|>",
31
+ "vision_eos_token": "<|vision_end|>"
32
+ }
types.py ADDED
@@ -0,0 +1,205 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Request / response types.
2
+
3
+ Field names deliberately mirror the TypeSafe `POST /v1/systemone` contract so that
4
+ code written against the TypeSafe SDK can be pointed at OpenThai-SystemOne unchanged:
5
+
6
+ state : str | dict | list -- the thing to judge
7
+ questions : {id: Choice|Score|Noul} -- typed questions
8
+ answers : {id: ChoiceAnswer|ScoreAnswer|NoulAnswer}
9
+ """
10
+ from __future__ import annotations
11
+
12
+ from typing import Any, Dict, List, Literal, Optional, Union
13
+
14
+ from pydantic import BaseModel, Field, field_validator, model_validator
15
+
16
+ MAX_OPTIONS = 255 # single-stage cardinality limit (slots 0..254); slot 255 = abstain
17
+ MIN_SCORE_LEVELS = 2
18
+ MAX_SCORE_LEVELS = 10
19
+
20
+
21
+ class Noul(BaseModel):
22
+ """A yes/no question. Returns p(yes)."""
23
+
24
+ type: Literal["noul"] = "noul"
25
+ instructions: str
26
+ criteria: Optional[Dict[str, Optional[str]]] = None # {"true": "...", "false": "..."}
27
+
28
+ @field_validator("criteria")
29
+ @classmethod
30
+ def _check_criteria(cls, v):
31
+ if v is None:
32
+ return v
33
+ extra = set(v) - {"true", "false"}
34
+ if extra:
35
+ raise ValueError(f"noul criteria keys must be 'true'/'false', got {sorted(extra)}")
36
+ return v
37
+
38
+
39
+ class Choice(BaseModel):
40
+ """Pick one option. `criteria` maps option name -> description (or null)."""
41
+
42
+ type: Literal["choice"] = "choice"
43
+ instructions: str
44
+ criteria: Dict[str, Optional[str]]
45
+
46
+ @field_validator("criteria")
47
+ @classmethod
48
+ def _check_criteria(cls, v):
49
+ if len(v) < 1:
50
+ raise ValueError("choice needs at least one option")
51
+ if len(v) > MAX_OPTIONS:
52
+ raise ValueError(f"choice supports at most {MAX_OPTIONS} options in one stage")
53
+ for k in v:
54
+ if not str(k).strip():
55
+ raise ValueError("option names must be non-empty")
56
+ return v
57
+
58
+
59
+ class Score(BaseModel):
60
+ """Rate the state against ordered levels; `criteria[i]` describes level i (low -> high)."""
61
+
62
+ type: Literal["score"] = "score"
63
+ instructions: str
64
+ criteria: List[str]
65
+
66
+ @field_validator("criteria")
67
+ @classmethod
68
+ def _check_criteria(cls, v):
69
+ if not (MIN_SCORE_LEVELS <= len(v) <= MAX_SCORE_LEVELS):
70
+ raise ValueError(f"score needs {MIN_SCORE_LEVELS}..{MAX_SCORE_LEVELS} levels")
71
+ return v
72
+
73
+
74
+ class Point(BaseModel):
75
+ """Vision variant: locate what `instructions` describes on image `image` -> normalised (x, y). Answer may abstain."""
76
+
77
+ type: Literal["point"] = "point"
78
+ instructions: str
79
+ image: Optional[str] = None # id of one of the request's images; None = the first image
80
+
81
+
82
+ Question = Union[Noul, Choice, Score, Point]
83
+
84
+
85
+ class NoulAnswer(BaseModel):
86
+ type: Literal["noul"] = "noul"
87
+ noul: float
88
+
89
+
90
+ class ChoiceAnswer(BaseModel):
91
+ type: Literal["choice"] = "choice"
92
+ choice: str
93
+ probabilities: Dict[str, float]
94
+ confidence: float
95
+ abstain: Optional[float] = None # OpenThai extension: p(none of the options); not in TypeSafe
96
+
97
+
98
+ class ScoreAnswer(BaseModel):
99
+ type: Literal["score"] = "score"
100
+ score: float
101
+ legend: Dict[int, str]
102
+ probabilities: Dict[str, float]
103
+ confidence: float
104
+
105
+
106
+ class PointCandidate(BaseModel):
107
+ x: float
108
+ y: float
109
+ p: float
110
+ bbox: Optional[List[float]] = None # normalised [x0, y0, x1, y1] of the token cluster
111
+
112
+
113
+ class PointAnswer(BaseModel):
114
+ type: Literal["point"] = "point"
115
+ x: float
116
+ y: float
117
+ confidence: float # probability mass of the winning cluster
118
+ candidates: List[PointCandidate] = []
119
+ abstain: Optional[float] = None # p(target not on the image)
120
+ image: Optional[str] = None
121
+
122
+
123
+ Answer = Union[NoulAnswer, ChoiceAnswer, ScoreAnswer, PointAnswer]
124
+
125
+
126
+ class ImageInput(BaseModel):
127
+ """One image attached to the state (vision variant). Give exactly one of data / url / path."""
128
+
129
+ id: str = "img"
130
+ data: Optional[str] = None # data URI or base64
131
+ url: Optional[str] = None
132
+ path: Optional[str] = None # local file (SDK use only; the server ignores it)
133
+ role: Optional[str] = None # screenshot | frame | photo | document | ...
134
+ detail: Literal["auto", "high"] = "auto"
135
+
136
+ @model_validator(mode="after")
137
+ def _one_source(self):
138
+ if sum(v is not None for v in (self.data, self.url, self.path)) != 1:
139
+ raise ValueError("image needs exactly one of data / url / path")
140
+ return self
141
+
142
+ @property
143
+ def source(self) -> str:
144
+ return self.data or self.url or self.path # type: ignore[return-value]
145
+
146
+
147
+ class Usage(BaseModel):
148
+ input_tokens: int
149
+ output_tokens: int = 0
150
+ permutations: int = 1 # OpenThai extension: number of option orders averaged (order-invariant mode)
151
+
152
+
153
+ class SystemOneRequest(BaseModel):
154
+ state: Union[str, Dict[str, Any], List[Any]]
155
+ model: str = "openthai-systemone"
156
+ questions: Dict[str, Question] = Field(discriminator=None)
157
+ # OpenThai extensions. order_invariant=True averages the answer over several option orders (removes position
158
+ # bias, ~2x latency); None = automatic (on for choice questions with > 10 options); permutations overrides the count.
159
+ order_invariant: Optional[bool] = None
160
+ permutations: Optional[int] = Field(default=None, ge=1, le=32)
161
+ # vision variant: up to 4 images; text-only models reject requests that carry images or point questions.
162
+ images: Optional[List[ImageInput]] = None
163
+
164
+ @model_validator(mode="after")
165
+ def _non_empty(self):
166
+ if not self.questions:
167
+ raise ValueError("at least one question is required")
168
+ if self.images is not None:
169
+ if len(self.images) > 4:
170
+ raise ValueError("at most 4 images per request")
171
+ ids = [im.id for im in self.images]
172
+ if len(set(ids)) != len(ids):
173
+ raise ValueError("image ids must be unique")
174
+ for im in self.images:
175
+ if im.path is not None:
176
+ raise ValueError("image 'path' is not accepted over the API; send data or url")
177
+ for qid, q in self.questions.items():
178
+ if isinstance(q, Point):
179
+ if not self.images:
180
+ raise ValueError(f"point question {qid!r} needs at least one image")
181
+ if q.image is not None and q.image not in {im.id for im in self.images}:
182
+ raise ValueError(f"point question {qid!r} references unknown image {q.image!r}")
183
+ return self
184
+
185
+
186
+ class SystemOneResponse(BaseModel):
187
+ model: str
188
+ answers: Dict[str, Answer]
189
+ usage: Usage
190
+
191
+
192
+ def parse_question(obj: Union[Question, Dict[str, Any]]) -> Question:
193
+ """Accept a dict (raw JSON) or an already-typed question."""
194
+ if isinstance(obj, (Noul, Choice, Score, Point)):
195
+ return obj
196
+ t = obj.get("type")
197
+ if t == "noul":
198
+ return Noul(**obj)
199
+ if t == "choice":
200
+ return Choice(**obj)
201
+ if t == "score":
202
+ return Score(**obj)
203
+ if t == "point":
204
+ return Point(**obj)
205
+ raise ValueError(f"unknown question type: {t!r}")