Aikimi commited on
Commit
effa534
·
verified ·
1 Parent(s): 6a1cddf

Add files using upload-large-folder tool

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
LICENSE ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ Apache License
3
+ Version 2.0, January 2004
4
+ http://www.apache.org/licenses/
5
+
6
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
7
+
8
+ 1. Definitions.
9
+
10
+ "License" shall mean the terms and conditions for use, reproduction,
11
+ and distribution as defined by Sections 1 through 9 of this document.
12
+
13
+ "Licensor" shall mean the copyright owner or entity authorized by
14
+ the copyright owner that is granting the License.
15
+
16
+ "Legal Entity" shall mean the union of the acting entity and all
17
+ other entities that control, are controlled by, or are under common
18
+ control with that entity. For the purposes of this definition,
19
+ "control" means (i) the power, direct or indirect, to cause the
20
+ direction or management of such entity, whether by contract or
21
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
22
+ outstanding shares, or (iii) beneficial ownership of such entity.
23
+
24
+ "You" (or "Your") shall mean an individual or Legal Entity
25
+ exercising permissions granted by this License.
26
+
27
+ "Source" form shall mean the preferred form for making modifications,
28
+ including but not limited to software source code, documentation
29
+ source, and configuration files.
30
+
31
+ "Object" form shall mean any form resulting from mechanical
32
+ transformation or translation of a Source form, including but
33
+ not limited to compiled object code, generated documentation,
34
+ and conversions to other media types.
35
+
36
+ "Work" shall mean the work of authorship, whether in Source or
37
+ Object form, made available under the License, as indicated by a
38
+ copyright notice that is included in or attached to the work
39
+ (an example is provided in the Appendix below).
40
+
41
+ "Derivative Works" shall mean any work, whether in Source or Object
42
+ form, that is based on (or derived from) the Work and for which the
43
+ editorial revisions, annotations, elaborations, or other modifications
44
+ represent, as a whole, an original work of authorship. For the purposes
45
+ of this License, Derivative Works shall not include works that remain
46
+ separable from, or merely link (or bind by name) to the interfaces of,
47
+ the Work and Derivative Works thereof.
48
+
49
+ "Contribution" shall mean any work of authorship, including
50
+ the original version of the Work and any modifications or additions
51
+ to that Work or Derivative Works thereof, that is intentionally
52
+ submitted to Licensor for inclusion in the Work by the copyright owner
53
+ or by an individual or Legal Entity authorized to submit on behalf of
54
+ the copyright owner. For the purposes of this definition, "submitted"
55
+ means any form of electronic, verbal, or written communication sent
56
+ to the Licensor or its representatives, including but not limited to
57
+ communication on electronic mailing lists, source code control systems,
58
+ and issue tracking systems that are managed by, or on behalf of, the
59
+ Licensor for the purpose of discussing and improving the Work, but
60
+ excluding communication that is conspicuously marked or otherwise
61
+ designated in writing by the copyright owner as "Not a Contribution."
62
+
63
+ "Contributor" shall mean Licensor and any individual or Legal Entity
64
+ on behalf of whom a Contribution has been received by Licensor and
65
+ subsequently incorporated within the Work.
66
+
67
+ 2. Grant of Copyright License. Subject to the terms and conditions of
68
+ this License, each Contributor hereby grants to You a perpetual,
69
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
70
+ copyright license to reproduce, prepare Derivative Works of,
71
+ publicly display, publicly perform, sublicense, and distribute the
72
+ Work and such Derivative Works in Source or Object form.
73
+
74
+ 3. Grant of Patent License. Subject to the terms and conditions of
75
+ this License, each Contributor hereby grants to You a perpetual,
76
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
77
+ (except as stated in this section) patent license to make, have made,
78
+ use, offer to sell, sell, import, and otherwise transfer the Work,
79
+ where such license applies only to those patent claims licensable
80
+ by such Contributor that are necessarily infringed by their
81
+ Contribution(s) alone or by combination of their Contribution(s)
82
+ with the Work to which such Contribution(s) was submitted. If You
83
+ institute patent litigation against any entity (including a
84
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
85
+ or a Contribution incorporated within the Work constitutes direct
86
+ or contributory patent infringement, then any patent licenses
87
+ granted to You under this License for that Work shall terminate
88
+ as of the date such litigation is filed.
89
+
90
+ 4. Redistribution. You may reproduce and distribute copies of the
91
+ Work or Derivative Works thereof in any medium, with or without
92
+ modifications, and in Source or Object form, provided that You
93
+ meet the following conditions:
94
+
95
+ (a) You must give any other recipients of the Work or
96
+ Derivative Works a copy of this License; and
97
+
98
+ (b) You must cause any modified files to carry prominent notices
99
+ stating that You changed the files; and
100
+
101
+ (c) You must retain, in the Source form of any Derivative Works
102
+ that You distribute, all copyright, patent, trademark, and
103
+ attribution notices from the Source form of the Work,
104
+ excluding those notices that do not pertain to any part of
105
+ the Derivative Works; and
106
+
107
+ (d) If the Work includes a "NOTICE" text file as part of its
108
+ distribution, then any Derivative Works that You distribute must
109
+ include a readable copy of the attribution notices contained
110
+ within such NOTICE file, excluding those notices that do not
111
+ pertain to any part of the Derivative Works, in at least one
112
+ of the following places: within a NOTICE text file distributed
113
+ as part of the Derivative Works; within the Source form or
114
+ documentation, if provided along with the Derivative Works; or,
115
+ within a display generated by the Derivative Works, if and
116
+ wherever such third-party notices normally appear. The contents
117
+ of the NOTICE file are for informational purposes only and
118
+ do not modify the License. You may add Your own attribution
119
+ notices within Derivative Works that You distribute, alongside
120
+ or as an addendum to the NOTICE text from the Work, provided
121
+ that such additional attribution notices cannot be construed
122
+ as modifying the License.
123
+
124
+ You may add Your own copyright statement to Your modifications and
125
+ may provide additional or different license terms and conditions
126
+ for use, reproduction, or distribution of Your modifications, or
127
+ for any such Derivative Works as a whole, provided Your use,
128
+ reproduction, and distribution of the Work otherwise complies with
129
+ the conditions stated in this License.
130
+
131
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
132
+ any Contribution intentionally submitted for inclusion in the Work
133
+ by You to the Licensor shall be under the terms and conditions of
134
+ this License, without any additional terms or conditions.
135
+ Notwithstanding the above, nothing herein shall supersede or modify
136
+ the terms of any separate license agreement you may have executed
137
+ with Licensor regarding such Contributions.
138
+
139
+ 6. Trademarks. This License does not grant permission to use the trade
140
+ names, trademarks, service marks, or product names of the Licensor,
141
+ except as required for reasonable and customary use in describing the
142
+ origin of the Work and reproducing the content of the NOTICE file.
143
+
144
+ 7. Disclaimer of Warranty. Unless required by applicable law or
145
+ agreed to in writing, Licensor provides the Work (and each
146
+ Contributor provides its Contributions) on an "AS IS" BASIS,
147
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148
+ implied, including, without limitation, any warranties or conditions
149
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150
+ PARTICULAR PURPOSE. You are solely responsible for determining the
151
+ appropriateness of using or redistributing the Work and assume any
152
+ risks associated with Your exercise of permissions under this License.
153
+
154
+ 8. Limitation of Liability. In no event and under no legal theory,
155
+ whether in tort (including negligence), contract, or otherwise,
156
+ unless required by applicable law (such as deliberate and grossly
157
+ negligent acts) or agreed to in writing, shall any Contributor be
158
+ liable to You for damages, including any direct, indirect, special,
159
+ incidental, or consequential damages of any character arising as a
160
+ result of this License or out of the use or inability to use the
161
+ Work (including but not limited to damages for loss of goodwill,
162
+ work stoppage, computer failure or malfunction, or any and all
163
+ other commercial damages or losses), even if such Contributor
164
+ has been advised of the possibility of such damages.
165
+
166
+ 9. Accepting Warranty or Additional Liability. While redistributing
167
+ the Work or Derivative Works thereof, You may choose to offer,
168
+ and charge a fee for, acceptance of support, warranty, indemnity,
169
+ or other liability obligations and/or rights consistent with this
170
+ License. However, in accepting such obligations, You may act only
171
+ on Your own behalf and on Your sole responsibility, not on behalf
172
+ of any other Contributor, and only if You agree to indemnify,
173
+ defend, and hold each Contributor harmless for any liability
174
+ incurred by, or claims asserted against, such Contributor by reason
175
+ of your accepting any such warranty or additional liability.
176
+
177
+ END OF TERMS AND CONDITIONS
178
+
179
+ APPENDIX: How to apply the Apache License to your work.
180
+
181
+ To apply the Apache License to your work, attach the following
182
+ boilerplate notice, with the fields enclosed by brackets "[]"
183
+ replaced with your own identifying information. (Don't include
184
+ the brackets!) The text should be enclosed in the appropriate
185
+ comment syntax for the file format. We also recommend that a
186
+ file or class name and description of purpose be included on the
187
+ same "printed page" as the copyright notice for easier
188
+ identification within third-party archives.
189
+
190
+ Copyright 2026 Alibaba Cloud
191
+
192
+ Licensed under the Apache License, Version 2.0 (the "License");
193
+ you may not use this file except in compliance with the License.
194
+ You may obtain a copy of the License at
195
+
196
+ http://www.apache.org/licenses/LICENSE-2.0
197
+
198
+ Unless required by applicable law or agreed to in writing, software
199
+ distributed under the License is distributed on an "AS IS" BASIS,
200
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201
+ See the License for the specific language governing permissions and
202
+ limitations under the License.
README.md ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: Cloudflare/clef-flash
4
+ base_model_relation: quantized
5
+ library_name: transformers
6
+ tags:
7
+ - clef
8
+ - bitsandbytes
9
+ - image-classification
10
+ - quantized
11
+ ---
12
+ # Clef-Flash 9B INT8
13
+
14
+ Quantized from [Cloudflare/clef-flash](https://huggingface.co/Cloudflare/clef-flash/tree/17f0b0ad64efb65d273590632833508766b2aae6)
15
+ at `17f0b0ad64efb65d273590632833508766b2aae6`. The original model, schema encoder and joint head are
16
+ by Cloudflare. This derived release packages INT8 decoder weights with the
17
+ original BF16 vision encoder, joint head and dense input/output vocabulary.
18
+ It includes every inference asset; the original BF16 checkpoint is not required.
19
+ The output vocabulary is in `lm-head.safetensors` and indexed separately.
20
+
21
+ Clef performs a single forward pass over a schema of choice, score and noul
22
+ (boolean) questions. It does not generate free-form text. Probabilities are
23
+ distributions over the supplied alternatives, not calibrated accuracy estimates.
24
+
25
+ ## Usage
26
+
27
+ Use Forge Neo's Clef setup and GUI, or this repository's standalone CLI:
28
+
29
+ ```powershell
30
+ python -m pip install torch==2.13.0 torchvision==0.28.0 --index-url https://download.pytorch.org/whl/cu130
31
+ python -m pip install -r requirements.txt
32
+ python inference.py --profile flash-int8 --image example.png
33
+ python inference.py --profile flash-int8 --image example.png --questions questions.json
34
+ ```
35
+
36
+ `questions.json` is the original Clef schema: each question ID maps to a `type`,
37
+ `instructions`, and, for choice/score, `criteria`. Use `--state` for context,
38
+ `--max-pixels` for image processing (default 262144), and `--max-length` for the
39
+ total input token cap (default 2048). Inputs exceeding the cap are refused.
40
+ An NVIDIA CUDA GPU with BF16 support is required. The standalone loader reads
41
+ weights locally and never downloads code during inference.
42
+
43
+ The 27B release is shared by `clef-24gb` and `clef-16gb`. The latter keeps the
44
+ input embedding on CPU. Both keep the dense output vocabulary on CPU and move
45
+ only requested rows to CUDA; decoder layers remain on CUDA. The 9B INT8 release
46
+ uses `flash-int8`. Packed lm_head replacement and whole-layer CPU offload are
47
+ not part of these profiles. The supplied loader is required for this placement.
48
+
49
+ ## Validation and limits
50
+
51
+ Verified on Windows, RTX 3090 24GB, RAM 64GB, with Transformers 5.10.2,
52
+ bitsandbytes 0.50.2, safetensors 0.8.0 and PyTorch 2.13.0+cu130. We use SDPA,
53
+ batch=1, no KV cache, and safetensors `pread` to avoid persistent Windows
54
+ copy-on-write mappings of all shards.
55
+
56
+ Four images and a text-only request were checked with the same schemas.
57
+ 27B 16GB and 24GB profiles produced identical answers and probabilities.
58
+ Peak image allocations were 13.38 GiB and 15.73 GiB respectively. The 16GB
59
+ profile passed with a 15 GiB PyTorch allocator budget on the 3090; an actual
60
+ 16GB card was not available. The 9B INT8 peak was 9.82 GiB versus 16.26 GiB
61
+ for BF16, with no choice changes in those four images and a maximum probability
62
+ difference of 0.0423. This small comparison is not a general accuracy evaluation.
63
+ INT8 was slower than BF16 on this test; its benefit here is lower memory use.
64
+ Longer inputs or more image pixels require more memory.
65
+
66
+ ## Provenance and license
67
+
68
+ Apache-2.0; see `LICENSE` and the upstream model card. The original
69
+ `joint_schema_model.py`, head, vocabulary values and processor assets are
70
+ retained. Decoder linear layers were converted with bitsandbytes and saved
71
+ using Transformers. Vision and the judgment head remain BF16. The included
72
+ `clef_runtime` and `inference.py` add CPU row gathering, profile selection,
73
+ input validation and Windows `pread` loading. `complete.json` records the
74
+ upstream commit, quantization settings, file sizes and SHA-256 hashes.
75
+ No user images, evaluation histories, local configuration or credentials are
76
+ included in this model release.
chat_template.jinja ADDED
@@ -0,0 +1,154 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- set image_count = namespace(value=0) %}
2
+ {%- set video_count = namespace(value=0) %}
3
+ {%- macro render_content(content, do_vision_count, is_system_content=false) %}
4
+ {%- if content is string %}
5
+ {{- content }}
6
+ {%- elif content is iterable and content is not mapping %}
7
+ {%- for item in content %}
8
+ {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
9
+ {%- if is_system_content %}
10
+ {{- raise_exception('System message cannot contain images.') }}
11
+ {%- endif %}
12
+ {%- if do_vision_count %}
13
+ {%- set image_count.value = image_count.value + 1 %}
14
+ {%- endif %}
15
+ {%- if add_vision_id %}
16
+ {{- 'Picture ' ~ image_count.value ~ ': ' }}
17
+ {%- endif %}
18
+ {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
19
+ {%- elif 'video' in item or item.type == 'video' %}
20
+ {%- if is_system_content %}
21
+ {{- raise_exception('System message cannot contain videos.') }}
22
+ {%- endif %}
23
+ {%- if do_vision_count %}
24
+ {%- set video_count.value = video_count.value + 1 %}
25
+ {%- endif %}
26
+ {%- if add_vision_id %}
27
+ {{- 'Video ' ~ video_count.value ~ ': ' }}
28
+ {%- endif %}
29
+ {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
30
+ {%- elif 'text' in item %}
31
+ {{- item.text }}
32
+ {%- else %}
33
+ {{- raise_exception('Unexpected item type in content.') }}
34
+ {%- endif %}
35
+ {%- endfor %}
36
+ {%- elif content is none or content is undefined %}
37
+ {{- '' }}
38
+ {%- else %}
39
+ {{- raise_exception('Unexpected content type.') }}
40
+ {%- endif %}
41
+ {%- endmacro %}
42
+ {%- if not messages %}
43
+ {{- raise_exception('No messages provided.') }}
44
+ {%- endif %}
45
+ {%- if tools and tools is iterable and tools is not mapping %}
46
+ {{- '<|im_start|>system\n' }}
47
+ {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
48
+ {%- for tool in tools %}
49
+ {{- "\n" }}
50
+ {{- tool | tojson }}
51
+ {%- endfor %}
52
+ {{- "\n</tools>" }}
53
+ {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
54
+ {%- if messages[0].role == 'system' %}
55
+ {%- set content = render_content(messages[0].content, false, true)|trim %}
56
+ {%- if content %}
57
+ {{- '\n\n' + content }}
58
+ {%- endif %}
59
+ {%- endif %}
60
+ {{- '<|im_end|>\n' }}
61
+ {%- else %}
62
+ {%- if messages[0].role == 'system' %}
63
+ {%- set content = render_content(messages[0].content, false, true)|trim %}
64
+ {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
65
+ {%- endif %}
66
+ {%- endif %}
67
+ {%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
68
+ {%- for message in messages[::-1] %}
69
+ {%- set index = (messages|length - 1) - loop.index0 %}
70
+ {%- if ns.multi_step_tool and message.role == "user" %}
71
+ {%- set content = render_content(message.content, false)|trim %}
72
+ {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
73
+ {%- set ns.multi_step_tool = false %}
74
+ {%- set ns.last_query_index = index %}
75
+ {%- endif %}
76
+ {%- endif %}
77
+ {%- endfor %}
78
+ {%- if ns.multi_step_tool %}
79
+ {{- raise_exception('No user query found in messages.') }}
80
+ {%- endif %}
81
+ {%- for message in messages %}
82
+ {%- set content = render_content(message.content, true)|trim %}
83
+ {%- if message.role == "system" %}
84
+ {%- if not loop.first %}
85
+ {{- raise_exception('System message must be at the beginning.') }}
86
+ {%- endif %}
87
+ {%- elif message.role == "user" %}
88
+ {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
89
+ {%- elif message.role == "assistant" %}
90
+ {%- set reasoning_content = '' %}
91
+ {%- if message.reasoning_content is string %}
92
+ {%- set reasoning_content = message.reasoning_content %}
93
+ {%- else %}
94
+ {%- if '</think>' in content %}
95
+ {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
96
+ {%- set content = content.split('</think>')[-1].lstrip('\n') %}
97
+ {%- endif %}
98
+ {%- endif %}
99
+ {%- set reasoning_content = reasoning_content|trim %}
100
+ {%- if loop.index0 > ns.last_query_index %}
101
+ {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
102
+ {%- else %}
103
+ {{- '<|im_start|>' + message.role + '\n' + content }}
104
+ {%- endif %}
105
+ {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
106
+ {%- for tool_call in message.tool_calls %}
107
+ {%- if tool_call.function is defined %}
108
+ {%- set tool_call = tool_call.function %}
109
+ {%- endif %}
110
+ {%- if loop.first %}
111
+ {%- if content|trim %}
112
+ {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
113
+ {%- else %}
114
+ {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
115
+ {%- endif %}
116
+ {%- else %}
117
+ {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
118
+ {%- endif %}
119
+ {%- if tool_call.arguments is defined %}
120
+ {%- for args_name, args_value in tool_call.arguments|items %}
121
+ {{- '<parameter=' + args_name + '>\n' }}
122
+ {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
123
+ {{- args_value }}
124
+ {{- '\n</parameter>\n' }}
125
+ {%- endfor %}
126
+ {%- endif %}
127
+ {{- '</function>\n</tool_call>' }}
128
+ {%- endfor %}
129
+ {%- endif %}
130
+ {{- '<|im_end|>\n' }}
131
+ {%- elif message.role == "tool" %}
132
+ {%- if loop.previtem and loop.previtem.role != "tool" %}
133
+ {{- '<|im_start|>user' }}
134
+ {%- endif %}
135
+ {{- '\n<tool_response>\n' }}
136
+ {{- content }}
137
+ {{- '\n</tool_response>' }}
138
+ {%- if not loop.last and loop.nextitem.role != "tool" %}
139
+ {{- '<|im_end|>\n' }}
140
+ {%- elif loop.last %}
141
+ {{- '<|im_end|>\n' }}
142
+ {%- endif %}
143
+ {%- else %}
144
+ {{- raise_exception('Unexpected message role.') }}
145
+ {%- endif %}
146
+ {%- endfor %}
147
+ {%- if add_generation_prompt %}
148
+ {{- '<|im_start|>assistant\n' }}
149
+ {%- if enable_thinking is defined and enable_thinking is false %}
150
+ {{- '<think>\n\n</think>\n\n' }}
151
+ {%- else %}
152
+ {{- '<think>\n' }}
153
+ {%- endif %}
154
+ {%- endif %}
clef_runtime/__init__.py ADDED
@@ -0,0 +1 @@
 
 
1
+ """Local Clef decision models and the shared Forge Studio interface."""
clef_runtime/bundle.py ADDED
@@ -0,0 +1,78 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Validate complete, self-contained quantized releases without importing torch."""
2
+
3
+ from __future__ import annotations
4
+
5
+ import json
6
+ from pathlib import Path
7
+
8
+ from .cache import cache_directory, identity
9
+ from .core import PROFILES, ClefError, sha256, source_manifest
10
+
11
+ SUPPORT_FILES = (
12
+ "joint_head.safetensors",
13
+ "joint_head_config.json",
14
+ "joint_schema_model.py",
15
+ "tokenizer.json",
16
+ "tokenizer_config.json",
17
+ "processor_config.json",
18
+ "chat_template.jinja",
19
+ "LICENSE",
20
+ )
21
+ PORTABLE_MODULES = ("__init__.py", "core.py", "cache.py", "bundle.py", "runtime.py")
22
+ REQUIRED_FILES = {
23
+ *SUPPORT_FILES,
24
+ "config.json",
25
+ "model.safetensors.index.json",
26
+ "lm-head.safetensors",
27
+ "README.md",
28
+ "inference.py",
29
+ "requirements.txt",
30
+ *("clef_runtime/" + name for name in PORTABLE_MODULES),
31
+ }
32
+
33
+
34
+ def bundle_directory(root, profile):
35
+ converted = cache_directory(root, profile)
36
+ return converted.with_name(converted.name + "-bundle") if converted is not None else None
37
+
38
+
39
+ def read_bundle(directory, profile, *, verify_hashes=False):
40
+ directory = Path(directory)
41
+ try:
42
+ manifest = json.loads((directory / "complete.json").read_text(encoding="utf-8"))
43
+ expected = {**identity(profile), "bundle_format": 1}
44
+ if any(manifest.get(key) != value for key, value in expected.items()):
45
+ raise ClefError("Clef量子化配布の版・精度が一致しません。セットアップを再実行してください。")
46
+ files = manifest["files"]
47
+ names = {item["path"] for item in files}
48
+ if not REQUIRED_FILES <= names or len(names) != len(files):
49
+ raise ClefError("Clef量子化配布の構成が不足しています。セットアップを再実行してください。")
50
+ for item in files:
51
+ path = directory / item["path"]
52
+ if (
53
+ not path.resolve().is_relative_to(directory.resolve())
54
+ or not path.is_file()
55
+ or path.stat().st_size != item["size"]
56
+ ):
57
+ raise ClefError(f"Clef量子化配布が不足・変更されています: {item['path']}")
58
+ if (verify_hashes or path.suffix == ".py") and sha256(path) != item["sha256"]:
59
+ raise ClefError(f"ClefのSHA-256が一致しません: {item['path']}")
60
+ index = json.loads((directory / "model.safetensors.index.json").read_text(encoding="utf-8"))["weight_map"]
61
+ if (
62
+ index.get("lm_head.weight") != "lm-head.safetensors"
63
+ or "language_model.embed_tokens.weight" not in index
64
+ or not set(index.values()) <= names
65
+ ):
66
+ raise ClefError("Clef量子化配布の語彙・本体の構成が一致しません。")
67
+ return directory, manifest
68
+ except ClefError:
69
+ raise
70
+ except (OSError, ValueError, KeyError, TypeError) as exc:
71
+ raise ClefError("Clef量子化モデル未導入です。aikimi-clef-setup.batを実行してください。") from exc
72
+
73
+
74
+ def bundle_manifest(root, profile, *, verify_hashes=False):
75
+ directory = bundle_directory(root, profile)
76
+ if directory is None:
77
+ return source_manifest(root, PROFILES[profile]["model"], verify_hashes=verify_hashes)
78
+ return read_bundle(directory, profile, verify_hashes=verify_hashes)
clef_runtime/cache.py ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Storage location and fixed conversion provenance shared by release tools."""
2
+
3
+ from __future__ import annotations
4
+
5
+ import json
6
+ from pathlib import Path
7
+
8
+ from .core import MODELS, PROFILES
9
+
10
+
11
+ def cache_directory(root, profile):
12
+ settings = PROFILES[profile]
13
+ if settings["precision"] == "bf16":
14
+ return None
15
+ root = Path(root)
16
+ try:
17
+ storage = json.loads((root / "storage.json").read_text(encoding="utf-8"))
18
+ except FileNotFoundError:
19
+ base = root / "quantized"
20
+ except (OSError, ValueError) as exc:
21
+ raise ValueError("Clefのstorage.jsonを読み込めません。--cache-dirで保存先を設定し直してください。") from exc
22
+ else:
23
+ destination = storage.get("quantized") if isinstance(storage, dict) else None
24
+ if not isinstance(destination, str) or not Path(destination).is_absolute():
25
+ raise ValueError("Clefの量子化キャッシュ保存先は絶対パスで指定してください。")
26
+ base = Path(destination)
27
+ return base / f"{settings['model']}-{settings['precision']}"
28
+
29
+
30
+ def identity(profile):
31
+ settings = PROFILES[profile]
32
+ return {
33
+ **MODELS[settings["model"]],
34
+ "precision": settings["precision"],
35
+ "format": 1,
36
+ "transformers": "5.10.2",
37
+ "bitsandbytes": "0.50.2",
38
+ "vision": "bf16",
39
+ }
clef_runtime/core.py ADDED
@@ -0,0 +1,339 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Small, torch-free contracts for images, schemas and immutable decision runs."""
2
+
3
+ from __future__ import annotations
4
+
5
+ import copy
6
+ import csv
7
+ import hashlib
8
+ import io
9
+ import json
10
+ import math
11
+ import os
12
+ from pathlib import Path
13
+
14
+ ROOT = Path(__file__).resolve().parents[2]
15
+ RUNTIME = ROOT / "models" / "Clef"
16
+ OUTPUTS = ROOT / "outputs" / "clef"
17
+ MODELS = {
18
+ "clef-flash": {"repo": "Cloudflare/clef-flash", "revision": "17f0b0ad64efb65d273590632833508766b2aae6"},
19
+ "clef": {"repo": "Cloudflare/clef", "revision": "2f3de3dd85f379784083b0814d997ab627200f0c"},
20
+ }
21
+ PROFILES = {
22
+ "flash-int8": {
23
+ "label": "Flash INT8 · 軽量",
24
+ "model": "clef-flash",
25
+ "precision": "int8",
26
+ "cpu_embeddings": False,
27
+ "min_vram_gib": 12,
28
+ },
29
+ "clef-24gb": {
30
+ "label": "Clef NF4 · 24GB",
31
+ "model": "clef",
32
+ "precision": "nf4",
33
+ "cpu_embeddings": False,
34
+ "min_vram_gib": 20,
35
+ },
36
+ "clef-16gb": {
37
+ "label": "Clef NF4 · 16GB",
38
+ "model": "clef",
39
+ "precision": "nf4",
40
+ "cpu_embeddings": True,
41
+ "min_vram_gib": 14,
42
+ },
43
+ "flash-bf16": {
44
+ "label": "Flash BF16 · 比較用",
45
+ "model": "clef-flash",
46
+ "precision": "bf16",
47
+ "cpu_embeddings": False,
48
+ "min_vram_gib": 20,
49
+ },
50
+ }
51
+ TEMPLATES = {
52
+ "画像の評価": {
53
+ "style": {
54
+ "type": "choice",
55
+ "instructions": "画像の主な表現形式を分類してください。",
56
+ "criteria": {
57
+ "photo": "写真または写実的な画像",
58
+ "illustration": "イラスト・アニメ・絵画",
59
+ "design": "図・ポスター・グラフィックデザイン",
60
+ },
61
+ },
62
+ "quality": {
63
+ "type": "score",
64
+ "instructions": "画像の見た目の完成度を評価してください。",
65
+ "criteria": [
66
+ "大きな破綻や欠落がある",
67
+ "目立つ問題があり修正が必要",
68
+ "おおむね整っている",
69
+ "細部まで整い完成度が高い",
70
+ ],
71
+ },
72
+ "text": {"type": "noul", "instructions": "画像に読める文字が含まれている。"},
73
+ },
74
+ "生成画像の確認": {
75
+ "artifact": {"type": "noul", "instructions": "画像に不自然な形、重複、形状の破綻が見られる。"},
76
+ "composition": {
77
+ "type": "score",
78
+ "instructions": "構図の明瞭さと主題の伝わりやすさを評価してください。",
79
+ "criteria": ["主題が不明瞭", "主題は分かるが構図に問題がある", "主題が明瞭で構図が整っている"],
80
+ },
81
+ },
82
+ "文章・JSONの分類": {
83
+ "department": {
84
+ "type": "choice",
85
+ "instructions": "問い合わせを担当するチームはどれですか。",
86
+ "criteria": {"billing": "請求や支払い", "technical": "技術的な問題や障害", "other": "その他"},
87
+ },
88
+ "outage": {"type": "noul", "instructions": "サービスが停止している。"},
89
+ },
90
+ }
91
+
92
+
93
+ class ClefError(ValueError):
94
+ """An input or installation problem to show directly in the Studio."""
95
+
96
+
97
+ def atomic_json(path, data):
98
+ path = Path(path)
99
+ path.parent.mkdir(parents=True, exist_ok=True)
100
+ temporary = path.with_name(path.name + ".tmp")
101
+ temporary.write_text(json.dumps(data, ensure_ascii=False, indent=2, allow_nan=False), encoding="utf-8")
102
+ os.replace(temporary, path)
103
+
104
+
105
+ def sha256(path):
106
+ digest = hashlib.sha256()
107
+ with Path(path).open("rb") as stream:
108
+ for chunk in iter(lambda: stream.read(8 * 1024 * 1024), b""):
109
+ digest.update(chunk)
110
+ return digest.hexdigest()
111
+
112
+
113
+ def canonical_hash(value):
114
+ return hashlib.sha256(
115
+ json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"), allow_nan=False).encode()
116
+ ).hexdigest()
117
+
118
+
119
+ def _unique(pairs):
120
+ result = {}
121
+ for key, value in pairs:
122
+ if key in result:
123
+ raise ClefError(f"IDが重複しています: {key}")
124
+ result[key] = value
125
+ return result
126
+
127
+
128
+ def parse_schema(value):
129
+ if isinstance(value, str):
130
+ if len(value) > 100_000:
131
+ raise ClefError("判断項目のJSONは100,000文字以内にしてください。")
132
+ try:
133
+ value = json.loads(value, object_pairs_hook=_unique)
134
+ except json.JSONDecodeError as exc:
135
+ raise ClefError(f"JSONの{exc.lineno}行目・{exc.colno}文字目を確認してください。") from exc
136
+ if not isinstance(value, dict) or not 1 <= len(value) <= 32:
137
+ raise ClefError("判断項目は1〜32件のオブジェクトにしてください。")
138
+ for qid, question in value.items():
139
+ if not isinstance(qid, str) or not qid.strip() or len(qid) > 80:
140
+ raise ClefError("項目IDは1〜80文字にしてください���")
141
+ if not isinstance(question, dict) or question.get("type") not in {"choice", "score", "noul"}:
142
+ raise ClefError(f"{qid}: 種類はchoice / score / noulを指定してください。")
143
+ if set(question) - {"type", "instructions", "criteria"}:
144
+ raise ClefError(f"{qid}: 未対応の設定があります。")
145
+ if not isinstance(question.get("instructions", ""), str):
146
+ raise ClefError(f"{qid}: 判断することは文章で指定してください。")
147
+ criteria, kind = question.get("criteria"), question["type"]
148
+ if kind == "choice":
149
+ if not isinstance(criteria, dict) or not 2 <= len(criteria) <= 64:
150
+ raise ClefError(f"{qid}: 選択肢はIDと説明の組を2〜64件指定してください。")
151
+ if any(
152
+ not isinstance(k, str) or not k.strip() or not isinstance(v, str) or not v.strip()
153
+ for k, v in criteria.items()
154
+ ):
155
+ raise ClefError(f"{qid}: 選択肢のIDと説明を入力してください。")
156
+ elif kind == "score":
157
+ if (
158
+ not isinstance(criteria, list)
159
+ or not 2 <= len(criteria) <= 32
160
+ or any(not isinstance(x, str) or not x.strip() for x in criteria)
161
+ ):
162
+ raise ClefError(f"{qid}: 段階の説明を低い順に2〜32件指定してください。")
163
+ elif criteria is not None and (
164
+ not isinstance(criteria, dict)
165
+ or set(criteria) - {"true", "false"}
166
+ or any(not isinstance(x, str) for x in criteria.values())
167
+ ):
168
+ raise ClefError(f"{qid}: 真偽の説明はtrue / falseで指定してください。")
169
+ return copy.deepcopy(value)
170
+
171
+
172
+ def edit_question(schema, previous, qid, kind, instructions, options):
173
+ result = copy.deepcopy(schema)
174
+ qid = (qid or "").strip()
175
+ if qid in result and qid != previous:
176
+ raise ClefError(f"IDが重複しています: {qid}")
177
+ question = {"type": kind, "instructions": (instructions or "").strip()}
178
+ lines = [x.strip() for x in (options or "").splitlines() if x.strip()]
179
+ if kind == "choice":
180
+ pairs = []
181
+ for line in lines:
182
+ if ":" not in line:
183
+ raise ClefError("選択肢は1行に「ID: 説明」の形で指定してください。")
184
+ key, description = line.split(":", 1)
185
+ pairs.append((key.strip(), description.strip()))
186
+ question["criteria"] = _unique(pairs)
187
+ elif kind == "score":
188
+ question["criteria"] = lines
189
+ if previous and previous != qid:
190
+ result.pop(previous, None)
191
+ result[qid] = question
192
+ return parse_schema(result)
193
+
194
+
195
+ def integer(value, label, lower, upper):
196
+ try:
197
+ number = int(value)
198
+ if (
199
+ isinstance(value, bool)
200
+ or not math.isfinite(float(value))
201
+ or float(value) != number
202
+ or not lower <= number <= upper
203
+ ):
204
+ raise ValueError
205
+ return number
206
+ except (ValueError, TypeError, OverflowError) as exc:
207
+ raise ClefError(f"{label}は{lower}〜{upper}の整数にしてください。") from exc
208
+
209
+
210
+ def validate_request(request):
211
+ profile = request.get("profile")
212
+ if profile not in PROFILES:
213
+ raise ClefError("実行プロファイルを選んでください。")
214
+ schema = parse_schema(request.get("questions"))
215
+ state = request.get("state", "")
216
+ if not isinstance(state, (str, dict, list, int, float, bool, type(None))):
217
+ raise ClefError("補足入力は文章またはJSONにしてください。")
218
+ if len(json.dumps(state, ensure_ascii=False)) > 60_000:
219
+ raise ClefError("補足入力は60,000文字以内にしてください。")
220
+ return {
221
+ "profile": profile,
222
+ "questions": schema,
223
+ "state": state,
224
+ "max_pixels": integer(request.get("max_pixels", 262144), "画像の最大画素数", 65536, 1048576),
225
+ "max_length": integer(request.get("max_length", 4096), "入力トークン上限", 512, 8192),
226
+ }
227
+
228
+
229
+ def fingerprint(request, items):
230
+ profile = PROFILES[request["profile"]]
231
+ return canonical_hash(
232
+ {
233
+ "request": request,
234
+ "model": MODELS[profile["model"]],
235
+ "profile": profile,
236
+ "images": [item["sha256"] for item in items],
237
+ "preprocessing": "exif-rgb-v1",
238
+ }
239
+ )
240
+
241
+
242
+ def summary(answer, question):
243
+ kind = answer["type"]
244
+ if kind == "noul":
245
+ return f"P(真) {answer['noul']:.3f}"
246
+ if kind == "choice":
247
+ choice = answer["choice"]
248
+ return f"{choice} · p={answer['confidence']:.3f}"
249
+ return f"期待値 {answer['score']:.2f} (0–{len(question['criteria']) - 1})"
250
+
251
+
252
+ def ambiguity(answer):
253
+ if answer["type"] == "noul":
254
+ return 1 - 2 * abs(answer["noul"] - 0.5)
255
+ probabilities = list(answer["probabilities"].values())
256
+ return -sum(p * math.log(p) for p in probabilities if p > 0) / math.log(len(probabilities))
257
+
258
+
259
+ def compare_runs(left, right):
260
+ for key in ("questions", "state", "max_pixels"):
261
+ if canonical_hash(left["request"].get(key)) != canonical_hash(right["request"].get(key)):
262
+ raise ClefError("比較には判断項目・補足入力・処理解像度の一致が必要です。")
263
+ other = {x["sha256"]: x for x in right["items"] if x["status"] == "done"}
264
+ rows = []
265
+ for item in left["items"]:
266
+ if item["status"] != "done" or item["sha256"] not in other:
267
+ continue
268
+ differences = {}
269
+ for qid, answer in item["answers"].items():
270
+ target = other[item["sha256"]]["answers"][qid]
271
+ if answer["type"] == "noul":
272
+ delta = target["noul"] - answer["noul"]
273
+ differences[qid] = {
274
+ "delta": delta,
275
+ "changed": (answer["noul"] >= 0.5) != (target["noul"] >= 0.5),
276
+ "max_probability_delta": abs(delta),
277
+ }
278
+ else:
279
+ delta = max(abs(p - target["probabilities"][key]) for key, p in answer["probabilities"].items())
280
+ changed = (
281
+ answer.get("choice") != target.get("choice")
282
+ if answer["type"] == "choice"
283
+ else max(answer["probabilities"], key=answer["probabilities"].get)
284
+ != max(target["probabilities"], key=target["probabilities"].get)
285
+ )
286
+ differences[qid] = {"changed": changed, "max_probability_delta": delta}
287
+ if answer["type"] == "score":
288
+ differences[qid]["delta"] = target["score"] - answer["score"]
289
+ rows.append(
290
+ {
291
+ "name": item["name"],
292
+ "sha256": item["sha256"],
293
+ "differences": differences,
294
+ "changed": any(x["changed"] for x in differences.values()),
295
+ "max_probability_delta": max(x["max_probability_delta"] for x in differences.values()),
296
+ }
297
+ )
298
+ return sorted(rows, key=lambda x: x["max_probability_delta"], reverse=True)
299
+
300
+
301
+ def export_csv(run):
302
+ schema = run["request"]["questions"]
303
+ columns = ["name", "sha256", "status", "error"] + [
304
+ col for qid in schema for col in ("answer:" + qid, "confidence:" + qid)
305
+ ]
306
+ stream = io.StringIO(newline="")
307
+ writer = csv.DictWriter(stream, fieldnames=columns)
308
+ writer.writeheader()
309
+ for item in run["items"]:
310
+ row = {key: item.get(key, "") for key in columns[:4]}
311
+ for qid, answer in item.get("answers", {}).items():
312
+ row["answer:" + qid] = answer.get("choice", answer.get("score", answer.get("noul", "")))
313
+ row["confidence:" + qid] = answer.get("confidence", "")
314
+ for key, value in row.items():
315
+ if isinstance(value, str) and value.lstrip().startswith(("=", "+", "-", "@")):
316
+ row[key] = "'" + value
317
+ writer.writerow(row)
318
+ return stream.getvalue()
319
+
320
+
321
+ def source_manifest(root, model, verify_hashes=False):
322
+ directory = Path(root) / "source" / model
323
+ try:
324
+ manifest = json.loads((directory / "release.json").read_text(encoding="utf-8"))
325
+ except (OSError, ValueError) as exc:
326
+ raise ClefError(f"{model}のBF16元重みは未導入です。aikimi-clef-setup.bat --source --model {model}を実行してください。") from exc
327
+ if manifest.get("revision") != MODELS[model]["revision"] or manifest.get("repo") != MODELS[model]["repo"]:
328
+ raise ClefError("Clefの固定リビジョンが一致しません。セットアップを再実行してください。")
329
+ for item in manifest["files"]:
330
+ path = directory / item["path"]
331
+ if (
332
+ not path.resolve().is_relative_to(directory.resolve())
333
+ or not path.is_file()
334
+ or path.stat().st_size != item["size"]
335
+ ):
336
+ raise ClefError(f"Clef配布物が不足・変更されています: {item['path']}")
337
+ if (verify_hashes or path.suffix == ".py") and sha256(path) != item["sha256"]:
338
+ raise ClefError(f"ClefのSHA-256が一致しません: {item['path']}")
339
+ return directory, manifest
clef_runtime/runtime.py ADDED
@@ -0,0 +1,203 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Keep the official joint head while moving only vocabulary rows over PCIe."""
2
+
3
+ from __future__ import annotations
4
+
5
+ import gc
6
+ import importlib.util
7
+ import json
8
+ import sys
9
+ from contextlib import contextmanager
10
+ from functools import partial
11
+
12
+ import torch
13
+
14
+ from .bundle import bundle_manifest, read_bundle
15
+ from .core import PROFILES
16
+
17
+
18
+ class LexicalRows:
19
+ def __init__(self, weight, *, device, dtype):
20
+ if weight.device.type != "cpu" or weight.is_meta:
21
+ raise ValueError("語彙行列には実体のあるCPUテンソルが必要です。")
22
+ self.weight, self.device, self.dtype = weight, device, dtype
23
+
24
+ def __getitem__(self, token_ids):
25
+ shape = (*token_ids.shape, self.weight.shape[1])
26
+ return self.weight.index_select(0, token_ids.reshape(-1).cpu()).reshape(shape).to(self.device, dtype=self.dtype)
27
+
28
+
29
+ class CPUEmbedding(torch.nn.Module):
30
+ def __init__(self, weight, *, device, dtype):
31
+ super().__init__()
32
+ self.rows = LexicalRows(weight, device=device, dtype=dtype)
33
+ self.weight = torch.nn.Parameter(weight, requires_grad=False)
34
+
35
+ def forward(self, tokens):
36
+ return self.rows[tokens]
37
+
38
+
39
+ def load_options(profile):
40
+ settings = PROFILES[profile]
41
+ result = {"device_map": {"": 0}}
42
+ if settings["cpu_embeddings"]:
43
+ result["device_map"] = {
44
+ "visual": 0,
45
+ "language_model.embed_tokens": "cpu",
46
+ "language_model.layers": 0,
47
+ "language_model.norm": 0,
48
+ "language_model.rotary_emb": 0,
49
+ }
50
+ if settings["precision"] != "bf16":
51
+ quantization = {
52
+ "llm_int8_skip_modules": ["visual", "lm_head"],
53
+ "llm_int8_enable_fp32_cpu_offload": settings["cpu_embeddings"],
54
+ }
55
+ if settings["precision"] == "int8":
56
+ quantization["load_in_8bit"] = True
57
+ else:
58
+ quantization.update(
59
+ load_in_4bit=True,
60
+ bnb_4bit_quant_type="nf4",
61
+ bnb_4bit_use_double_quant=True,
62
+ bnb_4bit_compute_dtype=torch.bfloat16,
63
+ )
64
+ result["quantization"] = quantization
65
+ return result
66
+
67
+
68
+ def encode_complete(official, processor, record, max_length):
69
+ # The release silently truncates state. Encode once without truncation and
70
+ # refuse overflow before collating or allocating decoder activations.
71
+ encoded = official.encode_record(processor.tokenizer, record, processor=processor, max_length=1_000_000)
72
+ if len(encoded.input_ids) > max_length:
73
+ raise ValueError(
74
+ f"入力は{len(encoded.input_ids)}トークンです。上限{max_length}を超えます。補足や項目を短くするか処理解像度を下げてください。"
75
+ )
76
+ return encoded
77
+
78
+
79
+ def tensor_from_source(source, key):
80
+ from safetensors import safe_open
81
+
82
+ index = json.loads((source / "model.safetensors.index.json").read_text())
83
+ with safe_open(source / index["weight_map"][key], framework="pt", device="cpu", backend="pread") as stream:
84
+ return stream.get_tensor(key)
85
+
86
+
87
+ @contextmanager
88
+ def stream_weights():
89
+ import transformers.modeling_utils as loading
90
+
91
+ # Windows reserves commit for every copy-on-write mmap, including unread
92
+ # pages. Keep TF's lazy tensor loading without retaining all shard maps.
93
+ original = loading.safe_open
94
+ loading.safe_open = partial(original, backend="pread")
95
+ try:
96
+ yield
97
+ finally:
98
+ loading.safe_open = original
99
+
100
+
101
+ class Runner:
102
+ @classmethod
103
+ def from_directory(cls, directory, profile):
104
+ return cls(None, profile, directory=directory)
105
+
106
+ def __init__(self, root, profile, *, directory=None):
107
+ from safetensors.torch import load_file
108
+ from transformers import AutoProcessor, BitsAndBytesConfig, Qwen3_5Model
109
+
110
+ if not torch.cuda.is_available() or not torch.cuda.is_bf16_supported():
111
+ raise RuntimeError("ClefにはBF16対応のNVIDIA CUDA GPUが必要です。")
112
+ self.settings, self.profile = PROFILES[profile], profile
113
+ total = torch.cuda.get_device_properties(0).total_memory / 2**30
114
+ if total < self.settings["min_vram_gib"]:
115
+ raise RuntimeError(
116
+ f"{self.settings['label']}のGPUメモリが不足しています。16GB設定またはFlashを選んでください。"
117
+ )
118
+ source, _ = read_bundle(directory, profile) if directory is not None else bundle_manifest(root, profile)
119
+ spec = importlib.util.spec_from_file_location(
120
+ "clef_release_" + self.settings["model"].replace("-", "_"), source / "joint_schema_model.py"
121
+ )
122
+ self.official = importlib.util.module_from_spec(spec)
123
+ sys.modules[spec.name] = self.official
124
+ spec.loader.exec_module(self.official)
125
+ options = load_options(profile)
126
+ saved = source if self.settings["precision"] != "bf16" else None
127
+ if "quantization" in options:
128
+ options["quantization_config"] = BitsAndBytesConfig(**options.pop("quantization"))
129
+ print(f"{self.settings['label']}: {'保存済み' if saved else '公式重み'}を読み込み中", flush=True) # noqa: T201
130
+ with stream_weights():
131
+ self.backbone, loading = Qwen3_5Model.from_pretrained(
132
+ saved or source,
133
+ dtype=torch.bfloat16,
134
+ local_files_only=True,
135
+ attn_implementation="sdpa",
136
+ output_loading_info=True,
137
+ **options,
138
+ )
139
+ expected_extra = {"lm_head.weight"}
140
+ if loading["missing_keys"] or set(loading["unexpected_keys"]) != expected_extra:
141
+ raise RuntimeError(f"Clef本体のキーが一致しません: {loading}")
142
+ self.backbone.eval()
143
+ self.backbone.config.use_cache = False
144
+ if self.settings["cpu_embeddings"]:
145
+ # Read the bundle's BF16 rows directly; CPU offload may cast to FP32.
146
+ # Discard the old embedding together with its hook. Detaching a
147
+ # hook can temporarily restore the entire vocabulary onto CUDA.
148
+ embedding = tensor_from_source(source, "language_model.embed_tokens.weight")
149
+ self.backbone.language_model.embed_tokens = CPUEmbedding(embedding, device="cuda:0", dtype=torch.bfloat16)
150
+ gc.collect()
151
+ self.lexical = LexicalRows(tensor_from_source(source, "lm_head.weight"), device="cuda:0", dtype=torch.bfloat16)
152
+ self.head = self.official.JointSchemaHead(**json.loads((source / "joint_head_config.json").read_text()))
153
+ self.head.load_state_dict(load_file(source / "joint_head.safetensors", backend="pread"), strict=True)
154
+ self.head = self.head.to(device="cuda:0", dtype=torch.bfloat16).eval()
155
+ self.processor = AutoProcessor.from_pretrained(source, local_files_only=True)
156
+ self.source = source
157
+ self.root, self.saved = root, saved
158
+
159
+ @torch.inference_mode()
160
+ def decide(self, request, image=None):
161
+ record = {
162
+ "model": self.settings["model"],
163
+ "state": request["state"],
164
+ "questions": request["questions"],
165
+ "media_kwargs": {"max_pixels": request["max_pixels"]},
166
+ }
167
+ if image is not None:
168
+ record["images"] = [image]
169
+ encoded = encode_complete(self.official, self.processor, record, request["max_length"])
170
+ batch = self.official.collate_records([encoded], self.processor.tokenizer.pad_token_id, torch.device("cuda:0"))
171
+ torch.cuda.reset_peak_memory_stats()
172
+ media = batch.get("media") or {}
173
+ model = self.backbone if media else self.backbone.language_model
174
+ outputs = model(
175
+ input_ids=batch["input_ids"],
176
+ attention_mask=batch["attention_mask"],
177
+ use_cache=False,
178
+ return_dict=True,
179
+ **media,
180
+ )
181
+ logits = self.head(
182
+ outputs.last_hidden_state, batch["input_ids"], batch["attention_mask"], batch["records"], self.lexical
183
+ )[0]
184
+ answers = {
185
+ q.question_id: self.official.systemone_answer(
186
+ request["questions"][q.question_id],
187
+ dict(zip(q.option_ids, logit.float().softmax(-1).tolist(), strict=True)),
188
+ )
189
+ for q, logit in zip(encoded.questions, logits, strict=True)
190
+ }
191
+ torch.cuda.synchronize()
192
+ return {
193
+ "model": self.settings["model"],
194
+ "answers": answers,
195
+ "usage": {"input_tokens": len(encoded.input_ids), "output_tokens": 0},
196
+ "peak_allocated_gib": torch.cuda.max_memory_allocated() / 2**30,
197
+ "peak_reserved_gib": torch.cuda.max_memory_reserved() / 2**30,
198
+ }
199
+
200
+ def close(self):
201
+ self.backbone = self.head = self.lexical = self.processor = None
202
+ gc.collect()
203
+ torch.cuda.empty_cache()
complete.json ADDED
@@ -0,0 +1,132 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "repo": "Cloudflare/clef-flash",
3
+ "revision": "17f0b0ad64efb65d273590632833508766b2aae6",
4
+ "precision": "int8",
5
+ "format": 1,
6
+ "transformers": "5.10.2",
7
+ "bitsandbytes": "0.50.2",
8
+ "vision": "bf16",
9
+ "bundle_format": 1,
10
+ "files": [
11
+ {
12
+ "path": "LICENSE",
13
+ "size": 11544,
14
+ "sha256": "bbedc3fda3305820b977265f01b8619d87570a6739de3a5582c3464840f1e57a"
15
+ },
16
+ {
17
+ "path": "README.md",
18
+ "size": 3913,
19
+ "sha256": "8c2553f158a27717a0e5128bebdb576b3e81fa089b821bc79d71a3cfde77a256"
20
+ },
21
+ {
22
+ "path": "chat_template.jinja",
23
+ "size": 7756,
24
+ "sha256": "a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715"
25
+ },
26
+ {
27
+ "path": "clef_runtime/__init__.py",
28
+ "size": 72,
29
+ "sha256": "8252a7c5618dd1324b6c4c307a0c7d574712eb5481d3e129947b99dba14775ca"
30
+ },
31
+ {
32
+ "path": "clef_runtime/bundle.py",
33
+ "size": 3307,
34
+ "sha256": "4abd7c664fbf94e7217b4554657a5ba5ea6e1fea633a43455b2c213cb9b09ba2"
35
+ },
36
+ {
37
+ "path": "clef_runtime/cache.py",
38
+ "size": 1351,
39
+ "sha256": "431f466374141ab80af41d4cc0ddca93dfc9fc74a28de8de792f5be690eed1cb"
40
+ },
41
+ {
42
+ "path": "clef_runtime/core.py",
43
+ "size": 14153,
44
+ "sha256": "15edc6a89e4a011a308bc283ac182c793c7c72227dbe06788d4f9888e377ab36"
45
+ },
46
+ {
47
+ "path": "clef_runtime/runtime.py",
48
+ "size": 8826,
49
+ "sha256": "69df25f5118190fe066199225f0c9c2b25f10b93e1e550ff50890252aafa02b6"
50
+ },
51
+ {
52
+ "path": "config.json",
53
+ "size": 3479,
54
+ "sha256": "ff6de63fbfcd26ce6d0fed3c40039ddcf2c6833c6f07a189473ec2eeb59ac884"
55
+ },
56
+ {
57
+ "path": "inference.py",
58
+ "size": 1521,
59
+ "sha256": "9667ef47476bd4e8dddaa66d7d02f45733d542e815629ebb30ec5fc7639fb135"
60
+ },
61
+ {
62
+ "path": "joint_head.safetensors",
63
+ "size": 243538016,
64
+ "sha256": "19cdcec8c81dc9212be320fff47462ab342fbc1278be4368fb3da71241cf5ba0"
65
+ },
66
+ {
67
+ "path": "joint_head_config.json",
68
+ "size": 119,
69
+ "sha256": "77efe959a38b5b17b241543e129e695f3c77465ece55a25985279bd8176279a0"
70
+ },
71
+ {
72
+ "path": "joint_schema_model.py",
73
+ "size": 23263,
74
+ "sha256": "0e304cf7c6500e8bb59bef7e2afd2c6373f82596dfb3b57d1aa93c175e2dc3a3"
75
+ },
76
+ {
77
+ "path": "lm-head.safetensors",
78
+ "size": 2034237568,
79
+ "sha256": "dd63614f1dc80dce2d83be3f3e69af1f8a0ebc8add9b8e0abf3910f563e44344"
80
+ },
81
+ {
82
+ "path": "model-00001-of-00005.safetensors",
83
+ "size": 2034237592,
84
+ "sha256": "9c5483aa32a952b3a9a5d75f3c000c9ff934d3cbbb317b228cadff89e25bcffa"
85
+ },
86
+ {
87
+ "path": "model-00002-of-00005.safetensors",
88
+ "size": 1984182833,
89
+ "sha256": "6999510bfc3c9ea6536090fb76b70bb786a0f4bd76f290ea35f35f58aa77e4f6"
90
+ },
91
+ {
92
+ "path": "model-00003-of-00005.safetensors",
93
+ "size": 1983829880,
94
+ "sha256": "efa24fbcede25d84116de286e544554513bd0400c40c66b99c32e846f5096cdd"
95
+ },
96
+ {
97
+ "path": "model-00004-of-00005.safetensors",
98
+ "size": 1983498867,
99
+ "sha256": "7a5dd174473310a090c74abe907de00da6b85c46f8a3b5bcaec20267449fbdbd"
100
+ },
101
+ {
102
+ "path": "model-00005-of-00005.safetensors",
103
+ "size": 1886989356,
104
+ "sha256": "1463c85bac6efd374bb5189d3beda19df628a786a9068b00de74dd3504f12df9"
105
+ },
106
+ {
107
+ "path": "model.safetensors.index.json",
108
+ "size": 112960,
109
+ "sha256": "493691fe586b33a34830abaf788dc940ec43fcf4913139c9ed5dd1ba069542e3"
110
+ },
111
+ {
112
+ "path": "processor_config.json",
113
+ "size": 1191,
114
+ "sha256": "d89ef49ce9cd37fbf510158e13c1ef063d9286411c1ec9049932dbe0487143b1"
115
+ },
116
+ {
117
+ "path": "requirements.txt",
118
+ "size": 194,
119
+ "sha256": "6ea662f95ec818e7f7a17742211ba0e6de4744ac61055565f01214e8a818588a"
120
+ },
121
+ {
122
+ "path": "tokenizer.json",
123
+ "size": 19989325,
124
+ "sha256": "06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
125
+ },
126
+ {
127
+ "path": "tokenizer_config.json",
128
+ "size": 1075,
129
+ "sha256": "91a08f825d370d085d692e04cf117cdd7faad7bf18e996f1e6031b6dab03db72"
130
+ }
131
+ ]
132
+ }
config.json ADDED
@@ -0,0 +1,128 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Qwen3_5Model"
4
+ ],
5
+ "dtype": "bfloat16",
6
+ "image_token_id": 248056,
7
+ "model_type": "qwen3_5",
8
+ "quantization_config": {
9
+ "_load_in_4bit": false,
10
+ "_load_in_8bit": true,
11
+ "bnb_4bit_compute_dtype": "float32",
12
+ "bnb_4bit_quant_storage": "uint8",
13
+ "bnb_4bit_quant_type": "fp4",
14
+ "bnb_4bit_use_double_quant": false,
15
+ "llm_int8_enable_fp32_cpu_offload": false,
16
+ "llm_int8_has_fp16_weight": false,
17
+ "llm_int8_skip_modules": [
18
+ "visual",
19
+ "lm_head"
20
+ ],
21
+ "llm_int8_threshold": 6.0,
22
+ "load_in_4bit": false,
23
+ "load_in_8bit": true,
24
+ "quant_method": "bitsandbytes"
25
+ },
26
+ "text_config": {
27
+ "attention_bias": false,
28
+ "attention_dropout": 0.0,
29
+ "attn_output_gate": true,
30
+ "bos_token_id": null,
31
+ "dtype": "bfloat16",
32
+ "eos_token_id": 248044,
33
+ "full_attention_interval": 4,
34
+ "head_dim": 256,
35
+ "hidden_act": "silu",
36
+ "hidden_size": 4096,
37
+ "initializer_range": 0.02,
38
+ "intermediate_size": 12288,
39
+ "layer_types": [
40
+ "linear_attention",
41
+ "linear_attention",
42
+ "linear_attention",
43
+ "full_attention",
44
+ "linear_attention",
45
+ "linear_attention",
46
+ "linear_attention",
47
+ "full_attention",
48
+ "linear_attention",
49
+ "linear_attention",
50
+ "linear_attention",
51
+ "full_attention",
52
+ "linear_attention",
53
+ "linear_attention",
54
+ "linear_attention",
55
+ "full_attention",
56
+ "linear_attention",
57
+ "linear_attention",
58
+ "linear_attention",
59
+ "full_attention",
60
+ "linear_attention",
61
+ "linear_attention",
62
+ "linear_attention",
63
+ "full_attention",
64
+ "linear_attention",
65
+ "linear_attention",
66
+ "linear_attention",
67
+ "full_attention",
68
+ "linear_attention",
69
+ "linear_attention",
70
+ "linear_attention",
71
+ "full_attention"
72
+ ],
73
+ "linear_conv_kernel_dim": 4,
74
+ "linear_key_head_dim": 128,
75
+ "linear_num_key_heads": 16,
76
+ "linear_num_value_heads": 32,
77
+ "linear_value_head_dim": 128,
78
+ "mamba_ssm_dtype": "float32",
79
+ "max_position_embeddings": 262144,
80
+ "mlp_only_layers": [],
81
+ "model_type": "qwen3_5_text",
82
+ "mtp_num_hidden_layers": 0,
83
+ "mtp_use_dedicated_embeddings": false,
84
+ "num_attention_heads": 16,
85
+ "num_hidden_layers": 32,
86
+ "num_key_value_heads": 4,
87
+ "pad_token_id": null,
88
+ "partial_rotary_factor": 0.25,
89
+ "rms_norm_eps": 1e-06,
90
+ "rope_parameters": {
91
+ "mrope_interleaved": true,
92
+ "mrope_section": [
93
+ 11,
94
+ 11,
95
+ 10
96
+ ],
97
+ "partial_rotary_factor": 0.25,
98
+ "rope_theta": 10000000,
99
+ "rope_type": "default"
100
+ },
101
+ "tie_word_embeddings": false,
102
+ "use_cache": true,
103
+ "vocab_size": 248320
104
+ },
105
+ "tie_word_embeddings": false,
106
+ "transformers_version": "5.10.2",
107
+ "use_cache": false,
108
+ "video_token_id": 248057,
109
+ "vision_config": {
110
+ "deepstack_visual_indexes": [],
111
+ "depth": 27,
112
+ "dtype": "bfloat16",
113
+ "hidden_act": "gelu_pytorch_tanh",
114
+ "hidden_size": 1152,
115
+ "in_channels": 3,
116
+ "initializer_range": 0.02,
117
+ "intermediate_size": 4304,
118
+ "model_type": "qwen3_5_vision",
119
+ "num_heads": 16,
120
+ "num_position_embeddings": 2304,
121
+ "out_hidden_size": 4096,
122
+ "patch_size": 16,
123
+ "spatial_merge_size": 2,
124
+ "temporal_patch_size": 2
125
+ },
126
+ "vision_end_token_id": 248054,
127
+ "vision_start_token_id": 248053
128
+ }
inference.py ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Run a downloaded Clef quantized bundle, without Forge or source checkpoints."""
2
+
3
+ from __future__ import annotations
4
+
5
+ import argparse
6
+ import json
7
+ from pathlib import Path
8
+
9
+
10
+ def main():
11
+ parser = argparse.ArgumentParser(description=__doc__)
12
+ parser.add_argument("--profile", choices=["flash-int8", "clef-24gb", "clef-16gb"], required=True)
13
+ parser.add_argument("--image", type=Path)
14
+ parser.add_argument("--questions", type=Path)
15
+ parser.add_argument("--state", default="添付画像を評価してください。")
16
+ parser.add_argument("--max-pixels", type=int, default=262144)
17
+ parser.add_argument("--max-length", type=int, default=2048)
18
+ args = parser.parse_args()
19
+ from clef_runtime.core import TEMPLATES, validate_request
20
+ from clef_runtime.runtime import Runner
21
+
22
+ questions = json.loads(args.questions.read_text(encoding="utf-8")) if args.questions else TEMPLATES["画像の評価"]
23
+ request = validate_request(
24
+ {
25
+ "profile": args.profile,
26
+ "questions": questions,
27
+ "state": args.state,
28
+ "max_pixels": args.max_pixels,
29
+ "max_length": args.max_length,
30
+ }
31
+ )
32
+ runner = Runner.from_directory(Path(__file__).resolve().parent, args.profile)
33
+ try:
34
+ result = runner.decide(request, str(args.image.resolve()) if args.image else None)
35
+ print(json.dumps(result, ensure_ascii=False, indent=2), flush=True) # noqa: T201
36
+ finally:
37
+ runner.close()
38
+
39
+
40
+ if __name__ == "__main__":
41
+ main()
joint_head.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:19cdcec8c81dc9212be320fff47462ab342fbc1278be4368fb3da71241cf5ba0
3
+ size 243538016
joint_head_config.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "hidden_size": 4096,
3
+ "width": 1024,
4
+ "routing_layers": 2,
5
+ "layers": 4,
6
+ "heads": 16,
7
+ "feedforward": 4096
8
+ }
joint_schema_model.py ADDED
@@ -0,0 +1,576 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Clef: a multimodal Qwen backbone with a joint schema head for typed decisions.
2
+
3
+ A record provides a ``state`` (any JSON value), optional ``images`` and ``videos``,
4
+ and ``questions``. Each question has a ``type`` (``noul``, ``choice``, or ``score``),
5
+ ``instructions``, and, for ``choice`` and ``score``, ``criteria`` describing the
6
+ allowed options. The model returns one logit per allowed option for every question.
7
+ ``systemone`` answers a Jev/SystemOne ``/v1/systemone`` request body with the same
8
+ response body.
9
+ """
10
+
11
+ from __future__ import annotations
12
+
13
+ import json
14
+ import math
15
+ from dataclasses import dataclass
16
+ from dataclasses import field as dataclass_field
17
+ from pathlib import Path
18
+ from typing import Any
19
+
20
+ import torch
21
+ import torch.nn.functional as functional
22
+
23
+
24
+ SYSTEM_PROMPT = (
25
+ "Read the complete state and schema. Decide every field jointly. Each answer "
26
+ "must be exactly one of that field's allowed options."
27
+ )
28
+ IMAGE_PLACEHOLDER = "<|vision_start|><|image_pad|><|vision_end|>"
29
+ VIDEO_PLACEHOLDER = "<|vision_start|><|video_pad|><|vision_end|>"
30
+ MEDIA_BATCH_KEYS = ("pixel_values", "image_grid_thw", "pixel_values_videos", "video_grid_thw")
31
+ MEDIA_TOKEN_KEYS = ("mm_token_type_ids",)
32
+ QUESTION_TYPES = {"noul": 0, "choice": 1, "score": 2}
33
+
34
+
35
+ def render(value: Any) -> str:
36
+ if isinstance(value, str):
37
+ return value
38
+ return json.dumps(
39
+ value,
40
+ ensure_ascii=False,
41
+ separators=(",", ":"),
42
+ sort_keys=True,
43
+ )
44
+
45
+
46
+ def question_options(question: dict[str, Any]) -> list[tuple[str, Any]]:
47
+ question_type = str(question["type"])
48
+ if question_type == "noul":
49
+ criteria = {
50
+ "true": "The proposition is true or the answer is yes.",
51
+ "false": "The proposition is false or the answer is no.",
52
+ }
53
+ criteria.update(question.get("criteria") or {})
54
+ return [(key, criteria[key]) for key in ("true", "false")]
55
+ if question_type == "choice":
56
+ return sorted((str(key), value) for key, value in question["criteria"].items())
57
+ return [(str(index), value) for index, value in enumerate(question["criteria"])]
58
+
59
+
60
+ @dataclass(frozen=True)
61
+ class EncodedQuestion:
62
+ question_id: str
63
+ question_type: int
64
+ question_span: tuple[int, int]
65
+ option_spans: tuple[tuple[int, int], ...]
66
+ option_ids: tuple[str, ...]
67
+
68
+
69
+ @dataclass(frozen=True)
70
+ class EncodedRecord:
71
+ input_ids: tuple[int, ...]
72
+ questions: tuple[EncodedQuestion, ...]
73
+ record_id: str
74
+ media: dict[str, Any] | None = dataclass_field(default=None, compare=False, repr=False)
75
+
76
+
77
+ def _tokens(tokenizer: Any, text: str) -> list[int]:
78
+ return tokenizer(text, add_special_tokens=False).input_ids
79
+
80
+
81
+ def _encode_media(processor: Any, record: dict[str, Any]) -> tuple[list[int], dict[str, Any] | None]:
82
+ images = list(record.get("images") or [])
83
+ videos = list(record.get("videos") or [])
84
+ if not images and not videos:
85
+ return [], None
86
+ if processor is None:
87
+ raise ValueError("records with images or videos require a processor")
88
+ text = IMAGE_PLACEHOLDER * len(images) + VIDEO_PLACEHOLDER * len(videos) + "\n"
89
+ encoded = processor(
90
+ text=[text],
91
+ images=images or None,
92
+ videos=videos or None,
93
+ return_tensors="pt",
94
+ **(record.get("media_kwargs") or {}),
95
+ )
96
+ media = {key: encoded[key] for key in MEDIA_BATCH_KEYS if key in encoded}
97
+ for key in MEDIA_TOKEN_KEYS:
98
+ if key in encoded:
99
+ media[key] = encoded[key][0].tolist()
100
+ return encoded["input_ids"][0].tolist(), media
101
+
102
+
103
+ def encode_record(
104
+ tokenizer: Any,
105
+ record: dict[str, Any],
106
+ max_length: int = 16384,
107
+ max_state_tokens: int | None = None,
108
+ processor: Any | None = None,
109
+ ) -> EncodedRecord:
110
+ schema_ids = _tokens(tokenizer, "\n\nSCHEMA FIELDS:\n")
111
+ questions: list[EncodedQuestion] = []
112
+ for question_index, (question_id, question) in enumerate(record["questions"].items()):
113
+ schema_ids.extend(
114
+ _tokens(
115
+ tokenizer,
116
+ f"\nFIELD {question_index + 1}\nID: {question_id}\nTYPE: {question['type']}\nINSTRUCTION: ",
117
+ )
118
+ )
119
+ question_start = len(schema_ids)
120
+ instructions = question.get("instructions")
121
+ if instructions is None or instructions == "":
122
+ instructions = str(question_id)
123
+ schema_ids.extend(_tokens(tokenizer, render(instructions)))
124
+ question_end = len(schema_ids)
125
+ schema_ids.extend(_tokens(tokenizer, "\nALLOWED OPTIONS:\n"))
126
+
127
+ option_spans: list[tuple[int, int]] = []
128
+ option_ids: list[str] = []
129
+ for option_index, (option_id, description) in enumerate(question_options(question)):
130
+ schema_ids.extend(_tokens(tokenizer, f"OPTION {option_index + 1}: "))
131
+ option_start = len(schema_ids)
132
+ semantics = {"option_id": option_id}
133
+ if description is not None:
134
+ semantics["description"] = description
135
+ schema_ids.extend(_tokens(tokenizer, render(semantics)))
136
+ option_spans.append((option_start, len(schema_ids)))
137
+ option_ids.append(option_id)
138
+ schema_ids.extend(_tokens(tokenizer, "\n"))
139
+ schema_ids.extend(_tokens(tokenizer, "END FIELD\n"))
140
+ questions.append(
141
+ EncodedQuestion(
142
+ question_id=str(question_id),
143
+ question_type=QUESTION_TYPES[str(question["type"])],
144
+ question_span=(question_start, question_end),
145
+ option_spans=tuple(option_spans),
146
+ option_ids=tuple(option_ids),
147
+ )
148
+ )
149
+
150
+ prefix_ids = _tokens(
151
+ tokenizer,
152
+ f"<|im_start|>system\n{SYSTEM_PROMPT}<|im_end|>\n<|im_start|>user\nSTATE:\n",
153
+ )
154
+ suffix_ids = _tokens(
155
+ tokenizer,
156
+ "\n<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\nJOINT SCHEMA DECISIONS:",
157
+ )
158
+ media_ids, media = _encode_media(processor, record)
159
+ if media is not None:
160
+ media["token_offset"] = len(prefix_ids)
161
+ prefix_ids = prefix_ids + media_ids
162
+ state_ids = _tokens(tokenizer, render(record["state"]))
163
+ if max_state_tokens is not None:
164
+ state_ids = state_ids[:max_state_tokens]
165
+ fixed_length = len(prefix_ids) + len(schema_ids) + len(suffix_ids)
166
+ if fixed_length > max_length:
167
+ raise ValueError(
168
+ f"schema requires {fixed_length} tokens before state; maximum is {max_length}"
169
+ )
170
+ state_ids = state_ids[: max_length - fixed_length]
171
+ schema_offset = len(prefix_ids) + len(state_ids)
172
+ shifted_questions = tuple(
173
+ EncodedQuestion(
174
+ question_id=question.question_id,
175
+ question_type=question.question_type,
176
+ question_span=(
177
+ question.question_span[0] + schema_offset,
178
+ question.question_span[1] + schema_offset,
179
+ ),
180
+ option_spans=tuple(
181
+ (start + schema_offset, end + schema_offset)
182
+ for start, end in question.option_spans
183
+ ),
184
+ option_ids=question.option_ids,
185
+ )
186
+ for question in questions
187
+ )
188
+ input_ids = tuple(prefix_ids + state_ids + schema_ids + suffix_ids)
189
+ if not input_ids or not shifted_questions:
190
+ raise ValueError("record produced no model input or questions")
191
+ return EncodedRecord(
192
+ input_ids=input_ids,
193
+ questions=shifted_questions,
194
+ record_id=str(record.get("id", "unknown")),
195
+ media=media,
196
+ )
197
+
198
+
199
+ def collate_records(
200
+ records: list[EncodedRecord],
201
+ pad_token_id: int,
202
+ device: torch.device,
203
+ ) -> dict[str, Any]:
204
+ maximum_length = max(len(record.input_ids) for record in records)
205
+ input_ids = torch.full(
206
+ (len(records), maximum_length),
207
+ pad_token_id,
208
+ dtype=torch.long,
209
+ device=device,
210
+ )
211
+ attention_mask = torch.zeros(
212
+ (len(records), maximum_length),
213
+ dtype=torch.long,
214
+ device=device,
215
+ )
216
+ for index, record in enumerate(records):
217
+ length = len(record.input_ids)
218
+ input_ids[index, :length] = torch.tensor(record.input_ids, device=device)
219
+ attention_mask[index, :length] = 1
220
+ media: dict[str, torch.Tensor] = {}
221
+ for key in MEDIA_BATCH_KEYS:
222
+ values = [record.media[key] for record in records if record.media and key in record.media]
223
+ if values:
224
+ media[key] = torch.cat(values, dim=0).to(device)
225
+ for key in MEDIA_TOKEN_KEYS:
226
+ if any(record.media and key in record.media for record in records):
227
+ token_values = torch.zeros((len(records), maximum_length), dtype=torch.long, device=device)
228
+ for index, record in enumerate(records):
229
+ if record.media and key in record.media:
230
+ offset = record.media["token_offset"]
231
+ values = torch.tensor(record.media[key], dtype=torch.long, device=device)
232
+ token_values[index, offset : offset + len(values)] = values
233
+ media[key] = token_values
234
+ return {
235
+ "input_ids": input_ids,
236
+ "attention_mask": attention_mask,
237
+ "records": records,
238
+ "media": media,
239
+ }
240
+
241
+
242
+ class EvidenceRoutingLayer(torch.nn.Module):
243
+ def __init__(
244
+ self,
245
+ width: int,
246
+ heads: int,
247
+ feedforward: int,
248
+ dropout: float = 0.0,
249
+ ) -> None:
250
+ super().__init__()
251
+ self.query_norm = torch.nn.LayerNorm(width)
252
+ self.memory_norm = torch.nn.LayerNorm(width)
253
+ self.attention = torch.nn.MultiheadAttention(
254
+ width,
255
+ heads,
256
+ dropout=dropout,
257
+ batch_first=True,
258
+ )
259
+ self.attention_dropout = torch.nn.Dropout(dropout)
260
+ self.feedforward_norm = torch.nn.LayerNorm(width)
261
+ self.feedforward = torch.nn.Sequential(
262
+ torch.nn.Linear(width, feedforward),
263
+ torch.nn.GELU(),
264
+ torch.nn.Dropout(dropout),
265
+ torch.nn.Linear(feedforward, width),
266
+ torch.nn.Dropout(dropout),
267
+ )
268
+
269
+ def forward(self, queries: torch.Tensor, memory: torch.Tensor) -> torch.Tensor:
270
+ normalized_queries = self.query_norm(queries)
271
+ routed, _ = self.attention(
272
+ normalized_queries,
273
+ self.memory_norm(memory),
274
+ self.memory_norm(memory),
275
+ need_weights=False,
276
+ )
277
+ queries = queries + self.attention_dropout(routed)
278
+ return queries + self.feedforward(self.feedforward_norm(queries))
279
+
280
+
281
+ class JointSchemaHead(torch.nn.Module):
282
+ def __init__(
283
+ self,
284
+ hidden_size: int,
285
+ width: int,
286
+ routing_layers: int,
287
+ layers: int,
288
+ heads: int,
289
+ feedforward: int,
290
+ dropout: float = 0.0,
291
+ ) -> None:
292
+ super().__init__()
293
+ self.hidden_norm = torch.nn.LayerNorm(hidden_size)
294
+ self.memory_projection = torch.nn.Linear(hidden_size, width, bias=False)
295
+ self.question_projection = torch.nn.Linear(hidden_size, width, bias=False)
296
+ self.option_question_projection = torch.nn.Linear(hidden_size, width, bias=False)
297
+ self.global_projection = torch.nn.Linear(hidden_size, width, bias=False)
298
+ self.option_context_projection = torch.nn.Linear(hidden_size, width, bias=False)
299
+ self.option_lexical_projection = torch.nn.Linear(hidden_size, width, bias=False)
300
+ self.type_embedding = torch.nn.Embedding(3, width)
301
+ self.evidence_layers = torch.nn.ModuleList(
302
+ [
303
+ EvidenceRoutingLayer(
304
+ width=width,
305
+ heads=heads,
306
+ feedforward=feedforward,
307
+ dropout=dropout,
308
+ )
309
+ for _ in range(routing_layers)
310
+ ]
311
+ )
312
+ self.option_summary_norm = torch.nn.LayerNorm(width)
313
+ self.layers = torch.nn.ModuleList(
314
+ [
315
+ torch.nn.TransformerDecoderLayer(
316
+ d_model=width,
317
+ nhead=heads,
318
+ dim_feedforward=feedforward,
319
+ dropout=dropout,
320
+ activation="gelu",
321
+ batch_first=True,
322
+ norm_first=True,
323
+ )
324
+ for _ in range(layers)
325
+ ]
326
+ )
327
+ self.field_norm = torch.nn.LayerNorm(width)
328
+ self.option_norm = torch.nn.LayerNorm(width)
329
+ self.residual_scorer = torch.nn.Sequential(
330
+ torch.nn.Linear(width * 4, width),
331
+ torch.nn.GELU(),
332
+ torch.nn.Dropout(dropout),
333
+ torch.nn.Linear(width, 1),
334
+ )
335
+ self.prior_logit_scale = torch.nn.Parameter(torch.zeros(()))
336
+ self.joint_logit_scale = torch.nn.Parameter(torch.zeros(()))
337
+ self.residual_gate = torch.nn.Parameter(torch.zeros(()))
338
+
339
+ @staticmethod
340
+ def _mean_span(values: torch.Tensor, span: tuple[int, int]) -> torch.Tensor:
341
+ start, end = span
342
+ return values[start:end].mean(dim=0)
343
+
344
+ def forward(
345
+ self,
346
+ hidden_states: torch.Tensor,
347
+ input_ids: torch.Tensor,
348
+ attention_mask: torch.Tensor,
349
+ records: list[EncodedRecord],
350
+ output_embedding_weight: torch.Tensor,
351
+ ) -> list[list[torch.Tensor]]:
352
+ results: list[list[torch.Tensor]] = []
353
+ normalized_hidden = self.hidden_norm(hidden_states)
354
+ for batch_index, record in enumerate(records):
355
+ sequence_length = int(attention_mask[batch_index].sum().item())
356
+ sequence_hidden = normalized_hidden[batch_index, :sequence_length]
357
+ memory = self.memory_projection(sequence_hidden).unsqueeze(0)
358
+ global_vector = sequence_hidden[-1]
359
+ question_vectors = torch.stack(
360
+ [
361
+ self._mean_span(sequence_hidden, question.question_span)
362
+ for question in record.questions
363
+ ]
364
+ )
365
+ type_ids = torch.tensor(
366
+ [question.question_type for question in record.questions],
367
+ device=hidden_states.device,
368
+ )
369
+ option_contexts: list[torch.Tensor] = []
370
+ lexical_options: list[torch.Tensor] = []
371
+ option_counts = []
372
+ for question in record.questions:
373
+ context_vectors = torch.stack(
374
+ [
375
+ self._mean_span(sequence_hidden, span)
376
+ for span in question.option_spans
377
+ ]
378
+ )
379
+ lexical_vectors = []
380
+ for start, end in question.option_spans:
381
+ token_ids = input_ids[batch_index, start:end]
382
+ lexical_vectors.append(output_embedding_weight[token_ids].mean(dim=0))
383
+ lexical = torch.stack(lexical_vectors)
384
+ option_contexts.append(context_vectors)
385
+ lexical_options.append(lexical)
386
+ option_counts.append(len(question.option_spans))
387
+
388
+ option_queries = []
389
+ for question_index, (context_vectors, lexical) in enumerate(
390
+ zip(option_contexts, lexical_options)
391
+ ):
392
+ option_queries.append(
393
+ self.option_context_projection(context_vectors)
394
+ + self.option_lexical_projection(lexical)
395
+ + self.option_question_projection(
396
+ question_vectors[question_index]
397
+ ).unsqueeze(0)
398
+ )
399
+ routed_options = torch.cat(option_queries, dim=0).unsqueeze(0)
400
+ for layer in self.evidence_layers:
401
+ routed_options = layer(routed_options, memory)
402
+ routed_options = routed_options[0]
403
+ split_options = list(torch.split(routed_options, option_counts, dim=0))
404
+
405
+ base_fields = self.question_projection(question_vectors)
406
+ option_summaries = []
407
+ for field, options in zip(base_fields, split_options):
408
+ routing_weights = torch.softmax(
409
+ torch.matmul(options, field) / math.sqrt(options.shape[-1]),
410
+ dim=0,
411
+ )
412
+ option_summaries.append(
413
+ torch.sum(routing_weights.unsqueeze(-1) * options, dim=0)
414
+ )
415
+ fields = (
416
+ base_fields
417
+ + self.option_summary_norm(torch.stack(option_summaries))
418
+ + self.global_projection(global_vector).unsqueeze(0)
419
+ + self.type_embedding(type_ids)
420
+ )
421
+ fields = fields.unsqueeze(0)
422
+ for layer in self.layers:
423
+ fields = layer(fields, memory)
424
+ fields = self.field_norm(fields[0])
425
+
426
+ record_logits: list[torch.Tensor] = []
427
+ for field, question, lexical, routed in zip(
428
+ fields,
429
+ record.questions,
430
+ lexical_options,
431
+ split_options,
432
+ ):
433
+ anchor = functional.normalize(
434
+ question_vectors[len(record_logits)] + global_vector,
435
+ dim=-1,
436
+ )
437
+ lexical_anchor = functional.normalize(lexical, dim=-1)
438
+ prior_scale = self.prior_logit_scale.clamp(max=math.log(100.0)).exp()
439
+ prior = prior_scale * torch.matmul(lexical_anchor, anchor)
440
+ options = self.option_norm(routed)
441
+ repeated_field = field.unsqueeze(0).expand_as(options)
442
+ cosine = functional.cosine_similarity(repeated_field, options, dim=-1)
443
+ features = torch.cat(
444
+ [
445
+ repeated_field,
446
+ options,
447
+ repeated_field * options,
448
+ torch.abs(repeated_field - options),
449
+ ],
450
+ dim=-1,
451
+ )
452
+ residual = self.residual_scorer(features).squeeze(-1)
453
+ joint_scale = self.joint_logit_scale.clamp(max=math.log(100.0)).exp()
454
+ joint = joint_scale * cosine + residual
455
+ record_logits.append(
456
+ prior + torch.sigmoid(self.residual_gate) * joint
457
+ )
458
+ results.append(record_logits)
459
+ return results
460
+
461
+
462
+ class ClefModel(torch.nn.Module):
463
+ def __init__(self, language_model: Any, head: JointSchemaHead) -> None:
464
+ super().__init__()
465
+ self.language_model = language_model
466
+ self.head = head
467
+
468
+ def forward(self, batch: dict[str, Any]) -> list[list[torch.Tensor]]:
469
+ base_model = (
470
+ self.language_model.get_base_model()
471
+ if hasattr(self.language_model, "get_base_model")
472
+ else self.language_model
473
+ )
474
+ media = batch.get("media") or {}
475
+ text_model = base_model.model
476
+ if not media and hasattr(text_model, "language_model"):
477
+ text_model = text_model.language_model
478
+ outputs = text_model(
479
+ input_ids=batch["input_ids"],
480
+ attention_mask=batch["attention_mask"],
481
+ use_cache=False,
482
+ return_dict=True,
483
+ **media,
484
+ )
485
+ return self.head(
486
+ outputs.last_hidden_state,
487
+ batch["input_ids"],
488
+ batch["attention_mask"],
489
+ batch["records"],
490
+ base_model.get_output_embeddings().weight,
491
+ )
492
+
493
+
494
+ def load_release_model(
495
+ model_path: str | Path,
496
+ device: str | torch.device = "cuda",
497
+ dtype: torch.dtype = torch.bfloat16,
498
+ **from_pretrained_kwargs: Any,
499
+ ) -> tuple[ClefModel, Any]:
500
+ """Load a Clef release (merged backbone, joint schema head, and processor)."""
501
+ from huggingface_hub import snapshot_download
502
+ from safetensors.torch import load_file
503
+ from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
504
+
505
+ path = Path(model_path)
506
+ if not path.is_dir():
507
+ path = Path(snapshot_download(str(model_path)))
508
+ backbone = Qwen3_5ForConditionalGeneration.from_pretrained(
509
+ path,
510
+ dtype=dtype,
511
+ device_map={"": str(device)},
512
+ **from_pretrained_kwargs,
513
+ )
514
+ backbone.config.use_cache = False
515
+ head_config = json.loads((path / "joint_head_config.json").read_text())
516
+ head = JointSchemaHead(**head_config)
517
+ head.load_state_dict(load_file(path / "joint_head.safetensors"), strict=True)
518
+ head = head.to(device=device, dtype=dtype)
519
+ processor = AutoProcessor.from_pretrained(path)
520
+ return ClefModel(backbone, head).eval(), processor
521
+
522
+
523
+ def systemone_answer(question: dict[str, Any], probabilities: dict[str, float]) -> dict[str, Any]:
524
+ """Convert per-option probabilities for one question into a SystemOne answer."""
525
+ if question["type"] == "noul":
526
+ return {"type": "noul", "noul": round(probabilities["true"], 4)}
527
+ if question["type"] == "choice":
528
+ options = [str(option) for option in question["criteria"]]
529
+ choice = max(options, key=probabilities.__getitem__)
530
+ return {
531
+ "type": "choice",
532
+ "choice": choice,
533
+ "confidence": round(probabilities[choice], 4),
534
+ "probabilities": {option: round(probabilities[option], 4) for option in options},
535
+ }
536
+ levels = [str(index) for index in range(len(question["criteria"]))]
537
+ return {
538
+ "type": "score",
539
+ "score": round(sum(index * probabilities[level] for index, level in enumerate(levels)), 4),
540
+ "confidence": round(max(probabilities[level] for level in levels), 4),
541
+ "legend": dict(zip(levels, question["criteria"])),
542
+ "probabilities": {level: round(probabilities[level], 4) for level in levels},
543
+ }
544
+
545
+
546
+ @torch.inference_mode()
547
+ def systemone(model: ClefModel, processor: Any, request: dict[str, Any], max_length: int = 16384) -> dict[str, Any]:
548
+ """Answer a Jev/SystemOne ``/v1/systemone`` request body with a SystemOne response body.
549
+
550
+ The request has ``model``, ``state``, and ``questions``, plus optional ``images`` and ``videos``.
551
+ """
552
+ questions = request.get("questions")
553
+ if not isinstance(request.get("model"), str) or "state" not in request:
554
+ raise ValueError("model and state are required")
555
+ if not isinstance(questions, dict) or not questions:
556
+ raise ValueError("at least one question is required")
557
+ for question_id, question in questions.items():
558
+ if question.get("type") not in QUESTION_TYPES:
559
+ raise ValueError(f"{question_id}: type must be noul, choice, or score")
560
+ if question["type"] != "noul" and not question.get("criteria"):
561
+ raise ValueError(f"{question_id}: criteria must not be empty")
562
+ encoded = encode_record(processor.tokenizer, request, max_length=max_length, processor=processor)
563
+ device = next(model.parameters()).device
564
+ logits = model(collate_records([encoded], processor.tokenizer.pad_token_id, device))[0]
565
+ answers = {
566
+ question.question_id: systemone_answer(
567
+ questions[question.question_id],
568
+ dict(zip(question.option_ids, question_logits.float().softmax(-1).tolist())),
569
+ )
570
+ for question, question_logits in zip(encoded.questions, logits)
571
+ }
572
+ return {
573
+ "model": request["model"],
574
+ "answers": answers,
575
+ "usage": {"input_tokens": len(encoded.input_ids), "output_tokens": 0},
576
+ }
lm-head.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dd63614f1dc80dce2d83be3f3e69af1f8a0ebc8add9b8e0abf3910f563e44344
3
+ size 2034237568
model-00001-of-00005.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9c5483aa32a952b3a9a5d75f3c000c9ff934d3cbbb317b228cadff89e25bcffa
3
+ size 2034237592
model-00002-of-00005.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6999510bfc3c9ea6536090fb76b70bb786a0f4bd76f290ea35f35f58aa77e4f6
3
+ size 1984182833
model-00003-of-00005.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:efa24fbcede25d84116de286e544554513bd0400c40c66b99c32e846f5096cdd
3
+ size 1983829880
model-00004-of-00005.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7a5dd174473310a090c74abe907de00da6b85c46f8a3b5bcaec20267449fbdbd
3
+ size 1983498867
model-00005-of-00005.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1463c85bac6efd374bb5189d3beda19df628a786a9068b00de74dd3504f12df9
3
+ size 1886989356
model.safetensors.index.json ADDED
The diff for this file is too large to render. See raw diff
 
processor_config.json ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor": {
3
+ "do_convert_rgb": true,
4
+ "do_normalize": true,
5
+ "do_rescale": true,
6
+ "do_resize": true,
7
+ "image_mean": [
8
+ 0.5,
9
+ 0.5,
10
+ 0.5
11
+ ],
12
+ "image_processor_type": "Qwen2VLImageProcessor",
13
+ "image_std": [
14
+ 0.5,
15
+ 0.5,
16
+ 0.5
17
+ ],
18
+ "merge_size": 2,
19
+ "patch_size": 16,
20
+ "resample": 3,
21
+ "rescale_factor": 0.00392156862745098,
22
+ "size": {
23
+ "longest_edge": 16777216,
24
+ "shortest_edge": 65536
25
+ },
26
+ "temporal_patch_size": 2
27
+ },
28
+ "processor_class": "Qwen3VLProcessor",
29
+ "video_processor": {
30
+ "do_convert_rgb": true,
31
+ "do_normalize": true,
32
+ "do_rescale": true,
33
+ "do_resize": true,
34
+ "do_sample_frames": true,
35
+ "fps": 2,
36
+ "image_mean": [
37
+ 0.5,
38
+ 0.5,
39
+ 0.5
40
+ ],
41
+ "image_std": [
42
+ 0.5,
43
+ 0.5,
44
+ 0.5
45
+ ],
46
+ "max_frames": 768,
47
+ "merge_size": 2,
48
+ "min_frames": 4,
49
+ "patch_size": 16,
50
+ "resample": 3,
51
+ "rescale_factor": 0.00392156862745098,
52
+ "return_metadata": false,
53
+ "size": {
54
+ "longest_edge": 25165824,
55
+ "shortest_edge": 4096
56
+ },
57
+ "temporal_patch_size": 2,
58
+ "video_processor_type": "Qwen3VLVideoProcessor"
59
+ }
60
+ }
requirements.txt ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ # Isolated from Forge and the diffusion-model runtimes.
2
+ transformers==5.10.2
3
+ bitsandbytes==0.50.2
4
+ accelerate==1.15.0
5
+ huggingface_hub==1.5.0
6
+ safetensors==0.8.0
7
+ Pillow==12.3.0
8
+ torchvision==0.28.0
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523
3
+ size 19989325
tokenizer_config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "audio_bos_token": "<|audio_start|>",
4
+ "audio_eos_token": "<|audio_end|>",
5
+ "audio_token": "<|audio_pad|>",
6
+ "backend": "tokenizers",
7
+ "bos_token": null,
8
+ "clean_up_tokenization_spaces": false,
9
+ "eos_token": "<|im_end|>",
10
+ "errors": "replace",
11
+ "image_token": "<|image_pad|>",
12
+ "model_max_length": 262144,
13
+ "model_specific_special_tokens": {
14
+ "audio_bos_token": "<|audio_start|>",
15
+ "audio_eos_token": "<|audio_end|>",
16
+ "audio_token": "<|audio_pad|>",
17
+ "image_token": "<|image_pad|>",
18
+ "video_token": "<|video_pad|>",
19
+ "vision_bos_token": "<|vision_start|>",
20
+ "vision_eos_token": "<|vision_end|>"
21
+ },
22
+ "pad_token": "<|endoftext|>",
23
+ "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
24
+ "split_special_tokens": false,
25
+ "tokenizer_class": "Qwen2Tokenizer",
26
+ "unk_token": null,
27
+ "video_token": "<|video_pad|>",
28
+ "vision_bos_token": "<|vision_start|>",
29
+ "vision_eos_token": "<|vision_end|>"
30
+ }