devilyouwei commited on
Commit
6aaa9cc
·
verified ·
1 Parent(s): 7a58a50

Replace fixed graph with dynamic sequence and option dimensions

Browse files

Retain upstream 32768-token input bound without truncation. CPU parity passed 12 cases, including exact 1024/2048 tokens, 1/2/3 options, and JSON. Standalone Python and native Swift verified. Native 32K inference remains untested; decision adapters remain outside this export. English and Chinese model cards describe the validated scope.

README.md CHANGED
@@ -18,20 +18,21 @@ tags:
18
  English | [简体中文](README.zh-CN.md)
19
 
20
  An Apple Core AI conversion of [vllm-sr/vela3-fast](https://huggingface.co/vllm-sr/vela3-fast) by hugging-mac.
21
- This release contains the validated **choice / noul decision backbone**: one sample, at most 128 tokens in total, and exactly two options.
 
22
  **It is an experimental subset, not the complete Vela3-fast model with all task adapters.**
23
 
24
  Neural inference runs through Core AI, with host-side tokenization and response formatting. The model does not generate text.
25
 
26
  ## Supported contract
27
 
28
- - `choice`: two ordered options; returns their probabilities and the selected option.
29
  - `noul`: fixed `no` / `yes` options; returns `p(yes)`.
30
- - The input text, question, option descriptions, and special markers together must fit within **128 tokens**. Longer inputs raise an error; they are not truncated.
31
- - One question per inference, with batch size 1. The Python adapter accepts text or JSON objects serialized according to upstream rules.
32
  - CPU execution is validated. Earlier automatic-device validation did not finish because of resource pressure on the test host; no GPU/ANE performance is claimed.
33
  - Original FP32 decision weights, with no quantization, training, or fine-tuning.
34
- - Router, span, relation, NER, evidence, domain/factcheck/feedback, and other upstream adapters are not included. More than two options, long contexts, and full `system_one` routing are not supported by this artifact. Untested capabilities are not known to be unconvertible.
35
  - This release does not provide complete `hugging_mac_sdk` registration or production acceptance. Probabilities are relative to the supplied options, not calibrated confidence scores.
36
 
37
  ## Python usage
@@ -74,7 +75,7 @@ asyncio.run(main())
74
 
75
  `predict` prepares token IDs, marker positions, and the pooling mask before calling Core AI.
76
  Each instance accepts one inference request at a time; await completion before making another call.
77
- Unknown tasks, invalid option counts, and inputs exceeding the token limit raise errors.
78
 
79
  Run the standalone package checks:
80
 
@@ -82,8 +83,8 @@ Run the standalone package checks:
82
  USE_OS_COREAI=1 uv run --locked python smoke.py
83
  ```
84
 
85
- This compares six cases with upstream reference probabilities, checks exact input-tensor equality against upstream encoding,
86
- and verifies rejection of five invalid-request categories. It writes `package_smoke.json` and `swift_fixture.json` for the Swift example.
87
 
88
  ## Swift usage
89
 
@@ -117,7 +118,7 @@ var result = try await function.run(inputs: inputs)
117
  guard let logits = result.remove("logits")?.ndArray else {
118
  fatalError("Missing logits")
119
  }
120
- // Apply stable softmax to the two logits and interpret them in input-option order.
121
  ```
122
 
123
  For new text, reuse `tokenizer.json`, `decision.json`, and the encoding rules in `vela_coreai.py`.
@@ -125,15 +126,15 @@ Tokenizing only the input body does not produce valid model inputs. The bundled
125
 
126
  ## Tensor interface
127
 
128
- Entry point: `main`. No persistent recurrent/KV state. Inputs are right-padded to 128 tokens.
129
 
130
  | Name | Type | Shape | Meaning |
131
  | --- | --- | --- | --- |
132
- | `input_ids` | int32 | `[1,128]` | Complete question/options/body sequence |
133
- | `attention_mask` | int32 | `[1,128]` | 1 for valid tokens, 0 for padding |
134
- | `option_positions` | int32 | `[1,2]` | Positions of the two `[O]` markers |
135
- | `pool_mask` | float32 | `[1,128]` | 1 from `[SEG_user]` through the end of the body, excluding EOS |
136
- | `logits` (output) | float32 | `[1,2]` | Unnormalized scores for the two ordered options |
137
 
138
  Upstream sequence format:
139
 
@@ -144,13 +145,15 @@ Upstream sequence format:
144
 
145
  ## Validation and limitations
146
 
147
- - **6/6** text cases matched the original model's decisions and the expected answers.
148
- - Maximum absolute probability error was approximately **5.57e-7**. Validation uses `atol=0.02, rtol=0.02` and also requires decision agreement.
149
- - The standalone Python package passed all six cases without PyTorch installed, with exact input-tensor parity against upstream.
150
  - Native Swift CPU results are in `swift_smoke.json`; standalone Python results are in `package_smoke.json`.
151
- - Six handcrafted examples are not a general accuracy benchmark. JSON-object encoding is supported but was not separately evaluated in these six semantic cases.
152
- - iPhone/iPad, iOS AOT, automatic device selection, GPU/ANE placement, long contexts, and other upstream tasks have not been validated.
153
- - Export and runtime environments differ; their versions are recorded in `manifest.json` and `environment.json`. Earlier automatic-device validation was interrupted; subsequent CPU evaluation used the same artifact. The interrupted result was not rewritten as a success.
 
 
154
 
155
  Full cases and probabilities: `validation_cases.json` and `evaluation.json`.
156
  `SHA256SUMS` covers the published files. Original weight copies, development caches, and virtual environments are not uploaded.
 
18
  English | [简体中文](README.zh-CN.md)
19
 
20
  An Apple Core AI conversion of [vllm-sr/vela3-fast](https://huggingface.co/vllm-sr/vela3-fast) by hugging-mac.
21
+ This release contains the validated **choice / noul decision backbone**: one sample per graph call, dynamic sequence length, and a dynamic number of options.
22
+ The export retains the upstream **32,768-token** input bound; actual CPU parity tests currently cover lengths through **2,048 tokens**, including exactly 1,024.
23
  **It is an experimental subset, not the complete Vela3-fast model with all task adapters.**
24
 
25
  Neural inference runs through Core AI, with host-side tokenization and response formatting. The model does not generate text.
26
 
27
  ## Supported contract
28
 
29
+ - `choice`: one or more ordered options; returns their probabilities and the selected option.
30
  - `noul`: fixed `no` / `yes` options; returns `p(yes)`.
31
+ - The input text, question, option descriptions, and special markers together must fit within the upstream **32,768-token** bound. Longer inputs raise an error; they are not truncated.
32
+ - Each graph call has batch size 1. `predict_questions(state, questions)` accepts a mapping of questions and returns `(answers, input_tokens)`, matching the upstream decision API; questions execute sequentially. The adapter preserves upstream JSON serialization and accepts mapping or list choice criteria.
33
  - CPU execution is validated. Earlier automatic-device validation did not finish because of resource pressure on the test host; no GPU/ANE performance is claimed.
34
  - Original FP32 decision weights, with no quantization, training, or fine-tuning.
35
+ - Router, span, relation, NER, evidence, domain/factcheck/feedback, and other upstream adapters are not included. Full `system_one` adapter routing is not implemented by this decision-only artifact. Untested capabilities are not known to be unconvertible.
36
  - This release does not provide complete `hugging_mac_sdk` registration or production acceptance. Probabilities are relative to the supplied options, not calibrated confidence scores.
37
 
38
  ## Python usage
 
75
 
76
  `predict` prepares token IDs, marker positions, and the pooling mask before calling Core AI.
77
  Each instance accepts one inference request at a time; await completion before making another call.
78
+ Unknown tasks, empty option sets, and inputs exceeding the token limit raise errors.
79
 
80
  Run the standalone package checks:
81
 
 
83
  USE_OS_COREAI=1 uv run --locked python smoke.py
84
  ```
85
 
86
+ This compares 12 cases with upstream reference probabilities, checks exact input-tensor equality against upstream encoding,
87
+ and verifies rejection of three invalid-request categories. It writes `package_smoke.json` and `swift_fixture.json` for the Swift example.
88
 
89
  ## Swift usage
90
 
 
118
  guard let logits = result.remove("logits")?.ndArray else {
119
  fatalError("Missing logits")
120
  }
121
+ // Apply stable softmax to the output logits and interpret them in input-option order.
122
  ```
123
 
124
  For new text, reuse `tokenizer.json`, `decision.json`, and the encoding rules in `vela_coreai.py`.
 
126
 
127
  ## Tensor interface
128
 
129
+ Entry point: `main`. No persistent recurrent/KV state. Sequence and option axes are dynamic; inputs use their actual encoded length without fixed padding.
130
 
131
  | Name | Type | Shape | Meaning |
132
  | --- | --- | --- | --- |
133
+ | `input_ids` | int32 | `[1,sequence]` | Complete question/options/body sequence |
134
+ | `attention_mask` | int32 | `[1,sequence]` | 1 for valid tokens, 0 for padding |
135
+ | `option_positions` | int32 | `[1,options]` | Positions of the `[O]` markers |
136
+ | `pool_mask` | float32 | `[1,sequence]` | 1 from `[SEG_user]` through the end of the body, excluding EOS |
137
+ | `logits` (output) | float32 | `[1,options]` | Unnormalized scores for the one or more ordered options |
138
 
139
  Upstream sequence format:
140
 
 
145
 
146
  ## Validation and limitations
147
 
148
+ - **12/12** text cases matched the original model's decisions and the expected answers.
149
+ - Maximum absolute probability error was approximately **7.80e-7**. Validation uses `atol=0.02, rtol=0.02` and also requires decision agreement.
150
+ - The standalone Python package passed all 12 cases without PyTorch installed, with exact input-tensor parity against upstream.
151
  - Native Swift CPU results are in `swift_smoke.json`; standalone Python results are in `package_smoke.json`.
152
+ - These handcrafted examples are not a general accuracy benchmark. Coverage includes a JSON object, 1/2/3 options, and exact 129/1,024/2,048-token inputs.
153
+ - iPhone/iPad, iOS AOT, automatic device selection, GPU/ANE placement, lengths above 2,048 tokens, and other upstream tasks have not been validated.
154
+ - The new dynamic artifact passed initial CPU tensor validation and subsequent task evaluation. Tool versions are recorded in `manifest.json` and `environment.json`.
155
+ - The 32K bound is retained in the export and host adapter, but end-to-end 32K inference has not been validated on this 8 GB test machine. Longer inputs require additional memory; 2,048 is a tested length, not a new hard limit.
156
+ - This revision replaces the earlier fixed 128-token, two-option export. The adapters listed above remain outside the current decision graph.
157
 
158
  Full cases and probabilities: `validation_cases.json` and `evaluation.json`.
159
  `SHA256SUMS` covers the published files. Original weight copies, development caches, and virtual environments are not uploaded.
README.zh-CN.md CHANGED
@@ -18,7 +18,8 @@ tags:
18
  [English](README.md) | 简体中文
19
 
20
  由 hugging-mac 从 [vllm-sr/vela3-fast](https://huggingface.co/vllm-sr/vela3-fast) 转换的 Apple Core AI 模型。
21
- 本仓库发布的是经过验证的 **choice / noul 决策主干**:单样本、总长度最多 128 tokens、恰好两个选项。
 
22
  **这是实验性子集,不是包含所有任务 adapter 的完整 Vela3-fast。**
23
 
24
  This is an experimental, fixed-shape decision-backbone export, not a complete replacement for upstream Vela3-fast.
@@ -26,14 +27,14 @@ It runs neural inference through Core AI with host-side tokenization and respons
26
 
27
  ## 支持范围 / Contract
28
 
29
- - `choice`:两个有序选项,返回选项概率和选择结果。
30
  - `noul`:固定 `no` / `yes` 两个选项,返回 `p(yes)`。
31
- - 输入正文、问题、选项描述和特殊标记合计不超过 **128 tokens**;超长输入报错,不截断。
32
- - 每次推理一个问题,batch size 为 1。Python 适配器支持文本或按上游规则序列化的 JSON 对象。
33
  - CPU 模式已验证。自动设备模式此前因测试主机资源压力未完成验证,不宣称 GPU/ANE 性能。
34
  - 原始 FP32 决策权重,无量化、训练或微调。
35
  - 原模型的 router、span、relation、NER、evidence、domain/factcheck/feedback 等 adapter 不包含在此制品中;
36
- 超过两个选项、长上下文和完整 `system_one` 路由也不受支持。未测试不代表这些能力无法转换。
37
  - 不提供完整 `hugging_mac_sdk` 模型注册或生产验收结论。概率是给定选项间的相对值,不是校准置信度。
38
 
39
  ## Python 基本调用
@@ -75,7 +76,7 @@ asyncio.run(main())
75
  ```
76
 
77
  `predict` 执行 tokenizer、标记位置和 pooling mask 的准备,再调用 Core AI。
78
- 同一个实例一次只接收一个推理请求;请等待前一次调用完成。未知任务、错误选项数和超长输入都会报错。
79
 
80
  运行独立包验证:
81
 
@@ -83,7 +84,7 @@ asyncio.run(main())
83
  USE_OS_COREAI=1 uv run --locked python smoke.py
84
  ```
85
 
86
- 该脚本比较六个用例与上游参考概率,校验输入张量与上游编码完全相同,验证五类无效请求拒绝行为,
87
  并生成 `package_smoke.json` 和用于 Swift 的 `swift_fixture.json`。
88
 
89
  ## Swift 基本调用
@@ -118,7 +119,7 @@ var result = try await function.run(inputs: inputs)
118
  guard let logits = result.remove("logits")?.ndArray else {
119
  fatalError("Missing logits")
120
  }
121
- // 对两个 logits 计算稳定 softmax,按输入选项顺序解释概率。
122
  ```
123
 
124
  对任意新文本调用时,必须复用本仓库 `tokenizer.json`、`decision.json` 和 `vela_coreai.py` 的编码规则;
@@ -126,15 +127,15 @@ guard let logits = result.remove("logits")?.ndArray else {
126
 
127
  ## 张量接口
128
 
129
- 入口:`main`,无持久 recurrent/KV 状态。右侧补齐到 128 tokens。
130
 
131
  | 名称 | 类型 | 形状 | 含义 |
132
  | --- | --- | --- | --- |
133
- | `input_ids` | int32 | `[1,128]` | 完整问题/选项/正文序列 |
134
- | `attention_mask` | int32 | `[1,128]` | 有效 token 为 1,padding 为 0 |
135
- | `option_positions` | int32 | `[1,2]` | 两个 `[O]` 标记的位置 |
136
- | `pool_mask` | float32 | `[1,128]` | `[SEG_user]` 到正文末尾为 1,不包括 EOS |
137
- | `logits`(输出) | float32 | `[1,2]` | 对应两个有序选项的未归一化分数 |
138
 
139
  上游序列格式:
140
 
@@ -145,14 +146,15 @@ guard let logits = result.remove("logits")?.ndArray else {
145
 
146
  ## 验证记录与限制
147
 
148
- - 对上游原始模型的 **6/6** 文本用例,转换后决策全部一致,预期答案全部正确。
149
- - 最大概率绝对误差约 **5.57e-7**;验证容差为 `atol=0.02, rtol=0.02`,同时要求决策一致。
150
- - 独立 Python 包在未安装 PyTorch 的环境中通过六个用例,输入张量逐项匹配上游。
151
  - Swift CPU 原生调用结果见 `swift_smoke.json`;Python 独立包记录见 `package_smoke.json`。
152
- - 六个手工样例不是通用准确率基准;JSON 对象仅有编码支持,未在这组六个语义用例中单独评估。
153
- - 未验证 iPhone/iPad、iOS AOT、自动设备模式、GPU/ANE 放置、长上下文或其他上游任务。
154
- - 转换环境与运行环境的版本不同,分别记录在 `manifest.json` 与 `environment.json`。
155
- 早期自动设备验证中止,后续在同一制品上完成 CPU 评测;没有把中止记录改写成成功。
 
156
 
157
  完整用例与概率:`validation_cases.json`、`evaluation.json`。
158
  `SHA256SUMS` 覆盖发布文件;本仓库不上传原始权重副本、开发缓存或虚拟环境。
 
18
  [English](README.md) | 简体中文
19
 
20
  由 hugging-mac 从 [vllm-sr/vela3-fast](https://huggingface.co/vllm-sr/vela3-fast) 转换的 Apple Core AI 模型。
21
+ 本仓库发布的是经过验证的 **choice / noul 决策主干**:每次图调用一个样本,序列长度和选项数量均为动态。
22
+ 导出保留原始 **32768-token** 上限;CPU 实际对照验证覆盖到 **2048 tokens**,包括精确 1024 tokens。
23
  **这是实验性子集,不是包含所有任务 adapter 的完整 Vela3-fast。**
24
 
25
  This is an experimental, fixed-shape decision-backbone export, not a complete replacement for upstream Vela3-fast.
 
27
 
28
  ## 支持范围 / Contract
29
 
30
+ - `choice`:一个或多个有序选项,返回选项概率和选择结果。
31
  - `noul`:固定 `no` / `yes` 两个选项,返回 `p(yes)`。
32
+ - 输入正文、问题、选项描述和特殊标记合计不超过 原始 **32768 tokens** 上限;超长输入报错,不截断。
33
+ - 每次图调用的 batch size 为 1。`predict_questions(state, questions)` 接受问题映射并返回 `(answers, input_tokens)`,与上游决策接口对应;内部逐个执行。保留原始 JSON 序列化,choice criteria 支持字典或选项列表。
34
  - CPU 模式已验证。自动设备模式此前因测试主机资源压力未完成验证,不宣称 GPU/ANE 性能。
35
  - 原始 FP32 决策权重,无量化、训练或微调。
36
  - 原模型的 router、span、relation、NER、evidence、domain/factcheck/feedback 等 adapter 不包含在此制品中;
37
+ 当前决策图未实现完整 `system_one` adapter 路由。未测试不代表这些能力无法转换。
38
  - 不提供完整 `hugging_mac_sdk` 模型注册或生产验收结论。概率是给定选项间的相对值,不是校准置信度。
39
 
40
  ## Python 基本调用
 
76
  ```
77
 
78
  `predict` 执行 tokenizer、标记位置和 pooling mask 的准备,再调用 Core AI。
79
+ 同一个实例一次只接收一个推理请求;请等待前一次调用完成。未知任务、空选项集合和超长输入都会报错。
80
 
81
  运行独立包验证:
82
 
 
84
  USE_OS_COREAI=1 uv run --locked python smoke.py
85
  ```
86
 
87
+ 该脚本比较12 个用例与上游参考概率,校验输入张量与上游编码完全相同,验证三类无效请求拒绝行为,
88
  并生成 `package_smoke.json` 和用于 Swift 的 `swift_fixture.json`。
89
 
90
  ## Swift 基本调用
 
119
  guard let logits = result.remove("logits")?.ndArray else {
120
  fatalError("Missing logits")
121
  }
122
+ // 对输出 logits 计算稳定 softmax,按输入选项顺序解释概率。
123
  ```
124
 
125
  对任意新文本调用时,必须复用本仓库 `tokenizer.json`、`decision.json` 和 `vela_coreai.py` 的编码规则;
 
127
 
128
  ## 张量接口
129
 
130
+ 入口:`main`,无持久 recurrent/KV 状态。序列轴和选项轴均为动态,按实际编码长度输入,不固定补齐。
131
 
132
  | 名称 | 类型 | 形状 | 含义 |
133
  | --- | --- | --- | --- |
134
+ | `input_ids` | int32 | `[1,sequence]` | 完整问题/选项/正文序列 |
135
+ | `attention_mask` | int32 | `[1,sequence]` | 有效 token 为 1,padding 为 0 |
136
+ | `option_positions` | int32 | `[1,options]` | 各个 `[O]` 标记的位置 |
137
+ | `pool_mask` | float32 | `[1,sequence]` | `[SEG_user]` 到正文末尾为 1,不包括 EOS |
138
+ | `logits`(输出) | float32 | `[1,options]` | 对应一个或多个有序选项的未归一化分数 |
139
 
140
  上游序列格式:
141
 
 
146
 
147
  ## 验证记录与限制
148
 
149
+ - 对上游原始模型的 **12/12** 文本用例,转换后决策全部一致,预期答案全部正确。
150
+ - 最大概率绝对误差约 **7.80e-7**;验证容差为 `atol=0.02, rtol=0.02`,同时要求决策一致。
151
+ - 独立 Python 包在未安装 PyTorch 的环境中通过12 个用例,输入张量逐项匹配上游。
152
  - Swift CPU 原生调用结果见 `swift_smoke.json`;Python 独立包记录见 `package_smoke.json`。
153
+ - 这些手工样例不是通用准确率基准;覆盖 JSON 对象、1/2/3 个选项,以及精确 129/1024/2048-token 输入。
154
+ - 未验证 iPhone/iPad、iOS AOT、自动设备模式、GPU/ANE 放置、超过 2048 tokens 的输入或其他上游任务。
155
+ - 新动态制品通过了初始 CPU 张量验证和后续任务对照评测。工具版本见 `manifest.json` 与 `environment.json`。
156
+ - 导出和输入适配器保留 32K 上限,但尚未在这台 8 GB 测试机器上验证完整 32K 推理;更长输入需要更多内存。2048 是已测长度,不是新增硬限制。
157
+ - 本版本替换先前固定 128-token、两个选项的制品;上述 adapters 仍不在当前决策图中。
158
 
159
  完整用例与概率:`validation_cases.json`、`evaluation.json`。
160
  `SHA256SUMS` 覆盖发布文件;本仓库不上传原始权重副本、开发缓存或虚拟环境。
SHA256SUMS CHANGED
@@ -1,24 +1,25 @@
1
- f6cd5f38bce95bd253b0d7bf4b4ac1f427c02e4a07dd7a8384fbd24f0bd98bab README.md
2
- d8922e9a8a5d799abfae6ec063c04d1a1159ebe3813f62a6c0a4e3a0e188431c README.zh-CN.md
3
  c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4 LICENSE
4
  46710ad6644d63748efc1d91895ed2f7d039ddd3e4fe9caf7351f2f7a18ae448 NOTICE
5
  7b55f8e67b5623c4bef3fa691288da9437d79d3aba156de48d481db32ac7d16d .python-version
6
  fe8ebeab92a5ecbca37f7fd768d480c122ceee326f5ac78f2e02cbf8c42adb85 pyproject.toml
7
  2f622f1a164e966f012193787cc8325837d96c2bd33f62f7a704606f0d0e17a9 uv.lock
8
- 74444a35a0b888acf65e3ef9687577c1f1cfd664c2b071a2820408c22af5bc1f vela_coreai.py
9
  83dd42bf29453dafe9ca9525e9f8085198dc01afee7338012643b900ed0ac57e example.py
10
- e21d2cd9f85b33194a0aefa2e69a572a80cc3d9e0497f4f793ef7afc813ca513 smoke.py
11
- c7364280e5c4173b277782fb403119f4485b4ea4fe441ca69b7d93bbe742491e verify_coreai.swift
12
  38734bb17ef19f84c67b5eb5692bea19801ffd02cb5c5ffc11429e76b0242e08 tokenizer.json
13
  649781df90d51837dd6ba951a89fae07c985a9eacfd781d84002d480c3cb0b6e tokenizer_config.json
14
  1150af4f64b7cec72e35ad2ad06db180be009d5d18ed05903c2af21f2a85ece3 decision.json
15
- 91668d28adb713c2241894c42e8e5115cd49949b28a2e234fb321ff36fbbe49e manifest.json
16
- a37c2903b39eef663e4f644f924dc234d483e66af7bc229cc86e4f337cd86bd5 validation_cases.json
17
- c9c5b9a7a7f214d7e2fc1f2478139b47e82bf65f000ec64483f727d49b9046bd evaluation.json
18
- e5800b20eaa3e72a09481cd4456051800b087f7c71466859c262e0c110d3df0a environment.json
19
- aef579fd2a256ae0f9aa817d6c8476379e8b9801471e67e6ac4ce685a9c01eae package_smoke.json
20
- 71f4627d7ddd18d1c4ec86361d8cdb5558b2bbf6cba458fa3a093511761be37e swift_fixture.json
21
- a42be9e045ef3544a8a74263b7c51b198c181dd6f044c922680ff7d8cd84ab0e swift_smoke.json
22
- f854ff014ef231c04a22951298a0c5560c68d951123b7943a9d232b34db6bee2 model.aimodel/main.hash
23
- e759a2108bf439ee87b10940f26d61a4078853e02f97d181d34165ae086b45ce model.aimodel/main.mlirb
24
- e9969e2a029f38890ec12496d8e5c2d46f90b6a7554de00fd84bc40db73c884c model.aimodel/metadata.json
 
 
1
+ b8c3735da9846be5e0c0ff4e7dcaa4724464b33d3b7044d69cc6edd7bdb39e22 README.md
2
+ 48a4f8a55205833272d55ec08bc74b025ae65b18eb8a18dd704036f52dd52929 README.zh-CN.md
3
  c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4 LICENSE
4
  46710ad6644d63748efc1d91895ed2f7d039ddd3e4fe9caf7351f2f7a18ae448 NOTICE
5
  7b55f8e67b5623c4bef3fa691288da9437d79d3aba156de48d481db32ac7d16d .python-version
6
  fe8ebeab92a5ecbca37f7fd768d480c122ceee326f5ac78f2e02cbf8c42adb85 pyproject.toml
7
  2f622f1a164e966f012193787cc8325837d96c2bd33f62f7a704606f0d0e17a9 uv.lock
8
+ 0e6f26d2bbe55ac16368da959bf07521e54be0276eddeeb1f65a64004ae6476e vela_coreai.py
9
  83dd42bf29453dafe9ca9525e9f8085198dc01afee7338012643b900ed0ac57e example.py
10
+ 32d69284eacb23ac0b6be8fe4de2e3604248d4c0adb377cbd8c0afac5153b4be smoke.py
11
+ 04d7528586eaa525bb0fe5c5952cefbf5f1363098cd941803f0560cace854cb8 verify_coreai.swift
12
  38734bb17ef19f84c67b5eb5692bea19801ffd02cb5c5ffc11429e76b0242e08 tokenizer.json
13
  649781df90d51837dd6ba951a89fae07c985a9eacfd781d84002d480c3cb0b6e tokenizer_config.json
14
  1150af4f64b7cec72e35ad2ad06db180be009d5d18ed05903c2af21f2a85ece3 decision.json
15
+ 6593400dbc76848462ffbc464d5f326fbf9a959f638c8f3c218df4a1b98ac804 manifest.json
16
+ c43550694f14e808e66044db2145b2780cc794336425f9674718dccf6f09022e validation_cases.json
17
+ 80793396bc59f00d88a7df5c75b3ce729ef089501702733fda50fed49d08a691 evaluation.json
18
+ c9c84393adbd3d190c0443fabae3bc313ecc807870287472bba318d5352b3cc7 environment.json
19
+ 328ec37a9e1d578ae789bc0930db16ac05cbb30b4dab87c4358b4318e1bacdbf package_smoke.json
20
+ 9f3a0299f9df5a090df083fc4862fbc79e72f1387db467efbd5ef4ac26927b2a swift_fixture.json
21
+ 92ebf2b041505c1acb0f1f68b5b2d2962f470785609b17d7ae54e57a0707b361 swift_smoke.json
22
+ d1e3ae70a989d8d38ccae6513bf0b4085f751545cb6cd4efbd5373fa26f74fc4 model.aimodel/main.hash
23
+ d7b63896067e505f67a03c1afe31e070b0a1f5d746e14d603d9130708798adad model.aimodel/main.mlirb
24
+ a19b297a1df17828004220bd2a85bdd86f6630fd3edca8e9d3853366147d96b7 model.aimodel/metadata.json
25
+ 636221d20e2bcaaded4299220632346f377231dd9fdaae006482f4f9faa350ad input_boundary.json
environment.json CHANGED
@@ -1,10 +1,10 @@
1
  {
2
  "python": "3.12.13",
3
  "platform": "macOS-27.2.1-arm64-arm-64bit",
4
- "pytorch_installed": false,
5
  "packages": {
6
  "coreai-core": "1.0.0b3",
7
  "numpy": "2.3.5",
8
  "tokenizers": "0.23.3"
9
- }
 
10
  }
 
1
  {
2
  "python": "3.12.13",
3
  "platform": "macOS-27.2.1-arm64-arm-64bit",
 
4
  "packages": {
5
  "coreai-core": "1.0.0b3",
6
  "numpy": "2.3.5",
7
  "tokenizers": "0.23.3"
8
+ },
9
+ "pytorch_installed": false
10
  }
evaluation.json CHANGED
@@ -12,7 +12,7 @@
12
  "coreai-core": "1.0.0b3",
13
  "coreai-torch": "0.4.3"
14
  },
15
- "cases_sha256": "779c07609db34ddd9b11a8649937ee7db4b45a3db5c1023ffda190339ca36aac",
16
  "evaluation_config": {
17
  "scope": "diagnostic",
18
  "cases": [
@@ -33,10 +33,10 @@
33
  "span_sets": "untested",
34
  "evidence": "untested",
35
  "router_task_adapters": "untested",
36
- "structured_state": "untested",
37
  "multiple_questions": "untested",
38
- "variable_sequence_length": "untested",
39
- "variable_options": "untested",
40
  "system_one_response": "untested",
41
  "domain_adapter": "untested",
42
  "factcheck_adapter": "untested",
@@ -47,7 +47,7 @@
47
  {
48
  "id": "coding-en",
49
  "capability": "choice",
50
- "inference_ms": 95.21100000711158,
51
  "status": "evaluated",
52
  "probabilities": [
53
  0.9964854118964818,
@@ -60,7 +60,7 @@
60
  {
61
  "id": "cooking-zh",
62
  "capability": "choice",
63
- "inference_ms": 132.54979200428352,
64
  "status": "evaluated",
65
  "probabilities": [
66
  0.22242194924412478,
@@ -73,7 +73,7 @@
73
  {
74
  "id": "noul-positive",
75
  "capability": "noul",
76
- "inference_ms": 86.67745799175464,
77
  "status": "evaluated",
78
  "probabilities": [
79
  0.04283435292391378,
@@ -86,7 +86,7 @@
86
  {
87
  "id": "noul-negative",
88
  "capability": "noul",
89
- "inference_ms": 92.96466699743178,
90
  "status": "evaluated",
91
  "probabilities": [
92
  0.9887530718985025,
@@ -99,7 +99,7 @@
99
  {
100
  "id": "coding-zh",
101
  "capability": "choice",
102
- "inference_ms": 130.05970799713396,
103
  "status": "evaluated",
104
  "probabilities": [
105
  0.9965400461728955,
@@ -112,7 +112,7 @@
112
  {
113
  "id": "cooking-en",
114
  "capability": "choice",
115
- "inference_ms": 100.37370900681708,
116
  "status": "evaluated",
117
  "probabilities": [
118
  0.0008233419586070937,
@@ -121,6 +121,88 @@
121
  "decision": "cooking",
122
  "expected": "cooking",
123
  "correct": true
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
124
  }
125
  ],
126
  "candidate": {
@@ -130,20 +212,20 @@
130
  {
131
  "id": "coding-en",
132
  "capability": "choice",
133
- "reference_ms": 95.21100000711158,
134
- "converted_ms": 5489.578958993661,
135
  "passed": true,
136
  "numeric_passed": true,
137
  "decision_agreement": true,
138
- "max_absolute_error": 2.504995120188036e-09,
139
- "mean_absolute_error": 2.504995110213376e-09,
140
  "reference_probabilities": [
141
  0.9964854118964818,
142
  0.003514588103518185
143
  ],
144
  "converted_probabilities": [
145
- 0.9964854093914867,
146
- 0.003514590608513305
147
  ],
148
  "reference_decision": "coding",
149
  "converted_decision": "coding",
@@ -155,20 +237,20 @@
155
  {
156
  "id": "cooking-zh",
157
  "capability": "choice",
158
- "reference_ms": 132.54979200428352,
159
- "converted_ms": 242.97170800855383,
160
  "passed": true,
161
  "numeric_passed": true,
162
  "decision_agreement": true,
163
- "max_absolute_error": 5.566675266299725e-07,
164
- "mean_absolute_error": 5.56667526602217e-07,
165
  "reference_probabilities": [
166
  0.22242194924412478,
167
  0.7775780507558753
168
  ],
169
  "converted_probabilities": [
170
- 0.22242250591165136,
171
- 0.7775774940883486
172
  ],
173
  "reference_decision": "cooking",
174
  "converted_decision": "cooking",
@@ -180,20 +262,20 @@
180
  {
181
  "id": "noul-positive",
182
  "capability": "noul",
183
- "reference_ms": 86.67745799175464,
184
- "converted_ms": 200.55320800747722,
185
  "passed": true,
186
  "numeric_passed": true,
187
  "decision_agreement": true,
188
- "max_absolute_error": 7.820040781592041e-08,
189
- "mean_absolute_error": 7.820040778469539e-08,
190
  "reference_probabilities": [
191
  0.04283435292391378,
192
  0.9571656470760861
193
  ],
194
  "converted_probabilities": [
195
- 0.042834274723506024,
196
- 0.9571657252764939
197
  ],
198
  "reference_decision": "yes",
199
  "converted_decision": "yes",
@@ -205,20 +287,20 @@
205
  {
206
  "id": "noul-negative",
207
  "capability": "noul",
208
- "reference_ms": 92.96466699743178,
209
- "converted_ms": 199.48370799829718,
210
  "passed": true,
211
  "numeric_passed": true,
212
  "decision_agreement": true,
213
- "max_absolute_error": 1.0605268028346382e-08,
214
- "mean_absolute_error": 1.060526795722272e-08,
215
  "reference_probabilities": [
216
  0.9887530718985025,
217
  0.011246928101497617
218
  ],
219
  "converted_probabilities": [
220
- 0.9887530825037704,
221
- 0.011246917496229589
222
  ],
223
  "reference_decision": "no",
224
  "converted_decision": "no",
@@ -230,20 +312,20 @@
230
  {
231
  "id": "coding-zh",
232
  "capability": "choice",
233
- "reference_ms": 130.05970799713396,
234
- "converted_ms": 190.3682919946732,
235
  "passed": true,
236
  "numeric_passed": true,
237
  "decision_agreement": true,
238
- "max_absolute_error": 1.1097869623277745e-08,
239
- "mean_absolute_error": 1.1097869601593702e-08,
240
  "reference_probabilities": [
241
  0.9965400461728955,
242
  0.0034599538271045662
243
  ],
244
  "converted_probabilities": [
245
- 0.9965400350750259,
246
- 0.003459964924974146
247
  ],
248
  "reference_decision": "coding",
249
  "converted_decision": "coding",
@@ -255,20 +337,20 @@
255
  {
256
  "id": "cooking-en",
257
  "capability": "choice",
258
- "reference_ms": 100.37370900681708,
259
- "converted_ms": 196.19354100723285,
260
  "passed": true,
261
  "numeric_passed": true,
262
  "decision_agreement": true,
263
- "max_absolute_error": 1.3729699288285782e-09,
264
- "mean_absolute_error": 1.3729698888757282e-09,
265
  "reference_probabilities": [
266
  0.0008233419586070937,
267
  0.9991766580413929
268
  ],
269
  "converted_probabilities": [
270
- 0.0008233433315770225,
271
- 0.999176656668423
272
  ],
273
  "reference_decision": "cooking",
274
  "converted_decision": "cooking",
@@ -276,6 +358,164 @@
276
  "reference_correct": true,
277
  "converted_correct": true,
278
  "status": "passed"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
279
  }
280
  ]
281
  }
 
12
  "coreai-core": "1.0.0b3",
13
  "coreai-torch": "0.4.3"
14
  },
15
+ "cases_sha256": "184b5ab437c84bea6c1517d84e145ecd68ba7e4fbb7b61d7f628b9d474076e17",
16
  "evaluation_config": {
17
  "scope": "diagnostic",
18
  "cases": [
 
33
  "span_sets": "untested",
34
  "evidence": "untested",
35
  "router_task_adapters": "untested",
36
+ "structured_state": "exercised-in-diagnostic",
37
  "multiple_questions": "untested",
38
+ "variable_sequence_length": "exercised-in-diagnostic",
39
+ "variable_options": "exercised-in-diagnostic",
40
  "system_one_response": "untested",
41
  "domain_adapter": "untested",
42
  "factcheck_adapter": "untested",
 
47
  {
48
  "id": "coding-en",
49
  "capability": "choice",
50
+ "inference_ms": 51.31262500071898,
51
  "status": "evaluated",
52
  "probabilities": [
53
  0.9964854118964818,
 
60
  {
61
  "id": "cooking-zh",
62
  "capability": "choice",
63
+ "inference_ms": 68.58845800161362,
64
  "status": "evaluated",
65
  "probabilities": [
66
  0.22242194924412478,
 
73
  {
74
  "id": "noul-positive",
75
  "capability": "noul",
76
+ "inference_ms": 48.34995899000205,
77
  "status": "evaluated",
78
  "probabilities": [
79
  0.04283435292391378,
 
86
  {
87
  "id": "noul-negative",
88
  "capability": "noul",
89
+ "inference_ms": 52.418874998693354,
90
  "status": "evaluated",
91
  "probabilities": [
92
  0.9887530718985025,
 
99
  {
100
  "id": "coding-zh",
101
  "capability": "choice",
102
+ "inference_ms": 70.75237500248477,
103
  "status": "evaluated",
104
  "probabilities": [
105
  0.9965400461728955,
 
112
  {
113
  "id": "cooking-en",
114
  "capability": "choice",
115
+ "inference_ms": 51.56712500320282,
116
  "status": "evaluated",
117
  "probabilities": [
118
  0.0008233419586070937,
 
121
  "decision": "cooking",
122
  "expected": "cooking",
123
  "correct": true
124
+ },
125
+ {
126
+ "id": "three-options",
127
+ "capability": "variable_options",
128
+ "inference_ms": 53.129124993574806,
129
+ "status": "evaluated",
130
+ "probabilities": [
131
+ 0.9929379587678295,
132
+ 0.0028259221336674827,
133
+ 0.004236119098503003
134
+ ],
135
+ "decision": "coding",
136
+ "expected": "coding",
137
+ "correct": true
138
+ },
139
+ {
140
+ "id": "one-option",
141
+ "capability": "variable_options",
142
+ "inference_ms": 49.474542000098154,
143
+ "status": "evaluated",
144
+ "probabilities": [
145
+ 1.0
146
+ ],
147
+ "decision": "coding",
148
+ "expected": "coding",
149
+ "correct": true
150
+ },
151
+ {
152
+ "id": "structured-json",
153
+ "capability": "structured_state",
154
+ "inference_ms": 69.43170799058862,
155
+ "status": "evaluated",
156
+ "probabilities": [
157
+ 0.9955305388600278,
158
+ 0.0022821065546790487,
159
+ 0.002187354585293287
160
+ ],
161
+ "decision": "coding",
162
+ "expected": "coding",
163
+ "correct": true
164
+ },
165
+ {
166
+ "id": "exact-129-tokens",
167
+ "capability": "variable_sequence_length",
168
+ "inference_ms": 141.63116700365208,
169
+ "status": "evaluated",
170
+ "probabilities": [
171
+ 0.6347567731488714,
172
+ 0.13812266903655834,
173
+ 0.22712055781457036
174
+ ],
175
+ "decision": "coding",
176
+ "expected": "coding",
177
+ "correct": true
178
+ },
179
+ {
180
+ "id": "exact-1024-tokens",
181
+ "capability": "variable_sequence_length",
182
+ "inference_ms": 968.6068749870174,
183
+ "status": "evaluated",
184
+ "probabilities": [
185
+ 0.4407633906736511,
186
+ 0.37238073879955924,
187
+ 0.18685587052678965
188
+ ],
189
+ "decision": "coding",
190
+ "expected": "coding",
191
+ "correct": true
192
+ },
193
+ {
194
+ "id": "exact-2048-tokens",
195
+ "capability": "variable_sequence_length",
196
+ "inference_ms": 2569.3836659920635,
197
+ "status": "evaluated",
198
+ "probabilities": [
199
+ 0.5216577502831695,
200
+ 0.34918566249456723,
201
+ 0.12915658722226325
202
+ ],
203
+ "decision": "coding",
204
+ "expected": "coding",
205
+ "correct": true
206
  }
207
  ],
208
  "candidate": {
 
212
  {
213
  "id": "coding-en",
214
  "capability": "choice",
215
+ "reference_ms": 51.31262500071898,
216
+ "converted_ms": 2374.499250014196,
217
  "passed": true,
218
  "numeric_passed": true,
219
  "decision_agreement": true,
220
+ "max_absolute_error": 5.427491678311469e-09,
221
+ "mean_absolute_error": 5.4274916770104265e-09,
222
  "reference_probabilities": [
223
  0.9964854118964818,
224
  0.003514588103518185
225
  ],
226
  "converted_probabilities": [
227
+ 0.9964854064689901,
228
+ 0.0035145935310098605
229
  ],
230
  "reference_decision": "coding",
231
  "converted_decision": "coding",
 
237
  {
238
  "id": "cooking-zh",
239
  "capability": "choice",
240
+ "reference_ms": 68.58845800161362,
241
+ "converted_ms": 91.65208299236838,
242
  "passed": true,
243
  "numeric_passed": true,
244
  "decision_agreement": true,
245
+ "max_absolute_error": 4.7419820070615515e-07,
246
+ "mean_absolute_error": 4.74198200650644e-07,
247
  "reference_probabilities": [
248
  0.22242194924412478,
249
  0.7775780507558753
250
  ],
251
  "converted_probabilities": [
252
+ 0.22242242344232538,
253
+ 0.7775775765576746
254
  ],
255
  "reference_decision": "cooking",
256
  "converted_decision": "cooking",
 
262
  {
263
  "id": "noul-positive",
264
  "capability": "noul",
265
+ "reference_ms": 48.34995899000205,
266
+ "converted_ms": 47.826540991081856,
267
  "passed": true,
268
  "numeric_passed": true,
269
  "decision_agreement": true,
270
+ "max_absolute_error": 1.3685062416612936e-07,
271
+ "mean_absolute_error": 1.3685062412102655e-07,
272
  "reference_probabilities": [
273
  0.04283435292391378,
274
  0.9571656470760861
275
  ],
276
  "converted_probabilities": [
277
+ 0.0428342160732897,
278
+ 0.9571657839267103
279
  ],
280
  "reference_decision": "yes",
281
  "converted_decision": "yes",
 
287
  {
288
  "id": "noul-negative",
289
  "capability": "noul",
290
+ "reference_ms": 52.418874998693354,
291
+ "converted_ms": 80.2467909961706,
292
  "passed": true,
293
  "numeric_passed": true,
294
  "decision_agreement": true,
295
+ "max_absolute_error": 2.6513179342962934e-09,
296
+ "mean_absolute_error": 2.6513178276107996e-09,
297
  "reference_probabilities": [
298
  0.9887530718985025,
299
  0.011246928101497617
300
  ],
301
  "converted_probabilities": [
302
+ 0.9887530745498202,
303
+ 0.011246925450179683
304
  ],
305
  "reference_decision": "no",
306
  "converted_decision": "no",
 
312
  {
313
  "id": "coding-zh",
314
  "capability": "choice",
315
+ "reference_ms": 70.75237500248477,
316
+ "converted_ms": 79.13508299679961,
317
  "passed": true,
318
  "numeric_passed": true,
319
  "decision_agreement": true,
320
+ "max_absolute_error": 8.426156995235135e-09,
321
+ "mean_absolute_error": 8.426156994801454e-09,
322
  "reference_probabilities": [
323
  0.9965400461728955,
324
  0.0034599538271045662
325
  ],
326
  "converted_probabilities": [
327
+ 0.9965400377467385,
328
+ 0.0034599622532615615
329
  ],
330
  "reference_decision": "coding",
331
  "converted_decision": "coding",
 
337
  {
338
  "id": "cooking-en",
339
  "capability": "choice",
340
+ "reference_ms": 51.56712500320282,
341
+ "converted_ms": 60.23716600611806,
342
  "passed": true,
343
  "numeric_passed": true,
344
  "decision_agreement": true,
345
+ "max_absolute_error": 1.5691086767584406e-09,
346
+ "mean_absolute_error": 1.5691086487860245e-09,
347
  "reference_probabilities": [
348
  0.0008233419586070937,
349
  0.9991766580413929
350
  ],
351
  "converted_probabilities": [
352
+ 0.0008233435277157704,
353
+ 0.9991766564722843
354
  ],
355
  "reference_decision": "cooking",
356
  "converted_decision": "cooking",
 
358
  "reference_correct": true,
359
  "converted_correct": true,
360
  "status": "passed"
361
+ },
362
+ {
363
+ "id": "three-options",
364
+ "capability": "variable_options",
365
+ "reference_ms": 53.129124993574806,
366
+ "converted_ms": 59.11033399752341,
367
+ "passed": true,
368
+ "numeric_passed": true,
369
+ "decision_agreement": true,
370
+ "max_absolute_error": 3.460630049190172e-08,
371
+ "mean_absolute_error": 2.3070867004286687e-08,
372
+ "reference_probabilities": [
373
+ 0.9929379587678295,
374
+ 0.0028259221336674827,
375
+ 0.004236119098503003
376
+ ],
377
+ "converted_probabilities": [
378
+ 0.99293799337413,
379
+ 0.002825909094017193,
380
+ 0.004236097531852771
381
+ ],
382
+ "reference_decision": "coding",
383
+ "converted_decision": "coding",
384
+ "expected": "coding",
385
+ "reference_correct": true,
386
+ "converted_correct": true,
387
+ "status": "passed"
388
+ },
389
+ {
390
+ "id": "one-option",
391
+ "capability": "variable_options",
392
+ "reference_ms": 49.474542000098154,
393
+ "converted_ms": 56.663583003683016,
394
+ "passed": true,
395
+ "numeric_passed": true,
396
+ "decision_agreement": true,
397
+ "max_absolute_error": 0.0,
398
+ "mean_absolute_error": 0.0,
399
+ "reference_probabilities": [
400
+ 1.0
401
+ ],
402
+ "converted_probabilities": [
403
+ 1.0
404
+ ],
405
+ "reference_decision": "coding",
406
+ "converted_decision": "coding",
407
+ "expected": "coding",
408
+ "reference_correct": true,
409
+ "converted_correct": true,
410
+ "status": "passed"
411
+ },
412
+ {
413
+ "id": "structured-json",
414
+ "capability": "structured_state",
415
+ "reference_ms": 69.43170799058862,
416
+ "converted_ms": 75.03466600610409,
417
+ "passed": true,
418
+ "numeric_passed": true,
419
+ "decision_agreement": true,
420
+ "max_absolute_error": 1.4141390947131072e-08,
421
+ "mean_absolute_error": 9.427594002917964e-09,
422
+ "reference_probabilities": [
423
+ 0.9955305388600278,
424
+ 0.0022821065546790487,
425
+ 0.002187354585293287
426
+ ],
427
+ "converted_probabilities": [
428
+ 0.9955305530014188,
429
+ 0.002282103594566704,
430
+ 0.0021873434040145696
431
+ ],
432
+ "reference_decision": "coding",
433
+ "converted_decision": "coding",
434
+ "expected": "coding",
435
+ "reference_correct": true,
436
+ "converted_correct": true,
437
+ "status": "passed"
438
+ },
439
+ {
440
+ "id": "exact-129-tokens",
441
+ "capability": "variable_sequence_length",
442
+ "reference_ms": 141.63116700365208,
443
+ "converted_ms": 134.83975001145154,
444
+ "passed": true,
445
+ "numeric_passed": true,
446
+ "decision_agreement": true,
447
+ "max_absolute_error": 7.795353781370196e-07,
448
+ "mean_absolute_error": 5.196902520913463e-07,
449
+ "reference_probabilities": [
450
+ 0.6347567731488714,
451
+ 0.13812266903655834,
452
+ 0.22712055781457036
453
+ ],
454
+ "converted_probabilities": [
455
+ 0.6347575526842495,
456
+ 0.13812251758558605,
457
+ 0.2271199297301645
458
+ ],
459
+ "reference_decision": "coding",
460
+ "converted_decision": "coding",
461
+ "expected": "coding",
462
+ "reference_correct": true,
463
+ "converted_correct": true,
464
+ "status": "passed"
465
+ },
466
+ {
467
+ "id": "exact-1024-tokens",
468
+ "capability": "variable_sequence_length",
469
+ "reference_ms": 968.6068749870174,
470
+ "converted_ms": 978.0740829883143,
471
+ "passed": true,
472
+ "numeric_passed": true,
473
+ "decision_agreement": true,
474
+ "max_absolute_error": 1.5057525626360757e-07,
475
+ "mean_absolute_error": 1.0038350412947909e-07,
476
+ "reference_probabilities": [
477
+ 0.4407633906736511,
478
+ 0.37238073879955924,
479
+ 0.18685587052678965
480
+ ],
481
+ "converted_probabilities": [
482
+ 0.4407635277058288,
483
+ 0.372380588224303,
484
+ 0.1868558840698681
485
+ ],
486
+ "reference_decision": "coding",
487
+ "converted_decision": "coding",
488
+ "expected": "coding",
489
+ "reference_correct": true,
490
+ "converted_correct": true,
491
+ "status": "passed"
492
+ },
493
+ {
494
+ "id": "exact-2048-tokens",
495
+ "capability": "variable_sequence_length",
496
+ "reference_ms": 2569.3836659920635,
497
+ "converted_ms": 2785.3902079950785,
498
+ "passed": true,
499
+ "numeric_passed": true,
500
+ "decision_agreement": true,
501
+ "max_absolute_error": 2.872474891835175e-07,
502
+ "mean_absolute_error": 1.914983261038413e-07,
503
+ "reference_probabilities": [
504
+ 0.5216577502831695,
505
+ 0.34918566249456723,
506
+ 0.12915658722226325
507
+ ],
508
+ "converted_probabilities": [
509
+ 0.5216574953587729,
510
+ 0.3491859497420564,
511
+ 0.12915655489917072
512
+ ],
513
+ "reference_decision": "coding",
514
+ "converted_decision": "coding",
515
+ "expected": "coding",
516
+ "reference_correct": true,
517
+ "converted_correct": true,
518
+ "status": "passed"
519
  }
520
  ]
521
  }
input_boundary.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "encoding_32768": "passed",
3
+ "encoding_32769": "rejected",
4
+ "truncation": false,
5
+ "native_32768_inference": "not tested"
6
+ }
manifest.json CHANGED
@@ -5,57 +5,63 @@
5
  "experimental": true,
6
  "complete_upstream_model": false,
7
  "sdk_integrated": false,
8
- "artifact_scope": "two-option-decision-only",
9
  "format": "Core AI .aimodel",
10
  "function": "main",
11
  "output": "logits",
12
- "sequence_length": 128,
 
13
  "batch_size": 1,
14
- "option_count": 2,
15
  "tested_device": "cpu",
16
  "inputs": {
17
  "input_ids": {
18
  "dtype": "int32",
19
  "shape": [
20
  1,
21
- 128
22
  ]
23
  },
24
  "attention_mask": {
25
  "dtype": "int32",
26
  "shape": [
27
  1,
28
- 128
29
  ]
30
  },
31
  "option_positions": {
32
  "dtype": "int32",
33
  "shape": [
34
  1,
35
- 2
36
  ]
37
  },
38
  "pool_mask": {
39
  "dtype": "float32",
40
  "shape": [
41
  1,
42
- 128
43
  ]
44
  }
45
  },
46
  "source_weights_sha256": "2294745881d9a01b7d049133b57e5fed2aec6ae1d34ef1ab9fa16f047fc35c8e",
47
  "artifact_files_sha256": {
48
- "model.aimodel/main.hash": "f854ff014ef231c04a22951298a0c5560c68d951123b7943a9d232b34db6bee2",
49
- "model.aimodel/main.mlirb": "e759a2108bf439ee87b10940f26d61a4078853e02f97d181d34165ae086b45ce",
50
- "model.aimodel/metadata.json": "e9969e2a029f38890ec12496d8e5c2d46f90b6a7554de00fd84bc40db73c884c"
 
 
 
 
 
51
  },
52
  "export_versions": {
53
- "torch": "2.13.0",
54
  "huggingface-hub": "1.33.0",
55
  "transformers": "5.17.0",
56
  "safetensors": "0.8.0",
57
- "coreai-torch": "0.4.2",
58
- "coreai-core": "1.0.0b2"
59
  },
60
  "evaluation_versions": {
61
  "torch": "2.14.0",
@@ -76,15 +82,21 @@
76
  "span_sets": "untested",
77
  "evidence": "untested",
78
  "router_task_adapters": "untested",
79
- "structured_state": "untested",
80
  "multiple_questions": "untested",
81
- "variable_sequence_length": "untested",
82
- "variable_options": "untested",
83
  "system_one_response": "untested",
84
  "domain_adapter": "untested",
85
  "factcheck_adapter": "untested",
86
  "feedback_adapter": "untested",
87
  "feedback_df_adapter": "untested"
88
  },
89
- "initial_build_validation": "Auto-device validation interrupted by resource limits; the same artifact subsequently passed CPU task evaluation."
 
 
 
 
 
 
90
  }
 
5
  "experimental": true,
6
  "complete_upstream_model": false,
7
  "sdk_integrated": false,
8
+ "artifact_scope": "decision-only",
9
  "format": "Core AI .aimodel",
10
  "function": "main",
11
  "output": "logits",
12
+ "sequence_length": 32768,
13
+ "dynamic_sequence_length": true,
14
  "batch_size": 1,
15
+ "option_count": "dynamic",
16
  "tested_device": "cpu",
17
  "inputs": {
18
  "input_ids": {
19
  "dtype": "int32",
20
  "shape": [
21
  1,
22
+ "sequence"
23
  ]
24
  },
25
  "attention_mask": {
26
  "dtype": "int32",
27
  "shape": [
28
  1,
29
+ "sequence"
30
  ]
31
  },
32
  "option_positions": {
33
  "dtype": "int32",
34
  "shape": [
35
  1,
36
+ "options"
37
  ]
38
  },
39
  "pool_mask": {
40
  "dtype": "float32",
41
  "shape": [
42
  1,
43
+ "sequence"
44
  ]
45
  }
46
  },
47
  "source_weights_sha256": "2294745881d9a01b7d049133b57e5fed2aec6ae1d34ef1ab9fa16f047fc35c8e",
48
  "artifact_files_sha256": {
49
+ "README.md": "aa3f9f987651372a96ba1ff633910ca5f499690e3f484ca790a94a247eb6ae19",
50
+ "assets/LICENSE": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4",
51
+ "assets/config.json": "a85b358ac11b71d780deb09e8a082efddc67938a2eb4fc16d68d8ae888c69d14",
52
+ "assets/tokenizer.json": "38734bb17ef19f84c67b5eb5692bea19801ffd02cb5c5ffc11429e76b0242e08",
53
+ "assets/tokenizer_config.json": "649781df90d51837dd6ba951a89fae07c985a9eacfd781d84002d480c3cb0b6e",
54
+ "model.aimodel/main.hash": "d1e3ae70a989d8d38ccae6513bf0b4085f751545cb6cd4efbd5373fa26f74fc4",
55
+ "model.aimodel/main.mlirb": "d7b63896067e505f67a03c1afe31e070b0a1f5d746e14d603d9130708798adad",
56
+ "model.aimodel/metadata.json": "a19b297a1df17828004220bd2a85bdd86f6630fd3edca8e9d3853366147d96b7"
57
  },
58
  "export_versions": {
59
+ "torch": "2.14.0",
60
  "huggingface-hub": "1.33.0",
61
  "transformers": "5.17.0",
62
  "safetensors": "0.8.0",
63
+ "coreai-torch": "0.4.3",
64
+ "coreai-core": "1.0.0b3"
65
  },
66
  "evaluation_versions": {
67
  "torch": "2.14.0",
 
82
  "span_sets": "untested",
83
  "evidence": "untested",
84
  "router_task_adapters": "untested",
85
+ "structured_state": "exercised-in-diagnostic",
86
  "multiple_questions": "untested",
87
+ "variable_sequence_length": "exercised-in-diagnostic",
88
+ "variable_options": "exercised-in-diagnostic",
89
  "system_one_response": "untested",
90
  "domain_adapter": "untested",
91
  "factcheck_adapter": "untested",
92
  "feedback_adapter": "untested",
93
  "feedback_df_adapter": "untested"
94
  },
95
+ "initial_build_validation": {
96
+ "samples": 1,
97
+ "max_absolute_error": 1.430511474609375e-06,
98
+ "atol": 0.02,
99
+ "rtol": 0.02,
100
+ "scope": "tensor smoke; task parity still required"
101
+ }
102
  }
model.aimodel/main.hash CHANGED
@@ -1 +1 @@
1
- �Y���9 @�ma��S�/�с�Ae�kE�
 
1
+ ׶8�~P_g�<�1�p����F�M`=�0p����
model.aimodel/main.mlirb CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:e759a2108bf439ee87b10940f26d61a4078853e02f97d181d34165ae086b45ce
3
- size 1522413878
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d7b63896067e505f67a03c1afe31e070b0a1f5d746e14d603d9130708798adad
3
+ size 1522191259
model.aimodel/metadata.json CHANGED
@@ -1,5 +1,5 @@
1
  {
2
- "assetVersion" : "2.0",
3
- "creationDate" : "20261116T131125Z",
4
- "producer" : "coreai-core 1.0.0b2"
5
  }
 
1
  {
2
+ "creationDate" : "20261116T155013Z",
3
+ "producer" : "coreai-core 1.0.0b3",
4
+ "assetVersion" : "2.0"
5
  }
package_smoke.json CHANGED
@@ -1,64 +1,148 @@
1
  {
2
  "status": "passed",
3
- "scope": "experimental two-option decision graph",
4
  "device": "cpu",
5
  "cases": [
6
  {
7
  "id": "coding-en",
 
 
8
  "decision": "coding",
9
- "max_probability_error": 2.504995120188036e-09,
10
  "probabilities": {
11
- "coding": 0.9964854093914867,
12
- "cooking": 0.003514590608513305
13
  }
14
  },
15
  {
16
  "id": "cooking-zh",
 
 
17
  "decision": "cooking",
18
- "max_probability_error": 5.566675266299725e-07,
19
  "probabilities": {
20
- "coding": 0.22242250591165136,
21
- "cooking": 0.7775774940883486
22
  }
23
  },
24
  {
25
  "id": "noul-positive",
 
 
26
  "decision": "yes",
27
- "max_probability_error": 7.820040781592041e-08,
28
  "probabilities": {
29
- "no": 0.042834274723506024,
30
- "yes": 0.9571657252764939
31
  }
32
  },
33
  {
34
  "id": "noul-negative",
 
 
35
  "decision": "no",
36
- "max_probability_error": 1.0605268028346382e-08,
37
  "probabilities": {
38
- "no": 0.9887530825037704,
39
- "yes": 0.011246917496229589
40
  }
41
  },
42
  {
43
  "id": "coding-zh",
 
 
44
  "decision": "coding",
45
- "max_probability_error": 1.1097869623277745e-08,
46
  "probabilities": {
47
- "coding": 0.9965400350750259,
48
- "cooking": 0.003459964924974146
49
  }
50
  },
51
  {
52
  "id": "cooking-en",
 
 
53
  "decision": "cooking",
54
- "max_probability_error": 1.3729699288285782e-09,
55
  "probabilities": {
56
- "coding": 0.0008233433315770225,
57
- "cooking": 0.999176656668423
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
58
  }
59
  }
60
  ],
61
- "invalid_requests_rejected": 5,
62
  "pytorch_imported": false,
63
- "requires_original_weights": false
 
 
64
  }
 
1
  {
2
  "status": "passed",
3
+ "scope": "dynamic decision graph",
4
  "device": "cpu",
5
  "cases": [
6
  {
7
  "id": "coding-en",
8
+ "sequence_length": 24,
9
+ "option_count": 2,
10
  "decision": "coding",
11
+ "max_probability_error": 5.427491678311469e-09,
12
  "probabilities": {
13
+ "coding": 0.9964854064689901,
14
+ "cooking": 0.0035145935310098605
15
  }
16
  },
17
  {
18
  "id": "cooking-zh",
19
+ "sequence_length": 36,
20
+ "option_count": 2,
21
  "decision": "cooking",
22
+ "max_probability_error": 4.7419820070615515e-07,
23
  "probabilities": {
24
+ "coding": 0.22242242344232538,
25
+ "cooking": 0.7775775765576746
26
  }
27
  },
28
  {
29
  "id": "noul-positive",
30
+ "sequence_length": 16,
31
+ "option_count": 2,
32
  "decision": "yes",
33
+ "max_probability_error": 1.3685062416612936e-07,
34
  "probabilities": {
35
+ "no": 0.0428342160732897,
36
+ "yes": 0.9571657839267103
37
  }
38
  },
39
  {
40
  "id": "noul-negative",
41
+ "sequence_length": 17,
42
+ "option_count": 2,
43
  "decision": "no",
44
+ "max_probability_error": 2.6513179342962934e-09,
45
  "probabilities": {
46
+ "no": 0.9887530745498202,
47
+ "yes": 0.011246925450179683
48
  }
49
  },
50
  {
51
  "id": "coding-zh",
52
+ "sequence_length": 42,
53
+ "option_count": 2,
54
  "decision": "coding",
55
+ "max_probability_error": 8.426156995235135e-09,
56
  "probabilities": {
57
+ "coding": 0.9965400377467385,
58
+ "cooking": 0.0034599622532615615
59
  }
60
  },
61
  {
62
  "id": "cooking-en",
63
+ "sequence_length": 26,
64
+ "option_count": 2,
65
  "decision": "cooking",
66
+ "max_probability_error": 1.5691086767584406e-09,
67
  "probabilities": {
68
+ "coding": 0.0008233435277157704,
69
+ "cooking": 0.9991766564722843
70
+ }
71
+ },
72
+ {
73
+ "id": "three-options",
74
+ "sequence_length": 31,
75
+ "option_count": 3,
76
+ "decision": "coding",
77
+ "max_probability_error": 3.460630049190172e-08,
78
+ "probabilities": {
79
+ "coding": 0.99293799337413,
80
+ "cooking": 0.002825909094017193,
81
+ "sports": 0.004236097531852771
82
+ }
83
+ },
84
+ {
85
+ "id": "one-option",
86
+ "sequence_length": 19,
87
+ "option_count": 1,
88
+ "decision": "coding",
89
+ "max_probability_error": 0.0,
90
+ "probabilities": {
91
+ "coding": 1.0
92
+ }
93
+ },
94
+ {
95
+ "id": "structured-json",
96
+ "sequence_length": 37,
97
+ "option_count": 3,
98
+ "decision": "coding",
99
+ "max_probability_error": 1.4141390947131072e-08,
100
+ "probabilities": {
101
+ "coding": 0.9955305530014188,
102
+ "cooking": 0.002282103594566704,
103
+ "sports": 0.0021873434040145696
104
+ }
105
+ },
106
+ {
107
+ "id": "exact-129-tokens",
108
+ "sequence_length": 129,
109
+ "option_count": 3,
110
+ "decision": "coding",
111
+ "max_probability_error": 7.795353781370196e-07,
112
+ "probabilities": {
113
+ "coding": 0.6347575526842495,
114
+ "cooking": 0.13812251758558605,
115
+ "sports": 0.2271199297301645
116
+ }
117
+ },
118
+ {
119
+ "id": "exact-1024-tokens",
120
+ "sequence_length": 1024,
121
+ "option_count": 3,
122
+ "decision": "coding",
123
+ "max_probability_error": 1.5057525626360757e-07,
124
+ "probabilities": {
125
+ "coding": 0.4407635277058288,
126
+ "cooking": 0.372380588224303,
127
+ "sports": 0.1868558840698681
128
+ }
129
+ },
130
+ {
131
+ "id": "exact-2048-tokens",
132
+ "sequence_length": 2048,
133
+ "option_count": 3,
134
+ "decision": "coding",
135
+ "max_probability_error": 2.872474891835175e-07,
136
+ "probabilities": {
137
+ "coding": 0.5216574953587729,
138
+ "cooking": 0.3491859497420564,
139
+ "sports": 0.12915655489917072
140
  }
141
  }
142
  ],
143
+ "invalid_requests_rejected": 3,
144
  "pytorch_imported": false,
145
+ "requires_original_weights": false,
146
+ "multiple_questions": "passed",
147
+ "list_criteria": "passed"
148
  }
smoke.py CHANGED
@@ -25,10 +25,14 @@ async def main() -> None:
25
  reference = np.array(case["reference_probabilities"])
26
  np.testing.assert_allclose(values, reference, atol=0.02, rtol=0.02)
27
  decision = max(probabilities, key=probabilities.__getitem__)
28
- assert decision == case["reference_decision"] == case["expected"]
 
 
29
  rows.append(
30
  {
31
  "id": case["id"],
 
 
32
  "decision": decision,
33
  "max_probability_error": float(np.max(np.abs(values - reference))),
34
  "probabilities": probabilities,
@@ -47,31 +51,46 @@ async def main() -> None:
47
  rejected = 0
48
  for state, question in [
49
  ("text", {"type": "span"}),
50
- ("text", {"type": "choice", "criteria": {"a": "a", "b": "b", "c": "c"}}),
51
- ("text", {"type": "choice", "criteria": {"a": "a"}}),
52
- ("word " * 500, first["payload"]["question"]),
53
- (["image"], first["payload"]["question"]),
54
  ]:
55
  try:
56
  model.pack(state, question)
57
  except ValueError:
58
  rejected += 1
59
- assert rejected == 5
60
  # Host adapter must preserve the public choice/noul response shape.
61
  answer = await model.predict(first["payload"]["state"], first["payload"]["question"])
62
  assert answer["type"] == "choice" and answer["choice"] == first["expected"]
63
  noul = cases[2]
64
  answer = await model.predict(noul["payload"]["state"], noul["payload"]["question"])
65
  assert answer["type"] == "noul" and 0 <= answer["noul"] <= 1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
66
  assert "torch" not in sys.modules
67
  report = {
68
  "status": "passed",
69
- "scope": "experimental two-option decision graph",
70
  "device": "cpu",
71
  "cases": rows,
72
  "invalid_requests_rejected": rejected,
73
  "pytorch_imported": False,
74
  "requires_original_weights": False,
 
 
75
  }
76
  (root / "package_smoke.json").write_text(json.dumps(report, indent=2) + "\n")
77
  print(json.dumps(report, indent=2))
 
25
  reference = np.array(case["reference_probabilities"])
26
  np.testing.assert_allclose(values, reference, atol=0.02, rtol=0.02)
27
  decision = max(probabilities, key=probabilities.__getitem__)
28
+ assert decision == case["reference_decision"]
29
+ if case.get("expected") is not None:
30
+ assert decision == case["expected"]
31
  rows.append(
32
  {
33
  "id": case["id"],
34
+ "sequence_length": int(arrays["attention_mask"].sum()),
35
+ "option_count": len(labels),
36
  "decision": decision,
37
  "max_probability_error": float(np.max(np.abs(values - reference))),
38
  "probabilities": probabilities,
 
51
  rejected = 0
52
  for state, question in [
53
  ("text", {"type": "span"}),
54
+ ("text", {"type": "choice", "criteria": {}}),
55
+ ("word " * (model.manifest["sequence_length"] + 1), first["payload"]["question"]),
 
 
56
  ]:
57
  try:
58
  model.pack(state, question)
59
  except ValueError:
60
  rejected += 1
61
+ assert rejected == 3
62
  # Host adapter must preserve the public choice/noul response shape.
63
  answer = await model.predict(first["payload"]["state"], first["payload"]["question"])
64
  assert answer["type"] == "choice" and answer["choice"] == first["expected"]
65
  noul = cases[2]
66
  answer = await model.predict(noul["payload"]["state"], noul["payload"]["question"])
67
  assert answer["type"] == "noul" and 0 <= answer["noul"] <= 1
68
+ questions = {
69
+ "topic": first["payload"]["question"],
70
+ "software": noul["payload"]["question"],
71
+ }
72
+ answers, usage = await model.predict_questions(first["payload"]["state"], questions)
73
+ assert list(answers) == list(questions) and usage > 0
74
+ for key, question in questions.items():
75
+ assert answers[key] == await model.predict(first["payload"]["state"], question)
76
+ list_question = {"type": "choice", "criteria": ["coding", "cooking", "sports"]}
77
+ map_question = {**list_question, "criteria": dict.fromkeys(list_question["criteria"])}
78
+ left, left_labels = model.pack(None, list_question)
79
+ right, right_labels = model.pack(None, map_question)
80
+ assert left_labels == right_labels
81
+ for name in left:
82
+ np.testing.assert_array_equal(left[name], right[name])
83
  assert "torch" not in sys.modules
84
  report = {
85
  "status": "passed",
86
+ "scope": "dynamic decision graph",
87
  "device": "cpu",
88
  "cases": rows,
89
  "invalid_requests_rejected": rejected,
90
  "pytorch_imported": False,
91
  "requires_original_weights": False,
92
+ "multiple_questions": "passed",
93
+ "list_criteria": "passed",
94
  }
95
  (root / "package_smoke.json").write_text(json.dumps(report, indent=2) + "\n")
96
  print(json.dumps(report, indent=2))
swift_fixture.json CHANGED
@@ -23,110 +23,6 @@
23
  5905,
24
  1517,
25
  30,
26
- 0,
27
- 0,
28
- 0,
29
- 0,
30
- 0,
31
- 0,
32
- 0,
33
- 0,
34
- 0,
35
- 0,
36
- 0,
37
- 0,
38
- 0,
39
- 0,
40
- 0,
41
- 0,
42
- 0,
43
- 0,
44
- 0,
45
- 0,
46
- 0,
47
- 0,
48
- 0,
49
- 0,
50
- 0,
51
- 0,
52
- 0,
53
- 0,
54
- 0,
55
- 0,
56
- 0,
57
- 0,
58
- 0,
59
- 0,
60
- 0,
61
- 0,
62
- 0,
63
- 0,
64
- 0,
65
- 0,
66
- 0,
67
- 0,
68
- 0,
69
- 0,
70
- 0,
71
- 0,
72
- 0,
73
- 0,
74
- 0,
75
- 0,
76
- 0,
77
- 0,
78
- 0,
79
- 0,
80
- 0,
81
- 0,
82
- 0,
83
- 0,
84
- 0,
85
- 0,
86
- 0,
87
- 0,
88
- 0,
89
- 0,
90
- 0,
91
- 0,
92
- 0,
93
- 0,
94
- 0,
95
- 0,
96
- 0,
97
- 0,
98
- 0,
99
- 0,
100
- 0,
101
- 0,
102
- 0,
103
- 0,
104
- 0,
105
- 0,
106
- 0,
107
- 0,
108
- 0,
109
- 0,
110
- 0,
111
- 0,
112
- 0,
113
- 0,
114
- 0,
115
- 0,
116
- 0,
117
- 0,
118
- 0,
119
- 0,
120
- 0,
121
- 0,
122
- 0,
123
- 0,
124
- 0,
125
- 0,
126
- 0,
127
- 0,
128
- 0,
129
- 0,
130
  0
131
  ],
132
  "attention_mask": [
@@ -153,111 +49,7 @@
153
  1,
154
  1,
155
  1,
156
- 1,
157
- 0,
158
- 0,
159
- 0,
160
- 0,
161
- 0,
162
- 0,
163
- 0,
164
- 0,
165
- 0,
166
- 0,
167
- 0,
168
- 0,
169
- 0,
170
- 0,
171
- 0,
172
- 0,
173
- 0,
174
- 0,
175
- 0,
176
- 0,
177
- 0,
178
- 0,
179
- 0,
180
- 0,
181
- 0,
182
- 0,
183
- 0,
184
- 0,
185
- 0,
186
- 0,
187
- 0,
188
- 0,
189
- 0,
190
- 0,
191
- 0,
192
- 0,
193
- 0,
194
- 0,
195
- 0,
196
- 0,
197
- 0,
198
- 0,
199
- 0,
200
- 0,
201
- 0,
202
- 0,
203
- 0,
204
- 0,
205
- 0,
206
- 0,
207
- 0,
208
- 0,
209
- 0,
210
- 0,
211
- 0,
212
- 0,
213
- 0,
214
- 0,
215
- 0,
216
- 0,
217
- 0,
218
- 0,
219
- 0,
220
- 0,
221
- 0,
222
- 0,
223
- 0,
224
- 0,
225
- 0,
226
- 0,
227
- 0,
228
- 0,
229
- 0,
230
- 0,
231
- 0,
232
- 0,
233
- 0,
234
- 0,
235
- 0,
236
- 0,
237
- 0,
238
- 0,
239
- 0,
240
- 0,
241
- 0,
242
- 0,
243
- 0,
244
- 0,
245
- 0,
246
- 0,
247
- 0,
248
- 0,
249
- 0,
250
- 0,
251
- 0,
252
- 0,
253
- 0,
254
- 0,
255
- 0,
256
- 0,
257
- 0,
258
- 0,
259
- 0,
260
- 0
261
  ],
262
  "option_positions": [
263
  7,
@@ -287,115 +79,11 @@
287
  1.0,
288
  1.0,
289
  1.0,
290
- 0.0,
291
- 0.0,
292
- 0.0,
293
- 0.0,
294
- 0.0,
295
- 0.0,
296
- 0.0,
297
- 0.0,
298
- 0.0,
299
- 0.0,
300
- 0.0,
301
- 0.0,
302
- 0.0,
303
- 0.0,
304
- 0.0,
305
- 0.0,
306
- 0.0,
307
- 0.0,
308
- 0.0,
309
- 0.0,
310
- 0.0,
311
- 0.0,
312
- 0.0,
313
- 0.0,
314
- 0.0,
315
- 0.0,
316
- 0.0,
317
- 0.0,
318
- 0.0,
319
- 0.0,
320
- 0.0,
321
- 0.0,
322
- 0.0,
323
- 0.0,
324
- 0.0,
325
- 0.0,
326
- 0.0,
327
- 0.0,
328
- 0.0,
329
- 0.0,
330
- 0.0,
331
- 0.0,
332
- 0.0,
333
- 0.0,
334
- 0.0,
335
- 0.0,
336
- 0.0,
337
- 0.0,
338
- 0.0,
339
- 0.0,
340
- 0.0,
341
- 0.0,
342
- 0.0,
343
- 0.0,
344
- 0.0,
345
- 0.0,
346
- 0.0,
347
- 0.0,
348
- 0.0,
349
- 0.0,
350
- 0.0,
351
- 0.0,
352
- 0.0,
353
- 0.0,
354
- 0.0,
355
- 0.0,
356
- 0.0,
357
- 0.0,
358
- 0.0,
359
- 0.0,
360
- 0.0,
361
- 0.0,
362
- 0.0,
363
- 0.0,
364
- 0.0,
365
- 0.0,
366
- 0.0,
367
- 0.0,
368
- 0.0,
369
- 0.0,
370
- 0.0,
371
- 0.0,
372
- 0.0,
373
- 0.0,
374
- 0.0,
375
- 0.0,
376
- 0.0,
377
- 0.0,
378
- 0.0,
379
- 0.0,
380
- 0.0,
381
- 0.0,
382
- 0.0,
383
- 0.0,
384
- 0.0,
385
- 0.0,
386
- 0.0,
387
- 0.0,
388
- 0.0,
389
- 0.0,
390
- 0.0,
391
- 0.0,
392
- 0.0,
393
- 0.0,
394
  0.0
395
  ],
396
  "expected": [
397
- 0.9964854093914867,
398
- 0.003514590608513305
399
  ],
400
  "labels": [
401
  "coding",
 
23
  5905,
24
  1517,
25
  30,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
  0
27
  ],
28
  "attention_mask": [
 
49
  1,
50
  1,
51
  1,
52
+ 1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
53
  ],
54
  "option_positions": [
55
  7,
 
79
  1.0,
80
  1.0,
81
  1.0,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
82
  0.0
83
  ],
84
  "expected": [
85
+ 0.9964854064689901,
86
+ 0.0035145935310098605
87
  ],
88
  "labels": [
89
  "coding",
swift_smoke.json CHANGED
@@ -7,7 +7,7 @@
7
  "maximum_probability_error_vs_python" : 5.9604644775390625e-08,
8
  "probabilities" : [
9
  0.99648535251617432,
10
- 0.0035145911388099194
11
  ],
12
  "swift_native_coreai" : true
13
  }
 
7
  "maximum_probability_error_vs_python" : 5.9604644775390625e-08,
8
  "probabilities" : [
9
  0.99648535251617432,
10
+ 0.0035145930014550686
11
  ],
12
  "swift_native_coreai" : true
13
  }
validation_cases.json CHANGED
The diff for this file is too large to render. See raw diff
 
vela_coreai.py CHANGED
@@ -49,27 +49,23 @@ class VelaCoreAI:
49
  return (
50
  value
51
  if isinstance(value, str)
52
- else json.dumps(value, ensure_ascii=False, separators=(",", ":"), allow_nan=False)
53
  )
54
 
55
  def pack(self, state: Any, question: Any) -> tuple[dict[str, np.ndarray[Any, Any]], list[str]]:
56
- if not isinstance(state, (str, dict)):
57
- raise ValueError("state must be text or a JSON object; media is not supported")
58
- if not isinstance(question, dict) or set(question) - {"type", "instructions", "criteria"}:
59
- raise ValueError("Provide one choice/noul question; unknown fields are rejected")
60
  kind = question.get("type")
61
  options: dict[str, Any]
62
  if kind == "noul":
63
- if "criteria" in question:
64
- raise ValueError("noul uses the fixed no/yes options")
65
  options = {"no": "", "yes": ""}
66
  elif kind == "choice":
67
  criteria = question.get("criteria")
68
- if not isinstance(criteria, dict) or len(criteria) != 2:
69
- raise ValueError("This experimental graph requires exactly two choice options")
70
- options = criteria
71
- if any(not isinstance(k, str) or not k for k in options):
72
- raise ValueError("Option names must be nonempty strings")
73
  else:
74
  raise ValueError("Only choice and noul are included in this experimental artifact")
75
 
@@ -86,12 +82,15 @@ class VelaCoreAI:
86
  ids += [markers["[O]"], *encode(f"{name}: {text}" if text else name)]
87
  ids.append(markers["[SEP_SCHEMA]"])
88
  start = len(ids)
89
- ids += [markers["[SEG_user]"], *encode(self._text(state))]
90
  end = len(ids)
91
  ids.append(self.config["eos_token_id"])
92
- width = self.manifest["sequence_length"]
93
- if len(ids) > width:
94
- raise ValueError(f"Prompt needs {len(ids)} tokens; capacity is {width}; no truncation")
 
 
 
95
  input_ids = np.full((1, width), self.config["pad_token_id"], dtype=np.int32)
96
  input_ids[0, : len(ids)] = ids
97
  attention_mask = np.zeros((1, width), dtype=np.int32)
@@ -117,7 +116,7 @@ class VelaCoreAI:
117
  try:
118
  output = await self._function({k: NDArray(v) for k, v in arrays.items()})
119
  logits = np.asarray(output[self.manifest["output"]].numpy(), dtype=np.float64)
120
- if logits.shape != (1, 2) or not np.isfinite(logits).all():
121
  raise FloatingPointError("Invalid Core AI logits")
122
  probabilities = np.exp(logits[0] - logits[0].max())
123
  probabilities /= probabilities.sum()
@@ -134,3 +133,14 @@ class VelaCoreAI:
134
  "choice": max(probabilities, key=probabilities.__getitem__),
135
  "probabilities": probabilities,
136
  }
 
 
 
 
 
 
 
 
 
 
 
 
49
  return (
50
  value
51
  if isinstance(value, str)
52
+ else json.dumps(value, ensure_ascii=False, separators=(",", ":"))
53
  )
54
 
55
  def pack(self, state: Any, question: Any) -> tuple[dict[str, np.ndarray[Any, Any]], list[str]]:
56
+ if not isinstance(question, dict):
57
+ raise ValueError("Provide a choice/noul question object")
 
 
58
  kind = question.get("type")
59
  options: dict[str, Any]
60
  if kind == "noul":
 
 
61
  options = {"no": "", "yes": ""}
62
  elif kind == "choice":
63
  criteria = question.get("criteria")
64
+ if isinstance(criteria, (list, tuple)):
65
+ criteria = {str(value): None for value in criteria}
66
+ if not isinstance(criteria, dict) or not criteria:
67
+ raise ValueError("choice requires a nonempty criteria mapping")
68
+ options = {str(k): v for k, v in criteria.items()}
69
  else:
70
  raise ValueError("Only choice and noul are included in this experimental artifact")
71
 
 
82
  ids += [markers["[O]"], *encode(f"{name}: {text}" if text else name)]
83
  ids.append(markers["[SEP_SCHEMA]"])
84
  start = len(ids)
85
+ ids += [markers["[SEG_user]"], *encode(self._text(state if state is not None else ""))]
86
  end = len(ids)
87
  ids.append(self.config["eos_token_id"])
88
+ capacity = self.manifest["sequence_length"]
89
+ if len(ids) > capacity:
90
+ raise ValueError(
91
+ f"Prompt needs {len(ids)} tokens; capacity is {capacity}; no truncation"
92
+ )
93
+ width = len(ids) if self.manifest.get("dynamic_sequence_length") else capacity
94
  input_ids = np.full((1, width), self.config["pad_token_id"], dtype=np.int32)
95
  input_ids[0, : len(ids)] = ids
96
  attention_mask = np.zeros((1, width), dtype=np.int32)
 
116
  try:
117
  output = await self._function({k: NDArray(v) for k, v in arrays.items()})
118
  logits = np.asarray(output[self.manifest["output"]].numpy(), dtype=np.float64)
119
+ if logits.shape != (1, len(labels)) or not np.isfinite(logits).all():
120
  raise FloatingPointError("Invalid Core AI logits")
121
  probabilities = np.exp(logits[0] - logits[0].max())
122
  probabilities /= probabilities.sum()
 
133
  "choice": max(probabilities, key=probabilities.__getitem__),
134
  "probabilities": probabilities,
135
  }
136
+
137
+ async def predict_questions(
138
+ self, state: Any, questions: dict[str, Any]
139
+ ) -> tuple[dict[str, Any], int]:
140
+ """Preserve DecisionEncoder.predict's question mapping and token usage."""
141
+ packed = {key: self.pack(state, question) for key, question in questions.items()}
142
+ answers = {}
143
+ for key, question in questions.items():
144
+ answers[key] = await self.predict(state, question)
145
+ tokens = sum(int(arrays["attention_mask"].sum()) for arrays, _ in packed.values())
146
+ return answers, tokens
verify_coreai.swift CHANGED
@@ -34,18 +34,18 @@ struct Fixture: Decodable {
34
  ("attention_mask", fixture.attention_mask),
35
  ("option_positions", fixture.option_positions),
36
  ] {
37
- guard case .ndArray(let shape) = descriptor.inputDescriptor(of: name) else {
38
  fatalError("Missing input \(name)")
39
  }
40
- var array = NDArray(descriptor: shape)
41
  var view = array.mutableView(as: Int32.self)
42
  view.copyElements(fromContentsOf: values)
43
  inputs[name] = array
44
  }
45
- guard case .ndArray(let shape) = descriptor.inputDescriptor(of: "pool_mask") else {
46
  fatalError("Missing pool_mask")
47
  }
48
- var pool = NDArray(descriptor: shape)
49
  var poolView = pool.mutableView(as: Float.self)
50
  poolView.copyElements(fromContentsOf: fixture.pool_mask)
51
  inputs["pool_mask"] = pool
@@ -54,7 +54,7 @@ struct Fixture: Decodable {
54
  fatalError("Missing logits")
55
  }
56
  let logits: [Float] = output.view(as: Float.self).withUnsafePointer { ptr, _, _ in
57
- Array(UnsafeBufferPointer(start: ptr, count: 2))
58
  }
59
  guard logits.allSatisfy({ $0.isFinite }) else { fatalError("Nonfinite logits") }
60
  let maximum = logits.max()!
 
34
  ("attention_mask", fixture.attention_mask),
35
  ("option_positions", fixture.option_positions),
36
  ] {
37
+ guard case .ndArray = descriptor.inputDescriptor(of: name) else {
38
  fatalError("Missing input \(name)")
39
  }
40
+ var array = NDArray(shape: [1, values.count], scalarType: .int32)
41
  var view = array.mutableView(as: Int32.self)
42
  view.copyElements(fromContentsOf: values)
43
  inputs[name] = array
44
  }
45
+ guard case .ndArray = descriptor.inputDescriptor(of: "pool_mask") else {
46
  fatalError("Missing pool_mask")
47
  }
48
+ var pool = NDArray(shape: [1, fixture.pool_mask.count], scalarType: .float32)
49
  var poolView = pool.mutableView(as: Float.self)
50
  poolView.copyElements(fromContentsOf: fixture.pool_mask)
51
  inputs["pool_mask"] = pool
 
54
  fatalError("Missing logits")
55
  }
56
  let logits: [Float] = output.view(as: Float.self).withUnsafePointer { ptr, _, _ in
57
+ Array(UnsafeBufferPointer(start: ptr, count: fixture.labels.count))
58
  }
59
  guard logits.allSatisfy({ $0.isFinite }) else { fatalError("Nonfinite logits") }
60
  let maximum = logits.max()!