Replace fixed graph with dynamic sequence and option dimensions
Browse filesRetain upstream 32768-token input bound without truncation. CPU parity passed 12 cases, including exact 1024/2048 tokens, 1/2/3 options, and JSON. Standalone Python and native Swift verified. Native 32K inference remains untested; decision adapters remain outside this export. English and Chinese model cards describe the validated scope.
- README.md +24 -21
- README.zh-CN.md +23 -21
- SHA256SUMS +16 -15
- environment.json +2 -2
- evaluation.json +286 -46
- input_boundary.json +6 -0
- manifest.json +29 -17
- model.aimodel/main.hash +1 -1
- model.aimodel/main.mlirb +2 -2
- model.aimodel/metadata.json +3 -3
- package_smoke.json +105 -21
- smoke.py +26 -7
- swift_fixture.json +3 -315
- swift_smoke.json +1 -1
- validation_cases.json +0 -0
- vela_coreai.py +27 -17
- verify_coreai.swift +5 -5
README.md
CHANGED
|
@@ -18,20 +18,21 @@ tags:
|
|
| 18 |
English | [简体中文](README.zh-CN.md)
|
| 19 |
|
| 20 |
An Apple Core AI conversion of [vllm-sr/vela3-fast](https://huggingface.co/vllm-sr/vela3-fast) by hugging-mac.
|
| 21 |
-
This release contains the validated **choice / noul decision backbone**: one sample
|
|
|
|
| 22 |
**It is an experimental subset, not the complete Vela3-fast model with all task adapters.**
|
| 23 |
|
| 24 |
Neural inference runs through Core AI, with host-side tokenization and response formatting. The model does not generate text.
|
| 25 |
|
| 26 |
## Supported contract
|
| 27 |
|
| 28 |
-
- `choice`:
|
| 29 |
- `noul`: fixed `no` / `yes` options; returns `p(yes)`.
|
| 30 |
-
- The input text, question, option descriptions, and special markers together must fit within **
|
| 31 |
-
-
|
| 32 |
- CPU execution is validated. Earlier automatic-device validation did not finish because of resource pressure on the test host; no GPU/ANE performance is claimed.
|
| 33 |
- Original FP32 decision weights, with no quantization, training, or fine-tuning.
|
| 34 |
-
- Router, span, relation, NER, evidence, domain/factcheck/feedback, and other upstream adapters are not included.
|
| 35 |
- This release does not provide complete `hugging_mac_sdk` registration or production acceptance. Probabilities are relative to the supplied options, not calibrated confidence scores.
|
| 36 |
|
| 37 |
## Python usage
|
|
@@ -74,7 +75,7 @@ asyncio.run(main())
|
|
| 74 |
|
| 75 |
`predict` prepares token IDs, marker positions, and the pooling mask before calling Core AI.
|
| 76 |
Each instance accepts one inference request at a time; await completion before making another call.
|
| 77 |
-
Unknown tasks,
|
| 78 |
|
| 79 |
Run the standalone package checks:
|
| 80 |
|
|
@@ -82,8 +83,8 @@ Run the standalone package checks:
|
|
| 82 |
USE_OS_COREAI=1 uv run --locked python smoke.py
|
| 83 |
```
|
| 84 |
|
| 85 |
-
This compares
|
| 86 |
-
and verifies rejection of
|
| 87 |
|
| 88 |
## Swift usage
|
| 89 |
|
|
@@ -117,7 +118,7 @@ var result = try await function.run(inputs: inputs)
|
|
| 117 |
guard let logits = result.remove("logits")?.ndArray else {
|
| 118 |
fatalError("Missing logits")
|
| 119 |
}
|
| 120 |
-
// Apply stable softmax to the
|
| 121 |
```
|
| 122 |
|
| 123 |
For new text, reuse `tokenizer.json`, `decision.json`, and the encoding rules in `vela_coreai.py`.
|
|
@@ -125,15 +126,15 @@ Tokenizing only the input body does not produce valid model inputs. The bundled
|
|
| 125 |
|
| 126 |
## Tensor interface
|
| 127 |
|
| 128 |
-
Entry point: `main`. No persistent recurrent/KV state.
|
| 129 |
|
| 130 |
| Name | Type | Shape | Meaning |
|
| 131 |
| --- | --- | --- | --- |
|
| 132 |
-
| `input_ids` | int32 | `[1,
|
| 133 |
-
| `attention_mask` | int32 | `[1,
|
| 134 |
-
| `option_positions` | int32 | `[1,
|
| 135 |
-
| `pool_mask` | float32 | `[1,
|
| 136 |
-
| `logits` (output) | float32 | `[1,
|
| 137 |
|
| 138 |
Upstream sequence format:
|
| 139 |
|
|
@@ -144,13 +145,15 @@ Upstream sequence format:
|
|
| 144 |
|
| 145 |
## Validation and limitations
|
| 146 |
|
| 147 |
-
- **
|
| 148 |
-
- Maximum absolute probability error was approximately **
|
| 149 |
-
- The standalone Python package passed all
|
| 150 |
- Native Swift CPU results are in `swift_smoke.json`; standalone Python results are in `package_smoke.json`.
|
| 151 |
-
-
|
| 152 |
-
- iPhone/iPad, iOS AOT, automatic device selection, GPU/ANE placement,
|
| 153 |
-
-
|
|
|
|
|
|
|
| 154 |
|
| 155 |
Full cases and probabilities: `validation_cases.json` and `evaluation.json`.
|
| 156 |
`SHA256SUMS` covers the published files. Original weight copies, development caches, and virtual environments are not uploaded.
|
|
|
|
| 18 |
English | [简体中文](README.zh-CN.md)
|
| 19 |
|
| 20 |
An Apple Core AI conversion of [vllm-sr/vela3-fast](https://huggingface.co/vllm-sr/vela3-fast) by hugging-mac.
|
| 21 |
+
This release contains the validated **choice / noul decision backbone**: one sample per graph call, dynamic sequence length, and a dynamic number of options.
|
| 22 |
+
The export retains the upstream **32,768-token** input bound; actual CPU parity tests currently cover lengths through **2,048 tokens**, including exactly 1,024.
|
| 23 |
**It is an experimental subset, not the complete Vela3-fast model with all task adapters.**
|
| 24 |
|
| 25 |
Neural inference runs through Core AI, with host-side tokenization and response formatting. The model does not generate text.
|
| 26 |
|
| 27 |
## Supported contract
|
| 28 |
|
| 29 |
+
- `choice`: one or more ordered options; returns their probabilities and the selected option.
|
| 30 |
- `noul`: fixed `no` / `yes` options; returns `p(yes)`.
|
| 31 |
+
- The input text, question, option descriptions, and special markers together must fit within the upstream **32,768-token** bound. Longer inputs raise an error; they are not truncated.
|
| 32 |
+
- Each graph call has batch size 1. `predict_questions(state, questions)` accepts a mapping of questions and returns `(answers, input_tokens)`, matching the upstream decision API; questions execute sequentially. The adapter preserves upstream JSON serialization and accepts mapping or list choice criteria.
|
| 33 |
- CPU execution is validated. Earlier automatic-device validation did not finish because of resource pressure on the test host; no GPU/ANE performance is claimed.
|
| 34 |
- Original FP32 decision weights, with no quantization, training, or fine-tuning.
|
| 35 |
+
- Router, span, relation, NER, evidence, domain/factcheck/feedback, and other upstream adapters are not included. Full `system_one` adapter routing is not implemented by this decision-only artifact. Untested capabilities are not known to be unconvertible.
|
| 36 |
- This release does not provide complete `hugging_mac_sdk` registration or production acceptance. Probabilities are relative to the supplied options, not calibrated confidence scores.
|
| 37 |
|
| 38 |
## Python usage
|
|
|
|
| 75 |
|
| 76 |
`predict` prepares token IDs, marker positions, and the pooling mask before calling Core AI.
|
| 77 |
Each instance accepts one inference request at a time; await completion before making another call.
|
| 78 |
+
Unknown tasks, empty option sets, and inputs exceeding the token limit raise errors.
|
| 79 |
|
| 80 |
Run the standalone package checks:
|
| 81 |
|
|
|
|
| 83 |
USE_OS_COREAI=1 uv run --locked python smoke.py
|
| 84 |
```
|
| 85 |
|
| 86 |
+
This compares 12 cases with upstream reference probabilities, checks exact input-tensor equality against upstream encoding,
|
| 87 |
+
and verifies rejection of three invalid-request categories. It writes `package_smoke.json` and `swift_fixture.json` for the Swift example.
|
| 88 |
|
| 89 |
## Swift usage
|
| 90 |
|
|
|
|
| 118 |
guard let logits = result.remove("logits")?.ndArray else {
|
| 119 |
fatalError("Missing logits")
|
| 120 |
}
|
| 121 |
+
// Apply stable softmax to the output logits and interpret them in input-option order.
|
| 122 |
```
|
| 123 |
|
| 124 |
For new text, reuse `tokenizer.json`, `decision.json`, and the encoding rules in `vela_coreai.py`.
|
|
|
|
| 126 |
|
| 127 |
## Tensor interface
|
| 128 |
|
| 129 |
+
Entry point: `main`. No persistent recurrent/KV state. Sequence and option axes are dynamic; inputs use their actual encoded length without fixed padding.
|
| 130 |
|
| 131 |
| Name | Type | Shape | Meaning |
|
| 132 |
| --- | --- | --- | --- |
|
| 133 |
+
| `input_ids` | int32 | `[1,sequence]` | Complete question/options/body sequence |
|
| 134 |
+
| `attention_mask` | int32 | `[1,sequence]` | 1 for valid tokens, 0 for padding |
|
| 135 |
+
| `option_positions` | int32 | `[1,options]` | Positions of the `[O]` markers |
|
| 136 |
+
| `pool_mask` | float32 | `[1,sequence]` | 1 from `[SEG_user]` through the end of the body, excluding EOS |
|
| 137 |
+
| `logits` (output) | float32 | `[1,options]` | Unnormalized scores for the one or more ordered options |
|
| 138 |
|
| 139 |
Upstream sequence format:
|
| 140 |
|
|
|
|
| 145 |
|
| 146 |
## Validation and limitations
|
| 147 |
|
| 148 |
+
- **12/12** text cases matched the original model's decisions and the expected answers.
|
| 149 |
+
- Maximum absolute probability error was approximately **7.80e-7**. Validation uses `atol=0.02, rtol=0.02` and also requires decision agreement.
|
| 150 |
+
- The standalone Python package passed all 12 cases without PyTorch installed, with exact input-tensor parity against upstream.
|
| 151 |
- Native Swift CPU results are in `swift_smoke.json`; standalone Python results are in `package_smoke.json`.
|
| 152 |
+
- These handcrafted examples are not a general accuracy benchmark. Coverage includes a JSON object, 1/2/3 options, and exact 129/1,024/2,048-token inputs.
|
| 153 |
+
- iPhone/iPad, iOS AOT, automatic device selection, GPU/ANE placement, lengths above 2,048 tokens, and other upstream tasks have not been validated.
|
| 154 |
+
- The new dynamic artifact passed initial CPU tensor validation and subsequent task evaluation. Tool versions are recorded in `manifest.json` and `environment.json`.
|
| 155 |
+
- The 32K bound is retained in the export and host adapter, but end-to-end 32K inference has not been validated on this 8 GB test machine. Longer inputs require additional memory; 2,048 is a tested length, not a new hard limit.
|
| 156 |
+
- This revision replaces the earlier fixed 128-token, two-option export. The adapters listed above remain outside the current decision graph.
|
| 157 |
|
| 158 |
Full cases and probabilities: `validation_cases.json` and `evaluation.json`.
|
| 159 |
`SHA256SUMS` covers the published files. Original weight copies, development caches, and virtual environments are not uploaded.
|
README.zh-CN.md
CHANGED
|
@@ -18,7 +18,8 @@ tags:
|
|
| 18 |
[English](README.md) | 简体中文
|
| 19 |
|
| 20 |
由 hugging-mac 从 [vllm-sr/vela3-fast](https://huggingface.co/vllm-sr/vela3-fast) 转换的 Apple Core AI 模型。
|
| 21 |
-
本仓库发布的是经过验证的 **choice / noul 决策主干**:
|
|
|
|
| 22 |
**这是实验性子集,不是包含所有任务 adapter 的完整 Vela3-fast。**
|
| 23 |
|
| 24 |
This is an experimental, fixed-shape decision-backbone export, not a complete replacement for upstream Vela3-fast.
|
|
@@ -26,14 +27,14 @@ It runs neural inference through Core AI with host-side tokenization and respons
|
|
| 26 |
|
| 27 |
## 支持范围 / Contract
|
| 28 |
|
| 29 |
-
- `choice`:
|
| 30 |
- `noul`:固定 `no` / `yes` 两个选项,返回 `p(yes)`。
|
| 31 |
-
- 输入正文、问题、选项描述和特殊标记合计不超过 **
|
| 32 |
-
- 每次
|
| 33 |
- CPU 模式已验证。自动设备模式此前因测试主机资源压力未完成验证,不宣称 GPU/ANE 性能。
|
| 34 |
- 原始 FP32 决策权重,无量化、训练或微调。
|
| 35 |
- 原模型的 router、span、relation、NER、evidence、domain/factcheck/feedback 等 adapter 不包含在此制品中;
|
| 36 |
-
|
| 37 |
- 不提供完整 `hugging_mac_sdk` 模型注册或生产验收结论。概率是给定选项间的相对值,不是校准置信度。
|
| 38 |
|
| 39 |
## Python 基本调用
|
|
@@ -75,7 +76,7 @@ asyncio.run(main())
|
|
| 75 |
```
|
| 76 |
|
| 77 |
`predict` 执行 tokenizer、标记位置和 pooling mask 的准备,再调用 Core AI。
|
| 78 |
-
同一个实例一次只接收一个推理请求;请等待前一次调用完成。未知任务、
|
| 79 |
|
| 80 |
运行独立包验证:
|
| 81 |
|
|
@@ -83,7 +84,7 @@ asyncio.run(main())
|
|
| 83 |
USE_OS_COREAI=1 uv run --locked python smoke.py
|
| 84 |
```
|
| 85 |
|
| 86 |
-
该脚本比较
|
| 87 |
并生成 `package_smoke.json` 和用于 Swift 的 `swift_fixture.json`。
|
| 88 |
|
| 89 |
## Swift 基本调用
|
|
@@ -118,7 +119,7 @@ var result = try await function.run(inputs: inputs)
|
|
| 118 |
guard let logits = result.remove("logits")?.ndArray else {
|
| 119 |
fatalError("Missing logits")
|
| 120 |
}
|
| 121 |
-
// 对
|
| 122 |
```
|
| 123 |
|
| 124 |
对任意新文本调用时,必须复用本仓库 `tokenizer.json`、`decision.json` 和 `vela_coreai.py` 的编码规则;
|
|
@@ -126,15 +127,15 @@ guard let logits = result.remove("logits")?.ndArray else {
|
|
| 126 |
|
| 127 |
## 张量接口
|
| 128 |
|
| 129 |
-
入口:`main`,无持久 recurrent/KV 状态。
|
| 130 |
|
| 131 |
| 名称 | 类型 | 形状 | 含义 |
|
| 132 |
| --- | --- | --- | --- |
|
| 133 |
-
| `input_ids` | int32 | `[1,
|
| 134 |
-
| `attention_mask` | int32 | `[1,
|
| 135 |
-
| `option_positions` | int32 | `[1,
|
| 136 |
-
| `pool_mask` | float32 | `[1,
|
| 137 |
-
| `logits`(输出) | float32 | `[1,
|
| 138 |
|
| 139 |
上游序列格式:
|
| 140 |
|
|
@@ -145,14 +146,15 @@ guard let logits = result.remove("logits")?.ndArray else {
|
|
| 145 |
|
| 146 |
## 验证记录与限制
|
| 147 |
|
| 148 |
-
- 对上游原始模型的 **
|
| 149 |
-
- 最大概率绝对误差约 **
|
| 150 |
-
- 独立 Python 包在未安装 PyTorch 的环境中通过
|
| 151 |
- Swift CPU 原生调用结果见 `swift_smoke.json`;Python 独立包记录见 `package_smoke.json`。
|
| 152 |
-
-
|
| 153 |
-
- 未验证 iPhone/iPad、iOS AOT、自动设备模式、GPU/ANE 放置、
|
| 154 |
-
-
|
| 155 |
-
|
|
|
|
| 156 |
|
| 157 |
完整用例与概率:`validation_cases.json`、`evaluation.json`。
|
| 158 |
`SHA256SUMS` 覆盖发布文件;本仓库不上传原始权重副本、开发缓存或虚拟环境。
|
|
|
|
| 18 |
[English](README.md) | 简体中文
|
| 19 |
|
| 20 |
由 hugging-mac 从 [vllm-sr/vela3-fast](https://huggingface.co/vllm-sr/vela3-fast) 转换的 Apple Core AI 模型。
|
| 21 |
+
本仓库发布的是经过验证的 **choice / noul 决策主干**:每次图调用一个样本,序列长度和选项数量均为动态。
|
| 22 |
+
导出保留原始 **32768-token** 上限;CPU 实际对照验证覆盖到 **2048 tokens**,包括精确 1024 tokens。
|
| 23 |
**这是实验性子集,不是包含所有任务 adapter 的完整 Vela3-fast。**
|
| 24 |
|
| 25 |
This is an experimental, fixed-shape decision-backbone export, not a complete replacement for upstream Vela3-fast.
|
|
|
|
| 27 |
|
| 28 |
## 支持范围 / Contract
|
| 29 |
|
| 30 |
+
- `choice`:一个或多个有序选项,返回选项概率和选择结果。
|
| 31 |
- `noul`:固定 `no` / `yes` 两个选项,返回 `p(yes)`。
|
| 32 |
+
- 输入正文、问题、选项描述和特殊标记合计不超过 原始 **32768 tokens** 上限;超长输入报错,不截断。
|
| 33 |
+
- 每次图调用的 batch size 为 1。`predict_questions(state, questions)` 接受问题映射并返回 `(answers, input_tokens)`,与上游决策接口对应;内部逐个执行。保留原始 JSON 序列化,choice criteria 支持字典或选项列表。
|
| 34 |
- CPU 模式已验证。自动设备模式此前因测试主机资源压力未完成验证,不宣称 GPU/ANE 性能。
|
| 35 |
- 原始 FP32 决策权重,无量化、训练或微调。
|
| 36 |
- 原模型的 router、span、relation、NER、evidence、domain/factcheck/feedback 等 adapter 不包含在此制品中;
|
| 37 |
+
当前决策图未实现完整 `system_one` adapter 路由。未测试不代表这些能力无法转换。
|
| 38 |
- 不提供完整 `hugging_mac_sdk` 模型注册或生产验收结论。概率是给定选项间的相对值,不是校准置信度。
|
| 39 |
|
| 40 |
## Python 基本调用
|
|
|
|
| 76 |
```
|
| 77 |
|
| 78 |
`predict` 执行 tokenizer、标记位置和 pooling mask 的准备,再调用 Core AI。
|
| 79 |
+
同一个实例一次只接收一个推理请求;请等待前一次调用完成。未知任务、空选项集合和超长输入都会报错。
|
| 80 |
|
| 81 |
运行独立包验证:
|
| 82 |
|
|
|
|
| 84 |
USE_OS_COREAI=1 uv run --locked python smoke.py
|
| 85 |
```
|
| 86 |
|
| 87 |
+
该脚本比较12 个用例与上游参考概率,校验输入张量与上游编码完全相同,验证三类无效请求拒绝行为,
|
| 88 |
并生成 `package_smoke.json` 和用于 Swift 的 `swift_fixture.json`。
|
| 89 |
|
| 90 |
## Swift 基本调用
|
|
|
|
| 119 |
guard let logits = result.remove("logits")?.ndArray else {
|
| 120 |
fatalError("Missing logits")
|
| 121 |
}
|
| 122 |
+
// 对输出 logits 计算稳定 softmax,按输入选项顺序解释概率。
|
| 123 |
```
|
| 124 |
|
| 125 |
对任意新文本调用时,必须复用本仓库 `tokenizer.json`、`decision.json` 和 `vela_coreai.py` 的编码规则;
|
|
|
|
| 127 |
|
| 128 |
## 张量接口
|
| 129 |
|
| 130 |
+
入口:`main`,无持久 recurrent/KV 状态。序列轴和选项轴均为动态,按实际编码长度输入,不固定补齐。
|
| 131 |
|
| 132 |
| 名称 | 类型 | 形状 | 含义 |
|
| 133 |
| --- | --- | --- | --- |
|
| 134 |
+
| `input_ids` | int32 | `[1,sequence]` | 完整问题/选项/正文序列 |
|
| 135 |
+
| `attention_mask` | int32 | `[1,sequence]` | 有效 token 为 1,padding 为 0 |
|
| 136 |
+
| `option_positions` | int32 | `[1,options]` | 各个 `[O]` 标记的位置 |
|
| 137 |
+
| `pool_mask` | float32 | `[1,sequence]` | `[SEG_user]` 到正文末尾为 1,不包括 EOS |
|
| 138 |
+
| `logits`(输出) | float32 | `[1,options]` | 对应一个或多个有序选项的未归一化分数 |
|
| 139 |
|
| 140 |
上游序列格式:
|
| 141 |
|
|
|
|
| 146 |
|
| 147 |
## 验证记录与限制
|
| 148 |
|
| 149 |
+
- 对上游原始模型的 **12/12** 文本用例,转换后决策全部一致,预期答案全部正确。
|
| 150 |
+
- 最大概率绝对误差约 **7.80e-7**;验证容差为 `atol=0.02, rtol=0.02`,同时要求决策一致。
|
| 151 |
+
- 独立 Python 包在未安装 PyTorch 的环境中通过12 个用例,输入张量逐项匹配上游。
|
| 152 |
- Swift CPU 原生调用结果见 `swift_smoke.json`;Python 独立包记录见 `package_smoke.json`。
|
| 153 |
+
- 这些手工样例不是通用准确率基准;覆盖 JSON 对象、1/2/3 个选项,以及精确 129/1024/2048-token 输入。
|
| 154 |
+
- 未验证 iPhone/iPad、iOS AOT、自动设备模式、GPU/ANE 放置、超过 2048 tokens 的输入或其他上游任务。
|
| 155 |
+
- 新动态制品通过了初始 CPU 张量验证和后续任务对照评测。工具版本见 `manifest.json` 与 `environment.json`。
|
| 156 |
+
- 导出和输入适配器保留 32K 上限,但尚未在这台 8 GB 测试机器上验证完整 32K 推理;更长输入需要更多内存。2048 是已测长度,不是新增硬限制。
|
| 157 |
+
- 本版本替换先前固定 128-token、两个选项的制品;上述 adapters 仍不在当前决策图中。
|
| 158 |
|
| 159 |
完整用例与概率:`validation_cases.json`、`evaluation.json`。
|
| 160 |
`SHA256SUMS` 覆盖发布文件;本仓库不上传原始权重副本、开发缓存或虚拟环境。
|
SHA256SUMS
CHANGED
|
@@ -1,24 +1,25 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4 LICENSE
|
| 4 |
46710ad6644d63748efc1d91895ed2f7d039ddd3e4fe9caf7351f2f7a18ae448 NOTICE
|
| 5 |
7b55f8e67b5623c4bef3fa691288da9437d79d3aba156de48d481db32ac7d16d .python-version
|
| 6 |
fe8ebeab92a5ecbca37f7fd768d480c122ceee326f5ac78f2e02cbf8c42adb85 pyproject.toml
|
| 7 |
2f622f1a164e966f012193787cc8325837d96c2bd33f62f7a704606f0d0e17a9 uv.lock
|
| 8 |
-
|
| 9 |
83dd42bf29453dafe9ca9525e9f8085198dc01afee7338012643b900ed0ac57e example.py
|
| 10 |
-
|
| 11 |
-
|
| 12 |
38734bb17ef19f84c67b5eb5692bea19801ffd02cb5c5ffc11429e76b0242e08 tokenizer.json
|
| 13 |
649781df90d51837dd6ba951a89fae07c985a9eacfd781d84002d480c3cb0b6e tokenizer_config.json
|
| 14 |
1150af4f64b7cec72e35ad2ad06db180be009d5d18ed05903c2af21f2a85ece3 decision.json
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
|
|
|
|
|
| 1 |
+
b8c3735da9846be5e0c0ff4e7dcaa4724464b33d3b7044d69cc6edd7bdb39e22 README.md
|
| 2 |
+
48a4f8a55205833272d55ec08bc74b025ae65b18eb8a18dd704036f52dd52929 README.zh-CN.md
|
| 3 |
c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4 LICENSE
|
| 4 |
46710ad6644d63748efc1d91895ed2f7d039ddd3e4fe9caf7351f2f7a18ae448 NOTICE
|
| 5 |
7b55f8e67b5623c4bef3fa691288da9437d79d3aba156de48d481db32ac7d16d .python-version
|
| 6 |
fe8ebeab92a5ecbca37f7fd768d480c122ceee326f5ac78f2e02cbf8c42adb85 pyproject.toml
|
| 7 |
2f622f1a164e966f012193787cc8325837d96c2bd33f62f7a704606f0d0e17a9 uv.lock
|
| 8 |
+
0e6f26d2bbe55ac16368da959bf07521e54be0276eddeeb1f65a64004ae6476e vela_coreai.py
|
| 9 |
83dd42bf29453dafe9ca9525e9f8085198dc01afee7338012643b900ed0ac57e example.py
|
| 10 |
+
32d69284eacb23ac0b6be8fe4de2e3604248d4c0adb377cbd8c0afac5153b4be smoke.py
|
| 11 |
+
04d7528586eaa525bb0fe5c5952cefbf5f1363098cd941803f0560cace854cb8 verify_coreai.swift
|
| 12 |
38734bb17ef19f84c67b5eb5692bea19801ffd02cb5c5ffc11429e76b0242e08 tokenizer.json
|
| 13 |
649781df90d51837dd6ba951a89fae07c985a9eacfd781d84002d480c3cb0b6e tokenizer_config.json
|
| 14 |
1150af4f64b7cec72e35ad2ad06db180be009d5d18ed05903c2af21f2a85ece3 decision.json
|
| 15 |
+
6593400dbc76848462ffbc464d5f326fbf9a959f638c8f3c218df4a1b98ac804 manifest.json
|
| 16 |
+
c43550694f14e808e66044db2145b2780cc794336425f9674718dccf6f09022e validation_cases.json
|
| 17 |
+
80793396bc59f00d88a7df5c75b3ce729ef089501702733fda50fed49d08a691 evaluation.json
|
| 18 |
+
c9c84393adbd3d190c0443fabae3bc313ecc807870287472bba318d5352b3cc7 environment.json
|
| 19 |
+
328ec37a9e1d578ae789bc0930db16ac05cbb30b4dab87c4358b4318e1bacdbf package_smoke.json
|
| 20 |
+
9f3a0299f9df5a090df083fc4862fbc79e72f1387db467efbd5ef4ac26927b2a swift_fixture.json
|
| 21 |
+
92ebf2b041505c1acb0f1f68b5b2d2962f470785609b17d7ae54e57a0707b361 swift_smoke.json
|
| 22 |
+
d1e3ae70a989d8d38ccae6513bf0b4085f751545cb6cd4efbd5373fa26f74fc4 model.aimodel/main.hash
|
| 23 |
+
d7b63896067e505f67a03c1afe31e070b0a1f5d746e14d603d9130708798adad model.aimodel/main.mlirb
|
| 24 |
+
a19b297a1df17828004220bd2a85bdd86f6630fd3edca8e9d3853366147d96b7 model.aimodel/metadata.json
|
| 25 |
+
636221d20e2bcaaded4299220632346f377231dd9fdaae006482f4f9faa350ad input_boundary.json
|
environment.json
CHANGED
|
@@ -1,10 +1,10 @@
|
|
| 1 |
{
|
| 2 |
"python": "3.12.13",
|
| 3 |
"platform": "macOS-27.2.1-arm64-arm-64bit",
|
| 4 |
-
"pytorch_installed": false,
|
| 5 |
"packages": {
|
| 6 |
"coreai-core": "1.0.0b3",
|
| 7 |
"numpy": "2.3.5",
|
| 8 |
"tokenizers": "0.23.3"
|
| 9 |
-
}
|
|
|
|
| 10 |
}
|
|
|
|
| 1 |
{
|
| 2 |
"python": "3.12.13",
|
| 3 |
"platform": "macOS-27.2.1-arm64-arm-64bit",
|
|
|
|
| 4 |
"packages": {
|
| 5 |
"coreai-core": "1.0.0b3",
|
| 6 |
"numpy": "2.3.5",
|
| 7 |
"tokenizers": "0.23.3"
|
| 8 |
+
},
|
| 9 |
+
"pytorch_installed": false
|
| 10 |
}
|
evaluation.json
CHANGED
|
@@ -12,7 +12,7 @@
|
|
| 12 |
"coreai-core": "1.0.0b3",
|
| 13 |
"coreai-torch": "0.4.3"
|
| 14 |
},
|
| 15 |
-
"cases_sha256": "
|
| 16 |
"evaluation_config": {
|
| 17 |
"scope": "diagnostic",
|
| 18 |
"cases": [
|
|
@@ -33,10 +33,10 @@
|
|
| 33 |
"span_sets": "untested",
|
| 34 |
"evidence": "untested",
|
| 35 |
"router_task_adapters": "untested",
|
| 36 |
-
"structured_state": "
|
| 37 |
"multiple_questions": "untested",
|
| 38 |
-
"variable_sequence_length": "
|
| 39 |
-
"variable_options": "
|
| 40 |
"system_one_response": "untested",
|
| 41 |
"domain_adapter": "untested",
|
| 42 |
"factcheck_adapter": "untested",
|
|
@@ -47,7 +47,7 @@
|
|
| 47 |
{
|
| 48 |
"id": "coding-en",
|
| 49 |
"capability": "choice",
|
| 50 |
-
"inference_ms":
|
| 51 |
"status": "evaluated",
|
| 52 |
"probabilities": [
|
| 53 |
0.9964854118964818,
|
|
@@ -60,7 +60,7 @@
|
|
| 60 |
{
|
| 61 |
"id": "cooking-zh",
|
| 62 |
"capability": "choice",
|
| 63 |
-
"inference_ms":
|
| 64 |
"status": "evaluated",
|
| 65 |
"probabilities": [
|
| 66 |
0.22242194924412478,
|
|
@@ -73,7 +73,7 @@
|
|
| 73 |
{
|
| 74 |
"id": "noul-positive",
|
| 75 |
"capability": "noul",
|
| 76 |
-
"inference_ms":
|
| 77 |
"status": "evaluated",
|
| 78 |
"probabilities": [
|
| 79 |
0.04283435292391378,
|
|
@@ -86,7 +86,7 @@
|
|
| 86 |
{
|
| 87 |
"id": "noul-negative",
|
| 88 |
"capability": "noul",
|
| 89 |
-
"inference_ms":
|
| 90 |
"status": "evaluated",
|
| 91 |
"probabilities": [
|
| 92 |
0.9887530718985025,
|
|
@@ -99,7 +99,7 @@
|
|
| 99 |
{
|
| 100 |
"id": "coding-zh",
|
| 101 |
"capability": "choice",
|
| 102 |
-
"inference_ms":
|
| 103 |
"status": "evaluated",
|
| 104 |
"probabilities": [
|
| 105 |
0.9965400461728955,
|
|
@@ -112,7 +112,7 @@
|
|
| 112 |
{
|
| 113 |
"id": "cooking-en",
|
| 114 |
"capability": "choice",
|
| 115 |
-
"inference_ms":
|
| 116 |
"status": "evaluated",
|
| 117 |
"probabilities": [
|
| 118 |
0.0008233419586070937,
|
|
@@ -121,6 +121,88 @@
|
|
| 121 |
"decision": "cooking",
|
| 122 |
"expected": "cooking",
|
| 123 |
"correct": true
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
}
|
| 125 |
],
|
| 126 |
"candidate": {
|
|
@@ -130,20 +212,20 @@
|
|
| 130 |
{
|
| 131 |
"id": "coding-en",
|
| 132 |
"capability": "choice",
|
| 133 |
-
"reference_ms":
|
| 134 |
-
"converted_ms":
|
| 135 |
"passed": true,
|
| 136 |
"numeric_passed": true,
|
| 137 |
"decision_agreement": true,
|
| 138 |
-
"max_absolute_error":
|
| 139 |
-
"mean_absolute_error":
|
| 140 |
"reference_probabilities": [
|
| 141 |
0.9964854118964818,
|
| 142 |
0.003514588103518185
|
| 143 |
],
|
| 144 |
"converted_probabilities": [
|
| 145 |
-
0.
|
| 146 |
-
0.
|
| 147 |
],
|
| 148 |
"reference_decision": "coding",
|
| 149 |
"converted_decision": "coding",
|
|
@@ -155,20 +237,20 @@
|
|
| 155 |
{
|
| 156 |
"id": "cooking-zh",
|
| 157 |
"capability": "choice",
|
| 158 |
-
"reference_ms":
|
| 159 |
-
"converted_ms":
|
| 160 |
"passed": true,
|
| 161 |
"numeric_passed": true,
|
| 162 |
"decision_agreement": true,
|
| 163 |
-
"max_absolute_error":
|
| 164 |
-
"mean_absolute_error":
|
| 165 |
"reference_probabilities": [
|
| 166 |
0.22242194924412478,
|
| 167 |
0.7775780507558753
|
| 168 |
],
|
| 169 |
"converted_probabilities": [
|
| 170 |
-
0.
|
| 171 |
-
0.
|
| 172 |
],
|
| 173 |
"reference_decision": "cooking",
|
| 174 |
"converted_decision": "cooking",
|
|
@@ -180,20 +262,20 @@
|
|
| 180 |
{
|
| 181 |
"id": "noul-positive",
|
| 182 |
"capability": "noul",
|
| 183 |
-
"reference_ms":
|
| 184 |
-
"converted_ms":
|
| 185 |
"passed": true,
|
| 186 |
"numeric_passed": true,
|
| 187 |
"decision_agreement": true,
|
| 188 |
-
"max_absolute_error":
|
| 189 |
-
"mean_absolute_error":
|
| 190 |
"reference_probabilities": [
|
| 191 |
0.04283435292391378,
|
| 192 |
0.9571656470760861
|
| 193 |
],
|
| 194 |
"converted_probabilities": [
|
| 195 |
-
0.
|
| 196 |
-
0.
|
| 197 |
],
|
| 198 |
"reference_decision": "yes",
|
| 199 |
"converted_decision": "yes",
|
|
@@ -205,20 +287,20 @@
|
|
| 205 |
{
|
| 206 |
"id": "noul-negative",
|
| 207 |
"capability": "noul",
|
| 208 |
-
"reference_ms":
|
| 209 |
-
"converted_ms":
|
| 210 |
"passed": true,
|
| 211 |
"numeric_passed": true,
|
| 212 |
"decision_agreement": true,
|
| 213 |
-
"max_absolute_error":
|
| 214 |
-
"mean_absolute_error":
|
| 215 |
"reference_probabilities": [
|
| 216 |
0.9887530718985025,
|
| 217 |
0.011246928101497617
|
| 218 |
],
|
| 219 |
"converted_probabilities": [
|
| 220 |
-
0.
|
| 221 |
-
0.
|
| 222 |
],
|
| 223 |
"reference_decision": "no",
|
| 224 |
"converted_decision": "no",
|
|
@@ -230,20 +312,20 @@
|
|
| 230 |
{
|
| 231 |
"id": "coding-zh",
|
| 232 |
"capability": "choice",
|
| 233 |
-
"reference_ms":
|
| 234 |
-
"converted_ms":
|
| 235 |
"passed": true,
|
| 236 |
"numeric_passed": true,
|
| 237 |
"decision_agreement": true,
|
| 238 |
-
"max_absolute_error":
|
| 239 |
-
"mean_absolute_error":
|
| 240 |
"reference_probabilities": [
|
| 241 |
0.9965400461728955,
|
| 242 |
0.0034599538271045662
|
| 243 |
],
|
| 244 |
"converted_probabilities": [
|
| 245 |
-
0.
|
| 246 |
-
0.
|
| 247 |
],
|
| 248 |
"reference_decision": "coding",
|
| 249 |
"converted_decision": "coding",
|
|
@@ -255,20 +337,20 @@
|
|
| 255 |
{
|
| 256 |
"id": "cooking-en",
|
| 257 |
"capability": "choice",
|
| 258 |
-
"reference_ms":
|
| 259 |
-
"converted_ms":
|
| 260 |
"passed": true,
|
| 261 |
"numeric_passed": true,
|
| 262 |
"decision_agreement": true,
|
| 263 |
-
"max_absolute_error": 1.
|
| 264 |
-
"mean_absolute_error": 1.
|
| 265 |
"reference_probabilities": [
|
| 266 |
0.0008233419586070937,
|
| 267 |
0.9991766580413929
|
| 268 |
],
|
| 269 |
"converted_probabilities": [
|
| 270 |
-
0.
|
| 271 |
-
0.
|
| 272 |
],
|
| 273 |
"reference_decision": "cooking",
|
| 274 |
"converted_decision": "cooking",
|
|
@@ -276,6 +358,164 @@
|
|
| 276 |
"reference_correct": true,
|
| 277 |
"converted_correct": true,
|
| 278 |
"status": "passed"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 279 |
}
|
| 280 |
]
|
| 281 |
}
|
|
|
|
| 12 |
"coreai-core": "1.0.0b3",
|
| 13 |
"coreai-torch": "0.4.3"
|
| 14 |
},
|
| 15 |
+
"cases_sha256": "184b5ab437c84bea6c1517d84e145ecd68ba7e4fbb7b61d7f628b9d474076e17",
|
| 16 |
"evaluation_config": {
|
| 17 |
"scope": "diagnostic",
|
| 18 |
"cases": [
|
|
|
|
| 33 |
"span_sets": "untested",
|
| 34 |
"evidence": "untested",
|
| 35 |
"router_task_adapters": "untested",
|
| 36 |
+
"structured_state": "exercised-in-diagnostic",
|
| 37 |
"multiple_questions": "untested",
|
| 38 |
+
"variable_sequence_length": "exercised-in-diagnostic",
|
| 39 |
+
"variable_options": "exercised-in-diagnostic",
|
| 40 |
"system_one_response": "untested",
|
| 41 |
"domain_adapter": "untested",
|
| 42 |
"factcheck_adapter": "untested",
|
|
|
|
| 47 |
{
|
| 48 |
"id": "coding-en",
|
| 49 |
"capability": "choice",
|
| 50 |
+
"inference_ms": 51.31262500071898,
|
| 51 |
"status": "evaluated",
|
| 52 |
"probabilities": [
|
| 53 |
0.9964854118964818,
|
|
|
|
| 60 |
{
|
| 61 |
"id": "cooking-zh",
|
| 62 |
"capability": "choice",
|
| 63 |
+
"inference_ms": 68.58845800161362,
|
| 64 |
"status": "evaluated",
|
| 65 |
"probabilities": [
|
| 66 |
0.22242194924412478,
|
|
|
|
| 73 |
{
|
| 74 |
"id": "noul-positive",
|
| 75 |
"capability": "noul",
|
| 76 |
+
"inference_ms": 48.34995899000205,
|
| 77 |
"status": "evaluated",
|
| 78 |
"probabilities": [
|
| 79 |
0.04283435292391378,
|
|
|
|
| 86 |
{
|
| 87 |
"id": "noul-negative",
|
| 88 |
"capability": "noul",
|
| 89 |
+
"inference_ms": 52.418874998693354,
|
| 90 |
"status": "evaluated",
|
| 91 |
"probabilities": [
|
| 92 |
0.9887530718985025,
|
|
|
|
| 99 |
{
|
| 100 |
"id": "coding-zh",
|
| 101 |
"capability": "choice",
|
| 102 |
+
"inference_ms": 70.75237500248477,
|
| 103 |
"status": "evaluated",
|
| 104 |
"probabilities": [
|
| 105 |
0.9965400461728955,
|
|
|
|
| 112 |
{
|
| 113 |
"id": "cooking-en",
|
| 114 |
"capability": "choice",
|
| 115 |
+
"inference_ms": 51.56712500320282,
|
| 116 |
"status": "evaluated",
|
| 117 |
"probabilities": [
|
| 118 |
0.0008233419586070937,
|
|
|
|
| 121 |
"decision": "cooking",
|
| 122 |
"expected": "cooking",
|
| 123 |
"correct": true
|
| 124 |
+
},
|
| 125 |
+
{
|
| 126 |
+
"id": "three-options",
|
| 127 |
+
"capability": "variable_options",
|
| 128 |
+
"inference_ms": 53.129124993574806,
|
| 129 |
+
"status": "evaluated",
|
| 130 |
+
"probabilities": [
|
| 131 |
+
0.9929379587678295,
|
| 132 |
+
0.0028259221336674827,
|
| 133 |
+
0.004236119098503003
|
| 134 |
+
],
|
| 135 |
+
"decision": "coding",
|
| 136 |
+
"expected": "coding",
|
| 137 |
+
"correct": true
|
| 138 |
+
},
|
| 139 |
+
{
|
| 140 |
+
"id": "one-option",
|
| 141 |
+
"capability": "variable_options",
|
| 142 |
+
"inference_ms": 49.474542000098154,
|
| 143 |
+
"status": "evaluated",
|
| 144 |
+
"probabilities": [
|
| 145 |
+
1.0
|
| 146 |
+
],
|
| 147 |
+
"decision": "coding",
|
| 148 |
+
"expected": "coding",
|
| 149 |
+
"correct": true
|
| 150 |
+
},
|
| 151 |
+
{
|
| 152 |
+
"id": "structured-json",
|
| 153 |
+
"capability": "structured_state",
|
| 154 |
+
"inference_ms": 69.43170799058862,
|
| 155 |
+
"status": "evaluated",
|
| 156 |
+
"probabilities": [
|
| 157 |
+
0.9955305388600278,
|
| 158 |
+
0.0022821065546790487,
|
| 159 |
+
0.002187354585293287
|
| 160 |
+
],
|
| 161 |
+
"decision": "coding",
|
| 162 |
+
"expected": "coding",
|
| 163 |
+
"correct": true
|
| 164 |
+
},
|
| 165 |
+
{
|
| 166 |
+
"id": "exact-129-tokens",
|
| 167 |
+
"capability": "variable_sequence_length",
|
| 168 |
+
"inference_ms": 141.63116700365208,
|
| 169 |
+
"status": "evaluated",
|
| 170 |
+
"probabilities": [
|
| 171 |
+
0.6347567731488714,
|
| 172 |
+
0.13812266903655834,
|
| 173 |
+
0.22712055781457036
|
| 174 |
+
],
|
| 175 |
+
"decision": "coding",
|
| 176 |
+
"expected": "coding",
|
| 177 |
+
"correct": true
|
| 178 |
+
},
|
| 179 |
+
{
|
| 180 |
+
"id": "exact-1024-tokens",
|
| 181 |
+
"capability": "variable_sequence_length",
|
| 182 |
+
"inference_ms": 968.6068749870174,
|
| 183 |
+
"status": "evaluated",
|
| 184 |
+
"probabilities": [
|
| 185 |
+
0.4407633906736511,
|
| 186 |
+
0.37238073879955924,
|
| 187 |
+
0.18685587052678965
|
| 188 |
+
],
|
| 189 |
+
"decision": "coding",
|
| 190 |
+
"expected": "coding",
|
| 191 |
+
"correct": true
|
| 192 |
+
},
|
| 193 |
+
{
|
| 194 |
+
"id": "exact-2048-tokens",
|
| 195 |
+
"capability": "variable_sequence_length",
|
| 196 |
+
"inference_ms": 2569.3836659920635,
|
| 197 |
+
"status": "evaluated",
|
| 198 |
+
"probabilities": [
|
| 199 |
+
0.5216577502831695,
|
| 200 |
+
0.34918566249456723,
|
| 201 |
+
0.12915658722226325
|
| 202 |
+
],
|
| 203 |
+
"decision": "coding",
|
| 204 |
+
"expected": "coding",
|
| 205 |
+
"correct": true
|
| 206 |
}
|
| 207 |
],
|
| 208 |
"candidate": {
|
|
|
|
| 212 |
{
|
| 213 |
"id": "coding-en",
|
| 214 |
"capability": "choice",
|
| 215 |
+
"reference_ms": 51.31262500071898,
|
| 216 |
+
"converted_ms": 2374.499250014196,
|
| 217 |
"passed": true,
|
| 218 |
"numeric_passed": true,
|
| 219 |
"decision_agreement": true,
|
| 220 |
+
"max_absolute_error": 5.427491678311469e-09,
|
| 221 |
+
"mean_absolute_error": 5.4274916770104265e-09,
|
| 222 |
"reference_probabilities": [
|
| 223 |
0.9964854118964818,
|
| 224 |
0.003514588103518185
|
| 225 |
],
|
| 226 |
"converted_probabilities": [
|
| 227 |
+
0.9964854064689901,
|
| 228 |
+
0.0035145935310098605
|
| 229 |
],
|
| 230 |
"reference_decision": "coding",
|
| 231 |
"converted_decision": "coding",
|
|
|
|
| 237 |
{
|
| 238 |
"id": "cooking-zh",
|
| 239 |
"capability": "choice",
|
| 240 |
+
"reference_ms": 68.58845800161362,
|
| 241 |
+
"converted_ms": 91.65208299236838,
|
| 242 |
"passed": true,
|
| 243 |
"numeric_passed": true,
|
| 244 |
"decision_agreement": true,
|
| 245 |
+
"max_absolute_error": 4.7419820070615515e-07,
|
| 246 |
+
"mean_absolute_error": 4.74198200650644e-07,
|
| 247 |
"reference_probabilities": [
|
| 248 |
0.22242194924412478,
|
| 249 |
0.7775780507558753
|
| 250 |
],
|
| 251 |
"converted_probabilities": [
|
| 252 |
+
0.22242242344232538,
|
| 253 |
+
0.7775775765576746
|
| 254 |
],
|
| 255 |
"reference_decision": "cooking",
|
| 256 |
"converted_decision": "cooking",
|
|
|
|
| 262 |
{
|
| 263 |
"id": "noul-positive",
|
| 264 |
"capability": "noul",
|
| 265 |
+
"reference_ms": 48.34995899000205,
|
| 266 |
+
"converted_ms": 47.826540991081856,
|
| 267 |
"passed": true,
|
| 268 |
"numeric_passed": true,
|
| 269 |
"decision_agreement": true,
|
| 270 |
+
"max_absolute_error": 1.3685062416612936e-07,
|
| 271 |
+
"mean_absolute_error": 1.3685062412102655e-07,
|
| 272 |
"reference_probabilities": [
|
| 273 |
0.04283435292391378,
|
| 274 |
0.9571656470760861
|
| 275 |
],
|
| 276 |
"converted_probabilities": [
|
| 277 |
+
0.0428342160732897,
|
| 278 |
+
0.9571657839267103
|
| 279 |
],
|
| 280 |
"reference_decision": "yes",
|
| 281 |
"converted_decision": "yes",
|
|
|
|
| 287 |
{
|
| 288 |
"id": "noul-negative",
|
| 289 |
"capability": "noul",
|
| 290 |
+
"reference_ms": 52.418874998693354,
|
| 291 |
+
"converted_ms": 80.2467909961706,
|
| 292 |
"passed": true,
|
| 293 |
"numeric_passed": true,
|
| 294 |
"decision_agreement": true,
|
| 295 |
+
"max_absolute_error": 2.6513179342962934e-09,
|
| 296 |
+
"mean_absolute_error": 2.6513178276107996e-09,
|
| 297 |
"reference_probabilities": [
|
| 298 |
0.9887530718985025,
|
| 299 |
0.011246928101497617
|
| 300 |
],
|
| 301 |
"converted_probabilities": [
|
| 302 |
+
0.9887530745498202,
|
| 303 |
+
0.011246925450179683
|
| 304 |
],
|
| 305 |
"reference_decision": "no",
|
| 306 |
"converted_decision": "no",
|
|
|
|
| 312 |
{
|
| 313 |
"id": "coding-zh",
|
| 314 |
"capability": "choice",
|
| 315 |
+
"reference_ms": 70.75237500248477,
|
| 316 |
+
"converted_ms": 79.13508299679961,
|
| 317 |
"passed": true,
|
| 318 |
"numeric_passed": true,
|
| 319 |
"decision_agreement": true,
|
| 320 |
+
"max_absolute_error": 8.426156995235135e-09,
|
| 321 |
+
"mean_absolute_error": 8.426156994801454e-09,
|
| 322 |
"reference_probabilities": [
|
| 323 |
0.9965400461728955,
|
| 324 |
0.0034599538271045662
|
| 325 |
],
|
| 326 |
"converted_probabilities": [
|
| 327 |
+
0.9965400377467385,
|
| 328 |
+
0.0034599622532615615
|
| 329 |
],
|
| 330 |
"reference_decision": "coding",
|
| 331 |
"converted_decision": "coding",
|
|
|
|
| 337 |
{
|
| 338 |
"id": "cooking-en",
|
| 339 |
"capability": "choice",
|
| 340 |
+
"reference_ms": 51.56712500320282,
|
| 341 |
+
"converted_ms": 60.23716600611806,
|
| 342 |
"passed": true,
|
| 343 |
"numeric_passed": true,
|
| 344 |
"decision_agreement": true,
|
| 345 |
+
"max_absolute_error": 1.5691086767584406e-09,
|
| 346 |
+
"mean_absolute_error": 1.5691086487860245e-09,
|
| 347 |
"reference_probabilities": [
|
| 348 |
0.0008233419586070937,
|
| 349 |
0.9991766580413929
|
| 350 |
],
|
| 351 |
"converted_probabilities": [
|
| 352 |
+
0.0008233435277157704,
|
| 353 |
+
0.9991766564722843
|
| 354 |
],
|
| 355 |
"reference_decision": "cooking",
|
| 356 |
"converted_decision": "cooking",
|
|
|
|
| 358 |
"reference_correct": true,
|
| 359 |
"converted_correct": true,
|
| 360 |
"status": "passed"
|
| 361 |
+
},
|
| 362 |
+
{
|
| 363 |
+
"id": "three-options",
|
| 364 |
+
"capability": "variable_options",
|
| 365 |
+
"reference_ms": 53.129124993574806,
|
| 366 |
+
"converted_ms": 59.11033399752341,
|
| 367 |
+
"passed": true,
|
| 368 |
+
"numeric_passed": true,
|
| 369 |
+
"decision_agreement": true,
|
| 370 |
+
"max_absolute_error": 3.460630049190172e-08,
|
| 371 |
+
"mean_absolute_error": 2.3070867004286687e-08,
|
| 372 |
+
"reference_probabilities": [
|
| 373 |
+
0.9929379587678295,
|
| 374 |
+
0.0028259221336674827,
|
| 375 |
+
0.004236119098503003
|
| 376 |
+
],
|
| 377 |
+
"converted_probabilities": [
|
| 378 |
+
0.99293799337413,
|
| 379 |
+
0.002825909094017193,
|
| 380 |
+
0.004236097531852771
|
| 381 |
+
],
|
| 382 |
+
"reference_decision": "coding",
|
| 383 |
+
"converted_decision": "coding",
|
| 384 |
+
"expected": "coding",
|
| 385 |
+
"reference_correct": true,
|
| 386 |
+
"converted_correct": true,
|
| 387 |
+
"status": "passed"
|
| 388 |
+
},
|
| 389 |
+
{
|
| 390 |
+
"id": "one-option",
|
| 391 |
+
"capability": "variable_options",
|
| 392 |
+
"reference_ms": 49.474542000098154,
|
| 393 |
+
"converted_ms": 56.663583003683016,
|
| 394 |
+
"passed": true,
|
| 395 |
+
"numeric_passed": true,
|
| 396 |
+
"decision_agreement": true,
|
| 397 |
+
"max_absolute_error": 0.0,
|
| 398 |
+
"mean_absolute_error": 0.0,
|
| 399 |
+
"reference_probabilities": [
|
| 400 |
+
1.0
|
| 401 |
+
],
|
| 402 |
+
"converted_probabilities": [
|
| 403 |
+
1.0
|
| 404 |
+
],
|
| 405 |
+
"reference_decision": "coding",
|
| 406 |
+
"converted_decision": "coding",
|
| 407 |
+
"expected": "coding",
|
| 408 |
+
"reference_correct": true,
|
| 409 |
+
"converted_correct": true,
|
| 410 |
+
"status": "passed"
|
| 411 |
+
},
|
| 412 |
+
{
|
| 413 |
+
"id": "structured-json",
|
| 414 |
+
"capability": "structured_state",
|
| 415 |
+
"reference_ms": 69.43170799058862,
|
| 416 |
+
"converted_ms": 75.03466600610409,
|
| 417 |
+
"passed": true,
|
| 418 |
+
"numeric_passed": true,
|
| 419 |
+
"decision_agreement": true,
|
| 420 |
+
"max_absolute_error": 1.4141390947131072e-08,
|
| 421 |
+
"mean_absolute_error": 9.427594002917964e-09,
|
| 422 |
+
"reference_probabilities": [
|
| 423 |
+
0.9955305388600278,
|
| 424 |
+
0.0022821065546790487,
|
| 425 |
+
0.002187354585293287
|
| 426 |
+
],
|
| 427 |
+
"converted_probabilities": [
|
| 428 |
+
0.9955305530014188,
|
| 429 |
+
0.002282103594566704,
|
| 430 |
+
0.0021873434040145696
|
| 431 |
+
],
|
| 432 |
+
"reference_decision": "coding",
|
| 433 |
+
"converted_decision": "coding",
|
| 434 |
+
"expected": "coding",
|
| 435 |
+
"reference_correct": true,
|
| 436 |
+
"converted_correct": true,
|
| 437 |
+
"status": "passed"
|
| 438 |
+
},
|
| 439 |
+
{
|
| 440 |
+
"id": "exact-129-tokens",
|
| 441 |
+
"capability": "variable_sequence_length",
|
| 442 |
+
"reference_ms": 141.63116700365208,
|
| 443 |
+
"converted_ms": 134.83975001145154,
|
| 444 |
+
"passed": true,
|
| 445 |
+
"numeric_passed": true,
|
| 446 |
+
"decision_agreement": true,
|
| 447 |
+
"max_absolute_error": 7.795353781370196e-07,
|
| 448 |
+
"mean_absolute_error": 5.196902520913463e-07,
|
| 449 |
+
"reference_probabilities": [
|
| 450 |
+
0.6347567731488714,
|
| 451 |
+
0.13812266903655834,
|
| 452 |
+
0.22712055781457036
|
| 453 |
+
],
|
| 454 |
+
"converted_probabilities": [
|
| 455 |
+
0.6347575526842495,
|
| 456 |
+
0.13812251758558605,
|
| 457 |
+
0.2271199297301645
|
| 458 |
+
],
|
| 459 |
+
"reference_decision": "coding",
|
| 460 |
+
"converted_decision": "coding",
|
| 461 |
+
"expected": "coding",
|
| 462 |
+
"reference_correct": true,
|
| 463 |
+
"converted_correct": true,
|
| 464 |
+
"status": "passed"
|
| 465 |
+
},
|
| 466 |
+
{
|
| 467 |
+
"id": "exact-1024-tokens",
|
| 468 |
+
"capability": "variable_sequence_length",
|
| 469 |
+
"reference_ms": 968.6068749870174,
|
| 470 |
+
"converted_ms": 978.0740829883143,
|
| 471 |
+
"passed": true,
|
| 472 |
+
"numeric_passed": true,
|
| 473 |
+
"decision_agreement": true,
|
| 474 |
+
"max_absolute_error": 1.5057525626360757e-07,
|
| 475 |
+
"mean_absolute_error": 1.0038350412947909e-07,
|
| 476 |
+
"reference_probabilities": [
|
| 477 |
+
0.4407633906736511,
|
| 478 |
+
0.37238073879955924,
|
| 479 |
+
0.18685587052678965
|
| 480 |
+
],
|
| 481 |
+
"converted_probabilities": [
|
| 482 |
+
0.4407635277058288,
|
| 483 |
+
0.372380588224303,
|
| 484 |
+
0.1868558840698681
|
| 485 |
+
],
|
| 486 |
+
"reference_decision": "coding",
|
| 487 |
+
"converted_decision": "coding",
|
| 488 |
+
"expected": "coding",
|
| 489 |
+
"reference_correct": true,
|
| 490 |
+
"converted_correct": true,
|
| 491 |
+
"status": "passed"
|
| 492 |
+
},
|
| 493 |
+
{
|
| 494 |
+
"id": "exact-2048-tokens",
|
| 495 |
+
"capability": "variable_sequence_length",
|
| 496 |
+
"reference_ms": 2569.3836659920635,
|
| 497 |
+
"converted_ms": 2785.3902079950785,
|
| 498 |
+
"passed": true,
|
| 499 |
+
"numeric_passed": true,
|
| 500 |
+
"decision_agreement": true,
|
| 501 |
+
"max_absolute_error": 2.872474891835175e-07,
|
| 502 |
+
"mean_absolute_error": 1.914983261038413e-07,
|
| 503 |
+
"reference_probabilities": [
|
| 504 |
+
0.5216577502831695,
|
| 505 |
+
0.34918566249456723,
|
| 506 |
+
0.12915658722226325
|
| 507 |
+
],
|
| 508 |
+
"converted_probabilities": [
|
| 509 |
+
0.5216574953587729,
|
| 510 |
+
0.3491859497420564,
|
| 511 |
+
0.12915655489917072
|
| 512 |
+
],
|
| 513 |
+
"reference_decision": "coding",
|
| 514 |
+
"converted_decision": "coding",
|
| 515 |
+
"expected": "coding",
|
| 516 |
+
"reference_correct": true,
|
| 517 |
+
"converted_correct": true,
|
| 518 |
+
"status": "passed"
|
| 519 |
}
|
| 520 |
]
|
| 521 |
}
|
input_boundary.json
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"encoding_32768": "passed",
|
| 3 |
+
"encoding_32769": "rejected",
|
| 4 |
+
"truncation": false,
|
| 5 |
+
"native_32768_inference": "not tested"
|
| 6 |
+
}
|
manifest.json
CHANGED
|
@@ -5,57 +5,63 @@
|
|
| 5 |
"experimental": true,
|
| 6 |
"complete_upstream_model": false,
|
| 7 |
"sdk_integrated": false,
|
| 8 |
-
"artifact_scope": "
|
| 9 |
"format": "Core AI .aimodel",
|
| 10 |
"function": "main",
|
| 11 |
"output": "logits",
|
| 12 |
-
"sequence_length":
|
|
|
|
| 13 |
"batch_size": 1,
|
| 14 |
-
"option_count":
|
| 15 |
"tested_device": "cpu",
|
| 16 |
"inputs": {
|
| 17 |
"input_ids": {
|
| 18 |
"dtype": "int32",
|
| 19 |
"shape": [
|
| 20 |
1,
|
| 21 |
-
|
| 22 |
]
|
| 23 |
},
|
| 24 |
"attention_mask": {
|
| 25 |
"dtype": "int32",
|
| 26 |
"shape": [
|
| 27 |
1,
|
| 28 |
-
|
| 29 |
]
|
| 30 |
},
|
| 31 |
"option_positions": {
|
| 32 |
"dtype": "int32",
|
| 33 |
"shape": [
|
| 34 |
1,
|
| 35 |
-
|
| 36 |
]
|
| 37 |
},
|
| 38 |
"pool_mask": {
|
| 39 |
"dtype": "float32",
|
| 40 |
"shape": [
|
| 41 |
1,
|
| 42 |
-
|
| 43 |
]
|
| 44 |
}
|
| 45 |
},
|
| 46 |
"source_weights_sha256": "2294745881d9a01b7d049133b57e5fed2aec6ae1d34ef1ab9fa16f047fc35c8e",
|
| 47 |
"artifact_files_sha256": {
|
| 48 |
-
"
|
| 49 |
-
"
|
| 50 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 51 |
},
|
| 52 |
"export_versions": {
|
| 53 |
-
"torch": "2.
|
| 54 |
"huggingface-hub": "1.33.0",
|
| 55 |
"transformers": "5.17.0",
|
| 56 |
"safetensors": "0.8.0",
|
| 57 |
-
"coreai-torch": "0.4.
|
| 58 |
-
"coreai-core": "1.0.
|
| 59 |
},
|
| 60 |
"evaluation_versions": {
|
| 61 |
"torch": "2.14.0",
|
|
@@ -76,15 +82,21 @@
|
|
| 76 |
"span_sets": "untested",
|
| 77 |
"evidence": "untested",
|
| 78 |
"router_task_adapters": "untested",
|
| 79 |
-
"structured_state": "
|
| 80 |
"multiple_questions": "untested",
|
| 81 |
-
"variable_sequence_length": "
|
| 82 |
-
"variable_options": "
|
| 83 |
"system_one_response": "untested",
|
| 84 |
"domain_adapter": "untested",
|
| 85 |
"factcheck_adapter": "untested",
|
| 86 |
"feedback_adapter": "untested",
|
| 87 |
"feedback_df_adapter": "untested"
|
| 88 |
},
|
| 89 |
-
"initial_build_validation":
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
}
|
|
|
|
| 5 |
"experimental": true,
|
| 6 |
"complete_upstream_model": false,
|
| 7 |
"sdk_integrated": false,
|
| 8 |
+
"artifact_scope": "decision-only",
|
| 9 |
"format": "Core AI .aimodel",
|
| 10 |
"function": "main",
|
| 11 |
"output": "logits",
|
| 12 |
+
"sequence_length": 32768,
|
| 13 |
+
"dynamic_sequence_length": true,
|
| 14 |
"batch_size": 1,
|
| 15 |
+
"option_count": "dynamic",
|
| 16 |
"tested_device": "cpu",
|
| 17 |
"inputs": {
|
| 18 |
"input_ids": {
|
| 19 |
"dtype": "int32",
|
| 20 |
"shape": [
|
| 21 |
1,
|
| 22 |
+
"sequence"
|
| 23 |
]
|
| 24 |
},
|
| 25 |
"attention_mask": {
|
| 26 |
"dtype": "int32",
|
| 27 |
"shape": [
|
| 28 |
1,
|
| 29 |
+
"sequence"
|
| 30 |
]
|
| 31 |
},
|
| 32 |
"option_positions": {
|
| 33 |
"dtype": "int32",
|
| 34 |
"shape": [
|
| 35 |
1,
|
| 36 |
+
"options"
|
| 37 |
]
|
| 38 |
},
|
| 39 |
"pool_mask": {
|
| 40 |
"dtype": "float32",
|
| 41 |
"shape": [
|
| 42 |
1,
|
| 43 |
+
"sequence"
|
| 44 |
]
|
| 45 |
}
|
| 46 |
},
|
| 47 |
"source_weights_sha256": "2294745881d9a01b7d049133b57e5fed2aec6ae1d34ef1ab9fa16f047fc35c8e",
|
| 48 |
"artifact_files_sha256": {
|
| 49 |
+
"README.md": "aa3f9f987651372a96ba1ff633910ca5f499690e3f484ca790a94a247eb6ae19",
|
| 50 |
+
"assets/LICENSE": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4",
|
| 51 |
+
"assets/config.json": "a85b358ac11b71d780deb09e8a082efddc67938a2eb4fc16d68d8ae888c69d14",
|
| 52 |
+
"assets/tokenizer.json": "38734bb17ef19f84c67b5eb5692bea19801ffd02cb5c5ffc11429e76b0242e08",
|
| 53 |
+
"assets/tokenizer_config.json": "649781df90d51837dd6ba951a89fae07c985a9eacfd781d84002d480c3cb0b6e",
|
| 54 |
+
"model.aimodel/main.hash": "d1e3ae70a989d8d38ccae6513bf0b4085f751545cb6cd4efbd5373fa26f74fc4",
|
| 55 |
+
"model.aimodel/main.mlirb": "d7b63896067e505f67a03c1afe31e070b0a1f5d746e14d603d9130708798adad",
|
| 56 |
+
"model.aimodel/metadata.json": "a19b297a1df17828004220bd2a85bdd86f6630fd3edca8e9d3853366147d96b7"
|
| 57 |
},
|
| 58 |
"export_versions": {
|
| 59 |
+
"torch": "2.14.0",
|
| 60 |
"huggingface-hub": "1.33.0",
|
| 61 |
"transformers": "5.17.0",
|
| 62 |
"safetensors": "0.8.0",
|
| 63 |
+
"coreai-torch": "0.4.3",
|
| 64 |
+
"coreai-core": "1.0.0b3"
|
| 65 |
},
|
| 66 |
"evaluation_versions": {
|
| 67 |
"torch": "2.14.0",
|
|
|
|
| 82 |
"span_sets": "untested",
|
| 83 |
"evidence": "untested",
|
| 84 |
"router_task_adapters": "untested",
|
| 85 |
+
"structured_state": "exercised-in-diagnostic",
|
| 86 |
"multiple_questions": "untested",
|
| 87 |
+
"variable_sequence_length": "exercised-in-diagnostic",
|
| 88 |
+
"variable_options": "exercised-in-diagnostic",
|
| 89 |
"system_one_response": "untested",
|
| 90 |
"domain_adapter": "untested",
|
| 91 |
"factcheck_adapter": "untested",
|
| 92 |
"feedback_adapter": "untested",
|
| 93 |
"feedback_df_adapter": "untested"
|
| 94 |
},
|
| 95 |
+
"initial_build_validation": {
|
| 96 |
+
"samples": 1,
|
| 97 |
+
"max_absolute_error": 1.430511474609375e-06,
|
| 98 |
+
"atol": 0.02,
|
| 99 |
+
"rtol": 0.02,
|
| 100 |
+
"scope": "tensor smoke; task parity still required"
|
| 101 |
+
}
|
| 102 |
}
|
model.aimodel/main.hash
CHANGED
|
@@ -1 +1 @@
|
|
| 1 |
-
�
|
|
|
|
| 1 |
+
8�~P_g�<�1�p����F�M`=�0p����
|
model.aimodel/main.mlirb
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d7b63896067e505f67a03c1afe31e070b0a1f5d746e14d603d9130708798adad
|
| 3 |
+
size 1522191259
|
model.aimodel/metadata.json
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
{
|
| 2 |
-
"
|
| 3 |
-
"
|
| 4 |
-
"
|
| 5 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"creationDate" : "20261116T155013Z",
|
| 3 |
+
"producer" : "coreai-core 1.0.0b3",
|
| 4 |
+
"assetVersion" : "2.0"
|
| 5 |
}
|
package_smoke.json
CHANGED
|
@@ -1,64 +1,148 @@
|
|
| 1 |
{
|
| 2 |
"status": "passed",
|
| 3 |
-
"scope": "
|
| 4 |
"device": "cpu",
|
| 5 |
"cases": [
|
| 6 |
{
|
| 7 |
"id": "coding-en",
|
|
|
|
|
|
|
| 8 |
"decision": "coding",
|
| 9 |
-
"max_probability_error":
|
| 10 |
"probabilities": {
|
| 11 |
-
"coding": 0.
|
| 12 |
-
"cooking": 0.
|
| 13 |
}
|
| 14 |
},
|
| 15 |
{
|
| 16 |
"id": "cooking-zh",
|
|
|
|
|
|
|
| 17 |
"decision": "cooking",
|
| 18 |
-
"max_probability_error":
|
| 19 |
"probabilities": {
|
| 20 |
-
"coding": 0.
|
| 21 |
-
"cooking": 0.
|
| 22 |
}
|
| 23 |
},
|
| 24 |
{
|
| 25 |
"id": "noul-positive",
|
|
|
|
|
|
|
| 26 |
"decision": "yes",
|
| 27 |
-
"max_probability_error":
|
| 28 |
"probabilities": {
|
| 29 |
-
"no": 0.
|
| 30 |
-
"yes": 0.
|
| 31 |
}
|
| 32 |
},
|
| 33 |
{
|
| 34 |
"id": "noul-negative",
|
|
|
|
|
|
|
| 35 |
"decision": "no",
|
| 36 |
-
"max_probability_error":
|
| 37 |
"probabilities": {
|
| 38 |
-
"no": 0.
|
| 39 |
-
"yes": 0.
|
| 40 |
}
|
| 41 |
},
|
| 42 |
{
|
| 43 |
"id": "coding-zh",
|
|
|
|
|
|
|
| 44 |
"decision": "coding",
|
| 45 |
-
"max_probability_error":
|
| 46 |
"probabilities": {
|
| 47 |
-
"coding": 0.
|
| 48 |
-
"cooking": 0.
|
| 49 |
}
|
| 50 |
},
|
| 51 |
{
|
| 52 |
"id": "cooking-en",
|
|
|
|
|
|
|
| 53 |
"decision": "cooking",
|
| 54 |
-
"max_probability_error": 1.
|
| 55 |
"probabilities": {
|
| 56 |
-
"coding": 0.
|
| 57 |
-
"cooking": 0.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
}
|
| 59 |
}
|
| 60 |
],
|
| 61 |
-
"invalid_requests_rejected":
|
| 62 |
"pytorch_imported": false,
|
| 63 |
-
"requires_original_weights": false
|
|
|
|
|
|
|
| 64 |
}
|
|
|
|
| 1 |
{
|
| 2 |
"status": "passed",
|
| 3 |
+
"scope": "dynamic decision graph",
|
| 4 |
"device": "cpu",
|
| 5 |
"cases": [
|
| 6 |
{
|
| 7 |
"id": "coding-en",
|
| 8 |
+
"sequence_length": 24,
|
| 9 |
+
"option_count": 2,
|
| 10 |
"decision": "coding",
|
| 11 |
+
"max_probability_error": 5.427491678311469e-09,
|
| 12 |
"probabilities": {
|
| 13 |
+
"coding": 0.9964854064689901,
|
| 14 |
+
"cooking": 0.0035145935310098605
|
| 15 |
}
|
| 16 |
},
|
| 17 |
{
|
| 18 |
"id": "cooking-zh",
|
| 19 |
+
"sequence_length": 36,
|
| 20 |
+
"option_count": 2,
|
| 21 |
"decision": "cooking",
|
| 22 |
+
"max_probability_error": 4.7419820070615515e-07,
|
| 23 |
"probabilities": {
|
| 24 |
+
"coding": 0.22242242344232538,
|
| 25 |
+
"cooking": 0.7775775765576746
|
| 26 |
}
|
| 27 |
},
|
| 28 |
{
|
| 29 |
"id": "noul-positive",
|
| 30 |
+
"sequence_length": 16,
|
| 31 |
+
"option_count": 2,
|
| 32 |
"decision": "yes",
|
| 33 |
+
"max_probability_error": 1.3685062416612936e-07,
|
| 34 |
"probabilities": {
|
| 35 |
+
"no": 0.0428342160732897,
|
| 36 |
+
"yes": 0.9571657839267103
|
| 37 |
}
|
| 38 |
},
|
| 39 |
{
|
| 40 |
"id": "noul-negative",
|
| 41 |
+
"sequence_length": 17,
|
| 42 |
+
"option_count": 2,
|
| 43 |
"decision": "no",
|
| 44 |
+
"max_probability_error": 2.6513179342962934e-09,
|
| 45 |
"probabilities": {
|
| 46 |
+
"no": 0.9887530745498202,
|
| 47 |
+
"yes": 0.011246925450179683
|
| 48 |
}
|
| 49 |
},
|
| 50 |
{
|
| 51 |
"id": "coding-zh",
|
| 52 |
+
"sequence_length": 42,
|
| 53 |
+
"option_count": 2,
|
| 54 |
"decision": "coding",
|
| 55 |
+
"max_probability_error": 8.426156995235135e-09,
|
| 56 |
"probabilities": {
|
| 57 |
+
"coding": 0.9965400377467385,
|
| 58 |
+
"cooking": 0.0034599622532615615
|
| 59 |
}
|
| 60 |
},
|
| 61 |
{
|
| 62 |
"id": "cooking-en",
|
| 63 |
+
"sequence_length": 26,
|
| 64 |
+
"option_count": 2,
|
| 65 |
"decision": "cooking",
|
| 66 |
+
"max_probability_error": 1.5691086767584406e-09,
|
| 67 |
"probabilities": {
|
| 68 |
+
"coding": 0.0008233435277157704,
|
| 69 |
+
"cooking": 0.9991766564722843
|
| 70 |
+
}
|
| 71 |
+
},
|
| 72 |
+
{
|
| 73 |
+
"id": "three-options",
|
| 74 |
+
"sequence_length": 31,
|
| 75 |
+
"option_count": 3,
|
| 76 |
+
"decision": "coding",
|
| 77 |
+
"max_probability_error": 3.460630049190172e-08,
|
| 78 |
+
"probabilities": {
|
| 79 |
+
"coding": 0.99293799337413,
|
| 80 |
+
"cooking": 0.002825909094017193,
|
| 81 |
+
"sports": 0.004236097531852771
|
| 82 |
+
}
|
| 83 |
+
},
|
| 84 |
+
{
|
| 85 |
+
"id": "one-option",
|
| 86 |
+
"sequence_length": 19,
|
| 87 |
+
"option_count": 1,
|
| 88 |
+
"decision": "coding",
|
| 89 |
+
"max_probability_error": 0.0,
|
| 90 |
+
"probabilities": {
|
| 91 |
+
"coding": 1.0
|
| 92 |
+
}
|
| 93 |
+
},
|
| 94 |
+
{
|
| 95 |
+
"id": "structured-json",
|
| 96 |
+
"sequence_length": 37,
|
| 97 |
+
"option_count": 3,
|
| 98 |
+
"decision": "coding",
|
| 99 |
+
"max_probability_error": 1.4141390947131072e-08,
|
| 100 |
+
"probabilities": {
|
| 101 |
+
"coding": 0.9955305530014188,
|
| 102 |
+
"cooking": 0.002282103594566704,
|
| 103 |
+
"sports": 0.0021873434040145696
|
| 104 |
+
}
|
| 105 |
+
},
|
| 106 |
+
{
|
| 107 |
+
"id": "exact-129-tokens",
|
| 108 |
+
"sequence_length": 129,
|
| 109 |
+
"option_count": 3,
|
| 110 |
+
"decision": "coding",
|
| 111 |
+
"max_probability_error": 7.795353781370196e-07,
|
| 112 |
+
"probabilities": {
|
| 113 |
+
"coding": 0.6347575526842495,
|
| 114 |
+
"cooking": 0.13812251758558605,
|
| 115 |
+
"sports": 0.2271199297301645
|
| 116 |
+
}
|
| 117 |
+
},
|
| 118 |
+
{
|
| 119 |
+
"id": "exact-1024-tokens",
|
| 120 |
+
"sequence_length": 1024,
|
| 121 |
+
"option_count": 3,
|
| 122 |
+
"decision": "coding",
|
| 123 |
+
"max_probability_error": 1.5057525626360757e-07,
|
| 124 |
+
"probabilities": {
|
| 125 |
+
"coding": 0.4407635277058288,
|
| 126 |
+
"cooking": 0.372380588224303,
|
| 127 |
+
"sports": 0.1868558840698681
|
| 128 |
+
}
|
| 129 |
+
},
|
| 130 |
+
{
|
| 131 |
+
"id": "exact-2048-tokens",
|
| 132 |
+
"sequence_length": 2048,
|
| 133 |
+
"option_count": 3,
|
| 134 |
+
"decision": "coding",
|
| 135 |
+
"max_probability_error": 2.872474891835175e-07,
|
| 136 |
+
"probabilities": {
|
| 137 |
+
"coding": 0.5216574953587729,
|
| 138 |
+
"cooking": 0.3491859497420564,
|
| 139 |
+
"sports": 0.12915655489917072
|
| 140 |
}
|
| 141 |
}
|
| 142 |
],
|
| 143 |
+
"invalid_requests_rejected": 3,
|
| 144 |
"pytorch_imported": false,
|
| 145 |
+
"requires_original_weights": false,
|
| 146 |
+
"multiple_questions": "passed",
|
| 147 |
+
"list_criteria": "passed"
|
| 148 |
}
|
smoke.py
CHANGED
|
@@ -25,10 +25,14 @@ async def main() -> None:
|
|
| 25 |
reference = np.array(case["reference_probabilities"])
|
| 26 |
np.testing.assert_allclose(values, reference, atol=0.02, rtol=0.02)
|
| 27 |
decision = max(probabilities, key=probabilities.__getitem__)
|
| 28 |
-
assert decision == case["reference_decision"]
|
|
|
|
|
|
|
| 29 |
rows.append(
|
| 30 |
{
|
| 31 |
"id": case["id"],
|
|
|
|
|
|
|
| 32 |
"decision": decision,
|
| 33 |
"max_probability_error": float(np.max(np.abs(values - reference))),
|
| 34 |
"probabilities": probabilities,
|
|
@@ -47,31 +51,46 @@ async def main() -> None:
|
|
| 47 |
rejected = 0
|
| 48 |
for state, question in [
|
| 49 |
("text", {"type": "span"}),
|
| 50 |
-
("text", {"type": "choice", "criteria": {
|
| 51 |
-
("
|
| 52 |
-
("word " * 500, first["payload"]["question"]),
|
| 53 |
-
(["image"], first["payload"]["question"]),
|
| 54 |
]:
|
| 55 |
try:
|
| 56 |
model.pack(state, question)
|
| 57 |
except ValueError:
|
| 58 |
rejected += 1
|
| 59 |
-
assert rejected ==
|
| 60 |
# Host adapter must preserve the public choice/noul response shape.
|
| 61 |
answer = await model.predict(first["payload"]["state"], first["payload"]["question"])
|
| 62 |
assert answer["type"] == "choice" and answer["choice"] == first["expected"]
|
| 63 |
noul = cases[2]
|
| 64 |
answer = await model.predict(noul["payload"]["state"], noul["payload"]["question"])
|
| 65 |
assert answer["type"] == "noul" and 0 <= answer["noul"] <= 1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 66 |
assert "torch" not in sys.modules
|
| 67 |
report = {
|
| 68 |
"status": "passed",
|
| 69 |
-
"scope": "
|
| 70 |
"device": "cpu",
|
| 71 |
"cases": rows,
|
| 72 |
"invalid_requests_rejected": rejected,
|
| 73 |
"pytorch_imported": False,
|
| 74 |
"requires_original_weights": False,
|
|
|
|
|
|
|
| 75 |
}
|
| 76 |
(root / "package_smoke.json").write_text(json.dumps(report, indent=2) + "\n")
|
| 77 |
print(json.dumps(report, indent=2))
|
|
|
|
| 25 |
reference = np.array(case["reference_probabilities"])
|
| 26 |
np.testing.assert_allclose(values, reference, atol=0.02, rtol=0.02)
|
| 27 |
decision = max(probabilities, key=probabilities.__getitem__)
|
| 28 |
+
assert decision == case["reference_decision"]
|
| 29 |
+
if case.get("expected") is not None:
|
| 30 |
+
assert decision == case["expected"]
|
| 31 |
rows.append(
|
| 32 |
{
|
| 33 |
"id": case["id"],
|
| 34 |
+
"sequence_length": int(arrays["attention_mask"].sum()),
|
| 35 |
+
"option_count": len(labels),
|
| 36 |
"decision": decision,
|
| 37 |
"max_probability_error": float(np.max(np.abs(values - reference))),
|
| 38 |
"probabilities": probabilities,
|
|
|
|
| 51 |
rejected = 0
|
| 52 |
for state, question in [
|
| 53 |
("text", {"type": "span"}),
|
| 54 |
+
("text", {"type": "choice", "criteria": {}}),
|
| 55 |
+
("word " * (model.manifest["sequence_length"] + 1), first["payload"]["question"]),
|
|
|
|
|
|
|
| 56 |
]:
|
| 57 |
try:
|
| 58 |
model.pack(state, question)
|
| 59 |
except ValueError:
|
| 60 |
rejected += 1
|
| 61 |
+
assert rejected == 3
|
| 62 |
# Host adapter must preserve the public choice/noul response shape.
|
| 63 |
answer = await model.predict(first["payload"]["state"], first["payload"]["question"])
|
| 64 |
assert answer["type"] == "choice" and answer["choice"] == first["expected"]
|
| 65 |
noul = cases[2]
|
| 66 |
answer = await model.predict(noul["payload"]["state"], noul["payload"]["question"])
|
| 67 |
assert answer["type"] == "noul" and 0 <= answer["noul"] <= 1
|
| 68 |
+
questions = {
|
| 69 |
+
"topic": first["payload"]["question"],
|
| 70 |
+
"software": noul["payload"]["question"],
|
| 71 |
+
}
|
| 72 |
+
answers, usage = await model.predict_questions(first["payload"]["state"], questions)
|
| 73 |
+
assert list(answers) == list(questions) and usage > 0
|
| 74 |
+
for key, question in questions.items():
|
| 75 |
+
assert answers[key] == await model.predict(first["payload"]["state"], question)
|
| 76 |
+
list_question = {"type": "choice", "criteria": ["coding", "cooking", "sports"]}
|
| 77 |
+
map_question = {**list_question, "criteria": dict.fromkeys(list_question["criteria"])}
|
| 78 |
+
left, left_labels = model.pack(None, list_question)
|
| 79 |
+
right, right_labels = model.pack(None, map_question)
|
| 80 |
+
assert left_labels == right_labels
|
| 81 |
+
for name in left:
|
| 82 |
+
np.testing.assert_array_equal(left[name], right[name])
|
| 83 |
assert "torch" not in sys.modules
|
| 84 |
report = {
|
| 85 |
"status": "passed",
|
| 86 |
+
"scope": "dynamic decision graph",
|
| 87 |
"device": "cpu",
|
| 88 |
"cases": rows,
|
| 89 |
"invalid_requests_rejected": rejected,
|
| 90 |
"pytorch_imported": False,
|
| 91 |
"requires_original_weights": False,
|
| 92 |
+
"multiple_questions": "passed",
|
| 93 |
+
"list_criteria": "passed",
|
| 94 |
}
|
| 95 |
(root / "package_smoke.json").write_text(json.dumps(report, indent=2) + "\n")
|
| 96 |
print(json.dumps(report, indent=2))
|
swift_fixture.json
CHANGED
|
@@ -23,110 +23,6 @@
|
|
| 23 |
5905,
|
| 24 |
1517,
|
| 25 |
30,
|
| 26 |
-
0,
|
| 27 |
-
0,
|
| 28 |
-
0,
|
| 29 |
-
0,
|
| 30 |
-
0,
|
| 31 |
-
0,
|
| 32 |
-
0,
|
| 33 |
-
0,
|
| 34 |
-
0,
|
| 35 |
-
0,
|
| 36 |
-
0,
|
| 37 |
-
0,
|
| 38 |
-
0,
|
| 39 |
-
0,
|
| 40 |
-
0,
|
| 41 |
-
0,
|
| 42 |
-
0,
|
| 43 |
-
0,
|
| 44 |
-
0,
|
| 45 |
-
0,
|
| 46 |
-
0,
|
| 47 |
-
0,
|
| 48 |
-
0,
|
| 49 |
-
0,
|
| 50 |
-
0,
|
| 51 |
-
0,
|
| 52 |
-
0,
|
| 53 |
-
0,
|
| 54 |
-
0,
|
| 55 |
-
0,
|
| 56 |
-
0,
|
| 57 |
-
0,
|
| 58 |
-
0,
|
| 59 |
-
0,
|
| 60 |
-
0,
|
| 61 |
-
0,
|
| 62 |
-
0,
|
| 63 |
-
0,
|
| 64 |
-
0,
|
| 65 |
-
0,
|
| 66 |
-
0,
|
| 67 |
-
0,
|
| 68 |
-
0,
|
| 69 |
-
0,
|
| 70 |
-
0,
|
| 71 |
-
0,
|
| 72 |
-
0,
|
| 73 |
-
0,
|
| 74 |
-
0,
|
| 75 |
-
0,
|
| 76 |
-
0,
|
| 77 |
-
0,
|
| 78 |
-
0,
|
| 79 |
-
0,
|
| 80 |
-
0,
|
| 81 |
-
0,
|
| 82 |
-
0,
|
| 83 |
-
0,
|
| 84 |
-
0,
|
| 85 |
-
0,
|
| 86 |
-
0,
|
| 87 |
-
0,
|
| 88 |
-
0,
|
| 89 |
-
0,
|
| 90 |
-
0,
|
| 91 |
-
0,
|
| 92 |
-
0,
|
| 93 |
-
0,
|
| 94 |
-
0,
|
| 95 |
-
0,
|
| 96 |
-
0,
|
| 97 |
-
0,
|
| 98 |
-
0,
|
| 99 |
-
0,
|
| 100 |
-
0,
|
| 101 |
-
0,
|
| 102 |
-
0,
|
| 103 |
-
0,
|
| 104 |
-
0,
|
| 105 |
-
0,
|
| 106 |
-
0,
|
| 107 |
-
0,
|
| 108 |
-
0,
|
| 109 |
-
0,
|
| 110 |
-
0,
|
| 111 |
-
0,
|
| 112 |
-
0,
|
| 113 |
-
0,
|
| 114 |
-
0,
|
| 115 |
-
0,
|
| 116 |
-
0,
|
| 117 |
-
0,
|
| 118 |
-
0,
|
| 119 |
-
0,
|
| 120 |
-
0,
|
| 121 |
-
0,
|
| 122 |
-
0,
|
| 123 |
-
0,
|
| 124 |
-
0,
|
| 125 |
-
0,
|
| 126 |
-
0,
|
| 127 |
-
0,
|
| 128 |
-
0,
|
| 129 |
-
0,
|
| 130 |
0
|
| 131 |
],
|
| 132 |
"attention_mask": [
|
|
@@ -153,111 +49,7 @@
|
|
| 153 |
1,
|
| 154 |
1,
|
| 155 |
1,
|
| 156 |
-
1
|
| 157 |
-
0,
|
| 158 |
-
0,
|
| 159 |
-
0,
|
| 160 |
-
0,
|
| 161 |
-
0,
|
| 162 |
-
0,
|
| 163 |
-
0,
|
| 164 |
-
0,
|
| 165 |
-
0,
|
| 166 |
-
0,
|
| 167 |
-
0,
|
| 168 |
-
0,
|
| 169 |
-
0,
|
| 170 |
-
0,
|
| 171 |
-
0,
|
| 172 |
-
0,
|
| 173 |
-
0,
|
| 174 |
-
0,
|
| 175 |
-
0,
|
| 176 |
-
0,
|
| 177 |
-
0,
|
| 178 |
-
0,
|
| 179 |
-
0,
|
| 180 |
-
0,
|
| 181 |
-
0,
|
| 182 |
-
0,
|
| 183 |
-
0,
|
| 184 |
-
0,
|
| 185 |
-
0,
|
| 186 |
-
0,
|
| 187 |
-
0,
|
| 188 |
-
0,
|
| 189 |
-
0,
|
| 190 |
-
0,
|
| 191 |
-
0,
|
| 192 |
-
0,
|
| 193 |
-
0,
|
| 194 |
-
0,
|
| 195 |
-
0,
|
| 196 |
-
0,
|
| 197 |
-
0,
|
| 198 |
-
0,
|
| 199 |
-
0,
|
| 200 |
-
0,
|
| 201 |
-
0,
|
| 202 |
-
0,
|
| 203 |
-
0,
|
| 204 |
-
0,
|
| 205 |
-
0,
|
| 206 |
-
0,
|
| 207 |
-
0,
|
| 208 |
-
0,
|
| 209 |
-
0,
|
| 210 |
-
0,
|
| 211 |
-
0,
|
| 212 |
-
0,
|
| 213 |
-
0,
|
| 214 |
-
0,
|
| 215 |
-
0,
|
| 216 |
-
0,
|
| 217 |
-
0,
|
| 218 |
-
0,
|
| 219 |
-
0,
|
| 220 |
-
0,
|
| 221 |
-
0,
|
| 222 |
-
0,
|
| 223 |
-
0,
|
| 224 |
-
0,
|
| 225 |
-
0,
|
| 226 |
-
0,
|
| 227 |
-
0,
|
| 228 |
-
0,
|
| 229 |
-
0,
|
| 230 |
-
0,
|
| 231 |
-
0,
|
| 232 |
-
0,
|
| 233 |
-
0,
|
| 234 |
-
0,
|
| 235 |
-
0,
|
| 236 |
-
0,
|
| 237 |
-
0,
|
| 238 |
-
0,
|
| 239 |
-
0,
|
| 240 |
-
0,
|
| 241 |
-
0,
|
| 242 |
-
0,
|
| 243 |
-
0,
|
| 244 |
-
0,
|
| 245 |
-
0,
|
| 246 |
-
0,
|
| 247 |
-
0,
|
| 248 |
-
0,
|
| 249 |
-
0,
|
| 250 |
-
0,
|
| 251 |
-
0,
|
| 252 |
-
0,
|
| 253 |
-
0,
|
| 254 |
-
0,
|
| 255 |
-
0,
|
| 256 |
-
0,
|
| 257 |
-
0,
|
| 258 |
-
0,
|
| 259 |
-
0,
|
| 260 |
-
0
|
| 261 |
],
|
| 262 |
"option_positions": [
|
| 263 |
7,
|
|
@@ -287,115 +79,11 @@
|
|
| 287 |
1.0,
|
| 288 |
1.0,
|
| 289 |
1.0,
|
| 290 |
-
0.0,
|
| 291 |
-
0.0,
|
| 292 |
-
0.0,
|
| 293 |
-
0.0,
|
| 294 |
-
0.0,
|
| 295 |
-
0.0,
|
| 296 |
-
0.0,
|
| 297 |
-
0.0,
|
| 298 |
-
0.0,
|
| 299 |
-
0.0,
|
| 300 |
-
0.0,
|
| 301 |
-
0.0,
|
| 302 |
-
0.0,
|
| 303 |
-
0.0,
|
| 304 |
-
0.0,
|
| 305 |
-
0.0,
|
| 306 |
-
0.0,
|
| 307 |
-
0.0,
|
| 308 |
-
0.0,
|
| 309 |
-
0.0,
|
| 310 |
-
0.0,
|
| 311 |
-
0.0,
|
| 312 |
-
0.0,
|
| 313 |
-
0.0,
|
| 314 |
-
0.0,
|
| 315 |
-
0.0,
|
| 316 |
-
0.0,
|
| 317 |
-
0.0,
|
| 318 |
-
0.0,
|
| 319 |
-
0.0,
|
| 320 |
-
0.0,
|
| 321 |
-
0.0,
|
| 322 |
-
0.0,
|
| 323 |
-
0.0,
|
| 324 |
-
0.0,
|
| 325 |
-
0.0,
|
| 326 |
-
0.0,
|
| 327 |
-
0.0,
|
| 328 |
-
0.0,
|
| 329 |
-
0.0,
|
| 330 |
-
0.0,
|
| 331 |
-
0.0,
|
| 332 |
-
0.0,
|
| 333 |
-
0.0,
|
| 334 |
-
0.0,
|
| 335 |
-
0.0,
|
| 336 |
-
0.0,
|
| 337 |
-
0.0,
|
| 338 |
-
0.0,
|
| 339 |
-
0.0,
|
| 340 |
-
0.0,
|
| 341 |
-
0.0,
|
| 342 |
-
0.0,
|
| 343 |
-
0.0,
|
| 344 |
-
0.0,
|
| 345 |
-
0.0,
|
| 346 |
-
0.0,
|
| 347 |
-
0.0,
|
| 348 |
-
0.0,
|
| 349 |
-
0.0,
|
| 350 |
-
0.0,
|
| 351 |
-
0.0,
|
| 352 |
-
0.0,
|
| 353 |
-
0.0,
|
| 354 |
-
0.0,
|
| 355 |
-
0.0,
|
| 356 |
-
0.0,
|
| 357 |
-
0.0,
|
| 358 |
-
0.0,
|
| 359 |
-
0.0,
|
| 360 |
-
0.0,
|
| 361 |
-
0.0,
|
| 362 |
-
0.0,
|
| 363 |
-
0.0,
|
| 364 |
-
0.0,
|
| 365 |
-
0.0,
|
| 366 |
-
0.0,
|
| 367 |
-
0.0,
|
| 368 |
-
0.0,
|
| 369 |
-
0.0,
|
| 370 |
-
0.0,
|
| 371 |
-
0.0,
|
| 372 |
-
0.0,
|
| 373 |
-
0.0,
|
| 374 |
-
0.0,
|
| 375 |
-
0.0,
|
| 376 |
-
0.0,
|
| 377 |
-
0.0,
|
| 378 |
-
0.0,
|
| 379 |
-
0.0,
|
| 380 |
-
0.0,
|
| 381 |
-
0.0,
|
| 382 |
-
0.0,
|
| 383 |
-
0.0,
|
| 384 |
-
0.0,
|
| 385 |
-
0.0,
|
| 386 |
-
0.0,
|
| 387 |
-
0.0,
|
| 388 |
-
0.0,
|
| 389 |
-
0.0,
|
| 390 |
-
0.0,
|
| 391 |
-
0.0,
|
| 392 |
-
0.0,
|
| 393 |
-
0.0,
|
| 394 |
0.0
|
| 395 |
],
|
| 396 |
"expected": [
|
| 397 |
-
0.
|
| 398 |
-
0.
|
| 399 |
],
|
| 400 |
"labels": [
|
| 401 |
"coding",
|
|
|
|
| 23 |
5905,
|
| 24 |
1517,
|
| 25 |
30,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
0
|
| 27 |
],
|
| 28 |
"attention_mask": [
|
|
|
|
| 49 |
1,
|
| 50 |
1,
|
| 51 |
1,
|
| 52 |
+
1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 53 |
],
|
| 54 |
"option_positions": [
|
| 55 |
7,
|
|
|
|
| 79 |
1.0,
|
| 80 |
1.0,
|
| 81 |
1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 82 |
0.0
|
| 83 |
],
|
| 84 |
"expected": [
|
| 85 |
+
0.9964854064689901,
|
| 86 |
+
0.0035145935310098605
|
| 87 |
],
|
| 88 |
"labels": [
|
| 89 |
"coding",
|
swift_smoke.json
CHANGED
|
@@ -7,7 +7,7 @@
|
|
| 7 |
"maximum_probability_error_vs_python" : 5.9604644775390625e-08,
|
| 8 |
"probabilities" : [
|
| 9 |
0.99648535251617432,
|
| 10 |
-
0.
|
| 11 |
],
|
| 12 |
"swift_native_coreai" : true
|
| 13 |
}
|
|
|
|
| 7 |
"maximum_probability_error_vs_python" : 5.9604644775390625e-08,
|
| 8 |
"probabilities" : [
|
| 9 |
0.99648535251617432,
|
| 10 |
+
0.0035145930014550686
|
| 11 |
],
|
| 12 |
"swift_native_coreai" : true
|
| 13 |
}
|
validation_cases.json
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
vela_coreai.py
CHANGED
|
@@ -49,27 +49,23 @@ class VelaCoreAI:
|
|
| 49 |
return (
|
| 50 |
value
|
| 51 |
if isinstance(value, str)
|
| 52 |
-
else json.dumps(value, ensure_ascii=False, separators=(",", ":")
|
| 53 |
)
|
| 54 |
|
| 55 |
def pack(self, state: Any, question: Any) -> tuple[dict[str, np.ndarray[Any, Any]], list[str]]:
|
| 56 |
-
if not isinstance(
|
| 57 |
-
raise ValueError("
|
| 58 |
-
if not isinstance(question, dict) or set(question) - {"type", "instructions", "criteria"}:
|
| 59 |
-
raise ValueError("Provide one choice/noul question; unknown fields are rejected")
|
| 60 |
kind = question.get("type")
|
| 61 |
options: dict[str, Any]
|
| 62 |
if kind == "noul":
|
| 63 |
-
if "criteria" in question:
|
| 64 |
-
raise ValueError("noul uses the fixed no/yes options")
|
| 65 |
options = {"no": "", "yes": ""}
|
| 66 |
elif kind == "choice":
|
| 67 |
criteria = question.get("criteria")
|
| 68 |
-
if
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
| 73 |
else:
|
| 74 |
raise ValueError("Only choice and noul are included in this experimental artifact")
|
| 75 |
|
|
@@ -86,12 +82,15 @@ class VelaCoreAI:
|
|
| 86 |
ids += [markers["[O]"], *encode(f"{name}: {text}" if text else name)]
|
| 87 |
ids.append(markers["[SEP_SCHEMA]"])
|
| 88 |
start = len(ids)
|
| 89 |
-
ids += [markers["[SEG_user]"], *encode(self._text(state))]
|
| 90 |
end = len(ids)
|
| 91 |
ids.append(self.config["eos_token_id"])
|
| 92 |
-
|
| 93 |
-
if len(ids) >
|
| 94 |
-
raise ValueError(
|
|
|
|
|
|
|
|
|
|
| 95 |
input_ids = np.full((1, width), self.config["pad_token_id"], dtype=np.int32)
|
| 96 |
input_ids[0, : len(ids)] = ids
|
| 97 |
attention_mask = np.zeros((1, width), dtype=np.int32)
|
|
@@ -117,7 +116,7 @@ class VelaCoreAI:
|
|
| 117 |
try:
|
| 118 |
output = await self._function({k: NDArray(v) for k, v in arrays.items()})
|
| 119 |
logits = np.asarray(output[self.manifest["output"]].numpy(), dtype=np.float64)
|
| 120 |
-
if logits.shape != (1,
|
| 121 |
raise FloatingPointError("Invalid Core AI logits")
|
| 122 |
probabilities = np.exp(logits[0] - logits[0].max())
|
| 123 |
probabilities /= probabilities.sum()
|
|
@@ -134,3 +133,14 @@ class VelaCoreAI:
|
|
| 134 |
"choice": max(probabilities, key=probabilities.__getitem__),
|
| 135 |
"probabilities": probabilities,
|
| 136 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
return (
|
| 50 |
value
|
| 51 |
if isinstance(value, str)
|
| 52 |
+
else json.dumps(value, ensure_ascii=False, separators=(",", ":"))
|
| 53 |
)
|
| 54 |
|
| 55 |
def pack(self, state: Any, question: Any) -> tuple[dict[str, np.ndarray[Any, Any]], list[str]]:
|
| 56 |
+
if not isinstance(question, dict):
|
| 57 |
+
raise ValueError("Provide a choice/noul question object")
|
|
|
|
|
|
|
| 58 |
kind = question.get("type")
|
| 59 |
options: dict[str, Any]
|
| 60 |
if kind == "noul":
|
|
|
|
|
|
|
| 61 |
options = {"no": "", "yes": ""}
|
| 62 |
elif kind == "choice":
|
| 63 |
criteria = question.get("criteria")
|
| 64 |
+
if isinstance(criteria, (list, tuple)):
|
| 65 |
+
criteria = {str(value): None for value in criteria}
|
| 66 |
+
if not isinstance(criteria, dict) or not criteria:
|
| 67 |
+
raise ValueError("choice requires a nonempty criteria mapping")
|
| 68 |
+
options = {str(k): v for k, v in criteria.items()}
|
| 69 |
else:
|
| 70 |
raise ValueError("Only choice and noul are included in this experimental artifact")
|
| 71 |
|
|
|
|
| 82 |
ids += [markers["[O]"], *encode(f"{name}: {text}" if text else name)]
|
| 83 |
ids.append(markers["[SEP_SCHEMA]"])
|
| 84 |
start = len(ids)
|
| 85 |
+
ids += [markers["[SEG_user]"], *encode(self._text(state if state is not None else ""))]
|
| 86 |
end = len(ids)
|
| 87 |
ids.append(self.config["eos_token_id"])
|
| 88 |
+
capacity = self.manifest["sequence_length"]
|
| 89 |
+
if len(ids) > capacity:
|
| 90 |
+
raise ValueError(
|
| 91 |
+
f"Prompt needs {len(ids)} tokens; capacity is {capacity}; no truncation"
|
| 92 |
+
)
|
| 93 |
+
width = len(ids) if self.manifest.get("dynamic_sequence_length") else capacity
|
| 94 |
input_ids = np.full((1, width), self.config["pad_token_id"], dtype=np.int32)
|
| 95 |
input_ids[0, : len(ids)] = ids
|
| 96 |
attention_mask = np.zeros((1, width), dtype=np.int32)
|
|
|
|
| 116 |
try:
|
| 117 |
output = await self._function({k: NDArray(v) for k, v in arrays.items()})
|
| 118 |
logits = np.asarray(output[self.manifest["output"]].numpy(), dtype=np.float64)
|
| 119 |
+
if logits.shape != (1, len(labels)) or not np.isfinite(logits).all():
|
| 120 |
raise FloatingPointError("Invalid Core AI logits")
|
| 121 |
probabilities = np.exp(logits[0] - logits[0].max())
|
| 122 |
probabilities /= probabilities.sum()
|
|
|
|
| 133 |
"choice": max(probabilities, key=probabilities.__getitem__),
|
| 134 |
"probabilities": probabilities,
|
| 135 |
}
|
| 136 |
+
|
| 137 |
+
async def predict_questions(
|
| 138 |
+
self, state: Any, questions: dict[str, Any]
|
| 139 |
+
) -> tuple[dict[str, Any], int]:
|
| 140 |
+
"""Preserve DecisionEncoder.predict's question mapping and token usage."""
|
| 141 |
+
packed = {key: self.pack(state, question) for key, question in questions.items()}
|
| 142 |
+
answers = {}
|
| 143 |
+
for key, question in questions.items():
|
| 144 |
+
answers[key] = await self.predict(state, question)
|
| 145 |
+
tokens = sum(int(arrays["attention_mask"].sum()) for arrays, _ in packed.values())
|
| 146 |
+
return answers, tokens
|
verify_coreai.swift
CHANGED
|
@@ -34,18 +34,18 @@ struct Fixture: Decodable {
|
|
| 34 |
("attention_mask", fixture.attention_mask),
|
| 35 |
("option_positions", fixture.option_positions),
|
| 36 |
] {
|
| 37 |
-
guard case .ndArray
|
| 38 |
fatalError("Missing input \(name)")
|
| 39 |
}
|
| 40 |
-
var array = NDArray(
|
| 41 |
var view = array.mutableView(as: Int32.self)
|
| 42 |
view.copyElements(fromContentsOf: values)
|
| 43 |
inputs[name] = array
|
| 44 |
}
|
| 45 |
-
guard case .ndArray
|
| 46 |
fatalError("Missing pool_mask")
|
| 47 |
}
|
| 48 |
-
var pool = NDArray(
|
| 49 |
var poolView = pool.mutableView(as: Float.self)
|
| 50 |
poolView.copyElements(fromContentsOf: fixture.pool_mask)
|
| 51 |
inputs["pool_mask"] = pool
|
|
@@ -54,7 +54,7 @@ struct Fixture: Decodable {
|
|
| 54 |
fatalError("Missing logits")
|
| 55 |
}
|
| 56 |
let logits: [Float] = output.view(as: Float.self).withUnsafePointer { ptr, _, _ in
|
| 57 |
-
Array(UnsafeBufferPointer(start: ptr, count:
|
| 58 |
}
|
| 59 |
guard logits.allSatisfy({ $0.isFinite }) else { fatalError("Nonfinite logits") }
|
| 60 |
let maximum = logits.max()!
|
|
|
|
| 34 |
("attention_mask", fixture.attention_mask),
|
| 35 |
("option_positions", fixture.option_positions),
|
| 36 |
] {
|
| 37 |
+
guard case .ndArray = descriptor.inputDescriptor(of: name) else {
|
| 38 |
fatalError("Missing input \(name)")
|
| 39 |
}
|
| 40 |
+
var array = NDArray(shape: [1, values.count], scalarType: .int32)
|
| 41 |
var view = array.mutableView(as: Int32.self)
|
| 42 |
view.copyElements(fromContentsOf: values)
|
| 43 |
inputs[name] = array
|
| 44 |
}
|
| 45 |
+
guard case .ndArray = descriptor.inputDescriptor(of: "pool_mask") else {
|
| 46 |
fatalError("Missing pool_mask")
|
| 47 |
}
|
| 48 |
+
var pool = NDArray(shape: [1, fixture.pool_mask.count], scalarType: .float32)
|
| 49 |
var poolView = pool.mutableView(as: Float.self)
|
| 50 |
poolView.copyElements(fromContentsOf: fixture.pool_mask)
|
| 51 |
inputs["pool_mask"] = pool
|
|
|
|
| 54 |
fatalError("Missing logits")
|
| 55 |
}
|
| 56 |
let logits: [Float] = output.view(as: Float.self).withUnsafePointer { ptr, _, _ in
|
| 57 |
+
Array(UnsafeBufferPointer(start: ptr, count: fixture.labels.count))
|
| 58 |
}
|
| 59 |
guard logits.allSatisfy({ $0.isFinite }) else { fatalError("Nonfinite logits") }
|
| 60 |
let maximum = logits.max()!
|