File size: 8,728 Bytes
e5e4742
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7a58a50
e5e4742
7a58a50
6aaa9cc
 
7a58a50
e5e4742
7a58a50
e5e4742
7a58a50
e5e4742
6aaa9cc
7a58a50
6aaa9cc
 
7a58a50
 
6aaa9cc
7a58a50
e5e4742
7a58a50
 
 
 
e5e4742
 
 
 
 
 
 
 
 
 
 
 
 
7a58a50
e5e4742
 
7a58a50
e5e4742
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7a58a50
 
6aaa9cc
e5e4742
7a58a50
e5e4742
 
 
 
 
6aaa9cc
 
e5e4742
7a58a50
e5e4742
7a58a50
 
 
e5e4742
 
7a58a50
e5e4742
 
 
 
 
 
7a58a50
e5e4742
 
 
 
 
 
 
 
 
 
 
 
7a58a50
e5e4742
 
 
 
6aaa9cc
e5e4742
 
7a58a50
 
e5e4742
7a58a50
e5e4742
6aaa9cc
e5e4742
7a58a50
e5e4742
6aaa9cc
 
 
 
 
e5e4742
7a58a50
e5e4742
 
 
 
 
 
7a58a50
e5e4742
6aaa9cc
 
 
7a58a50
6aaa9cc
 
 
 
 
e5e4742
7a58a50
 
e5e4742
7a58a50
e5e4742
7a58a50
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
---
language:
- en
- zh
license: apache-2.0
base_model: vllm-sr/vela3-fast
pipeline_tag: zero-shot-classification
tags:
- coreai
- apple-silicon
- aimodel
- experimental
- decision-model
---

# Vela3-fast Core AI — experimental decision graph

English | [简体中文](README.zh-CN.md)

An Apple Core AI conversion of [vllm-sr/vela3-fast](https://huggingface.co/vllm-sr/vela3-fast) by hugging-mac.
This release contains the validated **choice / noul decision backbone**: one sample per graph call, dynamic sequence length, and a dynamic number of options.
The export retains the upstream **32,768-token** input bound; actual CPU parity tests currently cover lengths through **2,048 tokens**, including exactly 1,024.
**It is an experimental subset, not the complete Vela3-fast model with all task adapters.**

Neural inference runs through Core AI, with host-side tokenization and response formatting. The model does not generate text.

## Supported contract

- `choice`: one or more ordered options; returns their probabilities and the selected option.
- `noul`: fixed `no` / `yes` options; returns `p(yes)`.
- The input text, question, option descriptions, and special markers together must fit within the upstream **32,768-token** bound. Longer inputs raise an error; they are not truncated.
- Each graph call has batch size 1. `predict_questions(state, questions)` accepts a mapping of questions and returns `(answers, input_tokens)`, matching the upstream decision API; questions execute sequentially. The adapter preserves upstream JSON serialization and accepts mapping or list choice criteria.
- CPU execution is validated. Earlier automatic-device validation did not finish because of resource pressure on the test host; no GPU/ANE performance is claimed.
- Original FP32 decision weights, with no quantization, training, or fine-tuning.
- Router, span, relation, NER, evidence, domain/factcheck/feedback, and other upstream adapters are not included. Full `system_one` adapter routing is not implemented by this decision-only artifact. Untested capabilities are not known to be unconvertible.
- This release does not provide complete `hugging_mac_sdk` registration or production acceptance. Probabilities are relative to the supplied options, not calibrated confidence scores.

## Python usage

Validated on Apple Silicon, macOS 27.2.1, Python 3.12.13, and `coreai-core==1.0.0b3`.
A macOS version supporting the Core AI system runtime is required. The example environment uses uv and **does not require PyTorch, transformers, or the original safetensors weights**.

```bash
uvx --from huggingface-hub hf download hugging-mac/vela3-fast-CoreAI --local-dir vela3-fast-CoreAI
cd vela3-fast-CoreAI
uv sync --locked
USE_OS_COREAI=1 uv run --locked python example.py
```

```python
import asyncio
from vela_coreai import VelaCoreAI

async def main():
    # Point to the downloaded repository; the adapter explicitly uses CPU mode.
    async with VelaCoreAI.open(".") as model:
        result = await model.predict(
            "Write a Python sorting function.",
            {
                "type": "choice",
                "instructions": "What is the topic?",
                "criteria": {"coding": "Software", "cooking": "Food"},
            },
        )
        print(result)  # type, choice, probabilities

        yes_no = await model.predict(
            "Python code",
            {"type": "noul", "instructions": "Is this about software?"},
        )
        print(yes_no)  # {"type": "noul", "noul": p_yes}

asyncio.run(main())
```

`predict` prepares token IDs, marker positions, and the pooling mask before calling Core AI.
Each instance accepts one inference request at a time; await completion before making another call.
Unknown tasks, empty option sets, and inputs exceeding the token limit raise errors.

Run the standalone package checks:

```bash
USE_OS_COREAI=1 uv run --locked python smoke.py
```

This compares 12 cases with upstream reference probabilities, checks exact input-tensor equality against upstream encoding,
and verifies rejection of three invalid-request categories. It writes `package_smoke.json` and `swift_fixture.json` for the Swift example.

## Swift usage

Requires Xcode 27 / the macOS 27 SDK or newer. `verify_coreai.swift` is a complete, compilable example that
loads pretokenized inputs from `swift_fixture.json`, runs native `CoreAI` inference, and compares probabilities with Python.
**The Swift example demonstrates native model invocation; it does not include a Swift tokenizer or a complete SDK for arbitrary text inputs.**

```bash
# Run inside the model directory. Adjust DEVELOPER_DIR if Xcode is installed elsewhere.
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
  xcrun swiftc -parse-as-library -target arm64-apple-macos27.0 \
  verify_coreai.swift -o /tmp/verify-vela
/tmp/verify-vela "$PWD"
```

The core invocation is shown below; see `verify_coreai.swift` for complete tensor construction:

```swift
import CoreAI
import Foundation

let model = try await AIModel(
    contentsOf: modelDirectory.appendingPathComponent("model.aimodel"),
    options: .cpuOnly
)
guard let function = try model.loadFunction(named: "main") else {
    fatalError("Missing main function")
}
// inputs: [String: NDArray], following the dtypes, shapes, and encoding rules below.
var result = try await function.run(inputs: inputs)
guard let logits = result.remove("logits")?.ndArray else {
    fatalError("Missing logits")
}
// Apply stable softmax to the output logits and interpret them in input-option order.
```

For new text, reuse `tokenizer.json`, `decision.json`, and the encoding rules in `vela_coreai.py`.
Tokenizing only the input body does not produce valid model inputs. The bundled fixture is a synthetic example and contains no user data.

## Tensor interface

Entry point: `main`. No persistent recurrent/KV state. Sequence and option axes are dynamic; inputs use their actual encoded length without fixed padding.

| Name | Type | Shape | Meaning |
| --- | --- | --- | --- |
| `input_ids` | int32 | `[1,sequence]` | Complete question/options/body sequence |
| `attention_mask` | int32 | `[1,sequence]` | 1 for valid tokens, 0 for padding |
| `option_positions` | int32 | `[1,options]` | Positions of the `[O]` markers |
| `pool_mask` | float32 | `[1,sequence]` | 1 from `[SEG_user]` through the end of the body, excluding EOS |
| `logits` (output) | float32 | `[1,options]` | Unnormalized scores for the one or more ordered options |

Upstream sequence format:

```text
[BOS] [Q] instructions [O] key_1: description_1 [O] key_2: description_2
[SEP_SCHEMA] [SEG_user] state [EOS]
```

## Validation and limitations

- **12/12** text cases matched the original model's decisions and the expected answers.
- Maximum absolute probability error was approximately **7.80e-7**. Validation uses `atol=0.02, rtol=0.02` and also requires decision agreement.
- The standalone Python package passed all 12 cases without PyTorch installed, with exact input-tensor parity against upstream.
- Native Swift CPU results are in `swift_smoke.json`; standalone Python results are in `package_smoke.json`.
- These handcrafted examples are not a general accuracy benchmark. Coverage includes a JSON object, 1/2/3 options, and exact 129/1,024/2,048-token inputs.
- iPhone/iPad, iOS AOT, automatic device selection, GPU/ANE placement, lengths above 2,048 tokens, and other upstream tasks have not been validated.
- The new dynamic artifact passed initial CPU tensor validation and subsequent task evaluation. Tool versions are recorded in `manifest.json` and `environment.json`.
- The 32K bound is retained in the export and host adapter, but end-to-end 32K inference has not been validated on this 8 GB test machine. Longer inputs require additional memory; 2,048 is a tested length, not a new hard limit.
- This revision replaces the earlier fixed 128-token, two-option export. The adapters listed above remain outside the current decision graph.

Full cases and probabilities: `validation_cases.json` and `evaluation.json`.
`SHA256SUMS` covers the published files. Original weight copies, development caches, and virtual environments are not uploaded.

## Attribution and license

Upstream: [vllm-sr/vela3-fast](https://huggingface.co/vllm-sr/vela3-fast/tree/fa60e3aa41132c60bf05a84cabfdced39cb6d4f7),
pinned to commit `fa60e3aa41132c60bf05a84cabfdced39cb6d4f7`, under Apache-2.0.
The converted graph and input rendering adapt `vela3_encoder.py` and `ub_d3_common.py` from that revision.
See `NOTICE` for attribution and modification details, and `LICENSE` for the license.
This is a hugging-mac community conversion, not an official release from Apple or the upstream team.