vela3-fast-CoreAI / README.md
devilyouwei's picture
Replace fixed graph with dynamic sequence and option dimensions
6aaa9cc verified
|
Raw History Blame Contribute Delete
8.73 kB
---
language:
- en
- zh
license: apache-2.0
base_model: vllm-sr/vela3-fast
pipeline_tag: zero-shot-classification
tags:
- coreai
- apple-silicon
- aimodel
- experimental
- decision-model
---
# Vela3-fast Core AI — experimental decision graph
English | [简体中文](README.zh-CN.md)
An Apple Core AI conversion of [vllm-sr/vela3-fast](https://huggingface.co/vllm-sr/vela3-fast) by hugging-mac.
This release contains the validated **choice / noul decision backbone**: one sample per graph call, dynamic sequence length, and a dynamic number of options.
The export retains the upstream **32,768-token** input bound; actual CPU parity tests currently cover lengths through **2,048 tokens**, including exactly 1,024.
**It is an experimental subset, not the complete Vela3-fast model with all task adapters.**
Neural inference runs through Core AI, with host-side tokenization and response formatting. The model does not generate text.
## Supported contract
- `choice`: one or more ordered options; returns their probabilities and the selected option.
- `noul`: fixed `no` / `yes` options; returns `p(yes)`.
- The input text, question, option descriptions, and special markers together must fit within the upstream **32,768-token** bound. Longer inputs raise an error; they are not truncated.
- Each graph call has batch size 1. `predict_questions(state, questions)` accepts a mapping of questions and returns `(answers, input_tokens)`, matching the upstream decision API; questions execute sequentially. The adapter preserves upstream JSON serialization and accepts mapping or list choice criteria.
- CPU execution is validated. Earlier automatic-device validation did not finish because of resource pressure on the test host; no GPU/ANE performance is claimed.
- Original FP32 decision weights, with no quantization, training, or fine-tuning.
- Router, span, relation, NER, evidence, domain/factcheck/feedback, and other upstream adapters are not included. Full `system_one` adapter routing is not implemented by this decision-only artifact. Untested capabilities are not known to be unconvertible.
- This release does not provide complete `hugging_mac_sdk` registration or production acceptance. Probabilities are relative to the supplied options, not calibrated confidence scores.
## Python usage
Validated on Apple Silicon, macOS 27.2.1, Python 3.12.13, and `coreai-core==1.0.0b3`.
A macOS version supporting the Core AI system runtime is required. The example environment uses uv and **does not require PyTorch, transformers, or the original safetensors weights**.
```bash
uvx --from huggingface-hub hf download hugging-mac/vela3-fast-CoreAI --local-dir vela3-fast-CoreAI
cd vela3-fast-CoreAI
uv sync --locked
USE_OS_COREAI=1 uv run --locked python example.py
```
```python
import asyncio
from vela_coreai import VelaCoreAI
async def main():
# Point to the downloaded repository; the adapter explicitly uses CPU mode.
async with VelaCoreAI.open(".") as model:
result = await model.predict(
"Write a Python sorting function.",
{
"type": "choice",
"instructions": "What is the topic?",
"criteria": {"coding": "Software", "cooking": "Food"},
},
)
print(result) # type, choice, probabilities
yes_no = await model.predict(
"Python code",
{"type": "noul", "instructions": "Is this about software?"},
)
print(yes_no) # {"type": "noul", "noul": p_yes}
asyncio.run(main())
```
`predict` prepares token IDs, marker positions, and the pooling mask before calling Core AI.
Each instance accepts one inference request at a time; await completion before making another call.
Unknown tasks, empty option sets, and inputs exceeding the token limit raise errors.
Run the standalone package checks:
```bash
USE_OS_COREAI=1 uv run --locked python smoke.py
```
This compares 12 cases with upstream reference probabilities, checks exact input-tensor equality against upstream encoding,
and verifies rejection of three invalid-request categories. It writes `package_smoke.json` and `swift_fixture.json` for the Swift example.
## Swift usage
Requires Xcode 27 / the macOS 27 SDK or newer. `verify_coreai.swift` is a complete, compilable example that
loads pretokenized inputs from `swift_fixture.json`, runs native `CoreAI` inference, and compares probabilities with Python.
**The Swift example demonstrates native model invocation; it does not include a Swift tokenizer or a complete SDK for arbitrary text inputs.**
```bash
# Run inside the model directory. Adjust DEVELOPER_DIR if Xcode is installed elsewhere.
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
xcrun swiftc -parse-as-library -target arm64-apple-macos27.0 \
verify_coreai.swift -o /tmp/verify-vela
/tmp/verify-vela "$PWD"
```
The core invocation is shown below; see `verify_coreai.swift` for complete tensor construction:
```swift
import CoreAI
import Foundation
let model = try await AIModel(
contentsOf: modelDirectory.appendingPathComponent("model.aimodel"),
options: .cpuOnly
)
guard let function = try model.loadFunction(named: "main") else {
fatalError("Missing main function")
}
// inputs: [String: NDArray], following the dtypes, shapes, and encoding rules below.
var result = try await function.run(inputs: inputs)
guard let logits = result.remove("logits")?.ndArray else {
fatalError("Missing logits")
}
// Apply stable softmax to the output logits and interpret them in input-option order.
```
For new text, reuse `tokenizer.json`, `decision.json`, and the encoding rules in `vela_coreai.py`.
Tokenizing only the input body does not produce valid model inputs. The bundled fixture is a synthetic example and contains no user data.
## Tensor interface
Entry point: `main`. No persistent recurrent/KV state. Sequence and option axes are dynamic; inputs use their actual encoded length without fixed padding.
| Name | Type | Shape | Meaning |
| --- | --- | --- | --- |
| `input_ids` | int32 | `[1,sequence]` | Complete question/options/body sequence |
| `attention_mask` | int32 | `[1,sequence]` | 1 for valid tokens, 0 for padding |
| `option_positions` | int32 | `[1,options]` | Positions of the `[O]` markers |
| `pool_mask` | float32 | `[1,sequence]` | 1 from `[SEG_user]` through the end of the body, excluding EOS |
| `logits` (output) | float32 | `[1,options]` | Unnormalized scores for the one or more ordered options |
Upstream sequence format:
```text
[BOS] [Q] instructions [O] key_1: description_1 [O] key_2: description_2
[SEP_SCHEMA] [SEG_user] state [EOS]
```
## Validation and limitations
- **12/12** text cases matched the original model's decisions and the expected answers.
- Maximum absolute probability error was approximately **7.80e-7**. Validation uses `atol=0.02, rtol=0.02` and also requires decision agreement.
- The standalone Python package passed all 12 cases without PyTorch installed, with exact input-tensor parity against upstream.
- Native Swift CPU results are in `swift_smoke.json`; standalone Python results are in `package_smoke.json`.
- These handcrafted examples are not a general accuracy benchmark. Coverage includes a JSON object, 1/2/3 options, and exact 129/1,024/2,048-token inputs.
- iPhone/iPad, iOS AOT, automatic device selection, GPU/ANE placement, lengths above 2,048 tokens, and other upstream tasks have not been validated.
- The new dynamic artifact passed initial CPU tensor validation and subsequent task evaluation. Tool versions are recorded in `manifest.json` and `environment.json`.
- The 32K bound is retained in the export and host adapter, but end-to-end 32K inference has not been validated on this 8 GB test machine. Longer inputs require additional memory; 2,048 is a tested length, not a new hard limit.
- This revision replaces the earlier fixed 128-token, two-option export. The adapters listed above remain outside the current decision graph.
Full cases and probabilities: `validation_cases.json` and `evaluation.json`.
`SHA256SUMS` covers the published files. Original weight copies, development caches, and virtual environments are not uploaded.
## Attribution and license
Upstream: [vllm-sr/vela3-fast](https://huggingface.co/vllm-sr/vela3-fast/tree/fa60e3aa41132c60bf05a84cabfdced39cb6d4f7),
pinned to commit `fa60e3aa41132c60bf05a84cabfdced39cb6d4f7`, under Apache-2.0.
The converted graph and input rendering adapt `vela3_encoder.py` and `ub_d3_common.py` from that revision.
See `NOTICE` for attribution and modification details, and `LICENSE` for the license.
This is a hugging-mac community conversion, not an official release from Apple or the upstream team.