vela3-fast-CoreAI / README.md
devilyouwei's picture
Replace fixed graph with dynamic sequence and option dimensions
6aaa9cc verified
|
Raw History Blame Contribute Delete
8.73 kB
metadata
language:
  - en
  - zh
license: apache-2.0
base_model: vllm-sr/vela3-fast
pipeline_tag: zero-shot-classification
tags:
  - coreai
  - apple-silicon
  - aimodel
  - experimental
  - decision-model

Vela3-fast Core AI — experimental decision graph

English | 简体中文

An Apple Core AI conversion of vllm-sr/vela3-fast by hugging-mac. This release contains the validated choice / noul decision backbone: one sample per graph call, dynamic sequence length, and a dynamic number of options. The export retains the upstream 32,768-token input bound; actual CPU parity tests currently cover lengths through 2,048 tokens, including exactly 1,024. It is an experimental subset, not the complete Vela3-fast model with all task adapters.

Neural inference runs through Core AI, with host-side tokenization and response formatting. The model does not generate text.

Supported contract

  • choice: one or more ordered options; returns their probabilities and the selected option.
  • noul: fixed no / yes options; returns p(yes).
  • The input text, question, option descriptions, and special markers together must fit within the upstream 32,768-token bound. Longer inputs raise an error; they are not truncated.
  • Each graph call has batch size 1. predict_questions(state, questions) accepts a mapping of questions and returns (answers, input_tokens), matching the upstream decision API; questions execute sequentially. The adapter preserves upstream JSON serialization and accepts mapping or list choice criteria.
  • CPU execution is validated. Earlier automatic-device validation did not finish because of resource pressure on the test host; no GPU/ANE performance is claimed.
  • Original FP32 decision weights, with no quantization, training, or fine-tuning.
  • Router, span, relation, NER, evidence, domain/factcheck/feedback, and other upstream adapters are not included. Full system_one adapter routing is not implemented by this decision-only artifact. Untested capabilities are not known to be unconvertible.
  • This release does not provide complete hugging_mac_sdk registration or production acceptance. Probabilities are relative to the supplied options, not calibrated confidence scores.

Python usage

Validated on Apple Silicon, macOS 27.2.1, Python 3.12.13, and coreai-core==1.0.0b3. A macOS version supporting the Core AI system runtime is required. The example environment uses uv and does not require PyTorch, transformers, or the original safetensors weights.

uvx --from huggingface-hub hf download hugging-mac/vela3-fast-CoreAI --local-dir vela3-fast-CoreAI
cd vela3-fast-CoreAI
uv sync --locked
USE_OS_COREAI=1 uv run --locked python example.py
import asyncio
from vela_coreai import VelaCoreAI

async def main():
    # Point to the downloaded repository; the adapter explicitly uses CPU mode.
    async with VelaCoreAI.open(".") as model:
        result = await model.predict(
            "Write a Python sorting function.",
            {
                "type": "choice",
                "instructions": "What is the topic?",
                "criteria": {"coding": "Software", "cooking": "Food"},
            },
        )
        print(result)  # type, choice, probabilities

        yes_no = await model.predict(
            "Python code",
            {"type": "noul", "instructions": "Is this about software?"},
        )
        print(yes_no)  # {"type": "noul", "noul": p_yes}

asyncio.run(main())

predict prepares token IDs, marker positions, and the pooling mask before calling Core AI. Each instance accepts one inference request at a time; await completion before making another call. Unknown tasks, empty option sets, and inputs exceeding the token limit raise errors.

Run the standalone package checks:

USE_OS_COREAI=1 uv run --locked python smoke.py

This compares 12 cases with upstream reference probabilities, checks exact input-tensor equality against upstream encoding, and verifies rejection of three invalid-request categories. It writes package_smoke.json and swift_fixture.json for the Swift example.

Swift usage

Requires Xcode 27 / the macOS 27 SDK or newer. verify_coreai.swift is a complete, compilable example that loads pretokenized inputs from swift_fixture.json, runs native CoreAI inference, and compares probabilities with Python. The Swift example demonstrates native model invocation; it does not include a Swift tokenizer or a complete SDK for arbitrary text inputs.

# Run inside the model directory. Adjust DEVELOPER_DIR if Xcode is installed elsewhere.
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
  xcrun swiftc -parse-as-library -target arm64-apple-macos27.0 \
  verify_coreai.swift -o /tmp/verify-vela
/tmp/verify-vela "$PWD"

The core invocation is shown below; see verify_coreai.swift for complete tensor construction:

import CoreAI
import Foundation

let model = try await AIModel(
    contentsOf: modelDirectory.appendingPathComponent("model.aimodel"),
    options: .cpuOnly
)
guard let function = try model.loadFunction(named: "main") else {
    fatalError("Missing main function")
}
// inputs: [String: NDArray], following the dtypes, shapes, and encoding rules below.
var result = try await function.run(inputs: inputs)
guard let logits = result.remove("logits")?.ndArray else {
    fatalError("Missing logits")
}
// Apply stable softmax to the output logits and interpret them in input-option order.

For new text, reuse tokenizer.json, decision.json, and the encoding rules in vela_coreai.py. Tokenizing only the input body does not produce valid model inputs. The bundled fixture is a synthetic example and contains no user data.

Tensor interface

Entry point: main. No persistent recurrent/KV state. Sequence and option axes are dynamic; inputs use their actual encoded length without fixed padding.

Name Type Shape Meaning
input_ids int32 [1,sequence] Complete question/options/body sequence
attention_mask int32 [1,sequence] 1 for valid tokens, 0 for padding
option_positions int32 [1,options] Positions of the [O] markers
pool_mask float32 [1,sequence] 1 from [SEG_user] through the end of the body, excluding EOS
logits (output) float32 [1,options] Unnormalized scores for the one or more ordered options

Upstream sequence format:

[BOS] [Q] instructions [O] key_1: description_1 [O] key_2: description_2
[SEP_SCHEMA] [SEG_user] state [EOS]

Validation and limitations

  • 12/12 text cases matched the original model's decisions and the expected answers.
  • Maximum absolute probability error was approximately 7.80e-7. Validation uses atol=0.02, rtol=0.02 and also requires decision agreement.
  • The standalone Python package passed all 12 cases without PyTorch installed, with exact input-tensor parity against upstream.
  • Native Swift CPU results are in swift_smoke.json; standalone Python results are in package_smoke.json.
  • These handcrafted examples are not a general accuracy benchmark. Coverage includes a JSON object, 1/2/3 options, and exact 129/1,024/2,048-token inputs.
  • iPhone/iPad, iOS AOT, automatic device selection, GPU/ANE placement, lengths above 2,048 tokens, and other upstream tasks have not been validated.
  • The new dynamic artifact passed initial CPU tensor validation and subsequent task evaluation. Tool versions are recorded in manifest.json and environment.json.
  • The 32K bound is retained in the export and host adapter, but end-to-end 32K inference has not been validated on this 8 GB test machine. Longer inputs require additional memory; 2,048 is a tested length, not a new hard limit.
  • This revision replaces the earlier fixed 128-token, two-option export. The adapters listed above remain outside the current decision graph.

Full cases and probabilities: validation_cases.json and evaluation.json. SHA256SUMS covers the published files. Original weight copies, development caches, and virtual environments are not uploaded.

Attribution and license

Upstream: vllm-sr/vela3-fast, pinned to commit fa60e3aa41132c60bf05a84cabfdced39cb6d4f7, under Apache-2.0. The converted graph and input rendering adapt vela3_encoder.py and ub_d3_common.py from that revision. See NOTICE for attribution and modification details, and LICENSE for the license. This is a hugging-mac community conversion, not an official release from Apple or the upstream team.