README.md

---
display_name: NeuDecide Q4 - ExecuTorch
license: apache-2.0
base_model: neuphonic/neudecide
base_model_relation: quantized
language:
  - en
tags:
  - audio-tool-calling
  - spoken-language-understanding
  - executorch
  - xnnpack
  - quantized
  - on-device
pipeline_tag: audio-classification
---

# NeuDecide Q4 - ExecuTorch

This package contains a per-channel Q4 conversion of
[`neuphonic/neudecide`](https://huggingface.co/neuphonic/neudecide) for English spoken-request
tool calling with ExecuTorch and XNNPACK. It generates JSON tool calls with arguments directly
from audio, without an intermediate transcript.

## ✨ Key Highlights

- **Incredibly fast tool calling from audio** — on one Arm® CPU inference thread with structured
  tool-call output:
  - Median completed request: **77.6** ms on a MacBook Pro® computer with an M4 Pro chip and
    **164.4** ms on an Android™ Vivo X300 smartphone.
  - RTFx: **21.63×** on the MacBook Pro® computer and **10.21×** on the Vivo X300 smartphone.
- **Tiny model for tool calling** — the four ExecuTorch model files total **99.14** MiB.

## 📦 Model Details

### Model Description

NeuDecide generates calls to configured assistant tools from an English spoken request. The package
provides ExecuTorch `.pte` model files, a SentencePiece tokenizer, and a JSON file with eight
assistant tool definitions, including the upstream example's `cleanRoom` action.

- **Developed by:** Neuphonic.
- **Model type:** English spoken-language understanding and audio tool calling.
- **Base model:** [`neuphonic/neudecide`](https://huggingface.co/neuphonic/neudecide), revision
  `892237127c492b51d97a1824c459854bcd6a4519`.
- **License:** Apache-2.0; see [LICENSE](LICENSE).
- **Package format:** ExecuTorch `.pte` files and a SentencePiece tokenizer.
- **Runtime model size:** 99.14 MiB across four `.pte` files.

### Model Sources

- [Upstream model](https://huggingface.co/neuphonic/neudecide/tree/892237127c492b51d97a1824c459854bcd6a4519).
- [Upstream implementation](https://github.com/neuphonic/neudecide/tree/548134f512f11dfe249acde3110c05c3a4774871).

## 🚀 Get Started with the Model

### Run with Compute Flow

Follow these steps to run NeuDecide Q4 - ExecuTorch on macOS® with Apple Silicon or on Linux® with
an Arm®-based CPU, such as a Raspberry Pi® 5.

#### Step 1: Clone Compute Flow and set up the virtual environment

```bash
git clone https://github.com/ARM-software/ComputeFlow.git && cd ComputeFlow

uv sync && source .venv/bin/activate
```

#### Step 2: Install the brick and model

```bash
compute-flow update
compute-flow model install audio_tool_calling:neu-decide neudecide-en
compute-flow build native
```

#### Step 3: Run the Python CLI example

```bash
python -m cli_examples.audio_tool_calling_cli \
  -m @manifest/neu-decide/neudecide-en/neudecide_manifest.json \
  -a @sample/command.wav \
  -t 1
```

The example extracts a tool call from the bundled `samples/command.wav` recording. Replace `-a`
with a path to your own WAV file.

## 📊 Quality evaluation

This package applies per-channel Q4 quantization to the released groupwise Q4 model. We compared
its tool selections with the [official Q4 implementation](https://github.com/neuphonic/neudecide)
on spoken requests using the same audio and tool definitions. In these limited checks, we did not
observe a substantial difference in overall tool calling. These checks do not establish general
quality or evaluate tool arguments. We recommend evaluating the model on representative audio and
tool definitions for the intended application.

## 🎯 Performance evaluation

The model artifacts were measured with structured tool-call output on a MacBook Pro® computer with
an M4 Pro chip running macOS® 26.6.2 and a Vivo X300 smartphone running Android™ 16 under the
following conditions:

- **One** Arm® CPU inference thread.
- **5** warmup requests.
- **30** measured requests with a **60-second** pause between runs.
- The MacBook Pro® computer was AC powered and kept awake; the Vivo X300 smartphone was USB powered
  and its screen was kept on.

The following methodology and definitions were used:

- **Input audio:** The packaged [upstream `command.wav` sample](samples/command.wav), a 1.679-second
  English vacuum-cleaning request at 16 kHz, with eight tool definitions and
  a 128-token output limit. The completed result was `cleanRoom` in every measured run on both
  devices.
- **End-to-end latency:** The summed execution time through a completed request. It
  includes preprocessing, tool tokenization, model execution, constrained decoding, and result
  collection; it excludes model setup, WAV file loading, and state reset.
- **RTFx:** Input audio duration divided by median latency; higher values are faster.
- **Average memory:** Mean process RSS sampled every 0.2 seconds across setup and inference.

| Metric | MacBook Pro® (M4 Pro) | Android™ Vivo X300 |
| --- | ---: | ---: |
| RTFx | 21.63× | 10.21× |
| End-to-end latency | 77.6 ms | 164.4 ms |
| Average memory | 347.9 MiB | 419.7 MiB |

The reported values describe this fixed utterance and tool list; other audio and tool schemas can
change the execution time and output.

## ⚠️ Known Limitations

The model can produce a tool call for silence or noise. Applications should check inputs and
generated arguments before executing tools.

## 🛠️ Technical Specifications

### Objective

Generate structured calls to configured assistant tools from an English spoken request.

### Runtime Architecture

Inference uses ExecuTorch with XNNPACK and SentencePiece tokenization.

### Precision and Quantization

- Linear weights use signed per-channel Q4 reconstructed from the released groupwise Q4 weights;
  linear activations use dynamic INT8 quantization with FP32 channel scales.
- Embeddings preserve the source packed Q4 codes and FP32 group scales.
- Source Q8 convolution weights are dequantized to FP32. Remaining floating-point operations and
  public audio inputs use FP32.

### Input Specification

| Input | Format | Limits |
| --- | --- | --- |
| Spoken request | Mono English audio at 16 kHz | The leading 30 seconds are processed; resample other rates to 16 kHz. |

### Output Specification

| Output | Format | Meaning |
| --- | --- | --- |
| `calls` | One UTF-8 JSON string | An array of tool calls in generation order. Each call has a `name` string and an `arguments` object. |

An empty completed response produces `[]`. If decoding stops before the full array is complete,
the brick returns `[]` without exposing partial calls.

### Manifest Configuration

The manifest's `max_output_tokens` setting defaults to `128` and accepts values from `2` to `128`.
A lower limit can truncate a response and produce `[]`.

[`assets/tools.json`](assets/tools.json) defines the available tools. Changing this file requires
reinitializing inference. Tool names must be unique and cannot be `no_tool`. The complete tool
definitions must fit within 1,536 tokens.

### Repository Contents

- `*.pte` — four ExecuTorch model artifacts.
- `tokenizer.model` — upstream SentencePiece tokenizer.
- `neudecide_manifest.json` — model and runtime asset configuration.
- `assets/tools.json` and `assets/token_bytes.json` — runtime assets.
- `samples/command.wav` — [upstream vacuum-cleaning request](https://github.com/neuphonic/neudecide/blob/548134f512f11dfe249acde3110c05c3a4774871/examples/command.wav);
  with the packaged tools, its expected tool name is `cleanRoom`.
- `metadata.yaml` — package metadata.
- `SHA256SUMS` — model-bundle checksums for reproducibility.

## 🗂️ Model and Asset Origin

- **Model files:** Converted from [Neuphonic's Apache-2.0 Q4 release](https://huggingface.co/neuphonic/neudecide).
- **Tokenizer:** From the pinned [upstream model revision](https://huggingface.co/neuphonic/neudecide/tree/892237127c492b51d97a1824c459854bcd6a4519).
- **Sample audio:** [`command.wav`](https://github.com/neuphonic/neudecide/blob/548134f512f11dfe249acde3110c05c3a4774871/examples/command.wav)
  from the upstream implementation, distributed as `samples/command.wav`.
- **Tool definitions:** Prepared for this package; `cleanRoom` follows the pinned
  [upstream example](https://github.com/neuphonic/neudecide/blob/548134f512f11dfe249acde3110c05c3a4774871/examples/tools.json).

## 🔐 Checksums

`SHA256SUMS` was generated by recursively hashing every regular file in the
model bundle, including files in subdirectories, except the generated root
`SHA256SUMS` and paths with a dotfile component.

From the model bundle root, verify the checked-out files with:

```sh
shasum -a 256 -c SHA256SUMS
```