| --- |
| display_name: NeuDecide Q4 - ExecuTorch |
| license: apache-2.0 |
| base_model: neuphonic/neudecide |
| base_model_relation: quantized |
| language: |
| - en |
| tags: |
| - audio-tool-calling |
| - spoken-language-understanding |
| - executorch |
| - xnnpack |
| - quantized |
| - on-device |
| pipeline_tag: audio-classification |
| --- |
| |
| # NeuDecide Q4 - ExecuTorch |
|
|
| This package contains a per-channel Q4 conversion of |
| [`neuphonic/neudecide`](https://huggingface.co/neuphonic/neudecide) for English spoken-request |
| tool calling with ExecuTorch and XNNPACK. It generates JSON tool calls with arguments directly |
| from audio, without an intermediate transcript. |
|
|
| ## ✨ Key Highlights |
|
|
| - **Incredibly fast tool calling from audio** — on one Arm® CPU inference thread with structured |
| tool-call output: |
| - Median completed request: **77.6** ms on a MacBook Pro® computer with an M4 Pro chip and |
| **164.4** ms on an Android™ Vivo X300 smartphone. |
| - RTFx: **21.63×** on the MacBook Pro® computer and **10.21×** on the Vivo X300 smartphone. |
| - **Tiny model for tool calling** — the four ExecuTorch model files total **99.14** MiB. |
| |
| ## 📦 Model Details |
|
|
| ### Model Description |
|
|
| NeuDecide generates calls to configured assistant tools from an English spoken request. The package |
| provides ExecuTorch `.pte` model files, a SentencePiece tokenizer, and a JSON file with eight |
| assistant tool definitions, including the upstream example's `cleanRoom` action. |
|
|
| - **Developed by:** Neuphonic. |
| - **Model type:** English spoken-language understanding and audio tool calling. |
| - **Base model:** [`neuphonic/neudecide`](https://huggingface.co/neuphonic/neudecide), revision |
| `892237127c492b51d97a1824c459854bcd6a4519`. |
| - **License:** Apache-2.0; see [LICENSE](LICENSE). |
| - **Package format:** ExecuTorch `.pte` files and a SentencePiece tokenizer. |
| - **Runtime model size:** 99.14 MiB across four `.pte` files. |
|
|
| ### Model Sources |
|
|
| - [Upstream model](https://huggingface.co/neuphonic/neudecide/tree/892237127c492b51d97a1824c459854bcd6a4519). |
| - [Upstream implementation](https://github.com/neuphonic/neudecide/tree/548134f512f11dfe249acde3110c05c3a4774871). |
|
|
| ## 🚀 Get Started with the Model |
|
|
| ### Run with Compute Flow |
|
|
| Follow these steps to run NeuDecide Q4 - ExecuTorch on macOS® with Apple Silicon or on Linux® with |
| an Arm®-based CPU, such as a Raspberry Pi® 5. |
|
|
| #### Step 1: Clone Compute Flow and set up the virtual environment |
|
|
| ```bash |
| git clone https://github.com/ARM-software/ComputeFlow.git && cd ComputeFlow |
| |
| uv sync && source .venv/bin/activate |
| ``` |
|
|
| #### Step 2: Install the brick and model |
|
|
| ```bash |
| compute-flow update |
| compute-flow model install audio_tool_calling:neu-decide neudecide-en |
| compute-flow build native |
| ``` |
|
|
| #### Step 3: Run the Python CLI example |
|
|
| ```bash |
| python -m cli_examples.audio_tool_calling_cli \ |
| -m @manifest/neu-decide/neudecide-en/neudecide_manifest.json \ |
| -a @sample/command.wav \ |
| -t 1 |
| ``` |
|
|
| The example extracts a tool call from the bundled `samples/command.wav` recording. Replace `-a` |
| with a path to your own WAV file. |
|
|
| ## 📊 Quality evaluation |
|
|
| This package applies per-channel Q4 quantization to the released groupwise Q4 model. We compared |
| its tool selections with the [official Q4 implementation](https://github.com/neuphonic/neudecide) |
| on spoken requests using the same audio and tool definitions. In these limited checks, we did not |
| observe a substantial difference in overall tool calling. These checks do not establish general |
| quality or evaluate tool arguments. We recommend evaluating the model on representative audio and |
| tool definitions for the intended application. |
|
|
| ## 🎯 Performance evaluation |
|
|
| The model artifacts were measured with structured tool-call output on a MacBook Pro® computer with |
| an M4 Pro chip running macOS® 26.6.2 and a Vivo X300 smartphone running Android™ 16 under the |
| following conditions: |
|
|
| - **One** Arm® CPU inference thread. |
| - **5** warmup requests. |
| - **30** measured requests with a **60-second** pause between runs. |
| - The MacBook Pro® computer was AC powered and kept awake; the Vivo X300 smartphone was USB powered |
| and its screen was kept on. |
|
|
| The following methodology and definitions were used: |
|
|
| - **Input audio:** The packaged [upstream `command.wav` sample](samples/command.wav), a 1.679-second |
| English vacuum-cleaning request at 16 kHz, with eight tool definitions and |
| a 128-token output limit. The completed result was `cleanRoom` in every measured run on both |
| devices. |
| - **End-to-end latency:** The summed execution time through a completed request. It |
| includes preprocessing, tool tokenization, model execution, constrained decoding, and result |
| collection; it excludes model setup, WAV file loading, and state reset. |
| - **RTFx:** Input audio duration divided by median latency; higher values are faster. |
| - **Average memory:** Mean process RSS sampled every 0.2 seconds across setup and inference. |
|
|
| | Metric | MacBook Pro® (M4 Pro) | Android™ Vivo X300 | |
| | --- | ---: | ---: | |
| | RTFx | 21.63× | 10.21× | |
| | End-to-end latency | 77.6 ms | 164.4 ms | |
| | Average memory | 347.9 MiB | 419.7 MiB | |
|
|
| The reported values describe this fixed utterance and tool list; other audio and tool schemas can |
| change the execution time and output. |
|
|
| ## ⚠️ Known Limitations |
|
|
| The model can produce a tool call for silence or noise. Applications should check inputs and |
| generated arguments before executing tools. |
|
|
| ## 🛠️ Technical Specifications |
|
|
| ### Objective |
|
|
| Generate structured calls to configured assistant tools from an English spoken request. |
|
|
| ### Runtime Architecture |
|
|
| Inference uses ExecuTorch with XNNPACK and SentencePiece tokenization. |
|
|
| ### Precision and Quantization |
|
|
| - Linear weights use signed per-channel Q4 reconstructed from the released groupwise Q4 weights; |
| linear activations use dynamic INT8 quantization with FP32 channel scales. |
| - Embeddings preserve the source packed Q4 codes and FP32 group scales. |
| - Source Q8 convolution weights are dequantized to FP32. Remaining floating-point operations and |
| public audio inputs use FP32. |
|
|
| ### Input Specification |
|
|
| | Input | Format | Limits | |
| | --- | --- | --- | |
| | Spoken request | Mono English audio at 16 kHz | The leading 30 seconds are processed; resample other rates to 16 kHz. | |
|
|
| ### Output Specification |
|
|
| | Output | Format | Meaning | |
| | --- | --- | --- | |
| | `calls` | One UTF-8 JSON string | An array of tool calls in generation order. Each call has a `name` string and an `arguments` object. | |
|
|
| An empty completed response produces `[]`. If decoding stops before the full array is complete, |
| the brick returns `[]` without exposing partial calls. |
|
|
| ### Manifest Configuration |
|
|
| The manifest's `max_output_tokens` setting defaults to `128` and accepts values from `2` to `128`. |
| A lower limit can truncate a response and produce `[]`. |
|
|
| [`assets/tools.json`](assets/tools.json) defines the available tools. Changing this file requires |
| reinitializing inference. Tool names must be unique and cannot be `no_tool`. The complete tool |
| definitions must fit within 1,536 tokens. |
|
|
| ### Repository Contents |
|
|
| - `*.pte` — four ExecuTorch model artifacts. |
| - `tokenizer.model` — upstream SentencePiece tokenizer. |
| - `neudecide_manifest.json` — model and runtime asset configuration. |
| - `assets/tools.json` and `assets/token_bytes.json` — runtime assets. |
| - `samples/command.wav` — [upstream vacuum-cleaning request](https://github.com/neuphonic/neudecide/blob/548134f512f11dfe249acde3110c05c3a4774871/examples/command.wav); |
| with the packaged tools, its expected tool name is `cleanRoom`. |
| - `metadata.yaml` — package metadata. |
| - `SHA256SUMS` — model-bundle checksums for reproducibility. |
|
|
| ## 🗂️ Model and Asset Origin |
|
|
| - **Model files:** Converted from [Neuphonic's Apache-2.0 Q4 release](https://huggingface.co/neuphonic/neudecide). |
| - **Tokenizer:** From the pinned [upstream model revision](https://huggingface.co/neuphonic/neudecide/tree/892237127c492b51d97a1824c459854bcd6a4519). |
| - **Sample audio:** [`command.wav`](https://github.com/neuphonic/neudecide/blob/548134f512f11dfe249acde3110c05c3a4774871/examples/command.wav) |
| from the upstream implementation, distributed as `samples/command.wav`. |
| - **Tool definitions:** Prepared for this package; `cleanRoom` follows the pinned |
| [upstream example](https://github.com/neuphonic/neudecide/blob/548134f512f11dfe249acde3110c05c3a4774871/examples/tools.json). |
|
|
| ## 🔐 Checksums |
|
|
| `SHA256SUMS` was generated by recursively hashing every regular file in the |
| model bundle, including files in subdirectories, except the generated root |
| `SHA256SUMS` and paths with a dotfile component. |
|
|
| From the model bundle root, verify the checked-out files with: |
|
|
| ```sh |
| shasum -a 256 -c SHA256SUMS |
| ``` |
|
|
