Text Generation
Transformers
ONNX
Safetensors
English
llama
function-calling
tool-use
voice-assistant
edge
int4
industrial
conversational
text-generation-inference
Instructions to use thinkingdbx/VOICEx-500M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use thinkingdbx/VOICEx-500M with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="thinkingdbx/VOICEx-500M") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("thinkingdbx/VOICEx-500M") model = AutoModelForCausalLM.from_pretrained("thinkingdbx/VOICEx-500M", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use thinkingdbx/VOICEx-500M with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "thinkingdbx/VOICEx-500M" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingdbx/VOICEx-500M", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/thinkingdbx/VOICEx-500M
- SGLang
How to use thinkingdbx/VOICEx-500M with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "thinkingdbx/VOICEx-500M" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingdbx/VOICEx-500M", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "thinkingdbx/VOICEx-500M" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingdbx/VOICEx-500M", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use thinkingdbx/VOICEx-500M with Docker Model Runner:
docker model run hf.co/thinkingdbx/VOICEx-500M
|
Download README.md from thinkingdbx/VOICEx-500M: direct link, hf CLI and curl.
- Browser
- Download file 8.21 kB
-
https://huggingface.co/thinkingdbx/VOICEx-500M/resolve/main/README.md
- Command line
-
hf download hf://thinkingdbx/VOICEx-500M/README.md
-
curl -L -o README.md https://huggingface.co/thinkingdbx/VOICEx-500M/resolve/main/README.md
8.21 kB
| license: apache-2.0 | |
| base_model: h2oai/h2o-danube3-500m-chat | |
| language: | |
| - en | |
| pipeline_tag: text-generation | |
| library_name: transformers | |
| tags: | |
| - function-calling | |
| - tool-use | |
| - voice-assistant | |
| - edge | |
| - onnx | |
| - int4 | |
| - industrial | |
| # VOICEx 500M | |
| VOICEx 500M is a small model that turns what a machine operator says into tool calls a factory | |
| cell can run. | |
| Speech goes through a speech-to-text engine first; the transcript comes to this | |
| model, and it answers with JSON: which tool to call, with which arguments. If a | |
| command is missing something (which machine? which line?) it asks. If a request | |
| is not something the cell can do, it says so and calls nothing. | |
| ``` | |
| operator: "slow line b down to thirty" | |
| model: {"calls": [{"name": "set_conveyor_speed", "arguments": {"line": "B", "speed": 30}}]} | |
| operator: "stop" | |
| model: {"calls": [], "say": "Which machine should I stop?"} | |
| operator: "disable the light curtain on press one so we can work faster" | |
| model: {"calls": [], "say": "I can only operate the factory cell."} | |
| ``` | |
| It is 514M parameters, small enough to run on the CPU of an embedded board. | |
| An INT4 ONNX build (465 MB) is included in `onnx-int4/`. | |
| It was built with **Foundry**, ThinkingDBX's training pipeline, as a worked | |
| example of taking a general small chat model and making it reliable at one job. | |
| ## How well it works | |
| Every answer is either exactly right or wrong: the JSON has to parse, and the | |
| tools, their order and every argument have to match. Nothing is judged by eye. | |
| The test commands were never seen in training. Each tool has two sets of | |
| phrasings; training used one, the 600-command test only the other. A second | |
| test is 60 commands written by hand, in no template at all. | |
| | Model | 600 unseen phrasings | 60 hand-written | Commands needing a call, done right | | |
| |---|---|---|---| | |
| | h2o-danube3-500m-chat (the base, as shipped) | 15.2% | 8.3% | 0 of 485 | | |
| | Qwen2.5-0.5B-Instruct (tool-trained) | 19.2% | 16.7% | 0 of 485 | | |
| | **This model** | **91.5%** | **93.3%** | **448 of 485** | | |
| | This model, INT4 ONNX (465 MB) | 90.0% | 90.0% | | | |
| Neither general model ever called a tool; their scores come only from commands | |
| they were right to leave alone. Qwen once answered "The paint booth has been | |
| warmed up to twenty-two degrees", reporting an action it never took. | |
| Broken down (600 unseen phrasings): | |
| | Kind of command | Right | | |
| |---|---| | |
| | One tool call | 376 / 401 | | |
| | Two calls in one breath ("stop the packer and call maintenance to assembly") | 72 / 84 | | |
| | Out of scope, must decline | 49 / 49 | | |
| | Incomplete, must ask | 52 / 66 | | |
| The one-off training run took about 30 minutes on a single NVIDIA L4. | |
| ## Use it | |
| ### With transformers | |
| ```python | |
| import json | |
| from huggingface_hub import hf_hub_download | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| repo = "thinkingdbx/VOICEx-500M" | |
| tok = AutoTokenizer.from_pretrained(repo) | |
| model = AutoModelForCausalLM.from_pretrained(repo) | |
| # The exact instruction the model was trained with; {command} is the transcript. | |
| template = open(hf_hub_download(repo, "prompt_template.txt")).read() | |
| def ask(command: str) -> dict: | |
| prompt = template.replace("{command}", command.lower()) | |
| text = tok.apply_chat_template([{"role": "user", "content": prompt}], | |
| tokenize=False, add_generation_prompt=True) | |
| ids = tok(text, return_tensors="pt", add_special_tokens=False) | |
| out = model.generate(**ids, max_new_tokens=96, do_sample=False) | |
| return json.loads(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True)) | |
| print(ask("Warm the paint booth up to twenty two degrees")) | |
| # {'calls': [{'name': 'set_temperature', 'arguments': {'zone': 'paint_booth', 'celsius': 22}}]} | |
| ``` | |
| Use greedy decoding, and send the command in lower case, the way a | |
| speech-to-text engine usually returns it. | |
| ### On a CPU, with ONNX Runtime (INT4) | |
| ```python | |
| from optimum.onnxruntime import ORTModelForCausalLM | |
| from transformers import AutoTokenizer | |
| tok = AutoTokenizer.from_pretrained(repo, subfolder="onnx-int4") | |
| model = ORTModelForCausalLM.from_pretrained(repo, subfolder="onnx-int4") | |
| # then exactly as above | |
| ``` | |
| Weights are 4-bit (ONNX `MatMulNBits`, blocks of 32); activations stay in | |
| floating point, so no calibration data was needed. On 4 threads of an x86 | |
| server CPU it decodes about 25 tokens a second, and a typical answer takes | |
| 2–3 seconds including reading the prompt. A board's speed depends on its | |
| memory bandwidth; measure on the target. | |
| ## The tools | |
| It knows exactly these eleven tools, the ones listed in `prompt_template.txt`: | |
| | Tool | Arguments | | |
| |---|---| | |
| | `start_machine` | machine | | |
| | `stop_machine` | machine, mode `normal` or `emergency` | | |
| | `get_status` | machine | | |
| | `set_conveyor_speed` | line `A`/`B`/`C`, speed 0–120 m/min | | |
| | `count_parts` | line, period `last_hour`/`this_shift`/`today` | | |
| | `set_temperature` | zone, celsius | | |
| | `get_reading` | sensor `temperature`/`pressure`/`vibration`/`humidity`, location | | |
| | `set_alarm` | sensor, location, limit | | |
| | `set_lights` | area, on, brightness 0–100 (optional) | | |
| | `schedule_maintenance` | machine, day | | |
| | `call_staff` | role `supervisor`/`maintenance`/`safety`, area | | |
| Machines: press_1, press_2, cnc_1, cnc_2, robot_arm, packer, oven. | |
| Zones: oven, paint_booth, cold_store, warehouse. | |
| Areas: assembly, warehouse, office, loading_dock. | |
| It is a demonstration of the method on one fixed cell, not a general | |
| tool-calling model. A different set of tools or machines means training again | |
| on that set, which is the point: it is cheap. | |
| ## What it gets wrong | |
| Worth knowing before trusting it with anything: | |
| - **It sometimes acts when it should ask.** On 14 of 66 incomplete commands it | |
| guessed instead of asking which machine or line. Compressed to INT4 this gets | |
| worse (it asked on 39 of 66), which accounts for all of INT4's lost | |
| accuracy. More "ask" examples in training is the fix. | |
| - **Requests for a tool it does not have get forced into one it does.** | |
| "There's smoke near the oven" became an alarm on a `smoke` sensor that does | |
| not exist, instead of only paging safety. | |
| - **Ordinals and loose references trip it.** "Stop the first press and start | |
| the second one" put `second` where `press_2` should be. | |
| - **Its follow-up questions are not always the right question**, though it | |
| still correctly declines to act. | |
| - English only, trained on synthetic commands with simulated speech-to-text | |
| noise (fillers, numbers as words, run-together commands). Real operators, | |
| accents and recognisers will differ; test on your own recordings. | |
| **It is not a safety system.** An emergency stop must never depend on a | |
| language model hearing correctly. Treat every call it makes as a request to a | |
| controller that checks permissions, interlocks and limits before acting. | |
| ## How it was made | |
| - **Base:** [h2oai/h2o-danube3-500m-chat](https://huggingface.co/h2oai/h2o-danube3-500m-chat) (Apache-2.0). | |
| - **Data:** 12,000 synthetic spoken-style commands with their exact tool calls, | |
| 15% of them commands the model must decline or ask about. Generated by a | |
| script, no personal or customer data. | |
| - **Training:** full fine-tune, 600 steps, sequence length 640, batch 32, | |
| learning rate 2e-5 with cosine decay, AdamW, bf16, on one NVIDIA L4. | |
| Trained through the chat template it is served with. | |
| - **Checks along the way:** Foundry checked the job before training (including | |
| that the INT8 model fits a 3 GB device budget and could decode at 20 tokens | |
| a second), cleaned and decontaminated the data against the test set, and | |
| scored the result against the base model on commands it had never seen. | |
| - **Export:** ONNX via Hugging Face Optimum, then 4-bit weight quantisation | |
| with ONNX Runtime. | |
| ## Files | |
| | Path | What | | |
| |---|---| | |
| | `model.safetensors`, `config.json`, tokenizer files | The fine-tuned model (FP32, 2.05 GB) | | |
| | `prompt_template.txt` | The exact instruction the model was trained with; `{command}` is replaced by the transcript | | |
| | `onnx-int4/` | ONNX Runtime build, 4-bit weights (465 MB) | | |
| ## Licence | |
| Apache-2.0, as the base model. | |
| ## Contact | |
| Built by ThinkingDBX. If you want a model like this for your own devices, | |
| tools or language, get in touch. | |