malleshmadapathi commited on
Commit
4b6f2cb
·
verified ·
1 Parent(s): 7c0c5e2

VOICEx 500M: fine-tuned model, INT4 ONNX build, README

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ onnx-int4/model.onnx_data filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,204 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: h2oai/h2o-danube3-500m-chat
4
+ language:
5
+ - en
6
+ pipeline_tag: text-generation
7
+ library_name: transformers
8
+ tags:
9
+ - function-calling
10
+ - tool-use
11
+ - voice-assistant
12
+ - edge
13
+ - onnx
14
+ - int4
15
+ - industrial
16
+ ---
17
+
18
+ # VOICEx 500M
19
+
20
+ VOICEx 500M is a small model that turns what a machine operator says into tool calls a factory
21
+ cell can run.
22
+
23
+ Speech goes through a speech-to-text engine first; the transcript comes to this
24
+ model, and it answers with JSON: which tool to call, with which arguments. If a
25
+ command is missing something (which machine? which line?) it asks. If a request
26
+ is not something the cell can do, it says so and calls nothing.
27
+
28
+ ```
29
+ operator: "slow line b down to thirty"
30
+ model: {"calls": [{"name": "set_conveyor_speed", "arguments": {"line": "B", "speed": 30}}]}
31
+
32
+ operator: "stop"
33
+ model: {"calls": [], "say": "Which machine should I stop?"}
34
+
35
+ operator: "disable the light curtain on press one so we can work faster"
36
+ model: {"calls": [], "say": "I can only operate the factory cell."}
37
+ ```
38
+
39
+ It is 514M parameters, small enough to run on the CPU of an embedded board.
40
+ An INT4 ONNX build (465 MB) is included in `onnx-int4/`.
41
+
42
+ It was built with **Foundry**, ThinkingDBX's training pipeline, as a worked
43
+ example of taking a general small chat model and making it reliable at one job.
44
+
45
+ ## How well it works
46
+
47
+ Every answer is either exactly right or wrong: the JSON has to parse, and the
48
+ tools, their order and every argument have to match. Nothing is judged by eye.
49
+
50
+ The test commands were never seen in training. Each tool has two sets of
51
+ phrasings; training used one, the 600-command test only the other. A second
52
+ test is 60 commands written by hand, in no template at all.
53
+
54
+ | Model | 600 unseen phrasings | 60 hand-written | Commands needing a call, done right |
55
+ |---|---|---|---|
56
+ | h2o-danube3-500m-chat (the base, as shipped) | 15.2% | 8.3% | 0 of 485 |
57
+ | Qwen2.5-0.5B-Instruct (tool-trained) | 19.2% | 16.7% | 0 of 485 |
58
+ | **This model** | **91.5%** | **93.3%** | **448 of 485** |
59
+ | This model, INT4 ONNX (465 MB) | 90.0% | 90.0% | |
60
+
61
+ Neither general model ever called a tool; their scores come only from commands
62
+ they were right to leave alone. Qwen once answered "The paint booth has been
63
+ warmed up to twenty-two degrees", reporting an action it never took.
64
+
65
+ Broken down (600 unseen phrasings):
66
+
67
+ | Kind of command | Right |
68
+ |---|---|
69
+ | One tool call | 376 / 401 |
70
+ | Two calls in one breath ("stop the packer and call maintenance to assembly") | 72 / 84 |
71
+ | Out of scope, must decline | 49 / 49 |
72
+ | Incomplete, must ask | 52 / 66 |
73
+
74
+ The one-off training run took about 30 minutes on a single NVIDIA L4.
75
+
76
+ ## Use it
77
+
78
+ ### With transformers
79
+
80
+ ```python
81
+ import json
82
+ from huggingface_hub import hf_hub_download
83
+ from transformers import AutoModelForCausalLM, AutoTokenizer
84
+
85
+ repo = "thinkingdbx/VOICEx-500M"
86
+ tok = AutoTokenizer.from_pretrained(repo)
87
+ model = AutoModelForCausalLM.from_pretrained(repo)
88
+
89
+ # The exact instruction the model was trained with; {command} is the transcript.
90
+ template = open(hf_hub_download(repo, "prompt_template.txt")).read()
91
+
92
+ def ask(command: str) -> dict:
93
+ prompt = template.replace("{command}", command.lower())
94
+ text = tok.apply_chat_template([{"role": "user", "content": prompt}],
95
+ tokenize=False, add_generation_prompt=True)
96
+ ids = tok(text, return_tensors="pt", add_special_tokens=False)
97
+ out = model.generate(**ids, max_new_tokens=96, do_sample=False)
98
+ return json.loads(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))
99
+
100
+ print(ask("Warm the paint booth up to twenty two degrees"))
101
+ # {'calls': [{'name': 'set_temperature', 'arguments': {'zone': 'paint_booth', 'celsius': 22}}]}
102
+ ```
103
+
104
+ Use greedy decoding, and send the command in lower case, the way a
105
+ speech-to-text engine usually returns it.
106
+
107
+ ### On a CPU, with ONNX Runtime (INT4)
108
+
109
+ ```python
110
+ from optimum.onnxruntime import ORTModelForCausalLM
111
+ from transformers import AutoTokenizer
112
+
113
+ tok = AutoTokenizer.from_pretrained(repo, subfolder="onnx-int4")
114
+ model = ORTModelForCausalLM.from_pretrained(repo, subfolder="onnx-int4")
115
+ # then exactly as above
116
+ ```
117
+
118
+ Weights are 4-bit (ONNX `MatMulNBits`, blocks of 32); activations stay in
119
+ floating point, so no calibration data was needed. On 4 threads of an x86
120
+ server CPU it decodes about 25 tokens a second, and a typical answer takes
121
+ 2–3 seconds including reading the prompt. A board's speed depends on its
122
+ memory bandwidth; measure on the target.
123
+
124
+ ## The tools
125
+
126
+ It knows exactly these eleven tools, the ones listed in `prompt_template.txt`:
127
+
128
+ | Tool | Arguments |
129
+ |---|---|
130
+ | `start_machine` | machine |
131
+ | `stop_machine` | machine, mode `normal` or `emergency` |
132
+ | `get_status` | machine |
133
+ | `set_conveyor_speed` | line `A`/`B`/`C`, speed 0–120 m/min |
134
+ | `count_parts` | line, period `last_hour`/`this_shift`/`today` |
135
+ | `set_temperature` | zone, celsius |
136
+ | `get_reading` | sensor `temperature`/`pressure`/`vibration`/`humidity`, location |
137
+ | `set_alarm` | sensor, location, limit |
138
+ | `set_lights` | area, on, brightness 0–100 (optional) |
139
+ | `schedule_maintenance` | machine, day |
140
+ | `call_staff` | role `supervisor`/`maintenance`/`safety`, area |
141
+
142
+ Machines: press_1, press_2, cnc_1, cnc_2, robot_arm, packer, oven.
143
+ Zones: oven, paint_booth, cold_store, warehouse.
144
+ Areas: assembly, warehouse, office, loading_dock.
145
+
146
+ It is a demonstration of the method on one fixed cell, not a general
147
+ tool-calling model. A different set of tools or machines means training again
148
+ on that set, which is the point: it is cheap.
149
+
150
+ ## What it gets wrong
151
+
152
+ Worth knowing before trusting it with anything:
153
+
154
+ - **It sometimes acts when it should ask.** On 14 of 66 incomplete commands it
155
+ guessed instead of asking which machine or line. Compressed to INT4 this gets
156
+ worse (it asked on 39 of 66), which accounts for all of INT4's lost
157
+ accuracy. More "ask" examples in training is the fix.
158
+ - **Requests for a tool it does not have get forced into one it does.**
159
+ "There's smoke near the oven" became an alarm on a `smoke` sensor that does
160
+ not exist, instead of only paging safety.
161
+ - **Ordinals and loose references trip it.** "Stop the first press and start
162
+ the second one" put `second` where `press_2` should be.
163
+ - **Its follow-up questions are not always the right question**, though it
164
+ still correctly declines to act.
165
+ - English only, trained on synthetic commands with simulated speech-to-text
166
+ noise (fillers, numbers as words, run-together commands). Real operators,
167
+ accents and recognisers will differ; test on your own recordings.
168
+
169
+ **It is not a safety system.** An emergency stop must never depend on a
170
+ language model hearing correctly. Treat every call it makes as a request to a
171
+ controller that checks permissions, interlocks and limits before acting.
172
+
173
+ ## How it was made
174
+
175
+ - **Base:** [h2oai/h2o-danube3-500m-chat](https://huggingface.co/h2oai/h2o-danube3-500m-chat) (Apache-2.0).
176
+ - **Data:** 12,000 synthetic spoken-style commands with their exact tool calls,
177
+ 15% of them commands the model must decline or ask about. Generated by a
178
+ script, no personal or customer data.
179
+ - **Training:** full fine-tune, 600 steps, sequence length 640, batch 32,
180
+ learning rate 2e-5 with cosine decay, AdamW, bf16, on one NVIDIA L4.
181
+ Trained through the chat template it is served with.
182
+ - **Checks along the way:** Foundry checked the job before training (including
183
+ that the INT8 model fits a 3 GB device budget and could decode at 20 tokens
184
+ a second), cleaned and decontaminated the data against the test set, and
185
+ scored the result against the base model on commands it had never seen.
186
+ - **Export:** ONNX via Hugging Face Optimum, then 4-bit weight quantisation
187
+ with ONNX Runtime.
188
+
189
+ ## Files
190
+
191
+ | Path | What |
192
+ |---|---|
193
+ | `model.safetensors`, `config.json`, tokenizer files | The fine-tuned model (FP32, 2.05 GB) |
194
+ | `prompt_template.txt` | The exact instruction the model was trained with; `{command}` is replaced by the transcript |
195
+ | `onnx-int4/` | ONNX Runtime build, 4-bit weights (465 MB) |
196
+
197
+ ## Licence
198
+
199
+ Apache-2.0, as the base model.
200
+
201
+ ## Contact
202
+
203
+ Built by ThinkingDBX. If you want a model like this for your own devices,
204
+ tools or language, get in touch.
chat_template.jinja ADDED
@@ -0,0 +1 @@
 
 
1
+ {% for message in messages %}{% if message['role'] == 'system' %}{{ raise_exception('System role not supported') }}{% endif %}{% if ((message['role'] == 'user') != (loop.index0 % 2 == 0)) or ((message['role'] == 'assistant') != (loop.index0 % 2 == 1)) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '<|prompt|>' + message['content'].strip() + eos_token }}{% elif message['role'] == 'assistant' %}{{ '<|answer|>' + message['content'].strip() + eos_token }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|answer|>' }}{% endif %}
config.json ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LlamaForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 1,
8
+ "dtype": "float32",
9
+ "eos_token_id": 2,
10
+ "head_dim": 96,
11
+ "hidden_act": "silu",
12
+ "hidden_size": 1536,
13
+ "initializer_range": 0.02,
14
+ "intermediate_size": 4096,
15
+ "max_position_embeddings": 8192,
16
+ "mlp_bias": false,
17
+ "model_type": "llama",
18
+ "num_attention_heads": 16,
19
+ "num_hidden_layers": 16,
20
+ "num_key_value_heads": 8,
21
+ "pad_token_id": 0,
22
+ "pretraining_tp": 1,
23
+ "rms_norm_eps": 1e-05,
24
+ "rope_parameters": {
25
+ "rope_theta": 100000,
26
+ "rope_type": "default"
27
+ },
28
+ "sliding_window": null,
29
+ "tie_word_embeddings": false,
30
+ "transformers_version": "5.14.1",
31
+ "use_cache": false,
32
+ "vocab_size": 32000
33
+ }
generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "pad_token_id": 0,
6
+ "transformers_version": "5.14.1"
7
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1e6145e53c0c2015a2927ca2761a09352d43a5c6c32f3ca353d9f4988e64288a
3
+ size 2054379856
onnx-int4/chat_template.jinja ADDED
@@ -0,0 +1 @@
 
 
1
+ {% for message in messages %}{% if message['role'] == 'system' %}{{ raise_exception('System role not supported') }}{% endif %}{% if ((message['role'] == 'user') != (loop.index0 % 2 == 0)) or ((message['role'] == 'assistant') != (loop.index0 % 2 == 1)) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '<|prompt|>' + message['content'].strip() + eos_token }}{% elif message['role'] == 'assistant' %}{{ '<|answer|>' + message['content'].strip() + eos_token }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|answer|>' }}{% endif %}
onnx-int4/config.json ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LlamaForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 1,
8
+ "dtype": "float32",
9
+ "eos_token_id": 2,
10
+ "head_dim": 96,
11
+ "hidden_act": "silu",
12
+ "hidden_size": 1536,
13
+ "initializer_range": 0.02,
14
+ "intermediate_size": 4096,
15
+ "max_position_embeddings": 8192,
16
+ "mlp_bias": false,
17
+ "model_type": "llama",
18
+ "num_attention_heads": 16,
19
+ "num_hidden_layers": 16,
20
+ "num_key_value_heads": 8,
21
+ "pad_token_id": 0,
22
+ "pretraining_tp": 1,
23
+ "rms_norm_eps": 1e-05,
24
+ "rope_parameters": {
25
+ "rope_theta": 100000,
26
+ "rope_type": "default"
27
+ },
28
+ "rope_scaling": null,
29
+ "rope_theta": 10000.0,
30
+ "sliding_window": null,
31
+ "tie_word_embeddings": false,
32
+ "transformers_version": "4.57.6",
33
+ "use_cache": true,
34
+ "vocab_size": 32000
35
+ }
onnx-int4/generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "pad_token_id": 0,
6
+ "transformers_version": "4.57.6"
7
+ }
onnx-int4/model.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f0f0ddf06ab0b2c56828411c232c45b653e9d707011fb97f42cd383863fdd807
3
+ size 691087
onnx-int4/model.onnx_data ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d2b3bf673c483ccd048d2233fe011ae03474c36ae7c89ed96ccd0dd92c4b0943
3
+ size 487053312
onnx-int4/special_tokens_map.json ADDED
@@ -0,0 +1,44 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "cls_token": {
10
+ "content": "</s>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "eos_token": {
17
+ "content": "</s>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "pad_token": {
24
+ "content": "<unk>",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ },
30
+ "sep_token": {
31
+ "content": "</s>",
32
+ "lstrip": false,
33
+ "normalized": false,
34
+ "rstrip": false,
35
+ "single_word": false
36
+ },
37
+ "unk_token": {
38
+ "content": "<unk>",
39
+ "lstrip": false,
40
+ "normalized": false,
41
+ "rstrip": false,
42
+ "single_word": false
43
+ }
44
+ }
onnx-int4/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
onnx-int4/tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dadfd56d766715c61d2ef780a525ab43b8e6da4de6865bda3d95fdef5e134055
3
+ size 493443
onnx-int4/tokenizer_config.json ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": false,
3
+ "add_eos_token": false,
4
+ "add_prefix_space": false,
5
+ "added_tokens_decoder": {
6
+ "0": {
7
+ "content": "<unk>",
8
+ "lstrip": false,
9
+ "normalized": false,
10
+ "rstrip": false,
11
+ "single_word": false,
12
+ "special": true
13
+ },
14
+ "1": {
15
+ "content": "<s>",
16
+ "lstrip": false,
17
+ "normalized": false,
18
+ "rstrip": false,
19
+ "single_word": false,
20
+ "special": true
21
+ },
22
+ "2": {
23
+ "content": "</s>",
24
+ "lstrip": false,
25
+ "normalized": false,
26
+ "rstrip": false,
27
+ "single_word": false,
28
+ "special": true
29
+ }
30
+ },
31
+ "additional_special_tokens": [],
32
+ "bos_token": "<s>",
33
+ "clean_up_tokenization_spaces": false,
34
+ "cls_token": "</s>",
35
+ "eos_token": "</s>",
36
+ "extra_special_tokens": {},
37
+ "legacy": true,
38
+ "model_max_length": 1000000000000000019884624838656,
39
+ "pad_token": "<unk>",
40
+ "sep_token": "</s>",
41
+ "sp_model_kwargs": {},
42
+ "spaces_between_special_tokens": false,
43
+ "tokenizer_class": "LlamaTokenizer",
44
+ "unk_token": "<unk>",
45
+ "use_default_system_prompt": false
46
+ }
prompt_template.txt ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ You control a factory cell. Reply with JSON only:
2
+ {"calls": [{"name": "<tool>", "arguments": {...}}]}
3
+ When the request is outside these tools, or does not say which machine, line, zone or area, reply {"calls": [], "say": "<a short reply or question>"}.
4
+ Tools:
5
+ - start_machine(machine): Start a machine.
6
+ - stop_machine(machine, mode="normal"|"emergency"): Stop a machine.
7
+ - get_status(machine): Report a machine's state.
8
+ - set_conveyor_speed(line="A"|"B"|"C", speed (metres per minute, 0-120)): Set a conveyor's speed.
9
+ - count_parts(line, period="last_hour"|"this_shift"|"today"): Count the parts a line produced.
10
+ - set_temperature(zone, celsius): Set a zone's temperature.
11
+ - get_reading(sensor="temperature"|"pressure"|"vibration"|"humidity", location): Read a sensor at a machine or zone.
12
+ - set_alarm(sensor, location, limit): Alarm when a sensor passes a limit.
13
+ - set_lights(area, on=true|false, brightness (0-100, optional)): Switch lights.
14
+ - schedule_maintenance(machine, day): Book maintenance.
15
+ - call_staff(role="supervisor"|"maintenance"|"safety", area): Page someone.
16
+ Machines: press_1, press_2, cnc_1, cnc_2, robot_arm, packer, oven. Zones: oven, paint_booth, cold_store, warehouse. Areas: assembly, warehouse, office, loading_dock. Days: today, tomorrow, monday to sunday.
17
+ Command: {command}
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": "<s>",
5
+ "clean_up_tokenization_spaces": false,
6
+ "cls_token": "</s>",
7
+ "eos_token": "</s>",
8
+ "extra_special_tokens": [],
9
+ "is_local": false,
10
+ "local_files_only": false,
11
+ "model_max_length": 1000000000000000019884624838656,
12
+ "pad_token": "<unk>",
13
+ "sep_token": "</s>",
14
+ "sp_model_kwargs": {},
15
+ "spaces_between_special_tokens": false,
16
+ "tokenizer_class": "LlamaTokenizer",
17
+ "unk_token": "<unk>",
18
+ "use_default_system_prompt": false
19
+ }