| # Qwen2.5-Instruct for mobile (ONNX) and the transcript refiner |
|
|
| Converts `Qwen/Qwen2.5-1.5B-Instruct` to int4 ONNX files that ONNX Runtime runs on Android and iOS, ports the |
| transcript refiner (punctuation, misheard words, fillers, numbers, dates and units, bullet lists) to Kotlin and Swift, checks everything |
| against the Python reference, and publishes the public Hugging Face repo `pdnhan/Qwen2.5-1.5B-Instruct-mobile` |
| (Apache 2.0, used by the app). The same steps convert `arcee-ai/Arcee-VyLinh`, a Vietnamese merge of VyLinh-3B |
| with Qwen2.5-3B-Instruct (same architecture and tokenizer), to `pdnhan/Arcee-VyLinh-mobile` (`--size VyLinh-3B`). |
| It is published under Apache 2.0, the license Arcee AI gives it. |
|
|
| ```sh |
| uv venv .venv --python 3.12 |
| uv pip install --python .venv/bin/python --extra-index-url https://download.pytorch.org/whl/cpu \ |
| "torch==2.6.*" transformers onnx onnx_ir onnxscript onnxruntime onnxruntime-genai numpy regex jiwer \ |
| protobuf safetensors tokenizers huggingface_hub |
| |
| hf download Qwen/Qwen2.5-1.5B-Instruct --local-dir data/Qwen2.5-1.5B-Instruct |
| hf download arcee-ai/Arcee-VyLinh --revision 5f9751c095cdfa14c2e77489f8aef8a037da1b1c --local-dir data/Arcee-VyLinh |
| cp data/Qwen2.5-1.5B-Instruct/LICENSE data/Arcee-VyLinh/LICENSE # the repo has no LICENSE file; this is the Apache 2.0 text |
| # FLEURS Vietnamese test clips and PhoWhisper's int8 exports: see ../mobile_export |
| |
| .venv/bin/python -I export.py # out/<size>/repo (sizes: 1.5B, VyLinh-3B): onnx/, vocab.txt, merges.txt, LICENSE |
| ../mobile_export/.venv/bin/python -I asr_hyps.py # data/asr_hyps.tsv: PhoWhisper tiny and small transcripts |
| .venv/bin/python -I validate.py # tokenizer, prompt, WER and punctuation -> out/<size>/validation.json, fixtures |
| (cd kotlin && gradle -q run --args="../out/1.5B/repo ../out/1.5B/fixtures") # QwenRefiner.kt on the JVM, real ONNX Runtime |
| sh swift/check/run.sh # the Swift package in a swift:6.0 container, with a stand-in for ONNX Runtime |
| sh swift/check/real-ort/run.sh 1.5B # the same sources with real ONNX Runtime (C library) on the real model |
| .venv/bin/python -I publish.py # model card + code + upload (needs hf auth login) |
| ``` |
|
|
| `export.py` uses the safetensors weights and runs no code from the model repos (`hf_remote=false`). |
|
|
| `qwen_refine.py` is the reference pipeline the Kotlin (`kotlin/src/main/kotlin/QwenRefiner.kt`) and Swift |
| (`swift/Sources/QwenRefiner/`) ports follow step by step. The app copies the Kotlin file to |
| `app/src/main/java/vn/thithom/stt/refine/QwenRefiner.kt`; keep them in sync. A prompt change changes the expected |
| outputs: rerun `validate.py` and both checks, and update the app's device test. |
|
|
| The Swift code targets ONNX Runtime's Objective-C API, which only exists on Apple platforms. `swift/check/run.sh` |
| replays the Python reference's tokens through a stand-in; `swift/check/real-ort/` implements the same Objective-C |
| API on ONNX Runtime's C library, so the package's unchanged sources run the real model on Linux. |
|
|
