| > The .litertlm models on this card already use the QAT that is discussed in the blog post. The most popular file, `gemma-4-E2B-it.litertlm`, uses a mixture of int2, int4 and int8 to keep it small, fast and efficient. |
|
|
| https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm/discussions/30#6a2c2e1ac539f50158631020 |
|
|
|
|
|
|
| # Voice Chatbot (Android) — interruptible, on-device voice chat pipeline |
|
|
| A Kotlin/Compose Android app implementing a fully local, **interruptible** |
| voice chatbot: |
|
|
| ``` |
| mic ─► Silero VAD (sherpa-onnx) |
| │ endpointing + barge-in |
| ▼ |
| Whisper tiny.en Q4 (whisper.cpp) ──► transcript |
| ▼ |
| LLM (streaming): |
| ├─ Local: LiteRT-LM (`.litertlm`, e.g. Qwen3-0.6B) |
| └─ Remote: OpenAI-compatible `/v1/chat/completions` (SSE) |
| ▼ |
| SentenceSplitter ──► per-sentence VITS TTS (sherpa-onnx + Piper voice) |
| ▼ |
| AudioTrack (playback; barge-in stops it instantly) |
| ``` |
|
|
| **Barge-in:** speak while the assistant is talking → playback and generation |
| stop immediately and your new utterance becomes the next turn. |
|
|
| ## Components |
|
|
| | Piece | Runtime | Model | Size | |
| |---|---|---|---| |
| | STT | whisper.cpp (JNI, built via NDK/CMake) | `ggml-tiny.en-q4_0.bin` (Q4) | ~45 MB | |
| | VAD | sherpa-onnx AAR (official prebuilt `sherpa-onnx-1.13.4.aar`) | `silero_vad.onnx` | ~2 MB | |
| | TTS | sherpa-onnx VITS (Piper) | `vits-piper-en_US-lessac-medium` (+ `espeak-ng-data`) | ~62 MB | |
| | LLM (local) | LiteRT-LM `com.google.ai.edge.litertlm:litertlm-android:0.15.0` | `litert-community/Qwen3-0.6B` → `Qwen3-0.6B.litertlm` (dynamic INT8) | 586 MB | |
| | LLM (remote) | OkHttp + kotlinx.serialization (OpenAI-compatible chat completions) | any remote model, e.g. `qwen2.5-0.5b-instruct` | n/a | |
|
|
| All models are Apache-2.0 or MIT except the Piper voice/GPL espeak-ng data |
| (bundled in the sherpa-onnx tts-model tarball) — fine for local testing. |
|
|
| ## Prerequisites |
|
|
| - macOS/Linux with Android SDK (platform 35, build-tools 36, NDK `27.2.12479018`, CMake 3.22.1), |
| JDK 17, and a connected Android device (arm64, 8 GB RAM recommended). |
| - `adb` on PATH. |
|
|
| ## Build |
|
|
| Toolchain: **AGP 8.13.2 · Gradle 8.13 (wrapper) · Kotlin 2.3.21 · JDK 17**. |
| whisper.cpp is vendored under `third_party/` (committed), so the project builds |
| out-of-the-box once the SDK/NDK are present. |
|
|
| ```bash |
| # 0. JDK 17 (Gradle 8.13 does not support newer JDKs) |
| # macOS/Homebrew: brew install openjdk@17 |
| export JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home |
| |
| # 0b. Point Gradle at the Android SDK (or set ANDROID_HOME) |
| echo "sdk.dir=$HOME/Library/Android/sdk" > local.properties |
| |
| # 1. (first time) install NDK/cmake if missing |
| "$ANDROID_HOME"/cmdline-tools/latest/bin/sdkmanager \ |
| "platforms;android-35" "ndk;27.2.12479018" "cmake;3.22.1" |
| |
| # 2. Build (Gradle downloads the sherpa-onnx AAR into app/libs on first run) |
| ./gradlew :app:assembleDebug |
| |
| # 3. Install |
| adb install -r app/build/outputs/apk/debug/app-debug.apk |
| ``` |
|
|
| Unit tests: `./gradlew :app:testDebugUnitTest` |
|
|
| > Only if `third_party/whisper.cpp` is ever missing (e.g. a fresh export that |
| > dropped it): `tools/fetch-deps.sh` re-clones it. |
| |
| ## Models |
| |
| **Models are bundled in the APK** (`assets/models/`) and copied to the app's |
| private internal storage on first launch — no adb push needed. Storage used on |
| the device: `~240 MB` under `/data/data/com.example.voicechatbot/files/models/`. |
| |
| > The APK is ~500 MB because of the bundled models; devices need **~1.5 GB free** |
| > to install it. |
| |
| **Preinstalled:** SmolLM2 135M (LLM) + Whisper tiny.en Q4 + Silero VAD + Piper TTS. |
| |
| **Download in-app:** bigger LLMs (SmolLM2 360M, Qwen3 0.6B, Gemma 4 E2B/E4B/12B) |
| and Whisper models are selected in Settings via **preset chips that |
| auto-download the file if it isn't already saved** (progress shown on the chip). |
| |
| |
| | id | Model | Size | Preinstalled | Thinking | License | |
| |---|---|---|---|---|---| |
| | `smollm2-135m` | SmolLM2 135M | 136 MB | ✅ | no | Apache-2.0 | |
| | `smollm2-360m` | SmolLM2 360M | 356 MB | download | no | Apache-2.0 | |
| | `qwen3-0.6b` | Qwen3 0.6B int8 | 586 MB | download | yes (stripped) | Apache-2.0 | |
| | `gemma-4-e2b` | Gemma 4 E2B (mobile 2/4/8-bit) | 2583 MB | download | yes (stripped) | Apache-2.0 | |
| | `gemma-4-e4b` | Gemma 4 E4B (mobile 2/4/8-bit) | 3654 MB | download | yes (stripped) | Apache-2.0 | |
| | `gemma-4-12b` | Gemma 4 12B | 6548 MB | download | yes (stripped) | Apache-2.0 | |
| |
| To rebuild with bundled models, the files must be present in `./models/`: |
| |
| ```bash |
| tools/download-models.sh # downloads all LLM presets + whisper/VAD/TTS |
| tools/download-models.sh smollm2-360m # or just one |
| tools/download-models.sh list # show presets |
| ``` |
| |
| (`tools/push-models.sh` is obsolete — models are in the APK now.) |
| |
| > **LFM-700M:** there is no `.litertlm` build of LFM2-700M on HF yet, and a |
| > conversion requires the `litert-torch` generic-HF-export pipeline (heavy, not |
| > guaranteed for the architecture). The SmolLM2 presets are the practical small, |
| > non-thinking, Apache-2.0 alternatives. |
| |
| > **Gemma 4 E2B / E4B:** official `litert-community` LiteRT-LM builds of |
| > `google/gemma-4-E2B-it` / `google/gemma-4-E4B-it` (Apache-2.0), using Google's |
| > "Gemma-4 mobile" 2/4/8-bit (LUT) quantization. LiteRT-LM **memory-maps the |
| > weights straight from the `.litertlm` file on disk** — the embedding tables are |
| > mmap'd and never fully loaded into RAM — so the ~2.6 GB / ~3.7 GB files don't |
| > need to fit in memory. Both support up to 32k context. Android benchmarks |
| > (S26 Ultra, 2048 ctx): E2B ~1.7 GB CPU / ~0.7 GB GPU process RAM at |
| > ~47–52 tok/s; E4B ~3.3 GB CPU / ~0.7 GB GPU at ~18–22 tok/s — E4B really wants |
| > a 12 GB-class phone **and** the GPU backend. Settings → *Context window* |
| > (LiteRT-LM `maxNumTokens`) caps the KV cache so bigger models fit in less RAM. |
| > **Gemma 4 12B** (`gemma-4-12B-it.litertlm`, ~6.5 GB) is the largest option — |
| > realistically it needs a 16 GB-class device with the GPU backend. |
| |
| > **TTS voices:** only the bundled `en_US-lessac-medium` voice is used. In-app |
| > voice downloads were removed; Settings → *Voice (VITS)* just shows the |
| > preinstalled voice (blank the model path for text-only chat). |
|
|
| > **Whisper (STT):** the bundled model is `ggml-tiny.en-q4_0.bin` (~45 MB). |
| > Better variants are available as Settings → *Whisper model* presets |
| > (auto-downloaded on selection): unquantized tiny.en / tiny (FP16, ~75 MB), |
| > tiny.en Q8_0 (~42 MB), or base.en Q8_0 (~78 MB). Q4_0 builds are **not** |
| > published on Hugging Face (`tools/download-models.sh` quantizes them locally), |
| > so Q8_0 is the closest official base.en quantized download. Larger models are |
| > slower but more accurate, especially for non-English speech (use the |
| > multilingual `tiny`). |
| |
| ## Running & testing |
| |
| 1. Grant the **Record audio** permission when prompted — listening starts |
| automatically, so there is no Start button. |
| 2. Say something. You should see `Listening → Transcribing → Thinking → |
| Speaking`, and hear the reply sentence-by-sentence. |
| 3. **Barge-in:** start talking mid-reply. Playback stops at once and your |
| question is picked up as the next turn. |
| 4. **Remote LLM:** open Settings → *Remote (OpenAI API)* → set base URL |
| (e.g. `http://<your-server>:8000/v1`) + model name, save & restart. |
| Works with llama.cpp server, vLLM, Ollama, etc. |
| **Context length:** Settings → *Context length* (512…32768). The app sends |
| it as `n_ctx`, which llama.cpp honours per-request. "Server default" sends |
| nothing — the server decides (vLLM/Ollama manage context server-side). |
| 5. **Stop** cancels TTS + generation and re-arms listening. |
| 6. **Visualizer mode:** tap the eye icon in the top bar to hide the transcript. |
| The screen then shows only a minimal audio-level bar visualizer and the |
| sentence that is currently being spoken, rendered in a cursive glowing font. |
| 7. **Mini screen + submit photo:** tap the PiP icon to enable a small draggable |
| floating screen (grant the *Display over other apps* permission) that shows |
| the currently-spoken sentence in the glowing cursive font, with a distinct |
| submit-photo button (opens the system photo picker, like the chat's photo |
| icon). While the mini screen is active a **foreground service with a |
| persistent "listening" notification** keeps the microphone running in the |
| background (Android 14+). Image input only works with a **Gemma 4** model on |
| the local LiteRT-LM backend; anything else shows an error. |
| 8. **Sampling:** Settings → *Sampling (local)* (temperature / top-p / top-k for |
| LiteRT-LM) and *Sampling (remote)* (temperature / top-p / max tokens for the |
| OpenAI-compatible endpoint). |
|
|
| Tips: |
|
|
| - Use headphones or a quiet room: without acoustic echo cancellation support |
| the assistant may "barge in" on itself. The capture uses |
| `VOICE_COMMUNICATION` + `AcousticEchoCanceler` when available. |
| - TTS can be disabled by blanking the TTS model path in Settings (text-only |
| chat) for quick LLM testing. |
| - GPU backend: for the **local** LiteRT-LM backend this is LiteRT's GPU |
| delegate ("ML Drift", OpenCL on Android) — roughly doubles decode speed and |
| cuts process RAM a lot (Gemma E4B: ~3.3 GB → ~0.7 GB). For the **remote** |
| backend the GPU is on the server (llama.cpp CUDA/Metal/OpenCL, vLLM, …), not |
| in the app. Enable/disable it in Settings → *GPU backend*. |
|
|
| ## Project layout |
|
|
| ``` |
| app/src/main/java/com/example/voicechatbot/ |
| MainActivity.kt Compose UI + settings dialog |
| config/AppConfig.kt persisted settings & model paths |
| pipeline/VoicePipelineEngine.kt state machine, barge-in, sentence streaming |
| audio/AudioRecorder.kt 16 kHz capture (+AEC) |
| audio/TtsPlayer.kt AudioTrack queue with generation-based barge-in |
| vad/VadEngine.kt Silero VAD (sherpa-onnx) |
| stt/WhisperTranscriber.kt whisper.cpp JNI wrapper |
| stt/WhisperNative.kt JNI declarations |
| llm/ChatClient.kt backend interface |
| llm/LitertChatClient.kt LiteRT-LM local inference (Flow streaming) |
| llm/OpenAiChatClient.kt OpenAI-compatible remote client (SSE) |
| llm/SentenceSplitter.kt token stream → sentence boundaries |
| llm/PromptBuilder.kt system prompt |
| tts/TtsEngine.kt VITS/Piper synthesis (sherpa-onnx) |
| app/src/main/cpp/ whisper.cpp CMake + JNI bridge |
| third_party/whisper.cpp vendored (committed, v1.7.5) |
| tools/download-models.sh model fetch + Q4 quantize (local) |
| tools/push-models.sh adb push models to device |
| ``` |
|
|
| ## Notes & caveats |
|
|
| - **LiteRT-LM** requires a high-end device (Pixel 8 / S23 class). Engine init |
| takes a few seconds and is done off the UI thread. |
| - On interrupt, the current conversation context is dropped (clean slate for |
| the next turn). |
| - This is a test app: no wake word, no streaming partial transcription, no |
| foreground service (screen must stay on), single ABI (`arm64-v8a`). |
|
|
