Scylla's Band Android sample

An on-device speech and singing studio for Scylla's Band (release v4-20261106, which speaks and sings). It runs the native libscyllasband runtime through LiteRT by default; nothing leaves the phone. Two tabs: Speak (a script with speaker points) and Sing (a melody built line by line).

What it demonstrates

All fourteen voices and each voice's languages, read from the runtime (scyllasband_voices_json) rather than hard-coded.
Delivery: three sliders, intensity (calm to intense) and pace (slower to faster, words and pauses) in the voice's own standard deviations and pitch in semitones, each from -2 to 2 in 0.1 steps; 0 is the voice's own delivery. A reset button sets all three back to 0 (see Delivery range).
A text editor with inline speaker points (voice, language, delivery). Double-tap or long-press to add a point, tap one to edit or remove it. A point applies until the next one; new points copy the last one's settings.
An example selector loading the repository's data/ files: walkthrough_demo.txt (tagged multi-voice, multilingual dialogue; also the first-launch document), test_document.txt (long narration) and emotional_text.txt (emotional prose). Tagged files become editable speaker points.
Streaming playback: each sentence the runtime finishes is queued to a mono float AudioTrack while the next renders. The playback transcript follows the audible sentence. Stop calls scyllasband_cancel, which ends synthesis at the next graph call.

Sing

The Sing tab builds a song as lines of notes and sings it with the runtime (ABI 3.1), as the iOS sample's Sing tab does:

Type a line's lyrics (ba-by splits a word into syllables, _ holds a syllable over another note), then tap a note chip to set its pitch on a one-octave keyboard, its length, or to make it a rest, split it, join it with the next note, insert a rest or delete it. Each line shows a sketch of its melody; rests of a quarter second or more separate the lines the voice sings one at a time.
The song card opens the settings: title, voice, style (pop or ballad), tempo, sound (Quality, the BigVGAN decoder, or Turbo, Vocos) and key (transpose, with how the melody sits in the voice's range and a suggested transpose).
Every change is checked by the runtime (ScyllasBand.parseScore, singFit; the note it objects to is outlined) without waiting for a song in progress. Preview plays the melody as tones (preview, no model runs).
Sing streams the song while it renders when rendering keeps ahead of playback, else renders it, then plays; the take is kept until the song changes, and Another take sings it with a new seed.
Examples loads the release's twenty example scores (melodies/, embedded from scyllasband/models/melodies); Songs starts a new song or opens one saved with Save (JSON in the app's files); Share sends the score (the runtime's format, which python -m scyllasband sing --score reads) and the last take as a WAV.

The song model (app/.../sing/SongDocument.kt, SongEditing.kt) is plain Kotlin with JVM tests against the runtime's own parse of every example (./gradlew :app:testDebugUnitTest). Debug builds take launch options for testing: adb shell am start -n org.scyllasband.demo/.MainActivity --es sing_example song_pop_gwen [--ez sing_turbo true] --ez autosing true opens the Sing tab, loads the example, sings it and logs SBSING <status> (tag ScyllasBandSing).

Dialogue tags

Imported text uses the same format as python -m scyllasband group-speak:

[ariadne:en_us:intensity=1] Good to see you. [es] Me alegra verte.
[ink:en_gb:intensity=-1,pace=-0.5] Keep this between us.

[voice:language:delivery] sets all three (an empty voice or language keeps the active one). A language-only tag such as [es] keeps the active voice and delivery. A tag with a delivery part replaces the delivery, and controls it does not name return to the voice's own. [voice] alone switches voice. Each line starts from the default settings. Values outside -2 to 2 are clamped into the range, as the runtime would; v2 delivery axes (energy, tension, valence, assertiveness) are rejected.

Build

Download the LiteRT bundle (about 470 MB) and the example scores from the repository root:
```
python -m scyllasband download --yes
```

Stage the LiteRT Android prebuilts (git-ignored, under libscyllasband/third_party/litert/lib/<platform>/):

python libscyllasband/scripts/stage_litert_sdk.py --platform android-arm64 --download-runtime --download-gpu-accelerator --overwrite
python libscyllasband/scripts/stage_litert_sdk.py --platform android-x86_64 --download-runtime --download-gpu-accelerator --overwrite

Build from this directory (needs the Android SDK, NDK 29.0.13113456 and CMake):

./gradlew :app:assembleDebug
./gradlew :app:testDebugUnitTest :scyllasband-android:testDebugUnitTest

The APK is app/build/outputs/apk/debug/app-debug.apk. It embeds the bundle as assets and the first launch copies it to noBackupFilesDir, because the runtime needs filesystem paths. Production apps should deliver the model as an install-time asset pack or a one-time download, and keep only the target ABI. arm64-v8a and x86_64 are built; arm64-v8a is the one that matters on devices.

libLiteRt.so and libLiteRtClGlAccelerator.so are packaged with jniLibs.useLegacyPackaging = true so they are extracted to disk: LiteRT loads its GPU accelerator from the directory of libLiteRt.so.

ONNX Runtime instead of LiteRT

python -m scyllasband download --flavor onnx --yes
./gradlew -Pscyllasband.backend=onnx :app:assembleDebug

This links libscyllasband against ONNX Runtime 1.30.0 (onnxruntime-android) and embeds scyllasband/models/onnx (about 1.1 GB, full precision). ONNX Runtime builds run on the CPU. LiteRT is the default.

Accelerator

ScyllasBand.initialize(threadCount, accelerator) takes CPU (XNNPACK, up to 4 threads; the default), GPU or AUTO. The sample uses AUTO: the graphs the bundle lists in controls.litert.gpu.graphs run on the GPU, everything else on the CPU, and a bundle that lists none runs entirely on the CPU.

LiteRT's GPU accelerator uses the vendor's OpenCL driver. Since Android 12 an app loads a vendor library only after declaring it, so the manifest has <uses-native-library android:name="libOpenCL.so" android:required="false" /> (and the -car and -pixel variants). Without it LiteRT falls back to its OpenGL path, which fails, and every graph runs on the CPU: the GPU worked from adb shell but never inside the app.

CPU is fast enough for speech: on a Galaxy Z Fold 8 (Snapdragon SM8850) the native runtime speaks a three-sentence passage (10.3 s of audio) with first audio about 0.52 s after launch (0.8 s on the first launch, which writes the weight cache) and 120 ms once warm, about 27 times faster than real time.

For singing with BigVGAN use AUTO with a bundle that names BigVGAN as its GPU graph (controls.litert.gpu.graphs, as the release's LiteRT bundle does with its channels-last BigVGAN graph): BigVGAN then runs on the GPU and every other graph on the CPU. With v4-20261106 (the flow distilled to 4 Euler steps without guidance) the Fold sings the 71 s example song in 19 s with Quality (3.7 times faster than real time, first audio after 1.9 s, 764 MB; 71 s on the CPU alone) and in 1.7 s with Turbo (42 times, first audio after 0.26 s; Vocos runs on the CPU either way). The first launch after an install compiles BigVGAN's GPU program (first audio about 5 s), later launches load it from the cache next to the bundle; the Sing tab warms up the song's decoder when it first opens. In the background (another app in front, or the screen locked) Android moves the app to its slower cores, so a song that should keep rendering there needs a foreground service.

Don't put every graph on the GPU with GPU (AUTO with a bundle that lists no GPU graphs stays on the CPU): with LiteRT 2.2.0 the duration predictor returns wrong durations there (speech plays too fast), the flow is slower than on the CPU, and a channels-first BigVGAN graph aborts the process in the GPU delegate.

Delivery range

The runtime supports intensity, pace and pitch from -2 to 2 (scyllasband_voices_json reports the ranges and each voice's defaults, read into ScyllasBandBundleInfo.deliveryRanges and ScyllasBandVoice.deliveryDefaults). The sliders cover exactly that range, and values read from tagged documents are clamped into it when they become speaker points. ScyllasBandDelivery rejects values outside it; a control left null takes the voice's default.

Minimal integration

private val scyllasband by lazy { ScyllasBand.create(applicationContext) }

// Off the main thread: installs the bundle, creates the runtime, warms it up.
val info = scyllasband.initialize()   // info.voices, info.sampleRate, info.backend, info.accelerator

val settings = ScyllasBandSegmentSettings(
    voiceId = "ariadne",
    language = "en_us",
    delivery = ScyllasBandDelivery(intensity = 1f, pace = -0.5f),
)
scyllasband.synthesizeStreaming(
    text = text,
    settings = settings,
    seed = 31_415L,                      // optional; null draws fresh noise
    onChunkStarted = { index, count, sentence -> true },
    onAudioChunk = { samples, index, count -> true },   // return false to stop
)

scyllasband.cancel()   // from any thread
scyllasband.close()

Synthesis uses the bundle defaults (Euler, 4 steps); steps is an optional parameter. ScyllasBandDelivery.spec() produces the ABI delivery string, for example intensity=1,pace=-0.5 (empty when every control takes the voice's default).

Modules

libscyllasband is the shared C++ runtime (ABI 3.1.0).
scyllasband-android builds it with -DSCYLLASBAND_BACKEND=litert|onnx, adds the JNI layer, installs the bundle and exposes the Kotlin API above.
app owns the marker editor, point dialog, document segmentation, lifecycle and audio playback (Speak), and the song editor (sing/: SongDocument, SongEditing, SingScreen, PianoKeysView, ContourView, SongStore).

The app needs no network, microphone or storage permission.