Moonshine Base Japanese: ONNX Runtime (`.ort`) with cross-attention outputs for word timestamps

Powered by Moonshine AI. This Moonshine AI Model is licensed under the Moonshine AI Community License, Copyright © Moonshine AI Ltd. All Rights Reserved. It is not MIT: research and non-commercial use is free; commercial use requires registering with Moonshine AI; organisations (with affiliates) above US$1M annual revenue have no licence under these terms and need an enterprise licence. Use restrictions apply. See LICENSE, NOTICE and License.

This repository contains Moonshine Base, a small (≈61 M parameters) encoder-decoder speech recognition model for Japanese, as quantized ONNX Runtime .ort files for on-device use (Android, desktop CPU). Its decoder also returns its cross-attention weights (cross_attentions.0 … cross_attentions.7, one per decoder layer).

Relation to Moonshine AI's files. Moonshine AI publishes an attention-enabled decoder only for English (base-en/…/decoder_with_attention.ort). This repo provides the same thing for Japanese: the same file name and the same cross_attentions.{i} outputs that Moonshine's runtime reads from the English file.

Why: the cross-attention maps show which part of the audio each generated token attended to. Aligning them with dynamic time warping (DTW) gives character-level timestamps without a separate alignment model and without running the decoder a second time. The attention values are computed anyway, and fetching them costs nothing measurable (see Speed under Benchmarks). This is the same idea as Whisper's word timestamps and Moonshine's own core/word-alignment.cpp.

Provenance and what was changed


Original model	`UsefulSensors/moonshine-base-ja` (Moonshine Base, Japanese) by Useful Sensors / Moonshine AI
Upstream files	Moonshine AI's quantized ONNX Runtime files `download.moonshine.ai/model/base-ja/quantized/base-ja/*.ort` (fetched with `moonshine_voice.download`)
Direct parent	`sherpa-onnx-moonshine-base-ja-quantized-2026-02-27.tar.bz2` from the k2-fsa/sherpa-onnx `asr-models` release: sherpa-onnx repackaged the upstream `encoder_model.ort` and `decoder_model_merged.ort` and converted Moonshine's `tokenizer.bin` into `tokens.txt`
`encoder_model.ort`	unmodified, byte-identical to the parent
`tokens.txt`	unmodified, byte-identical to the parent
`test.wav`	`test_wavs/0.wav` from the sherpa-onnx archive, which sherpa-onnx downloads from `https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/ja.wav`; its licence is not stated
`decoder_with_attention.ort`	parent `decoder_model_merged.ort` + 8 extra outputs (below)

What was changed (decoder). The merged decoder has one If node (use_cache_branch) with a no-cache branch and a with-cache branch. In both branches, an ONNX Identity node was attached to the output of each layer's cross-attention softmax (/model/decoder/layers.{i}/encoder_attn/Softmax_output_0, i = 0…7). These are exposed as new graph outputs cross_attentions.0 … cross_attentions.7 (16 Identity nodes in total). The graph was loaded and re-saved with ONNX Runtime graph optimisations disabled (ORT_DISABLE_ALL) so that the quantized operators are kept exactly as they were.

The quantized operators are DynamicQuantizeLinear, MatMulIntegerToFloat and DynamicQuantizeMatMul with uint8 weights.

What was not changed. Weights, quantization, every pre-existing input and output (names, shapes, dtypes), the encoder and the tokenizer. The attention probabilities were already computed internally; they are now also returned. With identical inputs the patched decoder produces bit-identical logits to the parent decoder_model_merged.ort, so transcripts are identical to the parent model (100/100 FLEURS utterances, see Benchmarks).

Files

File	Size	Description
`encoder_model.ort`	31.3 MB	Encoder: raw 16 kHz waveform → hidden states (ONNX Runtime format, quantized)
`decoder_with_attention.ort`	109.0 MB	Merged decoder with KV cache (`use_cache_branch`) + `cross_attentions.0-7` outputs
`tokens.txt`	0.5 MB	Vocabulary, 32,768 lines: `<base64 of the piece's UTF-8 bytes> <id>`
`test.wav`	0.7 MB	Sample clip (44100 Hz, 8.16 s; resample to 16 kHz before use)
`config.json`	–	Machine-readable model facts (dimensions, special ids, inputs, defaults)
`moonshine_timestamps.py`	–	Reference Python script: transcription + word timestamps (MIT)
`LICENSE`	–	Licence text (copied from moonshine-ai/moonshine)
`NOTICE`	–	Attribution and list of changes required by the Moonshine AI Community License

Model inputs and outputs

Encoder (encoder_model.ort)

Name	Type	Shape	Notes
in `input_values`	float32	`[batch, samples]`	raw mono waveform, 16 kHz, range [-1, 1]; no feature extraction
in `attention_mask`	int64	`[batch, samples]`	1 for every valid sample (all ones for a single clip)
out `last_hidden_state`	float32	`[batch, frames, 416]`	`frames = ((((samples − 127) // 64 + 1 − 7) // 3 + 1 − 3) // 2 + 1)` ≈ 41.7 frames/s (one frame ≈ 24 ms)

Decoder (decoder_with_attention.ort): 8 layers, 8 heads, head size 52, d_model 416

Name	Type	Shape	Notes
in `encoder_attention_mask`	int64	`[batch, samples]`	sample-level mask, i.e. the same tensor as the encoder's `attention_mask` (the graph downsamples it internally with stride 384). Passing a frame-level mask of length `frames` only works by accident for clips shorter than ≈9.2 s (≤ 384 frames) and fails for longer ones
in `input_ids`	int64	`[batch, dec_len]`	`[[1]]` (`<s>`) on the first call, then the previously generated token
in `encoder_hidden_states`	float32	`[batch, frames, 416]`	encoder output
in `past_key_values.{i}.decoder.key` / `.value`	float32	`[batch, 8, past_len, 52]`	self-attention cache, i = 0…7; zero-length (`past_len = 0`) on the first call
in `past_key_values.{i}.encoder.key` / `.value`	float32	`[batch, 8, frames, 52]`	cross-attention cache; zero-length on the first call, then the `present.{i}.encoder.` of the first* call
in `use_cache_branch`	bool	`[1]`	`false` on the first call, `true` afterwards
out `logits`	float32	`[batch, dec_len, 32768]`	greedy: `argmax(logits[0, -1])`
out `present.{i}.decoder.key` / `.value`	float32	`[batch, 8, past_len + dec_len, 52]`	feed back as `past_key_values.{i}.decoder.*`
out `present.{i}.encoder.key` / `.value`	float32	`[batch, 8, frames, 52]`	only meaningful when `use_cache_branch = false`; the with-cache branch returns a dummy `[0, 8, 1, 52]` tensor, so keep the first call's copy
out `cross_attentions.{i}` (new)	float32	`[batch, 8, dec_len, frames]`	softmax cross-attention probabilities of decoder layer i (declared `[?, 8, ?, ?]`); each row sums to 1 over `frames`

The decoder has 33 original outputs + 8 new ones = 41 outputs. Special token ids: 0 = <unk>, 1 = <s> (decoder start), 2 = </s> (end). No language or task prompt tokens are needed: the model is monolingual.

tokens.txt (sherpa-onnx format): one line per id, <base64> <id>, where the base64 string encodes the raw UTF-8 bytes of the SentencePiece piece (one piece, id 31353, is empty, so its line starts with a space). Byte-fallback pieces (ids 3–258) are stored as the single raw byte 0x00–0xFF they stand for (byte = id − 3). To detokenize: concatenate the bytes of all non-special tokens, decode the result as UTF-8 once at the end (a character may be split over several byte tokens), then replace ▁ (U+2581) with a space and strip. A piece that starts with ▁ starts a new word.

Vocabulary note. Ids 32000–32767 are unused placeholder pieces <<ST_0>> … <<ST_767>>; treat them as special tokens with no text. The script masks them before argmax.

How the timestamps are computed

The decoder is run greedily, one token per call with the KV cache. After each call, the attention row of the newest token, cross_attentions.{l}[0, :, -1, :], is stored for all 8 layers × 8 heads. Row k belongs to the step that produced token k (step 0 consumes <s>); the step that produced </s> is kept too. The rows form a tensor A[64 heads, steps, frames], and then:

Normalize: z-score every head over the token axis (--method moonshine instead z-scores every (head, token) row over the frame axis, as in Moonshine's core/word-alignment.cpp).
Smooth: median filter of width 7 along the frame axis.
Average the 64 heads, giving M[steps, frames].
DTW: compute the monotonic path through the cost −M from (first token, frame 0) to (last step, last frame).
Token times: a token starts at the frame where the path first enters its row and ends where the next token starts (the </s> row closes the last token). Seconds = frame × duration / frames (≈ 24 ms per frame). With --method moonshine a token spans the min…max+1 frames of its path cells.
Words: consecutive tokens are merged into a word at every piece that starts with ▁; overlaps are snapped to the midpoint. Japanese text has no ▁ word boundaries (only the first token carries one), so for this language the script outputs character-level segments instead: each chunk is the smallest run of tokens that decodes to whole UTF-8 characters (usually one character; byte-fallback tokens are merged). Closing punctuation such as 、。，」 is attached to the preceding character, opening brackets and quotes such as 「（“ to the following one.
Optional --trim-silence: each word is shrunk to the 10 ms frames inside it whose energy is above a threshold (95th percentile − 35 dB). DTW gives a pause to the word that follows it; trimming removes most of that error (see Limitations).

The output has the same shape as the Android app's JSON: {"text": ..., "chunks": [{"text": word, "timestamp": [start_s, end_s]}, ...]}, plus "truncated": true when a window ended without </s>.

Quickstart (Python)

Requirements: Python ≥ 3.9, numpy ≥ 1.20, onnxruntime (tested with 1.24.1), soundfile; librosa (or scipy) for resampling. The hf command needs huggingface_hub ≥ 0.34 (older versions: huggingface-cli download).

pip install onnxruntime numpy soundfile huggingface_hub librosa
hf download ipsilondev/moonshine-attention-ort-ja --local-dir moonshine-attention-ort-ja
python moonshine-attention-ort-ja/moonshine_timestamps.py --model-dir moonshine-attention-ort-ja moonshine-attention-ort-ja/test.wav
python moonshine-attention-ort-ja/moonshine_timestamps.py --model-dir moonshine-attention-ort-ja your_audio.wav --trim-silence -o result.json

# script only; the model is downloaded to the Hugging Face cache on first use
hf download ipsilondev/moonshine-attention-ort-ja moonshine_timestamps.py --local-dir .
python moonshine_timestamps.py --lang ja your_audio.wav

Options:

--trim-silence: shrink each chunk to its voiced frames. Recommended for subtitles (see Limitations).
--method {whisper,moonshine}: alignment variant (see above). The Android app uses moonshine.
--tokens-per-sec: token budget per window, ceil(duration × tokens_per_sec) + 10 (default 13; 20 for Vietnamese).
--max-chunk-sec 30: longer files are split into windows of at most 30 s, each cut at the quietest point in the last 5 s before the limit.
--normalize: RMS-normalise each window to −20 dBFS (helps quiet recordings).
--threads N, -v (timings), -o file.json.

If a window ends without </s> (token budget used up, or a repetition loop that the script cut off), the JSON also contains "truncated": true and a warning is printed to stderr.

Output for test.wav (default settings; first 4 of 46 chunks shown; this is raw model output, including any recognition errors):

{
  "text": "国があなたのために何ができるかを問うのではなく、あなたが国のために何ができるかを問うてください。",
  "chunks": [
    {"text": "国", "timestamp": [0.0, 0.531]},
    {"text": "が", "timestamp": [0.531, 0.531]},
    {"text": "あ", "timestamp": [0.531, 0.531]},
    {"text": "な", "timestamp": [0.531, 0.7]}
  ]
}

Use from Python. The script is also a small library:

import sys; sys.path.insert(0, "moonshine-attention-ort-ja")
from moonshine_timestamps import MoonshineAttention, load_audio, transcribe_long

model = MoonshineAttention("moonshine-attention-ort-ja")
result = transcribe_long(model, load_audio("moonshine-attention-ort-ja/test.wav"), lang="ja", trim=True)
print(result["text"])
print(result["chunks"][:3])

The decoding loop, step by step. Inputs are fed by name, so the same code works for every language repo. moonshine_timestamps.py adds window splitting, a repetition guard and logit masking.

import numpy as np, onnxruntime as ort, soundfile as sf, librosa

d = "moonshine-attention-ort-ja"
a, sr = sf.read(f"{d}/test.wav", dtype="float32", always_2d=True)
audio = librosa.resample(a.mean(1), orig_sr=sr, target_sr=16000)       # 16 kHz mono, at most ~30 s

cpu = ["CPUExecutionProvider"]
enc = ort.InferenceSession(f"{d}/encoder_model.ort", providers=cpu)
dec = ort.InferenceSession(f"{d}/decoder_with_attention.ort", providers=cpu)
feed = lambda sess, avail: {i.name: avail[i.name] for i in sess.get_inputs()}  # only declared inputs

x = audio[None, :].astype(np.float32)                                  # [1, samples]
hidden = enc.run(None, feed(enc, {"input_values": x,
                                  "attention_mask": np.ones(x.shape, np.int64)}))[0]  # [1, frames, 416]

past = {i.name: np.zeros((1, 8, 0, 52), np.float32)
        for i in dec.get_inputs() if i.name.startswith("past_key_values.")}
names = [o.name for o in dec.get_outputs()]
tokens, attn = [1], []                                                 # 1 = <s>
for step in range(int(np.ceil(x.shape[1] / 16000 * 13)) + 10):
    out = dict(zip(names, dec.run(names, feed(dec, {
         "input_ids": np.array([[tokens[-1]]], np.int64),
         "encoder_hidden_states": hidden,
         "encoder_attention_mask": np.ones(x.shape, np.int64),   # sample-level mask
         "use_cache_branch": np.array([step > 0]),
         **past}))))
    tokens.append(int(out["logits"][0, -1].argmax()))
    attn.append(np.stack([out[f"cross_attentions.{l}"][0, :, -1] for l in range(8)]))  # [8, 8, frames]
    if tokens[-1] == 2:                                                # 2 = </s>
        break
    for n in past:                                                     # encoder KV: first call only
        if step == 0 or ".decoder." in n:
            past[n] = out[n.replace("past_key_values", "present")]

# word timings from attn, with the helpers in moonshine_timestamps.py
import sys; sys.path.insert(0, d)
from moonshine_timestamps import Tokens, token_spans, group
text_ids = tokens[1:-1]                                                # drop <s> and </s>
spans = token_spans(np.stack(attn), len(text_ids), x.shape[1] / 16000 / hidden.shape[1])
print(group(Tokens(f"{d}/tokens.txt"), text_ids, spans, "chars"))      # [{"text": ..., "timestamp": [s, e]}, ...]

Android / Java usage

Add ONNX Runtime for Android (the full package loads .ort files; the reference app uses 1.23.2):

implementation 'com.microsoft.onnxruntime:onnxruntime-android:1.23.2'

Ship the three model files in assets/ (or a Play Asset Delivery pack) and create the sessions. ONNX Runtime accepts a byte[] or a file path; for ~140 MB of models, copying the asset to filesDir once and loading by path avoids holding a second copy in the Java heap.

import ai.onnxruntime.*;
import java.nio.*;
import java.util.*;

OrtEnvironment env = OrtEnvironment.getEnvironment();
OrtSession.SessionOptions opts = new OrtSession.SessionOptions();
opts.setIntraOpNumThreads(4);
// copyAssetToFile(): your helper that copies an asset to filesDir once and returns its absolute path
OrtSession encoder = env.createSession(copyAssetToFile(ctx, "moonshine-ja/encoder_model.ort"), opts);
OrtSession decoder = env.createSession(copyAssetToFile(ctx, "moonshine-ja/decoder_with_attention.ort"), opts);
boolean encNeedsMask = encoder.getInputNames().contains("attention_mask");
boolean decNeedsMask = decoder.getInputNames().contains("encoder_attention_mask");

Decode loop (greedy, KV cache, collecting cross-attention). ONNX Runtime Java does not free native memory on garbage collection, so every OnnxTensor and OrtSession.Result must be closed. The KV-cache tensors belong to the Result that produced them: the first step's Result holds the encoder KV and stays open until the end, and each later Result can be closed once the next step has replaced its decoder KV.

float[] pcm = ...;                                     // 16 kHz mono, [-1, 1], at most ~30 s
long n = pcm.length;
long[] onesArr = new long[(int) n];
Arrays.fill(onesArr, 1L);
OnnxTensor x = OnnxTensor.createTensor(env, FloatBuffer.wrap(pcm), new long[]{1, n});
OnnxTensor ones = OnnxTensor.createTensor(env, LongBuffer.wrap(onesArr), new long[]{1, n});
Map<String, OnnxTensor> encIn = new HashMap<>();
encIn.put("input_values", x);
if (encNeedsMask) encIn.put("attention_mask", ones);
OrtSession.Result encOut = encoder.run(encIn);
OnnxTensor hidden = (OnnxTensor) encOut.get("last_hidden_state").get();

OnnxTensor emptyKv = OnnxTensor.createTensor(env, FloatBuffer.allocate(0), new long[]{1, 8, 0, 52});
Map<String, OnnxTensor> past = new HashMap<>();
for (String name : decoder.getInputNames())
    if (name.startsWith("past_key_values.")) past.put(name, emptyKv);

List<Integer> tokens = new ArrayList<>();
tokens.add(1);                                         // <s>
List<float[][][]> attn = new ArrayList<>();            // per step: [layer][head][frame]
int maxNew = (int) Math.ceil(n / 16000.0 * 13) + 10;
OrtSession.Result first = null, prev = null;           // first: encoder KV; prev: current decoder KV
for (int step = 0; step < maxNew; step++) {
    long last = tokens.get(tokens.size() - 1);
    ByteBuffer flagBuf = ByteBuffer.allocateDirect(1).put(0, (byte) (step > 0 ? 1 : 0));
    OrtSession.Result out;
    try (OnnxTensor ids = OnnxTensor.createTensor(env, new long[][]{ {last} });
         OnnxTensor flag = OnnxTensor.createTensor(env, flagBuf, new long[]{1}, OnnxJavaType.BOOL)) {
        Map<String, OnnxTensor> in = new HashMap<>(past);
        in.put("input_ids", ids);
        in.put("use_cache_branch", flag);
        in.put("encoder_hidden_states", hidden);
        if (decNeedsMask) in.put("encoder_attention_mask", ones);   // sample-level mask, NOT frames
        out = decoder.run(in);
    }
    float[][][] logits = (float[][][]) out.get("logits").get().getValue();
    int next = argmax(logits[0][logits[0].length - 1]);   // argmax(): index of the largest value
    float[][][] stepAttn = new float[8][8][];
    for (int l = 0; l < 8; l++) {
        float[][][][] a = (float[][][][]) out.get("cross_attentions." + l).get().getValue();
        for (int h = 0; h < 8; h++) stepAttn[l][h] = a[0][h][a[0][h].length - 1];  // newest token row
    }
    attn.add(stepAttn);
    tokens.add(next);
    if (next == 2) { out.close(); break; }             // </s>
    for (String name : past.keySet())
        if (step == 0 || name.contains(".decoder."))   // encoder KV only from the first call
            past.put(name, (OnnxTensor) out.get(name.replace("past_key_values", "present")).get());
    if (step == 0) first = out;
    else if (prev != first) prev.close();              // its decoder KV was just replaced
    prev = out;
}
if (prev != null && prev != first) prev.close();
if (first != null) first.close();
encOut.close(); x.close(); ones.close(); emptyKv.close();
// attn + tokens -> DTW word timestamps (same steps as described above)

The Sub4Video AI Android app uses these models. Its implementation (MoonshineManager, MoonshineTokenizer, MoonshineDTWAligner) is not publicly available. What it does: FFmpeg cuts the audio into 8 s chunks; each call is limited to 30 s; the token budget is min(616, ceil(duration × 25) + 16); the decoder gets the sample-level mask; alignment follows --method moonshine (a Java port of Moonshine's core/word-alignment.cpp); and text in CJK scripts (Han, Hiragana, Katakana) is split into per-character segments. moonshine_timestamps.py --method moonshine --tokens-per-sec 25 --max-chunk-sec 8 comes close to the app's output; the script's defaults (--method whisper, 13 tokens/s, 20 for Vietnamese, 30 s windows) give somewhat different timestamps.

Benchmarks

Setup. FLEURS test split (google/fleurs), first 100 utterances, reference = transcription. Every utterance is cut into 8 s segments (a tail shorter than 1 s is merged, so segments are at most 9 s), as the Android app does. Greedy decoding with token budget ceil(segment_s × 13) + 10 (13 tokens/s), no repetition guard. Normalisation: NFKC, lowercase, Unicode punctuation (P*) removed, whitespace collapsed; WER for en/es/uk/vi, CER without spaces for ja/zh; corpus-level. ONNX Runtime 1.24 CPU execution provider, ORT_ENABLE_ALL unless stated.

Equivalence with the parent

Compared with	Identical transcripts (100 utts)	Logit max abs diff (first step / whole decode)
Parent `sherpa-onnx-moonshine-base-ja-quantized-2026-02-27`	100/100	0.0 / 0.0
This repo with vs. without fetching `cross_attentions.*`	100/100	–

encoder_model.ort and tokens.txt are byte-identical to the parent (sha256).

Accuracy (CER, 5,049 reference characters)

Model	CER %	Utterances with a segment that ran to the token budget
Float reference (transformers `UsefulSensors/moonshine-base-ja`, fp32)	28.60	16
Parent (`sherpa-onnx-moonshine-base-ja-quantized-2026-02-27`)	30.92	18
This repo	30.92	18

This repo and its parent produce identical transcripts, so their scores are identical. Absolute scores depend on the token cap. The segments that ran to the 13 tokens/s cap were repetition loops or other runaway text (for example hallucinated continuations on short tails), not speech cut short. With the Android app's budget of 25 tokens/s the scores get worse because runaway text continues longer (ja 43.22 % CER instead of 30.92 %, vi 29.35 % WER instead of 23.13 %, zh 24.29 % CER instead of 20.18 %).

The quantized files are the same checkpoint as the float reference, quantized. The float-vs-quantized differences come mostly from a few chunk-level failures on the 8 s cuts (an early </s> or a loop in one of the models); they do not show that quantization improves or harms accuracy. English decoded without segmentation scores 12.21 % WER quantized and 12.92 % float, and excluding utterances where either model hit the cap, float and quantized differ by at most 1.4 points in every language.

Timestamp accuracy (measured). There is no word-level ground truth in this benchmark. On test.wav repeated with 0.4 s pauses, the first word after each pause (国) started on average 382 ms early without --trim-silence and 3 ms late with it (n = 1). Monotonic, in-range timestamps hold by construction (the DTW path is monotonic and clipped to the window), so they say nothing about alignment quality.

For Japanese and Chinese many tokens share the same start frame, so token-level alignment is weaker than for the space-separated languages.

Attention output size. Each decode step returns 8 × 8 × frames float32 values = 256 × frames bytes (≈ 95 KB for an 8 s segment, ≈ 320 KB for 30 s). The largest total for one utterance in this benchmark was 20.4 MB.

Speed. Real-time factor (RTF) = (encoder + greedy decoding time) / audio duration; lower is faster. Intel Core i5-12400F, onnxruntime 1.24 CPU execution provider, 20 FLEURS utterances, 8 s segments, warm-up run, mean of 2 repetitions; at 4 threads one session ran at a time.

Threads	Parent	This repo, fetching `cross_attentions.*`	This repo, not fetching	This repo (not fetching) vs parent
1	0.0439	0.0473	0.0485	+10.5 %
4	0.0302	0.0328	0.0331	+9.6 %

Fetching the attention outputs costs about nothing: with and without fetching cross_attentions.* the RTF differs by less than ±3 %, which is within the run-to-run noise.
The patched decoder is slower than its parent even when the attention outputs are not fetched: here +10.5 % at 1 thread and +9.6 % at 4 threads (5–10 % in most measurements across the five languages with a sherpa-onnx parent). The likely cause, not verified, is that the .ort file was re-serialised with graph optimisations disabled, dropping optimisations the parent .ort carried.

Limitations

Quantized weights. int8/uint8 quantization (DynamicQuantizeLinear + MatMulIntegerToFloat/DynamicQuantizeMatMul) adds small numerical differences from the float model; in the benchmark, the float-vs-quantized score differences came mostly from a few chunk-level failures (see Benchmarks). The attention maps come from the quantized model as well.
Window length. Keep each call ≤ 30 s (Moonshine's own transcriber uses VAD segments of at most 15 s). When FLEURS utterances were joined into longer windows, accuracy held up to 30 s, but above 30 s it got worse in four of six languages (words dropped or repeated). The script cuts longer files at the quietest point in the last 5 s before each 30 s limit; for real long-form audio, segment with a VAD at pauses.
Repetition loops and truncation. Greedy decoding can loop. The script stops when a pattern of 2–4 tokens repeats 4 times in a row (a single token 10 times), keeps one copy and marks the result "truncated": true; it does the same when the token budget runs out. Loops over longer phrases are not caught.
Quiet speech. Speech that is much quieter than other speech in the same window can be dropped, and the transcript can even come back empty (two FLEURS English utterances with peaks 0.003 and 0.75, joined into 16 s, gave no tokens; each alone decodes correctly). Normalise the level per speech segment (for example after VAD). --normalize scales each whole window, which helps quiet recordings but not this case.
Approximate timestamps. Cross-attention is not trained for alignment; boundaries come from DTW on attention averaged over all heads. The frame resolution is ≈ 24 ms, but pauses cause a larger error: DTW gives a pause to the word that follows it, so a word after a pause starts early (by about 382 ms on average for the repeated test.wav, and up to about 0.5 s across languages). --trim-silence moves each word to its voiced frames and brought this error within ±10 ms on clean speech; on noisy audio the energy threshold may not separate speech from background. The DTW path always starts at frame 0, so without trimming the first word absorbs any leading silence and starts at 0.0 s. Two tokens can map to the same frame, which gives a zero-length segment.
No word segmentation for Japanese. Timestamps are per character (see above). For word-level timing, run a word segmenter (for example MeCab/SudachiPy for Japanese, jieba for Chinese) and merge the character spans.
Greedy decoding only; no beam search, language detection or speaker labels. The model is monolingual (Japanese).

License and citation

The non-streaming Japanese Moonshine Base model is one of the models that Moonshine AI keeps under the Moonshine AI Community License (Section 2 of the included LICENSE, copied from moonshine-ai/moonshine; see also moonshine.ai/license). This repository is a Derivative Work and is distributed under the same licence. In short:

research and non-commercial use is free;
commercial use requires registering at moonshine.ai/community-license;
organisations (with affiliates) above US$1,000,000 annual revenue have no licence under these terms, for any use (Section III: all licences granted terminate), and need an enterprise licence (moonshine.ai/enterprise);
use restrictions apply (Section IV(b)): Moonshine AI's Acceptable Use Policy, and no use of the models or their outputs to create or improve foundational generative AI models;
if you redistribute these files or a product that uses them, you must pass on a copy of the licence, keep the attribution in NOTICE, and display "Powered by Moonshine AI".

If you use this model, please cite the Moonshine and Flavors of Moonshine papers:

@misc{jeffries2024moonshinespeechrecognitionlive,
      title={Moonshine: Speech Recognition for Live Transcription and Voice Commands},
      author={Nat Jeffries and Evan King and Manjunath Kudlur and Guy Nicholson and James Wang and Pete Warden},
      year={2024},
      eprint={2410.15608},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2410.15608},
}

@misc{king2025flavorsofmoonshine,
      title={Flavors of Moonshine: Tiny Specialized ASR Models for Edge Devices},
      author={Evan King and Adam Sabra and Manjunath Kudlur and James Wang and Pete Warden},
      year={2025},
      eprint={2509.02523},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.02523},
}

Acknowledgements

Useful Sensors / Moonshine AI for the Moonshine models and the reference word-alignment code (core/word-alignment.cpp).
k2-fsa/sherpa-onnx for the packaged ONNX Runtime exports, tokens.txt and the test clips.
The DTW alignment follows the cross-attention approach popularised by OpenAI Whisper's word timestamps.

Downloads last month: -

Model tree for ipsilondev/moonshine-attention-ort-ja

Base model

moonshine-ai/moonshine-base-ja

Quantized

(6)

this model

Papers for ipsilondev/moonshine-attention-ort-ja

Flavors of Moonshine: Tiny Specialized ASR Models for Edge Devices

Paper • 2509.02523 • Published Sep 2, 2025 • 22

Moonshine: Speech Recognition for Live Transcription and Voice Commands

Paper • 2410.15608 • Published Oct 21, 2024 • 14

Moonshine Base Japanese: ONNX Runtime (.ort) with cross-attention outputs for word timestamps