Moonshine Base Spanish: ONNX Runtime (`.ort`) with cross-attention outputs for word timestamps

Powered by Moonshine AI. This Moonshine AI Model is licensed under the Moonshine AI Community License, Copyright © Moonshine AI Ltd. All Rights Reserved. It is not MIT: research and non-commercial use is free; commercial use requires registering with Moonshine AI; organisations (with affiliates) above US$1M annual revenue have no licence under these terms and need an enterprise licence. Use restrictions apply. See LICENSE, NOTICE and License.

This repository contains Moonshine Base, a small (≈61 M parameters) encoder-decoder speech recognition model for Spanish, as quantized ONNX Runtime .ort files for on-device use (Android, desktop CPU). Its decoder also returns its cross-attention weights (cross_attentions.0 … cross_attentions.7, one per decoder layer).

Relation to Moonshine AI's files. Moonshine AI publishes an attention-enabled decoder only for English (base-en/…/decoder_with_attention.ort). This repo provides the same thing for Spanish: the same file name and the same cross_attentions.{i} outputs that Moonshine's runtime reads from the English file.

Why: the cross-attention maps show which part of the audio each generated token attended to. Aligning them with dynamic time warping (DTW) gives word-level timestamps without a separate alignment model and without running the decoder a second time. The attention values are computed anyway, and fetching them costs nothing measurable (see Speed under Benchmarks). This is the same idea as Whisper's word timestamps and Moonshine's own core/word-alignment.cpp.

Provenance and what was changed


Original model	`UsefulSensors/moonshine-es` (Moonshine Base, Spanish) by Useful Sensors / Moonshine AI
Direct parent	`UsefulSensors/moonshine-es` → `onnx/merged/base/quantized/{encoder_model,decoder_model_merged}.onnx`, converted to `.ort` with ONNX Runtime graph optimisations disabled
`encoder_model.ort`	parent encoder, converted to `.ort`
`decoder_with_attention.ort`	parent merged decoder, converted to `.ort`, + 8 extra outputs (below)
`tokens.txt`	byte-identical to `tokens.txt` of `sherpa-onnx-moonshine-base-es-quantized-2026-02-27` (k2-fsa/sherpa-onnx)
`test.wav`	`test_wavs/0.wav` from the sherpa-onnx archive, which sherpa-onnx downloads from `https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/es.wav`; its licence is not stated

Quantization (Spanish only). The MatMul weights are uint8 with dynamic activation quantization, using standard ONNX operators: DynamicQuantizeLinear + MatMulInteger (+ Cast/Mul rescaling), ONNX opset 14. The encoder's 3 convolution layers use int8 weight-only quantization (dequantized to float, then a float Conv). This is not the quantization of the sherpa-onnx Spanish package (sherpa-onnx-moonshine-base-es-quantized-2026-02-27, whose .ort files are identical to UsefulSensors/moonshine-es → onnx/merged/base/float/*.ort): that package dequantizes int8 weights through Cast/Mul/Add and multiplies in float.

What was changed (decoder). The merged decoder has one If node (use_cache_branch) with a no-cache branch and a with-cache branch. In both branches, an ONNX Identity node was attached to the output of each layer's cross-attention softmax (/model/decoder/layers.{i}/encoder_attn/Softmax_output_0, i = 0…7). These are exposed as new graph outputs cross_attentions.0 … cross_attentions.7 (16 Identity nodes in total). The graph was loaded and re-saved with ONNX Runtime graph optimisations disabled (ORT_DISABLE_ALL) so that the quantized operators are kept exactly as they were.

What was not changed. Weights, quantization, every pre-existing input and output, and the tokenizer. The attention probabilities were already computed internally; they are now also returned.

Equivalence, stated precisely. These .ort files are bit-identical in behaviour to the parent ONNX files only when the parent runs with ONNX Runtime graph optimisations disabled (that is how the .ort files were saved): same tokens on 100/100 FLEURS utterances, encoder output and decoder logits identical (max abs difference 0.0). Loaded with the default ORT_ENABLE_ALL, the parent ONNX gives slightly different numbers (ORT's optimiser perturbs the quantized encoder output by about 5 % relative; 86/100 identical transcripts) and runs 30–48 % faster than this .ort; see Speed under Benchmarks.

Files

File	Size	Description
`encoder_model.ort`	21.0 MB	Encoder: raw 16 kHz waveform → hidden states (ONNX Runtime format, quantized)
`decoder_with_attention.ort`	43.8 MB	Merged decoder with KV cache (`use_cache_branch`) + `cross_attentions.0-7` outputs
`tokens.txt`	0.5 MB	Vocabulary, 32,000 lines: `<base64 of the piece's UTF-8 bytes> <id>`
`test.wav`	0.2 MB	Sample clip (22050 Hz, 5.33 s; resample to 16 kHz before use)
`config.json`	–	Machine-readable model facts (dimensions, special ids, inputs, defaults)
`moonshine_timestamps.py`	–	Reference Python script: transcription + word timestamps (MIT)
`LICENSE`	–	Licence text (copied from moonshine-ai/moonshine)
`NOTICE`	–	Attribution and list of changes required by the Moonshine AI Community License

Model inputs and outputs

Encoder (encoder_model.ort)

Name	Type	Shape	Notes
in `input_values`	float32	`[batch, num_samples]`	raw mono waveform, 16 kHz, range [-1, 1]; no feature extraction. No `attention_mask` input
out `last_hidden_state`	float32	`[batch, frames, 416]`	`frames = ((((samples − 127) // 64 + 1 − 7) // 3 + 1 − 3) // 2 + 1)` ≈ 41.7 frames/s (one frame ≈ 24 ms)

Decoder (decoder_with_attention.ort): 8 layers, 8 heads, head size 52, d_model 416

Name	Type	Shape	Notes
in `input_ids`	int64	`[batch, dec_len]`	`[[1]]` (`<s>`) on the first call, then the previously generated token
in `encoder_hidden_states`	float32	`[batch, frames, 416]`	encoder output
in `past_key_values.{i}.decoder.key` / `.value`	float32	`[batch, 8, past_len, 52]`	self-attention cache, i = 0…7; zero-length (`past_len = 0`) on the first call
in `past_key_values.{i}.encoder.key` / `.value`	float32	`[batch, 8, frames, 52]` (dim named `encoder_sequence_length_out`)	cross-attention cache; zero-length on the first call, then the `present.{i}.encoder.` of the first* call
in `use_cache_branch`	bool	`[1]`	`false` on the first call, `true` afterwards
out `logits`	float32	`[batch, dec_len, 32768]`	greedy: `argmax(logits[0, -1])`
out `present.{i}.decoder.key` / `.value`	float32	`[batch, 8, past_len + 1, 52]` (as declared; one token per call)	feed back as `past_key_values.{i}.decoder.*`
out `present.{i}.encoder.key` / `.value`	float32	`[batch, 8, frames, 52]`	only meaningful when `use_cache_branch = false`; the with-cache branch returns a dummy `[0, 8, 1, 52]` tensor, so keep the first call's copy
out `cross_attentions.{i}` (new)	float32	`[batch, 8, dec_len, frames]`	softmax cross-attention probabilities of decoder layer i (declared fully dynamic); each row sums to 1 over `frames`

The decoder has 33 original outputs + 8 new ones = 41 outputs. Special token ids: 0 = <unk>, 1 = <s> (decoder start), 2 = </s> (end). No language or task prompt tokens are needed: the model is monolingual.

tokens.txt (sherpa-onnx format): one line per id, <base64> <id>, where the base64 string encodes the raw UTF-8 bytes of the SentencePiece piece. Byte-fallback pieces (ids 3–258) are stored as the single raw byte 0x00–0xFF they stand for (byte = id − 3). To detokenize: concatenate the bytes of all non-special tokens, decode the result as UTF-8 once at the end (a character may be split over several byte tokens), then replace ▁ (U+2581) with a space and strip. A piece that starts with ▁ starts a new word.

Vocabulary note (Spanish). tokens.txt has 32,000 entries while logits has 32,768 columns. Ids 32000–32767 are unused placeholder slots (in the other languages' tokens.txt they are spelled <<ST_0>> … <<ST_767>>); treat them as special tokens with no text. The script masks them before argmax.

How the timestamps are computed

The decoder is run greedily, one token per call with the KV cache. After each call, the attention row of the newest token, cross_attentions.{l}[0, :, -1, :], is stored for all 8 layers × 8 heads. Row k belongs to the step that produced token k (step 0 consumes <s>); the step that produced </s> is kept too. The rows form a tensor A[64 heads, steps, frames], and then:

Normalize: z-score every head over the token axis (--method moonshine instead z-scores every (head, token) row over the frame axis, as in Moonshine's core/word-alignment.cpp).
Smooth: median filter of width 7 along the frame axis.
Average the 64 heads, giving M[steps, frames].
DTW: compute the monotonic path through the cost −M from (first token, frame 0) to (last step, last frame).
Token times: a token starts at the frame where the path first enters its row and ends where the next token starts (the </s> row closes the last token). Seconds = frame × duration / frames (≈ 24 ms per frame). With --method moonshine a token spans the min…max+1 frames of its path cells.
Words: consecutive tokens are merged into a word at every piece that starts with ▁; overlaps are snapped to the midpoint.
Optional --trim-silence: each word is shrunk to the 10 ms frames inside it whose energy is above a threshold (95th percentile − 35 dB). DTW gives a pause to the word that follows it; trimming removes most of that error (see Limitations).

The output has the same shape as the Android app's JSON: {"text": ..., "chunks": [{"text": word, "timestamp": [start_s, end_s]}, ...]}, plus "truncated": true when a window ended without </s>.

Quickstart (Python)

Requirements: Python ≥ 3.9, numpy ≥ 1.20, onnxruntime (tested with 1.24.1), soundfile; librosa (or scipy) for resampling. The hf command needs huggingface_hub ≥ 0.34 (older versions: huggingface-cli download).

pip install onnxruntime numpy soundfile huggingface_hub librosa
hf download ipsilondev/moonshine-attention-ort-es --local-dir moonshine-attention-ort-es
python moonshine-attention-ort-es/moonshine_timestamps.py --model-dir moonshine-attention-ort-es moonshine-attention-ort-es/test.wav
python moonshine-attention-ort-es/moonshine_timestamps.py --model-dir moonshine-attention-ort-es your_audio.wav --trim-silence -o result.json

# script only; the model is downloaded to the Hugging Face cache on first use
hf download ipsilondev/moonshine-attention-ort-es moonshine_timestamps.py --local-dir .
python moonshine_timestamps.py --lang es your_audio.wav

Options:

--trim-silence: shrink each chunk to its voiced frames. Recommended for subtitles (see Limitations).
--method {whisper,moonshine}: alignment variant (see above). The Android app uses moonshine.
--tokens-per-sec: token budget per window, ceil(duration × tokens_per_sec) + 10 (default 13; 20 for Vietnamese).
--max-chunk-sec 30: longer files are split into windows of at most 30 s, each cut at the quietest point in the last 5 s before the limit.
--normalize: RMS-normalise each window to −20 dBFS (helps quiet recordings).
--threads N, -v (timings), -o file.json.

If a window ends without </s> (token budget used up, or a repetition loop that the script cut off), the JSON also contains "truncated": true and a warning is printed to stderr.

Output for test.wav (default settings; first 4 of 17 chunks shown; this is raw model output, including any recognition errors):

{
  "text": "No preguntes qué puede hacer tu país por ti. Pregunta qué puedes hacer qué por tu país.",
  "chunks": [
    {"text": "No", "timestamp": [0.097, 0.218]},
    {"text": "preguntes", "timestamp": [0.218, 0.751]},
    {"text": "qué", "timestamp": [0.751, 0.993]},
    {"text": "puede", "timestamp": [0.993, 1.235]}
  ]
}

Use from Python. The script is also a small library:

import sys; sys.path.insert(0, "moonshine-attention-ort-es")
from moonshine_timestamps import MoonshineAttention, load_audio, transcribe_long

model = MoonshineAttention("moonshine-attention-ort-es")
result = transcribe_long(model, load_audio("moonshine-attention-ort-es/test.wav"), lang="es", trim=True)
print(result["text"])
print(result["chunks"][:3])

The decoding loop, step by step. Inputs are fed by name, so the same code works for every language repo. moonshine_timestamps.py adds window splitting, a repetition guard and logit masking.

import numpy as np, onnxruntime as ort, soundfile as sf, librosa

d = "moonshine-attention-ort-es"
a, sr = sf.read(f"{d}/test.wav", dtype="float32", always_2d=True)
audio = librosa.resample(a.mean(1), orig_sr=sr, target_sr=16000)       # 16 kHz mono, at most ~30 s

cpu = ["CPUExecutionProvider"]
enc = ort.InferenceSession(f"{d}/encoder_model.ort", providers=cpu)
dec = ort.InferenceSession(f"{d}/decoder_with_attention.ort", providers=cpu)
feed = lambda sess, avail: {i.name: avail[i.name] for i in sess.get_inputs()}  # only declared inputs

x = audio[None, :].astype(np.float32)                                  # [1, samples]
# this ES encoder declares only input_values; feed() drops attention_mask
hidden = enc.run(None, feed(enc, {"input_values": x,
                                  "attention_mask": np.ones(x.shape, np.int64)}))[0]  # [1, frames, 416]

past = {i.name: np.zeros((1, 8, 0, 52), np.float32)
        for i in dec.get_inputs() if i.name.startswith("past_key_values.")}
names = [o.name for o in dec.get_outputs()]
tokens, attn = [1], []                                                 # 1 = <s>
for step in range(int(np.ceil(x.shape[1] / 16000 * 13)) + 10):
    out = dict(zip(names, dec.run(names, feed(dec, {
         "input_ids": np.array([[tokens[-1]]], np.int64),
         "encoder_hidden_states": hidden,
         "encoder_attention_mask": np.ones(x.shape, np.int64),   # (not declared by the ES graph)
         "use_cache_branch": np.array([step > 0]),
         **past}))))
    tokens.append(int(out["logits"][0, -1].argmax()))
    attn.append(np.stack([out[f"cross_attentions.{l}"][0, :, -1] for l in range(8)]))  # [8, 8, frames]
    if tokens[-1] == 2:                                                # 2 = </s>
        break
    for n in past:                                                     # encoder KV: first call only
        if step == 0 or ".decoder." in n:
            past[n] = out[n.replace("past_key_values", "present")]

# word timings from attn, with the helpers in moonshine_timestamps.py
import sys; sys.path.insert(0, d)
from moonshine_timestamps import Tokens, token_spans, group
text_ids = tokens[1:-1]                                                # drop <s> and </s>
spans = token_spans(np.stack(attn), len(text_ids), x.shape[1] / 16000 / hidden.shape[1])
print(group(Tokens(f"{d}/tokens.txt"), text_ids, spans, "words"))      # [{"text": ..., "timestamp": [s, e]}, ...]

Android / Java usage

Add ONNX Runtime for Android (the full package loads .ort files; the reference app uses 1.23.2):

implementation 'com.microsoft.onnxruntime:onnxruntime-android:1.23.2'

Ship the three model files in assets/ (or a Play Asset Delivery pack) and create the sessions. ONNX Runtime accepts a byte[] or a file path; for ~140 MB of models, copying the asset to filesDir once and loading by path avoids holding a second copy in the Java heap.

import ai.onnxruntime.*;
import java.nio.*;
import java.util.*;

OrtEnvironment env = OrtEnvironment.getEnvironment();
OrtSession.SessionOptions opts = new OrtSession.SessionOptions();
opts.setIntraOpNumThreads(4);
// copyAssetToFile(): your helper that copies an asset to filesDir once and returns its absolute path
OrtSession encoder = env.createSession(copyAssetToFile(ctx, "moonshine-es/encoder_model.ort"), opts);
OrtSession decoder = env.createSession(copyAssetToFile(ctx, "moonshine-es/decoder_with_attention.ort"), opts);
boolean encNeedsMask = encoder.getInputNames().contains("attention_mask");
boolean decNeedsMask = decoder.getInputNames().contains("encoder_attention_mask");

For this Spanish model encNeedsMask and decNeedsMask are both false (the graphs do not declare the masks); checking the declared input names keeps one code path for all six language repos.

Decode loop (greedy, KV cache, collecting cross-attention). ONNX Runtime Java does not free native memory on garbage collection, so every OnnxTensor and OrtSession.Result must be closed. The KV-cache tensors belong to the Result that produced them: the first step's Result holds the encoder KV and stays open until the end, and each later Result can be closed once the next step has replaced its decoder KV.

float[] pcm = ...;                                     // 16 kHz mono, [-1, 1], at most ~30 s
long n = pcm.length;
long[] onesArr = new long[(int) n];
Arrays.fill(onesArr, 1L);
OnnxTensor x = OnnxTensor.createTensor(env, FloatBuffer.wrap(pcm), new long[]{1, n});
OnnxTensor ones = OnnxTensor.createTensor(env, LongBuffer.wrap(onesArr), new long[]{1, n});
Map<String, OnnxTensor> encIn = new HashMap<>();
encIn.put("input_values", x);
if (encNeedsMask) encIn.put("attention_mask", ones);
OrtSession.Result encOut = encoder.run(encIn);
OnnxTensor hidden = (OnnxTensor) encOut.get("last_hidden_state").get();

OnnxTensor emptyKv = OnnxTensor.createTensor(env, FloatBuffer.allocate(0), new long[]{1, 8, 0, 52});
Map<String, OnnxTensor> past = new HashMap<>();
for (String name : decoder.getInputNames())
    if (name.startsWith("past_key_values.")) past.put(name, emptyKv);

List<Integer> tokens = new ArrayList<>();
tokens.add(1);                                         // <s>
List<float[][][]> attn = new ArrayList<>();            // per step: [layer][head][frame]
int maxNew = (int) Math.ceil(n / 16000.0 * 13) + 10;
OrtSession.Result first = null, prev = null;           // first: encoder KV; prev: current decoder KV
for (int step = 0; step < maxNew; step++) {
    long last = tokens.get(tokens.size() - 1);
    ByteBuffer flagBuf = ByteBuffer.allocateDirect(1).put(0, (byte) (step > 0 ? 1 : 0));
    OrtSession.Result out;
    try (OnnxTensor ids = OnnxTensor.createTensor(env, new long[][]{ {last} });
         OnnxTensor flag = OnnxTensor.createTensor(env, flagBuf, new long[]{1}, OnnxJavaType.BOOL)) {
        Map<String, OnnxTensor> in = new HashMap<>(past);
        in.put("input_ids", ids);
        in.put("use_cache_branch", flag);
        in.put("encoder_hidden_states", hidden);
        if (decNeedsMask) in.put("encoder_attention_mask", ones);   // sample-level mask, NOT frames
        out = decoder.run(in);
    }
    float[][][] logits = (float[][][]) out.get("logits").get().getValue();
    int next = argmax(logits[0][logits[0].length - 1]);   // argmax(): index of the largest value
    float[][][] stepAttn = new float[8][8][];
    for (int l = 0; l < 8; l++) {
        float[][][][] a = (float[][][][]) out.get("cross_attentions." + l).get().getValue();
        for (int h = 0; h < 8; h++) stepAttn[l][h] = a[0][h][a[0][h].length - 1];  // newest token row
    }
    attn.add(stepAttn);
    tokens.add(next);
    if (next == 2) { out.close(); break; }             // </s>
    for (String name : past.keySet())
        if (step == 0 || name.contains(".decoder."))   // encoder KV only from the first call
            past.put(name, (OnnxTensor) out.get(name.replace("past_key_values", "present")).get());
    if (step == 0) first = out;
    else if (prev != first) prev.close();              // its decoder KV was just replaced
    prev = out;
}
if (prev != null && prev != first) prev.close();
if (first != null) first.close();
encOut.close(); x.close(); ones.close(); emptyKv.close();
// attn + tokens -> DTW word timestamps (same steps as described above)

The Sub4Video AI Android app uses these models. Its implementation (MoonshineManager, MoonshineTokenizer, MoonshineDTWAligner) is not publicly available. What it does: FFmpeg cuts the audio into 8 s chunks; each call is limited to 30 s; the token budget is min(616, ceil(duration × 25) + 16); the decoder gets the sample-level mask; alignment follows --method moonshine (a Java port of Moonshine's core/word-alignment.cpp); and text in CJK scripts (Han, Hiragana, Katakana) is split into per-character segments. moonshine_timestamps.py --method moonshine --tokens-per-sec 25 --max-chunk-sec 8 comes close to the app's output; the script's defaults (--method whisper, 13 tokens/s, 20 for Vietnamese, 30 s windows) give somewhat different timestamps.

Benchmarks

Setup. FLEURS test split (google/fleurs), first 100 utterances, reference = transcription. Every utterance is cut into 8 s segments (a tail shorter than 1 s is merged, so segments are at most 9 s), as the Android app does. Greedy decoding with token budget ceil(segment_s × 13) + 10 (13 tokens/s), no repetition guard. Normalisation: NFKC, lowercase, Unicode punctuation (P*) removed, whitespace collapsed; WER for en/es/uk/vi, CER without spaces for ja/zh; corpus-level. ONNX Runtime 1.24 CPU execution provider, ORT_ENABLE_ALL unless stated.

Equivalence with the parent

Compared with	Identical transcripts (100 utts)	Logit max abs diff
Parent quantized ONNX, ORT optimisations disabled	100/100	0.0 (encoder output also identical)
Parent quantized ONNX, `ORT_ENABLE_ALL` (default)	86/100	1.51 at the first step (the optimiser perturbs the encoder output by up to 1.91)
sherpa-onnx ES (other quantization)	84/100	1.34 at the first step

Accuracy (WER, 2,517 reference words)

Model	WER %	Utterances with a segment that ran to the token budget
Float reference (`UsefulSensors/moonshine-es` `onnx/merged/base/float/*.onnx`)	7.23	1
Parent quantized ONNX, `ORT_ENABLE_ALL`	9.26	4
Parent quantized ONNX, optimisations disabled	8.90	3
This repo	8.90	3
sherpa-onnx ES (`sherpa-onnx-moonshine-base-es-quantized-2026-02-27`, other quantization)	8.46	–

This repo and the parent with optimisations disabled produce identical transcripts, so their scores are identical. Absolute scores depend on the token cap. The segments that ran to the 13 tokens/s cap were repetition loops or other runaway text (for example hallucinated continuations on short tails), not speech cut short. The cap matters most for Japanese, Vietnamese and Chinese: at 25 tokens/s (the Android app's budget) their scores get worse because runaway text continues longer (ja 43.22 % CER instead of 30.92 %, vi 29.35 % WER instead of 23.13 %, zh 24.29 % CER instead of 20.18 %).

The quantized files are the same checkpoint as the float reference, quantized. The float-vs-quantized differences come mostly from a few chunk-level failures on the 8 s cuts (an early </s> or a loop in one of the models); they do not show that quantization improves or harms accuracy. English decoded without segmentation scores 12.21 % WER quantized and 12.92 % float, and excluding utterances where either model hit the cap, float and quantized differ by at most 1.4 points in every language.

Timestamp accuracy (measured). There is no word-level ground truth in this benchmark. On test.wav repeated with 0.4 s pauses, the first word after each pause (No) started on average 94 ms early without --trim-silence and 4 ms late with it (n = 3). Monotonic, in-range timestamps hold by construction (the DTW path is monotonic and clipped to the window), so they say nothing about alignment quality.

Attention output size. Each decode step returns 8 × 8 × frames float32 values = 256 × frames bytes (≈ 95 KB for an 8 s segment, ≈ 320 KB for 30 s). The largest total for one utterance in this benchmark was 12.5 MB.

Speed. Real-time factor (RTF) = (encoder + greedy decoding time) / audio duration; lower is faster. Intel Core i5-12400F, onnxruntime 1.24 CPU execution provider, 20 FLEURS utterances, 8 s segments, warm-up run, mean of 2 repetitions; at 4 threads one session ran at a time.

Threads	Parent ONNX, `ORT_ENABLE_ALL`	Parent ONNX, optimisations disabled	This repo, fetching `cross_attentions.*`	This repo, not fetching	This repo (not fetching) vs parent `ORT_ENABLE_ALL`
1	0.0504	0.0747	0.0749	0.0746	+48.0 %
4	0.0392	0.0511	0.0513	0.0511	+30.4 %

Fetching the attention outputs costs about nothing: with and without fetching cross_attentions.* the RTF differs by less than ±3 %, which is within the run-to-run noise.
This .ort runs at the speed of the unoptimised parent (it was saved with graph optimisations disabled so that it reproduces the parent exactly), which is 48 % slower at 1 thread and 30 % slower at 4 threads than the parent ONNX loaded with ORT_ENABLE_ALL.

Limitations

Quantized weights. int8/uint8 quantization (DynamicQuantizeLinear + MatMulInteger; int8 weight-only encoder convolutions) adds small numerical differences from the float model; in the benchmark, the float-vs-quantized score differences came mostly from a few chunk-level failures (see Benchmarks). The attention maps come from the quantized model as well.
Window length. Keep each call ≤ 30 s (Moonshine's own transcriber uses VAD segments of at most 15 s). When FLEURS utterances were joined into longer windows, accuracy held up to 30 s, but above 30 s it got worse in four of six languages (words dropped or repeated). The script cuts longer files at the quietest point in the last 5 s before each 30 s limit; for real long-form audio, segment with a VAD at pauses.
Repetition loops and truncation. Greedy decoding can loop. The script stops when a pattern of 2–4 tokens repeats 4 times in a row (a single token 10 times), keeps one copy and marks the result "truncated": true; it does the same when the token budget runs out. Loops over longer phrases are not caught.
Quiet speech. Speech that is much quieter than other speech in the same window can be dropped, and the transcript can even come back empty (two FLEURS English utterances with peaks 0.003 and 0.75, joined into 16 s, gave no tokens; each alone decodes correctly). Normalise the level per speech segment (for example after VAD). --normalize scales each whole window, which helps quiet recordings but not this case.
Approximate timestamps. Cross-attention is not trained for alignment; boundaries come from DTW on attention averaged over all heads. The frame resolution is ≈ 24 ms, but pauses cause a larger error: DTW gives a pause to the word that follows it, so a word after a pause starts early (by about 94 ms on average for the repeated test.wav, and up to about 0.5 s across languages). --trim-silence moves each word to its voiced frames and brought this error within ±10 ms on clean speech; on noisy audio the energy threshold may not separate speech from background. The DTW path always starts at frame 0, so without trimming the first word absorbs any leading silence and starts at 0.0 s. Two tokens can map to the same frame, which gives a zero-length segment.
Greedy decoding only; no beam search, language detection or speaker labels. The model is monolingual (Spanish).
Not the sherpa-onnx quantization. The Spanish files are not the sherpa-onnx Spanish export (see Provenance), so accuracy differs slightly from it; see Benchmarks.

License and citation

The non-streaming Spanish Moonshine Base model is one of the models that Moonshine AI keeps under the Moonshine AI Community License (Section 2 of the included LICENSE, copied from moonshine-ai/moonshine; see also moonshine.ai/license). This repository is a Derivative Work and is distributed under the same licence. In short:

research and non-commercial use is free;
commercial use requires registering at moonshine.ai/community-license;
organisations (with affiliates) above US$1,000,000 annual revenue have no licence under these terms, for any use (Section III: all licences granted terminate), and need an enterprise licence (moonshine.ai/enterprise);
use restrictions apply (Section IV(b)): Moonshine AI's Acceptable Use Policy, and no use of the models or their outputs to create or improve foundational generative AI models;
if you redistribute these files or a product that uses them, you must pass on a copy of the licence, keep the attribution in NOTICE, and display "Powered by Moonshine AI".

If you use this model, please cite the Moonshine paper:

@misc{jeffries2024moonshinespeechrecognitionlive,
      title={Moonshine: Speech Recognition for Live Transcription and Voice Commands},
      author={Nat Jeffries and Evan King and Manjunath Kudlur and Guy Nicholson and James Wang and Pete Warden},
      year={2024},
      eprint={2410.15608},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2410.15608},
}

Acknowledgements

Useful Sensors / Moonshine AI for the Moonshine models and the reference word-alignment code (core/word-alignment.cpp).
k2-fsa/sherpa-onnx for the packaged ONNX Runtime exports, tokens.txt and the test clips.
The DTW alignment follows the cross-attention approach popularised by OpenAI Whisper's word timestamps.

Downloads last month: 11

Model tree for ipsilondev/moonshine-attention-ort-es

Base model

moonshine-ai/moonshine-es

Quantized

(1)

this model

Paper for ipsilondev/moonshine-attention-ort-es

Moonshine: Speech Recognition for Live Transcription and Voice Commands

Paper • 2410.15608 • Published Oct 21, 2024 • 14

Moonshine Base Spanish: ONNX Runtime (.ort) with cross-attention outputs for word timestamps