ksingla025's picture
Model card: final results (full test sets, same scoring as the Arc Audio report)
6f683b8 verified
|
Raw History Blame Contribute Delete
9.4 kB
---
license: cc-by-nc-4.0
language:
- en
library_name: arc-transcriber
pipeline_tag: automatic-speech-recognition
base_model: nvidia/stt_en_fastconformer_hybrid_large_streaming_multi
tags:
- speech-recognition
- streaming
- named-entity-recognition
- entity-linking
- space
- moon
- onnx
---
# Arc Transcriber Space
Arc Transcriber Space is a streaming English speech model for space and Moon-exploration apps. In a single pass it gives:
- **Transcript words**, from a streaming FastConformer encoder with an RNNT decoder.
- **Inline entity tags** on the words: lunar features, sites, celestial bodies, spacecraft, missions, crew, numbers and
more. Each tag is checked by a span head.
- **Name linking to the app's own list** (`names`), so "show me tycho" returns your `Tycho` entry.
- **Long context**: a names list, previous text and session memory.
- **Audio events and device-directed speech**.
The encoder is an int8 ONNX graph (cache-aware streaming, 560 ms chunks). It runs on a CPU.
## Results
Final numbers from the [Arc Audio report](https://whissleai.github.io/Outpost-Luna/arc-audio.html), which has the full
details: how the test sets were built, how leaks were removed, and per-tag scores.
**Test data.** The main benchmark is a NASA speech corpus collected by Team Outpost Luna from NASA public audio
(Artemis audio and the podcasts Houston We Have a Podcast, Small Steps, Giant Leaps and Curious Universe). **This model
was not trained on any of it**: it is a held-out test set here, and its training split is reserved for the next model.
The podcast test set is 19 whole episodes (12.8 h, 10,410 segments); the Artemis test set is 15 whole recordings (1,141
utterances) kept out of every training set. Every test set is scored in full.
### Word error rate
Streaming, against the NVIDIA FastConformer this model started from
(`nvidia/stt_en_fastconformer_hybrid_large_streaming_multi`). Both models run streaming and are scored with the same text
normalisation. Lower is better.
| Test set | Size | NVIDIA base | Arc Transcriber Space |
|---|---|---|---|
| **NASA speech corpus (held out)** | | | |
| Houston We Have a Podcast | 11 episodes | 10.0% | 7.2% |
| Small Steps, Giant Leaps | 5 episodes | 9.7% | 6.4% |
| Curious Universe | 3 episodes | 6.0% | 4.8% |
| All podcast parts | 19 episodes, 12.8 h | 9.7% | 7.0% |
| Artemis audio | 1,141 utterances | 28.5% | 12.6% |
| **Other test sets** | | | |
| Apollo mission audio | 500 utterances | 35.5% | 15.7% |
| Open read speech (children's corpus) | 300 utterances | 42.0% | 18.5% |
| Space questions (synthetic speech) | 300 utterances | 48.1% | 9.1% |
| Common Voice 17 English | 2,000 clips | 12.6% | 17.1% |
| CREMA-D (acted adult speech) | 600 clips | 3.7% | 1.9% |
On Common Voice the base model is better (12.6% vs 17.1%): tuning for space speech costs some everyday dictation. Every
other test set improves.
### Names: hearing and linking them
The app's own path: the int8 ONNX encoder on a CPU, full test sets. Each utterance gets a names list made of its own
names plus 20 similar-sounding decoys. On sets with no names, any link is a false link. The base model cannot link names.
| Test set | Name recall | Names linked correctly | Linked to a wrong entry | Space-tag F1 |
|---|---|---|---|---|
| Artemis II crew audio (1,141 utterances) | 93.4% | 94.5% | 1.0% | 0.88 |
| Space questions, synthetic speech (300) | 82.8% | 91.4% | 0.0% | 0.83 |
| Open read speech, children's corpus (300) | – | 0.67 false links per 100 utterances (no names in this set) | – | – |
| Apollo mission audio (500) | – | 0.4 false links per 100 utterances (no gold names in this set) | – | – |
Inline tag precision / recall / F1 (gold tag count in brackets):
| Tag | Artemis II crew audio (1,141) | Space questions (300) |
|---|---|---|
| ENTITY_LUNAR | .88 / .88 / .88 (584) | .88 / .81 / .84 (124) |
| ENTITY_SPACECRAFT | .86 / 1 / .92 (6) | .83 / .83 / .83 (29) |
| ENTITY_MISSION | – | .81 / .79 / .80 (43) |
| ENTITY_CELESTIAL | .92 / .97 / .94 (34) | 1 / 1 / 1 (2) |
| ENTITY_TERM | .82 / .83 / .82 (115) | 1 / 1 / 1 (2) |
| ENTITY_PERSON | .22 / .94 / .36 (47) | .13 / 1 / .22 (1) |
| NUMBER | .51 / .67 / .58 (162) | .88 / 1 / .93 (7) |
| NUMBER_PERCENT | 0 / 0 / 0 (5) | 1 / 1 / 1 (13) |
| NUMBER_TIME | .18 / .06 / .09 (36) | – |
| NUMBER_MEASURE | 0 / 0 / 0 (14) | 0 / 0 / 0 (1) |
| All space name tags | .88 / .88 / .88 (624) | .86 / .81 / .83 (198) |
Weak spots: ENTITY_PERSON fires too often (low precision), and number kinds (time, measure) are rarely right. The four
space name tags (lunar, spacecraft, mission, celestial), the ones linked to the app's names lists, are the strong ones.
**Overlap with training.** Common Voice and CREMA-D are test only. Open read speech uses the corpus's own split (trained on
its train split only). The space questions and Apollo test utterances are held out. Artemis II crew audio (a separate
training source from the NASA corpus above) is split by whole recording: other recordings are in training, the 15 test
recordings never are.
### Speed
On CPU with 4 threads, the encoder and RNNT run at about 0.13× real time with a peak of about 2.3 GB RAM, model
included.
## Install
The code is in the [arc_transcriber](https://github.com/WhissleAI/arc_transcriber) package; use the `space` extra.
```bash
pip install "arc-transcriber[space] @ git+https://github.com/WhissleAI/arc_transcriber"
```
A C library is also available (see `c/README.md` in the same repository). It uses the ONNX Runtime C API for the encoder:
```bash
git clone https://github.com/WhissleAI/arc_transcriber && cd arc_transcriber/c
make ARC_ORT=/path/to/onnxruntime # builds libarc + arc-nft2
```
## Use
```python
import soundfile as sf
from arc_audio import Arc
arc = Arc.from_pretrained("WhissleAI/arc-transcriber-space", backbone="nft2", device="cpu", threads=4)
NAMES = [ # what is on screen / in the current lesson, most relevant first
{"name": "Tycho", "kind": "lunar_feature"},
{"name": "Apollo 12", "kind": "mission"},
{"name": "Ocean of Storms", "kind": "lunar_feature", "aliases": ["Oceanus Procellarum"]},
{"name": "Orion", "kind": "spacecraft"},
{"name": "Artemis II", "kind": "mission", "aliases": ["Artemis 2"]},
]
wav, sr = sf.read("examples/samples/sample_2.wav", dtype="float32") # 16 kHz mono
out = arc.predict(wav, domain="assistant", transcript="nft2", names=NAMES)
print(out["transcript"][0]["text"]) # "Show me the Apollo twelve landing site in the ocean of storms"
for n in out["names"]:
print(n["heard"], "->", n["entity"], n["state"]) # "Apollo" -> "Apollo 12" linked ; "ocean of storms" -> "Ocean of Storms" linked
# streaming (microphone): feed 16 kHz float32 chunks, change the list at any time
s = arc.stream(domain="assistant", transcript="nft2", names=NAMES)
for chunk in mic_chunks(): # e.g. 100 ms each
for update in filter(None, [s.feed(chunk)]):
... # partial / final transcript, names, events
s.set_context(names=NEW_SCREEN_NAMES, previous_text="...")
final = s.flush()
```
Each item in `out["names"]` has these fields:
- `tag`: the entity type, e.g. `entity.mention.lunar_feature`
- `heard`: the words in spoken form, e.g. `apollo twelve`
- `entity`: your list entry's `name`, e.g. `Apollo 12`
- `state`: one of
- `linked`: act on `entity`
- `abstain`: ambiguous or generic, so ask the user
- `not_in_list`: a name the user said that is not in your list
- `p`: confidence
## Names lists: read this before integrating
Linking compares normalized text:
- lower case;
- digits become spoken words: `Apollo 12` → `apollo twelve`, `1969` → `nineteen sixty nine`;
- `-` becomes a space;
- **accented and other non-ASCII letters are dropped**.
Rules for building lists:
- Always give a `kind`.
- Put the most relevant names first.
- Keep lists short.
- Add aliases for names that are not said the way they are written: `Artemis II` → `"Artemis 2"`, `S-IVB` → `"S 4 B"`.
- Add ASCII aliases for non-ASCII names: `Schrödinger` → `"Schrodinger"`.
The full guide is in [NAMES_LISTS.md](NAMES_LISTS.md).
The transcript is in spoken form ("apollo twelve"). For display and routing, use `names[i].entity`, which is your
list's own spelling.
## Notes
- Input is 16 kHz mono audio.
- **The model is built not to transcribe the app's own spoken answers**: text-to-speech playback leaking into the
microphone. Synthetic TTS voices, especially female Kokoro-style voices, can therefore be partly or fully ignored. Test
with real voices, not TTS. Real children's speech is transcribed normally.
- The `examples/samples/` clips are synthetic speech made with Kokoro-82M (Apache-2.0). They contain no recording of a
real person.
## Licence
© 2026 Whissle Inc. The model weights are licensed under CC BY-NC 4.0 together with the Whissle terms in
[LICENSE](LICENSE), for non-commercial use only. Commercial licences: ksingla@whissle.ai, https://whissle.ai.
Third-party components are listed in [NOTICE](NOTICE):
- a modified NVIDIA FastConformer (`nvidia/stt_en_fastconformer_hybrid_large_streaming_multi`, CC-BY-4.0);
- a modified Moonshine Streaming decoder (MIT; its licence is kept in `nft2/decoder/LICENSE`).
This model is not affiliated with or endorsed by NASA or NVIDIA.