SimulSense

Speech-to-text translation from English speech to Chinese, German, or Japanese text.

This release contains one SimulSense unit detector and three translation models. It accompanies the paper on IEEE Xplore. Speech synthesis and reinforcement learning models are outside the scope of this release.

Files

Directory	Contents
SimulSense/	Shared unit detector, configuration, and Whisper decoder adapter
translation/en-zh/	English → Chinese speech-to-text projection and Qwen adapter
translation/en-de/	English → German speech-to-text projection and Qwen adapter
translation/en-ja/	English → Japanese speech-to-text projection and Qwen adapter
simulsense-st-source.tar.gz	Matching source distribution, dependency lockfile, instructions, and licenses

WEIGHTS_MANIFEST.json records file sizes and SHA-256 checksums. WEIGHTS_VALIDATION.json records parameter/copy equality checks performed during export. Translation checkpoint selection labels in that report identify local checkpoints; they are not benchmark results.

Use

Download and extract the source archive, then follow its README:

hf download tan-ht/SimulSense simulsense-st-source.tar.gz --local-dir .
tar -xzf simulsense-st-source.tar.gz
cd simulsense-st
uv sync --frozen
uv run python scripts/download_models.py --target all
uv run python scripts/run.py evaluate \
  --target zh \
  --checkpoint checkpoints/translation/en-zh \
  --detector checkpoints/SimulSense \
  --source /path/to/source.txt --reference /path/to/target.txt \
  --output outputs/en-zh --device 0 --chunk-ms 2000 --threshold 1.0

source.txt contains one absolute mono 16 kHz WAV path per line, and target.txt contains the matching reference translations. Use the bundled streaming command for long continuous recordings. The source README also documents training and fixed/wait-k/VAD baselines.

These custom Fairseq + PEFT checkpoints require the provided source. They cannot be loaded directly with transformers.pipeline. The frozen Whisper large-v3 and Qwen3-8B backbones are downloaded separately. Linux, Python 3.10, and a CUDA GPU are required by the documented execution recipe. Exact GPU memory requirements have not been measured for this release.

Export details and limitations

The detector is shared across the three target languages. Its original expert tensor dimensions are preserved; the public interfaces accept English source speech only. Detector decoder adapter files are included for compatibility with the original model; the simultaneous ST policy uses the unit detector and frozen Whisper encoder.

Optimizer state, training metrics, and private filesystem paths are removed from the exported .pt metadata. Saved inference tensors are preserved, and copied adapters are byte-identical to the source checkpoints. These are inference exports, not resumable training snapshots. Load PyTorch checkpoints only from trusted sources.

Full GPU evaluation and replication of paper tables have not been performed as part of release preparation. The models can produce inaccurate or hallucinated translations; quality and latency depend on input conditions, decoding configuration, thresholds, and hardware. Training data and unit annotations are not bundled.

License

Project-authored code and additions retain the repository's MIT license. Base models and adapted code retain their upstream licenses, including Apache-2.0 for the two Hugging Face base models. See LICENSE, THIRD_PARTY_NOTICES.md, and licenses/. The project license does not replace upstream notices.

Citation

Please cite the associated paper. Use the publisher-provided citation available from the paper page.