| --- |
| license: other |
| library_name: nemo-speech.cpp |
| base_model: nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B |
| base_model_relation: quantized |
| tags: |
| - asr |
| - speech-recognition |
| - speech-to-text |
| - transcription |
| - gguf |
| - ggml |
| - llama.cpp |
| - nemo |
| - nvidia |
| - nemotron |
| - nemotron-3.5 |
| - cuda |
| - cuda-12.8 |
| - cuda-binary |
| - binary |
| - prebuilt-binary |
| - inference-server |
| - turing |
| - turing-gpu |
| - t4 |
| - tesla-t4 |
| - kaggle |
| - colab |
| --- |
| # nemo-speech (CUDA build for Tesla T4) |
|
|
| Prebuilt `nemo-speech` server from NeMo-Speech.cpp, so Kaggle/Colab notebooks can skip the ~40 min compile. It runs [NVIDIA Nemotron-3.5-Transcribe-30B-A4B](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B) (speech recognition) from the GGUF conversion [AtomicChat/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-GGUF](https://huggingface.co/AtomicChat/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-GGUF). This repo contains the binary only, not the model weights. |
|
|
| ## Parent model |
| | | | |
| |---|---| |
| | Original model | [nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B) | |
| | GGUF weights used | [AtomicChat/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-GGUF](https://huggingface.co/AtomicChat/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-GGUF) | |
| | Inference engine | NeMo-Speech.cpp (ggml / llama.cpp based) | |
|
|
| ## Build |
| | | | |
| |---|---| |
| | Source | NeMo-Speech.cpp, commit `20999cc` (with this notebook's two-GPU split patch) | |
| | Preset | `cuda-server`, `-DCMAKE_CUDA_ARCHITECTURES=75-real` (Turing: T4 only) | |
| | CUDA | Build cuda_12.8.r12.8/compiler.35583870_0 | |
| | Built on | Linux-6.18.55+-x86_64-with-glibc2.39, Tesla T4, 7.5, 580.178.04 | |
| |
| ## Use |
| ```python |
| from huggingface_hub import hf_hub_download |
| import tarfile, os |
| p = hf_hub_download("masterofaudio2077/nemo-speech-cpp-cuda-t4", "nemo-speech-cuda-t4.tar.gz") |
| tarfile.open(p).extractall("/tmp/nemo-bin") |
| os.chmod("/tmp/nemo-bin/bin/nemo-speech", 0o755) |
| # libraries sit next to the binary |
| os.environ["LD_LIBRARY_PATH"] = "/tmp/nemo-bin/bin:" + os.environ.get("LD_LIBRARY_PATH", "") |
| ``` |
| Then run `/tmp/nemo-bin/bin/nemo-speech serve --asr-model <model.gguf> ...`, with the GGUF files downloaded from the repo above. |
| |
| ## Limits |
| - Compiled for compute capability 7.5 only. It will not run on other GPUs (A100, L4, P100). |
| - Needs an NVIDIA driver with `libcuda.so.1` and a glibc/libstdc++ at least as new as the build machine's (see `ldd.txt`). |
| - Licence: the model is under NVIDIA's own licence (see the parent model page); the binary follows the NeMo-Speech.cpp, ggml and CUDA licences. Check them before redistributing. |
