File size: 2,580 Bytes
916f925 032ba9a 942c947 916f925 73984e0 916f925 73984e0 916f925 73984e0 916f925 73984e0 916f925 73984e0 916f925 73984e0 916f925 032ba9a 916f925 032ba9a 916f925 032ba9a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | ---
license: other
library_name: nemo-speech.cpp
base_model: nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B
base_model_relation: quantized
tags:
- asr
- speech-recognition
- speech-to-text
- transcription
- gguf
- ggml
- llama.cpp
- nemo
- nvidia
- nemotron
- nemotron-3.5
- cuda
- cuda-12.8
- cuda-binary
- binary
- prebuilt-binary
- inference-server
- turing
- turing-gpu
- t4
- tesla-t4
- kaggle
- colab
---
# nemo-speech (CUDA build for Tesla T4)
Prebuilt `nemo-speech` server from NeMo-Speech.cpp, so Kaggle/Colab notebooks can skip the ~40 min compile. It runs [NVIDIA Nemotron-3.5-Transcribe-30B-A4B](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B) (speech recognition) from the GGUF conversion [AtomicChat/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-GGUF](https://huggingface.co/AtomicChat/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-GGUF). This repo contains the binary only, not the model weights.
## Parent model
| | |
|---|---|
| Original model | [nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B) |
| GGUF weights used | [AtomicChat/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-GGUF](https://huggingface.co/AtomicChat/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-GGUF) |
| Inference engine | NeMo-Speech.cpp (ggml / llama.cpp based) |
## Build
| | |
|---|---|
| Source | NeMo-Speech.cpp, commit `20999cc` (with this notebook's two-GPU split patch) |
| Preset | `cuda-server`, `-DCMAKE_CUDA_ARCHITECTURES=75-real` (Turing: T4 only) |
| CUDA | Build cuda_12.8.r12.8/compiler.35583870_0 |
| Built on | Linux-6.18.55+-x86_64-with-glibc2.39, Tesla T4, 7.5, 580.178.04 |
## Use
```python
from huggingface_hub import hf_hub_download
import tarfile, os
p = hf_hub_download("masterofaudio2077/nemo-speech-cpp-cuda-t4", "nemo-speech-cuda-t4.tar.gz")
tarfile.open(p).extractall("/tmp/nemo-bin")
os.chmod("/tmp/nemo-bin/bin/nemo-speech", 0o755)
# libraries sit next to the binary
os.environ["LD_LIBRARY_PATH"] = "/tmp/nemo-bin/bin:" + os.environ.get("LD_LIBRARY_PATH", "")
```
Then run `/tmp/nemo-bin/bin/nemo-speech serve --asr-model <model.gguf> ...`, with the GGUF files downloaded from the repo above.
## Limits
- Compiled for compute capability 7.5 only. It will not run on other GPUs (A100, L4, P100).
- Needs an NVIDIA driver with `libcuda.so.1` and a glibc/libstdc++ at least as new as the build machine's (see `ldd.txt`).
- Licence: the model is under NVIDIA's own licence (see the parent model page); the binary follows the NeMo-Speech.cpp, ggml and CUDA licences. Check them before redistributing. |