| --- |
| language: |
| - en |
| pipeline_tag: text-generation |
| license: other |
| base_model: tiiuae/Falcon-H1-0.5B-Instruct |
| tags: |
| - falcon_h1 |
| - falcon-h1 |
| - rust |
| - candle |
| - cuda |
| - mamba2 |
| - inference |
| - text-generation |
| - conversational |
| --- |
| |
| # Falcon-H1-0.5B-Instruct — Rust CUDA Inference API (v0.4) |
|
|
| This repository hosts the **original Falcon-H1-0.5B-Instruct pretrained weights converted to an FP32 `.f32pack` checkpoint** for inference using a native **Rust/Candle implementation with NVIDIA CUDA support**. |
|
|
| The implementation preserves the model's **hybrid Mamba-2 and grouped-query attention architecture**. It is an inference port, **not a newly trained, fine-tuned, distilled, or quantized model**. |
|
|
| - **Rust source code and HTTP API:** https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API |
| - **This converted checkpoint:** https://huggingface.co/Angshul/Falcon-H1-0.5B-Rust |
| - **Original pretrained model (Technology Innovation Institute):** https://huggingface.co/tiiuae/Falcon-H1-0.5B-Instruct |
|
|
| ## Implementation |
|
|
| The Rust v0.4 backend includes: |
|
|
| - Native inference with Rust and Candle, supporting CPU and NVIDIA CUDA. |
| - Custom fused CUDA kernels for Mamba-2 state recurrence. |
| - Cached incremental decoding, retaining Mamba recurrent states, convolution history, and attention key/value states. |
| - Batched grouped-query attention (GQA). |
| - FP32 inference using the converted `.f32pack` weights. |
| - Fixed-length greedy generation of up to **512 new tokens**, subject to a **2,048-token combined input/output limit** on the CUDA backend. |
| - A lightweight HTTP API for inference and timing measurements. |
|
|
| The provided `.f32pack` file is intended for this Rust backend. It is **not** a native Hugging Face `safetensors` checkpoint and cannot be opened directly using `transformers.from_pretrained()`. Use the upstream repository for standard Transformers inference. |
|
|
| ## HTTP API |
|
|
| | Endpoint | Purpose | |
| | --- | --- | |
| | `GET /health` | Check service and execution backend | |
| | `GET /v1/model` | Retrieve model configuration | |
| | `POST /v1/logits` | Calculate next-token logits | |
| | `POST /v1/generate` | Generate tokens with greedy decoding | |
| | `POST /v1/generate_profile` | Generate tokens and report prefill/decode timing | |
|
|
| **The API uses token IDs, not raw text strings.** Use the original Falcon-H1 tokenizer to encode input and decode results. The server runs locally or on an externally managed deployment; this Hugging Face model repository does not itself provide a hosted endpoint. |
|
|
| ### Build and run |
|
|
| Install Rust and, for GPU execution, a compatible NVIDIA CUDA toolkit. Clone the source repository: |
|
|
| ```bash |
| git clone https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API.git |
| cd Falcon-H1-Rust-Transformer-API |
| cargo build --release --features cuda |
| ``` |
|
|
| Download `falcon-h1.f32pack` from this Hugging Face repository and put it in the directory where you launch the server. Then start the CUDA API: |
|
|
| ```bash |
| ./target/release/falcon-h1-api \ |
| --checkpoint falcon-h1.f32pack \ |
| --device cuda \ |
| --bind 127.0.0.1:8083 |
| ``` |
|
|
| Check that it is running: |
|
|
| ```bash |
| curl http://127.0.0.1:8083/health |
| ``` |
|
|
| Send a token-ID request: |
|
|
| ```bash |
| curl -X POST http://127.0.0.1:8083/v1/generate \ |
| -H "Content-Type: application/json" \ |
| -d '{"input_ids":[1,2,3],"max_new_tokens":16}' |
| ``` |
|
|
| The response contains `output_ids`, including the supplied prompt tokens and newly generated tokens. The example IDs above are only a smoke test, not a meaningful natural-language prompt. |
|
|
| For a natural-language request, obtain the tokenizer from `tiiuae/Falcon-H1-0.5B-Instruct` and use it to encode the prompt before calling the API. |
|
|
| ### Reproducibility checks |
|
|
| The GitHub repository includes kernel tests, cached-generation parity checks, and benchmarking scripts. |
|
|
| ```bash |
| cargo test --release --features cuda --lib fused_cuda_matches_reference_demo -- --nocapture |
| cargo test --release --features cuda --lib cached_cuda_generation_equals_full_prefix -- --nocapture |
| cargo test --release --features cuda --lib cached_cpu_generation_equals_full_prefix -- --nocapture |
| ``` |
|
|
| For benchmark options and dependencies, see the [GitHub README](https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API#benchmark). |
|
|
| ## Validation against the official Transformers model |
|
|
| The Rust CUDA backend was validated on an **NVIDIA Tesla T4** in November 2026 against the official Hugging Face Transformers implementation of Falcon-H1-0.5B-Instruct. |
|
|
| - The CUDA build and fused kernel correctness test passed. |
| - Cached generation parity tests passed on CPU and CUDA. |
| - FP32 logits agreed with the reference implementation, with a maximum absolute difference of approximately **5.6 × 10⁻⁴** over the tested prompts. |
| - Final long-generation checks matched the official implementation **exactly at every generated-token position**. |
|
|
| | Prompt tokens | Generated tokens | Exact matching tokens | |
| | ---: | ---: | ---: | |
| | 16 | 128 | 128/128 | |
| | 64 | 512 | 512/512 | |
| | 128 | 256 | 256/256 | |
| | 128 | 512 | 512/512 | |
|
|
| To compare the same number of tokens, early EOS stopping was disabled in the PyTorch reference. Rust v0.4 currently performs fixed-length generation and does not stop automatically at an EOS token. |
|
|
| These checks validate the tested configurations; they do not constitute a formal equivalence proof across all inputs or devices. |
|
|
| ## Performance |
|
|
| On a Tesla T4, across **12 tested prompt/output-length configurations**, Rust CUDA end-to-end HTTP generation was **2.00×–2.77× faster** than the official Hugging Face Transformers implementation running with **eager attention and reference/fallback Mamba kernels** in FP32. |
|
|
| **Important benchmark scope:** These results are **not** a comparison against optimized PyTorch Mamba kernels (`mamba_ssm` or `causal_conv1d`), FlashAttention, vLLM, or other production inference stacks. They should not be interpreted as evidence that this Rust implementation is universally faster. |
|
|
| ## Current limitations |
|
|
| - Inference only: no training, fine-tuning, or distillation. |
| - Token-ID API: no built-in tokenizer or text-to-text endpoint. |
| - Fixed-length greedy generation; no automatic EOS stopping. |
| - Fused sequential Mamba prefill, rather than a fully parallel SSD prefill implementation. |
| - Attention KV caches use on-device concatenation, not preallocated buffers. |
| - No CUDA Graphs, optimized FlashAttention, or production-grade serving features. |
| - Benchmark results are specific to the reported hardware, software configuration, and batch-size-one workload. |
|
|
| ## Attribution and license |
|
|
| **Falcon-H1-0.5B-Instruct, its architecture, and its original pretrained parameters were developed by the Technology Innovation Institute (TII).** This repository redistributes converted weights for use with the Rust inference implementation; it does not claim authorship of the underlying pretrained model. |
|
|
| The original Falcon-H1 model is distributed under the **Falcon-LLM License**. Review the [upstream model page and its license terms](https://huggingface.co/tiiuae/Falcon-H1-0.5B-Instruct) before using or redistributing the checkpoint. The separate Rust source repository has its own code license. |
|
|
| - **Upstream model:** https://huggingface.co/tiiuae/Falcon-H1-0.5B-Instruct |
| - **Rust/CUDA API and reproducibility code:** https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API |
|
|
