---
language:
- en
pipeline_tag: text-generation
license: other
base_model: tiiuae/Falcon-H1-0.5B-Instruct
tags:
- falcon_h1
- falcon-h1
- rust
- candle
- cuda
- mamba2
- inference
- text-generation
- conversational
---

# Falcon-H1-0.5B-Instruct — Rust CUDA Inference API (v0.4)

This repository hosts the **original Falcon-H1-0.5B-Instruct pretrained weights converted to an FP32 `.f32pack` checkpoint** for inference using a native **Rust/Candle implementation with NVIDIA CUDA support**.

The implementation preserves the model's **hybrid Mamba-2 and grouped-query attention architecture**. It is an inference port, **not a newly trained, fine-tuned, distilled, or quantized model**.

- **Rust source code and HTTP API:** https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API
- **This converted checkpoint:** https://huggingface.co/Angshul/Falcon-H1-0.5B-Rust
- **Original pretrained model (Technology Innovation Institute):** https://huggingface.co/tiiuae/Falcon-H1-0.5B-Instruct

## Implementation

The Rust v0.4 backend includes:

- Native inference with Rust and Candle, supporting CPU and NVIDIA CUDA.
- Custom fused CUDA kernels for Mamba-2 state recurrence.
- Cached incremental decoding, retaining Mamba recurrent states, convolution history, and attention key/value states.
- Batched grouped-query attention (GQA).
- FP32 inference using the converted `.f32pack` weights.
- Fixed-length greedy generation of up to **512 new tokens**, subject to a **2,048-token combined input/output limit** on the CUDA backend.
- A lightweight HTTP API for inference and timing measurements.

The provided `.f32pack` file is intended for this Rust backend. It is **not** a native Hugging Face `safetensors` checkpoint and cannot be opened directly using `transformers.from_pretrained()`. Use the upstream repository for standard Transformers inference.

## HTTP API

| Endpoint | Purpose |
| --- | --- |
| `GET /health` | Check service and execution backend |
| `GET /v1/model` | Retrieve model configuration |
| `POST /v1/logits` | Calculate next-token logits |
| `POST /v1/generate` | Generate tokens with greedy decoding |
| `POST /v1/generate_profile` | Generate tokens and report prefill/decode timing |

**The API uses token IDs, not raw text strings.** Use the original Falcon-H1 tokenizer to encode input and decode results. The server runs locally or on an externally managed deployment; this Hugging Face model repository does not itself provide a hosted endpoint.

### Build and run

Install Rust and, for GPU execution, a compatible NVIDIA CUDA toolkit. Clone the source repository:

```bash
git clone https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API.git
cd Falcon-H1-Rust-Transformer-API
cargo build --release --features cuda
```

Download `falcon-h1.f32pack` from this Hugging Face repository and put it in the directory where you launch the server. Then start the CUDA API:

```bash
./target/release/falcon-h1-api \
  --checkpoint falcon-h1.f32pack \
  --device cuda \
  --bind 127.0.0.1:8083
```

Check that it is running:

```bash
curl http://127.0.0.1:8083/health
```

Send a token-ID request:

```bash
curl -X POST http://127.0.0.1:8083/v1/generate \
  -H "Content-Type: application/json" \
  -d '{"input_ids":[1,2,3],"max_new_tokens":16}'
```

The response contains `output_ids`, including the supplied prompt tokens and newly generated tokens. The example IDs above are only a smoke test, not a meaningful natural-language prompt.

For a natural-language request, obtain the tokenizer from `tiiuae/Falcon-H1-0.5B-Instruct` and use it to encode the prompt before calling the API.

### Reproducibility checks

The GitHub repository includes kernel tests, cached-generation parity checks, and benchmarking scripts.

```bash
cargo test --release --features cuda --lib fused_cuda_matches_reference_demo -- --nocapture
cargo test --release --features cuda --lib cached_cuda_generation_equals_full_prefix -- --nocapture
cargo test --release --features cuda --lib cached_cpu_generation_equals_full_prefix -- --nocapture
```

For benchmark options and dependencies, see the [GitHub README](https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API#benchmark).

## Validation against the official Transformers model

The Rust CUDA backend was validated on an **NVIDIA Tesla T4** in November 2026 against the official Hugging Face Transformers implementation of Falcon-H1-0.5B-Instruct.

- The CUDA build and fused kernel correctness test passed.
- Cached generation parity tests passed on CPU and CUDA.
- FP32 logits agreed with the reference implementation, with a maximum absolute difference of approximately **5.6 × 10⁻⁴** over the tested prompts.
- Final long-generation checks matched the official implementation **exactly at every generated-token position**.

| Prompt tokens | Generated tokens | Exact matching tokens |
| ---: | ---: | ---: |
| 16 | 128 | 128/128 |
| 64 | 512 | 512/512 |
| 128 | 256 | 256/256 |
| 128 | 512 | 512/512 |

To compare the same number of tokens, early EOS stopping was disabled in the PyTorch reference. Rust v0.4 currently performs fixed-length generation and does not stop automatically at an EOS token.

These checks validate the tested configurations; they do not constitute a formal equivalence proof across all inputs or devices.

## Performance

On a Tesla T4, across **12 tested prompt/output-length configurations**, Rust CUDA end-to-end HTTP generation was **2.00×–2.77× faster** than the official Hugging Face Transformers implementation running with **eager attention and reference/fallback Mamba kernels** in FP32.

**Important benchmark scope:** These results are **not** a comparison against optimized PyTorch Mamba kernels (`mamba_ssm` or `causal_conv1d`), FlashAttention, vLLM, or other production inference stacks. They should not be interpreted as evidence that this Rust implementation is universally faster.

## Current limitations

- Inference only: no training, fine-tuning, or distillation.
- Token-ID API: no built-in tokenizer or text-to-text endpoint.
- Fixed-length greedy generation; no automatic EOS stopping.
- Fused sequential Mamba prefill, rather than a fully parallel SSD prefill implementation.
- Attention KV caches use on-device concatenation, not preallocated buffers.
- No CUDA Graphs, optimized FlashAttention, or production-grade serving features.
- Benchmark results are specific to the reported hardware, software configuration, and batch-size-one workload.

## Attribution and license

**Falcon-H1-0.5B-Instruct, its architecture, and its original pretrained parameters were developed by the Technology Innovation Institute (TII).** This repository redistributes converted weights for use with the Rust inference implementation; it does not claim authorship of the underlying pretrained model.

The original Falcon-H1 model is distributed under the **Falcon-LLM License**. Review the [upstream model page and its license terms](https://huggingface.co/tiiuae/Falcon-H1-0.5B-Instruct) before using or redistributing the checkpoint. The separate Rust source repository has its own code license.

- **Upstream model:** https://huggingface.co/tiiuae/Falcon-H1-0.5B-Instruct
- **Rust/CUDA API and reproducibility code:** https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API