Angshul/Falcon-H1-0.5B-Rust

Falcon-H1-0.5B-Instruct — Rust CUDA Inference API (v0.4)

This repository hosts the original Falcon-H1-0.5B-Instruct pretrained weights converted to an FP32 .f32pack checkpoint for inference using a native Rust/Candle implementation with NVIDIA CUDA support.

The implementation preserves the model's hybrid Mamba-2 and grouped-query attention architecture. It is an inference port, not a newly trained, fine-tuned, distilled, or quantized model.

• Rust source code and HTTP API: https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API
• This converted checkpoint: https://huggingface.co/Angshul/Falcon-H1-0.5B-Rust
• Original pretrained model (Technology Innovation Institute): https://huggingface.co/tiiuae/Falcon-H1-0.5B-Instruct

Implementation

The Rust v0.4 backend includes:

• Native inference with Rust and Candle, supporting CPU and NVIDIA CUDA.
• Custom fused CUDA kernels for Mamba-2 state recurrence.
• Cached incremental decoding, retaining Mamba recurrent states, convolution history, and attention key/value states.
• Batched grouped-query attention (GQA).
• FP32 inference using the converted .f32pack weights.
• Fixed-length greedy generation of up to 512 new tokens, subject to a 2,048-token combined input/output limit on the CUDA backend.
• A lightweight HTTP API for inference and timing measurements.

The provided .f32pack file is intended for this Rust backend. It is not a native Hugging Face safetensors checkpoint and cannot be opened directly using transformers.from_pretrained(). Use the upstream repository for standard Transformers inference.

HTTP API

Endpoint	Purpose
GET /health	Check service and execution backend
GET /v1/model	Retrieve model configuration
POST /v1/logits	Calculate next-token logits
POST /v1/generate	Generate tokens with greedy decoding
POST /v1/generate_profile	Generate tokens and report prefill/decode timing

The API uses token IDs, not raw text strings. Use the original Falcon-H1 tokenizer to encode input and decode results. The server runs locally or on an externally managed deployment; this Hugging Face model repository does not itself provide a hosted endpoint.

Build and run

Install Rust and, for GPU execution, a compatible NVIDIA CUDA toolkit. Clone the source repository:

git clone https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API.git
cd Falcon-H1-Rust-Transformer-API
cargo build --release --features cuda

Download falcon-h1.f32pack from this Hugging Face repository and put it in the directory where you launch the server. Then start the CUDA API:

./target/release/falcon-h1-api \
  --checkpoint falcon-h1.f32pack \
  --device cuda \
  --bind 127.0.0.1:8083

Check that it is running:

curl http://127.0.0.1:8083/health

Send a token-ID request:

curl -X POST http://127.0.0.1:8083/v1/generate \
  -H "Content-Type: application/json" \
  -d '{"input_ids":[1,2,3],"max_new_tokens":16}'

The response contains output_ids, including the supplied prompt tokens and newly generated tokens. The example IDs above are only a smoke test, not a meaningful natural-language prompt.

For a natural-language request, obtain the tokenizer from tiiuae/Falcon-H1-0.5B-Instruct and use it to encode the prompt before calling the API.

Reproducibility checks

The GitHub repository includes kernel tests, cached-generation parity checks, and benchmarking scripts.

cargo test --release --features cuda --lib fused_cuda_matches_reference_demo -- --nocapture
cargo test --release --features cuda --lib cached_cuda_generation_equals_full_prefix -- --nocapture
cargo test --release --features cuda --lib cached_cpu_generation_equals_full_prefix -- --nocapture

For benchmark options and dependencies, see the GitHub README.

Validation against the official Transformers model

The Rust CUDA backend was validated on an NVIDIA Tesla T4 in November 2026 against the official Hugging Face Transformers implementation of Falcon-H1-0.5B-Instruct.

• The CUDA build and fused kernel correctness test passed.
• Cached generation parity tests passed on CPU and CUDA.
• FP32 logits agreed with the reference implementation, with a maximum absolute difference of approximately 5.6 × 10⁻⁴ over the tested prompts.
• Final long-generation checks matched the official implementation exactly at every generated-token position.

Prompt tokens	Generated tokens	Exact matching tokens
16	128	128/128
64	512	512/512
128	256	256/256
128	512	512/512

To compare the same number of tokens, early EOS stopping was disabled in the PyTorch reference. Rust v0.4 currently performs fixed-length generation and does not stop automatically at an EOS token.

These checks validate the tested configurations; they do not constitute a formal equivalence proof across all inputs or devices.

Performance

On a Tesla T4, across 12 tested prompt/output-length configurations, Rust CUDA end-to-end HTTP generation was 2.00×–2.77× faster than the official Hugging Face Transformers implementation running with eager attention and reference/fallback Mamba kernels in FP32.

Important benchmark scope: These results are not a comparison against optimized PyTorch Mamba kernels (mamba_ssm or causal_conv1d), FlashAttention, vLLM, or other production inference stacks. They should not be interpreted as evidence that this Rust implementation is universally faster.

Current limitations

• Inference only: no training, fine-tuning, or distillation.
• Token-ID API: no built-in tokenizer or text-to-text endpoint.
• Fixed-length greedy generation; no automatic EOS stopping.
• Fused sequential Mamba prefill, rather than a fully parallel SSD prefill implementation.
• Attention KV caches use on-device concatenation, not preallocated buffers.
• No CUDA Graphs, optimized FlashAttention, or production-grade serving features.
• Benchmark results are specific to the reported hardware, software configuration, and batch-size-one workload.

Attribution and license

Falcon-H1-0.5B-Instruct, its architecture, and its original pretrained parameters were developed by the Technology Innovation Institute (TII). This repository redistributes converted weights for use with the Rust inference implementation; it does not claim authorship of the underlying pretrained model.

The original Falcon-H1 model is distributed under the Falcon-LLM License. Review the upstream model page and its license terms before using or redistributing the checkpoint. The separate Rust source repository has its own code license.

• Upstream model: https://huggingface.co/tiiuae/Falcon-H1-0.5B-Instruct
• Rust/CUDA API and reproducibility code: https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API