Instructions to use tiiuae/Falcon-H1-7B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use tiiuae/Falcon-H1-7B-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="tiiuae/Falcon-H1-7B-Instruct") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("tiiuae/Falcon-H1-7B-Instruct") model = AutoModelForCausalLM.from_pretrained("tiiuae/Falcon-H1-7B-Instruct", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use tiiuae/Falcon-H1-7B-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "tiiuae/Falcon-H1-7B-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tiiuae/Falcon-H1-7B-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/tiiuae/Falcon-H1-7B-Instruct
- SGLang
How to use tiiuae/Falcon-H1-7B-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "tiiuae/Falcon-H1-7B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tiiuae/Falcon-H1-7B-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "tiiuae/Falcon-H1-7B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tiiuae/Falcon-H1-7B-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use tiiuae/Falcon-H1-7B-Instruct with Docker Model Runner:
docker model run hf.co/tiiuae/Falcon-H1-7B-Instruct
Benchmarking Falcon-H1-7B vs Falcon3-7B & Mamba-7B on L4: vLLM CUDA graph speedup, state scaling, mixed NF4, and a 24GB vLLM config tip
Hi Falcon team,
Thank you for releasing 7B checkpoints for all three architectures (Falcon3-7B, Falcon3-Mamba-7B and Falcon-H1-7B). Having them at the same scale made a clean comparison on identical hardware possible.
I benchmarked the three on a single NVIDIA L4 (24 GB) at contexts from 512 to 16,384 tokens (batch size 1, 128 decode tokens, n=5 trials), logging latency, memory, quantization fidelity and 50 Hz NVML energy (J/token). Four findings are below, starting with the one most likely to save someone time.
1. Running H1 or Mamba in vLLM v1 on a 24 GB GPU
With the default max_num_seqs=256, vLLM pre-allocates Mamba state for 256 concurrent sequences (about 16.8 GiB on H1) during warmup and runs out of memory next to the 14.5 GiB of weights.
Fix: pass --max-num-seqs 8 (or 16) and --gpu-memory-utilization 0.92. On CUDA 12.4, also set VLLM_USE_FLASHINFER_SAMPLER=0, because FlashInfer's --compress-mode=size flag fails there.
2. Why the SSM models look about 2x slower in HF eager
In Hugging Face eager mode, Falcon-H1-7B (9.0 tok/s) and Falcon3-Mamba-7B (9.5 tok/s) look almost 2x slower than Falcon3-7B (14.2 to 16.4 tok/s). This is consistent with kernel launch overhead across 44 to 64 layers (the gap mostly closes under CUDA graphs), though I did not isolate it with a profiler timeline.
With CUDA graphs in vLLM v1, decode speeds up 1.8x to 1.9x for both SSM models (H1: 16.15 to 16.59 tok/s; Mamba: 17.26 to 17.29 tok/s), and decode energy per token falls by about 39 to 42%. At 8K context under vLLM:
| Model | tok/s | J/token |
|---|---|---|
| Falcon3-Mamba-7B | 17.27 ± 0.01 | 4.152 |
| Falcon3-7B | 16.55 ± 0.02 | 4.348 |
| Falcon-H1-7B | 16.15 ± 0.05 | 4.450 |
H1 stays flat across context lengths and sits about 2.4% behind Falcon3-7B.
3. Persistent state and peak VRAM
At 16K context, H1's persistent state (KV cache plus Mamba-2 state) is 771 MB, against 1,792 MB for Falcon3-7B (2.3x smaller). Falcon3-Mamba-7B's state is a constant 20 MB. Peak VRAM is roughly equal at 2K tokens (14,900.5 MB vs 14,902.9 MB). At 16K, H1 uses 1,671 MB less than Falcon3-7B (17,038 MB vs 18,709 MB).
4. Mixed NF4 on Falcon-H1-7B
I compared uniform bitsandbytes NF4 (int4_uniform) with a recipe that keeps the Mamba projections in BF16 and quantizes attention and MLP to NF4 (int4_mixed_ssm, using llm_int8_skip_modules=["mamba", "lm_head"]).
- Both recipes cut static weights by roughly half or more (14.48 GB to 6.74 GB for the mixed recipe, 4.87 GB for uniform).
- On a 24-passage EN/AR corpus (2,179 tokens), the mixed recipe has 17.4% lower token-level KL divergence than uniform NF4 (0.091 ± 0.004 vs 0.110 ± 0.006) and a smaller perplexity increase (+1.96% vs +2.38%).
- The cost: in bitsandbytes eager mode, SSM decoding drops to about 6.65 tok/s, energy rises to 8.16 J/token (uniform: 7.30), and VRAM is 1.87 GB higher than uniform NF4, because the Mamba weights stay in BF16. A fused mixed-precision kernel would be needed to make this practical at runtime.
I also ran a 200-item zero-shot benchmark, but the differences between recipes there are within noise at that sample size, so I'm not drawing conclusions from it.
Caveats
- All runs are batch size 1 on a single L4. Batching behaviour is not covered.
- The HF eager and vLLM runs used different software stacks (torch 2.5.1 / transformers 4.57.6 vs torch 2.13.0 / transformers 5.19.0), so the engine comparison also varies the stack.
- MBU figures are analytical estimates, not hardware counters.
- The 200-item benchmark has a standard error of 1.7 to 3.5 points, so I treat it as a "no collapse" check only.
Data and code
- Code, including a standalone
vllm_energy_mfu_profiler.py, and raw 50 Hz traces: https://github.com/HarithSami/falcon-hybrid-crossover-benchmark - Draft paper (PDF): https://github.com/HarithSami/falcon-hybrid-crossover-benchmark/blob/main/paper/falcon_7b_hybrid_crossover_paper.pdf
- Telemetry dataset: https://huggingface.co/datasets/HarithSami/falcon-7b-hybrid-crossover-telemetry
Question for the team
@JingweiZuo @ybelkada @DhiyaEddine @Slim-Frikha: for the next round, would it be more useful to benchmark multi-request batching in vLLM (H1's smaller state should allow larger batches before KV exhaustion), or FP8/AWQ/Marlin kernels with the SSM branch protected? If you've seen similar sensitivity in the Mamba projections at low bit widths, I'd be glad to hear it.
Thanks again for the models.
Harith Ibrahim, ML engineer, Dubai
@HarithSami very practical write-up. The max-num-seqs tip alone will save people hours.
One angle for Arabic users: with most tokenizers, Arabic text produces more tokens than the same content in English, so a 4K-token Arabic prompt carries less content than it seems. That makes your long-context results more relevant than they look, because the hybrid design's lower memory growth should matter most exactly where Arabic pushes context length up. A J/token comparison on an Arabic prompt set would be a great follow-up.