Update README.md

README.md

@@ -1,93 +1,147 @@
 ---
+language:
+- en
+pipeline_tag: text-generation
 license: other
-license_name: falcon-llm-license
-license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
 base_model: tiiuae/Falcon-H1-0.5B-Instruct
 tags:
+- falcon_h1
+- falcon-h1
 - rust
-- cuda
 - candle
+- cuda
 - mamba2
-- falcon-h1
+- inference
 - text-generation
+- conversational
 ---
 
-# Falcon-H1-0.5B — Rust CUDA Transformer
+# Falcon-H1-0.5B-Instruct — Rust CUDA Inference API (v0.4)
+
+This repository hosts the **original Falcon-H1-0.5B-Instruct pretrained weights converted to an FP32 `.f32pack` checkpoint** for inference using a native **Rust/Candle implementation with NVIDIA CUDA support**.
+
+The implementation preserves the model's **hybrid Mamba-2 and grouped-query attention architecture**. It is an inference port, **not a newly trained, fine-tuned, distilled, or quantized model**.
+
+- **Rust source code and HTTP API:** https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API
+- **This converted checkpoint:** https://huggingface.co/Angshul/Falcon-H1-0.5B-Rust
+- **Original pretrained model (Technology Innovation Institute):** https://huggingface.co/tiiuae/Falcon-H1-0.5B-Instruct
+
+## Implementation
+
+The Rust v0.4 backend includes:
+
+- Native inference with Rust and Candle, supporting CPU and NVIDIA CUDA.
+- Custom fused CUDA kernels for Mamba-2 state recurrence.
+- Cached incremental decoding, retaining Mamba recurrent states, convolution history, and attention key/value states.
+- Batched grouped-query attention (GQA).
+- FP32 inference using the converted `.f32pack` weights.
+- Fixed-length greedy generation of up to **512 new tokens**, subject to a **2,048-token combined input/output limit** on the CUDA backend.
+- A lightweight HTTP API for inference and timing measurements.
+
+The provided `.f32pack` file is intended for this Rust backend. It is **not** a native Hugging Face `safetensors` checkpoint and cannot be opened directly using `transformers.from_pretrained()`. Use the upstream repository for standard Transformers inference.
+
+## HTTP API
+
+| Endpoint | Purpose |
+| --- | --- |
+| `GET /health` | Check service and execution backend |
+| `GET /v1/model` | Retrieve model configuration |
+| `POST /v1/logits` | Calculate next-token logits |
+| `POST /v1/generate` | Generate tokens with greedy decoding |
+| `POST /v1/generate_profile` | Generate tokens and report prefill/decode timing |
 
-A Rust implementation of the pretrained Falcon-H1-0.5B-Instruct model.
+**The API uses token IDs, not raw text strings.** Use the original Falcon-H1 tokenizer to encode input and decode results. The server runs locally or on an externally managed deployment; this Hugging Face model repository does not itself provide a hosted endpoint.
 
-## Architecture
+### Build and run
 
-The implementation reproduces Falcon-H1's hybrid architecture:
+Install Rust and, for GPU execution, a compatible NVIDIA CUDA toolkit. Clone the source repository:
 
-- Parallel Mamba-2 and attention
-- Grouped-query attention
-- Rotary positional embeddings
-- RMSNorm
-- SwiGLU feed-forward networks
-- Original model dimensions and parameters
+```bash
+git clone https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API.git
+cd Falcon-H1-Rust-Transformer-API
+cargo build --release --features cuda
+```
 
-Implemented using Rust and Hugging Face Candle.
+Download `falcon-h1.f32pack` from this Hugging Face repository and put it in the directory where you launch the server. Then start the CUDA API:
 
-## Pretrained Weights
+```bash
+./target/release/falcon-h1-api \
+  --checkpoint falcon-h1.f32pack \
+  --device cuda \
+  --bind 127.0.0.1:8083
+```
 
-The checkpoint contains converted FP32 weights from:
+Check that it is running:
 
-https://huggingface.co/tiiuae/Falcon-H1-0.5B-Instruct
+```bash
+curl http://127.0.0.1:8083/health
+```
 
-No additional training or fine-tuning was performed.
+Send a token-ID request:
 
-The checkpoint uses a custom .f32pack format.
-It is not directly loadable using Transformers.from_pretrained.
+```bash
+curl -X POST http://127.0.0.1:8083/v1/generate \
+  -H "Content-Type: application/json" \
+  -d '{"input_ids":[1,2,3],"max_new_tokens":16}'
+```
 
-## GPU Validation
+The response contains `output_ids`, including the supplied prompt tokens and newly generated tokens. The example IDs above are only a smoke test, not a meaningful natural-language prompt.
 
-Tested on NVIDIA Tesla T4 using Google Colab.
+For a natural-language request, obtain the tokenizer from `tiiuae/Falcon-H1-0.5B-Instruct` and use it to encode the prompt before calling the API.
 
-Compared against the original Hugging Face implementation.
+### Reproducibility checks
 
-- Maximum absolute logit error: 7.0522e-5
-- Mean absolute logit error: 1.4631e-5
-- Identical top-5 token predictions
+The GitHub repository includes kernel tests, cached-generation parity checks, and benchmarking scripts.
 
-Validation used the three-token input [1, 2, 3].
+```bash
+cargo test --release --features cuda --lib fused_cuda_matches_reference_demo -- --nocapture
+cargo test --release --features cuda --lib cached_cuda_generation_equals_full_prefix -- --nocapture
+cargo test --release --features cuda --lib cached_cpu_generation_equals_full_prefix -- --nocapture
+```
 
-## Build
+For benchmark options and dependencies, see the [GitHub README](https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API#benchmark).
 
-Install Rust and CUDA, then execute:
+## Validation against the official Transformers model
 
-    cargo build --release --features cuda
+The Rust CUDA backend was validated on an **NVIDIA Tesla T4** in November 2026 against the official Hugging Face Transformers implementation of Falcon-H1-0.5B-Instruct.
 
-## Start HTTP API
+- The CUDA build and fused kernel correctness test passed.
+- Cached generation parity tests passed on CPU and CUDA.
+- FP32 logits agreed with the reference implementation, with a maximum absolute difference of approximately **5.6 × 10⁻⁴** over the tested prompts.
+- Final long-generation checks matched the official implementation **exactly at every generated-token position**.
 
-    ./target/release/falcon-h1-api --checkpoint falcon-h1.f32pack --device cuda
+| Prompt tokens | Generated tokens | Exact matching tokens |
+| ---: | ---: | ---: |
+| 16 | 128 | 128/128 |
+| 64 | 512 | 512/512 |
+| 128 | 256 | 256/256 |
+| 128 | 512 | 512/512 |
 
-## API Endpoints
+To compare the same number of tokens, early EOS stopping was disabled in the PyTorch reference. Rust v0.4 currently performs fixed-length generation and does not stop automatically at an EOS token.
 
-- GET /health
-- GET /v1/model
-- POST /v1/logits
-- POST /v1/generate
+These checks validate the tested configurations; they do not constitute a formal equivalence proof across all inputs or devices.
 
-The current API accepts token IDs.
+## Performance
 
-## CPU Support
+On a Tesla T4, across **12 tested prompt/output-length configurations**, Rust CUDA end-to-end HTTP generation was **2.00×–2.77× faster** than the official Hugging Face Transformers implementation running with **eager attention and reference/fallback Mamba kernels** in FP32.
 
-The architecture also supports CPU execution.
+**Important benchmark scope:** These results are **not** a comparison against optimized PyTorch Mamba kernels (`mamba_ssm` or `causal_conv1d`), FlashAttention, vLLM, or other production inference stacks. They should not be interpreted as evidence that this Rust implementation is universally faster.
 
-## Attribution
+## Current limitations
 
-Original pretrained model: Technology Innovation Institute (TII).
+- Inference only: no training, fine-tuning, or distillation.
+- Token-ID API: no built-in tokenizer or text-to-text endpoint.
+- Fixed-length greedy generation; no automatic EOS stopping.
+- Fused sequential Mamba prefill, rather than a fully parallel SSD prefill implementation.
+- Attention KV caches use on-device concatenation, not preallocated buffers.
+- No CUDA Graphs, optimized FlashAttention, or production-grade serving features.
+- Benchmark results are specific to the reported hardware, software configuration, and batch-size-one workload.
 
-Original model weights are governed by the Falcon-LLM License.
+## Attribution and license
 
-See the original model repository for licensing conditions.
+**Falcon-H1-0.5B-Instruct, its architecture, and its original pretrained parameters were developed by the Technology Innovation Institute (TII).** This repository redistributes converted weights for use with the Rust inference implementation; it does not claim authorship of the underlying pretrained model.
 
-## Implementation Status
+The original Falcon-H1 model is distributed under the **Falcon-LLM License**. Review the [upstream model page and its license terms](https://huggingface.co/tiiuae/Falcon-H1-0.5B-Instruct) before using or redistributing the checkpoint. The separate Rust source repository has its own code license.
 
-- Rust implementation: completed
-- CUDA inference: validated on T4
-- Numerical agreement: verified for a three-token input
-- HTTP inference API: implemented
-- Knowledge distillation: not yet implemented
-- GPU training optimization: pending
+- **Upstream model:** https://huggingface.co/tiiuae/Falcon-H1-0.5B-Instruct
+- **Rust/CUDA API and reproducibility code:** https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API