vllm version for deploy?
cookbooks are now for 2.5, i try vllm==0.26.0 on lfm3 and i didn't work
Here is how I ran LiquidAI/LFM3-1.7B-FP8 on an NVIDIA L40S using SGLang.
Inference was extremely fast, but the model tended to overthink and spend too many tokens reasoning, even on simple tasks. In my testing, answer quality was less impressive than speed.
Software versions:
SGLang: 0.5.22 with the complete source snapshot from PR #46858.
Pinned commit: b05dd4a01f34b9381f2c8f4a87742f32118edab4
PyTorch: 2.14.1 + CUDA 13.0
Transformers: 5.19.0
FlashInfer: 0.7.2
SGLang kernel: 0.4.10
LFM3 support was still under review when tested. Installing stock SGLang 0.5.22 alone was insufficient; this setup required the PR source and matching dependencies.
The model ran in Docker, with the checkpoint mounted at /model. These were the environment variables and launch command inside the container:
export SGLANG_FLASHINFER_USE_PAGED=1
export SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=python
python3 -m sglang.launch_server --model-path /model --served-model-name lfm3-1.7b-fp8 --host 0.0.0.0 --port 8000 --trust-remote-code --reasoning-parser qwen3 --attention-backend flashinfer --fp8-gemm-backend triton --mem-fraction-static 0.32 --context-length 24576 --max-running-requests 12 --chunked-prefill-size 2048 --cuda-graph-max-bs-decode 12 --max-mamba-cache-size 512
Hardware-specific configuration:
The L40S uses the SM89 architecture. FP8 matrix operations used Triton, while convolution used the PR’s compiled PyTorch fallback because its CuTe convolution path did not support SM89.
FlashInfer paged prefill preserved correct sliding-window attention behavior. The Python cache backend avoided a Rust extension ABI mismatch.
I selected the qwen3 reasoning parser because qwen3-thinking incorrectly placed the final answer in reasoning_content when thinking was disabled. With qwen3, both thinking modes returned the answer correctly in content.
General sampling settings were temperature 0.1, top_k 50, top_p 1.0, and repetition_penalty 1.1.
Measured performance after warmup:
Single-request decode: approximately 180 tokens per second.
Time to first streamed content: approximately 38 milliseconds.
Twelve concurrent requests: approximately 1,876 output tokens per second in aggregate.
VRAM usage, including allocated caches: approximately 10.5 GiB.
A 24,000-token input was processed successfully.
The concurrency test used short prompts and 128 generated tokens per request. These were synthetic performance checks, not a quality benchmark.
My takeaway: exceptionally fast inference, but excessive thinking and underwhelming answer quality for my use case. I subsequently stopped and removed this trial deployment.