Text Generation
Transformers
Safetensors
llama
fp8
compressed-tensors
vllm
quantized
text-generation-inference
Instructions to use liodon-ai/sqlcoder-7b-2-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use liodon-ai/sqlcoder-7b-2-FP8 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="liodon-ai/sqlcoder-7b-2-FP8")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("liodon-ai/sqlcoder-7b-2-FP8") model = AutoModelForCausalLM.from_pretrained("liodon-ai/sqlcoder-7b-2-FP8", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use liodon-ai/sqlcoder-7b-2-FP8 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "liodon-ai/sqlcoder-7b-2-FP8" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "liodon-ai/sqlcoder-7b-2-FP8", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/liodon-ai/sqlcoder-7b-2-FP8
- SGLang
How to use liodon-ai/sqlcoder-7b-2-FP8 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "liodon-ai/sqlcoder-7b-2-FP8" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "liodon-ai/sqlcoder-7b-2-FP8", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "liodon-ai/sqlcoder-7b-2-FP8" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "liodon-ai/sqlcoder-7b-2-FP8", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use liodon-ai/sqlcoder-7b-2-FP8 with Docker Model Runner:
docker model run hf.co/liodon-ai/sqlcoder-7b-2-FP8
File size: 2,048 Bytes
efb4936 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | ---
license: other
base_model: defog/sqlcoder-7b-2
base_model_relation: quantized
library_name: transformers
pipeline_tag: text-generation
tags:
- fp8
- compressed-tensors
- vllm
- quantized
quantized_by: liodon-ai
---
# sqlcoder-7b-2 — FP8 (dynamic)
FP8 quantization of [defog/sqlcoder-7b-2](https://huggingface.co/defog/sqlcoder-7b-2), published by [Liodon AI](https://huggingface.co/liodon-ai).
Quantized with [llm-compressor](https://github.com/vllm-project/llm-compressor) using the
`FP8_DYNAMIC` scheme: weights are cast to FP8 (E4M3) per-channel ahead of time, activations are
quantized to FP8 dynamically per-token at inference time. No calibration dataset is needed for this
scheme, so the quantized weights are numerically just a direct cast of the original — no calibration-set
bias to worry about. `lm_head` is left unquantized (standard practice — negligible size, disproportionate
quality impact if quantized).
Original size: 13.5 GB → Quantized: 7.0 GB.
## Quick Start
**vLLM**
```bash
vllm serve liodon-ai/sqlcoder-7b-2-FP8
```
**Text Generation Inference (TGI)**
```bash
docker run --gpus all -p 8080:80 ghcr.io/huggingface/text-generation-inference \
--model-id liodon-ai/sqlcoder-7b-2-FP8
```
**SGLang**
```bash
python -m sglang.launch_server --model-path liodon-ai/sqlcoder-7b-2-FP8
```
FP8 execution requires an NVIDIA GPU with compute capability ≥ 8.9 (Ada/Hopper/Blackwell — RTX 40-series,
L4/L40S, H100/H200, B100/B200/GB10). On older GPUs, vLLM/TGI will dequantize to run, which loses the
speed/memory benefit.
## Source
- **Model**: [defog/sqlcoder-7b-2](https://huggingface.co/defog/sqlcoder-7b-2)
- **License**: other
## Citation
```bibtex
@misc{liodonai_sqlcoder_7b_2_fp8,
title = {sqlcoder-7b-2 — FP8},
author = {{Liodon AI}},
year = {2026},
howpublished = {\url{https://huggingface.co/liodon-ai/sqlcoder-7b-2-FP8}},
note = {FP8 (dynamic) quantization of defog/sqlcoder-7b-2}
}
```
---
*Quantized by [Liodon AI](https://huggingface.co/liodon-ai)*
|