Instructions to use inclusionAI/Ling-3.1-flash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Inference
- HuggingChat
- Notebooks
- Google Colab
- Kaggle
Update SGLang quickstart to the day-0 image and cookbook
#3
by JustinTong - opened
README.md
CHANGED
|
@@ -45,60 +45,45 @@ We also introduce **LingCode**, a harness built on the open-source coding agent
|
|
| 45 |
|
| 46 |
## Quickstart
|
| 47 |
#### SGLang (Recommended)
|
| 48 |
-
**Install our SGLang**
|
| 49 |
|
| 50 |
-
|
| 51 |
-
pip install uv
|
| 52 |
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
source ~/my_ling_env/bin/activate
|
| 56 |
-
|
| 57 |
-
git clone -b ling-v3-open https://github.com/inclusionAI/sglang_ling_v3.git
|
| 58 |
-
cd sglang_ling_v3
|
| 59 |
-
|
| 60 |
-
pip install --upgrade pip
|
| 61 |
-
pip install -e "python"
|
| 62 |
```
|
| 63 |
|
| 64 |
##### Run Inference
|
| 65 |
-
Both FP8 and mxfp4 models are supported by SGLang now. It depends on the dtype of the model in `${MODEL_PATH}`. Here is the example to run Ling-3.1-flash with 8 GPUs, where the master node IP is `${MASTER_IP}` and server port is `${PORT}`:
|
| 66 |
|
| 67 |
-
**Server**
|
| 68 |
-
|
| 69 |
-
**1. Standard Inference**
|
| 70 |
|
| 71 |
```bash
|
| 72 |
-
|
| 73 |
-
--
|
| 74 |
-
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
--enable-metrics \
|
| 80 |
-
--mem-fraction-static 0.85 \
|
| 81 |
-
--max-running-requests 64 \
|
| 82 |
-
--chunked-prefill-size 8192 \
|
| 83 |
-
--speculative-algorithm NEXTN \
|
| 84 |
-
--allow-auto-truncate \
|
| 85 |
-
--tool-call-parser ling3 \
|
| 86 |
-
--reasoning-parser ling3
|
| 87 |
```
|
| 88 |
|
|
|
|
|
|
|
| 89 |
**Client**
|
| 90 |
|
| 91 |
```bash
|
| 92 |
-
curl -s http://
|
| 93 |
-H "Content-Type: application/json" \
|
| 94 |
-
-d '{"model": "
|
| 95 |
-
"messages": [{"role": "user", "content": "What is the capital of France?"}],
|
|
|
|
| 96 |
"stream": true,
|
| 97 |
-
"temperature": 1.0,
|
| 98 |
"top_k": 20,
|
| 99 |
"top_p": 0.95}'
|
| 100 |
```
|
| 101 |
|
|
|
|
|
|
|
| 102 |
#### vLLM
|
| 103 |
##### Environment Preparation
|
| 104 |
```bash
|
|
|
|
| 45 |
|
| 46 |
## Quickstart
|
| 47 |
#### SGLang (Recommended)
|
|
|
|
| 48 |
|
| 49 |
+
The day-0 SGLang image includes Ling-3.1-flash support, MTP (NEXTN) speculative decoding, and the Hopper decode optimizations:
|
|
|
|
| 50 |
|
| 51 |
+
```bash
|
| 52 |
+
docker pull lmsysorg/sglang:dev-ling-3.1-flash
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 53 |
```
|
| 54 |
|
| 55 |
##### Run Inference
|
|
|
|
| 56 |
|
| 57 |
+
**Server** (8x H200, TP8 + EP8):
|
|
|
|
|
|
|
| 58 |
|
| 59 |
```bash
|
| 60 |
+
docker run --gpus all --network host --ipc=host --rm \
|
| 61 |
+
lmsysorg/sglang:dev-ling-3.1-flash \
|
| 62 |
+
python3 -m sglang.launch_server \
|
| 63 |
+
--model-path inclusionAI/Ling-3.1-flash \
|
| 64 |
+
--tp 8 --ep 8 \
|
| 65 |
+
--speculative-algorithm NEXTN \
|
| 66 |
+
--host 0.0.0.0 --port 30000
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
```
|
| 68 |
|
| 69 |
+
On 4x GB300, use `--tp 4 --ep 4` instead. Hopper decode optimizations (optional, ~10% faster batch-1 decode) can be enabled with `SGLANG_ENABLE_FP8_BLOCK_GEMV=1` and `SGLANG_ENABLE_FUSED_KDA_OUTPUT_QUANT=1`.
|
| 70 |
+
|
| 71 |
**Client**
|
| 72 |
|
| 73 |
```bash
|
| 74 |
+
curl -s http://localhost:30000/v1/chat/completions \
|
| 75 |
-H "Content-Type: application/json" \
|
| 76 |
+
-d '{"model": "Ling-3.1-flash",
|
| 77 |
+
"messages": [{"role": "user", "content": "What is the capital of France?"}],
|
| 78 |
+
"chat_template_kwargs": {"enable_thinking": true},
|
| 79 |
"stream": true,
|
| 80 |
+
"temperature": 1.0,
|
| 81 |
"top_k": 20,
|
| 82 |
"top_p": 0.95}'
|
| 83 |
```
|
| 84 |
|
| 85 |
+
Verified recipes for H200, GB300, H800, H100, and B200, plus HiCache (Mooncake L3), reasoning on/off, tool calling, and measured accuracy/speed numbers, are in the [SGLang Ling-3.1-flash cookbook](https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-3.1-flash).
|
| 86 |
+
|
| 87 |
#### vLLM
|
| 88 |
##### Environment Preparation
|
| 89 |
```bash
|