Update SGLang quickstart to the day-0 image and cookbook

#3
by JustinTong - opened
Files changed (1) hide show
  1. README.md +20 -35
README.md CHANGED
@@ -45,60 +45,45 @@ We also introduce **LingCode**, a harness built on the open-source coding agent
45
 
46
  ## Quickstart
47
  #### SGLang (Recommended)
48
- **Install our SGLang**
49
 
50
- ```bash
51
- pip install uv
52
 
53
- uv venv ~/my_ling_env
54
-
55
- source ~/my_ling_env/bin/activate
56
-
57
- git clone -b ling-v3-open https://github.com/inclusionAI/sglang_ling_v3.git
58
- cd sglang_ling_v3
59
-
60
- pip install --upgrade pip
61
- pip install -e "python"
62
  ```
63
 
64
  ##### Run Inference
65
- Both FP8 and mxfp4 models are supported by SGLang now. It depends on the dtype of the model in `${MODEL_PATH}`. Here is the example to run Ling-3.1-flash with 8 GPUs, where the master node IP is `${MASTER_IP}` and server port is `${PORT}`:
66
 
67
- **Server**
68
-
69
- **1. Standard Inference**
70
 
71
  ```bash
72
- sglang serve \
73
- --model-path $MODEL_PATH \
74
- --port $PORT \
75
- --trust-remote-code \
76
- --tp 8 \
77
- --enable-cache-report \
78
- --log-level info \
79
- --enable-metrics \
80
- --mem-fraction-static 0.85 \
81
- --max-running-requests 64 \
82
- --chunked-prefill-size 8192 \
83
- --speculative-algorithm NEXTN \
84
- --allow-auto-truncate \
85
- --tool-call-parser ling3 \
86
- --reasoning-parser ling3
87
  ```
88
 
 
 
89
  **Client**
90
 
91
  ```bash
92
- curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
93
  -H "Content-Type: application/json" \
94
- -d '{"model": "auto",
95
- "messages": [{"role": "user", "content": "What is the capital of France?"}], "chat_template_kwargs": {"enable_thinking": true},
 
96
  "stream": true,
97
- "temperature": 1.0,
98
  "top_k": 20,
99
  "top_p": 0.95}'
100
  ```
101
 
 
 
102
  #### vLLM
103
  ##### Environment Preparation
104
  ```bash
 
45
 
46
  ## Quickstart
47
  #### SGLang (Recommended)
 
48
 
49
+ The day-0 SGLang image includes Ling-3.1-flash support, MTP (NEXTN) speculative decoding, and the Hopper decode optimizations:
 
50
 
51
+ ```bash
52
+ docker pull lmsysorg/sglang:dev-ling-3.1-flash
 
 
 
 
 
 
 
53
  ```
54
 
55
  ##### Run Inference
 
56
 
57
+ **Server** (8x H200, TP8 + EP8):
 
 
58
 
59
  ```bash
60
+ docker run --gpus all --network host --ipc=host --rm \
61
+ lmsysorg/sglang:dev-ling-3.1-flash \
62
+ python3 -m sglang.launch_server \
63
+ --model-path inclusionAI/Ling-3.1-flash \
64
+ --tp 8 --ep 8 \
65
+ --speculative-algorithm NEXTN \
66
+ --host 0.0.0.0 --port 30000
 
 
 
 
 
 
 
 
67
  ```
68
 
69
+ On 4x GB300, use `--tp 4 --ep 4` instead. Hopper decode optimizations (optional, ~10% faster batch-1 decode) can be enabled with `SGLANG_ENABLE_FP8_BLOCK_GEMV=1` and `SGLANG_ENABLE_FUSED_KDA_OUTPUT_QUANT=1`.
70
+
71
  **Client**
72
 
73
  ```bash
74
+ curl -s http://localhost:30000/v1/chat/completions \
75
  -H "Content-Type: application/json" \
76
+ -d '{"model": "Ling-3.1-flash",
77
+ "messages": [{"role": "user", "content": "What is the capital of France?"}],
78
+ "chat_template_kwargs": {"enable_thinking": true},
79
  "stream": true,
80
+ "temperature": 1.0,
81
  "top_k": 20,
82
  "top_p": 0.95}'
83
  ```
84
 
85
+ Verified recipes for H200, GB300, H800, H100, and B200, plus HiCache (Mooncake L3), reasoning on/off, tool calling, and measured accuracy/speed numbers, are in the [SGLang Ling-3.1-flash cookbook](https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-3.1-flash).
86
+
87
  #### vLLM
88
  ##### Environment Preparation
89
  ```bash