Update README.md
Browse files
README.md
CHANGED
|
@@ -111,6 +111,9 @@ Use a framework build that includes Step 5 Preview support, including the reason
|
|
| 111 |
| vLLM | BF16 | Two H200 nodes, 16 GPUs | TP16, TEP16, or DEP16 | OpenAI-compatible API |
|
| 112 |
| Transformers | FP8 | One H200 node, 8 GPUs | TP8 / EP8 | Offline generation |
|
| 113 |
| Transformers | BF16 | Two H200 nodes, 16 GPUs | TP16 / EP16 | Offline generation (see validation note below) |
|
|
|
|
|
|
|
|
|
|
| 114 |
|
| 115 |
DP, TP, and EP refer to data, tensor, and expert parallelism, respectively. TEP combines tensor and expert parallelism; DEP combines data and expert parallelism. These example configurations are sized for the deployments below. Actual memory requirements, usable context length, and concurrency vary with the checkpoint, inference framework, and KV cache allocation.
|
| 116 |
|
|
@@ -425,10 +428,6 @@ This launches 16 processes across the two nodes, matching the 16-shard checkpoin
|
|
| 425 |
|
| 426 |
### SGLang
|
| 427 |
|
| 428 |
-
| Framework | Precision | Hardware | Parallelism | Interface |
|
| 429 |
-
| --- | --- | --- | --- | --- |
|
| 430 |
-
| SGLang | FP8 | One B300 node, 4 GPUs | TP4/EP4 | OpenAI-compatible API |
|
| 431 |
-
| SGLang | FP8 | One H200 node, 8 GPUs | TP8/EP8 | OpenAI-compatible API |
|
| 432 |
|
| 433 |
#### Environment Setup
|
| 434 |
|
|
|
|
| 111 |
| vLLM | BF16 | Two H200 nodes, 16 GPUs | TP16, TEP16, or DEP16 | OpenAI-compatible API |
|
| 112 |
| Transformers | FP8 | One H200 node, 8 GPUs | TP8 / EP8 | Offline generation |
|
| 113 |
| Transformers | BF16 | Two H200 nodes, 16 GPUs | TP16 / EP16 | Offline generation (see validation note below) |
|
| 114 |
+
| SGLang | FP8 | One B300 node, 4 GPUs | TP4/EP4 | OpenAI-compatible API |
|
| 115 |
+
| SGLang | FP8 | One H200 node, 8 GPUs | TP8/EP8 | OpenAI-compatible API |
|
| 116 |
+
|
| 117 |
|
| 118 |
DP, TP, and EP refer to data, tensor, and expert parallelism, respectively. TEP combines tensor and expert parallelism; DEP combines data and expert parallelism. These example configurations are sized for the deployments below. Actual memory requirements, usable context length, and concurrency vary with the checkpoint, inference framework, and KV cache allocation.
|
| 119 |
|
|
|
|
| 428 |
|
| 429 |
### SGLang
|
| 430 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 431 |
|
| 432 |
#### Environment Setup
|
| 433 |
|