Preserve native mixed NVFP4/MXFP8 Inkling 2 without model-specific plugins
Browse files
README.md
CHANGED
|
@@ -35,8 +35,7 @@ Use SGLang with Inkling 2 support. No model-specific package is required.
|
|
| 35 |
|
| 36 |
```bash
|
| 37 |
SGLANG_BUILD_RUST_EXTS=none pip install 'git+https://github.com/sgl-project/sglang.git@f726172d760dea737c7f1e797c9f32efe14d1685#subdirectory=python'
|
| 38 |
-
|
| 39 |
-
sglang serve --model-path ./inkling-2-tiny --tp 2 --ep 2 \
|
| 40 |
--dtype bfloat16 --attention-backend fa4 --enable-multimodal \
|
| 41 |
--context-length 2048 --mem-fraction-static 0.2 \
|
| 42 |
--disable-custom-all-reduce \
|
|
|
|
| 35 |
|
| 36 |
```bash
|
| 37 |
SGLANG_BUILD_RUST_EXTS=none pip install 'git+https://github.com/sgl-project/sglang.git@f726172d760dea737c7f1e797c9f32efe14d1685#subdirectory=python'
|
| 38 |
+
sglang serve --model-path tiny-random/inkling-2 --tp 2 --ep 2 \
|
|
|
|
| 39 |
--dtype bfloat16 --attention-backend fa4 --enable-multimodal \
|
| 40 |
--context-length 2048 --mem-fraction-static 0.2 \
|
| 41 |
--disable-custom-all-reduce \
|