Instructions to use thinkingmachines/Inkling-Small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use thinkingmachines/Inkling-Small with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="thinkingmachines/Inkling-Small")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("thinkingmachines/Inkling-Small")
model = AutoModelForMultimodalLM.from_pretrained("thinkingmachines/Inkling-Small", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use thinkingmachines/Inkling-Small with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "thinkingmachines/Inkling-Small"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "thinkingmachines/Inkling-Small",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/thinkingmachines/Inkling-Small

SGLang

How to use thinkingmachines/Inkling-Small with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "thinkingmachines/Inkling-Small" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "thinkingmachines/Inkling-Small",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "thinkingmachines/Inkling-Small" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "thinkingmachines/Inkling-Small",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use thinkingmachines/Inkling-Small with Docker Model Runner:
```
docker model run hf.co/thinkingmachines/Inkling-Small
```

Vram required?

by jtvino - opened 1 day ago

Discussion

jtvino

1 day ago

Hi
I am trying to run this model on 2 H200 GPUs; however it looks like it keeps running out of VRAM?

vllm serve thinkingmachines/Inkling-Small-NVFP4
            --served-model-name thinkingmachines/Inkling-Small
            --reasoning-parser inkling
            --tool-call-parser inkling
            --enable-auto-tool-choice
            --tensor-parallel-size 2
            --max-model-len 200000
            --trust-remote-code
            --enable-chunked-prefill
            --enable-prefix-caching
            --kernel-config.enable_flashinfer_autotune=False
            --disable-custom-all-reduce

DEBUG 08-05 15:27:06 [plugins/__init__.py:44] No plugins for group vllm.platform_plugins found.
DEBUG 08-05 15:27:06 [platforms/__init__.py:36] Checking if TPU platform is available.
DEBUG 08-05 15:27:06 [platforms/__init__.py:55] TPU platform is not available because: No module named 'libtpu'
DEBUG 08-05 15:27:06 [platforms/__init__.py:61] Checking if CUDA platform is available.
DEBUG 08-05 15:27:06 [platforms/__init__.py:84] Confirmed CUDA platform is available.
DEBUG 08-05 15:27:06 [platforms/__init__.py:112] Checking if ROCm platform is available.
DEBUG 08-05 15:27:06 [platforms/__init__.py:126] ROCm platform is not available because: No module named 'amdsmi'
DEBUG 08-05 15:27:06 [platforms/__init__.py:133] Checking if XPU platform is available.
DEBUG 08-05 15:27:06 [platforms/__init__.py:164] Checking if CPU platform is available.
DEBUG 08-05 15:27:06 [platforms/__init__.py:61] Checking if CUDA platform is available.
DEBUG 08-05 15:27:06 [platforms/__init__.py:84] Confirmed CUDA platform is available.
DEBUG 08-05 15:27:06 [platforms/__init__.py:245] Automatically detected platform cuda.
DEBUG 08-05 15:27:18 [entrypoints/.../utils/api_utils.py:166] Setting VLLM_WORKER_MULTIPROC_METHOD to 'spawn'
DEBUG 08-05 15:27:18 [plugins/__init__.py:52] Available plugins for group vllm.general_plugins:
DEBUG 08-05 15:27:18 [plugins/__init__.py:54] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
DEBUG 08-05 15:27:18 [plugins/__init__.py:54] - lora_hf_hub_resolver -> vllm.plugins.lora_resolvers.hf_hub_resolver:register_hf_hub_resolver
DEBUG 08-05 15:27:18 [plugins/__init__.py:57] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
(APIServer pid=467) INFO 08-05 15:27:18 [entrypoints/.../utils/api_utils.py:345] 
(APIServer pid=467) INFO 08-05 15:27:18 [entrypoints/.../utils/api_utils.py:345]        █     █     █▄   ▄█
(APIServer pid=467) INFO 08-05 15:27:18 [entrypoints/.../utils/api_utils.py:345]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.26.1rc1.dev306+gcb8104839
(APIServer pid=467) INFO 08-05 15:27:18 [entrypoints/.../utils/api_utils.py:345]   █▄█▀ █     █     █     █  model   thinkingmachines/Inkling-Small-NVFP4
(APIServer pid=467) INFO 08-05 15:27:18 [entrypoints/.../utils/api_utils.py:345]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=467) INFO 08-05 15:27:18 [entrypoints/.../utils/api_utils.py:345] 
(APIServer pid=467) INFO 08-05 15:27:18 [entrypoints/.../utils/api_utils.py:273] non-default args: {'model_tag': 'thinkingmachines/Inkling-Small-NVFP4', 'enable_auto_tool_choice': True, 'tool_call_parser': 'inkling', 'model': 'thinkingmachines/Inkling-Small-NVFP4', 'trust_remote_code': True, 'max_model_len': 200000, 'served_model_name': ['thinkingmachines/Inkling-Small'], 'reasoning_parser': 'inkling', 'tensor_parallel_size': 2, 'disable_custom_all_reduce': True, 'gpu_memory_utilization': 0.85, 'enable_prefix_caching': True, 'enable_chunked_prefill': True, 'kernel_config': KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[]), enable_flashinfer_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')}
(APIServer pid=467) WARNING 08-05 15:27:18 [envs.py:2120] Unknown vLLM environment variable detected: VLLM_MULTI_SERVE_PORT_8000_TCP
(APIServer pid=467) WARNING 08-05 15:27:18 [envs.py:2120] Unknown vLLM environment variable detected: VLLM_MULTI_SERVE_SERVICE_PORT
(APIServer pid=467) WARNING 08-05 15:27:18 [envs.py:2120] Unknown vLLM environment variable detected: VLLM_MULTI_SERVE_PORT
(APIServer pid=467) WARNING 08-05 15:27:18 [envs.py:2120] Unknown vLLM environment variable detected: VLLM_MULTI_SERVE_PORT_8000_TCP_ADDR
(APIServer pid=467) WARNING 08-05 15:27:18 [envs.py:2120] Unknown vLLM environment variable detected: VLLM_MULTI_SERVE_PORT_8000_TCP_PORT
(APIServer pid=467) WARNING 08-05 15:27:18 [envs.py:2120] Unknown vLLM environment variable detected: VLLM_MULTI_SERVE_PORT_8000_TCP_PROTO
(APIServer pid=467) WARNING 08-05 15:27:18 [envs.py:2120] Unknown vLLM environment variable detected: VLLM_MULTI_SERVE_SERVICE_HOST
(APIServer pid=467) DEBUG 08-05 15:27:19 [model_executor/models/registry.py:990] Loaded model info for class vllm.models.inkling.InklingForConditionalGeneration from cache
(APIServer pid=467) DEBUG 08-05 15:27:19 [logging_utils/log_time.py:29] Registry inspect model class: Elapsed time 0.0039657 secs
(APIServer pid=467) INFO 08-05 15:27:19 [config/model.py:645] Resolved architecture: InklingForConditionalGeneration
(APIServer pid=467) INFO 08-05 15:27:19 [config/model.py:663] Defaulting to tokenizer_mode='inkling' for InklingForConditionalGeneration
(APIServer pid=467) INFO 08-05 15:27:19 [config/model.py:1883] Using max model len 200000
(APIServer pid=467) DEBUG 08-05 15:27:20 [utils/import_utils.py:73] Loading module triton_kernels from /usr/local/lib/python3.12/dist-packages/vllm/third_party/triton_kernels/__init__.py.
(APIServer pid=467) DEBUG 08-05 15:27:20 [utils/nccl.py:35] Found nccl from library libnccl.so.2
(APIServer pid=467) DEBUG 08-05 15:27:24 [tokenizers/registry.py:83] Loading CachedHfTokenizer for tokenizer_mode='inkling'
(APIServer pid=467) DEBUG 08-05 15:27:26 [compilation/decorators.py:221] Inferred dynamic dimensions for forward method of <class 'vllm.model_executor.models.qwen3_dflash.DFlashQwen3Model'>: ['input_ids', 'positions', 'input_embeds']
(APIServer pid=467) DEBUG 08-05 15:27:26 [compilation/decorators.py:221] Inferred dynamic dimensions for forward method of <class 'vllm.model_executor.models.deepseek_v2.DeepseekV2Model'>: ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds']
(APIServer pid=467) DEBUG 08-05 15:27:26 [compilation/decorators.py:221] Inferred dynamic dimensions for forward method of <class 'vllm.model_executor.models.deepseek_mtp.DeepSeekMTP'>: ['input_ids', 'positions', 'hidden_states', 'intermediate_tensors', 'inputs_embeds']
(APIServer pid=467) DEBUG 08-05 15:27:26 [config/model.py:1948] Generative models support chunked prefill.
(APIServer pid=467) DEBUG 08-05 15:27:26 [config/model.py:2003] Generative models support prefix caching.
(APIServer pid=467) WARNING 08-05 15:27:26 [utils/torch_utils.py:355] WARNING: Unknown kv_cache_quant_algo 'none' in model config. Supported values: ['fp8', 'nvfp4']. Falling back to 'auto'.
(APIServer pid=467) DEBUG 08-05 15:27:26 [config/parallel.py:953] Defaulting to use mp for distributed inference
(APIServer pid=467) DEBUG 08-05 15:27:26 [engine/arg_utils.py:2775] Defaulting max_num_batched_tokens to 8192 for OPENAI_API_SERVER usage context.
(APIServer pid=467) DEBUG 08-05 15:27:26 [engine/arg_utils.py:2785] Defaulting max_num_seqs to 1024 for OPENAI_API_SERVER usage context.
(APIServer pid=467) INFO 08-05 15:27:26 [config/scheduler.py:242] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=467) WARNING 08-05 15:27:26 [model_executor/.../quantization/modelopt.py:1019] Detected ModelOpt NVFP4 checkpoint (quant_algo=NVFP4). Please note that the format is experimental and could change in future.
(APIServer pid=467) INFO 08-05 15:27:26 [config/vllm.py:1231] Auto-enabling VLLM_USE_BREAKABLE_CUDAGRAPH=1. Set VLLM_USE_BREAKABLE_CUDAGRAPH=0 to opt out.
(APIServer pid=467) DEBUG 08-05 15:27:26 [config/kernel.py:288] Setting platform-specific IR op priority defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), user-defined: IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[])
(APIServer pid=467) INFO 08-05 15:27:26 [config/kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(APIServer pid=467) WARNING 08-05 15:27:26 [config/vllm.py:2267] Model Runner V2 does not yet support the thinking_token_budget request parameter. Set VLLM_USE_V2_MODEL_RUNNER=0 if this is required.
(APIServer pid=467) DEBUG 08-05 15:27:26 [config/vllm.py:1984] Max num batched tokens below allreduce-rms fusion threshold, allreduce-rms fusion will be enabled for all num_tokens.
(APIServer pid=467) INFO 08-05 15:27:26 [config/compilation.py:329] Enabled custom fusions: norm_quant, act_quant, allreduce_rms
(APIServer pid=467) DEBUG 08-05 15:27:26 [plugins/__init__.py:44] No plugins for group vllm.stat_logger_plugins found.
(APIServer pid=467) DEBUG 08-05 15:27:26 [renderers/registry.py:59] Loading InklingRenderer for renderer_mode='inkling'
(APIServer pid=467) DEBUG 08-05 15:27:26 [utils/torch_utils.py:167] OMP_NUM_THREADS is not set; defaulting Torch threads to 1.
DEBUG 08-05 15:27:28 [plugins/__init__.py:44] No plugins for group vllm.platform_plugins found.
DEBUG 08-05 15:27:28 [platforms/__init__.py:36] Checking if TPU platform is available.
DEBUG 08-05 15:27:28 [platforms/__init__.py:55] TPU platform is not available because: No module named 'libtpu'
DEBUG 08-05 15:27:28 [platforms/__init__.py:61] Checking if CUDA platform is available.
DEBUG 08-05 15:27:28 [platforms/__init__.py:84] Confirmed CUDA platform is available.
DEBUG 08-05 15:27:28 [platforms/__init__.py:112] Checking if ROCm platform is available.
DEBUG 08-05 15:27:28 [platforms/__init__.py:126] ROCm platform is not available because: No module named 'amdsmi'
DEBUG 08-05 15:27:28 [platforms/__init__.py:133] Checking if XPU platform is available.
DEBUG 08-05 15:27:28 [platforms/__init__.py:164] Checking if CPU platform is available.
DEBUG 08-05 15:27:28 [platforms/__init__.py:61] Checking if CUDA platform is available.
DEBUG 08-05 15:27:28 [platforms/__init__.py:84] Confirmed CUDA platform is available.
DEBUG 08-05 15:27:28 [platforms/__init__.py:245] Automatically detected platform cuda.
DEBUG 08-05 15:27:33 [utils/import_utils.py:73] Loading module triton_kernels from /usr/local/lib/python3.12/dist-packages/vllm/third_party/triton_kernels/__init__.py.
DEBUG 08-05 15:27:33 [utils/nccl.py:35] Found nccl from library libnccl.so.2
(EngineCore pid=770) DEBUG 08-05 15:27:34 [v1/engine/core.py:1252] Waiting for init message from front-end.
(APIServer pid=467) DEBUG 08-05 15:27:34 [v1/engine/utils.py:1341] HELLO from local core engine process 0.
(EngineCore pid=770) DEBUG 08-05 15:27:34 [v1/engine/core.py:1263] Received init message: EngineHandshakeMetadata(addresses=EngineZmqAddresses(inputs=['ipc:///tmp/424a4421-bd29-43eb-808a-88172bd80c8c'], outputs=['ipc:///tmp/aed3b494-d4ca-45d4-99d0-bea12685b322'], coordinator_input=None, coordinator_output=None, frontend_stats_publish_address=None), parallel_config={})
(EngineCore pid=770) DEBUG 08-05 15:27:34 [v1/engine/core.py:1056] Has DP Coordinator: False, stats publish address: None
(EngineCore pid=770) DEBUG 08-05 15:27:34 [plugins/__init__.py:52] Available plugins for group vllm.general_plugins:
(EngineCore pid=770) DEBUG 08-05 15:27:34 [plugins/__init__.py:54] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
(EngineCore pid=770) DEBUG 08-05 15:27:34 [plugins/__init__.py:54] - lora_hf_hub_resolver -> vllm.plugins.lora_resolvers.hf_hub_resolver:register_hf_hub_resolver
(EngineCore pid=770) DEBUG 08-05 15:27:34 [plugins/__init__.py:57] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
(EngineCore pid=770) INFO 08-05 15:27:34 [v1/engine/core.py:121] Initializing a V1 LLM engine (v0.26.1rc1.dev306+gcb8104839) with config: model='thinkingmachines/Inkling-Small-NVFP4', speculative_config=None, tokenizer='thinkingmachines/Inkling-Small-NVFP4', skip_tokenizer_init=False, tokenizer_mode=inkling, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=200000, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=modelopt_fp4, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='inkling', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=thinkingmachines/Inkling-Small, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
(EngineCore pid=770) WARNING 08-05 15:27:34 [v1/executor/multiproc_executor.py:1081] Reducing Torch parallelism from 128 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=770) INFO 08-05 15:27:34 [v1/executor/multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=10.130.1.190 (local), world_size=2, local_world_size=2
(EngineCore pid=770) DEBUG 08-05 15:27:34 [distributed/device_communicators/shm_broadcast.py:502] Binding to ipc:///tmp/0649717d-01d8-4049-a940-23b56a37ad2d
(EngineCore pid=770) DEBUG 08-05 15:27:34 [distributed/device_communicators/shm_broadcast.py:555] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1], buffer_handle=(2, 16777216, 10, 'psm_f81cdfac'), local_subscribe_addr='ipc:///tmp/0649717d-01d8-4049-a940-23b56a37ad2d', local_notify_addr='ipc:///tmp/39a78bbb-a8f0-4f9d-bc1a-0f6d70ad66ea', remote_subscribe_addr=None, remote_addr_ipv6=False)
DEBUG 08-05 15:27:35 [plugins/__init__.py:44] No plugins for group vllm.platform_plugins found.
DEBUG 08-05 15:27:35 [platforms/__init__.py:36] Checking if TPU platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:55] TPU platform is not available because: No module named 'libtpu'
DEBUG 08-05 15:27:35 [platforms/__init__.py:61] Checking if CUDA platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:84] Confirmed CUDA platform is available.
DEBUG 08-05 15:27:35 [plugins/__init__.py:44] No plugins for group vllm.platform_plugins found.
DEBUG 08-05 15:27:35 [platforms/__init__.py:36] Checking if TPU platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:55] TPU platform is not available because: No module named 'libtpu'
DEBUG 08-05 15:27:35 [platforms/__init__.py:61] Checking if CUDA platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:84] Confirmed CUDA platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:112] Checking if ROCm platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:126] ROCm platform is not available because: No module named 'amdsmi'
DEBUG 08-05 15:27:35 [platforms/__init__.py:133] Checking if XPU platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:164] Checking if CPU platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:61] Checking if CUDA platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:84] Confirmed CUDA platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:112] Checking if ROCm platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:126] ROCm platform is not available because: No module named 'amdsmi'
DEBUG 08-05 15:27:35 [platforms/__init__.py:133] Checking if XPU platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:164] Checking if CPU platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:61] Checking if CUDA platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:84] Confirmed CUDA platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:245] Automatically detected platform cuda.
DEBUG 08-05 15:27:35 [platforms/__init__.py:245] Automatically detected platform cuda.
DEBUG 08-05 15:27:40 [utils/import_utils.py:73] Loading module triton_kernels from /usr/local/lib/python3.12/dist-packages/vllm/third_party/triton_kernels/__init__.py.
DEBUG 08-05 15:27:40 [utils/import_utils.py:73] Loading module triton_kernels from /usr/local/lib/python3.12/dist-packages/vllm/third_party/triton_kernels/__init__.py.
DEBUG 08-05 15:27:40 [utils/nccl.py:35] Found nccl from library libnccl.so.2
DEBUG 08-05 15:27:40 [utils/nccl.py:35] Found nccl from library libnccl.so.2
DEBUG 08-05 15:27:41 [plugins/__init__.py:52] Available plugins for group vllm.general_plugins:
DEBUG 08-05 15:27:41 [plugins/__init__.py:54] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
DEBUG 08-05 15:27:41 [plugins/__init__.py:54] - lora_hf_hub_resolver -> vllm.plugins.lora_resolvers.hf_hub_resolver:register_hf_hub_resolver
DEBUG 08-05 15:27:41 [plugins/__init__.py:57] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
DEBUG 08-05 15:27:42 [plugins/__init__.py:52] Available plugins for group vllm.general_plugins:
DEBUG 08-05 15:27:42 [plugins/__init__.py:54] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
DEBUG 08-05 15:27:42 [plugins/__init__.py:54] - lora_hf_hub_resolver -> vllm.plugins.lora_resolvers.hf_hub_resolver:register_hf_hub_resolver
DEBUG 08-05 15:27:42 [plugins/__init__.py:57] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
DEBUG 08-05 15:27:42 [tokenizers/registry.py:83] Loading CachedHfTokenizer for tokenizer_mode='inkling'
DEBUG 08-05 15:27:42 [tokenizers/registry.py:83] Loading CachedHfTokenizer for tokenizer_mode='inkling'
DEBUG 08-05 15:27:44 [config/kernel.py:85] Setting IR op priority for rms_norm to ['vllm_c', 'native']
DEBUG 08-05 15:27:44 [ir/op.py:422] Priority for vllm.ir.rms_norm set to ['vllm_c', 'native']
DEBUG 08-05 15:27:44 [config/kernel.py:85] Setting IR op priority for fused_add_rms_norm to ['vllm_c', 'native']
DEBUG 08-05 15:27:44 [ir/op.py:422] Priority for vllm.ir.fused_add_rms_norm set to ['vllm_c', 'native']
(Worker pid=916) DEBUG 08-05 15:27:44 [distributed/parallel_state.py:1596] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:33065 backend=nccl
(Worker pid=916) INFO 08-05 15:27:44 [distributed/parallel_state.py:1640] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:33065 backend=nccl
DEBUG 08-05 15:27:44 [config/kernel.py:85] Setting IR op priority for rms_norm to ['vllm_c', 'native']
DEBUG 08-05 15:27:44 [ir/op.py:422] Priority for vllm.ir.rms_norm set to ['vllm_c', 'native']
DEBUG 08-05 15:27:44 [config/kernel.py:85] Setting IR op priority for fused_add_rms_norm to ['vllm_c', 'native']
DEBUG 08-05 15:27:44 [ir/op.py:422] Priority for vllm.ir.fused_add_rms_norm set to ['vllm_c', 'native']
(Worker pid=917) DEBUG 08-05 15:27:44 [distributed/parallel_state.py:1596] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:33065 backend=nccl
(Worker pid=917) INFO 08-05 15:27:44 [distributed/parallel_state.py:1640] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:33065 backend=nccl
(Worker pid=917) DEBUG 08-05 15:27:44 [distributed/parallel_state.py:1722] Detected 1 nodes in the distributed environment
(Worker pid=916) DEBUG 08-05 15:27:44 [distributed/parallel_state.py:1722] Detected 1 nodes in the distributed environment
(Worker pid=916) INFO 08-05 15:27:44 [distributed/device_communicators/pynccl.py:113] vLLM is using nccl==2.30.7
(APIServer pid=467) DEBUG 08-05 15:27:44 [v1/engine/utils.py:1243] Waiting for 1 local, 0 remote core engine proc(s) to start.
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclOsDlopen(libnccl-env.so) failed: libnccl-env.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Bootstrap: Using eth0:10.130.1.190<0>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:1117 [0] NCCL INFO Socket handshake magic 0x564ab9f2fc4b9d6c (built-in default)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO cudaDriverVersion 13000
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NCCL version 2.30.7+cuda13.3
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NCCL git version HEAD a5187fa2e
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclOsDlopen(libnccl-net.so) failed: libnccl-net.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NET/IB : No device found.
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NET/IB : Using [RO]; OOB eth0:10.130.1.190<0>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Failed to initialize NET plugin IB
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NET/Socket : Using [0]eth0:10.130.1.190<0>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Initialized NET plugin Socket
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Assigned NET plugin Socket to comm
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclOsDlopen(libnccl-rma.so) failed: libnccl-rma.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO RMA/Plugin: Could not find: libnccl-rma.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO RMA/Plugin: Failed to load external plugin libnccl-rma.so, dlHandle: (nil), ncclRma: (nil)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO RMA/Plugin: Failed to initialize any plugin
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclOsDlopen(libnccl-gin.so) failed: libnccl-gin.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO GIN/Plugin: Failed to load external plugin libnccl-gin.so, dlHandle: (nil), ncclGin: (nil)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO GIN/Plugin: Failed to initialize any GIN plugin
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Using network Socket
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Loaded NVML from libnvidia-ml.so.1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO [Rank 0] ncclCommInitRank comm 0x17245930 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId a1000 commId 0xb01e43a8871f0913 - Init START
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO RAS client listening socket at ::1<28028>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Bootstrap timings total 0.001047 (create 0.000038, send 0.000103, recv 0.000266, ring 0.000139, delay 0.000001)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978zthinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclOsDlopen(libnccl-env.so) failed: libnccl-env.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO cudaDriverVersion 13000
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Bootstrap: Using eth0:10.130.1.190<0>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NCCL version 2.30.7+cuda13.3
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NCCL git version HEAD a5187fa2e
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclOsDlopen(libnccl-net.so) failed: libnccl-net.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NET/IB : No device found.
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NET/IB : Using [RO]; OOB eth0:10.130.1.190<0>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Failed to initialize NET plugin IB
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NET/Socket : Using [0]eth0:10.130.1.190<0>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Initialized NET plugin Socket
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Assigned NET plugin Socket to comm
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclOsDlopen(libnccl-rma.so) failed: libnccl-rma.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO RMA/Plugin: Could not find: libnccl-rma.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO RMA/Plugin: Failed to load external plugin libnccl-rma.so, dlHandle: (nil), ncclRma: (nil)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO RMA/Plugin: Failed to initialize any plugin
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclOsDlopen(libnccl-gin.so) failed: libnccl-gin.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO GIN/Plugin: Failed to load external plugin libnccl-gin.so, dlHandle: (nil), ncclGin: (nil)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO GIN/Plugin: Failed to initialize any GIN plugin
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Using network Socket
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Loaded NVML from libnvidia-ml.so.1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO [Rank 1] ncclCommInitRank comm 0x1961ec10 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId d1000 commId 0xb01e43a8871f0913 - Init START
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Socket handshake magic 0x564ab9f2fc4b9d6c (built-in default)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO RAS client listening socket at ::1<28028>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Bootstrap timings total 0.142605 (create 0.000039, send 0.000088, recv 0.141837, ring 0.000030, delay 0.000000)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z55sg5:916:916 [0] NCCL INFO NCCL_P2P_DISABLE set by environment to 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1.
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 0 is 64-127,192-255. (GPU affinity = 64-127,192-255).
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NVLS multicast support is not available on dev 0
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Rank 0: 1 Net devices
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Rank 0: 0 CollNet devices
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Local Net device counts across ranks: min 1 max 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Local CollNet device counts across ranks: min 0 max 0
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO comm 0x17245930 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 00/04 : 0 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 01/04 : 0 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 02/04 : 0 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 03/04 : 0 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 [2] 1/-1/-1->0->-1 [3] -1/-1/-1->0->1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO P2P Chunksize set to 131072
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclOsDlopen(libnccl-profiler.so) failed: libnccl-profiler.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0 isAllCudaP2p 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:1121 [0] NCCL INFO [Proxy Service] Device 0 CPU core 220
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:1121 [0] NCCL INFO proxy listening socket at 10.130.1.190<39107>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:1122 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 76
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclOsDlopen(libnccl-tuner.so) failed: libnccl-tuner.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO CC Off, workFifoBytes 1048576
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclCommInitRank comm 0x17245930 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId a1000 commId 0xb01e43a8871f0913 - Init COMPLETE
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Init timings - ncclCommInitRank: rank 0 nranks 2 total 0.10 (kernels 0.09, alloc 0.00, bootstrap 0.00, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Symmetric VA size=14(Worker pid=917) WARNING 08-05 15:27:44 [distributed/device_communicators/symm_mem.py:107] SymmMemCommunicator: symmetric memory multicast operations are not supported.
(Worker pid=916) WARNING 08-05 15:27:44 [distributed/device_communicators/symm_mem.py:107] SymmMemCommunicator: symmetric memory multicast operations are not supported.
(Worker pid=916) INFO 08-05 15:27:44 [distributed/device_communicators/cuda_communicator.py:266] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=916) DEBUG 08-05 15:27:44 [distributed/device_communicators/shm_broadcast.py:502] Binding to ipc:///tmp/6236dad2-1642-4143-aa62-b7dbbf82befb
(Worker pid=916) DEBUG 08-05 15:27:44 [distributed/device_communicators/shm_broadcast.py:555] vLLM message queue communication handle: Handle(local_reader_ranks=[1], buffer_handle=(1, 4194304, 6, 'psm_172b091f'), local_subscribe_addr='ipc:///tmp/6236dad2-1642-4143-aa62-b7dbbf82befb', local_notify_addr='ipc:///tmp/d897c86c-6fa8-49f5-afe2-44f86787e380', remote_subscribe_addr=None, remote_addr_ipv6=False)
(Worker pid=917) DEBUG 08-05 15:27:44 [distributed/device_communicators/shm_broadcast.py:579] Connecting to ipc:///tmp/6236dad2-1642-4143-aa62-b7dbbf82befb
sg5:917:917 [1] NCCL INFO NCCL_P2P_DISABLE set by environment to 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1.
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 1 is 64-127,192-255. (GPU affinity = 64-127,192-255).
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NVLS multicast support is not available on dev 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Rank 1: 1 Net devices
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Rank 1: 0 CollNet devices
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO comm 0x1961ec10 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 [2] -1/-1/-1->1->0 [3] 0/-1/-1->1->-1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO P2P Chunksize set to 131072
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclOsDlopen(libnccl-profiler.so) failed: libnccl-profiler.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0 isAllCudaP2p 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:1124 [0] NCCL INFO [Proxy Service UDS] Device 1 CPU core 220
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:1123 [0] NCCL INFO [Proxy Service] Device 1 CPU core 115
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:1123 [1] NCCL INFO proxy listening socket at 10.130.1.190<39423>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclOsDlopen(libnccl-tuner.so) failed: libnccl-tuner.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclCommInitRank comm 0x1961ec10 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId d1000 commId 0xb01e43a8871f0913 - Init COMPLETE
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Init timings - ncclCommInitRank: rank 1 nranks 2 total 0.25 (kernels 0.09, alloc 0.00, bootstrap 0.14, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Symmetric VA size=140GB
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Channel 00 : 1[1] -> 0[0] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Channel 01 : 1[1] -> 0[0] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Channel 02 : 1[1] -> 0[0] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Channel 03 : 1[1] -> 0[0] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Initialized NET plugin Socket
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z50GB
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 00 : 0[0] -> 1[1] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 01 : 0[0] -> 1[1] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 02 : 0[0] -> 1[1] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 03 : 0[0] -> 1[1] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Initialized NET plugin Socket
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Assigned NET plugin Socket to comm
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO RMA/Plugin: Failed to initialize any plugin
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO GIN/Plugin: Failed to initialize any GIN plugin
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Using network Socket
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO [Rank 0] ncclCommInitRank comm 0x1f658970 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId a1000 commId 0x5c3ff7f5395fd00c - Init START
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Bootstrap timings total 0.000365 (create 0.000023, send 0.000070, recv 0.000145, ring 0.000006, delay 0.000000)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 0 is 64-127,192-255. (GPU affinity = 64-127,192-255).
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NVLS multicast support is not available on dev 0
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Rank 0: 1 Net devices
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Rank 0: 0 CollNet devices
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Local Net device counts across ranks: min 1 max 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Local CollNet device counts across ranks: min 0 max 0
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO comm 0x1f658970 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 00/04 : 0 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 01/04 : 0 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 02/04 : 0 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 03/04 : 0 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 [2] 1/-1/-1->0->-1 [3] -1/-1/-1->0->1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO P2P Chunksize set to 131072
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0 isAllCudaP2p 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:1180 [0] NCCL INFO [Proxy Service] Device 0 CPU core 230
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:1180 [0] NCCL INFO proxy listening socket at 10.130.1.190<38975>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:1183 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 201
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO 4 coll(Worker pid=916) INFO 08-05 15:27:45 [distributed/device_communicators/cuda_communicator.py:266] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=916) INFO 08-05 15:27:45 [distributed/parallel_state.py:1977] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=916) INFO 08-05 15:27:45 [v1/worker/gpu_worker.py:385] Using V2 Model Runner
(Worker pid=916) DEBUG 08-05 15:27:45 [v1/worker/gpu_worker.py:397] worker init memory snapshot: torch_peak=0.0GiB, torch_allocated=0.0GiB, free_memory=138.73GiB, total_memory=139.8GiB, cuda_memory=1.07GiB, torch_memory=0.0GiB, non_torch_memory=1.07GiB, timestamp=1785943665.4141102, auto_measure=True
(Worker pid=916) DEBUG 08-05 15:27:45 [v1/worker/gpu_worker.py:398] worker requested memory: 118.83GiB
(Worker pid=917) DEBUG 08-05 15:27:45 [v1/worker/gpu_worker.py:397] worker init memory snapshot: torch_peak=0.0GiB, torch_allocated=0.0GiB, free_memory=138.73GiB, total_memory=139.8GiB, cuda_memory=1.07GiB, torch_memory=0.0GiB, non_torch_memory=1.07GiB, timestamp=1785943665.4269004, auto_measure=True
(Worker pid=917) DEBUG 08-05 15:27:45 [v1/worker/gpu_worker.py:398] worker requested memory: 118.83GiB
(Worker_TP0 pid=916) INFO 08-05 15:27:45 [v1/worker/gpu/model_runner.py:326] Loading model from scratch...
(Worker_TP0 pid=916) DEBUG 08-05 15:27:46 [model_executor/.../fused_moe/layer.py:68] FusedMoEParallelConfig = FusedMoEParallelConfig(tp_size=2, pcp_size=1, dp_size=1, ep_size=1, tp_rank=0, pcp_rank=0, dp_rank=0, ep_rank=0, sp_size=1, use_ep=False, all2all_backend='allgather_reducescatter', enable_eplb=False)
(Worker_TP0 pid=916) DEBUG 08-05 15:27:46 [compilation/decorators.py:221] Inferred dynamic dimensions for forward method of <class 'vllm.model_executor.models.commandr.CohereModel'>: ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds']
(Worker_TP1 pid=917) DEBUG 08-05 15:27:46 [model_executor/.../fused_moe/layer.py:68] FusedMoEParallelConfig = FusedMoEParallelConfig(tp_size=2, pcp_size=1, dp_size=1, ep_size=1, tp_rank=1, pcp_rank=0, dp_rank=0, ep_rank=0, sp_size=1, use_ep=False, all2all_backend='allgather_reducescatter', enable_eplb=False)
...
(Worker_TP0 pid=916) DEBUG 08-05 15:27:49 [model_executor/models/utils.py:282] Loaded weight model.layers.7.attn.k_norm.weight with shape torch.Size([128])
(Worker_TP0 pid=916) DEBUG 08-05 15:27:49 [model_executor/models/utils.py:282] Loaded weight model.layers.7.attn.k_sconv.weight with shape torch.Size([512, 1, 4])
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.39.attn.k_sconv.weight with shape torch.Size([512, 1, 4])
(Worker_TP0 pid=916) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.7.mlp.gate.global_scale with shape torch.Size([1])
(Worker_TP0 pid=916) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.9.attn_sconv.weight with shape torch.Size([2048, 1, 4])
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.39.mlp.gate.bias with shape torch.Size([256])
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.4.attn.k_norm.weight with shape torch.Size([128])
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.40.attn.v_sconv.weight with shape torch.Size([512, 1, 4])
 channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO CC Off, workFifoBytes 1048576
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclCommInitRank comm 0x1f658970 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId a1000 commId 0x5c3ff7f5395fd00c - Init COMPLETE
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Init timings - ncclCommInitRank: rank 0 nranks 2 total 0.01 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.00, topo 0.00, graphs 0.00, connections 0.00, rest 0.00)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Symmetric VA size=140GB
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 00 : 0[0] -> 1[1] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 01 : 0[0] -> 1[1] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 02 : 0[0] -> 1[1] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 03 : 0[0] -> 1[1] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
[rank0]:[W805 15:27:50.651590942 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.652804488 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.653856615 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.654884212 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.655913239 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.657039706 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.658054933 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.659086670 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.660081537 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.661083304 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.662108681 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.663127598 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.664148955 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.665179332 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.666200629 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.667217606 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/layers/linear.py:1012] Loaded shard 1 with shape torch.Size([1024, 4096]) into model.layers.6.attn.qkvr.weight
[rank0]:[W805 15:27:50.668220333 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.669208870 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.670221587 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.671221284 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.6.attn.wo_ud.weight with shape torch.Size([4096, 2048])
[rank0]:[W805 15:27:50.672235131 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.6.mlp_norm.weight with shape torch.Size([4096])
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.7.attn.k_norm.weight with shape torch.Size([128])
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.7.attn.k_sconv.weight with shape torch.Size([512, 1, 4])
[rank0]:[W805 15:27:50.673279748 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.674318785 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
(Worker_TP0 pid=916) 
Loading safetensors checkpoint shards:  20% Completed | 2/10 [00:03<00:13,  1.64s/it]
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.7.mlp.gate.global_scale with shape torch.Size([1])
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.9.attn_sconv.weight with shape torch.Size([2048, 1, 4])
(Worker_TP0 pid=916) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.1.attn.v_sconv.weight with shape torch.Size([512, 1, 4])
(Worker_TP0 pid=916) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.1.mlp.global_scale with shape torch.Size([1])
sg5:917:917 [1] NCCL INFO Assigned NET plugin Socket to comm
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO RMA/Plugin: Failed to initialize any plugin
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO GIN/Plugin: Failed to initialize any GIN plugin
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Using network Socket
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO [Rank 1] ncclCommInitRank comm 0x1694f800 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId d1000 commId 0x5c3ff7f5395fd00c - Init START
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Bootstrap timings total 0.000363 (create 0.000026, send 0.000066, recv 0.000187, ring 0.000007, delay 0.000000)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 1 is 64-127,192-255. (GPU affinity = 64-127,192-255).
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NVLS multicast support is not available on dev 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Rank 1: 1 Net devices
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Rank 1: 0 CollNet devices
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO comm 0x1694f800 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 [2] -1/-1/-1->1->0 [3] 0/-1/-1->1->-1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO P2P Chunksize set to 131072
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0 isAllCudaP2p 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:1182 [0] NCCL INFO [Proxy Service UDS] Device 1 CPU core 94
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:1181 [0] NCCL INFO [Proxy Service] Device 1 CPU core 230
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:1181 [1] NCCL INFO proxy listening socket at 10.130.1.190<46167>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclCommInitRank comm 0x1694f800 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId d1000 commId 0x5c3ff7f5395fd00c - Init COMPLETE
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Init timings - ncclCommInitRank: rank 1 nranks 2 total 0.01 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.00, topo 0.00, graphs 0.00, connections 0.00, rest 0.00)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Symmetric VA size=140GB
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Channel 00 : 1[1] -> 0[0] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Channel 01 : 1[1] -> 0[0] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Channel 02 : 1[1] -> 0[0] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Channel 03 : 1[1] -> 0[0] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
[rank1]:[W805 15:27:50.850742647 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 1 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank1]:[W805 15:27:50.851908024 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 1 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank1]:[W805 15:27:50.852918241 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 1 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
(Worker_TP0 pid=916) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.11.attn_norm.weight with shape torch.Size([4096])
[rank1]:[W805 15:27:50.853893978 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 1 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
(Worker_TP0 pid=916) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.12.attn.q_norm.weight with shape torch.Size([128])
(Worker_TP0 pid=916) DEBUG 08-05 15:27:50 [model_executor/layers/linear.py:1012] Loaded shard 3 with shape torch.Size([512, 4096]) into model.layers.12.attn.qkvr.weight
[rank1]:[W805 15:27:50.854866665 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 1 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.855542863 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank1]:[W805 15:27:50.855957562 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 1 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).

aurick

Thinking Machines Lab org 1 day ago

This BF16 checkpoint requires 600GB of total VRAM. With 2x H200 GPUs, you may want to try the NVFP4 version.

aurick changed discussion status to closed 1 day ago

jtvino

1 day ago

@aurick I did use vllm serve thinkingmachines/Inkling-Small-NVFP4. Do you want me to repost inside the other model page?

aurick

Thinking Machines Lab org 1 day ago

Oh I see, I have seen this issue before with vLLM and the NVFP4 model. Could you try enabling pytorch expandable segments? If that doesn't work, perhaps the vLLM team can help if you post an issue on the vLLM Github.

aurick changed discussion status to open 1 day ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment